Train cedrpacrr using 5 folds BERT checkpoint, train_pairs and valid_run

[Pourbahman]

Hi Sean,

I want to train cedrpacrr using BERT checkpoint by the following command:

python train.py \
  --model cedr_pacrr \ # or cedr_knrm / cedr_drmm
  --datafiles data/queries.tsv data/documents.tsv \
  --qrels data/qrels \
  --train_pairs data/train_pairs \
  --valid_run data/valid_run \
  --initial_bert_weights models/vbert/weights.p \
  --model_out_dir models/cedrpacrr

As you know in your data directory train and validation data are in 5 folds. Also, weight of BERT check point has 5 folds.

Would you please guide me what I should do with train_pair, valid_run and weights?

Thanks in advance,
Kind Regards,
Zahra

[yiyaxiaozhi]

I used the following command to train a CEDR-KNRM model with fold 1 data on Robust 04 dataset:

python train.py   --model cedr_knrm  
--datafiles ../data/robust/queries.tsv ../data/robust/documents.tsv
--qrels ../data/robust/qrels   
--train_pairs ../data/robust/f1.train.pairs   
--valid_run ../data/robust/f1.valid.run   
--model_out_dir models/vbert1/cedr_knrm
--initial_bert_weights /models/cedr-models/vbert-robust-f1.p

where train_pairs value was the path of fold 1 training data and inital_bert_weights should also match the trained Vanilla_bert with fold 1 data.

[Pourbahman]

Thank you!

1. So, you evaluated average of values of each metric for 5 folds manually, am I right?

2. Would you please tell me whether you freezed the BERT layer when you trained the model? In other words, you make trainable of the BERT layer False? If yes, how did you do?

3. Also, would you please tell me whether you got the result of the paper? If yes, did you change any parts or any configs of parameters of the implementation on the repository?

4. Also, would you please tell me what was the configuration of the hardware you trained the model on it?

Thanks in advance,
Kind Regards

[seanmacavaney]

Hi @Pourbahman,

I recommend using a package like OpenNIR or Capreolus. This repository was meant to be as a simplification/demonstration of the main idea, rather than a comprehensive system for doing these types of experiments / tuning / etc. The original experiments in the paper were conducted on a precursor to OpenNIR.

To answer your questions:

• Yes, five models were trained and evaluated in a cross-validation setup and the performance reported in the paper is the average across the folds (as is common practice for Robust04). The trained models used in the paper can be found here.
• I don't have the exact hardware configuration (it was at my prior institution), but I can say it was trained on a 1080ti GPU, with 32-64GB RAM and an AMD CPU. If further details are needed, let me know which ones in particular are needed and I could contact somebody there.
• The weights of the model were not frozen. If you want to experiment with this on OpenNIR, you can set vocab.train=False.

[Pourbahman]

Hi Sean,

Thanks for your complete answer :)

Kind Regards,
Zahra