Reproduction of the experimental results with Waterbirds

[beotborry]

Hi, there. First of all, thank you for your wonderful work.

I'm struggling to reproduce the results with Waterbirds which you reported on the paper now.

So, I just want to ask about the hyperparameter settings you used for Waterbirds such as batch size, lr, etc.

Thank you,
Juhyeon

[aliasgharkhani]

Hi,

Thanks for your interest in our paper. You can use this bash file. All the specific hyperparameters used for this dataset are set in this bash file and others that are not set in this file, are the default values that we have used in the arguments.py file

Thanks

[beotborry]

Thanks for your response.
Actually, when I trained the model based on your bash file, its ERM accuracy is too low.

I used the following hyperparameter.

Namespace(dataset='waterbirds', 
workers=4, 
train=True, 
resume=False, 
test_batch=128, arch='resnet50', base_dir='/home/juhyeon/Masktune', dataset_dir='/home/juhyeon/Masktune/datasets/Waterbirds/', 
saved_mask_dir=None, 
best_erm_model_checkpoint_path=None,
last_erm_model_checkpoint_path=None, 
finetuned_model_checkpoint_path=None, 
use_cuda=True, 
gpu_ids=[0], 
optimizer='sgd',
lr=0.0001, 
schedule=[25, 50, 75, 100, 125, 150, 175, 200, 225, 250, 275], 
gamma=1.0, 
momentum=0.9, 
weight_decay=0.0001, 
lr_scheduler_name='multi_step', 
use_nesterov=False, 
masktune=True, 
epochs=100, 
train_batch=128, 
selective_classification=False, 
use_pretrained_weights=True, 
test_data_types=['biased'], 
masking_batch_size=128, 
masktune_iterations=1, 
accumulative_masking=False, 
use_random_masking=False, 
square_number=1, 
bias_type='square', 
train_bias_conflicting_data_ratio=0.2, 
test_bias_conflicting_data_ratio=1.0,
coverage=[80, 85, 90, 95, 100])

However, its ERM accuracy is as follows.

----------testing the model----------
test epoch: 0
accuracy of group 1: 0.952549889135255
accuracy of group 2: 0.2719449225473322
accuracy of group 3: 0.9545053003533569
accuracy of group 4: 0.2538593481989708
average accuracy: 0.8120710892134436

So, if possible, could you check if there's something wrong in my hyperparameters?

Thanks,
Juhyeon

[beotborry]

I add the experimental results of CelebA FYI.

Namespace(dataset='celeba', 
workers=4, 
train=True, 
resume=False, 
test_batch=128, 
arch='resnet50', 
base_dir='/home/juhyeon/Masktune', 
dataset_dir='/home/juhyeon/Masktune/datasets/CelebA', 
saved_mask_dir=None, 
best_erm_model_checkpoint_path=None, 
last_erm_model_checkpoint_path=None, 
finetuned_model_checkpoint_path=None, 
use_cuda=True, gpu_ids=[0], optimizer='sgd', 
lr=0.0001, schedule=[25, 50, 75, 100, 125, 150, 175, 200, 225, 250, 275], gamma=1.0, 
momentum=0.9, weight_decay=0.0001, lr_scheduler_name='multi_step', 
use_nesterov=False, masktune=True, epochs=20, 
train_batch=128, selective_classification=False, use_pretrained_weights=True, 
test_data_types=['biased'], masking_batch_size=128, masktune_iterations=1, 
accumulative_masking=False, use_random_masking=False, square_number=1, 
bias_type='square', train_bias_conflicting_data_ratio=0.2, 
test_bias_conflicting_data_ratio=1.0, coverage=[80, 85, 90, 95, 100])

----------testing the model----------
test epoch: 0
accuracy of group 1: 0.9612982492065117
accuracy of group 2: 0.7600806451612904
accuracy of group 3: 0.9964167219641672
accuracy of group 4: 0.18333333333333332
average accuracy: 0.9425408275723876
----------masking 162770 data----------
----------finetuning the erm model for one epoch----------
train epoch: 0
loss: 0.17620396162819538
accuracy: 0.925564036374444
val epoch: 0
loss: 0.17255067643806282
accuracy: 0.93122885505698
lr: 0.0001
----------testing the model----------
test epoch: 0
accuracy of group 1: 0.9386710351182553
accuracy of group 2: 0.839516129032258
accuracy of group 3: 0.9920371599203716
accuracy of group 4: 0.24444444444444444
average accuracy: 0.9402364492535819

[Yawen-Zhao-820]

Hi, there. Thank you for the wonderful work of MaskTune.

I'm struggling to reproduce the results with Waterbirds.

I used the corrected Waterbirds dataset and the bash file, but got the results below:

----------testing the model----------
test epoch 0: 100%|███████████████████████████████████████████████████████| 45/45 [00:05<00:00, 8.43it/s, loss=0.521, accuracy=0.784]
accuracy of group 1: 0.9991130820399113
accuracy of group 2: 0.0
accuracy of group 3: 1.0
accuracy of group 4: 0.0
average accuracy: 0.7948266760513814
----------masking 4756 data----------
100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 38/38 [02:18<00:00, 3.65s/it]
Masking Time: 138.51200103759766
---------- indexing train data ----------
----------finetuning the erm model for one epoch----------
train epoch 0: 100%|██████████████████████████████████████████████████████| 38/38 [00:10<00:00, 3.60it/s, loss=0.275, accuracy=0.891]
val epoch 0: 100%|████████████████████████████████████████████████████████| 10/10 [00:01<00:00, 5.27it/s, loss=0.828, accuracy=0.593]
lr: 0.0001
----------testing the model----------
test epoch 0: 100%|████████████████████████████████████████████████████████| 45/45 [00:05<00:00, 8.54it/s, loss=0.85, accuracy=0.598]
accuracy of group 1: 0.6802660753880266
accuracy of group 2: 0.3029259896729776
accuracy of group 3: 0.6704946996466431
accuracy of group 4: 0.3361921097770154
average accuracy: 0.6024986802745029
.

Could you please give me some advice on reproducing the results for Waterbirds dataset?

Thank you!!!