Accuracy on TUAB Lower Than Reported Despite Following Setup

[ChenHu-ML]

Hi,

First of all, thank you for your outstanding work—your research and codebase have been incredibly insightful and helpful.

I’m currently trying to reproduce the reported results on TUAB using the provided code. However, I’m seeing much lower accuracy than expected.

---

What I Did

1. Downloaded and preprocessed the TUAB dataset using your official preprocessing pipeline.
2. Ran the training script finetune_main.py using the same parameters described in the paper. Here is the exact command I used (for TUAB):

python /CBraMod/finetune_main.py \
  --seed 3407 \
  --cuda 0 \
  --epochs 50 \
  --batch_size 64 \
  --lr 1e-4 \
  --weight_decay 5e-2 \
  --optimizer AdamW \
  --clip_value 1 \
  --dropout 0.1 \
  --downstream_dataset TUAB\
  --datasets_dir /data/EEGdata/TUAB/edf/process_cbramod \
  --num_of_classes 1 \
  --model_dir ./models_weights/ \
  --num_workers 16 \
  --label_smoothing 0.1 \
  --frozen False \
  --use_pretrained_weights True \
  --foundation_dir pretrained_weights/pretrained_weights.pth

---

• I got result as below: ~epoch1_acc_0.77637_pr_0.86583_roc_0.86059
  These are significantly lower than the results reported in your paper. I confirmed that the number of samples is consistent with those reported in the paper. Any insights into what I might be missing would be greatly appreciated—thanks in advance for your support!

[ChenHu-ML]

I’m currently attempting to reproduce the TUEV results using the provided preprocessing code, but I noticed that the number of preprocessed samples does not match the figures reported in the paper. Specifically, after preprocessing, I obtained the following counts: Train: 67,850; Validation: 16,082; Test: 29,421; Total: 113,353. I believe these counts are inconsistent with those reported in the paper. I followed the same preprocessing pipeline, used the recommended parameters and seed, verified that the dataset splits (train/val/test) were generated as instructed, and checked for potential duplication or missing samples, but found none. Any insights into what I might be missing would be greatly appreciated—thanks in advance for your support!

[wjq-learning]

Hi,

Thank you for your kind words and for reaching out with your question!

We appreciate your efforts in reproducing the results on TUAB, and we're glad to hear that you've followed the preprocessing and training procedures closely.

We wanted to let you know that the codebase has been updated since the initial release, and several improvements have been made to enhance performance and usability — especially for fine-tuning tasks. One notable addition is the multi_lr flag, which enables separate learning rates for the pretrained backbone and the classification head. This can lead to significantly better performance on downstream tasks like TUAB.

We recommend setting:

--multi_lr True

and experimenting with slightly higher learning rates for the classifier head if needed.

Please ensure that you are using the latest version of the code from the repository, as it includes these and other important updates.

Feel free to reach out if you'd like more suggestions or run into any issues — we're happy to help!

[wjq-learning]

Regarding the TUEV dataset, in our preprocessing we obtained the following sample counts:

• Train: 65,805
• Validation: 17,265
• Test: 29,421
• Total: 112,491

We're not entirely sure why your results differ (especially the total number), as we've followed a fixed splitting procedure. Here are a couple of suggestions:

1. Please make sure that you're using the latest version of the preprocessing code from the repository — especially for TUEV and TUAB, we've made several refinements over time.
2. The version of MNE used may also affect the outcome slightly. We used MNE v1.6.1 during preprocessing, and we recommend using the same version to ensure consistency.

You may also want to double-check the dataset version and file integrity on your end.

If the problem persists, feel free to share more details — we’re happy to help!

[ChenHu-ML]

I ran the following command to fine-tune on TUAB, using the recommended settings:

python /CBraMod/finetune_main.py \
  --seed 3407 \
  --cuda 0 \
  --epochs 50 \
  --batch_size 64 \
  --lr 1e-4 \
  --weight_decay 5e-2 \
  --optimizer AdamW \
  --clip_value 1 \
  --dropout 0.1 \
  --downstream_dataset TUAB \
  --datasets_dir /data/EEGdata/TUAB/edf/process_cbramod \
  --num_of_classes 1 \
  --model_dir ./models_weights/ \
  --num_workers 16 \
  --label_smoothing 0.1 \
  --multi_lr True \
  --frozen False \
  --use_pretrained_weights True \
  --foundation_dir pretrained_weights/pretrained_weights.pth

For TUAB, I enabled multi_lr, used the latest code to train, and confirmed that the dataset was processed with the latest code.
I got the result as below: epoch1_acc_0.77989_pr_0.86844_roc_0.86268. I'm not sure what might be causing this issue? Thank you so much for your kind assistance! Your help has been invaluable.

[ChenHu-ML]

@wjq-learning I re-downloaded the dataset, but it still ended up with the same size and file count. On the TUEV dataset for the train folder, I currently have 16,514 files in total, distributed across 290 folders, with a total size of 12.5 GB (13,448,131,403 bytes). This seems consistent with expectations. I also have updated mne to version 1.6.1. Despite this update, I am still unable to get the same number of files as reported in the paper.

[wjq-learning]

Hi,

Thank you for your detailed description and for following all the recommended settings carefully.

Regarding TUAB:

We're also a bit puzzled by the results — especially since the best model was found at epoch 1, which is unusual. In our setup, the TUAB dataset contains the following number of samples:

• Train: 301,443
• Validation: 71,067
• Test: 36,945
• Total: 409,455

Could you please double-check whether your data split matches these numbers? A mismatch here could explain the performance discrepancy.

As a next step, you may want to try:

• Training with a different random seed
• Adjusting the learning rate (especially for the classification head when multi_lr=True)
• Increasing the number of training epochs

We’ll continue to investigate on our end as well.

Regarding TUEV:

The preprocessing code for TUEV was adapted from the BIOT repository, and we verified that our final sample counts match those reported in the BIOT paper. So it’s unlikely that the issue stems from our code.

We used TUEV version 2.0.0, while the latest official version is now 2.0.1. It's possible that some changes or additions have been made by the dataset maintainers, which might explain the differences in file count and total size you're observing.

You may want to check which version of the dataset you are using.

Thanks again for your careful work and for raising these questions — we really appreciate it!

Feel free to keep us posted, and we’ll be happy to assist further.

[ChenHu-ML]

@wjq-learning, thank you for your kind support these days. I've been continuing the reproduction of your code.

TUAB

Classification Head

For TUAB, I'm unsure whether the performance drop is due to the classification head. As you previously mentioned, it seems you introduced some robustness improvements in this component. When I replaced the classification head with average pooling:

feats = feats.mean(dim=[1, 2]).contiguous().view(bz, 200)
nn.Linear(200, 1)

I observed a notable performance boost compared to the original classification head. The current result is:
epoch3_acc_0.80005_pr_0.88734_roc_0.88473,
with all hyperparameters set exactly as you previously recommended.

Dataset Split

I’m not sure if the discrepancy is due to differences in subject splitting. Even when using the same random seed, the train/val subject splits may differ, likely due to ordering in os.listdir().
To better eliminate any performance gap caused by inconsistent data splits, could you kindly share the actual subject lists you used for training and validation? Specifically, the following lists:

train_a_sub, val_a_sub  
train_n_sub, val_n_sub

which were generated from:

# train, val abnormal subjects
train_val_abnormal = os.path.join(root, "train", "abnormal", channel_std)
train_val_a_sub = list(
    set([item.split("_")[0] for item in os.listdir(train_val_abnormal)])
)
np.random.shuffle(train_val_a_sub)
train_a_sub, val_a_sub = (
    train_val_a_sub[: int(len(train_val_a_sub) * 0.8)],
    train_val_a_sub[int(len(train_val_a_sub) * 0.8):],
)

# train, val normal subjects
train_val_normal = os.path.join(root, "train", "normal", channel_std)
train_val_n_sub = list(
    set([item.split("_")[0] for item in os.listdir(train_val_normal)])
)
np.random.shuffle(train_val_n_sub)
train_n_sub, val_n_sub = (
    train_val_n_sub[: int(len(train_val_n_sub) * 0.8)],
    train_val_n_sub[int(len(train_val_n_sub) * 0.8):],
)

Thanks again—I truly appreciate your time and support!

TUEV

Thank you also for your reply regarding TUEV. Indeed, I was using version v2.0.1, which appears to have introduced certain updates to a small number of files in the train dataset. My result is:

epoch30_acc_0.64823_kappa_0.64347_f1_0.81052

This significant variation might be due to differences in the dataset version or the subject splitting strategy.

To better eliminate any performance gap caused by inconsistent splits, would it also be possible for you to share the actual training and validation subject lists you used for TUEV?

Thank you again for your continued support!

[wjq-learning]

Thank you so much for your continued support and for sharing your detailed findings — it's really helpful for improving the codebase.

It's very encouraging to see that using an average pooling-based classification head leads to more stable training on TUAB. I truly appreciate your effort in exploring this improvement.

Regarding the dataset splits for both TUAB and TUEV: you're absolutely right. The preprocessing code used in BIOT and LaBraM contains sources of randomness (e.g., os.listdir() ordering and np.random.shuffle()), which has unfortunately been carried over into CBraMod. Since I directly adopted the results in BIOT and LaBraM without having direct access to their exact data splits, even I cannot guarantee that our experiments were conducted under the same split conditions. As a result, I also cannot ensure a fair comparison across others and CBraMod.

To fully resolve this issue, we are planning to refine the preprocessing code to eliminate all randomness in the dataset splitting process and make it fully reproducible.

Please stay tuned for our upcoming updates — I’ll do my best to provide a more stable and reliable preprocessing pipeline soon.

Thank you again for your valuable feedback and patience!