BackFed: An Efficient & Standardized Benchmark Suite for Backdoor Attacks in Federated Learning

About

BackFed is a comprehensive benchmark framework to efficiently and reliably evaluate backdoor attacks in Federated Learning (FL). This benchmark integrates Ray for parallel execution, Hydra for configuration management, and a modular architecture for easy extension of new attacks and defenses. Compared to existing codebases for backdoor attacks in FL, our framework could achieve 2X - 10X speedup in training time.

📄 Paper: BackFed: An Efficient & Standardized Benchmark Suite for Backdoor Attacks in Federated Learning

Updates

• [25/11/2025]: A new version of the paper has been released to Arxiv. Check it out for insights into the strengths and limitations of existing backdoor attacks and defenses!
• [05/09/2025]: Fix bugs for Flare and FLTrust defenses.
• [06/09/2025]: Add LocalDP client-side defense.
• [07/09/2025]: Add Anticipate Malicious client.

Features

• Modular Architecture: Each attack and defense belonges to a separate file, allowing for an easy extension with new attacks, defenses, models, and datasets.
• Parallel Execution: Supports both sequential and Ray-based parallel client training with training timeouts and selection threshold (select k% updates with earliest submission).
• Comprehensive Attack & Defense Library: Standardized implementations of diverse attacks and defenses for reliable benchmarking.
• Flexible Configuration: Hydra-based configuration system for easy and extensible experiment setups.
• Real-time Logging: Built-in real-time logging (WandB/CSV) and resource tracking (memory/time).

Supported Datasets and Models

Dataset	Task	Models	Data Distribution
CIFAR-10/100	image classification	ResNet + VGG models	IID (Uniform) + Simulated Non-IID (Dirichlet)
EMNIST (ByClass)	handwritten recognition	MnistNet + ResNet models	IID (Uniform) + Dirichlet Non-IID (Dirichlet)
Tiny-Imagenet	image classification	Any Pytorch models	IID (Uniform) + Dirichlet Non-IID (Dirichlet)
FEMNIST (Federated version of EMNIST)	handwritten recognition	MnistNet + ResNet models	Natural Non-IID (split by writers)
Reddit	next-word-prediction	LSTM + Transformer	Natural Non-IID (split by authors)
Sentiment140	sentiment analysis	LSTM + Transformer + ALBERT	Natural Non-IID (split by tweets)

Project Structure

BackFed/
├── config/                 # Configuration files
│   ├── base.yaml           # Base configuration
│   ├── *.yaml              # Dataset specific config
│   └── atk_config/         # Attack configurations
├── backfed/                # Core framework
│   ├── clients/            # Client module (benign and malicious clients)
│   ├── servers/            # Server module
│   ├── poisons/            # Trigger crafting module
│   ├── models/             # Model architectures
│   └── datasets/           # Dataset handling
├── experiments/            # Example experiment scripts
├── checkpoints/            # Model checkpoints
├── outputs/                # Hydra output logs + csv logs
└── main.py                 # Main entry point

Installation

Prerequisites

• Python 3.11
• PyTorch 2.6.0
• Ray 2.10.0 (Most reliable version, later versions could lead to high RAM usage)
• CUDA-compatible GPU (recommended)

Setup

1. Clone the repository:

git clone https://github.com/thinh-dao/BackFed.git
cd BackFed

2. Install dependencies in your environment:

pip install -r requirements.txt

3. Download pretrained models (Optional) Experiments in config folder required trained checkpoints, which are provided in Google Drive. You can download and put inside checkpoints folder (Note that you must preserve folder structure). Alternatively, you can use the following command to automate the download process:

chmod +x download_models.sh
./download_models.sh $dataset_name # Choose from cifar10, mnist, tinyimagenet, reddit, sentiment140
./download_models.sh all  # Download all models (skip if exists)

Training settings for these pretrained weights are given in experiments/clean_training.sh.

Usage

By default, all configuration files in config folder (e.g., emnist.yaml, cifar10.yaml, etc.) will inherit from base.yaml and override parameters. Please see config/base.yaml for a full description of parameters.

Basic Usage

Choose a configuration file to run experiment:

python main.py --config-name cifar10  # Equivalent to config/cifar10.yaml

For fine-grained control over parameters, you can:

1. Create custom configuration files that override key parameters from base.yaml
2. Use attack configuration files that override parameters from atk_config/base_attack.yaml
3. Override parameters directly via command line (see below)

Customizing Experiments in Command Line

Modify configuration parameters using Hydra's override syntax:

python main.py aggregator=unweighted_fedavg dataset=CIFAR10 model=ResNet18 num_rounds=600

To override attack configurations, use the following syntax:

python main.py atk_config=cifar10_multishot atk_config.model_poison_method=base atk_config.data_poison_method=pattern

Note that you can override specific parameters of an attack. For example, set data poison attack to IBA and change atk_eps of IBA to 0.1:

python main.py atk_config.data_poison_method=iba atk_config.data_poison_config.iba.atk_eps=0.1

Similarly, to override defense configurations, change aggregator and aggregator_config. For example, to use Trimmed-Mean defense with 20% trimming ratio:

python main.py aggregator=trimmed_mean aggregator_config.trimmed_mean.trim_ratio=0.2

Controlling Parallel Training Resources

In parallel training mode, multiple Ray actors (spawned processes) are created for concurrent client training. The framework automatically determines the optimal number of parallel actors based on available hardware resources.

Actor Calculation Formula:

num_parallel_actors = min(
    num_available_gpus / num_gpus_per_client,
    num_available_cpus / num_cpus_per_client
)

Where:

• num_available_gpus: Number of GPUs specified in cuda_visible_devices
• num_available_cpus: Total CPU cores available on the machine
• num_gpus_per_client: GPU fraction per client (set via num_gpus parameter)
• num_cpus_per_client: CPU cores per client (set via num_cpus parameter, default: 1)

Resource Configuration Examples

# Example 1: 4 GPUs, 0.5 GPU per client = 8 parallel actors
python main.py cuda_visible_devices="0,1,2,3" num_gpus=0.5

# Example 2: 2 GPUs, 1.0 GPU per client = 2 parallel actors
python main.py cuda_visible_devices="0,1" num_gpus=1.0

Performance Tuning

The most effective way to control parallelism is by adjusting:

1. cuda_visible_devices: Controls the number of available GPUs
2. num_gpus: Controls GPU allocation per client

⚠️ IMPORTANT: If the application freezes due to too many parallel Ray actors, try one of these solutions:

• Increase num_gpus (allocate more GPU memory per client)
• Decrease the number of GPUs in cuda_visible_devices
• Increase num_cpus (allocate more CPU cores per client)

This will reduce the total number of parallel actors and prevent resource contention.

Configuration

The framework uses Hydra for configuration management. Below are the key configuration parameters organized by category:

Core Experiment Settings

Parameter	Type	Default	Description
aggregator	str	unweighted_fedavg	Aggregation method for federated learning (see Available Aggregation Methods)
no_attack	bool	False	Disable attacks (set to True for clean training)
atk_config	dict	Null	Attack configuration file inside atk_config folder
training_mode	str	parallel	Training mode (parallel for Ray-based or sequential for single-threaded)
num_rounds	int	600	Number of federated learning rounds
num_clients	int	100	Total number of clients in the federation
num_clients_per_round	int	10	Number of clients selected per round

Note: If you use debuggers (e.g., ipdb or pdb), it is recommended to set training_mode=sequential and progress_bar=False.

Dataset and Model Configuration

Parameter	Type	Default	Description
dataset	str	Required	Dataset name (CIFAR10, CIFAR100, MNIST, EMNIST, FEMNIST, TinyImageNet, Reddit, Sentiment140)
model	str	Required	Model architecture (ResNet18, MNISTNet, Simple, Transformer, WordModel)
num_classes	int	Required	Number of classes in the dataset
partitioner	str	dirichlet	Data partitioning method (uniform or dirichlet)
alpha	float	0.5	Dirichlet distribution parameter (lower = more non-IID, higher = more IID)

Client Training Configuration

Parameter	Type	Default	Description
client_config.local_epochs	int	2	Number of local training epochs per client
client_config.batch_size	int	64	Training batch size for each client
client_config.lr	float	0.1	Learning rate for client training
client_config.optimizer	str	sgd	Optimizer type (sgd, adam, adamw)

Timeout Configuration

Parameter	Type	Default	Description
client_timeout	float	Null	Timeout for each client training & evaluation (in seconds)
selection_threshold	float	Null	Fraction of clients to select based on earliest completion time

Resource Management

Parameter	Type	Default	Description
cuda_visible_devices	str	"0,1,2,3,4"	GPU devices to use (comma-separated)
num_cpus	int	1	CPU cores allocated per client
num_gpus	float	0.5	GPU fraction allocated per client

Model Checkpointing and Resuming

Parameter	Type	Default	Description
checkpoint	str/int	Null	Resume from checkpoint (round number, file path, or "wandb")
save_model	bool	False	Save model to the same folder as the runtime logs
save_checkpoint	bool	False	Save model to checkpoints directory
save_checkpoint_rounds	list	[200,400,600,800,1000]	Specific rounds to save model
pretrain_model_path	str	Null	Path to pretrained weights or "IMAGENET1K_V2"

Note: save_model and save_checkpoint are different. save_model only saves model weights, while save_checkpoint saves the model weights with metadat (current round, metrics, and defense-specific intermediate state). checkpoint loads from a checkpoint while pretrain_model_path loads from model weights.

Logging and Visualization

Parameter	Type	Default	Description
save_logging	str	csv	Logging method (wandb, csv, both, or Null)
dir_tag	str	specified or auto-generated	Directory tag for organizing results
plot_data_distribution	bool	False	Generate data distribution plots
plot_client_selection	bool	False	Generate client selection plots
disable_progress_bar	bool	False	Disable progress bars during training

Attacks and Defenses

Implemented Attacks

Model Poisoning Attacks

Method	Description	Source
Neurotoxin	Selective parameter poisoning targeting specific neurons	Neurotoxin: Durable backdoors in federated learning (ICML 2022)
Chameleon	Use contrastive learning to adapt the feature extractor better to attacks	Chameleon: Adapting to peer images for planting durable backdoors in federated learning
Anticipate	Anticipates future aggregation steps to craft malicious updates	Thinking Two Moves Ahead: Anticipating Other Users Improves Backdoor Attacks in Federated Learning

Data Poisoning Attacks

Method	Description	Source
Pattern	White-patch trigger	How To Backdoor Federated Learning
Pixel	One pixel as trigger	BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain
BadNets	Classic BadNets trigger	BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain
Blended	Blends a trigger pattern with original images	Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning
Distributed	Distributed backdoor attack across multiple clients	DBA: Distributed Backdoor Attacks against Federated Learning
Edge-case	Use edge-case samples as backdoor triggers	Attack of the Tails: Yes, You Really Can Backdoor Federated Learning
A3FL	Optimize a trigger pattern	A3FL: Adversarially Adaptive Backdoor Attacks to Federated Learning
IBA	Use a trigger generator to inject triggers	IBA: Towards Irreversible Backdoor Attacks in Federated Learning

Implemented Defenses

Robust Aggregation

Defense	Description	Source
TrimmedMean	Removes extreme updates before aggregation	Byzantine-Robust
MultiKrum	Selects subset of updates closest to each other	Krum
GeometricMedian	Uses geometric median for aggregation	Geometric Median
CoordinateMedian	Uses coordinate-wise median aggregation	Coordinate-wise
FLTrust	Trust-based weighted aggregation with server dataset	FLTrust
RobustLR	Adaptive learning rate based on update trustworthiness	RobustLR
WeakDP	Clip aggregated model and add noise	WeakDP

Anomaly Detection

Defense	Description	Source
FoolsGold	Detects sybil attacks via update similarity	FoolsGold
DeepSight	Clustering-based backdoor detection	DeepSight
RFLBAT	PCA-based malicious update detection	RFLBAT
FLDetector	Sliding window approach for anomaly detection	FLDetector
FLARE	MMD-based anomaly detection with trust scores	FLARE
Indicator	Statistical anomaly detection method	Indicator
FLAME	Clustering, norm clipping, and noise addition	FLAME
AlignIns	Filters malicious updates using TDA/MPSA statistics	AlignIns
FedDLAD	COF-based reference client selection and norm clipping	FedDLAD
MultiMetrics	Combines multiple distance metrics to score and filter updates	Multi-metrics
Snowball	Clustering-based initial filtering followed by VAE-based refinement	Snowball

Client-side Defenses

Defense	Description	Source
FedProx	Adds proximal term to client optimization	FedProx
LocalDP	Adds DP noise at client-side after training	LocalDP

Methods Under Development

• Client-side defenses: FLIP, FL-WBC.
• Robust-aggregation defense: FedReDefense.
• Attacks: 3DFED, F3BA.

Examples

Check the experiments/ directory for example scripts:

• clean_training.sh: Train models without attacks
• fedavg_vs_attacks.sh: Evaluate FedAvg against various attacks
• anomaly_detection.sh: Test anomaly detection defenses
• robust_aggregation_multishot.sh: Test robust aggregation against multishot attacks
• model_replacement.sh: Model replacement attack experiments
• server_lr.sh: Experiment with different server learning rates
• sentiment140.sh: Sentiment140 dataset experiments
• weakdp_study.sh: Differential privacy defense studies

Acknowledgements

• Ray Backend follows the implementation of FL-Bench
• Attacks in NLP Backdoors101
• BackdoorIndicator

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.