Foresee: A Misbehaviour Forecasting tool for Focused Autonomous Driving Systems Testing

Contains code from the Transfuser repository: https://github.com/autonomousvision/transfuser/

OS: Ubuntu 20.04 (Also tested on Ubuntu 22.04)

Setting up and running the experiment:

mkdir ~/Documents/ADS/
mkdir ~/Documents/ADS/results/
mkdir ~/Documents/ADS/results/Interfuser/

Set up Transfuser in the ADS directory (for CARLA)

cd ~/Documents/ADS/
git clone https://github.com/autonomousvision/transfuser.git
cd transfuser
git checkout 2022
chmod +x setup_carla.sh
./setup_carla.sh

Set up InterFuser in the ADS directory

cd ~/Documents/ADS/
git clone https://github.com/opendilab/InterFuser.git
cd InterFuser
conda create -n interfuser python=3.7
conda activate interfuser
pip3 install -r requirements.txt
cd interfuser
python setup.py develop
pip install -U pip setuptools

After running the above commands, follow these steps:

• Download the model from http://43.159.60.142/s/p2CN (If the link does not work, try this)
• Copy the model to ~/Documents/ADS/InterFuser/leaderboard/team_code/
• Update the model_path variable in the ~/Documents/ADS/Interfuser/leaderboard/team_code/interfuser_config.py with the absolute path of the model. For example (replace with the username of the machine):

model_path = "/home/<user>/Documents/ADS/InterFuser/leaderboard/team_code/interfuser.pth.tar"

Now, copy the artifact (foresee folder) to ~/Documents/ADS/repository/ and change the current directory to the directory of this artifact. For example:

mkdir ~/Documents/ADS/repository/
cp -R foresee ~/Documents/ADS/repository/
cd ~/Documents/ADS/repository/foresee

Install CUDA and Tensorflow

Install CUDA on the machine, the experiment was run with CUDA 11.3 (Ubuntu 20.04) and tested with CUDA 11.7 (Ubuntu 22.04) as well. Make sure to install compatible nvidia drivers as well (For ubuntu 20.04, the driver version was nvidia-driver-525. For ubuntu 22.04, the driver version was nvidia-driver-545).

Note: Installing CUDA can be a trial and error procedure, the following is one example of how to install cuda, the instructions for you machine can vary.

Example:

sudo apt install nvidia-driver-545
sudo apt-get install cuda=11.7.0-1

After that, install tensorflow.

pip install tensorflow==2.11.0

Running the Experiment

Once set up, the experiment can be run with the conda environment "interfuser" following the following format:

bash Experiment.sh <Agent_Name> <Folder_Name> <o_b> <o_a> <n_rp> <scenario> <use_pre_run_seeds>

Here, <o_b> and <o_a> refers to the offset before and after a risky point for clipping, <n_rp> refers to the number of risky points per scenario, <scenario> refers to the name of the scenario (e.g. Scenario3, Scenario7 etc.) and <use_pre_run_seeds> is a flag that controls the usage of pre-run seeds (more details on this flag in the next section).

Example:

conda activate interfuser
cd ~/Documents/ADS/repository/foresee/src
bash Experiment.sh Interfuser Sc_4_10x4_Run_1 5 5 4 Scenario4 0

The script Experiment.sh has variables to adjust different experiment parameters inside.

Using pre-run seeds

By default, the Experiment.sh script runs all the seeds, but pre-run seeds can be used to run the technique as well. The seeds can be downloaded from this link. Once downloaded, extract the zip file and place the Plain_Seeds folder under the ~/Documents/ADS/results/Interfuser/ directory, and pass the flag <use_pre_run_seeds> as 1 while running the Experiemnt.sh script.

Example:

bash Experiment.sh Sc_4_10x4_Run_1 5 5 4 Scenario4 1

Run all experiments

By default, the scripts only run the technique described in the paper (Foresee). To run SelfOracle and Random, please set the flags RUN_SelfOracle=1 and RUN_Random=1. To run the exhaustive algorithm, use the flag RUN_Greedy=1 and set the other flags to 0.

To recreate results from the paper and run all experiments, run as follows:

bash Run_All_Experiments.sh

Running with pre-run seeds can save time, so to use the pre-run seeds (according to the previous section), run as follows:

bash Run_All_Experiments.sh 1

Generate tables from paper

After running the Run_All_Experiments.sh script, run the following to generate the tables used in the paper:

bash generate_tables_in_paper.sh 1 1 1

The flags are to generate f1 scores, table for effectiveness of foresee and table comparing foresee, selforacle and random respectively. To skip any of these tables, pass 0 as the corresponding flag.

To customize what directories this script uses, the csv file output_directories.csv needs to be modified, where the first column contains the names of the output directories, the second column contains the clip duration (i.e. o_a + o_b) and the third column contains the name of the scenario.

Run the ablation experiment from paper

To recreate results from the paper and run all experiments, run as follows:

bash Run_All_Experiments_ablation.sh

Running with pre-run seeds can save time, so to use the pre-run seeds, first create a ${AGENT}_ablation folder in results and copy the Plain_Seeds folder, created in the previous steps. Then run the following:

bash Run_All_Experiments_ablation.sh 1

Run the drivefuzz experiment from paper

To recreate results from the paper and run all experiments, run as follows:

bash Run_All_Experiments_drivefuzz.sh

This script expects the original experiment pre-run seeds to exist, as such run this script only if ${AGENT}/Plain_Seeds already exists.