Qihe Benchmark

This repository collects popular open-source Verilog/SystemVerilog projects curated for evaluating the Qihe static analysis framework.

It includes scripts to:

• Compile these open-source projects into Qihe IR for analysis
• Run Qihe's static analyses on the compiled IR for evaluation
• Distribute the compiled IR for other potential uses

Getting Started

Scripts in this repository run on Linux and require the following dependencies:

• Qihe CLI
  • See this Installation Guide. Note that the Slang tool, mentioned as optional in the guide, is required here.
• zip (optional, for packaging compiled files)
  • On Ubuntu, run: sudo apt install zip

Next, clone this repository (with submodules), enter the directory, and test the build script:

git clone --recursive https://github.com/QinlinChen/qihe-benchmark.git
cd qihe-benchmark
./build.sh -h

User Guide

Compilation

We provide scripts to compile the open-source projects in this repository into Qihe IR for analysis.

To see available projects:

./build.sh -h

To compile one or more projects:

./build.sh <project-name> ...

The <project-name> should match those listed in ./build.sh -h.

The compiled Qihe IR is placed in either:

• A single file: ./build/<project-name>.qh
• A directory: ./build/<project-name> (containing multiple .qh files)

To compile all projects:

./build.sh -a

Evaluation

Currently, we have two ways for evaluating Qihe's analyses:

• This repository provides the analyze.sh script to run Qihe's analyses on compiled IRs
  • Useful when human review is needed, such as for bug detection
• Qihe's repository contains JUnit tests to run Qihe's analyses on compiled IRs
  • Better for automatically processing and visualizing analysis results via Java code

Using this repository

If the compiled project has a single IR file:

# Compile the tiny_gpu project into a single Qihe file
# located at build/tiny-gpu.qh
./build.sh tiny_gpu

Run pre-configured analyses:

./analyze.sh build/tiny-gpu.qh

Then check the results:

• Analysis Reports: qihe-storage/tiny-gpu/report-(high|low).txt
• Logs: qihe-storage/tiny-gpu/run.log

If the compiled project has multiple IR files:

# Compile the hdl project into multiple Qihe files
# located in build/hdl
./build.sh hdl

Run analyses:

./analyze.sh build/hdl

Check the results:

• Analysis Reports: qihe-storage/hdl/report-(high|low).txt
• Logs: qihe-storage/hdl/run.log

Note: To customize analyze.sh behavior for a project (e.g., configuring which analyses to run), modify env.sh and options.toml in the corresponding qihe-storage/<project-name> directory.

If the qihe-storage directory has not been created yet, initialize it with:

./analyze.sh ./build/tiny-gpu.qh --init-only

Using Qihe's repository

Qihe's repository requires placing compiled projects in the directory: qihe/platform/src/test/resources/testcases/evaluation/qihe-benchmark.

You can achieve this using the script:

./install.sh --qihe [<dir>]

Here, <dir> is the root directory of the qihe repository (default: ../qihe).

Then, you can write and run JUnit tests in Qihe's repository to evaluate analyses on the compiled IRs.

Distribution

You can distribute the compiled IRs for other potential uses.

To distribute within the same machine (different directory):

./build.sh -a
./install.sh --prefix <dir>

This copies the Qihe IRs from ./build to the specified <dir> using the same directory hierarchy.

To distribute to other machines:

./package.sh

This creates a build.zip file containing all IR files from the ./build directory.

Developer Guide

To add a new Verilog project to this repository:

1. Add the project as a submodule: Add the project to the projects directory as a Git submodule.

2. Create a compilation function: Add a new function build in tasks/<project-name>.sh with the logic to compile the project into Qihe files.

3. Follow the output convention:

   • Single file: Write the .qh file to build/<project-name>.qh.
   • Multiple files: Place the .qh files in build/<project-name>/.

To define a new task, such as counting lines of code for each project, add a function with a specific name (e.g., loc) to each project's tasks/<project-name>.sh script. You can then run this task across all or selected projects with:

./batch_task.sh <task-name> (-a | [project-name]...)

INFO: Actually, build.sh internally invokes ./batch_task.sh build ...