asterinas/test/benchmark/README.md

# Introduction to benchmarks

## Overview of supported benchmarks
The benchmark suite contains several benchmarks that can be used to evaluate the performance of the Asterinas platform. The following benchmarks are supported:

- [Sysbench](#Sysbench)
- [Membench](#Membench)
- [Iperf](#Iperf)

### Sysbench
Sysbench is a scriptable benchmark tool that evaluates system performance. It includes five kinds of tests: CPU, memory, file I/O, mutex performance, and thread performance. Detailed usage and options can be found by using:
```shell
sysbench --help
sysbench --test=<test_name> help
``` 
Here we list some general commands for evaluation:
```shell
# CPU test
sysbench --test=cpu --cpu-max-prime=<N> --num-threads=<N> run

# Thread test
sysbench --test=threads --thread-yields=<N> --num-threads=<N> --max-time=<N> run

# Mutex test
sysbench --test=mutex --mutex-num=<N> --mutex-locks=<N> --num-threads=<N>

# File test, the file-total-size and file-num of prepare and run must be consistent 
sysbench --test=fileio --file-total-size=<N><K,M,G> --file-num=<N> prepare
sysbench --test=fileio --file-total-size=<N><K,M,G> --file-num=<N> --file-test-mode=<Type> --file-block-size=<N><K,M,G> --max-time=<N> run

# Memory test
sysbench --test=memory --memory-block-size=<N><K,M,G> --memory-access-mode=<Type> --memory-oper=<Type> run
```

### Membench
Membench is used to establish a baseline for memory bandwidth and latency. For specific usage and options, use:
```shell
membench --help
``` 
Here we list some general commands to use membench:
```shell
# Measure the latency of mmap
membench -runtime=5 -dir=/dev/zero -size=<N><K,M,G> -engine=mmap_lat

# Measure the latency of page fault handling. The size must be consistent with the file size.
membench -runtime=5 -dir=path_to_a_file -size=<N><K,M,G> -copysize=<N><K,M,G> -mode=<Type> -engine=page_fault 

# This is a easy way to generate a file with target size in Linux.
# The following command can create a file named 512K.file with the size 512K.
dd if=/dev/zero of=512K.file bs=1K count=512
```

### Iperf
iPerf is a tool for actively measuring the maximum achievable bandwidth on IP networks. Usage and options are detailed in:
```shell
iperf3 -h
``` 
iperf can run in the following instructions:
```shell
export HOST_ADDR=127.0.0.1
export HOST_PORT=8888
iperf3 -s -B $HOST_ADDR -p $HOST_PORT -D # Start the server as a daemon
iperf3 -c $HOST_ADDR -p $HOST_PORT # Start the client
```
Note that [a variant of iperf3](https://github.com/stefano-garzarella/iperf-vsock) can measure the performance of `vsock`. But the implemented `vsock` has not been verified to work well in it.

## Add benchmark to benchmark CI

To add a new benchmark to the Asternias Continuous Integration (CI) system, follow these detailed steps:

### Step 1: Add the Benchmark to the `asterinas/test/benchmarks` Directory

1. **Create the Benchmark Directory:**
   - Navigate to `asterinas/test/benchmarks`.
   - Create a new directory named after your benchmark, e.g., `getpid`.

2. **Create the Necessary Files:**
   - **config.json:**
     ```json
     {
         "alert_threshold": "125%",
         "pattern": "Syscall average latency:",
         "field": "4"
     }
     ```
     - `alert_threshold`: Set the threshold for alerting. If the benchmark result exceeds this threshold, an alert will be triggered.
     - `pattern`: Define the pattern to extract the benchmark result from the output.
     - `field`: Specify the index of the result in the extracted output.
    
      For example, if the benchmark output is "Syscall average latency: 1000 ns", the `pattern` is "Syscall average latency:", and the `field` is "4". `jq` will extract "1000" as the benchmark result.

   - **result_template.json:**
     ```json
     [
         {
             "name": "Average Syscall Latency on Linux",
             "unit": "ns",
             "value": 0,
             "extra": "linux_avg"
         },
         {
             "name": "Average Syscall Latency on Asterinas",
             "unit": "ns",
             "value": 0,
             "extra": "aster_avg"
         }
     ]
     ```
     - Adjust `name` and `unit` according to your benchmark specifics.

   - **run.sh:**
     ```bash
     #!/bin/bash

     /benchmark/bin/getpid
     ```
     - This script runs the benchmark. Ensure the path to the benchmark binary is correct. `asterinas/test/Makefile` handles the benchmark binaries.

### Step 2: Update the `asterinas/.github/benchmarks.yml` File

1. **Edit the Benchmarks Configuration:**
   - Open `asterinas/.github/benchmarks.yml`.
   - Add your benchmark to the `strategy.matrix.benchmark` list:
     ```yaml
     strategy:
       matrix:
         benchmark: [getpid]
       fail-fast: false
     ```

### Step 3: Test the Benchmark Locally

1. **Run the Benchmark:**
   - Execute the following command to test the benchmark locally:
     ```bash
     cd asterinas
     bash test/benchmark/bench_linux_aster.sh getpid
     ```
   - Ensure the benchmark runs successfully and check the results in `asterinas/result_getpid.json`.

### Additional Considerations

- **Validation:** After adding and testing the benchmark, ensure that the CI pipeline correctly integrates the new benchmark by triggering a CI build.
- **Documentation:** Update any relevant documentation to include the new benchmark, explaining its purpose and how to interpret its results.

By following these steps, you will successfully integrate a new benchmark into the Asternias CI system, enhancing its capability to evaluate platform performance.