The benchmark suite contains several benchmarks that can be used to evaluate the performance of the Asterinas platform. The following benchmarks are supported:

Sysbench
Membench
Iperf

Sysbench

Sysbench is a scriptable benchmark tool that evaluates system performance. It includes five kinds of tests: CPU, memory, file I/O, mutex performance, and thread performance. Detailed usage and options can be found by using:

sysbench --help
sysbench --test=<test_name> help

Here we list some general commands for evaluation:

# CPU test
sysbench --test=cpu --cpu-max-prime=<N> --num-threads=<N> run

# Thread test
sysbench --test=threads --thread-yields=<N> --num-threads=<N> --max-time=<N> run

# Mutex test
sysbench --test=mutex --mutex-num=<N> --mutex-locks=<N> --num-threads=<N>

# File test, the file-total-size and file-num of prepare and run must be consistent 
sysbench --test=fileio --file-total-size=<N><K,M,G> --file-num=<N> prepare
sysbench --test=fileio --file-total-size=<N><K,M,G> --file-num=<N> --file-test-mode=<Type> --file-block-size=<N><K,M,G> --max-time=<N> run

# Memory test
sysbench --test=memory --memory-block-size=<N><K,M,G> --memory-access-mode=<Type> --memory-oper=<Type> run

Membench

Membench is used to establish a baseline for memory bandwidth and latency. For specific usage and options, use:

membench --help

Here we list some general commands to use membench:

# Measure the latency of mmap
membench -runtime=5 -dir=/dev/zero -size=<N><K,M,G> -engine=mmap_lat

# Measure the latency of page fault handling. The size must be consistent with the file size.
membench -runtime=5 -dir=path_to_a_file -size=<N><K,M,G> -copysize=<N><K,M,G> -mode=<Type> -engine=page_fault 

# This is a easy way to generate a file with target size in Linux.
# The following command can create a file named 512K.file with the size 512K.
dd if=/dev/zero of=512K.file bs=1K count=512

Iperf

iPerf is a tool for actively measuring the maximum achievable bandwidth on IP networks. Usage and options are detailed in:

iperf3 -h

iperf can run in the following instructions:

export HOST_ADDR=127.0.0.1
export HOST_PORT=8888
iperf3 -s -B $HOST_ADDR -p $HOST_PORT -D # Start the server as a daemon
iperf3 -c $HOST_ADDR -p $HOST_PORT # Start the client

Note that a variant of iperf3 can measure the performance of vsock. But the implemented vsock has not been verified to work well in it.

Add benchmark to benchmark CI

To add a new benchmark to the Asternias Continuous Integration (CI) system, follow these detailed steps:

Step 1: Add the Benchmark to the `asterinas/test/benchmarks` Directory

Create the Benchmark Directory:
- Navigate to asterinas/test/benchmarks.
- Create a new directory named after your benchmark, e.g., getpid.
Create the Necessary Files:
- config.json:
```
{
    "alert_threshold": "125%",
    "pattern": "Syscall average latency:",
    "field": "4"
}
```
  - alert_threshold: Set the threshold for alerting. If the benchmark result exceeds this threshold, an alert will be triggered.
  - pattern: Define the pattern to extract the benchmark result from the output.
  - field: Specify the index of the result in the extracted output.
  For example, if the benchmark output is "Syscall average latency: 1000 ns", the pattern is "Syscall average latency:", and the field is "4". jq will extract "1000" as the benchmark result.
- result_template.json:
```
[
    {
        "name": "Average Syscall Latency on Linux",
        "unit": "ns",
        "value": 0,
        "extra": "linux_avg"
    },
    {
        "name": "Average Syscall Latency on Asterinas",
        "unit": "ns",
        "value": 0,
        "extra": "aster_avg"
    }
]
```
  - Adjust name and unit according to your benchmark specifics.
- run.sh:
```
#!/bin/bash

/benchmark/bin/getpid
```
  - This script runs the benchmark. Ensure the path to the benchmark binary is correct. asterinas/test/Makefile handles the benchmark binaries.

Step 2: Update the `asterinas/.github/benchmarks.yml` File

Edit the Benchmarks Configuration:
- Open asterinas/.github/benchmarks.yml.
- Add your benchmark to the strategy.matrix.benchmark list:
```
strategy:
  matrix:
    benchmark: [getpid]
  fail-fast: false
```

Step 3: Test the Benchmark Locally

Run the Benchmark:
- Execute the following command to test the benchmark locally:
```
cd asterinas
bash test/benchmark/bench_linux_aster.sh getpid
```
- Ensure the benchmark runs successfully and check the results in asterinas/result_getpid.json.

Additional Considerations

Validation: After adding and testing the benchmark, ensure that the CI pipeline correctly integrates the new benchmark by triggering a CI build.
Documentation: Update any relevant documentation to include the new benchmark, explaining its purpose and how to interpret its results.

By following these steps, you will successfully integrate a new benchmark into the Asternias CI system, enhancing its capability to evaluate platform performance.

README.md

Introduction to benchmarks

Overview of supported benchmarks