How to test your RISC-V emulator

Published on

Recently, I had a chance to learn about RISC-V emulation because I needed a CPU for my fantasy computer. Writing an emulator was, surprisingly, not a frustrating experience because of not only the ISA specification being open source and written in an understandable language, but also a great official test suite. Unfortunately, the suite's documentation isn't that comprehensive and is only rarely mentioned online. As such, here's a small guide on how to use it.

But first, a little disclaimer: As you can probably tell, I'm not a low-level development guru. There's probably a better way to do it, so take everything with a grain of salt.

Toolchain

In order to build the test cases (or well, any native RISC-V software), you need a toolchain. The 2 most popular options you have are GCC and LLVM, and I recommend using the former since that's what the test suite was written in mind. LLVM is likely possible to use with a minor makefile modification, but I didn't bother.

The easiest way to get the whole GCC toolchain is to either get it from your Linux distro (if it offers a riscv64-unknown-elf one) or build it manually using the official repo. For obvious reasons, I'm going to cover the second scenario.

Install the build dependencies for your specific distro using the command in the readme and clone the git repo. You don't really need --recursive here since there are more submodule dependencies than most targets need, and it's going to fetch everything at the build time anyway:

git clone https://github.com/riscv-collab/riscv-gnu-toolchain.git

Next, you need to build the Newlib flavor. I personally prefer using ~/.local as the prefix, so the whole thing is installed for my user only and requires no black magic other than putting ~/.local/bin in my $PATH:

./configure --prefix=$HOME/.local
make

Depending on how slow your hardware is, the whole process will take an hour or two. In the meanwhile, you can ensure that you're properly hydrated, your dishes are washed, and your laundry has been done.

Test suite

First of all, you need to clone the thing. Note that --recursive is needed here since the repo has a submodule that it doesn't fetch by itself:

git clone --recursive https://github.com/riscv-software-src/riscv-tests.git

In contrast to what I do with the toolchain, I prefer using a prefix pointing to a folder near the test suite itself. Since it doesn't install any executables native to your host, this shouldn't be an issue. Let's say you have the following file hierarchy in your project:

In order to build the tests and install them into output, you need to run the following:

cd riscv-tests

autoconf
./configure --prefix="$(realpath '../output')"

make
make install

Okay, now what?

As you can probably see, you now have a bunch of tests cases themselves:

The object of your interest here is the files with no extension (not counting Makefile, obviously). They're the native RISC-V binaries in the ELF format. .dump files are their disassembled versions, generated with objdump, that are actually quite handy because they let you quickly debug why a test case fails without having with manually objdump everything or rolling out your own full-fledged debugger.

Before even touching any of them, you should consider: does your emulator frontend support ELF? If not, you should first dump the contents of the test cases into a flat binary file:

riscv64-unknown-elf-objcopy -O binary test-case test-case.bin

Next, you need to pick the test cases that are actually relevant for your specific emulator (or "TVM" as the test suite calls them) by looking at their filenames. For example, here's what different parts of rv32mi-p-breakpoint mean:

XLEN bits is self-explanatory - pick 32 or 64 depending on which one you're emulating. There's no need to use both.

Privilege mode depends on which ones your emulator is capable to operate in, except for u that you're going to need anyway because it tests the instructions common to all privilege modes. So for example, if you're implementing the:

Extension/base instruction set depends on which ones you care about. So for example, if you're making an RV64IMAC emulator, you need to pick i, m, a, and c.

Target environment is a bit weird to define, so just consult the table in the test suite's readme. But in short, if you're making a simple single-hart emulator with no virtual memory support, you need p.

Tested instructions are easy to choose - just pick all of them. Or well, I guess you can omit ma_data if you don't implement misaligned loads/stores.

Running the tests

For this I'm going to recommend you the approach used by Max Nurzia's rv - make a tiny wrapper around your emulator's core and shove the unmodified test cases into it.

As test cases are just executables that do stuff and make an ecall when they're done, all you need to do is load them into memory, jump to the first instruction, and run until you encounter an ecall. In order to verify if the test passed, you need to look at the RVTEST_PASS/RVTEST_FAIL macros inside the riscv_test.h header file of your target environment. For example, here's what they look like for p:

TESTNUM: gp

RVTEST_PASS:
    fence
    li TESTNUM, 1
    li a7, 93
    li a0, 0
    ecall

RVTEST_FAIL:
    fence
1:  beqz TESTNUM, 1b
    sll TESTNUM, TESTNUM, 1
    or TESTNUM, TESTNUM, 1
    li a7, 93
    addi a0, TESTNUM, 0
    ecall

Which basically means that if a test passed, gp (aka x3) should be 1 and a0 (aka x10) should be 0, and if they aren't, the test failed.

The easiest way to provide feedback for test coverage is to repeat this process for each test case, one at a time, and either print the result or return it as an exit code and let a separate script do the printing for you. Here's a simplified version of the script I used in my emulator:

#!/bin/sh

set -ue

RUNNER=your-wrapper

RED='\033[0;31m'
GREEN='\033[0;32m'
NORMAL='\033[0m'

for CASE in relevant/cases/*
do
    if "${RUNNER}" "${CASE}"
    then
        echo -e "${GREEN}$(basename "${CASE}"): PASS${NORMAL}"
    else
        echo -e "${RED}$(basename "${CASE}"): FAIL${NORMAL}"
    fi
done