|Version 4 (modified by 5 years ago) ( diff ),
Automated testing of HelenOS in a virtual machine
Part of HelenOS continuous testing are also tests that are executed inside HelenOS. That is, we launch a VM (QEMU or MSIM at the moment) and check that command executed inside HelenOS gives the expected output. This page contains a brief overview of how to write such testing scenarios and how to run them.
For this to work, you need to have Python 3, QEMU, ImageMagick (i.e.
convert command) and SED. For testing on mips32, you will also need MSIM, Xvfb (or Xephyr), xterm and xdotool.
The testing script is in the root of the CI repository. To try it, prepare a HelenOS image (or grab one from ci.helenos.org) for one of the supported architectures. Here, we will assume you have downloaded HelenOS with binutils and PCC for amd64 into
helenos-amd64-with-binutils-pcc.iso into clone of CI repository.
For this image, we choose to run a scenario where we compile and run a Hello, World! program written in C. Scenarios are located under
scenarios folder in CI repository, their format will be described later on.
To actually run the scenario, we launch the
test-in-vm.py script and specify path to the boot image, architecture and path to the scenario. Advanced options will be described later.
./test-in-vm.py \ --arch amd64 \ --image helenos-amd64-with-binutils-pcc.iso \ --scenario scenarios/coast/pcc/hello.yml
When you run this script, you will see that it starts QEMU with HelenOS and waits until the terminal appears. Then it automatically types commands that compiles
hello.c and launches the compiled
a.out binary. QEMU is then terminated and if all went okay, we will see a message about successful pass of the scenario (or error message explaining the reason for failure).
The use described in the paragraphs above is the basic usage. Usually, we would like to run the scenario without actual QEMU window (as not to distract us) and maybe dump all the contents of the terminal session. Switches for just right that are here:
- Hides any windows, i.e. we can run the script on background or on CI server.
- Change the amount of memory available to the virtual machine.
- Extra options to pass through to the emulator (can be specified multiple times).
- Where to store dump of the whole terminal session.
- Where to store screenshot just before the VM is terminated.
- Prints (a lot of) debugging messages.
The scenario file is stored in YAML format and has rather simple structure that is best described on example. We will again use the example from above where the scenario compiles a simple C program.
meta: name: "Hello from PCC" harbours: - pcc - binutils tasks: - boot - command: args: "cp /src/c/demos/hello/hello.c /tmp/hello.c" - command: args: "cd /tmp" - command: args: "pcc hello.c" - command: args: "./a.out" assert: "hello!"
meta has one mandatory item:
name should briefly describe the scenario. This text is also used in CI output.
harbours key is used solely by CI script and lists packages that has to be part of the image. If you run
test-in-vm.py manually, it is your responsibility to prepare the right image!
The actual testing scenario is a list of tasks that are executed one after another. And unless stated otherwise, failure in one task aborts the whole run. The first task is always
boot that simply waits for the machine to boot into userspace/GUI. The only other available task (so far) is
command that executes single command in the terminal.
args is the actual command and without any other arguments, the script only checks that the command has not ended with a failure (i.e. exit code 0).
To check that a specific string appears in the output of the command, one can use the
assert key. If this string is not found in the output and the command terminates (i.e. the prompt reappears), the scenario is aborted. To check for a text that shall not appear (i.e. its appearance is a reason for abortion), use
negassert. If we expect that the program would abort (i.e. we intentionally dereference NULL pointer etc.), we can add
ignoreabort: True to skip checking the program exit code.
So far, the biggest scenario is
scenarios/base/fs.yml that checks that it is possible to create FAT image inside HelenOS, mount it and write data to it. This scenario also uses
name to better describe why we launch each command in the sequence. Use it as a good example for your own experimenting.