Testing
vpsAdminOS has a framework for writing and running tests. It is similar to tests on NixOS, but it is a different implementation.
Tests are run on one or more virtual machines running vpsAdminOS. These machines are managed by the test framework. All tests can be found in the vpsAdminOS repository.
Writing a test
Each test is a Nix file stored in directory tests/suite
. A test has a name,
one or more virtual machines to run on and a Ruby script that is run from
the host system and which can interact with the virtual machines.
import ../make-test.nix ({ pkgs }: {
name = "my-test";
description = ''
It's a great test indeed
'';
machine = import ../machines/empty.nix pkgs;
testScript = ''
machine.start
machine.succeeds("shell command that must succeed...")
'';
})
Different machines may be needed for various storage, configuration or clustering
tests. If only one machine is needed, it is simply called machine
and declared
as such. More machines can be defined as:
import ../make-test.nix ({ pkgs }: {
name = "my-test";
description = ''
It's a great test indeed
'';
machines = {
first = import ../machines/empty.nix pkgs;
second = import ../machines/empty.nix pkgs;
};
testScript = ''
first.start
second.start
'';
})
Disks can be added as:
import ../make-test.nix ({ pkgs }: {
name = "my-test";
description = ''
It's a great test indeed
'';
machine = {
# List of disk devices
disks = [
# 10 GB file sda.img will be created in the test's state directory
# and added to the virtual machine
{ type = "file"; device = "sda.img"; size = "10G"; }
];
# Machine configuration
config = {
imports = [ ../configs/base.nix ];
boot.zfs.pools.tank = {
layout = [
{ devices = [ "sda" ]; }
];
doCreate = true;
install = true;
};
};
};
testScript = ''
machine.start
machine.wait_for_zpool("tank")
'';
})
See template machine configs in tests/machines/
. vpsAdminOS configurations
used by machines for testing can be found in tests/configs
and the tests
themselves in tests/suite/
.
All tests have to be registered in tests/all-tests.nix
, otherwise they cannot
be run.
Running tests
To run the entire test suite, use:
./test-runner.sh test
Selected tests can be pattern-matched, e.g.:
./test-runner.sh test 'docker/*'
While developing a test, it is possible to start it with an interactive Ruby REPL:
./test-runner.sh debug my-test
The REPL can be used to issue the same commands as in the test script. The test
script itself can be run by calling method test_script
. You can call method
breakpoint
from inside the test to open the REPL from any point of execution.
Expected failure
A test can be expected to fail. The failure is shown, but it does not result in error exit status. If a test succeeds and we expected it to fail, it is considered as an error.
import ../make-test.nix ({ pkgs }: {
name = "my-failed-test";
description = ''
It's a great test indeed
'';
expectFailure = true;
machine = import ../machines/empty.nix pkgs;
testScript = ''
machine.start
machine.succeeds("shell command that fails...")
'';
})
Multiple test scripts
Each test can contain multiple test scripts. Test scripts are a part of test name and be run selectively.
import ../make-test.nix ({ pkgs }: {
name = "my-test";
description = ''
It's a great test indeed
'';
expectFailure = true;
machine = import ../machines/empty.nix pkgs;
testScripts = {
script1 = {
# Each script be expected to fail
# expectFailure = false;
# The test itself
script = ''
machine.start
machine.succeeds("uptime")
'';
};
script2 = {
script = ''
machine.succeeds("ps aux")
'';
};
};
})
./test-runner.sh ls 'my-test#*'
would show these tests as:
my-test#script1
my-test#script2
Script name is separated from test name by a hash (#
). Test scripts of
one test are run in the same environment one after the other -- they share
the same virtual machines, etc. Test scripts are executed in random order.
Test templates
Templates can be used to create multiple instances of a test. The difference between templates and multiple test scripts is that tests created by templates are isolated, have their own virtual machines and can run in parallel.
import ../make-template.nix ({ distribution, version }: rec {
instance = "${distribution}-${version}";
test = { pkgs }: {
name = "my-template@${instance}";
description = ''
Test something on ${distribution}-${version}
'';
machine = import ../machines/tank.nix pkgs;
testScript = ''
machine.wait_for_osctl_pool("tank")
machine.wait_until_online
machine.succeeds("osctl ct new --distribution ${distribution} --version ${version} testct")
'';
};
})
Within all-tests.nix
, the template would be listed as:
{ template = "my-template"; instances = distributions.all; }
./test-runner.sh ls 'my-template@*'
would show these tests as:
my-template@debian-stable
my-template@ubuntu-24.04
...
RSpec expectations
Test scripts can use RSpec expectations and optionally also example groups similar to RSpec. We reuse rspec-expectations and provide our own implementation of rspec-core. We aim to be compatible with RSpec when possible.
The following test demostrates available RSpec features.
import ../make-test.nix ({ pkgs }: {
name = "my-test";
description = ''
Test with RSpec expectations
'';
machine = import ../machines/tank.nix pkgs;
testScript = ''
# Optional global configuration for example groups / examples
configure_examples do |config|
# `:defined`, `:rand`, instance of `Random` or `Integer` used as a seed
# Defaults to `:rand` if not set
config.default_order = :defined
end
before(:suite) do
puts 'block executed before all examples'
end
# Create an example group
describe 'machine' do
before(:context) do
puts 'block executed before examples in this group'
end
after(:context) do
puts 'block executed after examples in this group'
end
before(:example) do
puts 'block executed before each example'
end
after(:example) do
puts 'block executed after each example'
end
it 'can execute commands' do
_, output = machine.succeeds('echo hello')
expect(output.strip).to eq('hello')
end
example 'without a block is skipped'
skip 'examples created by skip are also skipped' do
puts 'this will not run'
end
example 'can be skipped from the code block' do
skip
skip('with a reason')
end
pending 'examples are expected to fail' do
# This example will fail if expectations will be met
expect(0).to eq(1)
end
example 'can be marked as pending from the code block' do
pending
pending('this is expected to fail')
expect(0).to eq(1)
end
context 'nested example group with custom order of evaluation', order: :rand do
example '#1'
example '#2'
end
end
after(:suite) do
puts 'block executed after all examples'
end
'';
})
Groups and examples are evaluated in random order unless configured otherwise.
Temporary config changes
It is possible to change all test machine configurations by creating
os/configs/tests.nix
file, e.g. to change a kernel version used in tests:
{ config, ... }:
{
boot.kernelVersion = "6.1.30";
}