BSD.REGRESS.MK(5) | File Formats Manual | BSD.REGRESS.MK(5) |
bsd.regress.mk
—
regression test master Makefile fragment
.include <bsd.regress.mk>
bsd.regress.mk
holds the standard routines
used by the source regression tests. Some variables and targets are for its
internal use only. The rest are documented here.
Since this file also includes bsd.prog.mk
,
all of the usual OpenBSD Makefile variables may be
used to build the regression test programs.
depend
regress
regress
exits with an exit status of 0 even if
some of the tests failed.run-regress-*
REGRESS_CLEANUP
REGRESS_EXPECTED_FAILURES
REGRESS_FAIL_EARLY
regress
target will abort as soon as a test
fails.REGRESS_LOG
REGRESS_ROOT_TARGETS
SUDO
variable should be used in the test parts
that require root. See also SUDO
.REGRESS_SETUP
REGRESS_SETUP_ONCE
REGRESS_SKIP_SLOW
REGRESS_SLOW_TARGETS
variable.REGRESS_SKIP_TARGETS
REGRESS_SLOW_TARGETS
REGRESS_SKIP_SLOW
variable.REGRESS_TARGETS
run-regress-${PROG}
.SUDO
Some variables are intended to be set at runtime in the environment or in mk.conf(5), but not in the regress Makefile itself.
If an individual test passes, make testname should return with an exit status of 0. If it fails, it should return with a non-zero exit status.
If a test cannot be executed because a package is not installed or
some environment variable is not set, make
testname should print "SKIPPED" to stdout
and exit with status 0. To skip everything, implement the
regress
target with a command that prints
"SKIPPED".
Some tests may require a special setup on the test machine that has to be done manually before testing. This requirement has to be documented in the Makefile or in a README file. The test should find out whether the setup exists before running and print "SKIPPED" and exit if it is missing.
Tests should not fail because an intended feature has not been implemented yet. To avoid such false failures, a test should show the reason, print "DISABLED" to stdout and exit with status 0.
Tests must be able to run with an obj
directory. In case the test is not linked to the build or the tester forgot
to run make obj
before, this
directory or symlink may not exist. Then the test should run anyway.
Tests are executed with make
regress
, but running make
all
or make should have the same
effect. Tests must be runnable by root, and may also succeed when run as a
regular user. Tests must not assume they have a controlling tty, to allow
them to be run by cron(8). An individual
regress test may create a pseudo tty if it needs one.
Tests should use the binaries installed and the kernel running on
the local system. They may use environment variables to test alternative
binaries or remote kernels running on other machines. In some cases a test
may need binaries or libraries or object files to be present in
/usr/obj/ that exist only after
make build
was run in
/usr/src/. The test must not assume that they have
already been built, but should run make in the appropriate
source directory as a dependency. For missing generated source or header
files a target called make
generated
is common. The
/usr/src/ tree can be found with a relative path or
with the BSDSRCDIR
variable.
Tests should generally not set or modify
MALLOC_OPTIONS
unless it is testing specific
behaviour that depends on a particular flag or combination of flags. When
running tests, the appropriate sysctl (vm.malloc_conf) could be used
instead.
The regression system originally came from NetBSD, with many tests added by OpenBSD since. The current Makefile framework was written by Artur Grabowski and Marc Espie for OpenBSD 3.1.
March 31, 2022 | OpenBSD-current |