-
Notifications
You must be signed in to change notification settings - Fork 0
libtap Version 2 Programmer's Manual
libtap lets you write tests in C/C++ that implement the Test Anything Protocol
#include <tap.h>
void tap_plan(unsigned tests, unsigned flags, FILE * output);
void tap_init(FILE * out);
void plan_tests(unsigned tests);
void tap_test_case_name(const char * name);
unsigned is(unsigned got, unsigned expected, const char *format, ...);
unsigned is_eq(const char *got, const char *expected, const char *format, ...);
unsigned is_cmp(const void *got, const void * expected, int (*compare)(const void *, const void *),
const char *(*to_str)(const void *), const char *format, ...);
unsigned is_strncmp(const char *got, const char *expected, unsigned len, const char *format, ...);
unsigned is_strstr(const char *got, const char *expected, const char *format, ...);
unsigned isnt(unsigned got, unsigned expected, const char *format, ...);
unsigned isnt_eq(const char *got, const char *expected, const char *format, ...);
unsigned ok(unsigned boolean, const char *format, ...);
void skip(unsigned tests_to_skip, const char *format, ...);
int diag(const char *format, ...);
int exit_status(void);
void * tap_dup(const void *mem, size_t size);
char * tap_shell(const char *command, int *status);
bool tap_create_file(const char *fn, const char *format, ...) // printf-like parameters
char * tap_read_file(const char *fn, size_t *size_out);
void plan_no_plan(void);
void plan_skip_all(const char *reason);
unsigned ok1(unsigned boolean);
unsigned pass(const char *format, ...);
unsigned fail(const char *format, ...);
skip_if(unsigned boolean, unsigned tests_to_skip, const char *format, ...)
skip_start(unsigned boolean, unsigned tests_to_skip, const char *format, ...)
skip_end
void todo_start(const char *format, ...);
void todo_end(void);
The libtap library provides functions for writing unit test programs in C or C++. The API is similar to that of perl's Test::More, a powerful unit test library used extensively to test perl modules. Test programs written using libtap produce output consistent with the Test Anything Protocol. Though not required, a test harness that parses this protocol can run these tests and produce useful reports summarizing their success or failure.
For all function that take const char * format, ... as their last two parameters, format is a printf-like format string, and is followed by one or more arguments that are values to be formatted using that string.
tap_plan should be called with the number of tests that will be run, flags (0 for none, or the sum of 1 or more of the following: TAP_FLAG_DEBUG, TAP_FLAG_ON_FAILURE_EXIT, or TAP_FLAG_LINE_ON_OK), and a file to send output to (NULL for stdout). TAP_FLAG_LINE_ON_OK adds test program line number to OK messages to make tracing where you are in a test program easier from its output.
Otherwise, init_tap should be called before calling the plan_tests function if you want to redirect the test output to a file. Otherwise, output will go to stdout. Then plan_tests should be called with the number of tests that will be run before any tests are run. This allows libtap to notice if any tests were missed, or if the test program exited prematurely. plan_tests will cause your program to exit prematurely if you specify 0 tests or it's called more than once.
Before calling a test function, you can set a test case name by calling tap_test_case_name.
Each of the following test functions prints a line that includes the test number, either ok or not ok, the test program source file and line, and a comment describing the test, created using the printf-like format argument and any arguments that follow it. It is good practice to use the comment to describe the purpose of the test rather than its implementation. For example:
ok(db != NULL, "db is not NULL"); /* Not bad, but.. */
ok(db != NULL, "Database connection succeeded"); /* this is better */
The is test function passes if its first two parameters, got and expected, are numerically equal. The parameters can be single characters, integers or pointers. If the test fails, the values of expected and got will be printed to help you identify the cause of the failure. isnt tests whether two numbers are not equal.
The is_eq test function passes if its first two parameters point to identical '\0' terminated strings. On failure, the strings are printed. isnt_eq tests whether two strings are not equal.
The is_cmp test function allows you to compare arbitrary objects by passing comparison and print functions along with pointers to the actual and expected values. The test passes if its first two parameters point to equivalent objects when compared with the function pointed to by its compare argument. compare should return zero for equality and non-zero if the objects are different. On failure, the objects are printed after formatting them with the function pointed to by the to_str argument.
The is_strncmp test function passes if its first two parameters point to identical '\0' terminated strings that are length limited by the third parameter. On failure, the strings are printed.
The is_strstr test function passes if its first two parameters point to '\0' terminated strings and the first (got) is a substring of the second (expected). On failure, the strings are printed.
The ok test function passes if its boolean parameter is true, and fails if it is false. This allows you to test an arbitrarily complex expression, but doesn't give you any helpful output if the test fails. For example:
ok(init_subsystem() == 0, "Second initialization should fail");
The skip function allows you to skip tests with a diagnostic message. You pass it tests_to_skip, the number of tests you're skipping, and format the reason you're skipping the tests. Sometimes tests cannot be run because the test system lacks some feature. You can explicitly document that you're skipping tests using skip. Tests should be skipped when they require optional modules that aren't installed, are running under an OS that doesn't have some necessary feature (like fork or symbolic links), or need an Internet connection when one isn't available.
For example, suppose some tests should be run as root. If the test program is not being run as root then the tests should be skipped. Skipped tests are flagged as being ok, with a special message indicating that they were skipped. You must ensure that the number of tests skipped (the first parameter to skip) is the correct number of tests that will be skipped.
if (getuid() != 0) {
skip(1, "because test only works as root");
}
else {
ok(do_something_as_root(), "Did something as root");
}
If your tests need to produce diagnostic output, you can use the diag function. It ensures that the output will not be ignored by the TAP test harness. It adds the trailing newline character for you. It returns the number of characters written. For example:
diag("Expected return code 0, got return code %d", rcode);
For maximum compatibility with TAP test harnesses, your test program should return the correct exit code. This is calculated by exit_status. Return it from the test program's main function or pass it to exit from anywhere in the test program. For example:
exit(exit_status());
In my programming practice at Cisco, I began collecting utility functions that were repeated across many test programs.
tap_dup duplicates memory in a malloced buffer. This is useful in test functions called as callback functions from code under test to record data for later examination by the test program.
tap_shell runs a command in a shell, capturing its output and returning it in a malloced buffer.
tap_create_file creates a file, specifying the contents with printf-like arguments. The file is created atomically to allow testing with things like inotify.
tap_read_file reads a file into a malloced buffer.
The following functions are kept for backward compatibility to libtap version 1.
plan_no_plan let's you use libtap without specifying the number of tests that you will run. This prevents libtap from giving an error if the test program exits before running all the tests it should have. plan_no_plan will cause your test program to exit prematurely with a diagnostic message if called more than once.
plan_skip_all let's you skip all tests. If your test program detects at run time that some functionality that the entire test program requires is missing (for example, if it relies on a database connection which is not present, or a particular configuration option that has not been included in the running kernel), you can use plan_skip_all instead of plan_tests. For example:
if (!have_some_feature) {
plan_skip_all("Need some_feature support");
exit(exit_status());
}
plan_tests(13);
The ok1 macro uses the test expression as the test comment. These calls are equivalent:
ok( i == 5, "i == 5");
ok1(i == 5);
Sometimes you just want to say that the tests have passed. Usually the case is you've got some complicated condition that is difficult to wedge into an ok. In this case, you can simply use pass (to declare the test ok) or fail (for not ok). Use these very, very, very sparingly.
skip_if is a macro provided for convenience. The body of the skip_if will only be executed if the condition is false. Otherwise, it will be skipped. For example:
skip_if(getuid() != 0, 1, "because test only works as root") {
ok(do_something_as_root() == 0, "Did something as root");
}
The macros skip_start and skip_end can be used to skip whole groups of tests. If the tests are skipped, none of the code between the macros will be executed.
Sets of tests can be flagged as being "todo" by surrounding them with calls to todo_start and todo_end. Unlike skip_start and skip_end, the code between todo_start and todo_end is executed.
The tests directory in the source code distribution contains tests of tap functionality, written using tap. Examine them for examples of how to construct test suites.
libtap is used to unit test the sxe open source project, available from github.com. There are hundreds of examples in that project.
libtap strives to be compatible with the perl Test::More and Test::Harness modules. The test suite verifies that tap is bug-for-bug compatible with their behaviour. This is why some functions which would more naturally return nothing return constant values.
If the libpthread library is found at compile time, libtap should be thread safe. Indications to the contrary (and test cases that expose incorrect behaviour) are welcome.
Test::More, Test::Harness, prove
libtap requires a isoC-99 compiler. It is implemented using variadic macros, and that functionality was not formally codified until C99. Patches to use libtap with earlier compilers that have their own implementation of variadic macros will be gratefully received.
libtap was written to help improve the quality and coverage of the FreeBSD regression test suite, and released in the hope that others would find it a useful tool to help improve the quality of their code.
Nik Clayton ([email protected],[email protected])
Jim Belton (my.name@gmail.com)
libtap would not exist without the efforts of Michael G Schwern ([email protected]), Andy Lester ([email protected]), and the countless others who have worked on the Perl QA programme.
Ideally, running the tests should have no side effects on the behaviour of the application you are testing. However, it is not always possible to avoid them. The following side effects of using tap are known:
-
stdout is set to unbuffered mode after calling any of the
plan_functions.