Smokey API

A simple infrastructure for writing and running smoke tests. More...

A simple infrastructure for writing and running smoke tests.

Smokey is based on the Copperplate API, therefore is available over the single and dual kernel Xenomai configurations indifferently.

The API provides a set of services for declaring any number of test plugins, embodied into a test program. Each plugin usually implements a single smoke test, checking a particular feature of interest. Each plugin present in the running executable is automatically detected by the Smokey init routine. In addition, the Smokey API parses all arguments and options passed on the command line to the executable, running pre-defined actions which are therefore automatically recognized by all programs linked against the Smokey library.

Writing smoke tests with Smokey

A smoke test is composed of a routine which implements the test code, and a set of runtime settings/attributes for running such code. The routine prototype shall be:

int run_<test_name>(struct smokey_test *t, int argc, char *const argv[])

The test routine should return a zero value for success, or any negated POSIX error code for indicating the failure to the test driver (e.g. -EINVAL if some value is found to be wrong).

With t referring to the Smokey test descriptor, and argc, argv the argument count and vector expunged from all the inner options which may have been previously interpreted by the Smokey API and inner layers (such as Copperplate).

The Smokey API provides the services to declare a complete test (named foo in this example) as follows:

#include <smokey/smokey.h>

smokey_test_plugin(foo, // test name
                   SMOKEY_ARGLIST( // argument list
                        SMOKEY_INT(some_integer),
                        SMOKEY_STRING(some_string),
                        SMOKEY_BOOL(some_boolean),
              ),
                   // description
              "A dummy Smokey-based test plugin\n"
              "\taccepting three optional arguments:\n"
              "\tsome_integer=<value>\n"
              "\tsome_string=<string>\n"
              "\tsome_bool[=0/1]\n"
);

static int run_foo(struct smokey_test *t, int argc, char *const argv[])
{
     int i_arg = 0, nargs;
     char *s_arg = NULL;
     bool b_arg = false;

        nargs = smokey_parse_args(t, argc, argv);

     if (SMOKEY_ARG_ISSET(foo, some_integer))
        i_arg = SMOKEY_ARG_INT(foo, some_integer);
     if (SMOKEY_ARG_ISSET(foo, some_string))
        s_arg = SMOKEY_ARG_STRING(foo, some_string);
     if (SMOKEY_ARG_ISSET(foo, some_boolean))
        b_arg = SMOKEY_ARG_INT(foo, some_boolean);

     return run_some_hypothetical_smoke_test_code(i_arg, s_arg, b_arg);
}

As illustrated, a smoke test is at least composed of a test plugin descriptor (i.e. smokey_test_plugin()), and a run handler named after the test.

Test arguments

Smokey recognizes three argument declarators, namely: SMOKEY_INT(name) for a C (signed) integer, SMOKEY_BOOL(name) for a boolean value and SMOKEY_STRING(name) for a character string.

Each argument can be passed to the test code as a name=value pair, where name should match one of the declarators. Before the test-specific arguments can be accessed, a call to smokey_parse_args() must be issued by the test code, passing the parameters received in the run handler. This routine returns the number of arguments found on the command line matching the an entry in SMOKEY_ARGLIST().

Once smokey_parse_args() has returned with a non-zero value, each argument can be checked individually for presence. If a valid argument was matched on the command line, SMOKEY_ARG_ISSET(test_name, arg_name) returns non-zero. In the latter case, its value can be retrieved by a similar call to SMOKEY_ARG_INT(test_name, arg_name), SMOKEY_ARG_STRING(test_name, arg_name) or SMOKEY_ARG_BOOL(test_name, arg_name).

In the above example, passing "some_integer=3" on the command line of any program implementing such Smokey-based test would cause the variable i_arg to receive "3" as a value.

Pre-defined Smokey options

Any program linked against the Smokey API implicitly recognizes the following options:

• –list[=<id[,id...]>] dumps the list of tests implemented in the program to stdout. This list may be restricted to the tests matching the optional regular expression (see –run). The information given includes the description strings provided in the plugin declarators (smokey_test_plugin()). The position and symbolic name of each test is also issued, which may be used in id specifications with the –run option (see below).

Note

Test positions may vary depending on changes to the host program like adding or removing other tests, the symbolic name however is stable and identifies each test uniquely.

• –run[=<id[,id...]>] selects the tests to be run, determining the active test list among the overall set of tests detected in the host program. The test driver code (e.g. implementing a test harness program on top of Smokey) may then iterate over the smokey_test_list for accessing each active test individually, in the enumeration order specified by the user (Use for_each_smokey_test() for that).

  If no argument is passed to –run, Smokey assumes that all tests detected in the current program should be picked, filling smokey_test_list with tests by increasing position order.

  Otherwise, id may be a test position, a symbolic name, or a range thereof delimited by a dash character. A symbolic name may be matched using a glob(3) type regular expression.

  id specification may be:

  • 0-9, picks tests #0 to #9
  • -3, picks tests #0 to #3
  • 5-, picks tests #5 to the highest possible test position
  • 2-0, picks tests #2 to #0, in decreasing order
  • foo, picks test foo only
  • 0,1,foo- picks tests #0, #1, and any test from foo up to the last test defined
  • fo* picks any test with a name starting by "fo"
• –exclude=<id[,id...]> excludes the given tests from the test list. The format of the argument is identical to the one accepted by the –run option.
• –keep-going sets the boolean flag smokey_keep_going to a non-zero value, indicating to the test driver that receiving a failure code from a smoke test should not abort the test loop. This flag is not otherwise interpreted by the Smokey API.
• –verbose[=level] sets the integer smokey_verbose_mode to a non-zero value, which should be interpreted by all parties as the desired verbosity level (defaults to 1).
• –vm gives a hint to the test code, about running in a virtual environment, such as KVM. When passed, the boolean smokey_on_vm is set. Each test may act upon this setting, such as skipping time-dependent checks that may fail due to any slowdown induced by the virtualization.

Writing a test driver based on the Smokey API

A test driver provides the main() entry point, which should iterate over the test list (smokey_test_list) prepared by the Smokey API, for running each test individually. The for_each_smokey_test() helper is available for iterating over the active test list.

When this entry point is called, all the initialization chores, including the test detection and the active test selection have been performed by the Smokey API already.

Issuing information notices

The printf-like smokey_note() routine is available for issuing notices to the output device (currently stdout), unless –silent was detected on the command line. smokey_note() outputs a terminating newline character. Notes are enabled for any verbosity level greater than zero.

Issuing trace messages

The printf-like smokey_trace() routine is available for issuing progress messages to the output device (currently stdout), unless –silent was detected on the command line. smokey_trace() outputs a terminating newline character. Traces are enabled for any verbosity level greater than one.

Therefore, a possible implementation of a test driver could be as basic as:

#include <stdio.h>
#include <error.h>
#include <smokey/smokey.h>

int main(int argc, char *const argv[])
{
    struct smokey_test *t;
    int ret;

    if (pvlist_empty(&smokey_test_list))
        return 0;

     for_each_smokey_test(t) {
        ret = t->run(t, argc, argv);
        if (ret) {
                if (smokey_keep_going)
                        continue;
                error(1, -ret, "test %s failed", t->name);
        }
        smokey_note("%s OK", t->name);
     }

     return 0;
}