Using CLUnit: An Introduction and TutorialF. A. Adrian
|
![]() |
Insuring that code runs and continues to run after changes is extremely important in developing systems in a timely and effective manner. Unit testing is well known way to ensure that code works properly. Unit testing provides functions that are run to verify proper operation of the system under test. The CLUnit package provides a unit test environment for Common Lisp Implementations. It provides facilities for defining, organizing, and running tests.
CLUnit is provided under the terms of the LGPL. As such, it may be loaded and compiled into a product for general release. However, the code is distributed without any warranty (see the text of LGPL for all applicable licensing conditions) and the user assumes all risk of usage.
To start using CLUnit, load the file
CLUnit.lisp
:
(load "CLUnit.lisp")
In this document, we will show whatever the user types in bold and how the computer responds in normal face.
Dummy error occurred in test "Error test" 1 test run; 0 tests passed; 1 test failed. Dummy error occurred in test "Error test" 1 test run; 0 tests passed; 1 test failed. 2 tests run; 2 tests passed; 0 tests failed. 11 tests run; 11 tests passed; 0 tests failed. CLUnit self-test passed.
The first thing that CLUnit does is to run a self-test to ensure that it is running properly. The ancilary messages shown are outputs due to testing of error trapping and running of test sets. If the message "CLUnit self-test passed." is showing, then CLUnit is working!
Tests are defined using the
deftest
macro. In its simplest form, the macro takes a test
description and a test function that is invoked when the test is run.
If the test function returns a non-nil value, the test is assumed to
have succeeded. If it throws an error that escapes from the function
or returns nil, the test is assumed to have failed:
(deftest "Test car 1" :test-fn #'(lambda () (eq (car '(a b)) 'a))) #<ORG.ANCAR.CLUNIT::TEST Test car 1/*UNCATEGORIZED* #x1011B20> (run-named-test "Test car 1") T (run-all-tests) 1 test run; 1 test passed; 0 tests failed. T 0 1
As we can see, a test can be defined and run by name and all tests
thus far defined can be run. The
run-all-tests
function returns three values -- a boolean,
telling us if all tests have succeeded or not, the number of failed
tests and the number of tests that passed. If we define a test
that fails and then run-all-tests
again:
(deftest "Test car 2" :test-fn #'(lambda () (error "A test error"))) #<ORG.ANCAR.CLUNIT::TEST Test car 2/*UNCATEGORIZED* #x10AD190> (run-all-tests) A test error occurred in test "Test car 2" 2 tests run; 1 test passed; 1 test failed. NIL 1 1
In this case, the error is trapped and the
run-all-tests
function
returns the appropriate values.
Let's redefine the second test for a more graceful failure. By defining the test with the same name, the original test is replaced:
(deftest "Test car 2" :test-fn #'(lambda () nil)) #<ORG.ANCAR.CLUNIT::TEST Test car 2/*UNCATEGORIZED* #x10C85B0> (run-all-tests) Output did not match expected output in test "Test car 2" 2 tests run; 1 test passed; 1 test failed. NIL 1 1
Again, the error is detected. Note that, when a test fails, a
message is printed giving the description of the failed test and
the reason for failure. In addition, the
failed-tests
function will return a list of all tests that
failed:
(failed-tests) (#<ORG.ANCAR.CLUNIT::TEST Test car 2/*UNCATEGORIZED* # x10C85B0>)
Now we'll get rid of the failing test by using the
remove-test
function:
(remove-test "Test car 2") (#<ORG.ANCAR.CLUNIT::TEST Test car 1/*UNCATEGORIZED* #x1097258>) (run-all-tests) 1 test run; 1 test passed; 0 tests failed. T 0 1
And we see that the removed test is no longer in the system.
Finally, we can get rid of all of the tests by using the function
clear-tests
function:
(clear-tests) NIL (run-all-tests) 0 tests run; 0 tests passed; 0 tests failed. T 0 0
Categories are a useful tool to organize sets of tests. A category for
a test is specified by using the :category
keyword in the deftest
macro and tests
from a specific category are run using the
run-category
function:
(deftest "Test car of nil" :category "Test car" :test-fn #'(lambda () (eq (car nil) nil))) #<ORG.ANCAR.CLUNIT::TEST Test car of nil/Test car #x10F8630> (deftest "Test car of dotted cons" :category "Test car" :test-fn #'(lambda () (eq (car '(a . b)) 'a))) #<ORG.ANCAR.CLUNIT::TEST Test car of dotted cons/Test car #x110F688> (deftest "Another test" :category "General" :test-fn #'(lambda () t)) #<ORG.ANCAR.CLUNIT::TEST Another test/General #x11255C0> (run-category "Test car") 2 tests run; 2 tests passed; 0 tests failed. T 0 2 (run-category "General") 1 test run; 1 test passed; 0 tests failed. T 0 1 (run-all-tests) 3 tests run; 3 tests passed; 0 tests failed. T 0 3
By default, tests are categorized under the category named *UNCATEGORIZED*. Let's clear the tests again:
(clear-tests) NIL
All of the work of the tester can be done using only this functionality, but sometimes the testing becomes a bit clunky. Suppose we had a function returning multiple values:
(defun my-func () (values 1 2 3)) MY-FUNC (my-func) 1 2 3To test this function, a test would have to be written as follows:
(deftest "Test my-func" :test-fn #'(lambda () (equal (multiple-value-list (my-func)) '(1 2 3)))) #<ORG.ANCAR.CLUNIT::TEST Test my-func/*UNCATEGORIZED* #x103EA68> (run-all-tests) 1 test run; 1 test passed; 0 tests failed. T 0 1
In order to handle multiple values more gracefully, this test can be
written using the :output-fn
keyword:
(deftest "Test my-func" :test-fn #'my-func :output-fn #'(lambda () (values 1 2 3))) #<ORG.ANCAR.CLUNIT::TEST Test my-func/*UNCATEGORIZED* #x1096640> (run-all-tests) 1 test run; 1 test passed; 0 tests failed. T 0 1 (clear-tests) NIL
This feature can also be used for more mundane uses:
(deftest "Test car of list" :category "Test car" :test-fn (lambda () (car '(a b))) :output-fn #'(lambda () 'a)) #<ORG.ANCAR.CLUNIT::TEST Test car of list/Test car #x10CA018> (run-all-tests) 1 test run; 1 test passed; 0 tests failed. T 0 1
The expected output function is always a function of zero arguments
that returns the values that the output of the test function is
compared against. The comparison is done by turning each set of
values into a list using
multiple-value-list
and comparing the list with a comparison
function. This comparison function defaults to
#'equal
, but the user can
specify another comparison function using the
:compare-fn
keyword:
(clear-tests) NIL (deftest "Test compare-fn" :test-fn #'(lambda () "abc") :output-form "abc" :compare-fn #'(lambda (rlist1 rlist2) (reduce #'and (mapcar #'string-equal rlist1 rlist2)))) #<ORG.ANCAR.CLUNIT::TEST Test compare-fn/*UNCATEGORIZED* #xFB5500> (run-all-tests) 1 test run; 1 test passed; 0 tests failed. T 0 1
Note that the comparison function is comparing two lists holding the results of the output and test functions. As such, comparison functions can sometimes be quite hairy.
Rather than specifying the
:output-fn
keyword, one can use the
:output-form
keyword.
With respect to the test above, the test would change as follows:
(clear-tests) NIL (deftest "Test car of list" :category "Test car" :test-fn (lambda () (car '(a b))) :output-form 'a) #<ORG.ANCAR.CLUNIT::TEST Test car of list/Test car #x10E6370> (run-all-tests) 1 test run; 1 test passed; 0 tests failed. T 0 1
If both :output-fn
and
:output-form
keywords are
specified, the :output-fn
keyword takes precedence.
Analogously, :input-fn
and/or :input-form
functions
can be specified:
(clear-tests) NIL (deftest "Test +" :input-fn #'(lambda () (values 1 2 3)) :test-fn #'+ :output-form 6) #<ORG.ANCAR.CLUNIT::TEST Test +/*UNCATEGORIZED* #x1105888> (run-all-tests) 1 test run; 1 test passed; 0 tests failed. T 0 1
The test function is applied to the multiple values returned
by the input function. The :input-form
keyword returns only one value so the test function, in this
case, must be unary:
(clear-tests) NIL (deftest "Test car" :input-form '(a b) :test-fn #'car :output-form 'a) #<ORG.ANCAR.CLUNIT::TEST Test car/*UNCATEGORIZED* #x1121CD0> (run-all-tests) 1 test run; 1 test passed; 0 tests failed. T 0 1
Specifying an incorrect arity in the test function will result in an error:
(clear-tests) NIL (deftest "Test cons" :input-form '(a b) :test-fn #'cons :output-form '(a . b)) #<ORG.ANCAR.CLUNIT::TEST Test cons/*UNCATEGORIZED* #xFC0EB8> (run-all-tests) Wrong number of arguments occurred in test "Test cons" 1 test run; 0 tests passed; 1 test failed. NIL 1 0 (deftest "Test cons" :input-form (values 'a 'b) :test-fn #'cons :output-form '(a . b)) #<ORG.ANCAR.CLUNIT::TEST Test cons/*UNCATEGORIZED* #xFDA318> (run-all-tests) 1 test run; 1 test passed; 0 tests failed. T 0 1 (clear-tests) NIL
After a test failure, one may want to run a test outside the error
trapping code of run-all-tests
or run-category
to debug a
test failure. We use the function
run-named-test
to do this:
(run-named-test "Test cons") ;;; An error occurred in function _WRONG-NUMBER-OF-ARGS-ERROR: ;;; Error: Wrong number of arguments ;;; Entering Corman Lisp debug loop. ;;; Use :C followed by an option to exit. Type :HELP for help. ;;; Restart options: ;;; 1 Abort to top level. ;;; Debugging stuff...
The function
run-named-test
also takes an optional "protected mode" flag to allow the
user to try an individual test:
(run-named-test "Test cons" t) Wrong number of arguments occurred in test "Test cons" (NIL #<Simple-error #x101EDB8>)
This form of the
run-named-test
function returns two values, the first telling whether the function
succeeded, the second giving the error in case of failure.
Other miscellaneous functions include
list-tests
and
list-categories
:
(clear-tests) NIL (deftest "Test 1" :category "Category 1") #<ORG.ANCAR.CLUNIT::TEST Test 1/Category 1 #x1037060> (deftest "Test 1.2" :category "Category 1") #<ORG.ANCAR.CLUNIT::TEST Test 1.2/Category 1 #x1048DF0> (deftest "Test 2" :category "Category 2") #<ORG.ANCAR.CLUNIT::TEST Test 2/Category 2 #x105BB90> (list-tests) ("Test 2/Category 2" "Test 1.2/Category 1" "Test 1/Category 1") (list-tests "Category 1") ("Test 1.2/Category 1" "Test 1/Category 1") (list-categories) ("Category 1" "Category 2") (run-all-tests) 3 tests run; 3 tests passed; 0 tests failed. T 0 3
Note the other (somewhat useless) feature demonstrated above -- a test without a test function specified always succeeds.
string-equal
in the CLUnit
code. As such, case sensitivity is not necessarily an issue, but
proper spelling of category and test names (including embedded spaces)
is. It may be helpful to define constants for category names:
(defconstant +category-name+ "XXX") +CATEGORY-NAME+ (deftest "A test" :category +category-name+) #<ORG.ANCAR.CLUNIT::TEST A test/XXX #x1093B58>
remove-test
sparingly.
If the feature was important enough to test before, it's probably
important enough to tests now. Only in the cases when the
functionality has been removed is it usually necessary to remove the
test. When in doubt, redefine.run-named-test
and
run-category
sparingly. These
functions are provided for the convenience of running a subset of tests in
isolation for use in short development sessions. In general, you should run
all the system's unit tests as often as possible.
The more often you run all tests, the more often you insure that you
haven't broken anything else with your changes. Prefer
run-all-tests
.:input-fn
,
:input-form
,
:output-fn
,
and :output-form
to keep test
functions as simple as possible.