New GoLisp Testing Framework
by Dave Astels
What was
When I started working on GoLisp I decided to write tests of the runtime/internals in Go using GoCheck, and tests of Lisp level behavior in Lisp. To that end I wrote a very simple testing framework with one function: describe, which wrapped a sequence of predicate expressions, evaluated them and reported errors if they evaluated to something falsy. Generally, this took the form of a series of equality checks; it worked, but didn’t communicate very well.
With the impending version 1.0 release, I decided it was time to improve this.
What is
The describe form is gone, replaced by more rspec like context and it forms.
(context label setup it-form…)
label Is a symbol or string (a string is preferred, but a symbol is support to make converting old tests easier). It gets used in output to identify the particular context form.
setup is a list of sexprs used to initialize the environment within which each test runs. They are evaluated sequentially before each test.
it-form… is a sequence of it forms, each of which contain a single test. For each one, a new environment is created, the setup expressions are evaluated in this new environment, and the it-form is evaluated (also in the now set-up environment).
(it label sexpr…)
As with context, label is a symbol or sting that serves to identify a particular it form in output. The context and it _label_s should be written to form a coherent statement. Some examples:
- “Logical AND” “performs a logical and of it’s arguments”
- “Logical AND” “short-circuits”
Since the context’s setup code builds out everything needed by the contained it-forms, the it-forms, themselves, should only contain assertions. And ideally, very few assertions. Just enough to prove the point.
The assertions are very simple and are documented below. Currently only the basics are supported, but I expect this to grow over time. In truth, any test could be built using these, but if you find yourself doing the same thing repeatedly, it might be worth adding as more specific assertion. if nothing else, your tests code will be easier to read and understand.
(assert-true actual)
If the result of evaluating actual is truthy the assertion passes, otherwise it fails.
(assert-false actual)
If the result of evaluating actual is falsy the assertion passes, otherwise it fails.
(assert-eq actual expected)
If the result of evaluating actual is eq to the result of evaluating expected the assertion passes, otherwise it fails.
(assert-neq actual expected)
If the result of evaluating actual is neq to the result of evaluating expected the assertion passes, otherwise it fails.
(assert-nil actual)
If the result of evaluating actual is nil the assertion passes, otherwise it fails.
(assert-not-nil actual)
If the result of evaluating actual is not nil the assertion passes, otherwise it fails.
(assert-error actual)
If the evaluating actual causes an error the assertion passes, otherwise it fails.
In all cases, actual and expected are evaluated only once but, of course, should ideally be side-effect free.
Example
As an illustration of the differences, consider this test file from pre-v1.0 that tests the interval function:
(describe interval
(== (interval 1 1) '(1))
(== (interval 1 2) '(1 2))
(== (interval 1 5) '(1 2 3 4 5)))
(describe interval-with-step
(== (interval 1 4 1) '(1 2 3 4))
(== (interval 1 9 2) '(1 3 5 7 9))
(== (interval 1 10 2) '(1 3 5 7 9))
(== (interval 0 100 10) '(0 10 20 30 40 50 60 70 80 90 100)))
(describe reverse-interval
(== (interval 3 1) '(3 2 1))
(== (interval 10 1 -2) '(10 8 6 4 2)))
(describe single-arg-interval
(== (interval 1) '(1))
(== (interval 10) '(1 2 3 4 5 6 7 8 9 10)))
And here is the v1.0 version
(context "interval"
()
(it "makes simple increasing sequences"
(assert-eq (interval 1 1) '(1))
(assert-eq (interval 1 2) '(1 2))
(assert-eq (interval 1 5) '(1 2 3 4 5))
(assert-eq (interval 1 5) '(1 2 3 4 5)))
(it "lets you use a step with a sign in the direction of the interval"
(assert-eq (interval 1 4 1) '(1 2 3 4))
(assert-eq (interval 1 9 2) '(1 3 5 7 9))
(assert-eq (interval 1 10 2) '(1 3 5 7 9))
(assert-eq (interval 0 100 10) '(0 10 20 30 40 50 60 70 80 90 100))
(assert-error (interval 1 10 -2))
(assert-error (interval 1 10 5.3)))
(it "supports decreasing sequences, with optional step"
(assert-eq (interval 3 1) '(3 2 1))
(assert-eq (interval 10 1 -2) '(10 8 6 4 2))
(assert-error (interval 10 1 2)))
(it "supports a simple version for 1...n sequences"
(assert-eq (interval 1) '(1))
(assert-eq (interval 10) '(1 2 3 4 5 6 7 8 9 10))))
Testing predicates is even nicer, since now you can do something like
(assert-true (predicate))
instead of
(== (predicate) #t)
Setup
It was mentioned above that each context has setup code that builds the environment in which each of its tests run. Here’s an example of that in use:
(context "scope"
(
(define a 5)
(define (foo a)
(lambda (x)
(+ a x)))
)
(it global-env
(assert-eq a 5))
(it lambda-env
(assert-eq ((foo 1) 5) 6)
(assert-eq ((foo 2) 5) 7)
(assert-eq ((foo 10) 7) 17)))
The definitions of a and foo take place in the context’s environment. This environment is unique for each test, so the effects of any modification of bindings (using set! for example) would be limited to individual tests.
Running tests
You run tests by executing golisp in test mode, using the -t command line flag, and telling it what to run.
To run a single test file:
>:golisp -t tests/interval_test.lsp
Ran 15 tests in 0.002 seconds
15 passes, 0 failures, 0 errors
and to run all test files in a directory (files that match *_test.lsp):
>:golisp -t tests
Ran 949 tests in 0.107 seconds
949 passes, 0 failures, 0 errors
These commands produce only a summary. If failures or errors occur, they get output as well:
>:golisp -t tests/interval_test.lsp
Ran 13 tests in 0.003 seconds
11 passes, 1 failures, 1 errors
Failures:
interval lets you use a step with a sign in the direction of the interval:
(assert-eq (interval 1 10 2) '(1 3 5 7 9 10))
- expected (1 3 5 7 9 10), but was (1 3 5 7 9)
Errors:
interval lets you use a step with a sign in the direction of the interval:
ERROR: The sign of step has to match the direction of the interval
If you add the -v flag, you will get a fully detailed report, for example:
>:golisp -t -v tests/interval_test.lsp
interval
makes simple increasing sequences
(assert-eq (interval 1 1) '(1))
(assert-eq (interval 1 2) '(1 2))
(assert-eq (interval 1 5) '(1 2 3 4 5))
(assert-eq (interval 1 5) '(1 2 3 4 5))
lets you use a step with a sign in the direction of the interval
(assert-eq (interval 1 4 1) '(1 2 3 4))
(assert-eq (interval 1 9 2) '(1 3 5 7 9))
(assert-eq (interval 1 10 2) '(1 3 5 7 9 10))
- expected (1 3 5 7 9 10), but was (1 3 5 7 9)
ERROR: The sign of step has to match the direction of the interval
supports decreasing sequences, with optional step
(assert-eq (interval 3 1) '(3 2 1))
(assert-eq (interval 10 1 -2) '(10 8 6 4 2))
(assert-error (interval 10 1 2))
supports a simple version for 1...n sequences
(assert-eq (interval 1) '(1))
(assert-eq (interval 10) '(1 2 3 4 5 6 7 8 9 10))
Ran 13 tests in 0.002 seconds
11 passes, 1 failures, 1 errors
Failures:
interval lets you use a step with a sign in the direction of the interval:
(assert-eq (interval 1 10 2) '(1 3 5 7 9 10))
- expected (1 3 5 7 9 10), but was (1 3 5 7 9)
Errors:
interval lets you use a step with a sign in the direction of the interval:
ERROR: The sign of step has to match the direction of the interval
With no failures or errors:
interval
makes simple increasing sequences
(assert-eq (interval 1 1) '(1))
(assert-eq (interval 1 2) '(1 2))
(assert-eq (interval 1 5) '(1 2 3 4 5))
(assert-eq (interval 1 5) '(1 2 3 4 5))
lets you use a step with a sign in the direction of the interval
(assert-eq (interval 1 4 1) '(1 2 3 4))
(assert-eq (interval 1 9 2) '(1 3 5 7 9))
(assert-eq (interval 1 10 2) '(1 3 5 7 9))
(assert-eq (interval 0 100 10) '(0 10 20 30 40 50 60 70 80 90 100))
(assert-error (interval 1 10 -2))
(assert-error (interval 1 10 5.3))
supports decreasing sequences, with optional step
(assert-eq (interval 3 1) '(3 2 1))
(assert-eq (interval 10 1 -2) '(10 8 6 4 2))
(assert-error (interval 10 1 2))
supports a simple version for 1...n sequences
(assert-eq (interval 1) '(1))
(assert-eq (interval 10) '(1 2 3 4 5 6 7 8 9 10))
Ran 15 tests in 0.002 seconds
15 passes, 0 failures, 0 errors
Getting meta
I once wrote a blog post that is still as relevant as ever. It was called “One Assert Per Test” and argued that each test should ideally consist of a single assertion. Some people realized at the time that I was taking the idea to it’s logical extreme to make a point. The intent isn’t that you quite literally have a single assert statement in each test. Rather, each test should be razer focused on a single aspect of the system of which you are describing the behavior. With good factoring, you will find that a single assertion is quite natural.
The key to this is to keep your contexts focused. Create some state of the world (using the setup code) and then make simple statements about it.
The above example of this isn’t great at showing this since it was used partly to demonstrate how to port from the previous framework which didn’t support isolated test context with setup code.
Summary
After converting all of GoLisp’s tests over to the new framework, they read and communicate much better than before. The effort to convert them was relatively minor, and well worth it. As a bonus, there are now tests for error conditions.
One final note: the old framework was implemented in Go as primitives and special forms. The new one is written in GoLisp (other than the command line handling), making heavy use of macros. You can find it in the testing.lsp file in the repo. Note: testing.lsp is loaded automatically when you run golisp in test mode. You can, of course, manually load it into the REPL and evaluate the test running functions by hand.
I hope you enjoy the new framework.