Re: Gathering comprehensive SRFI test suites in one place
Lassi Kortela 26 Jan 2020 19:49 UTC
Thanks for chiming in and explaining your SRFI, Per. Do you have many
existing tests for SRFIs in Kawa using the 64 framework? If so, those
could be a useful starting point for a portable collection.
> Typical output on Kawa:
>
> $ ../bin/kawa "./lib-test.scm"
> %%%% Starting test libs (Writing full log to "libs.log")
> # of expected passes 269
> # of expected failures 9
This matches the output I get on various Schemes.
> (including 1 line for each unexpected failure or pass)
I guess Peter and I expected the output to explicitly say that all tests
have produced the expected result and that's what confused us.
For what it's worth, I also found the other test frameworks' output a
bit tricky to read. I guess these things come down to personal
preference, hence configurable output formats are a win.
For portable SRFI tests, we should probably eventually write a backend
that outputs a standardized S-expression format which everyone can then
format as they please :)
> Kawa also has another test framework, which is useful for
> short simple tests: The file is just an executable module,
> with expected output in comments:
>
> (let ((x ::int 1))
> (set! x 0)
> (try-finally
> (if x (display "x is true")
> (display "x is not true"))
> (format #t "~%finally-clause executed~%")))
> ;; Output: x is true
> ;; Output: finally-clause executed
>
> Comments can use plain text or regexps, specify error messages,
> and compiler options. This format is convenient for simple
> regression tests, but matching error messages is likely
> to be implementation-specific.
This is interesting, but parsing comments and matching on error message
text would seem brittle for portable stuff.
> For a portable testing framework, I recommend srfi 64.
It's a fine and widely supported choice as far as I'm concerned.
> The reference implementation is a bit complicated because
> it has a lot of bells and whistles, partly for the sake
> of the detailed log written to the log file.
> However, a minimal implementation that does not support
> custom test runners would be quite simple.
This is good to know. Indeed, as far as I can tell almost all Scheme
unit test frameworks revolve around the same few primitives for writing
tests. It's just the runners that are quite different.