On Mon, 15 Aug 2005, Per Bothner wrote:

> But first I'd like some feedback on an API for "test results":
>
> A test runner maintains a set of "result properties" associated with
> the current or most recent test.  (I.e. the properties of the
> most recent test are available as long as a new test hasn't started.)
>
> Each property has a name (a symbol) and a value (any value).
> Some properties are standard or set by the implementation.

This seems like a good general idea.

>
> (test-result-ref [runner] 'pname [default])

Hmm.  I think statically ambiguous interfaces are confusing.  If you see
the following (admittedly poorly written) code fragment:

    (test-result-ref x y)

you will need much more information (perhaps located arbitrarily far away
in the program) to understand what is actually being done.  Is the
property denoted by y of runner x being accessed, or is property x of the
current runner being accessed, with a default value of y?

And, of course, the compiler has to defer the disambiguation to runtime as
well.

> Returns the property value associated with the pname property name.
> If there is no value assocate with 'pname return default,
> or #t if default isn't specified.

Why #t?  I would think that #f, as the Most Distinguished Value, is a more
useful default default.

>
> (test-result-set! [runner] 'pname value)
> Sets the property value associated with the pname property name to value.
>
> (test-result-clear [runner] ['pname])

Same ambiguity problem as mentioned above.

> If 'pname is specified, remove the property with the name 'pname.
> If 'pname is not specified, remove all result properties.
> The implementation automatically calls test-result-clear
> at the start of a test-assert and similar procedures.

Do we need to say anything about when the standard properties might be
"added"?  My own implementation will not explicitly store standard
properties, so in effect they'll (mostly) always be present.

>
> (test-result-alist [runner])
> Returns an association list of the current result properties.
> It is unspecified if the result shares state with the test-runner.
> The result should not be modified; on the other hand the result
> may be inclicitly modified by future test-result-set! or
> test-result-clear calls, with one exception: A test-result-clear
> without a 'pname doesn't not modify the returned alist.  Thus you
> can "archive" result objects from previous runs.
> Issue: Or should we just say that test-result-alist returns a
> fresh alist?

I like archiving, but requiring a fresh alist is overly constraining.
What if my implementation can share structure where possible (sort of a
copy-on-write thing)?  I'd hate to always have to clone the alist...

>
> We also need a list of standard properties:
> 'kind
>  One of 'pass, 'xpass, 'fail, 'xfail, 'skip
>
> 'source-file
> 'source-line
> Location of test-assert etc in test suite source code.

+                  if meaningful/available

>
> 'source-form
> The source form, if meaningful/available.
>
> 'expected-result
> The expected result, if meaningful/available.
>
> 'actual-result
> The actual result, if meaningful/available.

I suggest the addition of (possibly optional) timestamps, including the
execution time duration and the absolute start time.

> The test-runner call-back functions (such as that bound using
> test-runner-on-test) is modified to only take a test-runner,
> rather than also take an explicit test-result, since the latter
> is now part of the former.

I'll try tweaking my meta-test suite to take these new concepts into
account...

-- Donovan Kolbly                    (  xxxxxx@rscheme.org
 				     (  http://www.rscheme.org/~donovan/