proosal: removing test-result objects Per Bothner (03 Apr 2005 08:11 UTC)
Re: proosal: removing test-result objects Donovan Kolbly (04 Apr 2005 14:18 UTC)
Re: proosal: removing test-result objects Per Bothner (05 Apr 2005 07:24 UTC)
Re: proosal: removing test-result objects Donovan Kolbly (05 Apr 2005 16:11 UTC)

Re: proosal: removing test-result objects Donovan Kolbly 05 Apr 2005 16:11 UTC

On Tue, 5 Apr 2005, Per Bothner wrote:

> Donovan Kolbly wrote: > So I'd probably go the other way and make test
> results (or, more likely,
> > test cases) first class, something like so:
> >
> >   (test-case [runner])  ->  current or most recent test case, or #f if none
> >
> >   (test-case-kind tcase) -> one of (pass xpass ...) [*]
> >   (test-case-passed? tcase) ; same as (memq (test-case-kind tcase) '(pass xpass))
> >   (test-case-property tcase 'property [value])   ; property setter/getter
> >   (test-case-name tcase) -> test name
> >
> > [* although 'kind' is looking like a strange name here; maybe
> > "disposition"?]
>
> I think you're being inprecise wrt test-case vs test-result.
> A test-case is something you execute and the result is a
> test-result (or a test-case-result).  I might re-run the asme
> test-case multiple times, in a different environment.  It's then
> a single test-case, but multiple results.

Perhaps the term "test case occurrence" or "instance" would be more
precise.  I do mean to refer to the concrete objects at the leaves of the
test tree (as opposed to the abstract test cases which may be present in
the test suite source code and possible reused in various ways).

I also meant to make a distinction from the *result* which is essentially
the output of the test case occurrence specifically because you want to
hook in before the occurrence executes and affect the result (e.g., by
deciding to skip it).

BTW, in the subproposal, how are you thinking someone would write a runner
that decides to skip some subsequent tests?  Would the runner itself,
during the on-test hook, invoke (test-skip (test-match-nth 1 999999))?

I guess if you want to make that kind of decision before running a test
case instead of after, you would make a custom specifier.  For example,
let's say I decide to skip all tests that have the substring "(only if
ABC)" in their name, e.g., because the ABC test failed, I might execute a
test-skip with a custom specifier when ABC fails.

>  Though perhaps you
> could argue if it's run with a different environment, it's really
> a different test-case.  Still, I don't think using test-case to
> mean a result is a good name.
>
> So the question: Is it useful to make test results first-class?
> Perhaps.  I can see some use cases: a testrunner that just saves
> every result.  Then perhaps you can have an interactive GUI,
> where you can click on a result to see details, for example.

Interestingly, that's exactly how my xmlrunner works.  I haven't tried to
port it to the bare API as a functionality test...

A post-mortem live debugger might be another use case.  e.g., when you run
the test suite the runner captures the context of any error conditions and
then you can drill in to debug any case that errored out.

Well, anyway, all that is still possible without first-class test case
occurrences in the API.  A custom runner can wrap things into whatever
first-class objects it wants, as long as the hooks are there.  And I don't
think there's anything in the subproposal to prevent an implementation
from using such a representation internally.

--
-- Donovan Kolbly                    (  xxxxxx@rscheme.org
				     (  http://www.rscheme.org/~donovan/