Donovan Kolbly wrote:
> So that, suppose:
>
>    (test-begin "a")
>    (test-begin "b")
>    (test-assert "x" #t)
>
> then in an on-test hook executing for the test-assert, we have:
>
>    (test-runner-test-name runner) ==> "x"
>    (test-runner-group-path runner) ==> ("a" "b")

Yes.

> Although, then a custom runner would have no way to know about an empty
> group.
>
> Would you intend that, for a custom runner, the following are
> indistinguishable:
>
>    (test-begin "a")
>    (test-assert "x" #t)
>    (test-end)
>    (test-begin "a")
>    (test-assert "y" #t)
>    (test-end)
>
> vs:
>
>    (test-begin "a")
>    (test-assert "x" #t)
>    (test-assert "y" #t)
>    (test-end)

I do think we should have call-back routins for test-begin/test-end.
For example a test-runner might want to write in a log:
[Entering group a]
Test x PASS.
[Exiting group a]
[Entering group a]
Test y PASS.
[Exiting group a]

> A "test suite" is a collection of "test cases" optionally organized into a
> hierarchy of "test groups".  A "test group" may be either implict, as
> introduced by test-begin and terminated by test-end, or explicit, as
> introduced by test-group.

Agreed.

> Furthermore, note that the tests within an
> explicit test group may be skipped using test-skip, but this is not true
> for tests within an implicit test group.
>
> [I think it's wierd that you can't skip an implicit test group, just for
> symmetry.  Although I can't see implementing it so that all the forms are
> skipped, it might make sense to actually skip *tests* inside a skippable
> implicit group.]

I'm inclined to agree.  I don't think the reference implementation does
it this way, but it should be difficult to fix.

For test-match-nth, should be count both the test-group/test-begin *and*
(assuming the test-group/test-begin is not skipped) the tests in it?

(test-skip 2) ;; Define this to mean skipping the 2nd following test.
(test-begin "a")
(test-assert "x1")
(test-assert "x2")
(test-end "a")
(test-assert "x3")

Should we skip "x2" or "x3"?  I.e. do we count 1 for "a" as a unit, or 1
for "a" and 1 each for "x1"..."x3".  The latter might be a little strange,
but perhaps more convenient - and easier to implement, since we can use
a single global counter.

> For the purposes of test counting as checked by test-end, a test group (of
> either variety) counts as a single test.

I'm open to discussion on this: perhaps it should count as a single test
*if it is skipped*.

 > Note that the registered on-test
> procedure is not involved in test group boundaries, which means that the
> sum of (test-runner-*-count) may be larger than the number of calls to the
> on-test procedure.

Yes, I think so.  I we add test-begin/test-end call-backs, as I think we
should, we might be able to be more specific.

 > Furthermore, if an explicit test group is skipped, the
> group as a whole is skipped, not each test within it, which means that
> (test-runner-skip-count) may be *smaller* than if the tests were
> individually skipped.

This seems unavoidable.
--
	--Per Bothner
xxxxxx@bothner.com   http://per.bothner.com/