Thien-Thi Nguyen wrote:
> Per Bothner <xxxxxx@bothner.com> writes:
>   >>Having a standard API makes this easier.
>
> but having a standard external (text) test report format is best.

Useful, yes, but not as importat as the API for writing test cases.

What one needs is a mechanism to so one notices and can investigate
if there is a change in test results.  A simple 'diff' will do that,
in a pinch.   But the test-runner framework does allow you to
customize the test report format.

The output format of the default "simple" test-runner is based on what
I'm used to from gcc and other dejagnu-based test suites:

$ make check-num
kawa -f "/home/bothner/Kawa/head/testsuite/num-test.scm"
%%%% Starting test numbers  (Writing full log to "numbers.log")
/home/bothner/Kawa/head/testsuite/num-test.scm:3: FAIL
# of expected passes      1663
# of unexpected failures  1

The "numbers.log" file has the complete test results in gory detail.
The line number in the FAIL line is nice but it does admittedly have
the problem that not all implementations can provide it.

The detailed log:

%%%% Starting test numbers
Group begin: numbers
Test begin:
   source-file: "/home/bothner/Kawa/head/testsuite/num-test.scm"
   source-line: 3
   source-form: (test-approximate 1.5 (sqrt 2) 0.02)
Test end:
   result-kind: fail
   actual-value: 1.4142135623730951
   expected-value: 1.5
Test begin:
   source-file: "/home/bothner/Kawa/head/testsuite/num-test.scm"
   source-line: 4
   source-form: (test-eqv 9223372036854775808 (- (- (expt 2 63))))
Test end:
   result-kind: pass
   actual-value: 9223372036854775808
   expected-value: 9223372036854775808
....
Group end
# of expected passes      1663
# of unexpected failures  1

[The failure is because I deliberately changed the expected value 1.4
to 1.5.]
--
	--Per Bothner
xxxxxx@bothner.com   http://per.bothner.com/