On 09/28/2015 12:55 PM, Taylan Ulrich Bayırlı/Kammer wrote:
> Per Bothner <xxxxxx@bothner.com> writes:

>> * I think I'd much rather see (current reference output):
>>
>> ./r7rs-tests.scm:494: FAIL
>>
>> than:
>>
>> [FAIL] R7RS/4.3 Macros/val
>>
>> though a combination might be even better:
>>
>> ./r7rs-tests.scm:494: FAIL (R7RS/4.3 Macros/val)
>>
>> Note having the line number first, and with this syntax, is both
>> traditional and handled better by tools such as Emacs.
>>
>> If line numbers aren't available, the following works:
>>
>> FAIL (R7RS/4.3 Macros/val)
>>
>> If you really prefer the square brackets, the following are acceptable:
>>
>> ./r7rs-tests.scm:494: [FAIL] R7RS/4.3 Macros/val
>> [FAIL] R7RS/4.3 Macros/val
>
> The source location is already reported as follows when available:
>
> Source:
> ./lib-test.scm:623
>
> That is an extra couple lines after the [FAIL] line.  I believe that
> should satisfy Emacs and other systems?

Maybe - probably not. Traditionally there is a colon followed by a message,
after the file:line - all on the same line:

./lib-test.scm:623: [FAIL] R7RS/4.3 Macros/val

Furthermore, using 3 lines for each "property" is excessive.
Instead of:

Expression:
(invoke a (quote getTargetAxisName))

Expected value:
field xxvalue

Got value:
field value

I'd rather see:

Expression: (invoke a (quote getTargetAxisName))
Expected value: field xxvalue
Got value: field value

and then a single blank line between tests,

The existing implementation print out these "properties"
even for passing tests.  I think at least the source position
should be in the log file, perhaps a single line:

file:line: [PASS] labels

>> * The standard output should only report unexpected failures (and passes),
>> so to make runs "more quiet".  That's the reason for the log file: to be able
>> to look at the details *when you want/need to*, not all the time.
>
> I just imitated the behavior I'm used to seeing from running 'make
> check' in most GNU programs.

DejaGnu only prints failing tests to the terminal.

>The rationale would probably be that the
> silence could become irritating when the test suite takes a long time
> and when no tests fail, which should be the normal case. ;-)

That is very non-Unix-y.  The traditional Unix philosophy is to say nothing
on success, and a terse message on failure.  I wouldn't go that far,
but a simple:
   NNN expected passes
is enough if all the expected tests pass (and no unexpected passes).

> In any case 'make check' (or whatever program is used to run the test
> suite) should probably report failure through a non-zero exit code.

Agree.

> After that the user can search for "FAIL", and the precise output format
> becomes mostly moot so long as all useful information is there in some
> way (though being Emacs-parsable is pretty neat).

That assumes you have a terminal emulator with a big scroll buffer.

You really don't want to write out 3000 lines of [PASS] to the terminal.

>> *I'd also skip (at least in standard output) the summary lines with count 0,
>> and the group begin/end messages.
>
> Hmm, I like to see a consistent format (same lines in same order every
> time, only counts changing),

You can do that in the log file.

> and the code is marginally simpler this
> way.  I'd like to keep the current format unless there's a good reason
> not to.

Unix is the reason - only output useful information, especially to the terminal.
"Unexpected passes: 0" is not useful information.

> By the way: testsuite/arr-test.scm lines 347 and 353 contain excess
> arguments to test-equal.  It thus tries to use an integer for the test
> name.

Oops.  Thanks.

> (This errors in my implementation; I guess the original somehow
> allows it although I don't think it should.)

Agreed.
--
	--Per Bothner
xxxxxx@bothner.com   http://per.bothner.com/