Per Bothner <xxxxxx@bothner.com> writes:

> On 09/28/2015 12:55 PM, Taylan Ulrich Bayırlı/Kammer wrote:
>> Per Bothner <xxxxxx@bothner.com> writes:
>>
>> The source location is already reported as follows when available:
>>
>> Source:
>> ./lib-test.scm:623
>>
>> That is an extra couple lines after the [FAIL] line.  I believe that
>> should satisfy Emacs and other systems?
>
> Maybe - probably not. Traditionally there is a colon followed by a message,
> after the file:line - all on the same line:
>
> ./lib-test.scm:623: [FAIL] R7RS/4.3 Macros/val
>
> Furthermore, using 3 lines for each "property" is excessive.
> Instead of:
>
> Expression:
> (invoke a (quote getTargetAxisName))
>
> Expected value:
> field xxvalue
>
> Got value:
> field value
>
> I'd rather see:
>
> Expression: (invoke a (quote getTargetAxisName))
> Expected value: field xxvalue
> Got value: field value
>
> and then a single blank line between tests,

OK, I now changed the format to:

[FAIL] group1: group2: group3: name
file.scm:line: <expression>
Expected value: <value>
Returned value: <value>
[blank line]

That keeps the invariant that lines start with [RESULT], reports the
file, line, and expression on one line in a familiar format, the
expected and returned values in one line each (and they neatly line
up!), and prints the one desired blank line.

Also the "foo: bar: baz" looks much nicer than "foo/bar/baz".  I stole
it from Guile.

I also dropped the group begin/end lines, since groups are made obvious
from the path labels.

> The existing implementation print out these "properties"
> even for passing tests.  I think at least the source position
> should be in the log file, perhaps a single line:
>
> file:line: [PASS] labels

For simplicity, I'd like to keep the same format in stdout and the log
file.  Is it really useful to see the location of a passing test?

>> I just imitated the behavior I'm used to seeing from running 'make
>> check' in most GNU programs.
>
> DejaGnu only prints failing tests to the terminal.

I guess most GNU programs don't use DejaGnu. :-)

Anyway, maybe familiarity was a weak argument on my side.  In any case I
think extended silence is not good.

>>The rationale would probably be that the
>> silence could become irritating when the test suite takes a long time
>> and when no tests fail, which should be the normal case. ;-)
>
> That is very non-Unix-y.  The traditional Unix philosophy is to say nothing
> on success, and a terse message on failure.  I wouldn't go that far,
> but a simple:
>   NNN expected passes
> is enough if all the expected tests pass (and no unexpected passes).

It still keeps the user in silence during the run of a long test suite.

I've been told that Perl's default, which is to print a dot for every
passing test, has been gaining acceptance.  I considered that, but it
would be unsuitable for a log file, and I don't want to complicate the
code much...

>> In any case 'make check' (or whatever program is used to run the test
>> suite) should probably report failure through a non-zero exit code.
>
> Agree.
>
>> After that the user can search for "FAIL", and the precise output format
>> becomes mostly moot so long as all useful information is there in some
>> way (though being Emacs-parsable is pretty neat).
>
> That assumes you have a terminal emulator with a big scroll buffer.
>
> You really don't want to write out 3000 lines of [PASS] to the terminal.

That's precisely why I resurrected logging.  I tend to run processes in
Emacs's async shell buffer where I can then trivially I-search for
"FAIL", but I figure some people have a terminal emulator-based workflow
where they can't easily retrieve the whole output of a command and
search through it.

Note that "Wrote log file ..." is the very last line of the output,
nicely directing the users there if they need it.

>> Hmm, I like to see a consistent format (same lines in same order every
>> time, only counts changing),
>
> You can do that in the log file.
>
>> and the code is marginally simpler this way.  I'd like to keep the
>> current format unless there's a good reason not to.
>
> Unix is the reason - only output useful information, especially to the
> terminal.  "Unexpected passes: 0" is not useful information.

Since those are just 3-5 extra lines at most, I don't think that
warrants changing the code to have different output between log file and
stdout.

If one day I separate out a logging test runner from the simple test
runner, I'll make sure to address your remaining concerns, but for now,
I think the new output I mentioned above is good enough.

By the way source information for Kawa has been fixed too now, with help
of the tip you gave in the other mail.

An updated testing.scm for Kawa is attached.

Thanks for the help & feedback!
Taylan