Per Bothner <xxxxxx@bothner.com> writes:

> I'm rather time-stressed until the end of this month, so I won't be able to fully
> evaluate this until then.
>
> One major difference I noticed:
>
> Consider (test-equal value expr).
> In your version the expr is wrapped in a lambda, and passed as a thunk.
> That has the advantage that the code size for the test-containing function
> is smaller, but the total code size is comparable - and on the JVM the
> generated .class files are slightly bigger.
>
> More importantly, you've changed evaluating an expression in-context
> to a procedure call.  There will be lots more procedures, more call overhead,
> worse cache locality, and more pressure on closure allocation machinery.
> Probably more compilation time.
>
> I don't know if that's a good trade-off (I'll have to think about it),
> but I suspect not.  (The more time it takes to run a test suite, the
> less frequently it will be run, which is not good.)

The changes I made brought a ~5 times code size reduction on Guile (from
2.5M to 440K for the external representation of the macro expand output
intermediate language) and a 20+ times compilation speed-up on Guile
(from 20s down to <1s to compile a 250-line file which exploded into a
ton of test cases via a macro).  This was apparently not particular to
Guile either, since the code size issue was most likely what caused the
build failure on Kawa where I blew the JVM's class size limit; that
issue disappeared around after these changes (I haven't made absolutely
sure but it's the most obvious deduction that it was that change that
fixed it).  I haven't tested compile times on Kawa.

Regarding run-time, both implementations run Kawa's 'make check' in
approximately 4 minutes.  The new implementation seems to have an edge
of ~10 seconds (faster) in absolute time.  Since only part of that time
is test suite overhead, the relative difference is probably a bit more.
I don't know what makes this difference, but evidently the thunks are
fine.  I also use SRFI-48 'format' gratuitously (for every executed test
case) which I would have expected to be slow actually.

I haven't tested run-time on Guile, or anything on Chibi and Larceny.
(On which two my implementation runs unchanged as an R7RS library.)

>> For simplicity, I'd like to keep the same format in stdout and the log file.
>
> Simplicity at the cost of a poor user experience is not a good tradeoff.
>
> You can do what you want of course.  However, if I were to replace
> the existing implementation used by Kawa, I would make a number of changes,
> staring with more quiet terminal output.
>
> But as I said, I'll be able to spend more time on this in November.
>
>> Is it really useful to see the location of a passing test?
>
> One reason is if there is a crash or some other uncaught error.
> In that case it is useful to see how far we've got before the failure.
>
>>> That is very non-Unix-y.  The traditional Unix philosophy is to say nothing
>>> on success, and a terse message on failure.  I wouldn't go that far,
>>> but a simple:
>>>    NNN expected passes
>>> is enough if all the expected tests pass (and no unexpected passes).
>>
>> It still keeps the user in silence during the run of a long test suite.
>
> That's ok.  You're running a batch program.  If I'm compiling a big
> program, I don't expect or want a lot of noise, either.

I'll try to address these issues eventually by splitting the terminal
and log output, making the former less verbose and the latter more.  I
see it as a low-priority issue though...

Taylan