On Wed, Mar 28, 2018 at 9:13 PM,  <xxxxxx@gmx.net> wrote:
>> Yes, I know about that, and its the first link on my list.
>
> Read again, https://ecraven.github.io/r7rs-coverage
>
> coverage, not benchmarks ;)

Sorry I miss-read.  :)

Indeed it ticks all my "requirements", as in they are simple to
implement support for them.

Are you open for a collaboration in filling them in?
And also creating a "small" standard to describe such tests?

I have extensive tests for some functions you are currently missing,
like you can see bellow:

  https://github.com/cipriancraciun/vonuvoli-scheme/blob/master/tests/scheme/control.sst
  https://github.com/cipriancraciun/vonuvoli-scheme/blob/master/tests/scheme/ports-memory.sst
  https://github.com/cipriancraciun/vonuvoli-scheme/blob/master/tests/scheme/ports-native.sst
  https://github.com/cipriancraciun/vonuvoli-scheme/blob/master/tests/scheme/quotation.sst
  https://github.com/cipriancraciun/vonuvoli-scheme/blob/master/tests/scheme/strings.sst

Other tests are found at:

  https://github.com/cipriancraciun/vonuvoli-scheme/blob/master/tests/scheme

(Symbols starting with `~` are non R7RS which can be removed;  and
there are some extensions particular for my implementation like
`#void` and `#string( ... )` which can be easily converted.)

I like the fact that you can "group" the tests, and the report you
generate based on them, but it seems it relies on specially crafted
comments.

If you are open I would propose two minor additions (and a few clarifications):

* base the grouping on s-expressions, something like `(test-group NAME
test-statement ...)`;  (where NAME can be any valid S-expression, and
if it is a symbol it should be quoted;  this is so that one can
implement it with a plain lambda;)

* introduce a few testing primitives for three (or four) common cases:

  * `(test= <expression> <expected> [<equal?>])` -- which would allow
the test harness to print something like: `test returned x, expected
y`;
  * `(test <expression> [<predicate?>])` -- which would be an
extension equivalent to `(test (<predicate?> <expression>))` but would
allow the harness to print something like `expression returned x which
doesn't satisfy <predicate?>);
  * `(test-error <expression> [<predicate?>])`;

* the `test` constructs have to be found at the top level;  (so that
one can parse and transform them into test cases native to the system
at hand without knowing anything about the tested language;)
* other statements are allowed at top level;
* everything is executed in top-down manner;

Implementing these constructs is trivial, and doesn't require anything
else except lambdas.  Moreover the clarifications allow the test files
to be treated just like simple Scheme source files, meanwhile also
allowing native test harnesses to create specific test cases.

I use the prefix `test` instead of `assert`, because most likely many
Scheme implementations would have `assert` as actual run-time
assertions, and using them would mean that the test harness would have
to "override" their meaning.  Moreover even inside tests, one could
use `assert` to check for invariants.  (Think of a loop that computes
a value:  the `test` construct should check that the final outcome is
correct, meanwhile the `assert` construct could be used to double
check some internal invariants at each cycle.)

I know about SRFI 64 and 78, and in fact my three constructs seem
similar to the simple ones proposed in SRFI 64, although my proposed
variant is compatible to what we both already have in our tests (and
it feels much more natural to say `(test <expression> <expected>)`
than the other way around `(test <expected> <expression>)`).

Ciprian.