> My example is not a test that fails to report a violation (this can
> always happen); it is a test that reports a violation where there is
> none.  This is far more serious and just shows that the test is wrong.

Aha, what you're saying is that your definition of `member` handles the
arity error in its own custom way. But `(test-error arity-error? (member
'a '(b c d) eq? 'junk))` will be a failing test because arity-error?
doesn't catch the custom style of error handling. That's a fair point.

>     Another example is, an arity error when calling `foo` does not
>     necessarily come from passing an incorrect number of args to `foo`. It
>     could also come from `foo` internally passing incorrect args to some
>     helper procedure. Or the outer call to `foo` has correct args but `foo`
>     calls itself recursively with incorrect args, etc. If we wanted to
>     handle all nuances like this, the tests would probably become so
>     verbose
>     that no-one would want to write them.
>
> That's why I am saying that `arity-error?' isn't helpful for checking
> that a procedure does not accept the wrong number of arguments.

It ought to be helpful in the cases (hopefully most of them) where
custom rest-arg destructuring tricks are not used, and a native facility
like `case-lambda`, `lambda/kw` or `lambda*` is used instead to define
optional/keyword args. `case-lambda` is even in RnRS.

> For a particular implementation, `arity-error?' may be helpful, but not
> for a test suite that is supposed to be (mostly) portable.  A test using
> `arity-error?' would depend on implementation details of the procedures
> being tested (see the examples above), which goes against the very basic
> principles of good tests.

I think a good test is whatever best avoids incorrect behavior. If that
means deliberately writing tests that exercise known tricky bits of
particular implementation(s), that's fine. If some implementations do
those internals differently, the offending tests can be turned off for
those implementations. Portability and standardization are useful in
degrees - a strict black-and-white approach is not always fruitful
(indeed, it is generally impossible to prove absolute conformance to a
standard when dealing with computers).