Philip McGrath <xxxxxx@philipmcgrath.com> writes:

> What is the line between what this SRFI wants to make portable, and
> what is to be left to the implementation?
>
> (snipped)
>
> Leaving the interpretation of "adjacent" up to the implementation
> makes documentation non-portable: if I write `(foo #|*doc*|# bar)`, I
> need to know, portably, whether I am documenting foo or bar. More
> realistically, the example:
>
>> ``` (define (add-one x)
>>   #|* Add one to x. *|# (+ x 1))
>> ```
>
> is not portable if it is treated as documentation for `add-one` by
> some implementations but for `(+ x 1)` by others.

The goal is to provide a convention that allows everyone to write
documentation in their preferred style, and an API that can be used by
tools.

A tool could be configured to handle comment-after-expression,
comment-before-expression, or comment-inside-expression. Perhaps a
variable (containing a symbol) should be introduced to specify which is
being used.

> There is a similar issue with the `documentation-format` parameter
> (though it doesn't seem to actually be used): the format documentation
> is written in is a property of the written documentation, not of the
> tool configuration. See [2] for a real (albeit small) problem caused
> by `the supplied <command> is` being parsed as plaintext in some
> contexts but markdown in others.
>
> [2]: https://github.com/cisco/ChezScheme/pull/816

documentation-format can be parameterized in multiple ways. Copying my
note from before:

Exporter programs can parameterize documentation-format based on the
value of a CLI flag. Exporter libraries can then use the parameterized
value. documentation-format can also be parameterized by code, by a
special string inside the documentation text, or the markup format can
be inferred from the documentation text.

See line 87 of doc-exporter.scm in the repo as a trivial example.

> In the other direction, I see little value in standardizing
> `read-documentation` or `read-documentation`, especially if they must
> be constrained to the lowest common denominator of R7RS Small without
> syntax objects. Tools seem very likely to want to use
> implementation-specific features, ... (snip)such as Chez Scheme's
> "annotations" [3] or Racket's syntax properties [4] and `srcloc`
> values [5]. (snip)

This is the purpose of the `documentation-alist` field of the
documentation record. Implementation-specific information can be placed
into it by the implementation when the `read-documentation` is called,
and made available to tools.

> I also have some concerns about the parts of the draft that do deal
> with writing, as opposed to processing, documentation.
>
> I am not aware of any existing documentation system that documents
> "expressions", and I am confused about the intent. (I also note that,
> in a language with macros, a `read`-based approach that doesn't
> perform macro expansion is really documenting source S-expressions,
> ...

Yes, source S-expressions are the subject of in-source documentation,
which is inherently text-based.

> not "expressions": macro expansion is needed to differentiate
> expressions per se from definitions, binding occurrences of variables,
> uses of syntax pattern variables with incorrect ellipsis depth, and
> other non-expressions.) Most documentation systems seem to document
> libraries and their exports: named functions, macros, structure types,
> and other values.
>
> There is no portable way to "import" extensions to lexical syntax, so
> they don't seem to me like a good solution in this case. A "magic"
> comment syntax avoids some of these problems, but not all, and, if
> enabled by default, can cause problems for source code that didn't
> expect the "magic", but expected all comments to be pure comments.
> While this is definitely a secondary concern, "magic" comments also
> require tool implementers to adjust their reader, at least in some
> modes, to avoid discarding comments.

The SRFI specifies that the reader's behavior must not be affected by
the magic:

The reader’s behavior shall not change when a documentation comment
sequence is encountered.

> As a "lowest common denominator", plain text (as comments or Scheme
> strings) definitely has a role to play in documentation. However, one
> of the major contributions from Scheme and the broader Lisp family of
> languages is rich support for creating embedded domain-specific
> languages. For documentation to *only* support character-based DSLs,
> whether in strings or "magic" comments, that must be parsed by
> external tools, seems to me like a weakness/restriction that ought to
> be removed.

As I mentioned to Wolfgang, this convention is compatible with attaching
documentation to identifiers via identifier properties or other
approaches.

> The docstring approach:
>
> On 5/25/24 07:35, Jakub T. Jankiewicz - jcubic at onet.pl (via
> srfi-discuss list) wrote:
>  >
>  > (define (foo x y)
>  >    "(foo x y)
>  >
>  >     x - this is a number y - this is a string" (+ x x))
>
> avoids the problems of "magic" comments, and it has the advantage of
> simplicity for plain text.

The docstring method requires implementations to provide special
`define` forms, and can only document expressions for which special
forms are provided. The comment method won't break implementations that
choose not to implement it, and can document any source S-expression
without having to implement special forms.

> I think a more promising route would be to
> allow syntactic extension. For example, a simple implementation might
> allow:
>
> ``` (define-syntax markdown-doc
>    (syntax-rules ()
>      ((_ str)
>       str))) (define (add x y)
>    (markdown-doc "Applies `+` to its two arguments.") (+ x y))
> ```
>
> and get the equivalent of a docstring by expanding to a string
> literal, while a different implementation might parse the markdown at
> a compile time and expand to implementation-defined primitive forms
> for documentation literals.

This introduces the concept of "phases" (documentation-time,
expand-time, compile-time, run-time), which the comment method was
intentionally made to avoid. I know Racket has a extensive phase system,
but this is intended as a lowest-common-denominator for any Scheme, so
long as they support block comments.

> There are two other resources I recommend to anyone thinking about
> this design space.
>
> (snip)
>
> Hope some of this is useful! Philip McGrath

Thank you for the review!