> As long as the text of the SRFIs doesn't change, the APIs won't
 > change, so we can make the changes without bothering the authors.

This is the biggest relief to us :) If we are permitted to change
markup at will, it's just a matter of hashing out the technical
approach (and any approach will be an improvement).

 > I'm okay if what we do breaks some authors' tools, but I'd like us
 > to try to minimize that.

What if we had a script that uses one of those popular tagsoup
libraries (lenient HTML -> strict HTML) and then does post-processing
to generate something close enough to the standardized (X)HTML
structure we want? If the script worked well enough, that could be
low-friction enough for SRFI authors.

 > I've settled on https://github.com/ericchiang/pup and exported the
 > whole thing as JSON and moved on from there...

 > I just checked, and the excellent tool Pandoc does indeed support
 > conversion from HTML to XHTML, so that process could be automated.

Always nice to have more tools at our disposal!

 > Given the current state of available Scheme developers, I doubt more
 > than a few people would actually "review" the output of such tools...
 > Most will just "use" the output as is.
 >
 > On the other side, the actual text of the HTML is reviewed at least by
 > the author and the SRFI index editor.

I agree that it would definitely be better to have visible markup
reviewed by the SRFI author in the final format, but in the meantime
for old SRFIs, the choice is between an index incompletely reviewed by
third parties and no index :)

 > I still think we can use HTML parsers instead of requiring strict
 > XHTML parsing, though.

Like Ciprian, I would be in favor of XHTML eventually because it's
easier to parse (XML libraries are absolutely everywhere and are more
reliable than tagsoup). We may be able to use a tagsoup library to
automatically convert HTML to XHTML.

 > I am all in favor of having more volunteers! Unless the conversion
 > is automated, I would prefer not to do it all myself.

I can empathize with this. I was thinking along the lines of there
being on average one new SRFI per month to format, which ought to take
30-60 minutes if the source HTML is reasonable. And if no-one can make
it on a particular month, there would be no time pressure since that
SRFI can simply be formatted later.

In my mind, the big issue re: volunteers would be the huge backlog of
existing SRFIs. But again there's no time pressure. If it takes 3
years to convert the whole backlog the result will still be useful.

Perhaps it would be best to first settle on the right format for new
SRFIs. Then once we convert the old ones, we'd stand a chance to get
the conversion right once and not have to go back and do a second
round of fixups to them.

 > We would have to agree on the format in which the index was stored,
 > e.g. Ciprian's suggested format, so that more than one tool could
 > make use of that data.

I would prefer to have the SRFI source documents in a standard format,
and then have an API server built upon that foundation. The API could
serve, index and query them in a number of different ways, whatever
people need. The API server would be open source so hardcode users can
also use it locally as a library. But I would strongly favor a "single
point of truth" in the documents themselves instead of a separately
maintained index (indexes auto-generated from the source documents are
fine, in any format). This based on observing a kind of Murphy's Law
that "Any information that can go out of sync, will go out of sync" :)

 > In my view, if you just want to "index" the existing body of SRFI
 > documents, then adding HTML classes to all the existing documents is
 > more work than taking one of the already "crawled" lists (by you, by
 > me, or by others), and just double-checking them to make sure they
 > are "complete" and "correct".
 >
 > Afterwards, starting from this index one could write a small program
 > that goes back to the SRFI, and based on some simple heuristics
 > manages to back-reference the place where these definitions are
 > "defined".

Not sure about this. I tried marking up a couple SRFIs with classes
and it was quite swift and easy work once I got the hang of it. Once
it becomes second nature, double-checking will be almost the same act
as just writing the extra tags/classes. (When I was writing the tags,
I continually ran a script to regenerate the index, so I was looking
at the source and the index side by side.) Particularly if the results
go right into the source documents, so one doesn't have to worry about
which version of which index one is using (just auto-generate index
from source).

This may just be some kind of bias in my personality and experiences
but I strongly feel that things are simpler if there is one officially
blessed place where we have the ground truth. I always get confused
managing different versions of things in different places. I can't
make an objective case for why I prefer this, apart from having many
experiences where it has proven simpler.

 > Namely when they are "ready-for-being-published", we just export
 > them into HTML and annotate them ourselves.
 >
 > And this is also why the XHTML approach won't be to cumbersome. We
 > can just "translate" even the HTML ones in XHTML making sure they
 > are conformant.

 > My suggestion for XHTML is purely pragmatical: XHTML is XML, then
 > one can just use any XML library to parse the document.
 >
 > Now I know that it "seems" that there are many HTML parsers out
 > there, unfortunately this is not true... There are a few, at least
 > for the most popular programming languages, however they are
 > "bloated" and full of issues...

I fully agree with these points. In particular, it'd be very nice if
the final format doesn't require a tagsoup parser library, but can use
a strict XML parser instead. IMHO it's fine if we use tagsoup to do
the initial conversion from the author's HTML to the final format, but
once an SRFI is in the final format, it would be nice to let tool
writers use simple and reliable parsers.

 > Thus I think that there are two different problems:
 > * "new" SRFI documents that have to be "structured", "annotated" and
"indexed";
 > * "old" SRFI documents that can be just "back-referenced" starting
 > from an existing "index";
 > * (and a third problem) "transforming" the old SRFI documents so that
 > they are in-line with the newly proposed format;

This is a good delineation :)

 > However thinking from a practical point-of-view, especially given
 > how "rudimentary" some XML/HTML parsing libraries are for various
 > languages, I think it's easier to search for just `^(.*[
 > ]+)?def-proc([ ]+.*)$` than trying to imagine in how many ways one
 > can combine the classes `def` and `proc`.
 >
 > A good exercise for this "annotation" experiment I think would be:
 > * take the whole HTML file, and replace newlines with spaces;  `tr
'\n' ' '`;
 > * write a `grep`-based line to extract, say all procedure names;

I would think it best not to worry about grep and regexps. In
principle it's nice to be able to grep anything, but those
line-oriented tools are so hopelessly ill-suited for dealing with
XML's nesting, quoting, escaping and variations in whitespace that
it's almost impossible to write anything reliable using them. I've
done that every once in a while for over a decade out of laziness, and
they always fail even with a simple corpus of text. XML/HTML tree
walking libraries are so widespread and easy to use now that I would
just start from the assumption that tools can access them.

There are also some tools that convert XML to a line-oriented syntax
for grepping, e.g. http://xmlstar.sourceforge.net/doc/UG/ch04s07.html

One point in favor of using single classes as you suggest, is that
XPath expressions don't support multiple classes easily. I don't know
how big a deal that is but it's good to keep in mind and explore. In
general, I would concede that XHTML documents support multiple classes
per element, and people commonly do that, so if we don't permit these
XHTML conventions like this it is likely to confuse people. Then
again, if we have a strict lint tool to warn people about it, problems
will be easier to avoid. Let's experiment until we find something
palatable :p

 > in my view the editor takes the text and more "actively" formats it,
 > just like the editor of a book or magazine would.

Arthur would be in the best position to estimate whether the workload
is reasonable. Having a very involved SRFI editor would obviously
produce the best documents, but volunteer work can also be pretty
draining and thankless at times, and most people need breaks and
lighter periods now and then to keep motivated.

 > > Would you and Lassi be willing to prepare a proposal together and
 > > submit it to <srfi-discuss> for public discussion?
 >
 > At the moment I think some more "brainstorming" would be better.

I agree with Ciprian -- it would probably be best to brainstorm and
iterate on some prototype scripts and (X)HTML structures until we find
something that works fairly well. There are too many unknowns to plan
ahead in detail. But once we have something functional then a formal
proposal would definitely be in order :)

Should we have more beta testers though? At least Per Bothner
expressed some measure of interest in one of last year's threads.

 > BTW, the IETF has just gone through a similar approach by moving
 > RFC's from plain text to an XML-based format:
 > https://xml2rfc.tools.ietf.org/
 >
 > Perhaps we could just reuse their tools? (They are / will be better
 > maintained than anything we can come up with.)

That's great news :)

Here's one example of their XML format:
<https://tools.ietf.org/tools/templates/davies-template-bare-06.xml>

Personally, I'd rather start with XHTML and add/change only what we
need. XHTML is familiar and has short tag names for everything. These
custom XML documentation formats tend to be quite verbose and
tailor-made for a specific type of document (I've long avoided DocBook
for the same reason -- just seems over-engineered to my taste).

In my experience, quality of tools is strongly correlated with the
simplicity of the format. People love to write lots of tools for
simple formats because it's immediately rewarding. On the other hand,
complex things tend to have poor tooling even with lots of industry
backing. So IMHO the priority would be to make the format simple and
familiar (hence similar to HTML). Not saying the IETF RFC XML is too
complex but it seems quite verbose and divergent from HTML.

I would suggest roughly the following design approach:

* Start with some version of XHTML
* Use only the XHTML tags we need in a rigid structure
* Add class attributes to signify everything else
* Specify a standard CSS stylesheet using those tags and classes

Lassi