> While workable, this seems to me to be less than ideal because any time
> one scrapes something the process is fragile, needing manual intervention
> to fix the scraper whenever some unforeseen change happens to the
> structure of what's being scraped.

The formats don't change all that much, and the big benefit of scrapers
is that they are executable documentation.

Before we had the current scrapers, there were various hand-written
listings of SRFI support around the net. It was impossible to tell where
they came from, how they had been assembled, and which parts were up to
date. There was no way for a newcomer to replicate the results.

Any large-scale data aggregation effort should absolutely use scrapers,
if only for documentation purposes. But it's also a good way to avoid
human error.

> There's also unnecessary bandwidth being wasted repeatedly downloading
> tar files, and time spent uncompressing and searching through them for
> what amounts to a relatively tiny bit of data.

These are non-issues. GitHub and GitLab have tons of bandwidth. Gambit
is one of the biggest Schemes and running listings/gambit-head.sh takes
only 4 seconds, including the time GitHub takes to generate us a
tailor-made tar archive of Gambit's git master.

We should scrape all this from different implementations and package
indexes, and aggregate it into one place where it's available as one
JSON and/or S-expression file.

But it pays to make a distinction between source data and aggregated
data. If one aggregator takes 5 seconds to scrape each source, it's not
a problem.

> Because there is no standard, the data you get from a an arbitrary
> Scheme's tar file is going to be unstructured, requiring more custom
> rules to extract it.
>
> Wouldn't it be so much simpler if every Scheme published the desired
> data in the desired format at could be directly, reliably consumed
> without having to write any custom code to deal with unstructured data
> in random locations?

It would, and this is most easily accomplished by adding an S-expression
file to each Scheme's git repo.

GitHub and GitLab can make you a link to each raw file stored in any Git
repo. E.g.
<https://raw.githubusercontent.com/schemedoc/implementation-metadata/master/schemes/chicken.scm>.
You can also change "master" in the URL to a different branch or tag. If
the aggregator could look for a standard file in the repo, it wouldn't
have to download the whole repo, and would take less than 1 second.

> Scanning through each package's metadata might be the most reliable way
> to do this, but there is still the question of how that metadata is made
> available.

The source metadata would be in each package. Each package manager would
scan all of its own packages and compile an index file. The aggregator
that compiles the full SRFI support table for all implementations would
then aggregate _that_ data :) We should serve the full table as HTML,
JSON, and S-expressions so people can save time and easily
machine-extract things directly from the full aggregated data.

> There is currently no standard, to my knowledge, of how
> packages are distributed or how their metadata is published.  Every
> Scheme does it in their own way.  This is an opportunity for
> standardization as well, with benefits to a metadata collection project.

All of that is correct.

> I would be happy help in the immediate future, though I'm afraid prior
> commitments might tear me away in the long run.

Don't worry about commitments; we can make a repo under
<https://github.com/pre-srfi> and gradually work on it.