Format of S-expression metadata for SRFI documents Lassi Kortela (11 Mar 2019 14:59 UTC)
Re: Format of S-expression metadata for SRFI documents Ciprian Dorin Craciun (11 Mar 2019 15:05 UTC)
Re: Format of S-expression metadata for SRFI documents Arthur A. Gleckler (11 Mar 2019 16:05 UTC)
Re: Format of S-expression metadata for SRFI documents Arthur A. Gleckler (11 Mar 2019 16:02 UTC)
Re: Format of S-expression metadata for SRFI documents Lassi Kortela (11 Mar 2019 16:02 UTC)
Re: Format of S-expression metadata for SRFI documents Arthur A. Gleckler (11 Mar 2019 16:12 UTC)
Re: Format of S-expression metadata for SRFI documents Lassi Kortela (11 Mar 2019 17:30 UTC)
Re: Format of S-expression metadata for SRFI documents Arthur A. Gleckler (11 Mar 2019 17:34 UTC)
Re: Format of S-expression metadata for SRFI documents Lassi Kortela (11 Mar 2019 17:49 UTC)
Re: Format of S-expression metadata for SRFI documents Arthur A. Gleckler (11 Mar 2019 20:35 UTC)
Re: Format of S-expression metadata for SRFI documents Lassi Kortela (12 Mar 2019 09:43 UTC)
Re: Format of S-expression metadata for SRFI documents Lassi Kortela (12 Mar 2019 13:22 UTC)
Re: Format of S-expression metadata for SRFI documents Arthur A. Gleckler (12 Mar 2019 17:02 UTC)
Re: Format of S-expression metadata for SRFI documents Lassi Kortela (12 Mar 2019 17:15 UTC)
Re: Format of S-expression metadata for SRFI documents Lassi Kortela (12 Mar 2019 17:35 UTC)
Re: Format of S-expression metadata for SRFI documents Lassi Kortela (12 Mar 2019 17:51 UTC)
Re: Format of S-expression metadata for SRFI documents Ciprian Dorin Craciun (13 Mar 2019 15:28 UTC)
Re: Format of S-expression metadata for SRFI documents Lassi Kortela (13 Mar 2019 17:01 UTC)
Re: Format of S-expression metadata for SRFI documents Ciprian Dorin Craciun (13 Mar 2019 15:41 UTC)
Re: Format of S-expression metadata for SRFI documents Lassi Kortela (13 Mar 2019 16:54 UTC)
Re: Format of S-expression metadata for SRFI documents Lassi Kortela (13 Mar 2019 17:17 UTC)
Re: Format of S-expression metadata for SRFI documents Ciprian Dorin Craciun (13 Mar 2019 19:00 UTC)
Re: Format of S-expression metadata for SRFI documents Lassi Kortela (14 Mar 2019 13:28 UTC)
Re: Format of S-expression metadata for SRFI documents Arthur A. Gleckler (14 Mar 2019 17:33 UTC)
Re: Format of S-expression metadata for SRFI documents Lassi Kortela (23 Mar 2019 10:35 UTC)
Re: Format of S-expression metadata for SRFI documents Arthur A. Gleckler (23 Mar 2019 16:37 UTC)
Re: Format of S-expression metadata for SRFI documents Lassi Kortela (24 Mar 2019 09:15 UTC)
Re: Format of S-expression metadata for SRFI documents Lassi Kortela (24 Mar 2019 09:26 UTC)
Re: Format of S-expression metadata for SRFI documents Lassi Kortela (24 Mar 2019 09:27 UTC)
Re: Format of S-expression metadata for SRFI documents Arthur A. Gleckler (25 Mar 2019 20:25 UTC)
Re: Format of S-expression metadata for SRFI documents Lassi Kortela (25 Mar 2019 22:04 UTC)
Re: Format of S-expression metadata for SRFI documents Arthur A. Gleckler (25 Mar 2019 22:13 UTC)
Re: Format of S-expression metadata for SRFI documents Lassi Kortela (25 Mar 2019 22:42 UTC)
Re: Format of S-expression metadata for SRFI documents Lassi Kortela (25 Mar 2019 22:50 UTC)
Re: Format of S-expression metadata for SRFI documents Arthur A. Gleckler (25 Mar 2019 22:54 UTC)
Re: Format of S-expression metadata for SRFI documents Lassi Kortela (25 Mar 2019 23:56 UTC)
Re: Format of S-expression metadata for SRFI documents Lassi Kortela (26 Mar 2019 00:16 UTC)
Re: Format of S-expression metadata for SRFI documents John Cowan (26 Mar 2019 01:27 UTC)
Re: Format of S-expression metadata for SRFI documents Lassi Kortela (26 Mar 2019 08:54 UTC)
Re: Format of S-expression metadata for SRFI documents Arthur A. Gleckler (26 Mar 2019 04:17 UTC)
Re: Format of S-expression metadata for SRFI documents Göran Weinholt (26 Mar 2019 21:38 UTC)
Re: Format of S-expression metadata for SRFI documents Arthur A. Gleckler (26 Mar 2019 23:36 UTC)
Re: Format of S-expression metadata for SRFI documents Lassi Kortela (27 Mar 2019 21:42 UTC)
Re: Format of S-expression metadata for SRFI documents Ciprian Dorin Craciun (13 Mar 2019 21:57 UTC)
Re: Format of S-expression metadata for SRFI documents Lassi Kortela (14 Mar 2019 10:49 UTC)
Re: Format of S-expression metadata for SRFI documents Lassi Kortela (13 Mar 2019 17:46 UTC)
Re: Format of S-expression metadata for SRFI documents Ciprian Dorin Craciun (13 Mar 2019 18:53 UTC)
Re: Format of S-expression metadata for SRFI documents Lassi Kortela (14 Mar 2019 11:03 UTC)
Re: Format of S-expression metadata for SRFI documents Ciprian Dorin Craciun (14 Mar 2019 11:07 UTC)
Re: Format of S-expression metadata for SRFI documents Lassi Kortela (14 Mar 2019 11:12 UTC)
Re: Format of S-expression metadata for SRFI documents Lassi Kortela (14 Mar 2019 11:34 UTC)
Re: Format of S-expression metadata for SRFI documents Arthur A. Gleckler (14 Mar 2019 17:24 UTC)
Re: Format of S-expression metadata for SRFI documents Lassi Kortela (14 Mar 2019 20:40 UTC)
Re: Format of S-expression metadata for SRFI documents Lassi Kortela (11 Mar 2019 18:09 UTC)
Re: Format of S-expression metadata for SRFI documents Arthur A. Gleckler (11 Mar 2019 20:37 UTC)
Re: Format of S-expression metadata for SRFI documents John Cowan (11 Mar 2019 22:20 UTC)
Re: Format of S-expression metadata for SRFI documents Lassi Kortela (12 Mar 2019 07:08 UTC)
Re: Format of S-expression metadata for SRFI documents Lassi Kortela (12 Mar 2019 07:45 UTC)
(missing)
Re: Format of S-expression metadata for SRFI documents Arthur A. Gleckler (12 Mar 2019 15:12 UTC)

Re: Format of S-expression metadata for SRFI documents Lassi Kortela 23 Mar 2019 10:35 UTC

OK, I'm done with my experiment:

> * Make a git repo for the (very work-in-progress) tool
> * Make a git repo combining the official SRFI repos via 'git subtree'
> * Edit some SRFIs in the combined repo using pull requests
> * Add a Travis CI job to the repo so it runs the tool to check pull
> requests.
> * Make the tool generate a dashboard web page of the current status of
> the master branch.

It didn't go quite that way :p But here's what I have:

* I tried using 'git subtree' to edit many things in one Git repo. The
experience is very intuitive for advanced version control stuff. So
there shouldn't be any problem e.g. making one huge Git repo containing
a subtree for every SRFI document, then mass-editing the SRFIs and
pushing the changes to their origin repos.

* Created a new GitHub organization, SRFI Explorations, for this
experimental stuff: <https://github.com/srfi-explorations> If anyone
wants to join, feel free to ask Arthur or me. I already sent an invite
to Ciprian in case he has use for it.

* Created a repo <https://github.com/srfi-explorations/srfi-tool> to
house the work-in-progress metadata/validation tool, written in R7RS
Scheme. It uses Alex Shinn's chibi-html-parser which has worked great
for this task so far.

* Created a repo <https://github.com/srfi-explorations/srfi-server> to
house a proof-of-concept web server for continuous integration and API.
The server is written in Racket and includes the above portable R7RS
tool as a library (I'm using 'git subtree' to have the tool as a
subdirectory of the server repo). Getting R7RS to play well with Racket
code wasn't trivial but with a little effort I was able to make it work.

* Installed an organization-wide GitHub webhook for srfi-explorations.
This one webhook gets every event pertaining to the organization,
including everything done to any repo under the org. So every time
someone pushes to master or does something to a pull request, the
webhook is run.

* Cloned a few SRFI documents under the srfi-explorations organization.
Added metadata HTML classes to them.

* The webhook on the server catches all of those cloned documents and
all changes to them. It caches copies of the latest SRFI HTML from the
repos' master branches, auto-generates S-expression metadata based on
it, caches all of that stuff in an extremely simple SQL database table
and serves it to the public.

* The server also uses the latest cached data to render a dashboard with
the latest information.

The server is running on a free Heroku dyno at:

     <http://srfi-test-server.herokuapp.com/>

When you go to that URL, you get the dashboard. The dashboard has
clickable links leading to the latest HTML and metadata files. Those
links have a stable URL structure so they can also be used as an API if
people want to write scripts to scrape the latest SRFI HTML/metadata.

The S-expression metadata files could be downloaded from the server and
committed into each SRFI document's own repo. The server (webhook) could
then verify that the committed versions in the repo match what it would
auto-generate from the HTML itself.

This is all experimental but I'm basically sold on this approach.
Everything works well with minimal hassle. I'm particularly pleased at
how easy the GitHub webhook API is. You only have to configure it in
only place, in the organization's global settings. Then it sends very
detailed JSON about everything. Whenever a commit is made, I check if
the repo's name is of the form "srfi-[0-9]+". Then I download the files
in that commit as a zip file (GitHub has a very easy API for that!). I
unpack the zip file into Lisp objects using Racket's unzip library. And
feed the HTML from that zip file into the portable R7RS library. The
library extracts metadata which the Racket code then saves into the SQL
database and serves over the web.

Overall this is a fun project and I'm pleased to be working with Racket
(which I learned from scratch for this job). R7RS has also worked well
for the purposes of the tool, and interoperates well with Racket after
some setup.