We get signal:
I admire the ability to get a POC running fast (I wish I had it). But I feel a bit buffaloed here, and I think the purpose of the database subprocess idea has been lost sight of. Can we take a step back here?
This is what I thought the subprocess design was all about:
0) Simplicity trumps efficiency: if you want efficiency, use a direct connection. That said, if some things are very inefficient, you may need more efficiency elsewhere to be usable at all.
1) Minimum requirements on the Scheme client, preferably no more than R5RS or R7RS-small.
2) Abililty to implement the subprocess in any language that can talk to the database.
3) The subprocess is in the same security context with the client, and they can trust each other, at least as much as anything else one loads from a package manager or
srfi.schemers.org. Subprocesses are not normally exposed to malicious users, as the database server may be.
To me, these mean:
Bit-diddling is possible in any Scheme with arithmetic operations (see the completely portable implementation of SRFI 151), but it involves a bunch of divides and modulos and looking things up in vector tables. (All kudos to Aubrey Jaffer for providing bitwise-and, bitwise-or, arithmetic-shift, and the rest of the bitwise core: see <
https://github.com/scheme-requests-for-implementation/srfi-151/blob/master/srfi-151/bitwise-core.scm>.) So from an efficiency standpoint, especially in an interpreted Scheme, using the built-in I/O like read and write makes a lot of sense. (See Note 1 and Note 2.) So: standard S-expressions both ways.
The Simplest Thing That Could Possibly Work is to have an outside manager (a bash script will do) create two named pipes and pass their names to both the client program and the subprocess (no longer "sub-") on the command line or in environment variables. (Note that there are named pipes on WIndows.) They open their files, the client writes to one while the subprocess reads it until it's done, then they reverse roles for the reply. *Any* R5RS Scheme can do this. The client can even close and reopen the "request" named pipe to force flushing if it doesn't have R7RS flush-output.
Assuming Python or Perl is safe on any Posix system, and it's actually easier to get going on Windows than a C development environment, especially since Visual C++ basically only has C99 (and until 2017 only C89). So if a subprocess is supplied in any of these languages, it's enough. Perl has a DBI module to provide the user interface and DBD modules to plug in (see Note Python just makes its DB-API a specification that all drivers are by convention expected to support.
Note 1: I accept as friendly amendments the use of (name . "value") instead of name=value, and of ordinary double-quoted strings instead of triple-quoted ones, in my proposed textual protocol.
Note 2: Writing a portable implementation of read that handles *exactly* what is described by the BNF for <datum> in R7RS-small, and lets you set limits on list/vector/bytevector length, on nesting depth, and on numeric tower support, would be a Good Thing. It could be packaged with a similar portable implementation of write to make a good SRFI with safe-read and safe-write, if anyone feels ambitious. I have implemented a zillion such parsers in the last 45 years, and I'm thoroughly sick of writing them.
Note 3: I seriously doubt that the wire protocol for either Oracle or Microsoft is unstable. The wire protocol for Oracle is the same back to 8.1.7 (2000), and for SQL Server back to 2005, IIUC.
Note 4: I'd love to have a pluggable architecture, what Java calls an SPI (Service Provider Interface): see <
https://itnext.io/java-service-provider-interface-understanding-it-via-code-30e1dd45a091>. But I have thought about it a lot, and I don't see how you can make one of those work in Scheme, because there is no way to ask a procedure what it expects at even the simplest level of "how many arguments". Procedures are opaque: all you can do is call them, with completely unpredictable results. If anyone has ideas, please pass them along.