Correct. And the driver lives in the subprocess written in C.
That doubles the amount of traffic, though presumably the subprocess will live on the same host as the application. Given that, I see no reason to write the subprocesses in C. Let's use Python or something similar, which already has plenty of drivers, is much easier to program in correctly, and can handle a simple protocol straightforwardly.
While I'm at it, how about this extremely simple protocol:
Request:
1. Bindings in the form name=value, one per line, where value can be a quoted string with R7RS \-escapes allowed (required, if the string contains a newline character), a number, a bytevector in R7RS notation, or (unquoted) null. Blank lines are ignored.
2. A query as a triple-quoted string. The first line starts with """, possibly with leading whitespace; the last line ends with """, possibly with trailing whitespace. No escapes are allowed.
3. Flush.
Response (in Lisp format):
1. A list of symbols wrapped in vertical bars, the column names.
2. Any number of lists whose elements are strings, numbers, bytevectors, and/or the symbol null.
3. The symbol "end".
4. Flush.
Possible addition: some wrapper around ISO 8601 strings representing dates, times, timestamps.
Comments?
> Also, are you talking to the driver over a socket (TCP or local, doesn't
> matter) or over a pair of pipes? If the latter, you have to make sure that
> both sides flush frequently, or you will get deadlocks.
I was thinking it would work identically over pipes or a socket. Scheme
would write its request, then flush and wait for the response. The C
side would read the request, write a response, flush, and wait for
another request. Does this simple arrangement cause concern of deadlocks?
A timeout should probably be applied on the Scheme side. Since the
subprocess doesn't have anything else to do, it may as well wait forever
for each request.
> It would also be possible to run the standard CLI client for the database
> using a PTY. That solves the flushing problem, but you'd need to carefully
> pick an output format (or small set of formats) that all CLI clients can
> produce. that doesn't throw away header names or manifest typing
> information, and unambiguously indicates the end of the result set somehow.
The subprocess would be an alternative to the standard CLI client. The
point would be to have a clean, generic binary interface optimized for
machine parsing. Parsing/quoting into REPLs - that way lies madness...
In a big Scheme with sockets, you'd probably want to write pure-Scheme
Postgres and MySQL clients that talk the native protocol of those
databases directly to the DB server. But for situations where a simple
implementation is important and performance is not an issue, the generic
implementation would do wonders.
I would also put SQLite into a subprocess instead of having in the
Scheme process. A Scheme with FFI that wants really fast SQLite (is
there a substantial speed difference, though?) would embed it in the
Scheme process and talk to it via FFI. The FFI may also be easier to
code. It all depends on the implementation.