So a Scheme implementation may be a very good idea if we need it for a
paricular purpose (and someone volunteers to write all that code). But
if we don't, then I'd recommend favoring simpler protocols. If ASN.1 is
needed, it may be worth to considering wrapping one of the many C
implementations.
You mean those horribly buggy, schema-dependent, statically typed, compile-time-specialized implementations? Include me out!
Your list of data types looks good if a comprehensive format is needed.
I think we do need a comprehensive format. Part of the goal of R7RS-large is to make a whole new crop of standard dateatypes all equal in the eyes of a Scheme programmer. Lisp programmers have always said things like: "Well, it would be faster and better to use vectors here, but the vector procedures in R7RS-small (which has more than any of its predecessors) are too impoverished compared to SRFI 1, so I'll use lists" (and similarly for CL). I want Scheme programmers to stop making those choices. SRFI 133 gives vectors substantial parity with lists, and including it in R7RS-large means that implementations will have them either as "batteries included" or as readily loadable from Snow etc., and as such, so that you can always reach for the right tool and not the most convenient one, because they are all equally convenient.
Part of that inclusiveness of types is to be able to serialize *all* our data structures, not just the simple int/float/stirng/boolean ones. Backward compatibility is going to make it difficult to do that with S-expressions, though I have some ideas for portable extensions to the S-expression format (more on that later). By no means all Schemes use a readtable approach as CL does, and a lot of implementations of `read` would be quite hard to extend. Interchanging a simple binary format makes things much more straightforward, as I pointed out in my not-evil post.
But I would leave out all of the space-optimized ones unless someone has measured that some specific task is too slow.
f32vectors aren't there because they are space-optimized, but because they are a data structure type of R7RS-large, per the vote on SRFI 160. (Wihtin a Scheme program, they are there because they are space-optimized and quite possibly more efficient to work on, if your implementation optimizes f32vector-map for certain known functional arguments.) Given that everything in an f32vector is an IEEE float, and that (r6rs bytevectors) aka (scheme vector), which as I mentioned has a portable implementation posted in the SRFI 4 repo, it's totally easy to convert from a pile of bytes in big-endian to an f32vector and back. (Well, we need bulk converters, for which I don't yet have even a pre-SRFI, but I will; it belongs to the tedious-but-trivial category.)
I'd go with varints for all numbers.
Varints are good for non-negative integers. For other numbers, not so much. There is actually a varint-like format for floats in ASN.1, but I left it out because interchanging in anything but IEEE format (which at most needs byte-swapping) is useless at this point. If you have to exchange with IBM z/Series machines (base 16 floats instead of base 2), use something else.
For anything subject to
network effects, a bigger network adds value much faster than technical
sophistication, as Lispers are all too aware.
Part of our whole raison d'etre is *not* to do what everyone else does: if you want that, use what everyone else uses, to resist Worse is Better. Or as Shaw put it: “The reasonable man adapts himself to the world: the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man.”
This all brings to mind the old saying: "Recursive-descent" is
computer-science jargon for "simple enough to write on a liter of Coke".
Never heard that before, but it's very true. I'd even add: "or to write on a whiteboard during a coding interview, though you may have to explain to your interviewer how it works." I did that when asked to check a string for balanced parentheses and on another occasion (I forget exactly for what) recently.