On 17/09/2019 13:05, Lassi Kortela wrote:

>
> Thanks for the support, Alaric :) Once again, I agree with all your
> points of view.
>

I am lulling you into a false sense of security, so you support MY
terrible ideas later... ;-)

> I would still prefer a binary encoding for interchange, since a text
> encoding using length prefixes is not really human-readable or editable.
> Also, people tend to open text files in text editors, which causes all
> the usual problems with character encodings and whitespace to start
> spreading. If we make binary S-expressions with the same data model as
> text ones, it'll be trivial to convert back and forth, so people can
> just convert to text for editing and inspection.

Yes. If you're not trying to be human-processible, then making the
format as convenient as possible for the machine (and programmer)
becomes paramount.

> If there is interest, I can write a spec about such an encoding, and
> solicit feedback. The basics are extremely easy to do with varints, as
> demonstrated.

I'd like that!

> Intricate data types like floats, timestamps, etc. are
> harder, and could be left out, or if not, people like John know a lot
> more about those than I do.

Just follow the Scheme numeric tower etc. and everything sort of falls
out naturally, IMHO.

Rationals? Store them as two integers.

Complex? Store them as two numbers.

Inexacts are the tricky one, as unlike all the other types, there's no
procedures to simply split them into parts that we can already represent
- and we would like to exactly represent inexacts, as it where,
reproducing them exactly the same way (assuming that the receiving
implementation's representation isn't less exact then the sending
one's). But peeking under the hood, one can note that most (all?)
inexact number representations are IEEE floating points, and either
provide a "raw IEEE double" (pick an endianness) value format, or avoid
problems with implementations that have inexact representations that
aren't a strict subset thereof by sending inexacts as two arbitrary
integers - mantissa and exponent - with a few special values for
infinities. Even the "subnormal" numbers very near 0 can be represented
in that form, but it's a bit of shuffling to convert between that and
Scheme numbers.

As for timestamps and stuff - you can store the arguments to make-time,
by using the accessors in srfi-19, etc!

> I'd like to have an encoding that has enough range that it can also be
> used seamlessly from Common Lisp, Clojure and Emacs Lisp. In fact, I'd
> like to have command-line program that converts between all the
> different text/binary S-expression representations. You could just pipe
> things to it as you went about your daily work.

It would be nice to add it as an input/output format in magic-pipes, my
sexprs-in-shell-pipelines toolkit:
https://www.kitten-technologies.co.uk/project/magic-pipes/doc/trunk/README.wiki

--
Alaric Snell-Pym   (M7KIT)
http://www.snell-pym.org.uk/alaric/