On 25/09/2019 10:53, Lassi Kortela wrote:

> Embedded XML is a good illustration of why embedding custom lexers
> causes things to quickly turn chaotic. (But XML is not the worst example
> by any means - it can and will get more baffling. Mix PHP, HTML, CSS
> and/or JavaScript in the same file and things get real :)

I've been there :-D

> Not to judge, but there is a segment of people who absolutely love
> lexers and parsers, and will jump at the chance to re-invent a
> poorly-thought-out custom syntax for the simplest of jobs. Then the rest
> of us have to scratch our heads at how to parse those things and how to
> implement basic tools like code walkers and syntax highlighters. So
> based on both principle (complexity budget) and repeated experience, I'd
> be strongly in favor of having very simple core S-exps, building any
> complex types by nesting those, and banhammering all custom lexer
> extensions in the format :) Let people keep any lexer extensions in
> their own Lisp code, where they are the only ones who will have to deal
> with them.

Sounds good to me!

>> (2) makes bits of a bytevector that happen to also be valid ascii or
>> utf-8 text "readable", but is more complicated to generate/parse and
>> ends up as a worse form of (1) for very unprintable stuff.
>
> The thing is that character encoding is easily messed up by running
> "iconv". (This problem concerns ordinary strings too.)

Yes. On a subtler level, it conflates the model of the s-expression as a
sequence of *characters* - and it being a sequence of bytes. A bit of a
layer violation!

--
Alaric Snell-Pym   (M7KIT)
http://www.snell-pym.org.uk/alaric/