Email list hosting service & mailing list manager

Re: scsh-0.6.7 -> scsh-0.7 has substantial changes, and introduces grievous error hga@xxxxxx (10 Jul 2019 19:43 UTC)
SQLite only SRFI? hga@xxxxxx (11 Jul 2019 01:49 UTC)
Re: SQLite only SRFI? John Cowan (11 Jul 2019 03:23 UTC)
Re: SQLite only SRFI? hga@xxxxxx (11 Jul 2019 08:19 UTC)
Re: SQLite only SRFI? John Cowan (11 Jul 2019 13:32 UTC)
Re: Systems lie about persistence and consistency promises. A lot. Arthur A. Gleckler (10 Jul 2019 22:51 UTC)
Schemepersist, anyone? hga@xxxxxx (10 Jul 2019 23:40 UTC)

Re: scsh-0.6.7 -> scsh-0.7 has substantial changes, and introduces grievous error hga@xxxxxx 10 Jul 2019 19:43 UTC

> From: Lassi Kortela <xxxxxx@lassi.io>
> Date: Wednesday, July 10, 2019 1:51 PM

> Ah, good old sync(2) :)
>
> > ;;; Amazingly bogus syscall -- doesn't *actually* sync the filesys.
> > (import-os-error-syscall sync-file-system () "scsh_sync")
>
> > ;;; Amazingly bogus syscall -- doesn't *actually* sync the filesys.
> > (import-lambda-definition-2 %sync-file-system (fd) "scsh_fsync")

> Does anyone lurking here know what's the story with sync() now that SSDs
> are gradually taking over?
>
> IIRC sync() was always kind of a weird syscall because it basically
> promises safety, but its behavior is so vague and complex that there's
> no clear criteria for how applications should rely on that promise....In
> practice, people resort to wishful thinking and do things like write a
> hundred files calling sync() after each file, or calling sync() then
> waiting a few seconds then calling sync() again to be sure. I've never
> met anybody who can lay out a principled approach for how to use it.

Applications shouldn't depend on sync....  But on e.g. Linux they
shouldn't depend on fsync either, for far too much of the software and
hardware stack lies through its teeth. Linux/Linus has just not cared
about this issue going back to the 1990s, and there are terrible
problems in many parts of its software stack.  E.g. the last time I
checked, the very handy Logical Volume Manager (LVM) did not honor the
promises made by software above or below and hardware below.

I suspect more reliable hardware and affordable UPSes have made
people complacent, with SSDs partly encourage that by being fast (but
many models take that all back by bricking if power is cut at the
wrong time; hypercapacitors are your friend, but even Intel is
dropping them from some of its datacenter line).

"Wishful" ... well, it's always been like that, back in the days of
PDP-11s, in my community the incantation before shutting down for real
was sync;sync;sync, then wait at least 5 seconds for the non-buffer
bloated hardware of the era to finish the job.

> The thing is, kernels typically sync() by themselves every 30-60
> seconds anyway, even if applications do not request it.

It's been that way for a very long time, at least to Research UNIX
Version 7, although looking at my 18.04 Ubuntu, the obviously named
separate process for doing this has probably been absorbed by systemd.

As I'm adding to the SRFI text, if you really care about best effort
writing, review the code in a paranoid database like SQLite.

- Harold