From: Amirouche Boubekki <xxxxxx@gmail.com>

Date: Saturday, June 29, 2019 11:41 AM

Thanks for the clarification / summary.

I have been slow to apply the requested changes partly because

I am not sure and because it is different from what I was expecting.

You're very welcome, and thanks for your patience.

More questions below:

Le sam. 29 juin 2019 à 17:21, <xxxxxx@ancell-ent.com> a écrit :

Summarizing the major points in the discussion that started 5 days ago:

• SRFI-167 forces all operations use transactions, rather than (strongly) suggests
  
• That requirement adds ceremony for isolated single operations when a database automatically executes them in a transaction, like (okvs-in-transaction db (lambda (txn) (okvs-ref txn (pack "hello"))))
  
• Some OKVSes don't support transactions; requiring transaction boilerplate creates a false sense of security
  
• For databases with a single operation automatic transaction feature, this requirement turns a single round trip into 3, start transaction, operation, commit transaction, which may be significantly more expensive
  

I disagree with that. In the case of LevelDB, transaction-begin et al. are nop procedures. I agree that it gives a false sense of security.

And requires ceremony that's not going to accomplish anything.  But for databases that do provide transactions, if the transaction operations require a context switch or more ... I prefer to not leave performance on the table, especially since some applications are rate limited by new data insertion:

 

• There are applications with very high required rates of write operations; adding round trips is bad, and for some applications a trade off of losing a portion of the incoming operations is perfectly acceptable
  

Many of these points are less strong when SRFI-167 is only used to provide the raw OKVS storage for SRFI-168, but SRFI-167 is high quality and I believe ought to support other uses, including ones where the schema had been defined by another, probably non-Scheme application. 

How does it relates to transactions?

Old schema, like ones that started with the original UNIX dbm mentioned below, likely don't take advantage of transactions which weren't available back then.  But I could see users of these applications upgrading to KVSes opened in auto-transaction mode, although they wouldn't necessarily be OKVSes.

This boils down to how much SRFI-167 wants to be a general OKVS interface, vs. one for new Scheme applications.  I suppose if it wanted to be very general, it would also support non-ordered KVSes.

Per Wikipedia, the original UNIX dbm library was released by AT&T in 1979 (no doubt part of the Research UNIX Version 7 release that same year).

AFAIK the original dbm library doesn't support transaction and is hash database.

I mention it because as far as I know that's when the ecosystems we generally use today started doing standardized KVS storage.

Counter arguments include:

• SRFI-167 doesn't have to be a general purpose OKVS store
  
• Requiring transactions is not necessarily a bad thing
  
• As application developers "climb the ladder of features", starting with transactions from the very beginning makes mistakes less likely and development cleaner
  

I agree with the above points.

 

• Bolting a transaction facility onto a non-transaction supporting OKVS is not tremendously difficult
  

I am not convinced the implementation of transactions is that easy.

I've done it, it's "easy" as long as you trust the underlying operating system and hardware.  Which you of course can't at all today, you're strongly advised to use code or a library from an existing paranoid database like SQLite, which could then give you the transaction facility "for free".

 

• I cannot see any way in which this requested change does not make SRF-167 more complicated, and probably more ugly
  

Maybe it is not ugly to add 'no-transaction. I am not sure as of right now.

Right now the database handle for the specific OKVS being manipulated is subsumed into the transaction handle.  My idea of an "in-your-face", remind you of the danger of using the no-op 'no-transaction handle cannot "cleanly" supply the database handle.  I suppose you could require setting the value of no-transaction to the database handle, and the code below eval-ing no-transaction, but that's tortuous, and "ugly".

My best idea is to disentangle the database and transaction handles, which will make the API more complicated or at least more ceremonial, but perhaps not "ugly".

For the 'no-transaction change, while I don't like it, other than requiring both a database and transaction handle, the only idea I've come up with is to have okvs return a procedure.  When called, it returns the equivalent of the current transaction handle, so from the tutorial:

(define db (okvs '((home . "/tmp/wt") (create? . #t))))

(okvs-in-transaction db (lambda (txn) (okvs-set! txn (pack "hello") (pack "world"))))

becomes:

(okvs-in-transaction (db 'no-transaction) (lambda (txn) (okvs-set! txn (pack "hello") (pack "world"))))

or with less ceremony, but not clean to add a second operation and put both in a single transaction:

(okvs-set! (db 'no-transaction) (pack "hello") (pack "world"))))

Not elegant, especially whatever is the opposite of (db 'no-transaction), but maybe it'll help spark a better idea.