From: Amirouche Boubekki <xxxxxx@gmail.com>
Date: Monday, July 08, 2019 2:42 AM
Date: Saturday, June 29, 2019 4:13 PM
[...]
- I cannot see any way in which this requested change does not make SRF-167 more complicated, and probably more ugly
Maybe it is not ugly to add 'no-transaction. I am not sure as of right now.
Right now the database handle for the specific OKVS being manipulated is subsumed into the transaction handle. My idea of an "in-your-face", remind you of the danger of using the no-op 'no-transaction handle cannot "cleanly" supply the database handle. I suppose you could require setting the value of no-transaction to the database handle, and the code below eval-ing no-transaction, but that's tortuous, and "ugly".
My best idea is to disentangle the database and transaction handles, which will make the API more complicated or at least more ceremonial, but perhaps not "ugly".
Ok, I agree with the idea of offering an upgrade path for dbm-like based applications.
Question: Should it be explicit in the specification that the library offers such a upgrade path?
Is anything needed, or would an example suffice?
This is in the context of adding a no transaction option to SRFI-167, and many old application databases not caring, since they were written before dbm databases added transaction.
I wish to not make the specification a
roman a tiroir, I intend to remove the sentence in the abstract that says that OKVS without transaction can be supported. The user can figure it out.
Might as well stay silent on it; if the user is serious about his data, he'll use a database with transactions, or add some sort of transaction log and check pointing. It can be very handy to create a transaction log outside of the scope of the database, that's tremendously useful in testing, and can with pain accommodate changes in business logic (the pain from "if we're past transaction 141 but before 213, we do X).
[ The argument placeholder to allow the auto-commit mode. ]
The above is something like the current "SOME" argument ... ah, if you want to keep that instead of indicating the alternatives in the argument list, "SOMETHING" is less grating to my ears.
I propose to use SOMETHING instead.
Since I sent that message, I thought of DB-OR-TRANSACTION-HANDLE. SOMETHING kinda implies "1" or "XYZZY" might suffice, this suggestion makes the options clear, although not why you'd chose one or the other.
Regarding okvs-range and okvs-prefix-range, I propose that when an okvs record
ie. db is passed as argument those procedures return a list of key-value pairs.
You've dropped that idea, but it prompts me to suggest avoiding having to know the maximum prefix or even beginning prefix, some set of arguments that explicitly generate the whole set of key value pairs. 'maximum-key might do, perhaps 'minimum-key as well. Not over-specifying, getting specific as your commit comment did, could allow a database to more efficiently return the whole set.
Not knowing having okvs-maximum-key is a problem whether one rely on a distributed OKVS or embedded OKVS alike. The thing is that the upper layers should know about the layout of the database
and should not require to scan the whole database to know what is in it.
Scanning the whole database is required for certain maintenance tasks like finding orphan keys, and is good for doing your own backups. A symbolic from 'minimum-key to 'maximum-key would make the recipe for this simple and crystal clear in intent.
Since dbm, would need to do a full "table scan" anyway to emulate those procedure
it does no harm. Also, since there is no transactions, the generator does not make sense,
because it would require to do generator->list then list->generator to be able to produce
the generator.
We could assume an automatic transaction for each key value pair generated. We also have to consider the case where old non-Scheme code is using the database at the same time new Scheme code using transactions does. Otherwise is there any point in using SRFI-167 for this conversion use case?
The case of no-transaction is loosely specified. I only want to keep it for the bad reason that it makes interaction at the REPL easier.
The auto-commit flavor is common in CLI shells that are REPLs in effect for databases, like psql for PostgeSQL, otherwise using them is painful.
And I want auto-commit for efficiency when 3 rounds trips, start transaction, the operation, and commit transaction, are expensive, requiring extra context switches or worse.
The change will allow much better user experience at the REPL.
That's something I'm going to be emphasizing in my reply to John on mutation triggers vs. constraints, safety when using the REPL to manipulate the database, for e.g. maintenance purposes.
To my mind, maintenance of the database should be done using a specific library or program that knows about the underlying schema. Relying on pure OKVS interface to do maintenance is, at some point, doomed to break the database.
You're 100% correct on all points ... but we know that's not the only way people will manipulate the database. Especially to try to fix errors in the database in an ad hoc manner, the temptation of a Scheme REPL and direct access to the okvs-* procedures will be irresistible, especially since that's a exponential force multiplier in first prototyping and exercising your library code, in learning the ins and outs of an implementation by doing.
But once you're used to exercising the database at the REPL level using raw low level procedures, the temptation to use them for Data Base Administrator (DBA) tasks is well neigh irresistible. Even if you have a firewall between development and production, the former is still likely to come up with REPL solutions to fix production problems, especially if there's time pressure. Like ITS TECO, "A moment of convenience, a lifetime of regret" which I don't believe we can forbid in the limited scope of these SRFIs, it would take making a full fledged OKVDBMS akin to conventional RDMBSes, adding abstraction layers on top.
- Harold