OK; especially since it's my idea, I volunteer to create the API, and code an implementation.

I think it's best to put it in just SRFI-167, and direct -168 and other potential users to use the -167 facility with their knowledge of the schema they're defining.

We need to key off prefixes, since I assume difference prefixes signal different schema.

For sanity I think we want separate validation procedures for each prefix, including the no-prefix case.  We also want an option to skip validation for maintenance and highest performance purposes.

Do we want to store the code in the database?

I don't think that'll be complicated or difficult, but executing code from an outside source is dangerous if the source is not properly protected, if so, warnings must be put in the SRFI(s).  But such "stored procedures" would make the validation ironclad, which is how the higher end RDBMS inspiration for this works, and would make it much simpler for non-Data Base Administrators (DBAs), all they'd need to know is that they might get validation errors, and how to skip validation.  This being Scheme, I don't think we need to make validation required and locked down except for DBAs.

If stored, we'll need to define a reserved prefix to store it under, like "VALIDATE_XYZZY_" or something even more random.

Or more general, this should also allow for the generic stored procedure concept where behavior is delegated to "the database".  We always need to be careful about being too prescriptive in use cases, users will find unanticipated ways to use functionality, and this is a particularly obvious example.  At some sites, DBA maintained stored procedures are the only way to access the database.

That also strongly suggests adding an "execute stored procedure" procedure to the SRFI-167 API.

Ah, one wrinkle in storing is Scheme implementations that require a C compilation step for highest performance, when it's not possible to store and then load upon okvs-open object code.  But the code could also be stored in an object file to be linked by everybody for the final binary, the API will need to be conceptually general to accommodate a variety of implementation methods.

Anyway, those are my half-awake thoughts as of now.

- Harold

----- Original message -----

From: Amirouche Boubekki <xxxxxx@gmail.com>

Date: Tuesday, July 02, 2019 7:26 AM

I have no idea how to achieve this.

I create this new topic to raise awareness.

---------- Forwarded message ---------

De : <xxxxxx@ancell-ent.com>

Date: lun. 1 juil. 2019 à 17:28

Subject: Re: Further comments on the SRFI

From: John Cowan <xxxxxx@ccil.org>

Date: Saturday, June 29, 2019 9:46 AM

On Sat, Jun 29, 2019 at 7:26 AM Amirouche Boubekki <xxxxxx@gmail.com> wrote:

That is very difficult to do validation at this level. OKVS manipulates bytevectors, to validate keys and values it would require to unpack the bytevector which is not know at this level. I acknowledged that validation is missing piece in the whole framework (SRFI-167 + SRFI-168), I am thinking about ways to improve the situation, but that will not be part of any current SRFI.

A simple approach would be to associate a simple Scheme function with each open database that is called before inserting any key-value pair.  It is passed the key and value as its arguments, and returns #t if the combination is valid and #f if it is not.  Of course the behavior of this function would depend on how the store was used.  That would be straightforward to implement within the library.

That's the sort of simple thing I was thinking of, and I think it would be in the scope of either or both SRFIs.

If the constraint is to be persistent, it would need to be stored in the database under some well-known key, which would mess up the complete freedom of the user to use any key desired.  If that were overcome, a fully general constraint would have to be stored as a lambda expression and eval'd in some minimal library (which would be worth defining anyway, probably).

That's a capital idea, although evaluating random external code invites mischief.  Arguably more beyond the scope of these SRFIs, especially since it would be nice to store multiple constraints, but it would be extremely useful, and help bridge the validation capability gap between simple OKVSes and the better RDBMSes.

[...]