Follow up after finalization of SRFI-167 and SRFI-168 Amirouche Boubekki 28 Nov 2019 18:22 UTC
OKVS is an abstraction that is *loosely* based on FoundationDB (FDB). That is, it does not expose *all* FDB features, for maximum performance in a very large database, roughly more than 1TB, you would be better served by relying on FDB bindings directly. Mind the fact, FDB works fine in standalone mode, it does also have an in-memory backend. The big difference between FDB and the embedded OKVS (RocksDB, WiredTiger, SQLite LSM extension) is that the latter rely on cursors with procedures like cursor-seek, cursor-previous, cursor-next, and cursor-close! In those embedded OKVS, it is trivial to implement leaderboards, sorted mapping or ranked set. In FDB, you can workaround it using engine-ref* (see below) and by implementing your own datastructure (see for example https://github.com/FoundationDB/fdb-record-layer/blob/master/fdb-extensions/src/main/java/com/apple/foundationdb/async/RankedSet.java). That is, one could specify a cursor-based OKVS that will be a complement to SRFI-167. I choose to base OKVS on a subset FDB interface, because there is less procedures. That will hopefully, help spread the idea that the ordered key-value stores are a better abstraction for persistent on-disk storage with strong guarantees compared to, so called, RDBMS. Here is a few notes after working more with those two SRFIs: - SRFI-168 nstore is good as a generic store. That is when you want to prototype quickly without any optimizations. The only procedures that I can see that can be good to have is a nstore->list procedure and list->nstore. - Related to versioned nstore (formerly fstore), it needs some love in particular I put too much features in that abstraction e.g. the snapshot should be in a higher level abstraction. [0] [0] https://git.sr.ht/~amz3/chez-scheme-arew/tree/master/src/arew/data/base/fstore.scm Regarding SRFI-167: - Regarding the engine's invoker procedure pass arguments as rest arguments and does not check the number of argument. This can be improved. That procedure is private and not specified. - It will be very handy to have alist->okvs and okvs->alist. - Taking inspiration from FDB, (engine-ref* engine okvs spec) could be added. SPEC can be a key, a selector or a pair of key or selector. A selector is a key associated with an offset and an equality flag. Similarly engine-remove* could be added. That covers several use cases and can avoid the need to (re) implement a ranked set on top of okvs. See https://apple.github.io/foundationdb/api-python.html#key-selectors - engine-prefix-range-remove! could be added, this is sugar syntax over engine-range-remove! Since the procedure strinc is not part of the public interface, it would make sense. Regarding engine-pack and engine-unpack that are in the end lexicographic packing and unpacking: - The sample implementation does not support nested lists, this is trivial to add, which in turns makes handling subspace prefixes much more easy (e.g. in nstore), - I think it requires to support pairs instead of lists - It might also support vector type - I think it would be better if engine-pack was taking its argument as a list instead of a rest argument. That will, I think, avoid a copy when doing something like (apply engine-pack engine args). That way, taking inspiration from FDB, it could take a raw bytevector prefix as optional second argument. Also, it will make it more symmetric with unpack that returns a list. Thanks a lot for the help. -- Amirouche ~ https://hyper.dev