On Tue, Oct 17, 2017 at 8:27 PM, Shiro Kawai <xxxxxx@gmail.com> wrote:

There are many good ideas, but before delving into each of them, let me share the background. While developing gauche.generator and playing with it in my code, I noticed one essential thing about having a generator library---every generator should follow exactly the same protocol. With that guarantee, we can provide generic generator combiners that are agnostic to the actual generators passed in. And since a procedure call is generally very lightweight in Scheme, we can create complex generator network that runs pretty efficiently.

I agree 100%, as should not be surprising, since I based the SRFI directly on gauche.generator. Generators are intended to be lightweight, a standard implementation pattern for functions with state.

I considered and tested multi-value generators and out-of-bound channel (by throwing exception from a generator) but eventually dropped them since it got in way to provide generic generator combiners. Either we should limit its domain (e.g. "this combiner can only accept single-value generator") or need to add extra code that would drag performance (e.g. always receive generator output by "receive", or add guard around a generator call in case it tries to communicate
with side channel). Sure it is the same as the general procedures, but generators are essentially a special-purpose procedures to make things easier to a specific purpose.

I agree with this reasoning. In generators (and accumulators), simplicity wins.

1. Enhancing EOF value

This works nicely with existing generator library. I think it's best to use auxiliary information carried in EOF sparingly; for example, what (gmerge < ga gb) should return at the end, if ga and/or gb returns enhanced EOF?

I agree that this is a good thing, but I think it should go in its own SRFI, as it cannot be written in 100% compatible R7RS-small in a pluggable way. That is, you can define a library to do this, but the enhanced EOF objects will not be recognized as such by other libraries that do not import it.

This is conceptually related to the problem that Chicken has historically had with extended numbers (everything but fixnums and flonums); there is library support, but you have to watch the boundaries carefully, because if an extended number is passed to a procedure or continuation that was compiled without support, it looks like a completely opaque object that is not even a number. The next major release of Chicken will have a fully integrated numeric tower.

On Python generators:

Python generators are by intention much more powerful than the kind of generators I am describing in SRFI 158. They are actually coroutines lwith two limitations: it must be the generator function itself that contains the yield, rather than a function that it calls; and a generator can only yield to the calling generator. (There are two ways to express the same idea in recent versions of Python, generator protocol and coroutine protocol, but they have the same power.) Lua coroutines have the second limitation but not the first. Both of these are attempts to avoid the full price of call/cc.

But in Scheme we have already paid that price: call/cc iis available in essentially all Schemes, except where limited by the implementation technology, and even there trampolines can be used as necessary. Lua coroutines can be made fully general by the use of a user-level trampoline, with the problem that trampolines do not compose with other trampolines. So I suggest that a new SRFI be written for general coroutines that use call/cc throughout, eliminates the use of native Scheme procedure calls to invoke, and has all the bells and whistles anyone might want. However, these coroutines would not interoperate with the generator operations of SRFI 158, at least not without a conversion function.

The Lua 5.3 coroutine library (part of the standard Lua library) and the Perl module Coro (in CPAN) are probably good base documents to work from. Note that Lua has multiple values with the same rules as Common Lisp (extra values are discarded, zero values become a single dummy value), and this is symmetric with the treatment of function arguments: missing ones become the dummy value, extra ones are discarded, so multiple values are already integrated into the Lua library.

John Cowan http://vrici.lojban.org/~cowan xxxxxx@ccil.org

Only do what only you can do.

--Edsger W. Dijkstra's advice to a student in search of a thesis