Now let's extend the Python version:
def f2():
yield 1
yield 2
yield 3
return 0
Generator objects created by this generator function differ from generator objects returned by f1 in that the StopIteration raised at the end carries the value 0 as a payload. SRFI 158 does not yet allow this, but it could be added easily by attaching a payload to eof objects (implementing this portable would mean to redefine eof-object? in the standard library).
In order to be precise, I should mention that there is a third way how a generator object in Python can yield a value to the caller, namely by throwing exceptions:
def f3():
yield 1
yield 2
yield 3
raise e
But this is nothing that we can't do in Scheme. (In fact, we can do more in Scheme because we can capture any continuations.)
Dually, there are several ways to feed values into a Python generator object:
def f4():
x = yield 1
yield x
Let g be the generator object returned by invoking f4. Invoking next(g) yields 1 and execution of the generator is halted at the first yield expression. Invoking g.send(2) afterwards feeds the value 2 into the generator, assigns that value to x and finally yields it. Any further invocation of next(g) or g.send(...) will cause a StopIteration.
There is another way to feed information into the generator by invoking g.close(). In that case, a GeneratorExit exception is raised at the suspended expression of the generator. The generator may or may not handle the exception, but it must not yield values after the exception was raised. As g.close() does not return a value, it is not usable for accumulators, but it can be used to clean up state in generators that have not yet been exhausted (but are not needed anymore). I think such a functionality is yet missing with SRFI 158 generators, namely stopping them prematurely to clean up state and free used resources.
Finally, one can invoke g.throw(exc, value), which raises the an exception of type exc and payload value inside the generator. The generator can handle this and yield or return a value, which will become the return value of the invokation of throw (a StopIteration in case of return vs yield).
Now, this allows us to implement accumulators, say:
def sum():
i = 0
while True:
try:
i += yield None
except MyExcepti
return i
Let c be the result of sum(). It is a generator object suitable for summing (an accumulator in SRFI 158's terminology). Calling next(c) lets the generator proceed to the first yield. Afterwards, we can do c.send(1), c.send(2), c.send(3), and finally c.throw(MyException, None). This in turn raises a StopIteration exception whose value will be 6.
How can all this be applied to the generator protocol of SRFI 158? First of all, an exhausted generator has to be able to return a non-trivial value. This can be done by adding a payload to eof objects. The beauty of this approach is that it is compatible with SRFI 121. Secondly, generators should be able to receive values. This can be implemented by allowing to call a generator that wants to accept values with one argument. E.g. if g is a generator (object), then the expression (g) is equivalent to Python's next(g), and the expression (g v) is equivalent to Python's g.send(v). The Scheme equivalent of stopping or closing a generator would then be to feed the eof object (possibly with a payload) into the generator. In that way, a SRFI 158 accumulator will just be a special type of generator with one exception: As a (generalized) generator the accumulator would return its final result packed into an eof object while the current protocol for SRFI 158 accumulators returns the bare value.
Although it is a bit more complicated to retrieve the value when it is packed into another object, it allows for many more applications. For example when implementing something like a pipe, which acts as an accumulator and generator at the same time and may want to return some accumulated value at the end of its life.
The only issue I still have with this proposed change (to the accumulators and to the general description of generators in SRFI 158) is that it still does not deal well with multiple values. (This makes it no worse than the draft's proposal because it has the same issues). It makes perfect sense to yield multiple values at each step of a generator's life. It makes even more sense to feed multiple values into an accumulator at each step. On the other hand, the eof object is just a single object, so an accumulator would have to be a procedure with a variable number of arguments (which is aesthetically not very pleasing and may have performance problems) and the consumer of a generator employing multiple values would need to be able to deal with different numbers of values coming out of the generator (same problems).
A more radical change (fitting more into Scheme's model of evaluation) would be to deliver an explicit continuation to each invocation of a generator. The generator would then invoke that continuation in tail context to signal that it is exhausted instead of returning normally. The explicit continuation could simply be (lambda () (eof-object)) giving back the behavior as specified in the draft. In general, however, the two different code paths (yielding a value and signalling the end of the sequence) would be clearly separated.
The same would have to be done for the feeding of values into a generalized generator (again, think of an accumulator): Whenever a generator is constructed, two procedures would have to be returned instead of one. Invoking the first procedure (with or without arguments) would simply send values into the generator. Invoking the second procedure would cause the generator to stop. By wrapping the first procedure into a procedure which calls the second whenever an eof object is received, we get back the protocol of SRFI 158.
Let me give two examples to illustrate my point where I use the prefix x- to denote the new versions of the SRFI 158 procedures:
(define g
(x-make-coroutine-generator
(lambda (yield)
(yield 1 2)
(yield 3 4)
5)))
Then, invoking (g f) would yield the multiple values 1, 2. Afterwards, 3, 4 are yielded. By invoking (g f) for a third time, finally (f 5) would be called in tail context. We can define make-coroutine-generator in terms of this model:
(define (make-coroutine-generator c)
(let ((g (x-make-coroutine-generator c)))
(lambda ()
(g (lambda args (eof-object))))))
(Here, due to the limitation of the SRFI 158 protocol, the return value 5 will have to be ignored if eof objects have no payload.)
An accumulator example would be:
(define-values (a s) (x-count-accumulator))
Invoking (a f 1), (a f 2), (a f 3) would feed the values 1, 2, 3 into the accumulator (the continuation f is just in case the accumulator, which would be a generalized generator in disguise, stops prematurely). Afterwards, we can invoke (s f). This would call f in tail context with the number of values fed in, that is 3.
We can get back (count-accumulator) as follows:
(define (count-accumulator)
(let-values ((a s) (x-count-accumulator))
(lambda (v)
(if (eof-object? v)
(s values)
(a values v)))))
If (x-)make-coroutine-generator is extended so that yield can return values into the generator, one can even implement (x-)count-accumulator very cleanly using (x-)make-coroutine-generator.
Marc