Thank you once again for the thoughtful and well expressed comments.
> Here are my current thoughts on keyword arguments in no particular order:
>
> (1)
>
> There seems to be consensus by those working on other SRFIs (for example
> the POSIX ones) that keyword arguments are very much needed for the API.
> (Personally, I would prefer other abstractions as I have discussed with
> John once, but I seem to be in a minority here.) Under this assumption
> it is important that we are able to talk about keyword arguments in
> R7RS-large.
These are like keyword args, but require more design and/or more code:
- record types
- little languages made out of combinators, pass a config object around
and mutate it with procedure sin that language
These are essentially keyword arguments done in a less
standard/efficient/flexible way:
- case-lambda / optional positional arguments (Emacs Lisp offers many
examples that this doesn't scale)
- split a single procedure into multiple procedures, which are variants
of the same behavior with different arguments
- roll your own keyword arguments (alist/plist/hashtable parsing)
(Am I missing any?)
This question is probably out of scope for SRFI 177, which exists to provide other SRFIs with a keyword argument mechanism. It would be a question for these other SRFIs whether, in a particular case, they should use keyword objects or another mechanism. In the latter case, another SRFI may be needed to provide a uniform way to create, say, these "little languages made out of combinators".
I think the main point here is that keywords args are about convenience
in programming-in-the-large situations where we can't spend a lot of
time to carefully plan every detail. The main disagreement is about
whether the language should exhort people to go ahead and plan anyway
(by using one of the more heavyweight alternatives above). From that
point of view, adding keyword args into the base language would
encourage people to be "sloppy" instead of using record types of little
languages made from reusable combinators.
While I strongly agree that too much focus on convenience is detrimental
to good programming, I think convenience is essential to
programming-in-the-large. Good programmers take pride in well-designed
code, but that can lead to a bit of a taboo against "just solving the
immediate problem and moving on".
In a language core, "just solving the problem" can derail the whole
language and a substantial portion of all the code written in it. But
the further you move away from the core, the less important each bit of
code becomes to a large software system.
If we want Scheme to be used to write large programs, we have to give
programmers some tools to do things that let them solve the immediate
problem without thinking about all the details. Keyword args are nice
because it's immediately obvious how to solve a problem in a way that is
unlikely to turn into a mess later. They have a decades-long track
record where they have proven themselves to be a good tool for that job.
A procedure taking keyword arguments can almost always be salvaged and
recover from design mistake (a badly-designed arg or two) without having
to be thrown away to be replaced by another procedure.
The marathon vs sprint metaphor comes to mind. Big programs are a
marathon, where endurance and covering the distance in the allotted time
are most important. Spending too much effort on individual segments is
actually detrimental to the whole project.
For endurance programming, things like GC, bignums, a library system and
keyword arguments are obvious wins that let people get on with the task
at hand without making an irrecoverable mess.
> For the specification of the consumers of the keyword system, the
> details of the keyword system are, however, not relevant. It is enough
> if the keyword system fulfills a few axioms. In particular, the exact
> syntax of the keyword system is not relevant for the specification of
> the consumers if that specification is written in a way that is
> transparent to the details.
Agreed.
> Likewise, it is probably irrelevant whether
> "procedures" taking keyword arguments are actual procedures or macro
> keywords. Furthermore, whether the keywords are identifiers, objects of
> a keyword type or hygienic identifiers doesn't have to play a role for
> those customers if they are specified with enough care.
I may misunderstand this point, but to me the main point of kw args is
that you don't need to care so much about things ("the luxury of
ignorance"). Hygienic keywords would have to be imported from somewhere;
would it be obvious to callers which modules to import, and how to avoid
keyword clashes?
By *convention*, we can choose not to bind any identifier of the form :XXX or XXX:. These identifiers won't be exported. Hygienic matching would then be by name. If one needs to introduce a new keyword argument that has to be disjoint to any other, you can then bind and export it.
Shiro and John argued earlier on the SRFI 177 list that the advantage of
non-hygienic keywords is not having to coordinate importing keywords.
> For SRFI 177, this means that it can serve as one model for the keyword
> system that is rich enough for the customers. Example code for the
> customers can be written in terms of SRFI 177. Depending on the
> adaption, how this discussion proceeds, and what syntactic transformers
> will be available in future iterations of R7RS-large, SRFI 177 can then
> easily be replaced by a successor. In particular, the successor may use
> a completely different syntax (so that answers to questions like whether
> it is called lambda/kw or keyword-lambda do not have to be set in
> stone).
Agreed. My feelings are not hurt if it's replaced by something else :)
The only thing I really care about is for every present and future
keyword system to be compatible with every other system. (At least for
the basic usage now covered by SRFI 177; for advanced usage,
compatibility is probably not critical.) Compatibility is a requirement
to make Scheme a contender for many people to write large programs, just
like compatibility of library systems has been a boon to Scheme recently.
> In fact, we can leave SRFI 177 can be underspecified, and this
> may be a good idea. For example, we may leave it open (at least for the
> time being) whether SRFI 177 keyword "procedures" are actual procedures
> or just macros.
Interesting idea. This didn't occur to me; I'll think about it.
My main concern is, will people have to think about difficult things
when changing old non-keyword procedures to keyword ones? Keywords are
about convenience, so having to think about subtle edge cases thoughts
every time one uses them would defeat their purpose.
> (2)
>
> When keywords are hygienic identifiers, "procedures" with keyword
> objects *have* to be special forms. This is probably not too bad. With
> keywords hygienic identifiers, we cannot have a generic `apply' for
> "procedures" taking keyword objects. This is possibly bad.
What causes this - can't a hygienic keyword be a self-evaluating unique
object (similar to a non-interned symbol, which is self-evaluating but
different from every other symbol)? E.g.
(import (rename (foo) (keyword-x keyword-1))
(rename (bar) (keyword-x keyword-2)))
Would lead to a situation where keyword-1 and keyword-2 could be passed
to apply/kw (keyword-apply in Racket).
I meant that you cannot write a general apply without having imported the keywords a priori. Maybe this is not that bad as I thought because there is no real use case.
(Tangential remark: IMHO keyword-apply is one of the most questionable
things I've encountered in Racket. I had to use it once or twice and
even basic code written using it is almost unreadable. To me, it's a
strong argument that their approach to keywords is too complex.)
> When keywords are non-hygienic objects, keyword procedure calls that
> should in fact be resolved at expand-time have to be resolved at
> run-time and it needs an intelligent compiler to rewrite the call into a
> fast one. This may also relevant for procedures with keyword arguments
> when they are exported from libraries. The Scheme implementation may
> have no way to look into a compiled library, which would make
> cross-library calls of procedures with keyword arguments slow.
Code like this:
(define/kw (foo &key a b c) (list a b c))
(foo :b 2 :a 1)
can be optimized into:
(define (foo/fast-path a b c) (list a b c))
(define/kw (foo &key a b c) (foo/fast-path a b c))
(foo/fast-path 1 2 #f)
Any compiler that can spot :a and :b are self-evaluating objects can
perform the optimization. What am I missing?
If the procedure definition of "foo" is exported from a library, the module where the foo call happens may not be able to look into the source of "foo" (that would need whole program optimization).
Thus, for a general Scheme implementation, we have to assume that calls to procedures with keyword arguments defined in other libraries are slow (where "slow" means slower than if it was an ordinary procedure or if it were a macro).
Base on my experience with Common Lisp, almost all keyword calls have
the self-evaluating keyword objects at the right places in the argument
list, so almost everything is ripe for optimization in this way.
I guess it can get complex if foo can be redefined into a different
procedure. But it shouldn't be any different from optimizing calls to
particular cases of a case-lambda.
Oh, there is a huge difference and this is why I urge everyone participating in these discussions to write one or at least read a lot of Scheme implementations. For case-lambda, it makes sense that your procedures (whose call sites are not known) to have a finite number of entry points (for, say, zero, one, two, three, four, and arbitrary many arguments). Each entry point is a fast path.
This works for case-lambda because the different paths are only distinguished by the number of arguments and there is a small subset of often used numbers of arguments. For keyword arguments, this does not work as there is no small subset of "usually" used keyword arguments.
As far as I understand it, that's one reason why case-lambda has made it into R6RS. You may want to take a look at the source code of Chez.
> (3)
>
> There should be a way to write wrappers for "procedures" taking keyword
> arguments. In particular, there has to be a way to pass keyword
> arguments down to callees. With unhygienic keyword arguments, there may
> be the problem that keywords conflict with each other. One way to solve
> it is to make sure that all keywords defined in R7RS-large are disjoint.
> But the way Scheme has solved these problems until now has been hygiene.
> So far, the only things that have to be disjoint are the Scheme library
> names. (The feature identifiers as well but (dummy) library names for
> this purpose would have been sufficient.)
The thing is, conflicting keyword can be either a problem or a solution
depending on what you want. There may be cases where someone's problem
is that keyword that look identical behave differently.
In any case, leaving &allow-other-keys and keyword-apply (or apply/kw)
out of SRFI 177 is almost certainly the right call. There are problems
with how to specify them that are not relevant to the basic everyday
usage of known keyword arguments.
> (4)
>
> If keyword arguments are hygienic, macros can easily generate disjoint
> keyword arguments that do not clash with other keyword arguments. If
> keyword arguments are symbols or keywords, something like `gensym' is
> needed. Furthermore, there has to be a way to define procedures that
> take keyword arguments whose names are made on the fly. The same goes
> for calls with keyword arguments.
Another thing we should think about is the culture that is likely to
develop around keywords. So far, all Lispers and Schemers have used
unhygienic keyword arguments; due to their simplicity, this is what
people are likely to continue using by default. That would lead to a
situation where real-world Scheme code is littered with unhygienic
keywords anyway.
In that case, hygienic keywords would have to be made as easy to use as
the unhygienic keywords that people are used to.
See my above remark about not binding the :XXX and XXX: versions, giving us the best of both worlds.
The R7RS library system is an inspiration here. It's quite
sophisticated, yet a Scheme newbie can still easily write a "Hello
world" library without any trouble. Any hygienic keyword system should
either be reserved for advanced usage and tie seamlessly into the
unhygienic keyword system, or be so easy to use that newbies use it
correctly by intuition (just as define-library is easy to use correctly
at the moment).
I agree that making up keyword names on the fly is probably useful from
macros, and should be supported in a compatible way. But IMHO we should
be careful not to end with something keyword-apply in Racket, whose
complexity does not seem to justify the problem it solves. Common Lisp
has a few corners that are just too complex, any way you slice it (loop
and format being the obvious examples). Scheme is so far refreshingly
simple, and that's one of the main parts of its appeal.
In my personal opinion, Racket is a good lesson for Scheme at large. I
find many aspects of it are simply too complex (keyword-apply, pathname
objects, the way immutable and mutable datatypes have incompatible APIs)
and detract from the beauty of Scheme, turning it into another Common
Lisp. But that's a topic for other threads.
I have the same motivation why I am writing on this list. I don't want Scheme to become another Common Lisp or Python by throwing a keyword system on it without thorough consideration.
> (5)
>
> Any macro that distinguishes the function of an identifier argument by
> its name (for example, whether it begins with ":" or not), is really an
> ugly hack. It may be used for some implementation, but the final
> specification should not depend on this. It won't work together well
> with the rest of the Scheme system. I gave one example with `let'.
In practice, we already have the problem that :foo is not portably read
as a symbol. I agree that parsing symbols is a hack, but it merely
levels the playing field so that all implementations then have the same
behavior that some prominent ones already do. With fewer differences
between implementation, the semantics are easier to remember.
I would be fine with having `:foo` read as a keyword object in standard
Scheme like it is in Common Lisp. Then the hack wouldn't be needed.
We don't want to incorporate an ugly wart knowingly into R7RS-large. If R7RS-large is allowed to relax R7RS-small in the regard that :XXX or XXX: may be read as a keyword instead of an identifier, that would be fine. If not, we have to find a better solution.
> Another example is a macro, which expands into a call with keyword
> arguments and that passes down an argument given by the macro user.
> Depending on whether this argument is of the form `:x' or just of the
> form `x' the macro may do completely different things and that would be
> opaque to the macro user.
OK, I see your point. That's true.
However, not all identifiers starting with `:` are parsed - only the
ones that are in "keyword position" in the argument list. Common Lisp
uses keyword args extensively, and it's rare to come across any real
usage that doesn't have the keywords at constant positions.
Above, we have talked about the beauty of the Scheme language. Let's preserve it and don't add things that work "most of the time".
This problem can be avoided with the (call/kw a b c (d 4 e 5 f 6))
syntax but other people didn't like it :)
This syntax is better than having to rely on a hack. As long SRFI 177 is compatible with native systems, people can still use native syntaxes if they like them better or add the hack with the ":" parsing themselves. For that, it doesn't have to be part of the language standard.
Having `call/kw` at the start also says that some magic is going on. If
the ordinary procedure call syntax of Scheme were extended, I would
agree with you that parsing `:` prefixes is too much.
call/kw can happen inside some macro expansion where we don't see it. If we were talking about PHP, I would consider your argument sufficient, though. :) But not for Scheme. ;)
> As long as the Scheme implementations cannot agree on a new fundamental
> type together with its syntax, I would rather add a space in between ":"
> and the identifier and export ":" as auxiliary syntax, e.g. as in
> `(call/kw 1 2 c : 3 d : 4)'. For this, even `syntax-rules' is
> sufficient.
IMHO the syntax where the colon is separate is hard to read and
needlessly different from other Lisps. You're right that the semantics
are easier. However, most real-world keyword usage in CL, Clojure is
quite pedestrian. A more difficult syntax would make advanced usage
easier but at the cost of detracting from everyday usage.
It is one way to get rid of the hack. What we can do is to define the syntax "(call/kw 1 2 (c 3 d 4))" or "(call/kw 1 2 c : 3 d : 4)" in this SRFI, which still allows implementations to provide a better syntax. When we have later agreed in R7RS-large on a, say "!keywords" flag for the reader that reads :XXX and XXX: as keyword objects (or even #:XXX), we can write and addendum to SRFI 177 (say SRFI 277) that adds the new syntax.
Maybe hash-table SRFIs are a good point of comparison here. SRFI 69 is
nice and simple; its successors can do more, but most of the time you
only need the basics.
I have no problem with more than one keyword SRFI as long as their
semantics are compatible for the basic use cases covered by 177.
> Furthermore, it would allow a syntax like `(call/kw 1 2 ,c
> : 3 d : 4)', where `c' here is an expression returning a keyword. This
> is important, see above, to generate new keywords on the fly for example
> in macros.
Good point. This should already possible with the current 177, but you
have to resort to gensym.
How does it work with SRFI 177?
; (R7RS records when they are understood with symbols and not
> hygienic identifiers as field names have the serious deficiency that
> hidden fields cannot be added transparently; we don't have to repeat the
> same mistake.)
Good point.
Clashing record field names are one of the best-known problems with
Haskell: <https://wiki.haskell.org/Name_clashes_in_record_fields>.
> (6) The whole discussion show to me that the foundations for R7RS-large
> that is provided by R7RS-small is very beautiful but also quite small.
Indeed. The more I explore it in different contexts, the more I
appreciate what an excellent standard it is. But no standard is the
ideal foundation for every job.
> If we postpone all serious additions to the language that cannot be
> implemented portably and may need some compromises from the R7RS
> implementers to the end of the R7RS-large specification process, the
> majority of the R7RS-large libraries won't use them.
Excellent point. This didn't occur to me. It could be kind of weird to
construct R7RS-large out of parts that use keywords and other parts that
use optional arguments or plists for similar purposes. Would be nice to
try to preserve some unity in standard Scheme, even though in a project
as ambitious as R7RS-large a few compromises have to be made.
> I wonder whether
> it is a good idea to squeeze an R8RS-small or a R7RS-small+ in between
> where the foundations are revised (e.g. by changing the meaning of the
> `:x' syntax).
I would advise against this on PR grounds. With R6RS and R7RS, Scheme is
already a bit confusing to outsiders. If another standard was added (for
any reason - even if it makes sense from a technical point of view) it
will make Scheme's public image more confusing and may decrease morale
for insiders.
I understand what you mean. I think it will depend partially on how it is advertised. ECMAScript has a new version every year and it doesn't seem to be confusing.
I see two possibilities here that may work: A R7RS' that just incorporates the R7RS errata and very few things where the common opinion is that R7RS would have been better with them (e.g. not parsing of :XXX or XXX: identifiers). The other possibility would be an R6+7RS, which brings together the two main Scheme standards. As Will Clinger has been pointing out, R7RS can almost be considered a superset of R6RS.
What would boost morale the most is to come to an agreement on
fundamental issues like keywords and strings, and ship R7RS-large using
them. John seems to share this opinion, and is doing an excellent job
juggling these conflicting demands.
To the extent that compromises need to be made, I would opt to add
further delay in shipping R7RS-large until we get it right. It will be a
serious blow to Scheme if -large turns out to make unfruitful choices on
core language issues. That's why I very much appreciate that you take
time to argue your points in detail, even against the majority.
Discussions like this are supposed to be difficult even in the ideal
case, and R7RS-large needs to be future-proof on issues like immutable
data and programming-in-the-large, so it's to be expected that it takes
a long time to figure out how it should be.
Thank you very much for giving me the time and space to raise my concerns.
While I like to see a conclusion soon (and I believe that we will be able to agree at some point), we mustn't rush. The addition of (and the form of) keyword arguments will be a *global* change to the language as they will be used pervasively by other SRFIs. For the Scheme user, the impact will be more global than the module system or the future decision what low-level macro system to add to the language.
Thus, we need much more thought here than, say, when discussing hash tables. The exact form of, say, SRFI 125 is only relevant for users of SRFI 125 but not for the language at large.
> (7)
>
> Lassi, do you have a most current version of your SRFI 177 draft? If you
> can give it to me, I will try to make it into an (alternative) proposal
> that addresses some of my points and then we discuss.
I don't have anything more recent than the published first draft. If you
want to draft another document, I can submit a draft #2 that is just
like the current draft but with the names changed to call/kw and
lambda/kw, and adding a define/kw. Those are the only things everyone
has agreed on so far :)
But it seems we have quite a lot of perplexing problems and
disagreements still. It's a long discussion and goes around in circles a
bit, but I don't think that's a problem. There are not that many
difficult issues like this left in the language; if we get keywords,
strings and immutable datatypes sorted, probably the rest of the
language will be easier.
I don't think that we are necessarily going in circles. We would if we didn't learn anything new by talking to each other. Or if we stick to old arguments in spite of the new things we have learned.
See, I have already accepted the necessity of standardizing some form of keyword arguments (in spite of my feeling that at least for the standard library there is no need to use them because we do not have to make up an API in a rush).