The Lisp Curse

by Rudolf Winestock

This essay is yet another attempt to reconcile the power of the Lisp
programming language with the inability of the Lisp community to
reproduce their pre-AI Winter achievements. Without doubt, Lisp has been
an influential source of ideas even during its time of retreat. That
fact, plus the brilliance of the different Lisp Machine architectures,
and the current Lisp renaissance after more than a decade in the
wilderness demonstrate that Lisp partisans must have some justification
for their smugness. Nevertheless, they have not been able to translate
the power of Lisp into a movement with overpowering momentum.

In this essay, I argue that Lisp's expressive power is actually a cause
of its lack of momentum.

The power of Lisp is its own worst enemy.

Here's a thought experiment to prove it: Take two programming languages,
neither of which are object-oriented. Your mission, if you choose to
accept it, is to make them object-oriented, keeping them
backward-compatible with the original languages, modulo some edge cases.
Inserting any pair of programming languages into this thought experiment
will show that this is easier with some languages than with others.
That's the point of the thought experiment. Here's a trivial example:
Intercal and Pascal.

Now make this thought experiment interesting: Imagine adding object
orientation to the C and Scheme programming languages. Making Scheme
object-oriented is a sophomore homework assignment. On the other hand,
adding object orientation to C requires the programming chops of Bjarne
Stroustrup.

The consequences of this divergence in needed talent and effort cause
The Lisp Curse:

Lisp is so powerful that problems which are technical issues in other
programming languages are social issues in Lisp.

Consider the case of Scheme, again. Since making Scheme object-oriented
is so easy, many Scheme hackers have done so. More to the point, many
individual Scheme hackers have done so. In the 1990s, this led to
a veritable warehouse inventory list of object-oriented packages for the
language. The Paradox of Choice, alone, guaranteed that none of them
would become standard. Now that some Scheme implementations have their
own object orientation facilities, it's not so bad. Nevertheless, the
fact that many of these packages were the work of lone individuals led
to problems which Olin Shivers wrote about in documenting the Scheme
Shell, scsh.

Programs written by individual hackers tend to follow the
scratch-an-itch model. These programs will solve the problem that the
hacker, himself, is having without necessarily handling related parts of
the problem which would make the program more useful to others.
Furthermore, the program is sure to work on that lone hacker's own
setup, but may not be portable to other Scheme implementations or to the
same Scheme implementation on other platforms. Documentation may be
lacking. Being essentially a project done in the hacker's copious free
time, the program is liable to suffer should real-life responsibilities
intrude on the hacker. As Olin Shivers noted, this means that these
one-man-band projects tend to solve eighty-percent of the problem.

Dr. Mark Tarver's essay, The Bipolar Lisp Programmer, has an apt
description of this phenomenon. He writes of these lone-wolf Lisp
hackers and their

...inability to finish things off properly. The phrase 'throw-away
design' is absolutely made for the BBM and it comes from the Lisp
community. Lisp allows you to just chuck things off so easily, and it is
easy to take this for granted. I saw this 10 years ago when looking for
a GUI to my Lisp. No problem, there were 9 different offerings. The
trouble was that none of the 9 were properly documented and none were
bug free. Basically each person had implemented his own solution and it
worked for him so that was fine. This is a BBM attitude; it works for me
and I understand it. It is also the product of not needing or wanting
anybody else's help to do something.

Once again, consider the C programming language in that thought
experiment. Due to the difficulty of making C object oriented, only two
serious attempts at the problem have made any traction: C++ and
Objective-C. Objective-C is most popular on the Macintosh, while C++
rules everywhere else. That means that, for a given platform, the
question of which object-oriented extension of C to use has already been
answered definitively. That means that the object-orientated facilities
for those languages have been documented, that integrated development
environments are aware of them, that code libraries are compatible
with them, and so forth.

Dr. Mark Tarver's essay on bipolar Lispers makes the point:

Now in contrast, the C/C++ approach is quite different. It's so damn
hard to do anything with tweezers and glue that anything significant you
do will be a real achievement. You want to document it. Also you're
liable to need help in any C project of significant size; so you're
liable to be social and work with others. You need to, just to get
somewhere.

And all that, from the point of view of an employer, is attractive. Ten
people who communicate, document things properly and work together are
preferable to one BBM hacking Lisp who can only be replaced by another
BBM (if you can find one) in the not unlikely event that he will, at
some time, go down without being rebootable.

Therefore, those who already know C don't ask "What object system should
I learn?" Instead, they use C++ or Objective-C depending on what their
colleagues are using, then move on to "How do I use object-oriented
feature X?" Answer: "Goog it and ye shall find."

Real Hackers, of course, have long known that object-oriented
programming is not the panacea that its partisans have claimed. Real
Hackers have moved on to more advanced concepts such as immutable data
structures, type inferencing, lazy evaluation, monads, arrows, pattern
matching, constraint-based programming, and so forth. Real Hackers have
also known, for a while, that C and C++ are not appropriate for most
programs that don't need to do arbitrary bit-fiddling. Nevertheless, the
Lisp Curse still holds.

Some smug Lisp-lovers have surveyed the current crop of academic
languages (Haskell, Ocaml, et cetera) and found them wanting, saying
that any feature of theirs is either already present in Lisp or can be
easily implemented — and improved upon — with Lisp macros. They're
probably right.

Pity the Lisp hackers.

Dr. Mark Tarver — twice-quoted, above — wrote a dialect of Lisp called
Qi. It is less than ten thousand lines of macros running atop Clisp. It
implements most of the unique features of Haskell and OCaml. In some
respects, Qi surpasses them. For instance, Qi's type inferencing engine
is Turing complete. In a world where teams of talented academics were
needed to write Haskell, one man, Dr. Tarver wrote Qi all by his
lonesome.

Read that paragraph, again, and extrapolate.

Exercise for the reader: Imagine that a strong rivalry develops between
Haskell and Common Lisp. What happens next?

Answer: The Lisp Curse kicks in. Every second or third serious Lisp
hacker will roll his own implementation of lazy evaluation, functional
purity, arrows, pattern matching, type inferencing, and the rest. Most
of these projects will be lone-wolf operations. Thus, they will have
eighty percent of the features that most people need (a different eighty
percent in each case). They will be poorly documented. They will not be
portable across Lisp systems. Some will show great promise before being
abandoned while the project maintainer goes off to pay his bills.
Several will beat Haskell along this or that dimension (again,
a different one in each case), but their acceptance will be hampered by
flame wars on the comp.lang.lisp Usenet group.

Endgame: A random old-time Lisp hacker's collection of macros will add
up to an undocumented, unportable, bug-ridden implementation of 80% of
Haskell because Lisp is more powerful than Haskell.

The moral of this story is that secondary and tertiary effects matter.
Technology not only affects what we can do with respect to technological
issues, it also affects our social behavior. This social behavior can
loop back and affect the original technological issues under
consideration.

Lisp is a painfully eloquent exemplar of this lesson. Lisp is so
powerful, that it encourages individual independence to the point of
bloody-mindedness. This independence has produced stunningly good
innovation as in the Lisp Machine days. This same independence also
hampers efforts to revive the "Lisp all the way down" systems of old; no
"Lisp OS" project has gathered critical mass since the demise of
Symbolics and LMI.

One result of these secondary and tertiary effects is that, even if Lisp
is the most expressive language ever, such that it is theoretically
impossible to make a more expressive language, Lispers will still have
things to learn from other programming languages. The Smalltalk guys
taught everyone — including Lisp hackers — a thing or two about object
oriented programming. The Clean programming language and the Mozart/Oz
combo may have a few surprises of their own.

The Lisp Curse does not contradict the maxim of Stanislav Datskovskiy:
Employers much prefer that workers be fungible, rather than maximally
productive. Too true. With great difficulty does anyone plumb the
venality of the managerial class. However, the last lines of his essay
are problematic. To wit:

As for the “free software” world, it eagerly opposes industrial dogmas
in rhetoric but not at all in practice. No concept shunned by cube farm
hells has ever gained real traction among the amateur masses.

In a footnote, he offers Linux as an example of this unwillingness to
pursue different ideas. To be sure, he has a point when it comes to
operating systems (the topmost comment, in particular, is infuriatingly
obtuse). He does not have a point when it comes to programming
languages. Python and Ruby were influenced by Lisp. Many of their fans
express respect for Lisp and some of their interest has augmented the
Lisp renaissance. With some justice, JavaScript has been described as
"Scheme in C's clothing" despite originating in those cube farm hells.

Nevertheless, in spite of this influence, in both the corporate and open
source worlds, Lisp still has only a fraction of the developer mind
share which the current crop of advanced scripting languages have
attracted. The closed-mindedness of MBA's cannot be the only explanation
for this. The Lisp Curse has more explanatory power.

The free development environments available for Lisp further exemplify
the Lisp Curse.

It's embarrassing to point this out, but it must be done. Forget about
the Lisp Machine; we don't even have development systems that match what
the average Smalltalk hacker takes for granted ("I've always felt Lisp
is the superior language and Smalltalk is the superior environment."
- Ramon Leon). Unless they pay thousands of dollars, Lisp hackers are
still stuck with Emacs.

James Gosling, the author of the first Emacs that ran on Unix, has
correctly pointed out that Emacs has not fundamentally changed in more
than twenty years. This is because the Emacs maintainers are still
layering code atop a design which was settled back when Emacs was
a grad-student project at the MIT AI Lab, i.e., when Emacs development
was still being indirectly financed by the national debt. A Slashdotter
may object that Emacs is already quite capable and can do anything that
any other development environment can do, only better. Those who have
used Lisp Machines say otherwise.

So why don't the Lisp hackers put the Smalltalk guys in their proper
place? Why don't they make a free development system that calls to mind
some of the lost glories of the LispM, even if they can't reproduce
another LispM?

The reason why this doesn't happen is because of the Lisp Curse. Large
numbers of Lisp hackers would have to cooperate with each other. Look
more closely: Large numbers of the kind of people who become Lisp
hackers would have to cooperate with each other. And they would have to
cooperate with each other on a design which was not already a given from
the beginning. And there wouldn't be any external discipline, such as
a venture capitalist or other corporate master, to keep them on track.

Every project has friction between members, disagreements, conflicts
over style and philosophy. These social problems are counter-acted by
the fact that no large project can be accomplished otherwise. "We must
all hang together, or we will all hang separately." But the
expressiveness of Lisp makes this countervailing force much weaker; one
can always start one's own project. Thus, individual hackers decide that
the trouble isn't worth it. So they either quit the project, or don't
join the project to begin with. This is the Lisp Curse.

One could even hack Emacs to get something that's good enough. Thus, the
Lisp Curse is the ally of Worse is Better.

The expressive power of Lisp has drawbacks. There is no such thing as
a free lunch.