Why Ruby is an acceptable LISP

Posted by Eric Kidd Sat, 03 Dec 2005 16:30:00 GMT

Years ago, I looked at Ruby and decided to ignore it. Ruby wasn’t as popular as Python, and it wasn’t as powerful as LISP. So why should I bother?

Of course, we could turn those criteria around. What if Ruby were more popular than LISP, and more powerful than Python? Would that be enough to make Ruby interesting?

Before answering this question, we should decide what makes LISP so powerful. Paul Graham has written eloquently about LISP’s virtues. But, for the sake of argument, I’d like to boil them down to two things:

LISP is a dense functional language.
LISP has programmatic macros.

As it turns out, Ruby compares well as a functional language, and it fakes macros better than I’d thought.

Ruby is a denser functional language than LISP

A dense language lets you say things concisely, without obfuscation. You can see more of your program in one glance, and there aren’t as many places for bugs to hide. Beyond a certain point, the only way to make programs denser is to use more powerful abstractions.

One particularly powerful abstraction is lambda. Using lambda, you can create a new function on the fly, pass it to other functions, and even store it for later use. For example, if you wanted to double each number in a list, you might write:

(mapcar (lambda (n) (* n 2)) mylist)

mapcar creates a new list by transforming each element of mylist. The transformation, in this case, could be read as “for each value n, multiply n by two.” In JavaScript, you’d write lambda as function, which is perhaps a bit clearer:

map(function (n) { return n*2 }, mylist)

Of course, this is only a hint of what you can do with lambda. Languages which favor this style of programming are called functional languages, because they work with functions. A dense functional language can be very concise indeed, and quite clear once you learn to read it.

How does Ruby stack up against LISP for functional programming? Let’s consider Paul Graham’s canonical example, a function which creates an accumulator:

(defun foo (n) (lambda (i) (incf n i)))

This code is marginally shorter in Ruby, and the notation will be more familiar to C hackers:

def foo(n) lambda {|i| n+=i} end

acc = foo 3
acc.call(1)   # --> 4
acc.call(10)  # --> 14
acc.call(0)   # --> 14

But there’s an interesting special case in Ruby which saves us even more typing. Consider a (very silly) function which takes a lambda as an argument:

;; Call 'fn' once for each natural number.
(defun each-natural-number (fn)
  (loop for n from 1 do (funcall fn n)))

;; Print 1, 2, 3...
(each-natural-number
 (lambda (n) (format t "~D~%" n)))

Now, we could write the same function in Ruby:

def each_natural_number(fn)
  n = 0
  loop { fn.call(n += 1) }
end

each_natural_number(lambda {|n| puts n })

But we can do better. Let’s get rid of lambda and fn using yield:

def each_natural_number
  n = 0
  loop { yield n += 1 }
end

each_natural_number {|n| puts n }

Yes, yield is a special-purpose hack, and yes, it only works for functions which take a single lambda. But in heavily functional code, yield buys us a lot. Compare:

[1,2,3].map {|n| n*n }.reject {|n| n%3==1 }

(remove-if (lambda (n) (= (mod n 3) 1))
           (mapcar (lambda (n) (* n n))
                   '(1 2 3)))

In a large program, the difference adds up. (In LISP’s defense, it’s possible to write a reader macro which makes lambda more concise. But this is rarely done.)

Ruby gives you about 80% of what you want from macros

At this point, the LISP hackers are saying, “A good syntax for lambda is nice, but what about macros?” And this is a good question. LISP macros are functions that:

Run in the compiler, and
Transform custom syntax into raw LISP.

The most common use of LISP macros is to avoid typing lambda quite so much:

(defmacro with-each-natural-number (n expr)
  `(each-natural-number (lambda (,n) ,expr)))

(with-each-natural-number n
  (format t "~D~%" n))

defmacro defines a function that takes a list as an argument, and returns another list. In this example, our macro is called every time the compiler sees with-each-natural-number. It uses LISP’s “backquote” syntax to quickly construct a list from a template, filling in n and expr. The list is then passed back to the compiler.

Of course, this macro would be useless in Ruby, because it’s working around a problem we don’t have.

The second most common use of LISP macros is to create mini-languages for defining stuff:

;; Generate some bindings to our database
;; using a hypothetical "LISP on Rails."
(defmodel <order> ()
  (belongs-to <customer>)
  (has-many <item> :dependent? t))

Using Ruby on Rails, we could write:

class Order < ActiveRecord::Base
  belongs_to :customer
  has_many :items, :dependent => true
end

Here, belongs_to is a class function. When called, it adds a bunch of member functions to Order. The implementation is pretty ugly, but the interface is excellent.

The real test of any macro-like functionality is how often it gets used to build mini-languages. And Ruby scores well here: In addition to Rails, there’s Rake (for writing Makefiles), Needle (for connecting components), OptionParser (for parsing command-line options), DL (for talking to C APIs), and countless others. Ruby programmers write everything in Ruby.

Of course, there’s lots of advanced LISP macros which can’t be easily ported to Ruby. In particular, macros which actually compile mini-languages haven’t appeared yet, although they might be possible with enough work. (Ryan Davis has done some promising work in this direction with ParseTree and RubyInline, and I’ll be writing about related techniques as I discover them.)

Ruby’s libraries, community, and momentum are good

So if LISP is still more powerful than Ruby, why not use LISP? The typical objections to programming in LISP are:

There aren’t enough libraries.
We can’t hire LISP programmers.
LISP has gone nowhere in the past 20 years.

These aren’t overwhelming objections, but they’re certainly worth considering.

Once upon a time, Common Lisp’s standard library was considered huge. But today, it seems painfully tiny. Java’s manuals fill a wall, and Perl’s CPAN archive has a module for anything you can imagine. Common Lisp, in comparison, doesn’t even have standard way to talk to the network.

Similarly, LISP programmers are scarce. If you’re around Boston, there’s a small pool of grizzled hackers who can very nearly work magic. Elsewhere, there’s a thin scattering of curious young hackers. But LISP has always been a minority language.

Ruby, on the other hand, is growing rapidly in popularity. The big driver seems to be Rails, and the ramp-up started in late 2004. If you’re trying to launch a company, it’s more-or-less a cliché that every potential employee is a Rails nut. Rails will soon trickle back into ordinary web consulting, and from there–eventually–into big business.

Ruby has also been around long enough to develop a good standard library, and a large archive of add-on libraries. If you need to download a web page, parse RSS, generate graphs, or call a SOAP API, you’re all set.

Now, given a choice between a powerful language, and popular language, it may make excellent sense to pick the powerful one. But if the difference in power is minor, being popular has all sorts of nice advantages. In 2005, I’d think long and hard before choosing LISP over Ruby. I’d probably only do it if I needed optimized code, or macros which acted as full-fledged compilers.

(Thank you to Michael Fromberger for reviewing an early draft of this essay.)

Tags LISP, Macros, Recommended, Ruby

Comments

nuke edit
Excellent Article said about 4 hours later:

Eric, thank you for this excellent article. I have toyed with Lisp for several years now, and while I agree it is more powerful, I have made the decision to build my sites in Ruby for many of the reasons you have cited.

I am constantly re-evaluating my choice of language tools, and have wondered if I made a mistake not building my site in Lisp. (It’s still in development now, but keep an eye our for Zifus around the first part of 2006).

I recently read an article by the developers of reddit that suggested they are considering re-writing their site at some point in the future. He cited as a specific difficulty with Lisp that many of the libraries that quickly become available in other languages are slower to appear in Lisp because of the smaller community.

Who knows if he was dissembling to throw potential competitors (like me) off track, but in the end it doesn’t matter, the point is valid. Whereas I have been incredibly happy with the progress of Rails as a framework, and I have been utterly delighted with my personal productivity in this wonderful framework.

Excellent article, too bad I can post it on Zifus yet, but soon, very soon.
nuke edit
zlxcgrogdss said about 5 hours later:

I’m looking forward to your next editorial, perhaps it could be titled something like “Why Java is an acceptable Forth”. Maybe, if I’m lucky, “Why assembly is an acceptable binary”.

What I’m, err, hinting at, is that Ruby is not Lisp. A Lisp is a Lisp because it’s code is represented as its basic data structure. Ruby’s isn’t (unless you argue that Ruby’s code consists of strings, and string is a baic data structure; though I’m sure you’ll agree that this would be pure sophistry). Lisp without this property is like a wheel that isn’t round.

The above distinction, along with all of its implications, is the reason Ruby can not be a substitute for Lisp, which, I am assuming, was what you meant to say when you erroneously called Ruby “a Lisp”. Ruby is neat and all, but let’s not get carried away.
nuke edit
bill a said about 6 hours later:

I took basically the opposite path – I abanboned Ruby in favor of Lisp.

You have some good points, and I agree that Ruby is the most Lispy language that isn’t actually a Lisp (CL, Scheme, Dylan). But although blocks can get you surprisingly close to macros, they can’t take you all the way there. As you pointed out, the implementation of macro-like code in Ruby tends to be ugly behind the scenes. Ruby’s syntax isn’t regular, and so can’t be operated on with macros. This means you’ve got to shoehorn various abstractions into doing what you want. Instead of processing the code passed to a block as a list and simply transforming it into something else, you have to define the elements of your mini-language as methods in some class. The pervasiveness of Ruby’s OO becomes a hassle here. (Sure, lots of defmacro’s end up being ugly too, but at least they’re conceptually cleaner). And all user-defined languages and structures immediately stand out because they have to be set off with the “do” keyword.

There are too many things about Lisp that no other language will ever get unless it is also a Lisp. The ability to move through sexp’s with Emacs keychords is priceless. The simple structure of Lisp and its lack of arbitrary syntax makes it easy to provide good editing support. The whole idea of interactive development rocks. The ability to, in most implementations, compile to native code rocks as well. Lisp is pretty mature, and while there are a few quirks to it, 50 years of existence will go a long way toward making a programming language great.

I guess what bothers me about Ruby is its quirks. That might sound crazy, since it’s well-known that Common Lisp has more than its share of historical cruft. But I really cannot get my mind back into the framework of arbitrary syntax. It is ultimately useless. It prevents you from writing macros, it prevents editors from having flawless support for your language, and it prevents you from expressing yourself as concisely as you need to.

Your lambda sample is indeed shorter in Ruby than in Lisp. But Ruby’s OO is not quite so clean in this situation:
```
(defmethod sum-children ((c item))
  (loop for child in (children-of item)
        sum (height-of child)))
```
```
(mapcar #'sum-children list-o-children)
```
I don’t think that can be done quite so easily in Ruby. The best I can come up with is:
```
class Item
  def sum_children
    children.inject( 0 ) { |c, s| c.height + s }
  end
end
```
```
list_o_children.map { |c| c.sum_children }
```
It isn’t possible to (easily?) treat the concept of summing the children’s height as a function to pass to others.

Also, blocks are little more than callable code. You can try to impose some structure on them by defining appropriate methods and evaluating the block in a special context, but this is not the same as the code transformation macros provide. You cannot, for instance, selectively evaluate different expressions in different contexts. Consider:
```
(let ((x 45))
  (defclass foo ()
    ((name :initform x))))
```
That will work because defclass will expand the initform (and only the initform!) into a lambda-expression and thus close over the x variable; nothing else will be evaluated in that context. It’s also important to notice that defclass does not look any different than let, even though defclass is simply a macro, and let is a built-in special form.

I will clearly go on for days unless I stop now. I was about to write about how let’s rock because you can limit a variable’s scope with absolute precision (not possible in Ruby) and then about how Marco Baringer implemented continuations in straight Lisp. To sum up: yes, Ruby is not that bad (maybe even “acceptable”) but Lisp owns.
nuke edit
Eric said about 7 hours later:

Ruby’s most frustrating quirks, from a LISP perspective, tend to involve its scoping rules. Methods, variables and constants each have subtly different behavior. And as for block locals, even Matz admits that they’re pretty broken.

Let’s consider a Ruby version of LET:
```
def let(*args) yield *args end
let(1,2,3) {|x,y,z| z }
```
This does exactly what you want if the variables are unbound: It introduces a new scope with three local variables. On the other hand, if any of the variables were previously bound, Ruby clobbers the existing bindings. Nasty.

The biggest limitation of trading LISP’s compile-time macro expansion for Ruby’s metaclass hackery is launch time. Rails works around this using FCGI and on-demand class loading.

These headaches aside, I’ve had very good luck porting LISP macros to Ruby. At this point, I could probably reimplement about half of the macros in On LISP without much trouble. Another quarter could be brute-forced using ParseTree, but that’s (arguably) cheating. I’ll talk more about these techniques in the coming months.
nuke edit
Anonymous said about 7 hours later:

Hi. I’ll just assume here you always meant Common Lisp when you wrote LISP (it isn’t capitalised there, just like PERL and JAVA aren’t right).

Before answering this question, we should decide what makes LISP so powerful. Paul Graham has written eloquently about LISP’s virtues. But, for the sake of argument, I’d like to boil them down to two things: LISP is a dense functional language. LISP has programmatic macros.

Sorry to say, but that doesn’t reflect reality. If those were the two main reasons to use Common Lisp, most people probably wouldn’t. Incidentally, Paul Graham’s Common Lisp style is definitely atypical (google for “Graham Crackers”). His Scheme background/affection certainly shows, and he usually doesn’t mention all the other things that make Common Lisp so useful (unlike in Scheme, avoiding side-effects and using recursion instead of iteration in Common Lisp is rather rare). Despite the roots, Common Lisp isn’t much of a “functional” language. It has all the necessary features, but imperative constructs and OOP play a major role in the language. It’s not Common Lisp if it doesn’t have the Common Lisp Object System; it also wouldn’t be anywhere as powerful without the condition system and all the other things (full numeric tower; the type system; etc.).

I have no doubt that Ruby might match CL in the functional department, but that won’t get you very far. Common Lisp has come a long way, and even though many people seem to reduce it to macros and higher-order functions, that view does certainly not reflect the way people use it in reality—it’s not very attractive or particularly useful (in comparison) when reduced to that set of features.

2005, I’d think long and hard before choosing LISP over Ruby. I’d probably only do it if I needed optimized code, or macros which acted as full-fledged compilers.

I’m sure Ruby is a good choice, but if you’re using Lisp just for the macros, you’ve probably missed all the interesting features… (In case you’re curious, Practical Common Lisp is a good, freely available book that covers a lot more ground than just macros and higher-order functions)
nuke edit
null said about 9 hours later:

You guys don’t understand…your precious little languages is a moving target. You’ll never write code that’ll survive 20 years with them. If all you’re doing is web pages, I suppose that’s all right. But only to a point where data integration and AI comes to the web. Than, Ruby, Perl and Python are gone.

What Lisp needs is more people. If the lisp community had /half/ of the people who enthusiastically throw themselves full hearted in every new hype, we’d have those libraries.

But don’t kid yourself, we already have our mini CPAN, some stuff for web, and free stuff is progressing all the time.
nuke edit
Erik Enge said about 10 hours later:

Nice article, though I think zlxcgrogdss (second comment) is spot on with why Ruby is not a Lisp. Anyways, “there aren’t enough libraries” is being worked on and you can see current progress at http://common-lisp.net/projects.shtml and http://cliki.net/Library for those who might be curious.

Hope that’s an ok comment to leave, if not , feel free to pull it.
nuke edit
Dookus Binglebib said about 10 hours later:

But don’t kid yourself, we already have our mini CPAN

Wow, already?
nuke edit
eh said about 11 hours later:

try embedding prolog in ruby.
nuke edit
Eric said about 11 hours later:

Anonymous, I agree that CLOS is a Good Thing.™ In particular, generic functions lead to much cleaner designs than traditional message-passing. This is a subject near and dear to my heart. You’re also right to call out Common Lisp’s condition system. It’s an elegant approach to exception-handling, and one that future language designers should study carefully.

That said, if your Common Lisp style doesn’t rely heavily on macros, you’re missing out on half the fun, not to mention your number one sales pitch. :-)
nuke edit
Eric said about 11 hours later:

Null, Dookus, could you please be polite? Heated arguments are perfectly OK, but please address each other with respect.

EH, I’ve been wrestling with the Prolog-in-Ruby problem for a few weeks now. The backtracking is easy, and I’ve gotten a decent pattern matcher running. The right-hand side of the rules, though, is proving a little tricky.
nuke edit
Garrett Snider said about 13 hours later:

Thanks Eric for the article and everyone else for the comments. I’ve been coming at Ruby as a long-time Java guy with a now and again interest in LISP and wandering what the LISP community thinks of Ruby. I love it as a step up from Java and I’ve been thinking that it is reasonably close to LISP, at least in power (not in language design). However, I wanted to get a deeper undestanding of the (practical) differences.

Also, I’ve taken the pill that allows me to love the “arbitrary syntax” mentioned by ‘bill a’ above. I find it hard to convince others who have the PERL vs. Python dualing philosophies in mind and stand on the Python (/Java) side, but as a way to implement Domain Specific Languages, I think Ruby’s open syntax shines. Where I’ve had a question is on the following issue:

If the additional syntax options are layered on top of a solid consistant core (like with LISP), it seems to be solid advantage, perhaps even outshining the venerable LISP in this respect. (Specifically, I’m thinking of Executable DSLs.) So commentary here regarding the fundamental soundness and consistancy of the Ruby language vs LISP is especially interesting and appreciated.
nuke edit
bill a said about 13 hours later:

OK, guys….you can say whatever you’d like about Lisp, but please, please, please stop capitalizing it. It’s Lisp, not LISP.
nuke edit
jimmy said about 13 hours later:

Thanks, Bill. That was bugging me too.
nuke edit
Smalltalking with a Lisp said about 13 hours later:

Ruby was inspired mainly by Smalltalk and Lisp. If you really want your hair blown back, check them out. Ruby and Python are gateway languages to the above.
nuke edit
bill a said about 14 hours later:

I am curious how much of Ruby comes from Lisp. I would tend to think that people who are familiar with Lisp do not go around writing languages. :)
nuke edit
Unordinary Programmer said about 14 hours later:

“Some may say Ruby is a bad rip-off of Lisp or Smalltalk, and I admit that. But it is nicer to ordinary people.”

- Matz, LL2
nuke edit
Anonymous Coward said about 16 hours later:

Hmm. Lisp macros are far more powerful than the trivial use cases you’ve listed. I could give a lot of examples here, but just ask yourself: why is most of the programming community so fond with “Design Patterns”, while the Lisp community generally isn’t?

Well, that’s because patterns are nothing but high-level specifications for code being rewritten again and again and again. The Lisp approach is to create some macros and auxiliary functions that actually implement the pattern, thus extending the language capabilities and avoiding continuous reinvention of the wheel.

It has happened for object orientation (CLOS anyone?), aspect-oriented programming, OO<->SQL mapping… All without touching the basic language syntax and implementations.

Ruby features may compete in some specific cases, but I bet that the same consideration would apply for other languages too.

Finally, regarding the Lisp libraries: it’s true that the language specs are full of historical cruft (e.g. mandatory support for the now-dead versioning filesystems), and lack things like standard sockets. But it’s also true that there’s a good amount of free (as in freedom) libraries that fill the gap. Have a tour on http://cliki.net/ for an appetizer.
nuke edit
Beowulf said about 16 hours later:

I’ve made a relatively lengthy response to the posts here on my blog. http://zifus.blogspot.com/2005/12/power-vs-popularity.html

You may not agree with my reasoning, but realize this is the reasoning process I went through when I decided to build my system in Rails instead of Lisp. Realize that I love Lisp and I would have used it if I could’ve focused on solving my actual problems rather than trying to integrate various frameworks together and struggling to get it all working as a cohesive whole. The biggest reason I abandoned Java and C# is that I wanted to focus on my problem, and not fight with getting the language and framework to do what I want. This is coming from a lover of Lisp, imagine what someone who either isn’t a fan of the language or doesn’t even know it exists is thinking.
nuke edit
Tayssir John Gabbour said about 20 hours later:

Ruby’s creator is outspokenly influenced by Lisp and Smalltalk, yet Ruby’s success is supposed to be a terrible thing to Lisp users? Particularly since Smalltalk’s creator also speaks of his influence and admiration of Lisp? ;)

I (perhaps badly) recall that things in motion tend to have lower friction coefficients. If Ruby’s technical advantages get the mainstream moving, great.

I think this is a more reasoned article than I remember seeing from the Java and Python worlds.

However, I think it has two honest errors. For one thing, I wouldn’t call Common Lisp a functional language, if by that we mean it is biased towards recursion over other techniques. I use iteration heavily; Lisp’s LOOP macro is pretty impressive in this regard, and there are even more powerful 3rd party iteration constructs.

Perhaps Rubists focus on functions because they correctly find a lot of power in them, but then they’re viewing Lisp through Ruby-colored glasses.

Second, as for plain macros, these are just functions which (at some time like “compiletime”) take code and return code. They take a sexp and return a sexp.

What’s the purpose? Usually expressiveness. Readability.

Since they can use other aspects of the language like dynamic scope, they multiply the expressive power of the language. But they must be used tastefully, as power requires justification.

Since I don’t write lambdas much in the first place, my macro usage certainly ain’t to save effort typing lambda. ;) Completely orthogonal concepts.
nuke edit
Anonymous (again) said about 21 hours later:

From Eric: That said, if your Common Lisp style doesn’t rely heavily on macros, you’re missing out on half the fun, not to mention your number one sales pitch.

Ok, I agree with that; my initial response may have been a little rash :-) My point remains, though, I’d say that more of half of the fun comes from all the other parts often not mentioned. Macros are an essential part, but just one of many.

To Beowulf: It sounds to me like you are playing the bitter ex-martyr yourself (no offense intended) there. Multiple commercial Lisp vendors are doing just fine, even though their implementations are not cheap and even though there are various actively developed high-quality open-source implementations. That alone shows that the market exists. As David Thornley put it, “Lisp doesn’t look any deader than usual to me”, and all the people saying it the opposite won’t change that (funny, back when I used FreeBSD, “FreeBSD is dead” was the typical (periodically occuring) outcry of many not-users. I guess it’s not particular to Lisp after all). I agree that it isn’t as widely used for open-source software as many contemporary languages are, but it is used surprisingly often once you look further; in particular, the harder the problems you’re solving, the more it seems to be used. The nature of most free Lisp implementations tends to favour long-running programs (they don’t focus on small initial memory footprints or delivering small binaries), and the benefits of the language help you more the harder your problems get—abstractions scale (in dealing with the complexity of your program), syntactic shortcuts don’t. I suspect that this is also why most/many Lispers don’t care too much about getting a shorter syntax for (lambda (...) ...): It simply doesn’t win you much (in terms of shorter code size) in non-trivial programs; other factors become much, much more important. (This is also why micro-benchmarks are usually not representative unless you write fibonacci or factorial programs all day)

You must occupy a niche if you will [...]

No, (in my humble opinion) the whole power of Common Lisp comes exactly from the fact that it doesn’t focus on niches. It focuses on power. If you want to occupy a niche, you can easily extend Common Lisp to do that (I think it’s called “writing your own DSL” nowadays :-); Prolog-in-Lisp, Lisa, AllegroCache and AspectL are just a few examples of that (all implemented as normal libraries, IIRC). I’m certain that Common Lisp couldn’t have survived for this long weren’t it so extensible: Perl, Python, Java, they and most other new languages all reinvent themselves every other year, and I’m sure they would have been succeeded by “competing” languages by now if they didn’t — most niches are moving targets and the competition doesn’t sleep. The Common Lisp standard, in comparison, has not changed for a long time and probably won’t (need to) anytime soon. Sure, that means that writing a simple webpage in PHP may be simpler; Perl is certainly more comfortable to use for a quick one-liner; Erlang should beat Common Lisp easily in the domain Erlang was created for, and so on. Where Common Lisp shines, however, is when your problem is not covered by a typical niche. It shines when your problem is hard and your requirements aren’t simple, when you actually can make use of the advanced features. If another language has a library/framework that does just what you want to do, and Common Lisp doesn’t, why use Common Lisp, anyway? I think using one language for everything is usually a bad sign (lack of flexibility on part of the developer), not a good one (no language is best at everything, and I never understood people who think that. Also, life would be terribly boring if that were so). It’s simply not true that people didn’t/don’t write frameworks in Common Lisp, but I suspect that what you consider a framework is usually just called a library in Common Lisp. If you think it is dead in the commercial world, franz.com (one commercial Lisp vendor) alone has more than enough success stories/impressive products to prove that wrong (I guess Lord of the Rings (the movie), Jak and Dexter and Orbitz are typical modern examples). cliki.net / lispwire.com have plenty of links to open-source libraries/software.

I’ll close with a quote from Kent Pitman: “…Please don’t assume Lisp is only useful for Animation and Graphics, AI, Bioinformatics, B2B and E-Commerce, Data Mining, EDA/Semiconductor applications, Expert Systems, Finance, Intelligent Agents, Knowledge Management, Mechanical CAD, Modeling and Simulation, Natural Language, Optimization, Research, Risk Analysis, Scheduling, Telecom, and Web Authoring just because these are the only things they happened to list.”
nuke edit
Eric said about 22 hours later:

Bill, the lowercase spelling “Lisp” is certainly correct for Common Lisp (and I’ve fixed it in the main article—thanks!). The uppercase spelling, however, has a long and respectable history. The new way certainly is easier on the eyes, but my LISP teachers were a bit old-school, and I haven’t overcome my nostalgia yet. :-)

As for your query about how often LISP hackers write new languages: Oddly enough, it’s a bit of an obsession. Yeah, I know, it makes no sense. But H. sapiens always did tend to mess with perfection.

Tayssir, I would be delighted if Ruby encouraged programmers to take a good look at LISP (and SmallTalk). And Ruby is certainly good news for metaprogramming in general. Maybe we’ll finally stop hearing about macros being “too powerful,” a personal peeve of mine.
nuke edit
Ben said 1 day later:

Nice article. I’m not a Ruby user but I will say one language feature it has which Common Lisp doesn’t—first class continuations.

Also, given the success which Python has had in porting to the CLR, has anyone seriously thought about making a Lisp-like language for .Net? I mean, there are some things out there, but they seem either not well supported or rather ML-like (which is fine but not exactly like Lisp.)
nuke edit
Joku said 1 day later:

The article didn’t really compare how things could be done in Python, even though it started from the premise that Ruby might be more powerful. Not really a surprise though, because it would’ve turned out that Python code can be nearly identical in complexity to Ruby code (sometimes slightly more complex, sometimes slightly less). And neither Ruby nor Python come close to providing the same power as macros in Lisps.

I don’t think you gave any reason to consider Ruby over Python.
nuke edit
bill a said 1 day later:

Eric,

Sure, many years ago “LISP” was the correct rendition, but I don’t think that’s been the case for quite some time.

Incidentally, all of those examples you pointed out (Dylan, Scheme, Goo, Logo) are all dialects of Lisp. :-)
nuke edit
Anonymous Coward said 1 day later:

About continuations in Lisp: here’s an example of a language extensions provided by using macros.

Maybe Common Lisp doesn’t have “native” continuations because they can be created in several ways. See “On Lisp” by Paul Graham for an example (the book is downloadable from his website, if you don’t want to spend money).

Maybe they are not syntactically as nice as the Scheme built-in continuations, but for practical pourposes they are just “good enough”. And more syntactic sugar is always available by using some macro magic.
nuke edit
First Rest said 1 day later:

This article sounds like it was written for folks who really want to use Lisp, but have chosen Ruby because all the cool kids are using it and want to reasonably justify an emotional decision.

Lisp has been around a long time and has watched other, more popular languages attempt to catch up to it. Here’s a clue: they still haven’t. Figuring that out involves a difficult learning process that challenges most of the things you have learned and involves looking at things in an entirely new way. If you want to learn a language that can change with the times and incorporate whatever latest fad the programming cool kids have to offer, Lisp is the choice. Lisp is the red pill.

One question to ask one of the other commenters (who put his comments on a blog that doesn’t allow anonymous posting) that mentioned the small Lisp community: If everyone does like you did and decide to move on, how would the community ever grow? Low hanging fruit is so much easier to pick.
nuke edit
Lisp + Python = Lython said 1 day later:

What about a sexpr-based language built on top of Python, that generates Python bytecode, can access Python libraries and even supports macros?

http://lemonodor.com/archives/000648.html

(Ok, it’s just a prototype, but here’s my 2c about the Ruby vs Python vs Lisp issue).
nuke edit
Eric said 1 day later:

It’s true, Joku, that Python metaprogramming has become more common recently, using techniques very similar to those popular in the Ruby world. This is a good thing, and deserves broader notice.

Guido, however, has never been much a fan of functional programming, and has stated (PDF, page 4) that he doesn’t like lambda, map, filter, reduce, and similar constructs. Normally, this isn’t a problem, because Python has slick list comprehensions:
```
[ n*n for n in [1,2,3] if (n*n)%3!=1 ]
```
In a lot of common cases, list comprehensions will look better than either the Ruby or LISP examples above. But there’s an underlying problem: List comprehensions are a special-purpose feature, and I can’t use the same mechanisms for building my own domain languages. And since Python’s lambda is very limited, it generally won’t help much.

Effectively, Python allows me to create new kinds of definitions (by hacking metaclasses), but it doesn’t allow me to define new control constructs. Normally, this doesn’t matter much, but it’s a nuisance if you’re trying to build something like Rake.
nuke edit
bill a said 1 day later:

One big difference between Lisp and Ruby is that Ruby gets its dynamicity and its metaprogramming capabilities by doing a lot of code evaluation at runtime. To people who aren’t familiar with Lisp, it might seem that this is the only way to do it.

So for instance in Lisp, I might write a macro to define part of a domain-specific language. That macro is then expanded into other code at compile-time, and that’s all there is to it. Ruby, on the other hand, takes a much more runtime-centric approach; instead of transforming code, I just have to make sure that a given block is evaluated in a context where the methods it calls can be accessed. So Ruby can’t make many assumptions about code it reads until that code is actually running.

The two approaches have advantages and disadvantages, but I (of course ;-) find Lisp’s approach cleaner. It means that the compiler can warn about calling undefined functions, and that development environments can easily do things like locate code definitions. With Ruby’s approach, in general the semantics of any given snippet of code can only be determined at runtime. To be sure, Lisp does have an eval function, but it doesn’t get used much at all outside of Lisp itself. This is a good thing. When I was using Ruby, I was sometimes frustrated about this situation – that all decisions about code are made at the last possible minute. Of course, this lets you do things like method_missing and so on, but I have yet to miss that (no pun intended) since switching to Lisp. I don’t mean to suggest that everything in Lisp must be statically prepared (as in C or Java), only that in general Lisp takes a different approach to dynamicity.

Just another point to keep in mind.
nuke edit
Eric said 1 day later:

Sweet! Lython’s definitely a slick hack, and—judging from the sample code—the author noticed that Python’s bytecode is actually more LISP-like than its official syntax. Python’s if statement really is an expression; the parser just refuses to nest it. In general, the Python Language Services are a pretty promising framework for implementing a macro system; somebody just needs to roll up their sleeves and make it happen.

Bill, you’re absolutely right about LISP and compile-time versus runtime performance. Exhibit A: Ruby on Rails takes a couple of seconds to launch a new server process. With a good compiler, LISP could do the same thing as fast as the OS could read pages into memory.

The long-term trends probably favor the LISP (or even Scheme) approach, mostly for reasons of IDE support and better compile-time checking. The Ruby approach is actually dangerous if your team doesn’t maintain unit tests.
nuke edit
Beowulf said 1 day later:

First Rest, I enabled anonymous comments, I didn’t realize you had to turn that on. :(

I agree with you that Low hanging fruit is easier to pick, which was exactly why I chose Rails for my system. That doesn’t mean I’ve moved on though, rather it means for this system at this time, I didn’t feel like Lisp was the right choice.

As someone else mentioned, increasing popularity of Ruby is a good thing for Lisp. Anything that gets people looking at languages more powerful than the Java/C/C++ paradigm is a good thing for Lisp.

Eric, my point wasn’t that Lisp should become a niche language. Although in most people’s mind Lisp already is a niche language, it’s the AI language to them. I was suggesting that Lisp needs to find a new place to occupy in people’s minds in order to increase in popularity. The best way to do that imho is to create a shiny new object (ala Rails) for people to play with and promote the heck out of it.

Lol, nice retort with the bitter ex-martyr. It made me laugh if nothing else. :-p.

I’ve followed and have been involved in numerous Lisp is better than language “x” discussions. Everyone that I have seen devolves into the all too familiar battle of syntax. I just think that people won’t switch languages because the solution to summing children is better in Lisp. I think that misses the point entirely. People are after real world, tangible benefits. Little 3 and 5 line snippets of code simply do not show the actual power and expressiveness of Lisp. Most people cannot extrapolate a 3 line snippet into a real system.

I think there is great news that Rails is what is driving adoption of Ruby. It serves to illustrate that people can and will pick up a new language, even an obscure one with relatively wonky syntax.
nuke edit
bill a said 1 day later:

Beowulf, I don’t think I ever said the solution to summing children was a reason to switch to Lisp. I was countering Eric’s assertion that Ruby is more “functional” and “functionally dense” than Lisp. Of course it “misses the point entirely” if that’s how you’re going to interpret it.

Real-word, tangible benefits? OK, here you are:

- macros give you the power to increase the readablility, concision, and correctness of code. Consider the LOOP and ITERATE macros. Common patterns of looping are expressed with amazing simplicity simply because LOOP and ITERATE can perform code transformations. For instance, let’s say you want to loop over the elements in a sequence and return the one with the highest absolute value:
```
(iter (for x in seq)
      (finding x maximizing (abs x)))
```
I think that’s fancy. And when you consider that the ITER form is going to expand into wicked-fast code at compile-time (and then be compiled to machine code, in some LISP implementations), you realize that you’ve just gotten a powerful abstraction and increased readability for free. No runtime efficiency or spacecompletely is wasted with this construct, because ITERATE has efficient idioms to transform common looping patterns into damn fast code. Directly using the faster code would mean a huge loss in readability and maintainability – with Lisp macros, I don’t have to worry about it. For the curious, the macroexpansion for that ITER form is at the bottom of this common.

For more macros, take a look at Peter Seibel’s book Practical Common Lisp ( available free at http://www.gigamonkeys.com/book). In one of the final chapters he builds a binary file parsing system that’s plain amazing – and he does it with macros. Earlier in the book, he writes a test framework in something like 26 lines – using macros too. Here is a sample of the code used to parse ID3 tags in MP3’s:
```
(define-tagged-binary-class id3-tag ()
  ((identifier     (iso-8859-1-string :length 3))
   (major-version  u1)
   (revision       u1)
   (flags          u1)
   (size           id3-tag-size))
  (:dispatch 
    (ecase major-version
     (2 'id3v2.2-tag)
     (3 'id3v2.3-tag))))
```
It specifices the types and order of the data to be read from the file, and then passes control to another binary-class based on the value of major-version – and even automatically throws an error if the major-version is invalid.

- interactive development. If you haven’t used this, you’re missing out. Most languages are about writing lines of code, feeding them to either an interpreter or a compiler, and then watching what happens. In Lisp, development is always done live and interactively. You write functions in files and then send them to a Lisp image. You develop each component of your system one by one and test it live. There is no compile-run-debug cycle – there isn’t even an interpret-debug cycle. It’s more like a write-test cycle.

Interactive development also means you have access to your running system. Not sure what parameters a fucntion takes? Start typing “(func ” and the Lisp environment will remind you what arguments can legally follow that. Want information on an object you’re playing with? Type “(inspect x)” and an interactive, recursive inspector will let you walk through it, even making changes as you go.

Once you get your program working, you refactor a little, do some benchmarks, take advantage of Lisp’s completely optional type system to speed up bottlenecks, and sit back and admire your work.

- the condition/restart system. In most languages, there is not much that can be done after an exception is raised. Log it, send it to the user, end the program. In Lisp, an error can be restarted. For instance, in the UncommonWeb framework, when you browse to a URL and an error is signaled, you get the backtrace in your environment. You can now make changes to your code, recompile it, and then invoke the RESTART-REQUEST restart. UCW starts processing your request from the beginning, and the browser gets the result of running your latest, changed code.

- regular syntax. In Lisp, all forms can be broken down into (operator [args]). There is nothing else. This means that I can write macros, but it also means that editors can do really neat things with code, because Lisp is regular. I can move up, down, forward, backward, into, and out of lists with Emacs keychords. This might be difficult to appreciate until you’ve tried it, but I really recommend it. Refactoring code becomes trivial when you can treat control structures as a single entity and cut and paste them.

- almost 50 years of use and development. Lisp has survived longer than every language except FORTRAN. It survived C, C++, Ada, COBOL, and it will survive Java and Ruby. It is, to borrow Paul Graham’s term, “the hundred-year language.”

- more!

All of this said, I ought to admit that I do use Ruby and Rails in all of my professional projects. This is not because I don’t think Lisp is ready, but simply because I hadn’t learned Lisp when I started these projects. I now regret that, and my next web application will be in Lisp.

As promised, here is the crazy but efficient expansion of the ITERATE form above:
```
(LET* ((#:LIST6 NIL)
      (X NIL)
      (#:TEMP7 NIL)
      (#:RESULT5 NIL)
      (#:MAX8 NIL)
      (#:FIRST-TIME9 T))
 (BLOCK NIL
   (TAGBODY
     (SETQ #:LIST6 SEQ)
    LOOP-TOP-NIL
     (IF (ATOM #:LIST6) (GO LOOP-END-NIL))
     (SETQ X (CAR #:LIST6))
     (SETQ #:LIST6 (CDR #:LIST6))
     (SETQ #:TEMP7 (SQRT X))
     (COND
      (#:FIRST-TIME9 (SETQ #:FIRST-TIME9 NIL)
                     (SETQ #:MAX8 #:TEMP7)
                     (SETQ #:RESULT5 X))
      (T
       (COND ([snipped to prevent
               overflow -ed.])
             (T #:RESULT5))))
     (GO LOOP-TOP-NIL)
    LOOP-END-NIL)
   #:RESULT5))
```
- Bill (who is heartily enjoying this discussion)
nuke edit
Johnny said 1 day later:

Ruby is more than just a dynamic or scripting language. But feel free to snob it. More “Rails-like” projects are on the way.
nuke edit
First Rest said 1 day later:

Lisp lets YOU be Gosling or Matz or Guido. If you want to extend the language, it’s possible:

“DOLIST is similar to Perl’s foreach or Python’s for. Java added a similar kind of loop construct with the “enhanced” for loop in Java 1.5, as part of JSR-201. Notice what a difference macros make. A Lisp programmer who notices a common pattern in their code can write a macro to give themselves a source-level abstraction of that pattern. A Java programmer who notices the same pattern has to convince Sun that this particular abstraction is worth adding to the language. Then Sun has to publish a JSR and convene an industry-wide “expert group” to hash everything out. That process-~~according to Sun~~-takes an average of 18 months. After that, the compiler writers all have to go upgrade their compilers to support the new feature. And even once the Java programmer’s favorite compiler supports the new version of Java, they probably still can’t use the new feature until they’re allowed to break source compatibility with older versions of Java. So an annoyance that Common Lisp programmers can resolve for themselves within five minutes plagues Java programmers for years.”

—Peter Siebel from Practial Common Lisp
nuke edit
Garrett said 1 day later:

Thanks Bill. That’s very helpful stuff.
nuke edit
null said 1 day later:

Some things not mentioned: meta-object protocol; sexpr (code is data, data is code); for commercial lisps, amazing IDEs.

Schemes also have advantages: smaller language – philosophy is move the rest of code to libraries; free near-C fast compilers (Bigloo) (free Schemes have better IDEs than Lisp); possibility prototype algorithms with call/cc and finally translating to efficient iterative algorithms in imperative languages (like C).

A final example: look at the 2 open source Computer Algebra Systems written in Common Lisp available as Open Source: 1) Axiom – originally from IBM Thomas Watson Research Center; 2) Maxima – originally Department of Energy (US). This is software written in the 70s. You can’t write software that lasts so long with a language that is a moving target. This is the kind of survival and complex domain which shows the power of Lisp.

All these scripting languages are really about people experimenting with language syntax. This is nice, but look at it for what it is. Additionally, almost none of those languages have any theory attached to them. Lisp has (lambda calculus), Smalltalk has (Actors), ML has (type theory). This garantees good, lasting design. Take a look at Perl6 for an example of extreme design experiment. Where it’s going, nobody knows, but – hey! – they’re using Haskell in the implementation (I always thought Perl people were smart).

Lisp was very expensive and demanding of hardware in the 70s and 80s. This is not so anymore. There is no great shiny free Lisp IDE (yet), but at least one vendor sells a good one for a relatively good price. Surely, this can’t keep people from using lisp, since people have taken to relatively “rough” stuff like Perl. By the way, there’s a great Perl book called Higher Order Perl that is really about learning a lot of techniques from Lisp and applying to Perl. I bet a lot of them could be applied to Ruby.

All these scripting languages do is create a problem and sell you a solution. But to be fair, scripting languages empowered programmers like Lisp had not (“batteries included”, etc).

I think the problem with Common Lisp and Smalltalk is the size of the stuff you have to learn, syntax-wise. Some jobs that required scripting languages in the 90s required learning something fast (everything is more complex nowadays – but at least you’ve got frameworks). Also, Perl, Python, etc, were very portable. Syntax-wise, however, I think the approach taken by Scheme and Eiffel are more sensible (there are reasons for the size of Common Lisp as well as Smalltalk – but I believe it is an impediment).

Who knows, maybe people will try Lisp because of Ruby. I bet more people will get frustrated than contribute back. Also try Scheme. It’s a different philosophy and gets even less press than Common Lisp, but it’s worth it.
nuke edit
bill a said 1 day later:

Boy I wish there was a comment editing facility here. More typos:

“the bottom of this comment” – not “the bottom of this common” an extra “completely” in “no extra space or time is wasted” “even allowing you to make changes’ instead of “making changes as you go”

Also, a clearer version of the iterate example would be:
```
(iter (for x in '(-1 2 -10 4))
      (finding x maximizing (abs x)))
```
```
;; => -10
```
Sorry, guys. I guess these don’t make too much difference, but hopefully this comment will clear up some confusion.
nuke edit
bill a said 1 day later:

null, that’s good advice but those not familiar with Scheme should remember that Common Lisp and Scheme are very different languages. The superficial similarities in their syntax don’t make that obvious, and so some people perceive them as pretty much interchangeable. I personally do not care for Scheme, and yet Common Lisp is by far my favorite language. So if you’ve had exposure to Scheme and didn’t like it, don’t dismiss Common Lisp out of hand (and vice-versa, too).

I agree that it does take a lot of work and mind-warping to “get” Lisp and I also agree that that’s probably helped to hold it back. I think Peter Seibel’s book Practical Common Lisp (I cannot recommend this book strongly enough!) goes a long way to resolving that.
nuke edit
Johnny said 1 day later:

You remember what the Japanese car companies did to GM? :-) Big, powerful, shiny, etc, aren’t enough to match simplicity and hardwork. Lisp people ignore 90% of the Ruby features.
nuke edit
bill a said 1 day later:

Sigh. Johnny, have you even read the discussion? Do you know what’s being discussed, or are you just making emotionally-driven, knee-jerk reactions because the superiority of your favorite language is being called into quexstion? Lisp people are aware of all of Ruby’s features because many of them come from Lisp. No one here is attacking Ruby (at least I’m not). The point of the discussion is simply that Lisp is awesome. Ruby’s geography has nothing to do with its properties as a language, and many of the concepts in Lisp are actually, believe it or not, much simpler than Ruby’s. It’s really not intelligent to make claims about the Lisp community, especially when you quite plainly don’t know Lisp. Also, what exactly is wrong with being a “dynamic language”? You use the term as if it were an insult. What is Ruby if not a dynamic language?
nuke edit
Beowulf said 1 day later:

Bill, that was a very well written comment. One definite pain point for me has been the lack of macros. There is enough similarity in my comment and scoring system in different areas that I really have wanted a better method for abstraction. Rails has the concept of Partials and Components, which can be used to abstract your design, however they also carry a pretty significant performance overhead. That overhead has forced me to abandon them on some pages because they simply were to slow with good abstractions in place. Several good macros on the other hand would have allowed me an elegant level of abstraction without that same type of performance hit.

Practical Common Lisp (PCL) is definitely an important book for Lisp. It took me a long time to find a good book to learn Lisp from (this was prior to the release of PCL), and I finally had to go back to Lisp by Winston and Horn. A book written in 80’s which is no longer in print. In PCL there is finally a well written introduction to Lisp that is relevant to solving the problems of the Internet age.

I honestly hope Lisp gets an upsurge in popularity. Things seem to be moving more and more in the direction of power and expressiveness. It just appears to be happening at a relatively glacial pace. Lisp clearly has the power to stay the course though.

As Eric mentioned in the original article though, Ruby really is a good next choice. Paul Graham once said if you can’t use Lisp, use Python. I’ve used Python for a long time now, and I definitely prefer Ruby to Python at this point. As a language it is much closer to Lisp than Python, and it’s OO syntax is much cleaner. That being said, it’s not a bad thing to know all three (Lisp, Ruby, and Python).
nuke edit
Eric said 1 day later:

Johnny, Bill, please keep it polite. We’re having a good discussion here, and I don’t want to take away anyone’s vowels. :-)

Bill’s point is excellent, and deserves elaboration. I can easily translate his code to Ruby:
```
class Array
  def find_maximizing
    result, f_result = nil, nil
    each do |x|
      f_x = yield x
      if f_result.nil? || f_x > f_result
        result, f_result = x, f_x
      end
    end
    result
  end
end

[1,2,-3].find_maximizing {|x| x.abs }
```
But my code is going to run much slower than Bill’s, because the iter macro lives in the compiler. The same goes for the IDv3 parser: It would look fine in Ruby, but get absolutely clobbered on performance.

A year from now, this could actually be a pretty good sales pitch for Common Lisp. “It’s like Ruby, but it runs at full hardware speed!”
nuke edit
Johnny said 1 day later:

Bill, you are one of the few who claim to be proficient at both, Ruby and Lisp. The problem is that the features that you or other people are going to use in Lisp are just a subset of the features that Ruby is used for. For example, Ruby is very suitable for shell programming, instead of Bash, Sed, Awk, Perl, etc. I just recently started using Rake, and it’s a really awesome tool to use for shell programming. RubyC, mkmf.rb, setup.rb, etc, really make it easy to create extensions for Ruby that bind to external libraries. I don’t think people enjoy creating bindings for Lisp as much as people do in Ruby.

So, if all you want to do is to create WebApps in Lisp, fine. But that use is just a subset of what Ruby is normally used for. I know that Lisp is quite powerful as I said that myself. I know it’s used for tons of different things. But WebApps really are what most people seem to be doing these days. Though with Ruby you can use it for more than just Web site handling.

So, when I claim people don’t know most Ruby features, it’s just because they are spread among many use-cases, libraries, etc, just like in Lisp.

Code RESTful in Ruby!
nuke edit
Eric said 1 day later:

Null wrote: I think the problem with Common Lisp and Smalltalk is the size of the stuff you have to learn, syntax-wise.

I’ve taught Scheme to a lot of non-programmers over the years. The syntax is problematic, but only until they find a good editor. (Emacs, sadly, doesn’t work for many non-programmers.) The other big impediment is usually LET, which introduces too much nesting for some folks.

The larger obstacle is learning to think in LISP (or SmallTalk, or Haskell, or any other powerful language). Bill can post a dozen excellent macros, and most people will say, “Huh?” Until you know how to use it, it’s just a weird feature.

It’s actually easier to sell a language by solving small problems (my web apps are ugly!) than by proposing profound new features. Ruby gets in the door because (1) Rails is slick, and (2) the syntax doesn’t look too odd. And once people have played with ActiveRecord for a while, they begin to understand why metaprogramming is cool.
nuke edit
Eric said 1 day later:

Johnny, even though few people write Makefiles or shell scripts in Lisp, it’s certainly possible. Lisp is basically a toolkit for building programs like Rake.

So why don’t you see more great Lisp frameworks? Mostly because (until very recently) the Lisp community was small, fragmented, and didn’t have anything like CPAN or RubyGems. Most of the Good Tricks™ had been written down only once or twice, and the books had gone out of print.

Fortunately, a lot of the missing pieces have appeared in the last five years, and many of the rest will be along soon.
nuke edit
timsuth said 1 day later:

Eric: “def find_maximizing”

Ruby 1.9 has “max_by”.
[1, 2, -3].max_by { |x| x.abs }
nukeedit bill a said 1 day later: OK, here’s another example of ITERATE that I hope showcases a little more of macros’ sweetness: (iter (for x in '(1 2 3 -4 -10 -43 49 49 8934)) (until (= (sqrt 7) x)) (collect x into collection) (finding x maximizing (abs x) into y) (finally (return (list collection y)))) ;; ((1 2 3 -4 -10 -43) -43) This is a lot more complicated than the other example, and still not really useful, but I hope it illustrates how ITERATE provides a whole sublanguage for expressing any kind of iteration you can think of. I’d also like to point out that ITERATE can be extended with user-defined macros. For instance, I’ve written a macro called AVERAGING that works with ITERATE: (defmacro averaging (var) "Add a clause to ITERATE to average the value of VAR." (with-unique-names (count total) `(progn (summing ,var :into ,total) (counting ,var :into ,count) (finally (return (/ ,total ,count)))))) (iter (for x in '(1 2 3 4 5)) (averaging x)) ;; => 3 There’s more Lisping going on here, and some of this probably looks pretty esoteric (incidentally, this example also shows off Lisp’s docstrings – that string after the defmacro line will be associated with AVERAGING as its documentation, so I can run (describe ‘averaging) and get that string at runtime). But notice that I’m working with an extensible sublanguage for expressing the most common concepts in iteration. The examples Eric and timsuth gave could be defined in any language with first-class functions, but no language but Lisp could define a custom mini-construct like this. Note also that even though ITERATE provides new coding constructs, I still have full access to Lisp within: (iter (for x initially 0 then (1+ x)) (if (evenp x) (print x)) (when (= x 10) (return x))) In this (contrived) example, everything except the first line is straight Lisp. This sort of thing isn’t possible in Ruby. Here’s another example: (iter (for line in-file "foobaz" using #'read-line) (collect line)) This will collect every line in file “foobaz” into a list. (iter (for line in-file "foobaz" using #'read-line) (collect line at beginning)) will collect the lines in reverse by adding each to the beginning of the result instead of the end. (iter (for line in-file "foobaz" using #'read-line) (collect line at beginning result-type 'vector)) will collect the lines into a vector instead of a list. And so on and so on, ad infinitum (or nauseam, since my comments are starting to get repetitive). The point is that the authors of ITERATE have built a mini-language that is: 1) able to capture the most common iteration constructs, even relatively complex ones, like maximizing a function and collecting items based on some condition 2) user-extensible 3) written in Lisp and not part of the Common Lisp language itself – that is, you or I could sit down and write ITERATE and no one would know it wasn’t part of the language 4) fast, and yet still high-level There is much more to macros than just ITERATE, but I think ITEREATE is such a good example of the kind of power that regular syntax and macros can give you. And keep in mind that all this ends up as fast, tight machine code. And if it still isn’t fast enough, throw in some type declarations and optimization settings: (iter (for x in '(1 2 3 4 5)) (declare (type fixnum x)) (declare (optimize (speed 3) (safety 0) (debug 0) (compilation-speed 0))) (averaging x)) I hope that these comments are useful to people who are curious about Lisp. I wish I’d known about some of these things many years ago. All code untested. - Bill P.S. While I appreciate Eric’s request that we keep it civil here, Johnny, saying that Lisp only has a subset of Ruby’s features is outrageous. It is quite, quite the opposite.
nukeedit bill a said 1 day later: OK, since I’ve spent the majority of my day commenting here, it is time to go do some other things, but I would like to also call attention to the fact that those optimizations in the last example are local only to the block that contains them – I think that’s pretty damn cool. And just one more example of macros. AllegroCache is a commercial database for Lisp that lets you make queries in Prolog. DISCLAIMER: I do not know Prolog very well, and I remember the examples I’ve seen very vaguely, so this may not be exactly and completely accurate, but it gives you a rough idea: (<-- (parent ?x ?y) (db person ?x children ?c) (member ?y ?c)) (<-- (grandparent ?x ?y) (parent ?x ?z) (parent ?z ?y)) Here, the <—macro indicates that the following forms will be in prolog. What we’ve done here is define parent and grandparent relations using Prolog. So we declare roughly how the two relate to each other, and then we can make queries on the database with these. So say I have a variable x with myself in it. I can then run the Prolog query macro, ?-, like so: (?- (parent me ?x)) and AllegroCache will return the name of anyone in the database who has me as a member of their children list. I could do the same with grandparents. Folks, I think this is one of the coolest things I’ve ever seen. Again, these are macros, so all the prolog gets reduced to Lisp code at compile-time, but really have you ever seen anything this cool? Can you picture the hideousness of the SQL that would do this? Lisp owns. OK, for real, there are many more things in life than the RandomHacks weblog, and I must go and attend to them. Hope I’ve made a difference.
nukeedit Johnny said 1 day later: Alright, Bill. Sorry about my “car” analogy. Sorry for pretending that Lisp users use a subset of what’s normally available for Ruby. It’s just that the world is too polite sometimes. If you use an “obscure” language and nobody talks about it, you are guilty of not spreading the world. If you use an “obscure” language and people talk about it, you are guilty of “hyping” it. It seems that we can only “hype” it if we present a revolutionary tool like Rails. Fine. Maybe in the future we will have some more of that. Till then, see you.
nukeedit bill a said 1 day later: Nope, there will be one more comment. First of all, the Prolog clauses do not actually expand into any serious code – they expand into calls to AllegroProlog functions and a little bit of bookkeeping. My bad. I think I gave the impression that they directly become code that is equivalent to the Prolog.
nukeedit JAP said 1 day later: I could start this comment with LOL, because the proposed statement is showing so perfectly the real deep misunderstanding of many of us regarding Lisp: So let me shortly explain my point of view: The real reason that brought me to Lisp (and never will get me away from it) is simply this: You can build it out of 7 (s-e-v-e-n) primitive operators! And, as a consequence, no other language can be expressed in itself as short as Lisp. (As you probably know, quite any language can be expressed in itself.) So let me repeat: really no other language can be expressed in itself that short, and (as a natural consequence) can be built out of less primitive operators. All the other features of Lisp are only natural consequences of the above. Because McCarthy did not “create” Lisp (as any other language designer does), he did “discover” it!! (If you read this sentence for the first time, please read it another 2 times). So the real point is: no other language can be more simple and more flexible than Lisp at the same time. BASTA! JAP (just another Paul [not Graham]) P.S. For the philosophers of us: we are all together very difficult, and really have a hard time to become more simple for the rest of our lifes ;-)
nukeedit Johnny said 1 day later: One last comment: maybe “Lisp is the most algorithmic language”.
nukeedit Haskell Junkie said 1 day later: Well, macros are fine and dandy, but don’t let the lisp diehards lead you astray. Learning the Haskell language will show you how to do all the DSL and other tricks using just higher order functions. After putting about 3 months effort into learning Haskell, you’ll probably come to see most macro usages as unnecessary hacks. I’d even bet that someone’s already done monads in Ruby. Interesting examples to get you motivated include parser combinators and backtracking.
nukeedit Eric said 1 day later: Bill, if you’re interested in combining Prolog with a Lisp dialect, I’ve heard that The Reasoned Schemer is an excellent introduction to logical programming in Scheme. Haskell Junkie, thank you for the links to the parser and backtracking papers. Haskell is often unrepresented in these discussions, which is too bad, because it’s a powerful and unique language. And yes, somebody is working on monads in Ruby. Once again, a big thank you to everyone for having kept this discussion friendly. Good manners make it easier to have heated, passionate arguments about important stuff. :-)
nukeedit bill a said 1 day later: Eric, I’ve got PAIP here when I finally get around to learning Prolog in Lisp. I was mostly just impressed that it could be used to make complex queries on a database. Haskell Junkie, can you give some examples?
nukeedit SB said 1 day later: “So let me repeat: really no other language can be expressed in itself that short, and (as a natural consequence) can be built out of less primitive operators.” Except Forth. Or Factor. Or Joy. Let’s not be ignorant here :)
nukeedit Ralph said 1 day later: As a humble current user of PHP and MySQL – but having been around since FORTRAN and Assembler and happy IBM 1620 Machine Language days – I want to say that I am enjoying the discussion here. In a way it sounds like a conversation among chess players, or artists. The “coolness” of a language, its elegance, its expressive power, are all important to language designers. I have constructed a few little languages, one of which even went into production in a small way. Recursive descent compiler, byte code interpreter, multi-threading at the application level. I understand the basics. But most of us, even those of us who do love elegance, nevertheless gravitate toward whatever language environment seems to get the job done with the least total effort. That means common operations must be expressed as easy-to-read idioms. If you consider the spoken form of natural language for a moment, I think you will find that it is essentially all made out of idiom. In writing, I can express something like, “The ship did not contain any containers containing containers, but it did contain several containers containing parts used to construct containers”—and here I use a bit of abstraction, made possible by a few minor regularities in the English language. No one who has been reading this thread is likely to have had any trouble immediately understanding what I just wrote. But in spoken language, it is quite difficult to express the above both clearly and concisely. Even people frequenting this forum might not be able to express or understand that thought quickly in spoken language unless something of the context had been pre-established. The reason for that, I think, is that there is probably no established idiom for even such a very slightly complex thought. If such thoughts had to be expressed with any frequency, an idiom of some kind would immediately and “spontaneously” arise. Then the above sentence would become trivial to express and to understand. Syntactic sugar provides idioms for commonly performed actions. Obviously, then, it must also reduce generality. Idiom is supposed to reduce generality. “Admitting,” then, that one uses Perl, or even (gasp!) PHP, is like admitting that one uses ordinary spoken English. But we all do that. So, I’m eagerly awaiting Lisp on Rails, and all sorts of other great stuff, plus a hosting company that lets me write my web pages in that great stuff. Please. Use some of that “hard problem solving” ability to get it all accomplished, and I will be thrilled to be your very first customer. My current provider gets about $20 from me each month, depending on disk space and network bandwidth used. I am not trying to be facetious, much less rude. I am trying to raise a serious point, even though I did end on a rather familiar, even caustic note, for which I apologize if it jangled anyone’s nerves.

Random Hacks

Technology and Other Fun Stuff