So what was the original purpose of Lispy? I’ll give you the tag line that I wrote when Lispy was nothing but an idea… Lispy is a distributed self-optimizing program by example language.

What does that mean? When I first became interested in programming I had 2 goals; to create a program like MetaStock (I’ve since written pyTrade) and to create Artificial Intelligence. Modest goals, I know. In my AI research I came across genetic programming and immediately took a liking to it, probably because I was still new to writing code and the idea that I could write code that wrote code fascinated me.

The main problem with genetic programming is that it is often difficult to write fitness tests for your problems. Somewhere along the way I noticed that programmers were writing fitness tests all the time in the form of unit tests for their code. In addition, the fact that 99% of the time the CPU sits idle while we browse reddit, kept rattling around in my head. Why not take advantage of those wasted cycles by using a programming language that used those cycles to optimize the code you just wrote?

Then I had another idea. Why write code at all? Why not just write the unit tests and let the code evolve on its own? It took me about an hour to realize that this wasn’t going to work on anything but the smallest applications. However another few weeks and I realized that, maybe it could work, given enough computers were running Lispy.

Anyway I have about 25 half-finished papers on Evo which I might get around to finishing and posting here. I’ll give the highlights and post a link to the github account where you can find more info. Evo is meant to be integrated with Lispy (though I don’t know when I’ll get around to that).

Evo is an evolutionary search based function optimizer. It provides an interface for defining new modules and functions as well as an evolutionary programming based method for optimizing those functions. Evo does not use the standard generational approach to GP, rather it uses “gene pools”, that contain functions of indefinite life, from which new functions can be evolved and tested one at a time.

When defining an Evo function you can choose to optimize it for speed, space or code length. Each function can have multiple gene pools that are each optimized for a different purpose. Using gene pools instead of a single population generational approach helps to avoid stagnation within the population as is a common occurrence with the standard generational model.

Evo is very much unfinished, however it can successfully run on its own and evolve solutions to problems. Because of this I am going to put the code out there in case anyone wishes to contribute. I used the implementation of Evo as an excuse to firm up my understanding of closures, as a result the majority of the program is written in a slightly OOP fashion. There is also a Tk based GUI for browsing modules and adding new functions and fitness tests, though its use is completely optional.

Without further ado, here’s the link to the Evo repo.

Implementing Scheme will let us test our implementation with a language that is already specified.

Create a new file, I’ll call it scheme_in_lispy.lispy, but you can name it whatever you like. We’ll be doing most of our work here so I’ll leave out the main lispy file.

We’ll start with define-primitive which we’ll simply copy from the old syntax.chicken, though we won’t use it until later.

(scheme-syntax define-primitive
  (lambda (expr env)
    (set-symbol! (car expr)
                 (make-primitive (eval (cadr expr)))
                 env)))

The first primitive form we’ll introduce is define. For this we’re going to do a little more copy and paste. Scheme’s define form is just a combination of our previous define and function. define sets a value to a symbol, but it also provides a shorthand for setting a lambda to a symbol (define (a x) x) is equivalent to (define a (lambda (x) x)). To figure out which one we need to produce, we only have to look at the first value in expr. If it is a symbol we just set the symbol name to the evaluated value. If it is a list we create a proc and set that as the value to symbol.

(scheme-syntax define
  (lambda (expr env)
    (if (list? (car expr))
        (set-symbol! (caar expr)
                     (make-proc (cdar expr)
                                (cdr expr)
                                env) env)
        (set-symbol! (car expr) (lispy-eval (cadr expr) env) env))))

(define a 42)
;===> #<unspecified>
a
;===> 42
(define (b x) x)
;===> #<unspecified>
(b 42)
;===> 42

lambda is simple after that. All we have to do is rip out the (make-proc …) procedure and drop it into lambda with one small change. With define we received a procedure definition as ((name arg-1 arg-2 arg-n) body), however an anonymous function does not have a name. We receive a lambda definition as ((arg-1 arg-2 arg-n) body).

(scheme-syntax lamb
  (lambda (expr env)
    (make-proc (car expr)
               (cdr expr)
               env)))

(define c (lambda (x) x))
;===> #<unspecified>
(c 42)
;===> 42
((lambda (y) y) 42)
;===> 42

if is a straight copy/paste from our old syntax file.

(scheme-syntax if
  (lambda (expr env)
    (if (lispy-eval (car expr) env)
        (lispy-eval (cadr expr) env)
        (lispy-eval (caddr expr) env))))

(if 1 2 3)
;===> 2
(if #f 2 3)
;===> 3

quote is one of the simplest forms in Scheme. All quote does is return its argument unevaluated.

(scheme-syntax quote
  (lambda (expr env)
    (car expr)))

(quote (+ 1 2))
;===> (+ 1 2)

set! is pretty simple too, it’s basically just a limited form of define.

(scheme-syntax set!
  (lambda (expr env)
    (set-symbol! (car expr) (lispy-eval (cadr expr) env) env)))

(set! a 42)
;===> #<unspecified>
a
;===> 42
(set! b (if #f 2 3))
;===> #<unspecified>
b
;===> 3
(set! c (lambda (x) x))
;===> #<unspecified>
(c 42)
;===> 42

begin is pretty straightforward, in fact we have already implemented it as eval-body (we used it for procedures).

(scheme-syntax begin
  (lambda (expr env)
    (eval-body expr env)))

(begin 1 2 3)
;===> 3
(begin (define x 42) x)
;===> 42

let is almost the same as begin, except we have to extend the environment with the given bindings before we evaluate the body of the let. This version uses cons cells instead of 2 element lists to set symbols. You could add support for (let ((x 35) (y 7)) …) if you like. Since making a Scheme is not my goal, I will not do that now.

(scheme-syntax let
  (lambda (expr env)
    (eval-body (cdr expr) (extend-environment (car expr) env))))

(let ((x . 35) (y . 7)) (if x x y))
;===> 35
x
;===> Error: Unbound symbol: x

equal? is easily snarfed from the underlying Scheme. equal? could also be defined using define-primitive (define-primitive equal? equal?).

(scheme-syntax equal?
  (lambda (expr env)
    (equal? (lispy-eval (car expr) env)
            (lispy-eval (cadr expr) env))))

(equal? 2 2)
;===> #t
(equal? 2 (if 1 2 3))
;===> #t
(equal? 1 2)
;===> #f

Finally we’ll snarf some primitives from the underlying Scheme to make our mini-Scheme a little more usable.

(define-primitive + +)
(define-primitive - -)
(define-primitive < <)
(define-primitive > >)
(define-primitive car car)
(define-primitive cdr cdr)
(define-primitive cons cons)
(define-primitive print print)

(+ 1 2)
;===> 3
(- 3 2)
;===> 1
(< 1 2)
;===> #t
(> 1 2)
;===> #f
(cons 1 2)
;===> (1 . 2)
(car (cons 1 2))
;===> 1
(cdr (cons 1 2))
;===> 2
(define (loop x) (if (< x 0) (print 'finished) (begin (print x) (loop (- x 1)))))
;===> #<unspecified>
(loop 3)
3
2
1
0
finished
;===> #<unspecified>

There is a lot that is left out, error handling and advanced features like call-with-current-continuation or macros, for example. But for a simple Scheme to help test the implementation of our interpreter this is just about all we need.

Most of the rest of Scheme can be implemented as derived forms from the primitives we just defined. What cannot be, can be implemented using our scheme-syntax macro.

Finally, here’s the full source of scheme_in_lispy.lispy for you to play with.

(scheme-syntax define-primitive
  (lambda (expr env)
    (set-symbol! (car expr)
                 (make-primitive (eval (cadr expr)))
                 env)))

(scheme-syntax define
  (lambda (expr env)
    (if (list? (car expr))
        (set-symbol! (caar expr)
                     (make-proc (cdar expr)
                                (cdr expr)
                                env) env)
        (set-symbol! (car expr) (lispy-eval (cadr expr) env) env))))

(scheme-syntax lambda
  (lambda (expr env)
    (make-proc (car expr)
               (cdr expr)
               env)))

(scheme-syntax if
  (lambda (expr env)
    (if (lispy-eval (car expr) env)
        (lispy-eval (cadr expr) env)
        (lispy-eval (caddr expr) env))))

(scheme-syntax quote
  (lambda (expr env)
    (car expr)))

(scheme-syntax set!
  (lambda (expr env)
    (set-symbol! (car expr) (lispy-eval (cadr expr) env) env)))

(scheme-syntax begin
  (lambda (expr env)
    (eval-body expr env)))

(scheme-syntax let
  (lambda (expr env)
    (eval-body (cdr expr) (extend-environment (car expr) env))))

(scheme-syntax equal?
  (lambda (expr env)
    (equal? (lispy-eval (car expr) env)
            (lispy-eval (cadr expr) env))))

(define-primitive + +)
(define-primitive - -)
(define-primitive < <)
(define-primitive > >)
(define-primitive car car)
(define-primitive cdr cdr)
(define-primitive cons cons)
(define-primitive print print)

GOTO Table of Contents

GOTO Table of Contents

To begin implementing Lispy procedures we have to define a procedure type, just like we did with primitives. A procedure has 3 basic parts. A list of parameters, a body and an environment. So we’ll set up our new procedure type with these fields and call it proc to avoid a name clash with Scheme’s procedure?.

Like primitives, procedures are handled by the lispy-apply procedure. We’ll change lispy-apply to use a cond instead of nested ifs and temporarily put in a placeholder for procedures. In addition we need a way to define new procedures from within Lispy. If we were writing a Scheme, the define form would allow us to both set symbols and procedures. For the moment we’ll separate the two usual jobs of define and create a new form (function (parameters) body) to set procedures.

(use srfi-69)

(define global-syntax-definitions (make-hash-table))
(define-record primitive function)
(define-record proc parameters body environment)

(define (current-environment env) (car env))
(define (enclosing-environment env) (cdr env))

(define (extend-environment bindings base-environment)
  (cons (alist->hash-table bindings) base-environment))

(define the-global-environment (extend-environment '() '()))

(define (set-symbol! symbol value env)
  (hash-table-set! (current-environment env) symbol value))

(define (lookup-symbol-value symbol environment)
  (if (null? environment)
    (error 'unbound-symbol "Unbound symbol:  " symbol)
    (if (hash-table-exists? (current-environment environment) symbol)
        (hash-table-ref (current-environment environment) symbol)
        (lookup-symbol-value symbol (enclosing-environment environment)))))

(define (self-evaluating? expr)
  (or (number? expr) (string? expr) (char? expr) (boolean? expr)))

(define (lispy-eval expr env)
  (cond ((self-evaluating? expr) expr)
        ((symbol? expr) (lookup-symbol-value expr env))
        (else
          (if (hash-table-exists? global-syntax-definitions (car expr))
              ((hash-table-ref global-syntax-definitions (car expr)) (cdr expr) env)
              (lispy-apply (lispy-eval (car expr) env) (eval-arguments (cdr expr) env))))))

(define (eval-arguments args env)
  (map (lambda (x) (lispy-eval x env)) args))

(define (lispy-apply procedure arguments)
  (cond ((primitive? procedure)
           (apply (primitive-function procedure) arguments))
        ((proc? procedure)
           "Attempted to apply a Lispy procedure")
        (else
           "Error: Undefined procedure")))

(hash-table-set! global-syntax-definitions 'scheme-syntax
  (lambda (expr env)
    (hash-table-set! global-syntax-definitions (car expr) (eval (cadr expr)))))

(hash-table-set! global-syntax-definitions 'load
  (lambda (expr env)
    (define f (open-input-file (car expr)))
    (let loop ((e (read f)))
      (if (equal? e #!eof) "Successfully Loaded!"
                           (begin
                             (lispy-eval e env)
                             (loop (read f)))))))

((hash-table-ref global-syntax-definitions 'load) '("syntax.chicken") the-global-environment)

(define (repl)
  (define input (read))
  (print ";===> " (lispy-eval input the-global-environment))
  (repl))

syntax.chicken

(scheme-syntax define
  (lambda (expr env)
    (set-symbol! (car expr) (lispy-eval (cadr expr) env) env)))

(scheme-syntax if
  (lambda (expr env)
    (if (lispy-eval (car expr) env)
        (lispy-eval (cadr expr) env)
        (lispy-eval (caddr expr) env))))

(scheme-syntax define-primitive
  (lambda (expr env)
    (set-symbol! (car expr)
                 (make-primitive (eval (cadr expr))))))

(scheme-syntax function
  (lambda (expr env)
    (set-symbol! (caar expr)
                 (make-procedure (cdar expr)
                                 (cdr expr)
                                 env) env)))

(function (a x) x)
;===> <unspecified>
a
;===> #<proc>
(a 42)
;===> Attempted to apply a Lispy procedure

Now that we have a basic outline for what our procedures will look like, we can focus on making them work!

To apply a procedure we have to evaluate the body. In the example above the body of the procedure is just x. However we can’t just evaluate x because x is not bound to anything yet.

First we need to create a new environment and bind the parameters to the arguments supplied in the procedure call. In this case we have to bind x to the value 42. For this we’ll use a helper procedure called assign-values.

The body is evaluated in the same way that we evaluate arguments. The difference is that we only return the last expression of the body. For now we’ll create a procedure named eval-body that will call eval-arguments, then return the last evaluated argument (it’s not the most efficient implementation, but it is simple and reuses code that we have already wrote).

(use srfi-69)
(use srfi-1)

(define global-syntax-definitions (make-hash-table))
(define-record primitive function)
(define-record proc parameters body environment)

(define (current-environment env) (car env))
(define (enclosing-environment env) (cdr env))

(define (extend-environment bindings base-environment)
  (cons (alist->hash-table bindings) base-environment))

(define the-global-environment (extend-environment '() '()))

(define (set-symbol! symbol value env)
  (hash-table-set! (current-environment env) symbol value))

(define (lookup-symbol-value symbol environment)
  (if (null? environment)
    "Error: Unbound symbol";(error 'unbound-symbol "Unbound symbol:  " symbol)
    (if (hash-table-exists? (current-environment environment) symbol)
        (hash-table-ref (current-environment environment) symbol)
        (lookup-symbol-value symbol (enclosing-environment environment)))))

(define (self-evaluating? expr)
  (or (number? expr) (string? expr) (char? expr) (boolean? expr)))

(define (lispy-eval expr env)
  (cond ((self-evaluating? expr) expr)
        ((symbol? expr) (lookup-symbol-value expr env))
        (else
          (if (hash-table-exists? global-syntax-definitions (car expr))
              ((hash-table-ref global-syntax-definitions (car expr)) (cdr expr) env)
              (lispy-apply (lispy-eval (car expr) env) (eval-arguments (cdr expr) env))))))

(define (eval-arguments args env)
  (map (lambda (x) (lispy-eval x env)) args))

(define (eval-body args env)
  (last (eval-arguments args env)))

(define (assign-values procedure args)
  (map cons (proc-parameters procedure) args))

(define (lispy-apply procedure arguments)
  (cond ((primitive? procedure)
           (apply (primitive-function procedure) arguments))
        ((proc? procedure)
           (eval-body (proc-body procedure)
                      (extend-environment (assign-values procedure arguments)
                                          (proc-environment procedure))))
        (else
           "Error: Undefined procedure")))

(hash-table-set! global-syntax-definitions 'scheme-syntax
  (lambda (expr env)
    (hash-table-set! global-syntax-definitions (car expr) (eval (cadr expr)))))

(hash-table-set! global-syntax-definitions 'load
  (lambda (expr env)
    (define f (open-input-file (car expr)))
    (let loop ((e (read f)))
      (if (equal? e #!eof) "Successfully Loaded!"
                           (begin
                             (lispy-eval e env)
                             (loop (read f)))))))

((hash-table-ref global-syntax-definitions 'load) '("syntax.chicken") the-global-environment)

(define (repl)
  (define input (read))
  (print ";===> " (lispy-eval input the-global-environment))
  (repl))

(function (test pred conseq alt)
  (if pred conseq alt))
;===> #<unspecified>
(test 1 2 3)
;===> 2

With that, we have implemented procedures in Lispy! In 58 lines we have defined an interpreter framework that we can use to write just about any parenthesized, applicative-order, lexically scoped language. In the next few parts we’ll implement a very basic Scheme, then McCarthy’s LISP and finally begin work on the implementation of Lispy.

Also note that there is very little error-checking going on here. For instance calling a procedure with the wrong number of arguments results in the whole interpreter crashing. This would not be good in a production language. However including error checking in this code would probably triple its size at least. The point of this exercise is to learn the concepts, not write a bullet-proof language.

GOTO Part 8 | Scheme in Lispy

GOTO Table of Contents

(define x 5)
(define (func x)
  x)
(func 100)
;===> 100

With our frame this wouldn’t work as expected. Because we have already defined x in the global environment, there will be one of two problems with the x in func. Our function could use the global value of x (5) instead of the supplied value (100) or when the supplied value (100) is mapped onto the function body it will overwrite the value of the global value of x (5) with the supplied value (100).

Currently we do not make any distinction between variables in a function body and in the global environment. To properly implement procedures we will have to figure out a way to do that. Consider the next bit of code.

(define x 5)
(define (func y)
  (+ x y))
(func 10)
;===> 15

Even though we need to make a distinction between environments they also need to be connected. In the example above we use the global variable x in func without ever declaring it. This is accomplished by a method called chaining environments.

(define x 1)
(define (outer-func y)
  (define (inner-func z)
    (+ x y z)
  (inner-func 3))
(outer-func 2)
;===> 6

The environments produced by the above code are best described by the diagram below.

So when x appears in inner-func we have to check inner-func’s environment for x, then outer-func’s environment for x and finally the global environment for x before we raise an undefined symbol error.

You may have noticed that the diagram above looks a lot like a linked list. We can take our frame and combine it with linked lists to form an environment. It is more informative (and closer to the actual design) to view environments from the point of view of eval. This diagram is a view of the environment structure when the code (+ x y z) is evaluated.

I used the names current-environment to refer to the inner-func environment and enclosing-environment to refer to the outer-func environment. We will use these same names in our source code. Note that the global-environment is just the enclosing-environment of outer-func, the only thing special about it is that its enclosing-environment is the empty list.

In addition we need to replace our global frame with the-global-environment, which we’ll do by providing a procedure extend-environment, and change lookup-symbol-value to check enclosing-environments if the symbol is not found.

(use srfi-69)

(define global-syntax-definitions (make-hash-table))
(define-record primitive function)

(define (current-environment env) (car env))
(define (enclosing-environment env) (cdr env))

(define (extend-environment bindings base-environment)
  (cons (alist->hash-table bindings) base-environment))

(define the-global-environment (extend-environment '() '()))

(define (set-symbol! symbol value)
  (hash-table-set! (current-environment the-global-environment) symbol value))

(define (lookup-symbol-value symbol environment)
  (if (null? environment)
    (error 'unbound-symbol "Unbound symbol:  " symbol)
    (if (hash-table-exists? (current-environment environment) symbol)
        (hash-table-ref (current-environment environment) symbol)
        (lookup-symbol-value symbol (enclosing-environment environment)))))

(define (self-evaluating? expr)
  (or (number? expr) (string? expr) (char? expr) (boolean? expr)))

(define (lispy-eval expr)
  (cond ((self-evaluating? expr) expr)
        ((symbol? expr) (lookup-symbol-value expr the-global-environment))
        (else
          (if (hash-table-exists? global-syntax-definitions (car expr))
              ((hash-table-ref global-syntax-definitions (car expr)) (cdr expr))
              (lispy-apply (lispy-eval (car expr)) (eval-arguments (cdr expr)))))))

(define (eval-arguments args)
  (map (lambda (x) (lispy-eval x)) args))

(define (lispy-apply procedure arguments)
  (if (primitive? procedure)
    (apply (primitive-function procedure) arguments)
    "Error: Undefined procedure")) 

(hash-table-set! global-syntax-definitions 'scheme-syntax
  (lambda (expr)
    (hash-table-set! global-syntax-definitions (car expr) (eval (cadr expr)))))

(hash-table-set! global-syntax-definitions 'load
  (lambda (expr)
    (define f (open-input-file (car expr)))
    (let loop ((e (read f)))
      (if (equal? e #!eof) "Successfully Loaded!"
                           (begin
                             (lispy-eval e)
                             (loop (read f)))))))

((hash-table-ref global-syntax-definitions 'load) '("syntax.chicken"))

(define (repl)
  (define input (read))
  (print ";===> " (lispy-eval input))
  (repl))

(define a 5)
;===> #<unspecified>
a
;===> 5

To use the full potential of our new environments we are going to have to rewrite our program a little bit to pass them around correctly. Since eval is the main method of dispatch for our interpreter we’ll supply our environment as an argument to eval. We’ll also have to pass an environment variable to procedures like set-symbol! and lookup-symbol-value. One final place we’ll need environments is in scheme-syntax, we’ll have to change the stored procedures there to use (lambda (expr env) …) instead of just (lambda (expr) …).

(use srfi-69)

(define global-syntax-definitions (make-hash-table))
(define-record primitive function)

(define (current-environment env) (car env))
(define (enclosing-environment env) (cdr env))

(define (extend-environment bindings base-environment)
  (cons (alist->hash-table bindings) base-environment))

(define the-global-environment (extend-environment '() '()))

(define (set-symbol! symbol value env)
  (hash-table-set! (current-environment env) symbol value))

(define (lookup-symbol-value symbol environment)
  (if (null? environment)
    (error 'unbound-symbol "Unbound symbol:  " symbol)
    (if (hash-table-exists? (current-environment environment) symbol)
        (hash-table-ref (current-environment environment) symbol)
        (lookup-symbol-value symbol (enclosing-environment environment)))))

(define (self-evaluating? expr)
  (or (number? expr) (string? expr) (char? expr) (boolean? expr)))

(define (lispy-eval expr env)
  (cond ((self-evaluating? expr) expr)
        ((symbol? expr) (lookup-symbol-value expr env))
        (else
          (if (hash-table-exists? global-syntax-definitions (car expr))
              ((hash-table-ref global-syntax-definitions (car expr)) (cdr expr) env)
              (lispy-apply (lispy-eval (car expr)) (eval-arguments (cdr expr)))))))

(define (eval-arguments args)
  (map (lambda (x) (lispy-eval x)) args))

(define (lispy-apply procedure arguments)
  (if (primitive? procedure)
    (apply (primitive-function procedure) arguments)
    "Error: Undefined procedure")) 

(hash-table-set! global-syntax-definitions 'scheme-syntax
  (lambda (expr env)
    (hash-table-set! global-syntax-definitions (car expr) (eval (cadr expr)))))

(hash-table-set! global-syntax-definitions 'load
  (lambda (expr env)
    (define f (open-input-file (car expr)))
    (let loop ((e (read f)))
      (if (equal? e #!eof) "Successfully Loaded!"
                           (begin
                             (lispy-eval e env)
                             (loop (read f)))))))

((hash-table-ref global-syntax-definitions 'load) '("syntax.chicken") the-global-environment)

(define (repl)
  (define input (read))
  (print ";===> " (lispy-eval input the-global-environment))
  (repl))

syntax.chicken

(scheme-syntax define
  (lambda (expr env)
    (set-symbol! (car expr) (lispy-eval (cadr expr) env) env)))

(scheme-syntax if
  (lambda (expr env)
    (if (lispy-eval (car expr) env)
        (lispy-eval (cadr expr) env)
        (lispy-eval (caddr expr) env))))

(scheme-syntax define-primitive
  (lambda (expr env)
    (set-symbol! (car expr)
                 (make-primitive (eval (cadr expr))))))

(define a 5)
;===> #<unspecified>
a
;===> 5

If you still don’t fully understand environments, that’s OK. You’ll get a chance to play around with them in the next part where we’ll finally cover implementing procedures in Lispy!

GOTO Part 7 | Lispy Procedures

GOTO Table of Contents

The first thing we need to do is figure out a way to represent our primitive procedures. SICP uses tagged lists, which are just regular lists with the first element designated as a tag. We’re going to use define-record to define a primitive type, which is more flexible and a little easier to use.

If the symbol of an expression of the form (symbol args …) is not found in global-syntax-definitions it must be a primitive procedure application (or not exist). To apply a primitive procedure we evaluate all the arguments using lispy-eval. We use that list as the args value to (apply primitive args). The primitive returns a Lispy value and we continue on.

For now we’re going to hardcode one primitive + into our interpreter. We’ll come up with something a little more elegant next.

(use srfi-69)

(define global-syntax-definitions (make-hash-table))
(define frame (make-hash-table))
(define-record primitive function)

(define (set-symbol! symbol value)
  (hash-table-set! frame symbol value))

(define (lookup-variable-value symbol)
  (if (hash-table-exists? frame symbol)
      (hash-table-ref frame symbol)
      "Error: Unbound variable"))

(define (self-evaluating? expr)
  (or (number? expr) (string? expr) (char? expr) (boolean? expr)))

(define (lispy-eval expr)
  (cond ((self-evaluating? expr) expr)
        ((symbol? expr) (lookup-variable-value expr))
        (else
          (if (hash-table-exists? global-syntax-definitions (car expr))
              ((hash-table-ref global-syntax-definitions (car expr)) (cdr expr))
              (lispy-apply (lispy-eval (car expr)) (eval-arguments (cdr expr)))))))

(define (eval-arguments args)
  (map (lambda (x) (lispy-eval x)) args))

(define (lispy-apply procedure arguments)
  (if (primitive? procedure)
    (apply (primitive-function procedure) arguments)
    "Error: Undefined procedure"))

(set-symbol! '+ (make-primitive +))

(hash-table-set! global-syntax-definitions 'scheme-syntax
  (lambda (expr)
    (hash-table-set! global-syntax-definitions (car expr) (eval (cadr expr)))))

(hash-table-set! global-syntax-definitions 'load
  (lambda (expr)
    (define f (open-input-file (car expr)))
    (let loop ((e (read f)))
      (if (equal? e #!eof) "Successfully Loaded!"
                           (begin
                             (lispy-eval e)
                             (loop (read f)))))))

((hash-table-ref global-syntax-definitions 'load) '("syntax.chicken"))

(define (repl)
  (define input (read))
  (print ";>>> " (lispy-eval input))
  (repl))

(repl)

(+ 3 4)
;===> 7
(- 3 4)
;===> Error: Undefined procedure

The next step is to open up that functionality to Lispy. We’ll implement define-primitive as a scheme-syntax macro. First remove the line that sets the + symbol. We’re only going to be modifying the syntax.chicken file for now so that’s all I’ll show you.

We need to create a scheme-syntax macro that makes a procedure type and sets it as the value of a symbol. This macro is a lot like the define macro that we wrote earlier. The only difference is instead of calling lispy-eval on the cadr we call eval and pass that result to make-primitive.

syntax.chicken

(scheme-syntax define
  (lambda (expr)
    (set-symbol! (car expr) (lispy-eval (cadr expr)))))

(scheme-syntax if
  (lambda (expr)
    (if (lispy-eval (car expr))
        (lispy-eval (cadr expr))
        (lispy-eval (caddr expr)))))

(scheme-syntax define-primitive
  (lambda (expr)
    (set-symbol! (car expr)
                 (make-primitive (eval (cadr expr))))))

With that we can define all sorts of primitives for our language…

(repl)
(define-primitive + +)
;===> #<unspecified>
(+ 3 4)
;===> 7
(- 3 4)
;===> Error: Undefined procedure

(define-primitive - -)
;===> #<unspecified>
(- 3 4)
;===> -1

(define-primitive square (lambda (x) (* x x)))
;===> #<unspecified>
(square 4)
;===> 16

You can start a new file and define a bunch of primitives, load it the same way you load syntax.chicken. I’m not going to define any primitives just yet but you are more than welcome to. Instead in the next post we’ll implement environments and then it’s on to implementing lambda!

GOTO Part 6 – Environments

GOTO Table of Contents

(use srfi-69)

(define global-syntax-definitions (make-hash-table))
(define frame (make-hash-table))

(define (set-symbol! symbol value)
  (hash-table-set! frame symbol value))

(define (lookup-variable-value symbol)
  (if (hash-table-exists? frame symbol)
      (hash-table-ref frame symbol)
      "Error: Unbound variable"))

(define (self-evaluating? expr)
  (or (number? expr) (string? expr) (char? expr) (boolean? expr)))

(define (lispy-eval expr)
  (cond ((self-evaluating? expr) expr)
        ((symbol? expr) (lookup-variable-value expr))
        (else
          (if (hash-table-exists? global-syntax-definitions (car expr))
              ((hash-table-ref global-syntax-definitions (car expr)) (cdr expr))
              "Not implemented"))))

(hash-table-set! global-syntax-definitions 'scheme-syntax
  (lambda (expr)
    (hash-table-set! global-syntax-definitions (car expr) (eval (cadr expr)))))

(define (repl)
  (define input (read))
  (print ";===> " (lispy-eval input))
  (repl))

Let’s take a moment to consider what these 25 lines of code can do. We now have a language that can understand self-evaluating Scheme values and lookup symbol values. In addition to that it has exactly one primitive, a macro that defines other primitives and gives us access to the underlying Scheme.

You may consider this language lacking, and I admit it does not have many features. However it is now possible to define any language construct (including procedures, eval and apply) from within Lispy with the help of scheme-syntax and the Scheme underneath. After writing these 25 lines we could define the rest of the language 100% from within Lispy.

However there are a few more things that conceptually belong in the core language. One of those things is the ability to load a Lispy file. This will let us break out our syntax definitions into another file and load it automatically before we start the repl.

(use srfi-69)

(define global-syntax-definitions (make-hash-table))
(define frame (make-hash-table))

(define (set-symbol! symbol value)
  (hash-table-set! frame symbol value))

(define (lookup-variable-value symbol)
  (if (hash-table-exists? frame symbol)
      (hash-table-ref frame symbol)
      "Error: Unbound variable"))

(define (self-evaluating? expr)
  (or (number? expr) (string? expr) (char? expr) (boolean? expr)))

(define (lispy-eval expr)
  (cond ((self-evaluating? expr) expr)
        ((symbol? expr) (lookup-variable-value expr))
        (else
          (if (hash-table-exists? global-syntax-definitions (car expr))
              ((hash-table-ref global-syntax-definitions (car expr)) (cdr expr))
              "Not implemented"))))

(hash-table-set! global-syntax-definitions 'scheme-syntax
  (lambda (expr)
    (hash-table-set! global-syntax-definitions (car expr) (eval (cadr expr)))))

(hash-table-set! global-syntax-definitions 'load
  (lambda (expr)
    (define f (open-input-file (car expr)))
    (let loop ((e (read f)))
      (if (equal? e #!eof) "Successfully Loaded!"
                           (begin
                             (lispy-eval e)
                             (loop (read f)))))))

(define (repl)
  (define input (read))
  (print ";===> " (lispy-eval input))
  (repl))

Now that we have load, we need something to load. Create a new file called syntax.chicken, or whatever you prefer. This is where we will define our primitive syntax forms like define, if and lambda.

(use srfi-69)

(define global-syntax-definitions (make-hash-table))
(define frame (make-hash-table))

(define (set-symbol! symbol value)
  (hash-table-set! frame symbol value))

(define (lookup-variable-value symbol)
  (if (hash-table-exists? frame symbol)
      (hash-table-ref frame symbol)
      "Error: Unbound variable"))

(define (self-evaluating? expr)
  (or (number? expr) (string? expr) (char? expr) (boolean? expr)))

(define (lispy-eval expr)
  (cond ((self-evaluating? expr) expr)
        ((symbol? expr) (lookup-variable-value expr))
        (else
          (if (hash-table-exists? global-syntax-definitions (car expr))
              ((hash-table-ref global-syntax-definitions (car expr)) (cdr expr))
              "Not implemented"))))

(hash-table-set! global-syntax-definitions 'scheme-syntax
  (lambda (expr)
    (hash-table-set! global-syntax-definitions (car expr) (eval (cadr expr)))))

(hash-table-set! global-syntax-definitions 'load
  (lambda (expr)
    (define f (open-input-file (car expr)))
    (let loop ((e (read f)))
      (if (equal? e #!eof) "Successfully Loaded!"
                           (begin
                             (lispy-eval e)
                             (loop (read f)))))))

((hash-table-ref global-syntax-definitions 'load) '("syntax.chicken"))

(define (repl)
  (define input (read))
  (print ";===> " (lispy-eval input))
  (repl))

syntax.chicken

(scheme-syntax define
  (lambda (expr)
    (set-symbol! (car expr) (lispy-eval (cadr expr)))))

(scheme-syntax if
  (lambda (expr)
    (if (lispy-eval (car expr))
        (lispy-eval (cadr expr))
        (lispy-eval (caddr expr)))))

(repl)
(define a 5)
;===> #<unspecified>
a
;===> 5
(if 1 2 3)
;===> 2

Now that we have implemented load we are one step closer to having a platform that we can build any language we want with. To change languages all we would have to do is load a different syntax file.

I encourage you to play around with that concept for a little while. Write two different syntax.chicken files and implement a couple of basic forms like define, or and if. Try to make your two implementations as different as possible even if making it different doesn’t make sense. For instance in one implementation if could take a mandatory keyword before the alternate branch (if pred consequent else alt) or it could act like and, taking two predicates and only evaluating the consequent branch if both predicates evaluate to true.

Since one of the goals of Lispy is to experiment with language features this ability will come in very useful. In the future we will extend this concept to make our interpreter more of a language platform than a language itself.

This article is a little short on code because we made some larger conceptual changes that I did not want to breeze past. In the next article we’ll implement primitive procedures and apply. The ability to define procedures from within Lispy is just around the corner.

GOTO Part 5 | Primitive Procedures and apply

GOTO Table of Contents

We’re going to start by implementing if the same way we implemented define. The first thing to do is test if the car of the pair equals if, then call apply-if.

We know that if will have to evaluate the predicate argument to determine whether to evaluate the consequent or alternate arguments. If the predicate is true we will evaluate the consequent expression and not the alternate. If the predicate is false we will evaluate the alternate expression and not the consequent.

(use srfi-69)

(define frame (make-hash-table))

(define (set-symbol! symbol value)
  (hash-table-set! frame symbol value))

(set-symbol! 'test "Value retrieved successfully")

(define (lookup-variable-value symbol)
  (if (hash-table-exists? frame symbol)
      (hash-table-ref frame symbol)
      "Error: Unbound variable"))

(define (self-evaluating? expr)
  (or (number? expr) (string? expr) (char? expr) (boolean? expr)))

(define (lispy-eval expr)
  (cond ((self-evaluating? expr) expr)
        ((symbol? expr) (lookup-variable-value expr))
        ((pair? expr)
          (if (equal? (car expr) 'define)
              (set-symbol! (cadr expr) (lispy-eval (caddr expr)))
              (if (equal? (car expr) 'if)
                  (apply-if (lispy-eval (cadr expr)) (caddr expr) (cadddr expr))
                  "Not implemented")))
        (else "Not implemented")))

(define (apply-if pred consequent alternate)
  (if pred
      (lispy-eval consequent)
      (lispy-eval alternate)))

(define (repl)
  (define input (read))
  (print ";===>" (lispy-eval input))
  (repl))

(repl)
(if 1 2 3)
;===> 2

Anyone who has ever used Scheme or Lisp is probably cringing at the use of nested ifs here, when I could have used a cond (like SICP). I did this to illustrate a point, using if or cond is fine while you have a small number of primitive syntax forms. However once you implement a large number of primitive syntax forms this method of dispatch becomes cumbersome. Also note that it is not possible to extend the primitive syntax forms except by modifying the original source by hand.

Reusing the Frame

On top of that we have already implemented a mechanism of associating and storing symbol and value pairs. Let’s try something like that first, we’ll start by making another global hash table called global-syntax-definitions. Then we’ll modify eval to check these definitions.

(use srfi-69)

(define global-syntax-definitions (make-hash-table))
(define frame (make-hash-table))

(define (set-symbol! symbol value)
  (hash-table-set! frame symbol value))

(set-symbol! 'test "Value retrieved successfully")

(define (lookup-variable-value symbol)
  (if (hash-table-exists? frame symbol)
      (hash-table-ref frame symbol)
      "Error: Unbound variable"))

(define (self-evaluating? expr)
  (or (number? expr) (string? expr) (char? expr) (boolean? expr)))

(define (lispy-eval expr)
  (cond ((self-evaluating? expr) expr)
        ((symbol? expr) (lookup-variable-value expr))
        (else
          (if (hash-table-exists? global-syntax-definitions (car expr))
              ((hash-table-ref global-syntax-definitions (car expr)) (cdr expr))
              "Not implemented"))))

(hash-table-set! global-syntax-definitions 'define
  (lambda (expr)
    (set-symbol! (car expr) (lispy-eval (cadr expr)))))

(hash-table-set! global-syntax-definitions 'if
  (lambda (expr)
    (if (lispy-eval (car expr))
        (lispy-eval (cadr expr))
        (lispy-eval (caddr expr)))))

(define (repl)
  (define input (read))
  (print ";===>" (lispy-eval input))
  (repl))

(repl)
(define a 5)
;===> #<unspecified>
a
;===> 5
(if 1 2 3)
;===> 2
(hello)
;===> Not implemented

With that addition we are finished with the first level of the eval function. An expression can only be one of three things: self-evaluating, a symbol or an application (either syntax or procedure which we’ll cover later).

How does it work? global-syntax-definitions is used to store anonymous procedures that each take a single argument (the rest of the expression after the name). If a given symbol matches a key in global-syntax-definitions the value of that key is called with the unevaluated list of arguments as the only argument.

In other words, when lispy-eval comes across a primitive syntax definition it transfers control to a Scheme procedure located in the global-syntax-definitions table. That Scheme procedure is responsible for evaluating the arguments and returning a Lispy value.

Now that we have a data structure to hold our global-syntax-definitions it would be a good idea to supply an easy mechanism to set a new syntax definition. For this purpose we’ll design a macro in Lispy that will set a new syntax definition called scheme-syntax.

(use srfi-69)

(define global-syntax-definitions (make-hash-table))
(define frame (make-hash-table))

(define (set-symbol! symbol value)
  (hash-table-set! frame symbol value))

(set-symbol! 'test "Value retrieved successfully")

(define (lookup-variable-value symbol)
  (if (hash-table-exists? frame symbol)
      (hash-table-ref frame symbol)
      "Error: Unbound variable"))

(define (self-evaluating? expr)
  (or (number? expr) (string? expr) (char? expr) (boolean? expr)))

(define (lispy-eval expr)
  (cond ((self-evaluating? expr) expr)
        ((symbol? expr) (lookup-variable-value expr))
        (else
          (if (hash-table-exists? global-syntax-definitions (car expr))
              ((hash-table-ref global-syntax-definitions (car expr)) (cdr expr))
              "Not implemented"))))

(hash-table-set! global-syntax-definitions 'scheme-syntax
  (lambda (expr)
    (hash-table-set! global-syntax-definitions (car expr) (eval (cadr expr)))))

(hash-table-set! global-syntax-definitions 'define
  (lambda (expr)
    (set-symbol! (car expr) (lispy-eval (cadr expr)))))

(hash-table-set! global-syntax-definitions 'if
  (lambda (expr)
    (if (lispy-eval (car expr))
        (lispy-eval (cadr expr))
        (lispy-eval (caddr expr)))))

(define (repl)
  (define input (read))
  (print ";===>" (lispy-eval input))
  (repl))

(repl)
(scheme-syntax quote (lambda (expr) (car expr)))
;===> #<unspecified>
(quote (1 2 3))
;===> (1 2 3)
(1 2 3)
;===> Not implemented

With the addition of scheme-syntax it is now possible to write Scheme directly in Lispy. Because of this we can now implement all of the primitive syntax forms in Lispy. Some might consider this cheating, but eventually we’re going to want a way to interface with Chicken. Why not implement it right away, use it, test it and make it solid by implementing everything else in it? We’ll still have to make some modifications to the core interpreter to implement procedures and a load form later.

We haven’t even gotten to the mutually recursive yin-yang that is the eval/apply cycle yet. However, thanks to piggy-backing on Chicken, we already have a somewhat useful language. A language without procedures is a pretty bad one, so we’ll keep pressing on, but check out some of the things that are possible already…

(repl)
(scheme-syntax print
  (lambda (expr)
    (print (car expr))))
;===> #<unspecified>
(scheme-syntax loop
  (lambda (expr)
    (let loop ((n (car expr)))
      (begin
        (lispy-eval (cadr expr))
        (if (> n 0)
            (loop (- n 1)))))))
;===> #<unspecified>
(loop 2 (print "Hi"))
;===> Hi
;===> Hi
;===> Hi
;===> #<unspecified>

In the next article we’ll do a little refactoring, remove define and if from the core of the interpreter and implement a load function so we can remove our syntax definitions out of the main interpreter file.

GOTO Part 4 | Refactor and load

GOTO Table of Contents

Starting with a Stub

To start with we need to modify our eval procedure to recognize symbols. We’ll call a procedure lookup-variable-value that will eventually give the value of the symbol. Right now we’ll just have it return a string stating our intentions.

(define (lookup-variable-value expr)
  "Here's where we lookup the value of the symbol.")

(define (self-evaluating? expr)
  (or (number? expr) (string? expr) (char? expr) (boolean? expr)))

(define (lispy-eval expr)
  (cond ((self-evaluating? expr) expr)
        ((symbol? expr) (lookup-variable-value expr))
        (else "Not implemented")))

(define (repl)
  (define input (read))
  (print ";===>" (lispy-eval input))
  (repl))

(repl)
"hello"
;===> hello
hi
;===> Here's where we lookup the value of the symbol.

The Global Frame

To set the value of a symbol we need to have somewhere to set it. That somewhere is going to be a hash table. The hash table that stores a set of names and their associated values is referred to as a frame.

For now we’ll define our frame as a global variable and also define a procedure set-variable! to set the value of a symbol. Since we don’t have any method of assigning values to symbols yet, we define a symbol “test” when we make the frame so we can see if it works.

(use srfi-69)

(define frame (make-hash-table))

(define (set-symbol! symbol value)
  (hash-table-set! frame symbol value))

(set-symbol! 'test "Value retrieved successfully")

(define (lookup-variable-value symbol)
  (if (hash-table-exists? frame symbol)
      (hash-table-ref frame symbol)
      "Error: Unbound variable"))

(define (self-evaluating? expr)
  (or (number? expr) (string? expr) (char? expr) (boolean? expr)))

(define (lispy-eval expr)
  (cond ((self-evaluating? expr) expr)
        ((symbol? expr) (lookup-variable-value expr))
        (else "Not implemented")))

(define (repl)
  (define input (read))
  (print ";===>" (lispy-eval input))
  (repl))

(repl)
test
;===> Value retrieved successfully
hello
;===> Error: Unbound variable

(define symbol value)

Now that we can store and retrieve the value of a symbol, we need a way to do it from within Lispy. Lispy uses the form (define symbol value) to define a new symbol or set the value of an existing symbol.

The form (define symbol value) is a pair (also a list, but all lists are also pairs) so the first step to detecting its usage is to look for pairs. The next step is to check the symbol in the car of that pair (or the first item of the list), if that symbol is “define” we run the code to define a symbol. For now we’ll just return a string stating our intentions.

(use srfi-69)

(define frame (make-hash-table))

(define (set-symbol! symbol value)
  (hash-table-set! frame symbol value))

(set-symbol! 'test "Value retrieved successfully")

(define (lookup-variable-value symbol)
  (if (hash-table-exists? frame symbol)
      (hash-table-ref frame symbol)
      "Error: Unbound variable"))

(define (self-evaluating? expr)
  (or (number? expr) (string? expr) (char? expr) (boolean? expr)))

(define (lispy-eval expr)
  (cond ((self-evaluating? expr) expr)
        ((symbol? expr) (lookup-variable-value expr))
        ((pair? expr)
          (if (equal? (car expr) 'define)
              "Set the value of the symbol."
              "Not implemented"))
        (else "Not implemented")))

(define (repl)
  (define input (read))
  (print ";===>" (lispy-eval input))
  (repl))

(repl)
test
;===> Value retrieved successfully
(define a 1)
;===> Set the value of the symbol.
(set a 1)
;===> Not implemented

Setting the Value of Symbols

To actually set the value of the symbol we need to get the arguments to define. Getting the symbol argument is straightforward, however we need to eval the value argument as it might not be self-evaluating.

(use srfi-69)

(define frame (make-hash-table))

(define (set-symbol! symbol value)
  (hash-table-set! frame symbol value))

(set-symbol! 'test "Value retrieved successfully")

(define (lookup-variable-value symbol)
  (if (hash-table-exists? frame symbol)
      (hash-table-ref frame symbol)
      "Error: Unbound variable"))

(define (self-evaluating? expr)
  (or (number? expr) (string? expr) (char? expr) (boolean? expr)))

(define (lispy-eval expr)
  (cond ((self-evaluating? expr) expr)
        ((symbol? expr) (lookup-variable-value expr))
        ((pair? expr)
          (if (equal? (car expr) 'define)
              (set-symbol! (cadr expr) (lispy-eval (caddr expr)))
              "Not implemented"))
        (else "Not implemented")))

(define (repl)
  (define input (read))
  (print ";===>" (lispy-eval input))
  (repl))

(repl)
(define a 5)
;===> #<unspecified>
a
;===> 5

In addition to being able to set symbols, we now have a glimpse into how basic forms (like define) are handled. It really is no more complicated than checking the procedure name, evaluating the arguments that need to be, then passing the arguments to a Scheme procedure.

* To any Schemers that may be reading this: I avoided implementing environments here for two reasons. I wanted to show the simplest implementation possible and in an upcoming post we will transform our global frame into a lookup table for primitive syntax.

GOTO Part 3 | scheme-syntax macro

GOTO Table of Contents

To force myself to come up with my own implementations, instead of just blindly reading through the series, I decided to write a version of Scheme that has some Python influences. I wrote generic functions and list/vector/string comprehensions and I hooked up the Hans Boehm garbage collector to stop it from crashing all the time.

Writing Lispy in C was a fun, challenging and rewarding experience. However as I began to answer my questions about how basic language features could be implemented, I began to have new questions. I started wondering about sequence abstractions, parallel execution, optimization and code generation. As my questions got more in depth I began to realize that C was not the easiest language for me to solve them in.

That leads us to Chicken. Since Lispy is a language that is based on Scheme, why not write it in Scheme? The advantages are obvious, we get a full blown reader (that we now only have to modify), compilation to C and tons of other features like macros. Most of Lispy is simply syntax or functional extensions of Scheme and map directly into Scheme code.

This series will build Lispy incrementally. Each stage of this process will generate a fully working, if not featureful, language. Each stage we implement will be the simplest code possible to implement a feature. Error handling or efficiency will not be a concern (though may be taken into account if absolutely necessary). The interpreter that we will build borrows a lot from the meta-circular evaluator in The Structure and Interpretation of Computer Programs.

You will need to install Chicken to follow along (this may work on other Schemes but that is not an objective for this series), if you’re running a recent version of Ubuntu you can get a fairly recent version of Chicken with apt-get or you can download it here. This series will build an interpreter that is similar to this meta-circular evaluator that I previously wrote.

A Simple Scheme Meta-Circular Evaluator

The first thing we’re going to do is implement a REPL, also called an interpreter. REPL stands for Read, Eval, Print Loop and that’s just what it does. First the repl reads in an expression from stdin, then it evaluates the expression and prints the return value to stdout. When it is finished with these tasks it loops and does it all again. To get something up and running so we can play with it, let’s make a simple Chicken repl.

Most repls will print an indicator before the input line (>>> in Python, #;1> in Chicken). Lispy will instead print an indicator before the output. This makes sense for now because it lets us copy and paste interpreter sessions. This will make our lives easier for now and is trivial to swap later.

(define (repl)
  (define input (read))
  (print ";===>" (eval input))
  (repl))

(repl)
42
;===> 42
"hi"
;===> hi
(+ 2 2)
;===> 4

Since Lispy is very close to Scheme in syntax we can continue to use the default (read) function for now. The thing that we’re most interested in at the moment is the eval function and we need to implement that in order to start implementing everything else.

Our first version of eval will be as simple as possible. We’ll simply echo the input back to the screen. Our simple reader already understands basic Scheme data types such as integers, strings and lists.

(define (lispy-eval expr)
  expr)

(define (repl)
  (define input (read))
  (print ";===>" (lispy-eval input))
  (repl))

(repl)
42
;===> 42
"hi"
;===> hi
(+ 2 2)
;===> (+ 2 2)

A Language of Numbers

The simplest language is a language of numbers. Numbers are self-evaluating and we’re going to use native Chicken data types wherever possible to simplify implementation, so if the expression is a number we’ll just return it for now.

(define (lispy-eval expr)
  (cond ((number? expr) expr)
        (else "Not implemented")))

(define (repl)
  (define input (read))
  (print ";===>" (lispy-eval input))
  (repl))

(repl)
42
;===> 42
"hi"
;===> Not implemented

A Language of Self-Evaluating Values

Continuing with our language of self-evaluating values is just as simple. Booleans, strings and characters are also self-evaluating so we’ll add them now. More complex data structures like hash tables are also self evaluating, however we’ll leave them for later.

To get something up and running quickly we’ll stick with the Scheme versions of these data types. Eventually we will modify the boolean and string data types and will probably remove the character type completely.

(define (self-evaluating? expr)
  (or (number? expr) (string? expr) (char? expr) (boolean? expr)))

(define (lispy-eval expr)
  (cond ((self-evaluating? expr) expr)
        (else "Not implemented")))

(define (repl)
  (define input (read))
  (print ";===>" (lispy-eval input))
  (repl))

(repl)
42
;===> 42
"hi"
;===> hi
#t
;===> #t

The first stage of our language is complete. We now have a functional language of self-evaluating values. Though one could argue exactly how functional it really is. The next step along our path will be to introduce symbols and implement assignment.

GOTO Part 2 | Assignment and define

GOTO Table of Contents

My general idea for Lispy was to have a system that behaved like Scheme, but looked and felt more like Python.  Most of the core functions in Lispy are generic and comprehensions play a large role in looping, though tail-recursion is still available in the general case.

However I’ve always wanted more from Lispy.  I really like the sequence abstractions and immutable data structures of Clojure.  I want to play around with lazy, concurrent and parallel evaluation.  However the idea of implementing all that in C is daunting to me.*

The solution is obvious.  Since Lispy is to be a Scheme-like language, and Scheme is a language that is designed for writing programming languages, the next version of Lispy will be written in Chicken Scheme.  It was a natural choice, and would have been my first, had I not done the original implementation for academic purposes.

The switch to Chicken should make it easier to scratch another itch of mine, compiling.  I can envision a number of paths to the compilation of Lispy, from macro compiling to Chicken all the way down to generating assembly.  I haven’t started laying out a plan for compilation yet, but my bet would be that Lispy will become a thin language layer on top of Chicken.

This new version will focus on extending the sequence/constructor concept to support lazy sequences, hash-tables and generic operations for all sequences.

There are three excerpts that I think succinctly describe the motivations behind Lispy.

Programming languages should be designed not by piling feature on top of feature, but by removing the weaknesses and restrictions that make additional features appear necessary.

- R4RS

Most data structures exist because of speed. For example, many languages today have both strings and lists. Semantically, strings are more or less a subset of lists in which the elements are characters. So why do you need a separate data type? You don’t, really. Strings only exist for efficiency. But it’s lame to clutter up the semantics of the language with hacks to make programs run faster. Having strings in a language seems to be a case of premature optimization.

- Paul Graham “The Hundred-Year Language”

Thus, programs must be written for people to read, and only incidentally for machines to execute.

- SICP

Lispy’s main goal is to make a language that is “shockingly inefficient” but also a joy to program in.  Lispy is at least a 10 year language.  Lispy is designed to be simple at the expense of efficiency.  Lispy is research, mainly research into the type of programming language that I want to use (as I haven’t quite figured that out yet).

Note that this post is to help encourage me to start developing this version.  I haven’t written any code yet.

* It may be that I’m just inexperienced with the language, but I prefer coding in Scheme.