First install the readline-dev packages for your OS. For Ubuntu I installed the libreadline6-dev package using apt-get, but this may vary a bit depending on which OS you’re running.

sudo apt-get install libreadline6-dev

Then you need to get the readline egg from the egg repository and install it. Fortunately chicken-install makes this a snap!

chicken-install readline

Then create a new file called .csirc in your home directory and also a .csi.history file:

touch ~/.csirc
touch ~/.csi.history

And put the following in .csirc:

(use readline irregex)
(current-input-port (make-gnu-readline-port))
(gnu-history-install-file-manager
  (string-append (or (get-environment-variable "HOME") ".") "/.csi.history"))

That’s it! Readline and history should now work in Chicken!

We are finally going to implement lambda and procedures.

To begin implementing procedures we have to define a procedure type, just like we did with primitives. A procedure has 3 basic parts. A list of parameters, a body and an environment. So we’ll set up a new procedure type with these fields and call it Procedure.

scheme_types.py

class Symbol(str):
  """A symbol is an immutable string, but must be its own type."""
  pass

class The_Empty_List():
  """The empty list is used as a list terminator and should be a singleton"""
  def __repr__(self):
    return "()"

the_empty_list = The_Empty_List()

class Pair(object):
  """A pair is a classic Lisp cons cell used to implement lists"""
  def __init__(self, car, cdr):
    self.car = car
    self.cdr = cdr
  def __iter__(self):
    x = self
    while not isinstance(x, The_Empty_List):
      yield x.car
      x = x.cdr
  def __repr__(self):
    return "(" + ' '.join([str(i) for i in list(self)]) + ")"

class Primitive(object):
  def __init__(self, fn):
    self.fn = fn

class Procedure(object):
  def __init__(self, parameters, body, environment):
    self.parameters = parameters
    self.body = body
    self.environment = environment

Like primitives, procedures are handled by the scheme_apply procedure. We’ll temporarily add a placeholder in scheme_apply for procedures. In addition we need new syntax to create a procedure. The syntax for procedures in Scheme is (lambda (parameters) body). Most Schemes allow the syntax (define (name params …) body …) but this is really just shorthand for (define name (lambda (params …) body …)). For now we won’t overload the define keyword, choosing instead to define functions the more verbose way.

It is important to take note of this before moving on. In many programming languages defining a function is a single step. However logically, there are 2 different operations that take place. First a function object is created, then that function object is bound to a symbol. By requiring the (define name (lambda …)) syntax we are making these two steps explicit.

scheme_eval.py

from scheme_types import Symbol, Pair, Primitive, the_empty_list, Procedure
from buffered_input import Buff
from scheme_read import scheme_read

##############################################################################
## Environments
##############################################################################

def current_environment(env):
  """Return the frame of the current environment"""
  return env.car

def enclosing_environment(env):
  """Return the environment stack with the first frame removed"""
  return env.cdr

def extend_environment(bindings, base_environment):
  """Push a new frame onto the given base_environment"""
  return Pair(bindings, base_environment)

global_environment = extend_environment({}, the_empty_list)
special_forms = {}

def set_symbol(symbol, val, env):
  """Set the binding (symbol, val) in the current frame of env"""
  current_environment(env)[symbol] = val

def lookup_symbol_value(symbol, environment):
  """Return the value of symbol or Unbound Symbol error"""
  env = environment
  while env != the_empty_list:
    if symbol in current_environment(env):
      return current_environment(env)[symbol]
    else:
      env = enclosing_environment(env)
  return "Error: Unbound symbol: " + symbol

##############################################################################
## Eval and Apply
##############################################################################

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr, env):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr, env)
  elif type(expr) is Pair:
    if expr.car in special_forms:
      return special_forms[expr.car](expr.cdr, env)
    else:
      return scheme_apply(scheme_eval(expr.car, env), [scheme_eval(a, env) for a in expr.cdr])
  else:
    return "scheme_eval: not implemented"

def scheme_apply(proc, args):
  if type(proc) is Primitive:
    return apply(proc.fn, args)
  elif type(proc) is Procedure:
    return "Attempted to apply a procedure"
  else:
    return "Error: Undefined procedure"

##############################################################################
## Builtin Syntax
##############################################################################

def special_form_handler(expr, env):
  """Register a symbol with a Python function named "f" that implements a special form"""
  exec(expr.cdr.car)
  special_forms[expr.car] = f

def load(expr):
  """Given a filename, open it and eval each expression in the global_environment"""
  f = open(expr.car, 'r')
  b = Buff(f)
  while b.peek():
    scheme_eval(scheme_read(b), global_environment)
    b.remove_whitespace()
  f.close()

special_forms['scheme-syntax'] = special_form_handler
special_forms['load'] = load

syntax.scm

(scheme-syntax define-primitive "
def f(expr, env):
  set_symbol(expr.car, Primitive(eval(expr.cdr.car)), env)
")

(scheme-syntax define "
def f(expr, env):
  set_symbol(expr.car, scheme_eval(expr.cdr.car, env), env)
")

(scheme-syntax if "
def f(expr, env):
  if scheme_eval(expr.car, env):
    return scheme_eval(expr.cdr.car, env)
  else:
    return scheme_eval(expr.cdr.cdr.car, env)
")

(scheme-syntax lambda "
def f(expr, env):
  return Procedure(expr.car, expr.cdr.car, env)
")

(define a (lambda (x) x))
a
;===> <scheme_types.Procedure object at 0xb733b10c>
(a 42)
;===> Attempted to apply a procedure

Now that we have a basic outline for what our procedures will look like, we can focus on making them work!

To apply a procedure we have to evaluate the body. In the example above the body of the procedure is just x. However we can’t just evaluate x because x is not bound to anything yet.

First we need to create a new environment and bind the parameters to the arguments supplied in the procedure call. In this case we have to bind x to the value 42. Then we just evaluate each expression in the body within the new environment and return the result of evaluating the last expression.

from scheme_types import Symbol, Pair, Primitive, the_empty_list, Procedure
from buffered_input import Buff
from scheme_read import scheme_read

##############################################################################
## Environments
##############################################################################

def current_environment(env):
  """Return the frame of the current environment"""
  return env.car

def enclosing_environment(env):
  """Return the environment stack with the first frame removed"""
  return env.cdr

def extend_environment(bindings, base_environment):
  """Push a new frame onto the given base_environment"""
  return Pair(bindings, base_environment)

global_environment = extend_environment({}, the_empty_list)
special_forms = {}

def set_symbol(symbol, val, env):
  """Set the binding (symbol, val) in the current frame of env"""
  current_environment(env)[symbol] = val

def lookup_symbol_value(symbol, environment):
  """Return the value of symbol or Unbound Symbol error"""
  env = environment
  while env != the_empty_list:
    if symbol in current_environment(env):
      return current_environment(env)[symbol]
    else:
      env = enclosing_environment(env)
  return "Error: Unbound symbol: " + symbol

##############################################################################
## Eval and Apply
##############################################################################

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr, env):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr, env)
  elif type(expr) is Pair:
    if expr.car in special_forms:
      return special_forms[expr.car](expr.cdr, env)
    else:
      return scheme_apply(scheme_eval(expr.car, env), [scheme_eval(a, env) for a in expr.cdr])
  else:
    return "scheme_eval: not implemented"

def scheme_apply(proc, args):
  if type(proc) is Primitive:
    return apply(proc.fn, args)
  elif type(proc) is Procedure:
    new_env = extend_environment(dict(zip(proc.parameters, args)), proc.environment)
    return [scheme_eval(e,new_env) for e in proc.body][-1]
  else:
    return "Error: Undefined procedure"

##############################################################################
## Builtin Syntax
##############################################################################

def special_form_handler(expr, env):
  """Register a symbol with a Python function named "f" that implements a special form"""
  exec(expr.cdr.car)
  special_forms[expr.car] = f

def load(expr):
  """Given a filename, open it and eval each expression in the global_environment"""
  f = open(expr.car, 'r')
  b = Buff(f)
  while b.peek():
    scheme_eval(scheme_read(b), global_environment)
    b.remove_whitespace()
  f.close()

special_forms['scheme-syntax'] = special_form_handler
special_forms['load'] = load

(define test (pred conseq alt)
  (if pred conseq alt))
(test 1 2 3)
;===> 2
(define-primitive + "lambda x,y: x+y")
(define-primitive < "lambda x,y: x<y")
(define a (lambda (x y) (if (< x y) (a (+ x 1) y) x)))
(a 1 3)
;===> 3

With that, we have implemented lambda in Scheme! However there are some glaring omissions from our implementation. For example ((lambda () 42)) will result in an iteration over non-sequence error from trying to iterate over the_empty_list. Also, tail-call elimination is not implemented as required by the Scheme standard.

Also note that there is very little error-checking going on here. For instance calling a procedure with the wrong number of arguments results in the whole interpreter crashing. This would not be good in a production language. However including error checking in this code would probably triple its size at least. The point of this exercise is to learn the concepts, not write a bullet-proof language.

I will likely come back to this at some point in the future and address some of these concerns. The rest is left for you to experiment with. You can get more information on how and what to implement from R5RS.

I will be Jumping back into Javascript for the next few posts at least, so this series will not be updated for a while (with the exception of bug fixes). When I do get back to language design and implementation, I will probably pick back up with the Scheme version of this interpreter, or possibly with a new project on compiling.

If you have any questions about how to proceed from here, or just want to show off your own version, please leave a comment.

Checkout this version on GitHub (v0.10)

GOTO | Table of Contents

We’ve been dealing with an environment ever since part 5 in this series when we implemented define, except we’ve been calling it a frame. Until now our frame has served us well, it has provided a simple environment where we could set and retrieve the value of symbols. However implementing procedures requires us to add a few features to our frame. Consider the code below.

(define x 5)
(define (func x)
  x)
(func 100)
;===> 100

With our frame this wouldn’t work as expected. Because we have already defined x in the global environment, there will be one of two problems with the x in func. Our function could use the global value of x (5) instead of the supplied value (100) or when the supplied value (100) is mapped onto the function body it will overwrite the value of the global value of x (5) with the supplied value (100).

Currently we do not make any distinction between variables in a function body and in the global environment. To properly implement procedures we will have to figure out a way to do that. Consider the next bit of code.

(define x 5)
(define (func y)
  (+ x y))
(func 10)
;===> 15

Even though we need to make a distinction between environments they also need to be connected. In the example above we use the global variable x in func without ever declaring it. This is accomplished by a method called chaining environments.

(define x 1)
(define (outer-func y)
  (define (inner-func z)
    (+ x y z)
  (inner-func 3))
(outer-func 2)
;===> 6

The environments produced by the above code are best described by the diagram below.

So when x appears in inner-func we have to check inner-func’s environment for x, then outer-func’s environment for x and finally the global environment for x before we raise an undefined symbol error.

You may have noticed that the diagram above looks a lot like a linked list. We can take our frame and combine it with linked lists to form an environment. It is more informative (and closer to the actual design) to view environments from the point of view of eval. This diagram is a view of the environment structure when the code (+ x y z) is evaluated.

I used the names current-environment to refer to the inner-func environment and enclosing-environment to refer to the outer-func environment. We will use these same names in our source code. Note that the global-environment is just the enclosing-environment of outer-func, the only thing special about it is that its enclosing-environment is the empty list.

In addition we need to replace our global frame with global_environment, which we’ll do by providing a procedure extend_environment, and change lookup_symbol_value to check enclosing-environments if the symbol is not found.

scheme_eval.py

from scheme_types import Symbol, Pair, Primitive, the_empty_list
from buffered_input import Buff
from scheme_read import scheme_read

def current_environment(env):
  """Return the frame of the current environment"""
  return env.car

def enclosing_environment(env):
  """Return the environment stack with the first frame removed"""
  return env.cdr

def extend_environment(bindings, base_environment):
  """Push a new frame onto the given base_environment"""
  return Pair(bindings, base_environment)

global_environment = extend_environment({}, the_empty_list)
special_forms = {}

def set_symbol(symbol, val, env):
  """Set the binding (symbol, val) in the current frame of env"""
  current_environment(env)[symbol] = val

def lookup_symbol_value(symbol, environment):
  """Return the value of symbol or Unbound Symbol error"""
  env = environment
  while env != the_empty_list:
    if symbol in current_environment(env):
      return current_environment(env)[symbol]
    else:
      env = enclosing_environment(env)
  return "Error: Unbound symbol: " + symbol

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr, global_environment)
  elif type(expr) is Pair:
    if expr.car in special_forms:
      return special_forms[expr.car](expr.cdr)
    else:
      return scheme_apply(scheme_eval(expr.car), [scheme_eval(a) for a in expr.cdr])
  else:
    return "scheme_eval: not implemented"

def scheme_apply(proc, args):
  if type(proc) is Primitive:
    return apply(proc.fn, args)
  else:
    return "Error: Undefined procedure"

def special_form_handler(expr):
  """Register a symbol with a Python function named "f" that implements a special form"""
  exec(expr.cdr.car)
  special_forms[expr.car] = f

def load(expr):
  """Given a filename, open it and eval each expression in the global_environment"""
  f = open(expr.car, 'r')
  b = Buff(f)
  while b.peek():
    scheme_eval(scheme_read(b))
    b.remove_whitespace()
  f.close()

special_forms['scheme-syntax'] = special_form_handler
special_forms['load'] = load

We also need to make some changes to syntax.scm to use the new set_symbol function.

syntax.scm

(scheme-syntax define-primitive "
def f(expr):
  set_symbol(expr.car, Primitive(eval(expr.cdr.car)), global_environment)
")

(scheme-syntax define "
def f(expr):
  set_symbol(expr.car, scheme_eval(expr.cdr.car), global_environment)
")

(scheme-syntax if "
def f(expr):
  if scheme_eval(expr.car):
    return scheme_eval(expr.cdr.car)
  else:
    return scheme_eval(expr.cdr.cdr.car)
")

(define a 5)
;===> #<unspecified>
a
;===> 5

Before we go any further I’m going to add in some comments to the scheme_eval.py file, mainly to group code into logical sections. I won’t provide a highlighted diff here, but you will notice the sections in the next bit of code.

To use the full potential of our new environments we are going to have to rewrite our program a little bit to pass them around correctly. Since eval is the main method of dispatch for our interpreter we’ll supply the environment as an argument to eval, which means finding every use of scheme_eval and adding the env argument. We’ll also have to pass an environment variable to procedures like set_symbol and lookup_symbol_value. One final place we’ll need environments is in scheme-syntax, we’ll have to change the stored procedures there to use “def f(expr, env):” instead of just “def f(expr):”.

from scheme_types import Symbol, Pair, Primitive, the_empty_list
from buffered_input import Buff
from scheme_read import scheme_read

##############################################################################
## Environments
##############################################################################

def current_environment(env):
  """Return the frame of the current environment"""
  return env.car

def enclosing_environment(env):
  """Return the environment stack with the first frame removed"""
  return env.cdr

def extend_environment(bindings, base_environment):
  """Push a new frame onto the given base_environment"""
  return Pair(bindings, base_environment)

global_environment = extend_environment({}, the_empty_list)
special_forms = {}

def set_symbol(symbol, val, env):
  """Set the binding (symbol, val) in the current frame of env"""
  current_environment(env)[symbol] = val

def lookup_symbol_value(symbol, environment):
  """Return the value of symbol or Unbound Symbol error"""
  env = environment
  while env != the_empty_list:
    if symbol in current_environment(env):
      return current_environment(env)[symbol]
    else:
      env = enclosing_environment(env)
  return "Error: Unbound symbol: " + symbol

##############################################################################
## Eval and Apply
##############################################################################

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr, env):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr, env)
  elif type(expr) is Pair:
    if expr.car in special_forms:
      return special_forms[expr.car](expr.cdr, env)
    else:
      return scheme_apply(scheme_eval(expr.car, env), [scheme_eval(a, env) for a in expr.cdr])
  else:
    return "scheme_eval: not implemented"

def scheme_apply(proc, args):
  if type(proc) is Primitive:
    return apply(proc.fn, args)
  else:
    return "Error: Undefined procedure"

##############################################################################
## Builtin Syntax
##############################################################################

def special_form_handler(expr, env):
  """Register a symbol with a Python function named "f" that implements a special form"""
  exec(expr.cdr.car)
  special_forms[expr.car] = f

def load(expr):
  """Given a filename, open it and eval each expression in the global_environment"""
  f = open(expr.car, 'r')
  b = Buff(f)
  while b.peek():
    scheme_eval(scheme_read(b), global_environment)
    b.remove_whitespace()
  f.close()

special_forms['scheme-syntax'] = special_form_handler
special_forms['load'] = load

repl.py

import sys
from scheme_read import scheme_read
from scheme_eval import scheme_eval, special_forms, global_environment
from scheme_types import Pair
from buffered_input import Buff

special_forms['load'](Pair("syntax.scm", None))

while True:
  inp = scheme_eval(scheme_read(Buff(sys.stdin)), global_environment)
  if inp != None:
    print ';===>', inp

syntax.chicken

(scheme-syntax define-primitive "
def f(expr, env):
  set_symbol(expr.car, Primitive(eval(expr.cdr.car)), env)
")

(scheme-syntax define "
def f(expr, env):
  set_symbol(expr.car, scheme_eval(expr.cdr.car, env), env)
")

(scheme-syntax if "
def f(expr, env):
  if scheme_eval(expr.car, env):
    return scheme_eval(expr.cdr.car, env)
  else:
    return scheme_eval(expr.cdr.cdr.car, env)
")

(define a 5)
a
;===> 5
(if 1 2 3)
;===> 2
(if #f 1 2)
;===> 2
(define-primitive + "lambda x,y: x+y")
(+ 3 5)
;===> 8

If you still don’t fully understand environments, that’s OK. You’ll get a chance to play around with them in the next part where we’ll finally cover implementing procedures in Lispy!

Checkout this version on GitHub (v0.09)

GOTO Part 10 | lambda

GOTO | Table of Contents

The goal for this part is to implement primitive procedures, apply and a method of setting new primitives.

The first thing we need to do is figure out a way to represent our primitive procedures. SICP uses tagged lists, which are just regular lists with the first element designated as a tag. Python uses objects to define new types and since we’ve already been doing that, we’ll continue here.

class Symbol(str):
  """A symbol is an immutable string, but must be its own type."""
  pass

class The_Empty_List():
  """The empty list is used as a list terminator and should be a singleton"""
  def __repr__(self):
    return "()"

the_empty_list = The_Empty_List()

class Pair(object):
  """A pair is a classic Lisp cons cell used to implement lists"""
  def __init__(self, car, cdr):
    self.car = car
    self.cdr = cdr
  def __iter__(self):
    x = self
    while not isinstance(x, The_Empty_List):
      yield x.car
      x = x.cdr
  def __repr__(self):
    return "(" + ' '.join([str(i) for i in list(self)]) + ")"

class Primitive(object):
  def __init__(self, fn):
    self.fn = fn

If the symbol of an expression of the form (symbol args …) is not found in special_forms it must be a procedure application (or not exist). To apply a primitive procedure first we have to evaluate all the arguments using scheme_eval. Then we use that list as the args value to apply(primitive, args). The primitive returns a Scheme value and we continue on.

For now we’re going to hardcode one primitive + into our interpreter. We’ll come up with something a little more elegant next.

from scheme_types import Symbol, Pair, Primitive
from buffered_input import Buff
from scheme_read import scheme_read

frame = {'+':Primitive(lambda x,y: x+y)}
special_forms = {}

def lookup_symbol_value(symbol):
  try:
    return frame[symbol]
  except KeyError:
    return "Error: Unbound variable"

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr)
  elif type(expr) is Pair:
    if expr.car in special_forms:
      return special_forms[expr.car](expr.cdr)
    else:
      return scheme_apply(scheme_eval(expr.car), [scheme_eval(a) for a in expr.cdr])
  else:
    return "scheme_eval: not implemented"

def scheme_apply(proc, args):
  if type(proc) is Primitive:
    return apply(proc.fn, args)
  else:
    return "Error: Undefined procedure"

def special_form_handler(expr):
  """Register a symbol with a Python function named "f" that implements a special form"""
  exec(expr.cdr.car)
  special_forms[expr.car] = f

def load(expr):
  """Given a filename, open it and eval each expression in the global_environment"""
  f = open(expr.car, 'r')
  b = Buff(f)
  while b.peek():
    scheme_eval(scheme_read(b))
    b.remove_whitespace()
  f.close()

special_forms['scheme-syntax'] = special_form_handler
special_forms['load'] = load

(+ 3 4)
;===> 7
(- 3 4)
;===> Error: Undefined procedure

The next step is to allow defining primitive procedures from within Scheme, as we did with special forms. We’ll implement define-primitive as a scheme-syntax macro. First remove the line that sets the + symbol. We’re only going to be modifying the syntax.scm file for now so that’s all I’ll show you.

We need to create a scheme-syntax macro that makes a Primitive object and sets it as the value of a symbol. This macro is a lot like the define macro that we wrote earlier. The only difference is instead of calling scheme_eval on the cadr we call eval and pass that result to make-primitive.

syntax.scm

(scheme-syntax define-primitive "
def f(expr):
  frame[expr.car] = Primitive(eval(expr.cdr.car))
")

(scheme-syntax define "
def f(expr):
  frame[expr.car] = scheme_eval(expr.cdr.car)
")

(scheme-syntax if "
def f(expr):
  if scheme_eval(expr.car):
    return scheme_eval(expr.cdr.car)
  else:
    return scheme_eval(expr.cdr.cdr.car)
")

With that we can define all sorts of primitives for our language…

(define-primitive + "lambda x,y: x+y")
(+ 3 4)
;===> 7
(- 3 4)
;===> Error: Undefined procedure

(define-primitive - "lambda x,y: x-y")
(- 3 4)
;===> -1

(define-primitive square "lambda x: x*x")
(square 4)
;===> 16

(define-primitive len "len")
(len "hello")
;===> 5

You can start a new file and define a bunch of primitives, load it the same way you load syntax.scm. I’m not going to define any primitives just yet but you are more than welcome to. Instead in the next post we’ll implement environments and then it’s on to implementing lambda!

Checkout this version on GitHub (v0.08)

GOTO Part 9 | Environments

GOTO | Table of Contents

In this part we’re going to be doing a bit of refactoring, so let’s start with that and a recap. The first thing that we’re going to do is remove define and if from our language. We’ll put them back in later, but first let’s look at what our language looks like with only scheme-syntax defined.

from scheme_types import Symbol, Pair

frame = {}
special_forms = {}

def lookup_symbol_value(symbol):
  try:
    return frame[symbol]
  except KeyError:
    return "Error: Unbound variable"

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr)
  elif type(expr) is Pair:
    if expr.car in special_forms:
      return special_forms[expr.car](expr.cdr)
    else:
      return "Error: unbound special form"
  else:
    return "scheme_eval: not implemented"

def special_form_handler(expr):
  """Register a symbol with a Python function named "f" that implements a special form"""
  exec(expr.cdr.car)
  special_forms[expr.car] = f

special_forms['scheme-syntax'] = special_form_handler

Let’s take a moment to consider what we can do so far. We now have a language that can understand self-evaluating Scheme values and lookup symbol values. In addition to that it has exactly one primitive, a macro that defines other primitives by giving us access to Python.

You may consider this language lacking, and I admit it does not have many features. However it is now possible to define any language construct (including procedures, eval and apply) from within Scheme with the help of scheme-syntax and Python underneath. After writing this code we could define the rest of the language 100% from within Scheme (though it may not be easy).

However there are a few more things that conceptually belong in the core language. One of those things is the ability to load a Scheme file. This will let us break out our syntax definitions into another file and load it automatically before we start the repl.

from scheme_types import Symbol, Pair
from buffered_input import Buff
from scheme_read import scheme_read

frame = {}
special_forms = {}

def lookup_symbol_value(symbol):
  try:
    return frame[symbol]
  except KeyError:
    return "Error: Unbound variable"

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr)
  elif type(expr) is Pair:
    if expr.car in special_forms:
      return special_forms[expr.car](expr.cdr)
    else:
      return "Error: unbound special form"
  else:
    return "scheme_eval: not implemented"

def special_form_handler(expr):
  """Register a symbol with a Python function named "f" that implements a special form"""
  exec(expr.cdr.car)
  special_forms[expr.car] = f

def load(expr):
  """Given a filename, open it and eval each expression in the global_environment"""
  f = open(expr.car, 'r')
  b = Buff(f)
  while b.peek():
    scheme_eval(scheme_read(b))
    b.remove_whitespace()
  f.close()

special_forms['scheme-syntax'] = special_form_handler
special_forms['load'] = load

Now that we have load, we need something to load. Create a new file called syntax.scm, or whatever you prefer. This is where we will define our primitive syntax forms like define, if and lambda. For this moment we’re going to leave it blank, and make the repl load it by default so we don’t have to do it manually each time.

Also I’m going to make some changes to the repl to make our sessions look more like the examples I’ve been giving (so ;===> will come before the output and there will be no prompt for input). Later I may go back to the beginning of this series and change this, as this is how it should have been from the start. I’ll also take this opportunity to skip printing any None value which will make using expressions that don’t return values more aesthetically pleasing.

repl.py

import sys
from scheme_read import scheme_read
from scheme_eval import scheme_eval, special_forms
from scheme_types import Pair
from buffered_input import Buff

special_forms['load'](Pair("syntax.scm", None))

while True:
  inp = scheme_read(Buff(sys.stdin))
  if inp != None:
    print ';===>', scheme_eval(inp)

Now we can define some special forms in syntax.scm to load by default. For now we’ll just reimplement define and if.

syntax.scm

(scheme-syntax define "
def f(expr):
  special_forms[expr.car] = scheme_eval(expr.cdr.car)
")

(scheme-syntax if "
def f(expr):
  if scheme_eval(expr.car):
    return scheme_eval(expr.cdr.car)
  else:
    return scheme_eval(expr.cdr.cdr.car)
")

(define a 5)
a
;===> 5
(if 1 2 3)
;===> 2

Now that we have implemented load we are one step closer to having a platform that we can build almost any language we want with. To change languages all we would have to do is load a different syntax file.

I encourage you to play around with that concept for a little while. Write two different syntax.scm files and implement a couple of basic forms like define, or and if. Try to make your two implementations as different as possible even if making it different doesn’t make sense. For instance in one implementation if could take a mandatory keyword before the alternate branch (if pred consequent else alt) or it could act like and, taking two predicates and only evaluating the consequent branch if both predicates evaluate to true.

This article is a little short on code because we made some larger conceptual changes that I did not want to breeze past. In the next article we’ll implement primitive procedures and apply. The ability to define procedures from within Scheme is just around the corner.

Checkout this version on GitHub (v0.07)

GOTO Part 8 | Primitive procedures and apply

GOTO | Table of Contents

In the last part we implemented our first special form and the first expression that was not self-evaluating, define. The goal for this part is to implement (if predicate consequent alternate).

We’re going to start by implementing if the same way we implemented define. The first thing to do is test if the car of the pair equals if, then call a helper procedure we’ll name apply_if.

We know that if will have to evaluate the predicate argument to determine whether to evaluate the consequent or alternate arguments. If the predicate is true we will evaluate the consequent expression and not the alternate. If the predicate is false we will evaluate the alternate expression and not the consequent.

from scheme_types import Symbol, Pair

frame = {}

def lookup_symbol_value(symbol):
  try:
    return frame[symbol]
  except KeyError:
    return "Error: Unbound variable"

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr)
  elif type(expr) is Pair:
    if expr.car == "define":
      frame[expr.cdr.car] = scheme_eval(expr.cdr.cdr.car)
      return "set symbol -> value"
    elif expr.car == "if":
      return apply_if(expr.cdr.car, expr.cdr.cdr.car, expr.cdr.cdr.cdr.car)
    else:
      return "scheme_eval: not implemented"
  else:
    return "scheme_eval: not implemented"

def apply_if(predicate, consequent, alternate):
  if scheme_eval(predicate):
    return scheme_eval(consequent)
  else:
    return scheme_eval(alternate)

(if 1 2 3)
;===> 2
(if #f 2 3)
;===> 3

Dispatching special forms using if is fine while you have a small number of primitive syntax forms. However once you implement a large number of primitive syntax forms this method of dispatch becomes cumbersome. Also note that it is not possible to extend the primitive syntax forms except by modifying the original source by hand.

Reusing the Frame

On top of that we have already implemented a mechanism of associating and storing symbol and value pairs. Let’s try something like that first, we’ll start by making another global dictionary called special_forms. Then we’ll modify eval to check this dictionary instead of using if/elif.

from scheme_types import Symbol, Pair

frame = {}
special_forms = {}

def lookup_symbol_value(symbol):
  try:
    return frame[symbol]
  except KeyError:
    return "Error: Unbound variable"

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr)
  elif type(expr) is Pair:
    if expr.car in special_forms:
      return special_forms[expr.car](expr.cdr)
    else:
      return "Error: unbound special form"
  else:
    return "scheme_eval: not implemented"

def define(expr):
  frame[expr.car] = scheme_eval(expr.cdr.car)
  return "set symbol -> value"

def apply_if(expr):
  if scheme_eval(expr.car):
    return scheme_eval(expr.cdr.car)
  else:
    return scheme_eval(expr.cdr.cdr.car)

special_forms['define'] = define
special_forms['if'] = apply_if

(repl)
(define a 5)
;===> #<unspecified>
a
;===> 5
(if 1 2 3)
;===> 2
(hello)
;===> Not implemented

With that addition we are finished with the first level of the eval function. An expression can only be one of three things: self-evaluating, a symbol or an application (either syntax or procedure which we’ll cover later).

How does it work? special_forms stores (symbol, procedure) pairs where the symbol is a keyword (like define or if) and the procedure is the Python code that implements that special form. Every procedure in special_forms takes a single argument, an unevaluated pair representing the rest of the expression passed to scheme_eval. Because special forms have their own evaluation rules, it is the responsibility of this procedure to evaluate the expression and return a result.

In other words, when lispy-eval comes across a special form it transfers control to a Python procedure located in the special_forms dict. That Python procedure is responsible for evaluating the arguments and returning a Scheme value.

Now that we have a data structure to hold our special_forms it would be a good idea to supply an easy mechanism to set a new special form. For this purpose we’ll design a special form in Scheme that will set new special forms called scheme-syntax.

from scheme_types import Symbol, Pair

frame = {}
special_forms = {}

def lookup_symbol_value(symbol):
  try:
    return frame[symbol]
  except KeyError:
    return "Error: Unbound variable"

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr)
  elif type(expr) is Pair:
    if expr.car in special_forms:
      return special_forms[expr.car](expr.cdr)
    else:
      return "Error: unbound special form"
  else:
    return "scheme_eval: not implemented"

def special_form_handler(expr):
  """Register a symbol with a Python function named "f" that implements a special form"""
  exec(expr.cdr.car)
  special_forms[expr.car] = f

special_forms['scheme-syntax'] = special_form_handler

def define(expr):
  frame[expr.car] = scheme_eval(expr.cdr.car)
  return "set symbol -> value"

def apply_if(expr):
  if scheme_eval(expr.car):
    return scheme_eval(expr.cdr.car)
  else:
    return scheme_eval(expr.cdr.cdr.car)

special_forms['define'] = define
special_forms['if'] = apply_if

(scheme-syntax quote "def f(expr): return expr.car")
;===> None
(1 2 3)
;===> Error: unbound special form
(quote (1 2 3))
;===> (1 2 3)

With the addition of scheme-syntax it is now possible to write Python directly in our interpreter. Because of this we can now implement all of the primitive syntax forms in Scheme. Some might consider this cheating, but eventually we’re going to want a way to interface with Python. Why not implement it right away, use it, test it and make it solid by implementing everything else in it? We’ll still have to make some modifications to the core interpreter to implement procedures and a load form later.

Because of Python’s restrictions on lambda expressions, this solution is not as elegant as the scheme-syntax form in Scheme in Scheme. For this version you must define a named function, and it must be named “f” to make this implementation work. If you know of a more elegant way to approach this please let me know in the comments.

We haven’t even gotten to the mutually recursive yin-yang that is the eval/apply cycle yet. However, thanks to piggy-backing on Python, we already have a somewhat useful language. A language without procedures is a pretty bad one, so we’ll keep pressing on, but check out some of the things that are possible already…

(scheme-syntax print "
def f(expr):
  print scheme_eval(expr.car)
  return None
")
;===> None
(scheme-syntax loop "
def f(expr):
  for n in range(scheme_eval(expr.car)):
    scheme_eval(expr.cdr.car)
")
;===> None
(loop 3 (print "Hi"))
;===> Hi
;===> Hi
;===> Hi
;===> None

In the next article we’ll do a little refactoring, remove define and if from the core of the interpreter and implement a load function so we can remove our syntax definitions out of the main interpreter file.

Checkout this version on GitHub (v0.06)

GOTO Part 7 | Refactor and load

GOTO | Table of Contents

Our goal in this part is to implement assignment via the form (define symbol value). There are three challenges to this goal. First we need to come up with a system to store symbol names and their values. Then we need to display those values when the symbol is eval’d. Finally we need to recognize the define form and set the value of its symbol.

Starting with a Stub

To start with we need to modify our eval procedure to recognize symbols. We’ll call a procedure lookup_symbol_value that will eventually give the value of the symbol. Right now we’ll just have it return a string stating our intentions.

from scheme_types import Symbol

def lookup_symbol_value(symbol):
  return "Here's where we lookup the value of the symbol."

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr)
  else:
    return "scheme_eval: not implemented"

The Global Frame

To set the value of a symbol we need to have somewhere to set it. That somewhere is going to be a dictionary. The dictionary that stores a set of names and their associated values is referred to as a frame.

For now we’ll define our frame as a global variable named frame. Since we don’t have any method of assigning values to symbols yet, we define a symbol “test” when we make the frame so we can see if it works.

from scheme_types import Symbol

frame = {'test':"Value retrieved successfully"}

def lookup_symbol_value(symbol):
  try:
    return frame[symbol]
  except KeyError:
    return "Error: Unbound variable"

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr)
  else:
    return "scheme_eval: not implemented"

test
;===> Value retrieved successfully
hello
;===> Error: Unbound variable

(define symbol value)

Now that we can store and retrieve the value of a symbol, we need a way to do it from within Scheme. Scheme uses the form (define symbol value) to define a new symbol or set the value of an existing symbol.

The form (define symbol value) is a pair (also a list, but all lists are also pairs) so the first step to detecting its usage is to look for pairs. The next step is to check the symbol in the car of that pair (or the first item of the list), if that symbol is “define” we run the code to define a symbol. For now we’ll just return a string stating our intentions.

from scheme_types import Symbol, Pair

frame = {'test':"Value retrieved successfully"}

def lookup_symbol_value(symbol):
  try:
    return frame[symbol]
  except KeyError:
    return "Error: Unbound variable"

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr)
  elif type(expr) is Pair:
    if expr.car == "define":
      return "set the value of the symbol"
    else:
      return "scheme_eval: not implemented"
  else:
    return "scheme_eval: not implemented"

test
;===> Value retrieved successfully
(define a 1)
;===> Set the value of the symbol.
(set a 1)
;===> scheme_eval: not implemented

Setting the Value of Symbols

To actually set the value of the symbol we need to get the arguments to define. Getting the symbol argument is straightforward, however we need to eval the value argument as it might not be self-evaluating.

Usually (define symbol value) will return void, or nothing. I have chosen to return a string here to give some visual feedback that setting the symbol happened as well as avoid having to modify the repl to handle None. You can choose to take the more standard approach and return None here, then modify the repl to skip printing None values.

from scheme_types import Symbol, Pair

frame = {}

def lookup_symbol_value(symbol):
  try:
    return frame[symbol]
  except KeyError:
    return "Error: Unbound variable"

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr)
  elif type(expr) is Pair:
    if expr.car == "define":
      frame[expr.cdr.car] = scheme_eval(expr.cdr.cdr.car)
      return "set symbol -> value"
    else:
      return "scheme_eval: not implemented"
  else:
    return "scheme_eval: not implemented"

a
;===> Error: Unbound variable
(define a 5)
;===> set symbol -> value
a
;===> 5

In addition to being able to set symbols, we now have a glimpse into how basic forms (like define) are handled. It really is no more complicated than checking the procedure name, evaluating the arguments that need to be, then passing the arguments to a Python procedure that implements the desired functionality.

* To any Schemers that may be reading this: I avoided implementing environments here for two reasons. I wanted to show the simplest implementation possible and in an upcoming post we will transform our global frame into a lookup table for primitive syntax.

Checkout this version on GitHub (v0.05)

GOTO Part 6 | scheme-syntax macro

GOTO | Table of Contents

In the last post we finished implementing the read layer. There is a lot of room for improvement, however it will serve our purposes for the moment. With the read layer complete we can move on to the eval layer.

The eval layer is where the Scheme in Scheme series starts and from here on there will be many similarities between these two series. In fact a sizable portion of this post was simply copy/pasted from the Scheme in Scheme series. With the exception of the code and language specific explanation of it, the rest of the series will also be a close replica.

Create a new file called scheme_eval.py, the rest of this series will focus mainly on this file. In it we will define a function scheme_eval. For now we’re simply going to echo the input back without processing it at all.

scheme_eval.py

def scheme_eval(expr):
  return expr

repl.py

import sys
from scheme_read import scheme_read
from scheme_eval import scheme_eval
from buffered_input import Buff

while True:
  print "> ",
  print scheme_eval(scheme_read(Buff(sys.stdin)))

With that we have the basic layout of the interpreter. If you read the last line backward you will see the layers of the interpreter print(eval(read(stdin))) all wrapped up in an infinite loop. From this point we can concentrate on evaluating the expressions that scheme_read returns.

A Language of Numbers

The simplest language is a language of numbers. Numbers are self-evaluating, so if the expression is a number we’ll just return it for now. If the expression is anything else, we’ll return an error string, just as we did in the read layer. If you are following along, you will probably want to add proper errors/error handling to your version.

def scheme_eval(expr):
  if type(expr) is int:
    return expr
  else:
    return "scheme_eval: not implemented"

A Language of Self-Evaluating Values

Continuing with our language of self-evaluating values is just as simple. Booleans, strings and characters are also self-evaluating so we’ll add them now. More complex data structures like hash tables are also self evaluating, however we’ll leave them for later.

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  else:
    return "scheme_eval: not implemented"

The first stage of the eval layer is complete. We now have a functional language of self-evaluating values. Though one could argue exactly how functional it really is. The next step along our path will be to introduce symbols and implement assignment.

Checkout this version on GitHub (v0.04)

GOTO Part 5 | Assignment and define

GOTO | Table of Contents

In the last post we added support for booleans, characters, symbols and strings. There is only one major data type left to add in the read layer, the pair/linked list.

A pair is a structure which contains 2 elements normally referred to as the car and cdr in Scheme. Lists are built on top of pairs where the car is some value and the cdr is another pair or the empty list. In a proper list the cdr of the last pair must be the empty list.

To illustrate this concept a bit more clearly here is a diagram of the list ‘(1 2 3) in Scheme:

This list is constructed out of 3 pairs. The car of the first pair is the value 1 and the cdr is the second pair. The car of the second pair has the value 2 and the cdr is the third pair. The car of the third pair has the value 3 and the cdr contains the empty list (Nil in the diagram).

class Pair:
  def __init__(self, car, cdr):
    self.car = car
    self.cdr = cdr

Above is the class definition for a very simple Pair object in Python. We can construct the above list using the Pair class as follows:

lst = Pair(1, Pair(2, Pair(3, None)))

This implementation of a list is called a linked list and you can find a more detailed description in my article WITH Linked Lists. Python does not have an implementation of a list in the core language, the list data type in Python is called an array in most other languages.

Scheme sourcecode is written using the linked list data type. This is one of the most powerful features of the language as the sourcecode is just a regular Scheme data type. This allows some really interesting features such as macros, but we won’t get into that just yet.

To start with we’ll implement the empty list. Since this is a distinct object in Scheme we’ll make a new type called The_Empty_List and since we only need one empty list we’ll also make a global variable to refer to it called the_empty_list. We’ll do this in the scheme_types.py file.

class Symbol(str):
  """A symbol is an immutable string, but must be its own type."""
  pass

class The_Empty_List():
  """The empty list is used as a list terminator and should be a singleton"""
  def __repr__(self):
    return "()"

the_empty_list = The_Empty_List()

Now that we have an empty list type we need to modify our reader to recognize it. When the “(” character is encountered, scheme_read will call read_pair. Since the only pair we’re going to allow for the moment is the empty list, we just need to check that the next character is “)” and return the_empty_list.

from scheme_types import Symbol, the_empty_list

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def is_initial(c):
  """Is c a valid initial character for an identifier?"""
  return c.isalpha() or c in '-+*/><=?!&'

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if '.' in buf:
    return float(buf)
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def read_expected_string(f, s):
  """Consume expected characters from the input buffer."""
  expected_chars = list(s)
  c = f.getc()
  while expected_chars:
    if c == expected_chars[0]:
      expected_chars.pop(0)
      c = f.getc()
    else:
      return "unexpected character"

def read_character(f):
  c = f.getc()
  if c == 's' and f.peek() == 'p':
    read_expected_string(f, "pace")
    return ' '
  elif c == 'n' and f.peek() == 'e':
    read_expected_string(f, "ewline")
    return '\n'
  else:
    return c

def read_symbol(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  return ''.join(buf)

def read_string(f):
  buf = []
  c = f.getc()
  while c != '\"':
    buf.append(c)
    c = f.getc()
  return ''.join(buf)

def read_pair(f):
  f.remove_whitespace()
  c = f.getc()
  if c == ')':
    return the_empty_list

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == '-' and f.peek().isdigit() or f.peek() == '.')
      or (c == '.' and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  elif c == '#':
    c = f.getc()
    if c == 't':
      return True
    elif c == 'f':
      return False
    elif c == '\\':
      return read_character(f)
    else:
      return "Error boolean value must be #t or #f not #" + c
  elif is_initial(c):
    f.ungetc(c)
    return Symbol(read_symbol(f))
  elif c == '\"':
    return read_string(f)
  elif c == '(':
    return read_pair(f)
  else:
    return "scheme_read: not implemented"

With that done, we’re only two steps away from being able to read linked lists. First we have to create a Pair type that will represent a Scheme pair. To do this, we’ll add the following to the scheme_types.py file.

class Symbol(str):
  """A symbol is an immutable string, but must be its own type."""
  pass

class The_Empty_List():
  """The empty list is used as a list terminator and should be a singleton"""
  def __repr__(self):
    return "()"

the_empty_list = The_Empty_List()

class Pair(object):
  """A pair is a classic Lisp cons cell used to implement lists"""
  def __init__(self, car, cdr):
    self.car = car
    self.cdr = cdr
  def __iter__(self):
    x = self
    while not isinstance(x, The_Empty_List):
      yield x.car
      x = x.cdr
  def __repr__(self):
    return "(" + ' '.join([str(i) for i in list(self)]) + ")"

In addition to a car and cdr field I have also implemented the __iter__ and __repr__ special methods. __iter__ will help us later on when we work with these pairs by allowing easy conversion to a Python list data type or allowing the use of for/in or map. For more information on __iter__ see the Python docs on iterators.

The only thing left to do is modify read_pair. The procedure to read a pair is as follows:

1. Get a value for the car slot by calling scheme_read on the input file
2. Remove any whitespace
3. Get a value for the cdr by calling read_pair on the input file
   • If the next character is “)” set the cdr to the_empty_list
   • If the next character is anything else set the cdr to a new pair by calling read_pair

from scheme_types import Symbol, the_empty_list, Pair

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def is_initial(c):
  """Is c a valid initial character for an identifier?"""
  return c.isalpha() or c in '-+*/><=?!&'

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if '.' in buf:
    return float(buf)
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def read_expected_string(f, s):
  """Consume expected characters from the input buffer."""
  expected_chars = list(s)
  c = f.getc()
  while expected_chars:
    if c == expected_chars[0]:
      expected_chars.pop(0)
      c = f.getc()
    else:
      return "unexpected character"

def read_character(f):
  c = f.getc()
  if c == 's' and f.peek() == 'p':
    read_expected_string(f, "pace")
    return ' '
  elif c == 'n' and f.peek() == 'e':
    read_expected_string(f, "ewline")
    return '\n'
  else:
    return c

def read_symbol(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  return ''.join(buf)

def read_string(f):
  buf = []
  c = f.getc()
  while c != '\"':
    buf.append(c)
    c = f.getc()
  return ''.join(buf)

def read_pair(f):
  f.remove_whitespace()
  c = f.getc()
  if c == ')':
    return the_empty_list
  f.ungetc(c)
  car = scheme_read(f)
  f.remove_whitespace()
  cdr = read_pair(f)
  return Pair(car, cdr)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == '-' and f.peek().isdigit() or f.peek() == '.')
      or (c == '.' and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  elif c == '#':
    c = f.getc()
    if c == 't':
      return True
    elif c == 'f':
      return False
    elif c == '\\':
      return read_character(f)
    else:
      return "Error boolean value must be #t or #f not #" + c
  elif is_initial(c):
    f.ungetc(c)
    return Symbol(read_symbol(f))
  elif c == '\"':
    return read_string(f)
  elif c == '(':
    return read_pair(f)
  else:
    return "scheme_read: not implemented"

There are two things to note about this implementation. First, () is matched as the_empty_list when it really shouldn’t be, the correct syntax is ‘(). Second, any literal list over 1,000 elements will hit Python’s recursion limit. This shouldn’t be a problem in the vast majority of cases, however it is something to be aware of.

With that, the read layer of our interpreter is complete (for the purposes of this series, there is a lot more that could be done). From this point on we will be concentrating on the eval layer of the interpreter.

Checkout this version on GitHub (v0.03)

GOTO Part 4 | Self-evaluating values

GOTO | Table of Contents

In the last post we created a read function that worked with integers and floating point numbers. We also covered the basic layout of the read function. In this post we will extend the read function to handle booleans, characters, strings and symbols.

We’ll start with booleans as they are the smallest set. There are 2 boolean values in Scheme #t and #f, which represent true and false respectively. You probably noticed that both start with a # character and that is how we will recognize them in the reader.

We will represent #t and #f by the Python values True and False. Since the code for this is so simple, we will handle this within the scheme_read function.

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if '.' in buf:
    return float(buf)
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == '-' and f.peek().isdigit() or f.peek() == '.')
      or (c == '.' and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  elif c == '#':
    c = f.getc()
    if c == 't':
      return True
    elif c == 'f':
      return False
    else:
      return "Error boolean value must be #t or #f not #" + c
  else:
    return "scheme_read: not implemented"

The code to handle booleans is pretty straightforward. If the first character is #, then we get the next character. If the next character is t or f, we return the appropriate boolean. Otherwise, we return an error string.

The next thing we’ll handle is characters. Characters in Scheme also start with the # symbol followed by a \, which gives us #\c where c is any character.

There are some special characters, but we’ll start by implementing the basics. We’ll check if an expression is a character in the same place as we check for booleans since they’re so similar.

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if '.' in buf:
    return float(buf)
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def read_character(f):
  c = f.getc()
  return c

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == '-' and f.peek().isdigit() or f.peek() == '.')
      or (c == '.' and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  elif c == '#':
    c = f.getc()
    if c == 't':
      return True
    elif c == 'f':
      return False
    elif c == '\\':
      return read_character(f)
    else:
      return "Error boolean value must be #t or #f not #" + c
  else:
    return "scheme_read: not implemented"

Scheme has a few special characters that we must deal with, namely #\space and #\newline. These will require a bit more work to process. To help, we’ll define a function read_expected_string which will consume characters from the input buffer, provided they match what’s expected (and return an error if they don’t).

I won’t go over the workings of read_expected_string any further than to say it compares characters from the expected string and the input buffer one-by-one until it reaches the end. We also need to add a check in read_character if the input buffer contains #\s or #\n. Both of these could be either the characters s or n or the special characters #\space or #\newline. To tell the difference we need to check the next character.

There are a few problems with this implementation, namely if you enter an invalid special character such as #\spxce it will leave unconsumed characters on the input buffer and will not return the error string. Handling these error conditions would add a lot of bloat to the code, though you would definitely want to do this in your own implementation.

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if '.' in buf:
    return float(buf)
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def read_expected_string(f, s):
  """Consume expected characters from the input buffer."""
  expected_chars = list(s)
  c = f.getc()
  while expected_chars:
    if c == expected_chars[0]:
      expected_chars.pop(0)
      c = f.getc()
    else:
      return "unexpected character"

def read_character(f):
  c = f.getc()
  if c == 's' and f.peek() == 'p':
    read_expected_string(f, "pace")
    return ' '
  elif c == 'n' and f.peek() == 'e':
    read_expected_string(f, "ewline")
    return '\n'
  else:
    return c

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == '-' and f.peek().isdigit() or f.peek() == '.')
      or (c == '.' and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  elif c == '#':
    c = f.getc()
    if c == 't':
      return True
    elif c == 'f':
      return False
    elif c == '\\':
      return read_character(f)
    else:
      return "Error boolean value must be #t or #f not #" + c
  else:
    return "scheme_read: not implemented"

Next up are symbols, which are the names of variables/constants. In most other languages symbols are not given their own type and you, as a user, are not allowed to manipulate them as first-class values. However this is not the case in Scheme.

We will represent symbols in our implementation as strings. However since we also need to implement strings, it is necessary to make symbols a distinct type. Since we will need to make other types as well as have access to them outside the read layer, make a new file called scheme_types.py and enter the following definition.

class Symbol(str):
  """A symbol is an immutable string, but must be its own type."""
  pass

How do we know if an expression is a symbol? There is a limited set of values that a symbol may start with. To test if a character is part of this set, we’ll provide a function is_initial.

Once we’ve identified that the expression is a symbol, the next step is to read each character into a buffer until we’ve hit a delimiter (just like reading a number). Then we turn the buffer into a string and pass it to Symbol to create a new symbol.

from scheme_types import Symbol

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def is_initial(c):
  """Is c a valid initial character for an identifier?"""
  return c.isalpha() or c in '-+*/><=?!&'

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if '.' in buf:
    return float(buf)
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def read_expected_string(f, s):
  """Consume expected characters from the input buffer."""
  expected_chars = list(s)
  c = f.getc()
  while expected_chars:
    if c == expected_chars[0]:
      expected_chars.pop(0)
      c = f.getc()
    else:
      return "unexpected character"

def read_character(f):
  c = f.getc()
  if c == 's' and f.peek() == 'p':
    read_expected_string(f, "pace")
    return ' '
  elif c == 'n' and f.peek() == 'e':
    read_expected_string(f, "ewline")
    return '\n'
  else:
    return c

def read_symbol(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  return ''.join(buf)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == '-' and f.peek().isdigit() or f.peek() == '.')
      or (c == '.' and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  elif c == '#':
    c = f.getc()
    if c == 't':
      return True
    elif c == 'f':
      return False
    elif c == '\\':
      return read_character(f)
    else:
      return "Error boolean value must be #t or #f not #" + c
  elif is_initial(c):
    f.ungetc(c)
    return Symbol(read_symbol(f))
  else:
    return "scheme_read: not implemented"

The reader does not lookup the value of symbols, that’s the job of the eval layer, so we’re all done for handling symbols.

With symbols done we only have strings left to implement for this part in the series. Luckily the implementation for strings is almost exactly the same as the one for symbols. Strings in Scheme start and end with “, so an expression is a string if the first character is a “.

The read_string function we’ll provide is a copy/paste of the read_symbol function with only one change. Instead of while not is_delimiter(c) we will use while c != ‘\”‘ since delimiter characters are allowed to appear in strings. Since the functionality is identical, you could abstract this into one function by adding a parameter that passes the stopping condition. However I feel that it’s more clear, for this series, as two.

from scheme_types import Symbol

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def is_initial(c):
  """Is c a valid initial character for an identifier?"""
  return c.isalpha() or c in '-+*/><=?!&'

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if '.' in buf:
    return float(buf)
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def read_expected_string(f, s):
  """Consume expected characters from the input buffer."""
  expected_chars = list(s)
  c = f.getc()
  while expected_chars:
    if c == expected_chars[0]:
      expected_chars.pop(0)
      c = f.getc()
    else:
      return "unexpected character"

def read_character(f):
  c = f.getc()
  if c == 's' and f.peek() == 'p':
    read_expected_string(f, "pace")
    return ' '
  elif c == 'n' and f.peek() == 'e':
    read_expected_string(f, "ewline")
    return '\n'
  else:
    return c

def read_symbol(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  return ''.join(buf)

def read_string(f):
  buf = []
  c = f.getc()
  while c != '\"':
    buf.append(c)
    c = f.getc()
  return ''.join(buf)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == '-' and f.peek().isdigit() or f.peek() == '.')
      or (c == '.' and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  elif c == '#':
    c = f.getc()
    if c == 't':
      return True
    elif c == 'f':
      return False
    elif c == '\\':
      return read_character(f)
    else:
      return "Error boolean value must be #t or #f not #" + c
  elif is_initial(c):
    f.ungetc(c)
    return Symbol(read_symbol(f))
  elif c == '\"':
    return read_string(f)
  else:
    return "scheme_read: not implemented"

With string handling completed there is only one major feature left for scheme_read, pairs. Pairs are used to implement linked lists in Scheme. The linked list is a very important data structure as all Scheme code is written and parsed into linked list format. This code/data duality is one of the things that gives Scheme (and other Lisps) its great power. The next post will be about the implementation of linked lists and how to parse them.

Checkout this version on GitHub (v0.02)

GOTO Part 3 | Pairs and linked lists

GOTO | Table of Contents

Stage 1 of implementing Scheme in Python is to write the reader. The entire read layer is self-contained so we don’t need to worry about evaluating and printing values just yet.

To start with let’s take a look at the basic shape of the read function. We’re going to call it scheme_read to avoid any name clashes that might arise, but other than that it should be more-or-less equivalent to any other Scheme read function.

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if c:
    return "Representation of a Scheme type."

scheme_read(f) is called with a buffered file-like object f that must support getc, ungetc, peek and remove_whitespace methods. First we remove any leading whitespace as whitespace in Scheme is only significant as a delimiter.

Before we get started let’s implement a simple repl so that we can play with our new features as we add them. Create a new file named repl.py and insert the following code.

#!/usr/bin/env python
import sys
from scheme_read import scheme_read
from buffered_input import Buff

while True:
  print "> ",
  print scheme_read(Buff(sys.stdin))

The next step is to get the first non-whitespace character, which we store in the variable c. We will use this character (and possibly the next) to determine the type of the expression. For instance if c is a digit then the remaining characters (until the next delimiter) must form a number, since digits are not allowed as the first character of an identifier. We’ll do this check in the if c: portion of our code. Let’s look at that code.

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if c.isdigit():
    return "Representation of a Scheme number."

What if c is not a digit? In that case, our read function does not recognize the input and we should return something that states this fact. For now let’s just return a string that says “scheme_read: not implemented”. Later on you may wish to make this a proper error, but returning a string here keeps things simple.

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if c.isdigit():
    return "Representation of a Scheme number."
  else:
    return "scheme_read: not implemented"

Now obviously, “Representation of a Scheme number.” is not actually a representation of a Scheme number. To get that we have to read the number from f. Instead of putting all the logic of reading a number into the if block, we should use a helper function here called read_number.

def read_number(f):
  return "Representation of a Scheme number."

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if c.isdigit():
    return read_number(f)
  else:
    return "scheme_read: not implemented"

We pass the file object f to read_number so that it can read and return the rest of the number. But wait, we already read the first digit! Before we pass the work off to read_number, we have to put the first digit back onto f. We do this using the ungetc method.

def read_number(f):
  return "Representation of a Scheme number."

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if c.isdigit():
    f.ungetc(c)
    return read_number(f)
  else:
    return "scheme_read: not implemented"

Now to read a number we simply read a character at a time, placing each into a buffer, until we hit a delimiter. To make that job easier we’ll provide a function is_delimiter that takes a character and returns True if that character is a delimiter and False if not. Then we’ll use that to build a buffer and return an integer.

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  return int(buf)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if c.isdigit():
    f.ungetc(c)
    return read_number(f)
  else:
    return "scheme_read: not implemented"

That takes care of positive integers, however the positive integers are a very small subset of numbers. We are going to want to handle negative integers as well as positive and negative floating point numbers at least. Luckily, this is very simple to do in Python.

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if '.' in buf:
    return float(buf)
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if c.isdigit():
    f.ungetc(c)
    return read_number(f)
  else:
    return "scheme_read: not implemented"

However, we’re not quite done yet. Our implementation won’t handle negative numbers. To do that we’re going to have to go back to the reader because “-” is not a digit and our current implementation will skip right over it.

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if '.' in buf:
    return float(buf)
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if c.isdigit() or (c == '-' and f.peek().isdigit()):
    f.ungetc(c)
    return read_number(f)
  else:
    return "scheme_read: not implemented"

We’re still not quite done though. What about floating point values where 0 < n < 1 that start with a decimal point?

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (‘ ‘, ‘(‘, ‘)’, ‘\"’, ‘;’, ‘\n’) or not c

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if ‘.’ in buf:
    return float(buf)
  elif ‘/’ in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == ‘-’ and f.peek().isdigit())
      or (c == ‘.’ and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  else:
    return "scheme_read: not implemented"

But wait, there's more! What about negative floating point numbers where -1 < n < 0 which start with a "-."?

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if '.' in buf:
    return float(buf)
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == '-' and f.peek().isdigit() or f.peek() == '.')
      or (c == '.' and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  else:
    return "scheme_read: not implemented"

As you can see, a lot of the work of implementing a programming language goes into dealing with corner cases. In the next post we'll extend our reader to handle booleans, characters, strings and symbols.

Checkout this version on GitHub (v0.01)

GOTO Part 2 | Extending the read layer

GOTO | Table of Contents

1) Read Layer
2) Eval Layer
3) Language Layer

The read layer

The read layer is responsible for consuming source code and transforming it into a data structure so that it can be passed to the eval layer.

The eval layer

After the reader returns a data structure representing the source code, the eval layer evaluates it according to a set of rules and returns a value.

After an expression has been evaluated it’s value is often printed to the screen. Immediately after that the program usually loops back to reading the next expression, hence the term REPL (Read, Eval, Print, Loop).

The language layer

This interpreter is designed using the “languages as libraries” concept. This approach is currently being used by the Racket and PyPy projects (though in a much more advanced form).

A language is usually defined as the core components, or primitives, plus additional libraries written using those primitives. In most languages it is an error to attempt to redefine a primitive procedure and even in languages where this is permissible, there is usually no way to swap all of the primitives.

Using the languages as libraries approach, we implement as little as possible in the source code of the interpreter and allow a file to be loaded with definitions of the primitives in the language itself. This implementation only requires one primitive to be defined at the interpreter level¹, define-syntax, which allows the user to define new primitive syntax.

Before we begin

The read function presented here is a port of the read function in Bootstrap Scheme by Peter Michaux. His implementation relies on the C functions getc and ungetc. Unfortunately input in Python is line-buffered and not character-buffered and Python does not provide ungetc.

To accommodate for this I have created a class (Buff) which wraps stdin (or any other file-like object) and provides the functions getc, ungetc and peek. Reading from input in this manner is not in the least bit Pythonic, however this method of reading is very simple and clearly shows the steps involved in lexing and parsing input strings.

The operation of Buff is simple. getc reads a single character from the given file if there are no characters waiting in the buffer, otherwise it pops off the last character added to the buffer. ungetc simply adds a character to the buffer, and peek uses getc to read a character and store it in a variable, then ungetc to put the character back in the buffer.

class Buff:
  def __init__(self, f):
    self.f = f
    self.last = []

  def getc(self):
    if self.last:
      return self.last.pop(0)
    else:
      return self.f.read(1)

  def ungetc(self, c):
    self.last.insert(0, c)

  def peek(self):
    if self.last:
      return self.last[0]
    else:
      c = self.f.read(1)
      self.last.append(c)
      return c

  def remove_whitespace(self):
    c = self.getc()
    while c:
      if c == ' ':
        c = self.getc()
      elif c == '\n':
        c = self.getc()
      elif c == ';':
        c = self.getc()
        while c and c != '\n':
          c = self.getc()
        self.ungetc(c)
      else:
        self.ungetc(c)
        break

You may notice that there is also a function: remove_whitespace. This simply reads and discards any characters which are whitespace (space and newline) and discards comments ;starting at the semicolon and continuing until a newline is reached.

With that we have the tools we need to get started building the read level.

[1] In theory, in practice we also define “load” to avoid the necessity of redefining it every time we restart the interpreter.

Checkout this project on GitHub

GOTO Part 1 | Numbers

GOTO | Table of Contents

Here’s the original question…

I’m new to scheme/racket macros and here is a simplified version of the problem I’m having:

> (define-syntax let-five
(syntax-rules ()
[(let-five expr ...)
(let ((five 5)) expr ...)]))
> (let-five (+ five 2))

This should return 7 but instead I get an error saying that the identifier “five” is unbound.
All I want let-five to do is to bind the identifier “five” to the number 5 so that I can use “five” within let-five. The reason I want to do this is more complicated than this example so don’t tell me that this macro is doesn’t do anything useful.
I am fairly sure that I know what is wrong, but my question is what would be the correct way to write this code so that it does what I intend?
Additional Details
Jack Trades, I don’t understand what x, r, and c are and what the % and @ are. It would be nice if you could explain that.

And here’s my response…

I don’t use Racket, but Chicken gives the same error (but probably a different error message).

The problem here is that syntax-rules is a hygienic macro transformer. Which basically means that the symbol five is being renamed by syntax-rules to avoid name clashes during expansion. You can see this clearly by looking at the error message (like I said they’re probably different from yours)…

    <syntax>          (##core#let ((five119 5)) (+ five 2))

As you can see syntax-rules renames five to five119 (or something completely different). So when you try to use five it is unbound.

In Chicken I would use explicit renaming macros which preserve hygiene but allow you to break it where necessary. I’m not sure if Racket has this option (check into an implementation of syntax-case, that will allow you to do something similar).

(define-syntax (let-five x r c)
  (let ((%let (r 'let)))
    `(,%let ((five 5)) ,@(cdr x))))

(let-five (+ five 2))
;===> 7

***** EDIT *****

As I already said I don’t think Racket has explicit renaming macros so this probably won’t work for you in that case. You need to look into an implementation of syntax-case for Racket. This will provide you with the ability to do what you want. To the best of my knowledge this is not possible with syntax-rules.

That being said here’s the explanation you asked for…

x, r and c are parameter names the same as if you defined a function (lambda (x r c) …).

The x parameter is the expression, which would be (let-five (+ five 2)) in your example.

The r parameter is a renaming procedure. This is the “explicit renaming” part of the explicit renaming macro. To maintain hygiene you must use this procedure to rename every symbol that you use (except for the ones you don’t want renamed). You do this by calling (r ‘let) for example.

The c parameter is a comparison function that can compare renamed symbols for equality.

You do not provide any of those parameters. Rather the implementation of ER macros does that for you. All you need to do is provide a function that accepts 3 parameters. Another equivalent way to write the ER macro above would be this…

(define-syntax let-five
  (lambda (expression rename compare)
    (let ((%let (r 'let)))
      `(,%let ((five 5)) ,@(cdr x))))

The % sign is just an ordinary character no different from the #l #e or #t. The reason I use that is to remind the reader that “%let” is the renamed symbol “let”. You could use (renamed-let (r ‘let)) if you want instead of (%let (r ‘let)).

The @ is the syntax for unquote-splicing. If you look at the last line in the ER macro above you’ll notice three forms of symbols (` , @).

` the backquote is syntax for (quasiquote (expression)). quasiquote is similar to the regular quote, except that instead of simply returning the whole expression unevaluated, it allows you to evaluate some things using the (,) syntax.

, the comma is syntax for (unquote expression) this tells Scheme to evaluate the expression that immediately follows it. So if we had the list `(1 (+ 1 1) ,(+ 1 2)) when it is evaluated it would return the list (1 (+ 1 1) 3). The expression (+ 1 1) was not evaluated because it was never unquoted, however the expression (+ 1 2) was unquoted and so it was evaluated and returned 3.

@ is syntax for splicing which is a way to evaluate an expression that returns a list and then put that list into the quasiquoted list. In other words it weaves a list into the quasiquoted list.

Here is some more information on Scheme macro systems. I read each of these articles at least a dozen times and eventually it started to come together. Even now though, I’d say that I only really understand about 10-20% of it.

Scheme macros are difficult to understand, but coming to that understanding (even if it is limited) is incredibly rewarding. It will definitely make you a better Scheme programmer and will probably help you to realize why the vast majority of other languages are simply inadequate.

This is the easiest read and the highest level overview.
A Scheme Syntax Rules Primer

This is probably the most complete approach to macros from the relatively simple to the complex. Read this until you don’t understand what it’s talking about, then play around with some of the things you learned. Then read it again from the beginning until you don’t know what it’s talking about, then play around. Then read it again… Eventually you’ll make it at least half way through and by then you’ll know more about macros than you ever thought possible.
define-syntax Primer

Here’s an interesting (and long) post on Chicken’s macro system. This covers ER macros.
Macro Systems and Chicken (long)

And finally here’s an advanced discussion of macros that may just leave you without hair.
An Advanced Syntax-Rules Primer for the Mildly Insane

So what was the original purpose of Lispy? I’ll give you the tag line that I wrote when Lispy was nothing but an idea… Lispy is a distributed self-optimizing program by example language.

What does that mean? When I first became interested in programming I had 2 goals; to create a program like MetaStock (I’ve since written pyTrade) and to create Artificial Intelligence. Modest goals, I know. In my AI research I came across genetic programming and immediately took a liking to it, probably because I was still new to writing code and the idea that I could write code that wrote code fascinated me.

The main problem with genetic programming is that it is often difficult to write fitness tests for your problems. Somewhere along the way I noticed that programmers were writing fitness tests all the time in the form of unit tests for their code. In addition, the fact that 99% of the time the CPU sits idle while we browse reddit, kept rattling around in my head. Why not take advantage of those wasted cycles by using a programming language that used those cycles to optimize the code you just wrote?

Then I had another idea. Why write code at all? Why not just write the unit tests and let the code evolve on its own? It took me about an hour to realize that this wasn’t going to work on anything but the smallest applications. However another few weeks and I realized that, maybe it could work, given enough computers were running Lispy.

Anyway I have about 25 half-finished papers on Evo which I might get around to finishing and posting here. I’ll give the highlights and post a link to the github account where you can find more info. Evo is meant to be integrated with Lispy (though I don’t know when I’ll get around to that).

Evo is an evolutionary search based function optimizer. It provides an interface for defining new modules and functions as well as an evolutionary programming based method for optimizing those functions. Evo does not use the standard generational approach to GP, rather it uses “gene pools”, that contain functions of indefinite life, from which new functions can be evolved and tested one at a time.

When defining an Evo function you can choose to optimize it for speed, space or code length. Each function can have multiple gene pools that are each optimized for a different purpose. Using gene pools instead of a single population generational approach helps to avoid stagnation within the population as is a common occurrence with the standard generational model.

Evo is very much unfinished, however it can successfully run on its own and evolve solutions to problems. Because of this I am going to put the code out there in case anyone wishes to contribute. I used the implementation of Evo as an excuse to firm up my understanding of closures, as a result the majority of the program is written in a slightly OOP fashion. There is also a Tk based GUI for browsing modules and adding new functions and fitness tests, though its use is completely optional.

Without further ado, here’s the link to the Evo repo.

Command Line or REPL

chicken-doc
Explore Chicken documentation from the command line or in the REPL

grep
The documentation for Chicken is bundled with the source.  You may use a tool such as grep to search these files.

Online

Chicken User’s Manual
The “Official Chicken User’s Manual”.  Hosted on the Chicken Wiki.

Chickadee
Search Chicken documentation by keyword or function name.

Egg Index
A list of eggs available for Chicken.  Many have documentation available when you click on their name.

Paste
The Chicken paste service provides a method of sharing code snippets.

Chicken Trac
Bug tracker for Chicken, also has a source code browser.

Mailing Lists and Forums

chicken-users Mailing List
chicken-users Archive
Mailing list for users of Chicken.

chicken-hackers Mailing List
chicken-hackers Archive
Mailing list for developers working on Chicken itself.

Chat

IRC on Freenode:  Channel #Chicken
Chicken’s chat channel

Blogs

Drinkable Chicken
Though not focusing as much on Chicken anymore, Drinkable Chicken is host to a bunch of great articles for someone new to Scheme.  Many of the articles compare Chicken to Python code snippets.

Schemely Blog
A few articles about Chicken and Scheme in general.

The Adventures of a Pythonista in Schemeland
A good series for an introduction to Scheme from a Python background.

Chicken for Blub Programmers

Python

Pointless Programming Language Reference
Pointless Programming’s language comparison database has comparisons between many languages including Chicken Scheme.

Drinkable Chicken
Though not focusing as much on Chicken anymore, Drinkable Chicken is host to a bunch of great articles for someone new to Scheme.  Many of the articles compare Chicken to Python code snippets.

Chicken Wiki – Chicken-for-Python-Programmers
A quick overview of common operations and their solutions in Python and Chicken.

The Adventures of a Pythonista in Schemeland
A good series for an introduction to Scheme from a Python background.

Other Languages

Chicken Wiki – Chicken-for-Ruby-Programmers
A through overview of the concepts behind Scheme from the prospective of a Ruby programmer.

Chicken Wiki – Chicken-for-C-Programmers
A very brief overview of Chicken and the equivalent programs in C.

Chicken Wiki – Chicken-for-PHP-Programmers
A very brief overview of Chicken and the equivalent programs in PHP.

Online Chicken Source Browsing

Chicken Source Code
This page provides links to releases, repos and news.

Chicken core git repository
Access and browse Chicken source online.

Egg Subversion Repository
Access and browse Egg sources online.  This link goes directly to eggs compatible with Chicken version 4.x or higher.

Chicken Trac
Bug tracker for Chicken, also has a source code browser.

General Scheme Documentation

SRFIs
A list of SRFIs (Scheme Request For Implementation) and their documentation.  Chicken makes use of a number of these, however you should check other Chicken-based sources first as they may be implemented differently.

Structure and Interpretation of Computer Programs
SICP is the canonical book on computer science. It can be a difficult read depending upon your experience, but this book will teach you a lot about Scheme and computer science in general.

And you can read the whole book online here.