We are finally going to implement lambda and procedures.

To begin implementing procedures we have to define a procedure type, just like we did with primitives. A procedure has 3 basic parts. A list of parameters, a body and an environment. So we’ll set up a new procedure type with these fields and call it Procedure.

scheme_types.py

class Symbol(str):
  """A symbol is an immutable string, but must be its own type."""
  pass

class The_Empty_List():
  """The empty list is used as a list terminator and should be a singleton"""
  def __repr__(self):
    return "()"

the_empty_list = The_Empty_List()

class Pair(object):
  """A pair is a classic Lisp cons cell used to implement lists"""
  def __init__(self, car, cdr):
    self.car = car
    self.cdr = cdr
  def __iter__(self):
    x = self
    while not isinstance(x, The_Empty_List):
      yield x.car
      x = x.cdr
  def __repr__(self):
    return "(" + ' '.join([str(i) for i in list(self)]) + ")"

class Primitive(object):
  def __init__(self, fn):
    self.fn = fn

class Procedure(object):
  def __init__(self, parameters, body, environment):
    self.parameters = parameters
    self.body = body
    self.environment = environment

Like primitives, procedures are handled by the scheme_apply procedure. We’ll temporarily add a placeholder in scheme_apply for procedures. In addition we need new syntax to create a procedure. The syntax for procedures in Scheme is (lambda (parameters) body). Most Schemes allow the syntax (define (name params …) body …) but this is really just shorthand for (define name (lambda (params …) body …)). For now we won’t overload the define keyword, choosing instead to define functions the more verbose way.

It is important to take note of this before moving on. In many programming languages defining a function is a single step. However logically, there are 2 different operations that take place. First a function object is created, then that function object is bound to a symbol. By requiring the (define name (lambda …)) syntax we are making these two steps explicit.

scheme_eval.py

from scheme_types import Symbol, Pair, Primitive, the_empty_list, Procedure
from buffered_input import Buff
from scheme_read import scheme_read

##############################################################################
## Environments
##############################################################################

def current_environment(env):
  """Return the frame of the current environment"""
  return env.car

def enclosing_environment(env):
  """Return the environment stack with the first frame removed"""
  return env.cdr

def extend_environment(bindings, base_environment):
  """Push a new frame onto the given base_environment"""
  return Pair(bindings, base_environment)

global_environment = extend_environment({}, the_empty_list)
special_forms = {}

def set_symbol(symbol, val, env):
  """Set the binding (symbol, val) in the current frame of env"""
  current_environment(env)[symbol] = val

def lookup_symbol_value(symbol, environment):
  """Return the value of symbol or Unbound Symbol error"""
  env = environment
  while env != the_empty_list:
    if symbol in current_environment(env):
      return current_environment(env)[symbol]
    else:
      env = enclosing_environment(env)
  return "Error: Unbound symbol: " + symbol

##############################################################################
## Eval and Apply
##############################################################################

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr, env):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr, env)
  elif type(expr) is Pair:
    if expr.car in special_forms:
      return special_forms[expr.car](expr.cdr, env)
    else:
      return scheme_apply(scheme_eval(expr.car, env), [scheme_eval(a, env) for a in expr.cdr])
  else:
    return "scheme_eval: not implemented"

def scheme_apply(proc, args):
  if type(proc) is Primitive:
    return apply(proc.fn, args)
  elif type(proc) is Procedure:
    return "Attempted to apply a procedure"
  else:
    return "Error: Undefined procedure"

##############################################################################
## Builtin Syntax
##############################################################################

def special_form_handler(expr, env):
  """Register a symbol with a Python function named "f" that implements a special form"""
  exec(expr.cdr.car)
  special_forms[expr.car] = f

def load(expr):
  """Given a filename, open it and eval each expression in the global_environment"""
  f = open(expr.car, 'r')
  b = Buff(f)
  while b.peek():
    scheme_eval(scheme_read(b), global_environment)
    b.remove_whitespace()
  f.close()

special_forms['scheme-syntax'] = special_form_handler
special_forms['load'] = load

syntax.scm

(scheme-syntax define-primitive "
def f(expr, env):
  set_symbol(expr.car, Primitive(eval(expr.cdr.car)), env)
")

(scheme-syntax define "
def f(expr, env):
  set_symbol(expr.car, scheme_eval(expr.cdr.car, env), env)
")

(scheme-syntax if "
def f(expr, env):
  if scheme_eval(expr.car, env):
    return scheme_eval(expr.cdr.car, env)
  else:
    return scheme_eval(expr.cdr.cdr.car, env)
")

(scheme-syntax lambda "
def f(expr, env):
  return Procedure(expr.car, expr.cdr.car, env)
")

(define a (lambda (x) x))
a
;===> <scheme_types.Procedure object at 0xb733b10c>
(a 42)
;===> Attempted to apply a procedure

Now that we have a basic outline for what our procedures will look like, we can focus on making them work!

To apply a procedure we have to evaluate the body. In the example above the body of the procedure is just x. However we can’t just evaluate x because x is not bound to anything yet.

First we need to create a new environment and bind the parameters to the arguments supplied in the procedure call. In this case we have to bind x to the value 42. Then we just evaluate each expression in the body within the new environment and return the result of evaluating the last expression.

from scheme_types import Symbol, Pair, Primitive, the_empty_list, Procedure
from buffered_input import Buff
from scheme_read import scheme_read

##############################################################################
## Environments
##############################################################################

def current_environment(env):
  """Return the frame of the current environment"""
  return env.car

def enclosing_environment(env):
  """Return the environment stack with the first frame removed"""
  return env.cdr

def extend_environment(bindings, base_environment):
  """Push a new frame onto the given base_environment"""
  return Pair(bindings, base_environment)

global_environment = extend_environment({}, the_empty_list)
special_forms = {}

def set_symbol(symbol, val, env):
  """Set the binding (symbol, val) in the current frame of env"""
  current_environment(env)[symbol] = val

def lookup_symbol_value(symbol, environment):
  """Return the value of symbol or Unbound Symbol error"""
  env = environment
  while env != the_empty_list:
    if symbol in current_environment(env):
      return current_environment(env)[symbol]
    else:
      env = enclosing_environment(env)
  return "Error: Unbound symbol: " + symbol

##############################################################################
## Eval and Apply
##############################################################################

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr, env):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr, env)
  elif type(expr) is Pair:
    if expr.car in special_forms:
      return special_forms[expr.car](expr.cdr, env)
    else:
      return scheme_apply(scheme_eval(expr.car, env), [scheme_eval(a, env) for a in expr.cdr])
  else:
    return "scheme_eval: not implemented"

def scheme_apply(proc, args):
  if type(proc) is Primitive:
    return apply(proc.fn, args)
  elif type(proc) is Procedure:
    new_env = extend_environment(dict(zip(proc.parameters, args)), proc.environment)
    return [scheme_eval(e,new_env) for e in proc.body][-1]
  else:
    return "Error: Undefined procedure"

##############################################################################
## Builtin Syntax
##############################################################################

def special_form_handler(expr, env):
  """Register a symbol with a Python function named "f" that implements a special form"""
  exec(expr.cdr.car)
  special_forms[expr.car] = f

def load(expr):
  """Given a filename, open it and eval each expression in the global_environment"""
  f = open(expr.car, 'r')
  b = Buff(f)
  while b.peek():
    scheme_eval(scheme_read(b), global_environment)
    b.remove_whitespace()
  f.close()

special_forms['scheme-syntax'] = special_form_handler
special_forms['load'] = load

(define test (pred conseq alt)
  (if pred conseq alt))
(test 1 2 3)
;===> 2
(define-primitive + "lambda x,y: x+y")
(define-primitive < "lambda x,y: x<y")
(define a (lambda (x y) (if (< x y) (a (+ x 1) y) x)))
(a 1 3)
;===> 3

With that, we have implemented lambda in Scheme! However there are some glaring omissions from our implementation. For example ((lambda () 42)) will result in an iteration over non-sequence error from trying to iterate over the_empty_list. Also, tail-call elimination is not implemented as required by the Scheme standard.

Also note that there is very little error-checking going on here. For instance calling a procedure with the wrong number of arguments results in the whole interpreter crashing. This would not be good in a production language. However including error checking in this code would probably triple its size at least. The point of this exercise is to learn the concepts, not write a bullet-proof language.

I will likely come back to this at some point in the future and address some of these concerns. The rest is left for you to experiment with. You can get more information on how and what to implement from R5RS.

I will be Jumping back into Javascript for the next few posts at least, so this series will not be updated for a while (with the exception of bug fixes). When I do get back to language design and implementation, I will probably pick back up with the Scheme version of this interpreter, or possibly with a new project on compiling.

If you have any questions about how to proceed from here, or just want to show off your own version, please leave a comment.

Checkout this version on GitHub (v0.10)

GOTO | Table of Contents

We’ve been dealing with an environment ever since part 5 in this series when we implemented define, except we’ve been calling it a frame. Until now our frame has served us well, it has provided a simple environment where we could set and retrieve the value of symbols. However implementing procedures requires us to add a few features to our frame. Consider the code below.

(define x 5)
(define (func x)
  x)
(func 100)
;===> 100

With our frame this wouldn’t work as expected. Because we have already defined x in the global environment, there will be one of two problems with the x in func. Our function could use the global value of x (5) instead of the supplied value (100) or when the supplied value (100) is mapped onto the function body it will overwrite the value of the global value of x (5) with the supplied value (100).

Currently we do not make any distinction between variables in a function body and in the global environment. To properly implement procedures we will have to figure out a way to do that. Consider the next bit of code.

(define x 5)
(define (func y)
  (+ x y))
(func 10)
;===> 15

Even though we need to make a distinction between environments they also need to be connected. In the example above we use the global variable x in func without ever declaring it. This is accomplished by a method called chaining environments.

(define x 1)
(define (outer-func y)
  (define (inner-func z)
    (+ x y z)
  (inner-func 3))
(outer-func 2)
;===> 6

The environments produced by the above code are best described by the diagram below.

So when x appears in inner-func we have to check inner-func’s environment for x, then outer-func’s environment for x and finally the global environment for x before we raise an undefined symbol error.

You may have noticed that the diagram above looks a lot like a linked list. We can take our frame and combine it with linked lists to form an environment. It is more informative (and closer to the actual design) to view environments from the point of view of eval. This diagram is a view of the environment structure when the code (+ x y z) is evaluated.

I used the names current-environment to refer to the inner-func environment and enclosing-environment to refer to the outer-func environment. We will use these same names in our source code. Note that the global-environment is just the enclosing-environment of outer-func, the only thing special about it is that its enclosing-environment is the empty list.

In addition we need to replace our global frame with global_environment, which we’ll do by providing a procedure extend_environment, and change lookup_symbol_value to check enclosing-environments if the symbol is not found.

scheme_eval.py

from scheme_types import Symbol, Pair, Primitive, the_empty_list
from buffered_input import Buff
from scheme_read import scheme_read

def current_environment(env):
  """Return the frame of the current environment"""
  return env.car

def enclosing_environment(env):
  """Return the environment stack with the first frame removed"""
  return env.cdr

def extend_environment(bindings, base_environment):
  """Push a new frame onto the given base_environment"""
  return Pair(bindings, base_environment)

global_environment = extend_environment({}, the_empty_list)
special_forms = {}

def set_symbol(symbol, val, env):
  """Set the binding (symbol, val) in the current frame of env"""
  current_environment(env)[symbol] = val

def lookup_symbol_value(symbol, environment):
  """Return the value of symbol or Unbound Symbol error"""
  env = environment
  while env != the_empty_list:
    if symbol in current_environment(env):
      return current_environment(env)[symbol]
    else:
      env = enclosing_environment(env)
  return "Error: Unbound symbol: " + symbol

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr, global_environment)
  elif type(expr) is Pair:
    if expr.car in special_forms:
      return special_forms[expr.car](expr.cdr)
    else:
      return scheme_apply(scheme_eval(expr.car), [scheme_eval(a) for a in expr.cdr])
  else:
    return "scheme_eval: not implemented"

def scheme_apply(proc, args):
  if type(proc) is Primitive:
    return apply(proc.fn, args)
  else:
    return "Error: Undefined procedure"

def special_form_handler(expr):
  """Register a symbol with a Python function named "f" that implements a special form"""
  exec(expr.cdr.car)
  special_forms[expr.car] = f

def load(expr):
  """Given a filename, open it and eval each expression in the global_environment"""
  f = open(expr.car, 'r')
  b = Buff(f)
  while b.peek():
    scheme_eval(scheme_read(b))
    b.remove_whitespace()
  f.close()

special_forms['scheme-syntax'] = special_form_handler
special_forms['load'] = load

We also need to make some changes to syntax.scm to use the new set_symbol function.

syntax.scm

(scheme-syntax define-primitive "
def f(expr):
  set_symbol(expr.car, Primitive(eval(expr.cdr.car)), global_environment)
")

(scheme-syntax define "
def f(expr):
  set_symbol(expr.car, scheme_eval(expr.cdr.car), global_environment)
")

(scheme-syntax if "
def f(expr):
  if scheme_eval(expr.car):
    return scheme_eval(expr.cdr.car)
  else:
    return scheme_eval(expr.cdr.cdr.car)
")

(define a 5)
;===> #<unspecified>
a
;===> 5

Before we go any further I’m going to add in some comments to the scheme_eval.py file, mainly to group code into logical sections. I won’t provide a highlighted diff here, but you will notice the sections in the next bit of code.

To use the full potential of our new environments we are going to have to rewrite our program a little bit to pass them around correctly. Since eval is the main method of dispatch for our interpreter we’ll supply the environment as an argument to eval, which means finding every use of scheme_eval and adding the env argument. We’ll also have to pass an environment variable to procedures like set_symbol and lookup_symbol_value. One final place we’ll need environments is in scheme-syntax, we’ll have to change the stored procedures there to use “def f(expr, env):” instead of just “def f(expr):”.

from scheme_types import Symbol, Pair, Primitive, the_empty_list
from buffered_input import Buff
from scheme_read import scheme_read

##############################################################################
## Environments
##############################################################################

def current_environment(env):
  """Return the frame of the current environment"""
  return env.car

def enclosing_environment(env):
  """Return the environment stack with the first frame removed"""
  return env.cdr

def extend_environment(bindings, base_environment):
  """Push a new frame onto the given base_environment"""
  return Pair(bindings, base_environment)

global_environment = extend_environment({}, the_empty_list)
special_forms = {}

def set_symbol(symbol, val, env):
  """Set the binding (symbol, val) in the current frame of env"""
  current_environment(env)[symbol] = val

def lookup_symbol_value(symbol, environment):
  """Return the value of symbol or Unbound Symbol error"""
  env = environment
  while env != the_empty_list:
    if symbol in current_environment(env):
      return current_environment(env)[symbol]
    else:
      env = enclosing_environment(env)
  return "Error: Unbound symbol: " + symbol

##############################################################################
## Eval and Apply
##############################################################################

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr, env):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr, env)
  elif type(expr) is Pair:
    if expr.car in special_forms:
      return special_forms[expr.car](expr.cdr, env)
    else:
      return scheme_apply(scheme_eval(expr.car, env), [scheme_eval(a, env) for a in expr.cdr])
  else:
    return "scheme_eval: not implemented"

def scheme_apply(proc, args):
  if type(proc) is Primitive:
    return apply(proc.fn, args)
  else:
    return "Error: Undefined procedure"

##############################################################################
## Builtin Syntax
##############################################################################

def special_form_handler(expr, env):
  """Register a symbol with a Python function named "f" that implements a special form"""
  exec(expr.cdr.car)
  special_forms[expr.car] = f

def load(expr):
  """Given a filename, open it and eval each expression in the global_environment"""
  f = open(expr.car, 'r')
  b = Buff(f)
  while b.peek():
    scheme_eval(scheme_read(b), global_environment)
    b.remove_whitespace()
  f.close()

special_forms['scheme-syntax'] = special_form_handler
special_forms['load'] = load

repl.py

import sys
from scheme_read import scheme_read
from scheme_eval import scheme_eval, special_forms, global_environment
from scheme_types import Pair
from buffered_input import Buff

special_forms['load'](Pair("syntax.scm", None))

while True:
  inp = scheme_eval(scheme_read(Buff(sys.stdin)), global_environment)
  if inp != None:
    print ';===>', inp

syntax.chicken

(scheme-syntax define-primitive "
def f(expr, env):
  set_symbol(expr.car, Primitive(eval(expr.cdr.car)), env)
")

(scheme-syntax define "
def f(expr, env):
  set_symbol(expr.car, scheme_eval(expr.cdr.car, env), env)
")

(scheme-syntax if "
def f(expr, env):
  if scheme_eval(expr.car, env):
    return scheme_eval(expr.cdr.car, env)
  else:
    return scheme_eval(expr.cdr.cdr.car, env)
")

(define a 5)
a
;===> 5
(if 1 2 3)
;===> 2
(if #f 1 2)
;===> 2
(define-primitive + "lambda x,y: x+y")
(+ 3 5)
;===> 8

If you still don’t fully understand environments, that’s OK. You’ll get a chance to play around with them in the next part where we’ll finally cover implementing procedures in Lispy!

Checkout this version on GitHub (v0.09)

GOTO Part 10 | lambda

GOTO | Table of Contents

The goal for this part is to implement primitive procedures, apply and a method of setting new primitives.

The first thing we need to do is figure out a way to represent our primitive procedures. SICP uses tagged lists, which are just regular lists with the first element designated as a tag. Python uses objects to define new types and since we’ve already been doing that, we’ll continue here.

class Symbol(str):
  """A symbol is an immutable string, but must be its own type."""
  pass

class The_Empty_List():
  """The empty list is used as a list terminator and should be a singleton"""
  def __repr__(self):
    return "()"

the_empty_list = The_Empty_List()

class Pair(object):
  """A pair is a classic Lisp cons cell used to implement lists"""
  def __init__(self, car, cdr):
    self.car = car
    self.cdr = cdr
  def __iter__(self):
    x = self
    while not isinstance(x, The_Empty_List):
      yield x.car
      x = x.cdr
  def __repr__(self):
    return "(" + ' '.join([str(i) for i in list(self)]) + ")"

class Primitive(object):
  def __init__(self, fn):
    self.fn = fn

If the symbol of an expression of the form (symbol args …) is not found in special_forms it must be a procedure application (or not exist). To apply a primitive procedure first we have to evaluate all the arguments using scheme_eval. Then we use that list as the args value to apply(primitive, args). The primitive returns a Scheme value and we continue on.

For now we’re going to hardcode one primitive + into our interpreter. We’ll come up with something a little more elegant next.

from scheme_types import Symbol, Pair, Primitive
from buffered_input import Buff
from scheme_read import scheme_read

frame = {'+':Primitive(lambda x,y: x+y)}
special_forms = {}

def lookup_symbol_value(symbol):
  try:
    return frame[symbol]
  except KeyError:
    return "Error: Unbound variable"

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr)
  elif type(expr) is Pair:
    if expr.car in special_forms:
      return special_forms[expr.car](expr.cdr)
    else:
      return scheme_apply(scheme_eval(expr.car), [scheme_eval(a) for a in expr.cdr])
  else:
    return "scheme_eval: not implemented"

def scheme_apply(proc, args):
  if type(proc) is Primitive:
    return apply(proc.fn, args)
  else:
    return "Error: Undefined procedure"

def special_form_handler(expr):
  """Register a symbol with a Python function named "f" that implements a special form"""
  exec(expr.cdr.car)
  special_forms[expr.car] = f

def load(expr):
  """Given a filename, open it and eval each expression in the global_environment"""
  f = open(expr.car, 'r')
  b = Buff(f)
  while b.peek():
    scheme_eval(scheme_read(b))
    b.remove_whitespace()
  f.close()

special_forms['scheme-syntax'] = special_form_handler
special_forms['load'] = load

(+ 3 4)
;===> 7
(- 3 4)
;===> Error: Undefined procedure

The next step is to allow defining primitive procedures from within Scheme, as we did with special forms. We’ll implement define-primitive as a scheme-syntax macro. First remove the line that sets the + symbol. We’re only going to be modifying the syntax.scm file for now so that’s all I’ll show you.

We need to create a scheme-syntax macro that makes a Primitive object and sets it as the value of a symbol. This macro is a lot like the define macro that we wrote earlier. The only difference is instead of calling scheme_eval on the cadr we call eval and pass that result to make-primitive.

syntax.scm

(scheme-syntax define-primitive "
def f(expr):
  frame[expr.car] = Primitive(eval(expr.cdr.car))
")

(scheme-syntax define "
def f(expr):
  frame[expr.car] = scheme_eval(expr.cdr.car)
")

(scheme-syntax if "
def f(expr):
  if scheme_eval(expr.car):
    return scheme_eval(expr.cdr.car)
  else:
    return scheme_eval(expr.cdr.cdr.car)
")

With that we can define all sorts of primitives for our language…

(define-primitive + "lambda x,y: x+y")
(+ 3 4)
;===> 7
(- 3 4)
;===> Error: Undefined procedure

(define-primitive - "lambda x,y: x-y")
(- 3 4)
;===> -1

(define-primitive square "lambda x: x*x")
(square 4)
;===> 16

(define-primitive len "len")
(len "hello")
;===> 5

You can start a new file and define a bunch of primitives, load it the same way you load syntax.scm. I’m not going to define any primitives just yet but you are more than welcome to. Instead in the next post we’ll implement environments and then it’s on to implementing lambda!

Checkout this version on GitHub (v0.08)

GOTO Part 9 | Environments

GOTO | Table of Contents

In this part we’re going to be doing a bit of refactoring, so let’s start with that and a recap. The first thing that we’re going to do is remove define and if from our language. We’ll put them back in later, but first let’s look at what our language looks like with only scheme-syntax defined.

from scheme_types import Symbol, Pair

frame = {}
special_forms = {}

def lookup_symbol_value(symbol):
  try:
    return frame[symbol]
  except KeyError:
    return "Error: Unbound variable"

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr)
  elif type(expr) is Pair:
    if expr.car in special_forms:
      return special_forms[expr.car](expr.cdr)
    else:
      return "Error: unbound special form"
  else:
    return "scheme_eval: not implemented"

def special_form_handler(expr):
  """Register a symbol with a Python function named "f" that implements a special form"""
  exec(expr.cdr.car)
  special_forms[expr.car] = f

special_forms['scheme-syntax'] = special_form_handler

Let’s take a moment to consider what we can do so far. We now have a language that can understand self-evaluating Scheme values and lookup symbol values. In addition to that it has exactly one primitive, a macro that defines other primitives by giving us access to Python.

You may consider this language lacking, and I admit it does not have many features. However it is now possible to define any language construct (including procedures, eval and apply) from within Scheme with the help of scheme-syntax and Python underneath. After writing this code we could define the rest of the language 100% from within Scheme (though it may not be easy).

However there are a few more things that conceptually belong in the core language. One of those things is the ability to load a Scheme file. This will let us break out our syntax definitions into another file and load it automatically before we start the repl.

from scheme_types import Symbol, Pair
from buffered_input import Buff
from scheme_read import scheme_read

frame = {}
special_forms = {}

def lookup_symbol_value(symbol):
  try:
    return frame[symbol]
  except KeyError:
    return "Error: Unbound variable"

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr)
  elif type(expr) is Pair:
    if expr.car in special_forms:
      return special_forms[expr.car](expr.cdr)
    else:
      return "Error: unbound special form"
  else:
    return "scheme_eval: not implemented"

def special_form_handler(expr):
  """Register a symbol with a Python function named "f" that implements a special form"""
  exec(expr.cdr.car)
  special_forms[expr.car] = f

def load(expr):
  """Given a filename, open it and eval each expression in the global_environment"""
  f = open(expr.car, 'r')
  b = Buff(f)
  while b.peek():
    scheme_eval(scheme_read(b))
    b.remove_whitespace()
  f.close()

special_forms['scheme-syntax'] = special_form_handler
special_forms['load'] = load

Now that we have load, we need something to load. Create a new file called syntax.scm, or whatever you prefer. This is where we will define our primitive syntax forms like define, if and lambda. For this moment we’re going to leave it blank, and make the repl load it by default so we don’t have to do it manually each time.

Also I’m going to make some changes to the repl to make our sessions look more like the examples I’ve been giving (so ;===> will come before the output and there will be no prompt for input). Later I may go back to the beginning of this series and change this, as this is how it should have been from the start. I’ll also take this opportunity to skip printing any None value which will make using expressions that don’t return values more aesthetically pleasing.

repl.py

import sys
from scheme_read import scheme_read
from scheme_eval import scheme_eval, special_forms
from scheme_types import Pair
from buffered_input import Buff

special_forms['load'](Pair("syntax.scm", None))

while True:
  inp = scheme_read(Buff(sys.stdin))
  if inp != None:
    print ';===>', scheme_eval(inp)

Now we can define some special forms in syntax.scm to load by default. For now we’ll just reimplement define and if.

syntax.scm

(scheme-syntax define "
def f(expr):
  special_forms[expr.car] = scheme_eval(expr.cdr.car)
")

(scheme-syntax if "
def f(expr):
  if scheme_eval(expr.car):
    return scheme_eval(expr.cdr.car)
  else:
    return scheme_eval(expr.cdr.cdr.car)
")

(define a 5)
a
;===> 5
(if 1 2 3)
;===> 2

Now that we have implemented load we are one step closer to having a platform that we can build almost any language we want with. To change languages all we would have to do is load a different syntax file.

I encourage you to play around with that concept for a little while. Write two different syntax.scm files and implement a couple of basic forms like define, or and if. Try to make your two implementations as different as possible even if making it different doesn’t make sense. For instance in one implementation if could take a mandatory keyword before the alternate branch (if pred consequent else alt) or it could act like and, taking two predicates and only evaluating the consequent branch if both predicates evaluate to true.

This article is a little short on code because we made some larger conceptual changes that I did not want to breeze past. In the next article we’ll implement primitive procedures and apply. The ability to define procedures from within Scheme is just around the corner.

Checkout this version on GitHub (v0.07)

GOTO Part 8 | Primitive procedures and apply

GOTO | Table of Contents

In the last part we implemented our first special form and the first expression that was not self-evaluating, define. The goal for this part is to implement (if predicate consequent alternate).

We’re going to start by implementing if the same way we implemented define. The first thing to do is test if the car of the pair equals if, then call a helper procedure we’ll name apply_if.

We know that if will have to evaluate the predicate argument to determine whether to evaluate the consequent or alternate arguments. If the predicate is true we will evaluate the consequent expression and not the alternate. If the predicate is false we will evaluate the alternate expression and not the consequent.

from scheme_types import Symbol, Pair

frame = {}

def lookup_symbol_value(symbol):
  try:
    return frame[symbol]
  except KeyError:
    return "Error: Unbound variable"

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr)
  elif type(expr) is Pair:
    if expr.car == "define":
      frame[expr.cdr.car] = scheme_eval(expr.cdr.cdr.car)
      return "set symbol -> value"
    elif expr.car == "if":
      return apply_if(expr.cdr.car, expr.cdr.cdr.car, expr.cdr.cdr.cdr.car)
    else:
      return "scheme_eval: not implemented"
  else:
    return "scheme_eval: not implemented"

def apply_if(predicate, consequent, alternate):
  if scheme_eval(predicate):
    return scheme_eval(consequent)
  else:
    return scheme_eval(alternate)

(if 1 2 3)
;===> 2
(if #f 2 3)
;===> 3

Dispatching special forms using if is fine while you have a small number of primitive syntax forms. However once you implement a large number of primitive syntax forms this method of dispatch becomes cumbersome. Also note that it is not possible to extend the primitive syntax forms except by modifying the original source by hand.

Reusing the Frame

On top of that we have already implemented a mechanism of associating and storing symbol and value pairs. Let’s try something like that first, we’ll start by making another global dictionary called special_forms. Then we’ll modify eval to check this dictionary instead of using if/elif.

from scheme_types import Symbol, Pair

frame = {}
special_forms = {}

def lookup_symbol_value(symbol):
  try:
    return frame[symbol]
  except KeyError:
    return "Error: Unbound variable"

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr)
  elif type(expr) is Pair:
    if expr.car in special_forms:
      return special_forms[expr.car](expr.cdr)
    else:
      return "Error: unbound special form"
  else:
    return "scheme_eval: not implemented"

def define(expr):
  frame[expr.car] = scheme_eval(expr.cdr.car)
  return "set symbol -> value"

def apply_if(expr):
  if scheme_eval(expr.car):
    return scheme_eval(expr.cdr.car)
  else:
    return scheme_eval(expr.cdr.cdr.car)

special_forms['define'] = define
special_forms['if'] = apply_if

(repl)
(define a 5)
;===> #<unspecified>
a
;===> 5
(if 1 2 3)
;===> 2
(hello)
;===> Not implemented

With that addition we are finished with the first level of the eval function. An expression can only be one of three things: self-evaluating, a symbol or an application (either syntax or procedure which we’ll cover later).

How does it work? special_forms stores (symbol, procedure) pairs where the symbol is a keyword (like define or if) and the procedure is the Python code that implements that special form. Every procedure in special_forms takes a single argument, an unevaluated pair representing the rest of the expression passed to scheme_eval. Because special forms have their own evaluation rules, it is the responsibility of this procedure to evaluate the expression and return a result.

In other words, when lispy-eval comes across a special form it transfers control to a Python procedure located in the special_forms dict. That Python procedure is responsible for evaluating the arguments and returning a Scheme value.

Now that we have a data structure to hold our special_forms it would be a good idea to supply an easy mechanism to set a new special form. For this purpose we’ll design a special form in Scheme that will set new special forms called scheme-syntax.

from scheme_types import Symbol, Pair

frame = {}
special_forms = {}

def lookup_symbol_value(symbol):
  try:
    return frame[symbol]
  except KeyError:
    return "Error: Unbound variable"

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr)
  elif type(expr) is Pair:
    if expr.car in special_forms:
      return special_forms[expr.car](expr.cdr)
    else:
      return "Error: unbound special form"
  else:
    return "scheme_eval: not implemented"

def special_form_handler(expr):
  """Register a symbol with a Python function named "f" that implements a special form"""
  exec(expr.cdr.car)
  special_forms[expr.car] = f

special_forms['scheme-syntax'] = special_form_handler

def define(expr):
  frame[expr.car] = scheme_eval(expr.cdr.car)
  return "set symbol -> value"

def apply_if(expr):
  if scheme_eval(expr.car):
    return scheme_eval(expr.cdr.car)
  else:
    return scheme_eval(expr.cdr.cdr.car)

special_forms['define'] = define
special_forms['if'] = apply_if

(scheme-syntax quote "def f(expr): return expr.car")
;===> None
(1 2 3)
;===> Error: unbound special form
(quote (1 2 3))
;===> (1 2 3)

With the addition of scheme-syntax it is now possible to write Python directly in our interpreter. Because of this we can now implement all of the primitive syntax forms in Scheme. Some might consider this cheating, but eventually we’re going to want a way to interface with Python. Why not implement it right away, use it, test it and make it solid by implementing everything else in it? We’ll still have to make some modifications to the core interpreter to implement procedures and a load form later.

Because of Python’s restrictions on lambda expressions, this solution is not as elegant as the scheme-syntax form in Scheme in Scheme. For this version you must define a named function, and it must be named “f” to make this implementation work. If you know of a more elegant way to approach this please let me know in the comments.

We haven’t even gotten to the mutually recursive yin-yang that is the eval/apply cycle yet. However, thanks to piggy-backing on Python, we already have a somewhat useful language. A language without procedures is a pretty bad one, so we’ll keep pressing on, but check out some of the things that are possible already…

(scheme-syntax print "
def f(expr):
  print scheme_eval(expr.car)
  return None
")
;===> None
(scheme-syntax loop "
def f(expr):
  for n in range(scheme_eval(expr.car)):
    scheme_eval(expr.cdr.car)
")
;===> None
(loop 3 (print "Hi"))
;===> Hi
;===> Hi
;===> Hi
;===> None

In the next article we’ll do a little refactoring, remove define and if from the core of the interpreter and implement a load function so we can remove our syntax definitions out of the main interpreter file.

Checkout this version on GitHub (v0.06)

GOTO Part 7 | Refactor and load

GOTO | Table of Contents

Our goal in this part is to implement assignment via the form (define symbol value). There are three challenges to this goal. First we need to come up with a system to store symbol names and their values. Then we need to display those values when the symbol is eval’d. Finally we need to recognize the define form and set the value of its symbol.

Starting with a Stub

To start with we need to modify our eval procedure to recognize symbols. We’ll call a procedure lookup_symbol_value that will eventually give the value of the symbol. Right now we’ll just have it return a string stating our intentions.

from scheme_types import Symbol

def lookup_symbol_value(symbol):
  return "Here's where we lookup the value of the symbol."

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr)
  else:
    return "scheme_eval: not implemented"

The Global Frame

To set the value of a symbol we need to have somewhere to set it. That somewhere is going to be a dictionary. The dictionary that stores a set of names and their associated values is referred to as a frame.

For now we’ll define our frame as a global variable named frame. Since we don’t have any method of assigning values to symbols yet, we define a symbol “test” when we make the frame so we can see if it works.

from scheme_types import Symbol

frame = {'test':"Value retrieved successfully"}

def lookup_symbol_value(symbol):
  try:
    return frame[symbol]
  except KeyError:
    return "Error: Unbound variable"

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr)
  else:
    return "scheme_eval: not implemented"

test
;===> Value retrieved successfully
hello
;===> Error: Unbound variable

(define symbol value)

Now that we can store and retrieve the value of a symbol, we need a way to do it from within Scheme. Scheme uses the form (define symbol value) to define a new symbol or set the value of an existing symbol.

The form (define symbol value) is a pair (also a list, but all lists are also pairs) so the first step to detecting its usage is to look for pairs. The next step is to check the symbol in the car of that pair (or the first item of the list), if that symbol is “define” we run the code to define a symbol. For now we’ll just return a string stating our intentions.

from scheme_types import Symbol, Pair

frame = {'test':"Value retrieved successfully"}

def lookup_symbol_value(symbol):
  try:
    return frame[symbol]
  except KeyError:
    return "Error: Unbound variable"

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr)
  elif type(expr) is Pair:
    if expr.car == "define":
      return "set the value of the symbol"
    else:
      return "scheme_eval: not implemented"
  else:
    return "scheme_eval: not implemented"

test
;===> Value retrieved successfully
(define a 1)
;===> Set the value of the symbol.
(set a 1)
;===> scheme_eval: not implemented

Setting the Value of Symbols

To actually set the value of the symbol we need to get the arguments to define. Getting the symbol argument is straightforward, however we need to eval the value argument as it might not be self-evaluating.

Usually (define symbol value) will return void, or nothing. I have chosen to return a string here to give some visual feedback that setting the symbol happened as well as avoid having to modify the repl to handle None. You can choose to take the more standard approach and return None here, then modify the repl to skip printing None values.

from scheme_types import Symbol, Pair

frame = {}

def lookup_symbol_value(symbol):
  try:
    return frame[symbol]
  except KeyError:
    return "Error: Unbound variable"

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr)
  elif type(expr) is Pair:
    if expr.car == "define":
      frame[expr.cdr.car] = scheme_eval(expr.cdr.cdr.car)
      return "set symbol -> value"
    else:
      return "scheme_eval: not implemented"
  else:
    return "scheme_eval: not implemented"

a
;===> Error: Unbound variable
(define a 5)
;===> set symbol -> value
a
;===> 5

In addition to being able to set symbols, we now have a glimpse into how basic forms (like define) are handled. It really is no more complicated than checking the procedure name, evaluating the arguments that need to be, then passing the arguments to a Python procedure that implements the desired functionality.

* To any Schemers that may be reading this: I avoided implementing environments here for two reasons. I wanted to show the simplest implementation possible and in an upcoming post we will transform our global frame into a lookup table for primitive syntax.

Checkout this version on GitHub (v0.05)

GOTO Part 6 | scheme-syntax macro

GOTO | Table of Contents

In the last post we finished implementing the read layer. There is a lot of room for improvement, however it will serve our purposes for the moment. With the read layer complete we can move on to the eval layer.

The eval layer is where the Scheme in Scheme series starts and from here on there will be many similarities between these two series. In fact a sizable portion of this post was simply copy/pasted from the Scheme in Scheme series. With the exception of the code and language specific explanation of it, the rest of the series will also be a close replica.

Create a new file called scheme_eval.py, the rest of this series will focus mainly on this file. In it we will define a function scheme_eval. For now we’re simply going to echo the input back without processing it at all.

scheme_eval.py

def scheme_eval(expr):
  return expr

repl.py

import sys
from scheme_read import scheme_read
from scheme_eval import scheme_eval
from buffered_input import Buff

while True:
  print "> ",
  print scheme_eval(scheme_read(Buff(sys.stdin)))

With that we have the basic layout of the interpreter. If you read the last line backward you will see the layers of the interpreter print(eval(read(stdin))) all wrapped up in an infinite loop. From this point we can concentrate on evaluating the expressions that scheme_read returns.

A Language of Numbers

The simplest language is a language of numbers. Numbers are self-evaluating, so if the expression is a number we’ll just return it for now. If the expression is anything else, we’ll return an error string, just as we did in the read layer. If you are following along, you will probably want to add proper errors/error handling to your version.

def scheme_eval(expr):
  if type(expr) is int:
    return expr
  else:
    return "scheme_eval: not implemented"

A Language of Self-Evaluating Values

Continuing with our language of self-evaluating values is just as simple. Booleans, strings and characters are also self-evaluating so we’ll add them now. More complex data structures like hash tables are also self evaluating, however we’ll leave them for later.

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  else:
    return "scheme_eval: not implemented"

The first stage of the eval layer is complete. We now have a functional language of self-evaluating values. Though one could argue exactly how functional it really is. The next step along our path will be to introduce symbols and implement assignment.

Checkout this version on GitHub (v0.04)

GOTO Part 5 | Assignment and define

GOTO | Table of Contents

In the last post we added support for booleans, characters, symbols and strings. There is only one major data type left to add in the read layer, the pair/linked list.

A pair is a structure which contains 2 elements normally referred to as the car and cdr in Scheme. Lists are built on top of pairs where the car is some value and the cdr is another pair or the empty list. In a proper list the cdr of the last pair must be the empty list.

To illustrate this concept a bit more clearly here is a diagram of the list ‘(1 2 3) in Scheme:

This list is constructed out of 3 pairs. The car of the first pair is the value 1 and the cdr is the second pair. The car of the second pair has the value 2 and the cdr is the third pair. The car of the third pair has the value 3 and the cdr contains the empty list (Nil in the diagram).

class Pair:
  def __init__(self, car, cdr):
    self.car = car
    self.cdr = cdr

Above is the class definition for a very simple Pair object in Python. We can construct the above list using the Pair class as follows:

lst = Pair(1, Pair(2, Pair(3, None)))

This implementation of a list is called a linked list and you can find a more detailed description in my article WITH Linked Lists. Python does not have an implementation of a list in the core language, the list data type in Python is called an array in most other languages.

Scheme sourcecode is written using the linked list data type. This is one of the most powerful features of the language as the sourcecode is just a regular Scheme data type. This allows some really interesting features such as macros, but we won’t get into that just yet.

To start with we’ll implement the empty list. Since this is a distinct object in Scheme we’ll make a new type called The_Empty_List and since we only need one empty list we’ll also make a global variable to refer to it called the_empty_list. We’ll do this in the scheme_types.py file.

class Symbol(str):
  """A symbol is an immutable string, but must be its own type."""
  pass

class The_Empty_List():
  """The empty list is used as a list terminator and should be a singleton"""
  def __repr__(self):
    return "()"

the_empty_list = The_Empty_List()

Now that we have an empty list type we need to modify our reader to recognize it. When the “(” character is encountered, scheme_read will call read_pair. Since the only pair we’re going to allow for the moment is the empty list, we just need to check that the next character is “)” and return the_empty_list.

from scheme_types import Symbol, the_empty_list

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def is_initial(c):
  """Is c a valid initial character for an identifier?"""
  return c.isalpha() or c in '-+*/><=?!&'

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if '.' in buf:
    return float(buf)
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def read_expected_string(f, s):
  """Consume expected characters from the input buffer."""
  expected_chars = list(s)
  c = f.getc()
  while expected_chars:
    if c == expected_chars[0]:
      expected_chars.pop(0)
      c = f.getc()
    else:
      return "unexpected character"

def read_character(f):
  c = f.getc()
  if c == 's' and f.peek() == 'p':
    read_expected_string(f, "pace")
    return ' '
  elif c == 'n' and f.peek() == 'e':
    read_expected_string(f, "ewline")
    return '\n'
  else:
    return c

def read_symbol(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  return ''.join(buf)

def read_string(f):
  buf = []
  c = f.getc()
  while c != '\"':
    buf.append(c)
    c = f.getc()
  return ''.join(buf)

def read_pair(f):
  f.remove_whitespace()
  c = f.getc()
  if c == ')':
    return the_empty_list

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == '-' and f.peek().isdigit() or f.peek() == '.')
      or (c == '.' and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  elif c == '#':
    c = f.getc()
    if c == 't':
      return True
    elif c == 'f':
      return False
    elif c == '\\':
      return read_character(f)
    else:
      return "Error boolean value must be #t or #f not #" + c
  elif is_initial(c):
    f.ungetc(c)
    return Symbol(read_symbol(f))
  elif c == '\"':
    return read_string(f)
  elif c == '(':
    return read_pair(f)
  else:
    return "scheme_read: not implemented"

With that done, we’re only two steps away from being able to read linked lists. First we have to create a Pair type that will represent a Scheme pair. To do this, we’ll add the following to the scheme_types.py file.

class Symbol(str):
  """A symbol is an immutable string, but must be its own type."""
  pass

class The_Empty_List():
  """The empty list is used as a list terminator and should be a singleton"""
  def __repr__(self):
    return "()"

the_empty_list = The_Empty_List()

class Pair(object):
  """A pair is a classic Lisp cons cell used to implement lists"""
  def __init__(self, car, cdr):
    self.car = car
    self.cdr = cdr
  def __iter__(self):
    x = self
    while not isinstance(x, The_Empty_List):
      yield x.car
      x = x.cdr
  def __repr__(self):
    return "(" + ' '.join([str(i) for i in list(self)]) + ")"

In addition to a car and cdr field I have also implemented the __iter__ and __repr__ special methods. __iter__ will help us later on when we work with these pairs by allowing easy conversion to a Python list data type or allowing the use of for/in or map. For more information on __iter__ see the Python docs on iterators.

The only thing left to do is modify read_pair. The procedure to read a pair is as follows:

1. Get a value for the car slot by calling scheme_read on the input file
2. Remove any whitespace
3. Get a value for the cdr by calling read_pair on the input file
   • If the next character is “)” set the cdr to the_empty_list
   • If the next character is anything else set the cdr to a new pair by calling read_pair

from scheme_types import Symbol, the_empty_list, Pair

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def is_initial(c):
  """Is c a valid initial character for an identifier?"""
  return c.isalpha() or c in '-+*/><=?!&'

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if '.' in buf:
    return float(buf)
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def read_expected_string(f, s):
  """Consume expected characters from the input buffer."""
  expected_chars = list(s)
  c = f.getc()
  while expected_chars:
    if c == expected_chars[0]:
      expected_chars.pop(0)
      c = f.getc()
    else:
      return "unexpected character"

def read_character(f):
  c = f.getc()
  if c == 's' and f.peek() == 'p':
    read_expected_string(f, "pace")
    return ' '
  elif c == 'n' and f.peek() == 'e':
    read_expected_string(f, "ewline")
    return '\n'
  else:
    return c

def read_symbol(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  return ''.join(buf)

def read_string(f):
  buf = []
  c = f.getc()
  while c != '\"':
    buf.append(c)
    c = f.getc()
  return ''.join(buf)

def read_pair(f):
  f.remove_whitespace()
  c = f.getc()
  if c == ')':
    return the_empty_list
  f.ungetc(c)
  car = scheme_read(f)
  f.remove_whitespace()
  cdr = read_pair(f)
  return Pair(car, cdr)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == '-' and f.peek().isdigit() or f.peek() == '.')
      or (c == '.' and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  elif c == '#':
    c = f.getc()
    if c == 't':
      return True
    elif c == 'f':
      return False
    elif c == '\\':
      return read_character(f)
    else:
      return "Error boolean value must be #t or #f not #" + c
  elif is_initial(c):
    f.ungetc(c)
    return Symbol(read_symbol(f))
  elif c == '\"':
    return read_string(f)
  elif c == '(':
    return read_pair(f)
  else:
    return "scheme_read: not implemented"

There are two things to note about this implementation. First, () is matched as the_empty_list when it really shouldn’t be, the correct syntax is ‘(). Second, any literal list over 1,000 elements will hit Python’s recursion limit. This shouldn’t be a problem in the vast majority of cases, however it is something to be aware of.

With that, the read layer of our interpreter is complete (for the purposes of this series, there is a lot more that could be done). From this point on we will be concentrating on the eval layer of the interpreter.

Checkout this version on GitHub (v0.03)

GOTO Part 4 | Self-evaluating values

GOTO | Table of Contents

In the last post we created a read function that worked with integers and floating point numbers. We also covered the basic layout of the read function. In this post we will extend the read function to handle booleans, characters, strings and symbols.

We’ll start with booleans as they are the smallest set. There are 2 boolean values in Scheme #t and #f, which represent true and false respectively. You probably noticed that both start with a # character and that is how we will recognize them in the reader.

We will represent #t and #f by the Python values True and False. Since the code for this is so simple, we will handle this within the scheme_read function.

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if '.' in buf:
    return float(buf)
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == '-' and f.peek().isdigit() or f.peek() == '.')
      or (c == '.' and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  elif c == '#':
    c = f.getc()
    if c == 't':
      return True
    elif c == 'f':
      return False
    else:
      return "Error boolean value must be #t or #f not #" + c
  else:
    return "scheme_read: not implemented"

The code to handle booleans is pretty straightforward. If the first character is #, then we get the next character. If the next character is t or f, we return the appropriate boolean. Otherwise, we return an error string.

The next thing we’ll handle is characters. Characters in Scheme also start with the # symbol followed by a \, which gives us #\c where c is any character.

There are some special characters, but we’ll start by implementing the basics. We’ll check if an expression is a character in the same place as we check for booleans since they’re so similar.

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if '.' in buf:
    return float(buf)
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def read_character(f):
  c = f.getc()
  return c

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == '-' and f.peek().isdigit() or f.peek() == '.')
      or (c == '.' and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  elif c == '#':
    c = f.getc()
    if c == 't':
      return True
    elif c == 'f':
      return False
    elif c == '\\':
      return read_character(f)
    else:
      return "Error boolean value must be #t or #f not #" + c
  else:
    return "scheme_read: not implemented"

Scheme has a few special characters that we must deal with, namely #\space and #\newline. These will require a bit more work to process. To help, we’ll define a function read_expected_string which will consume characters from the input buffer, provided they match what’s expected (and return an error if they don’t).

I won’t go over the workings of read_expected_string any further than to say it compares characters from the expected string and the input buffer one-by-one until it reaches the end. We also need to add a check in read_character if the input buffer contains #\s or #\n. Both of these could be either the characters s or n or the special characters #\space or #\newline. To tell the difference we need to check the next character.

There are a few problems with this implementation, namely if you enter an invalid special character such as #\spxce it will leave unconsumed characters on the input buffer and will not return the error string. Handling these error conditions would add a lot of bloat to the code, though you would definitely want to do this in your own implementation.

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if '.' in buf:
    return float(buf)
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def read_expected_string(f, s):
  """Consume expected characters from the input buffer."""
  expected_chars = list(s)
  c = f.getc()
  while expected_chars:
    if c == expected_chars[0]:
      expected_chars.pop(0)
      c = f.getc()
    else:
      return "unexpected character"

def read_character(f):
  c = f.getc()
  if c == 's' and f.peek() == 'p':
    read_expected_string(f, "pace")
    return ' '
  elif c == 'n' and f.peek() == 'e':
    read_expected_string(f, "ewline")
    return '\n'
  else:
    return c

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == '-' and f.peek().isdigit() or f.peek() == '.')
      or (c == '.' and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  elif c == '#':
    c = f.getc()
    if c == 't':
      return True
    elif c == 'f':
      return False
    elif c == '\\':
      return read_character(f)
    else:
      return "Error boolean value must be #t or #f not #" + c
  else:
    return "scheme_read: not implemented"

Next up are symbols, which are the names of variables/constants. In most other languages symbols are not given their own type and you, as a user, are not allowed to manipulate them as first-class values. However this is not the case in Scheme.

We will represent symbols in our implementation as strings. However since we also need to implement strings, it is necessary to make symbols a distinct type. Since we will need to make other types as well as have access to them outside the read layer, make a new file called scheme_types.py and enter the following definition.

class Symbol(str):
  """A symbol is an immutable string, but must be its own type."""
  pass

How do we know if an expression is a symbol? There is a limited set of values that a symbol may start with. To test if a character is part of this set, we’ll provide a function is_initial.

Once we’ve identified that the expression is a symbol, the next step is to read each character into a buffer until we’ve hit a delimiter (just like reading a number). Then we turn the buffer into a string and pass it to Symbol to create a new symbol.

from scheme_types import Symbol

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def is_initial(c):
  """Is c a valid initial character for an identifier?"""
  return c.isalpha() or c in '-+*/><=?!&'

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if '.' in buf:
    return float(buf)
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def read_expected_string(f, s):
  """Consume expected characters from the input buffer."""
  expected_chars = list(s)
  c = f.getc()
  while expected_chars:
    if c == expected_chars[0]:
      expected_chars.pop(0)
      c = f.getc()
    else:
      return "unexpected character"

def read_character(f):
  c = f.getc()
  if c == 's' and f.peek() == 'p':
    read_expected_string(f, "pace")
    return ' '
  elif c == 'n' and f.peek() == 'e':
    read_expected_string(f, "ewline")
    return '\n'
  else:
    return c

def read_symbol(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  return ''.join(buf)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == '-' and f.peek().isdigit() or f.peek() == '.')
      or (c == '.' and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  elif c == '#':
    c = f.getc()
    if c == 't':
      return True
    elif c == 'f':
      return False
    elif c == '\\':
      return read_character(f)
    else:
      return "Error boolean value must be #t or #f not #" + c
  elif is_initial(c):
    f.ungetc(c)
    return Symbol(read_symbol(f))
  else:
    return "scheme_read: not implemented"

The reader does not lookup the value of symbols, that’s the job of the eval layer, so we’re all done for handling symbols.

With symbols done we only have strings left to implement for this part in the series. Luckily the implementation for strings is almost exactly the same as the one for symbols. Strings in Scheme start and end with “, so an expression is a string if the first character is a “.

The read_string function we’ll provide is a copy/paste of the read_symbol function with only one change. Instead of while not is_delimiter(c) we will use while c != ‘\”‘ since delimiter characters are allowed to appear in strings. Since the functionality is identical, you could abstract this into one function by adding a parameter that passes the stopping condition. However I feel that it’s more clear, for this series, as two.

from scheme_types import Symbol

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def is_initial(c):
  """Is c a valid initial character for an identifier?"""
  return c.isalpha() or c in '-+*/><=?!&'

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if '.' in buf:
    return float(buf)
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def read_expected_string(f, s):
  """Consume expected characters from the input buffer."""
  expected_chars = list(s)
  c = f.getc()
  while expected_chars:
    if c == expected_chars[0]:
      expected_chars.pop(0)
      c = f.getc()
    else:
      return "unexpected character"

def read_character(f):
  c = f.getc()
  if c == 's' and f.peek() == 'p':
    read_expected_string(f, "pace")
    return ' '
  elif c == 'n' and f.peek() == 'e':
    read_expected_string(f, "ewline")
    return '\n'
  else:
    return c

def read_symbol(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  return ''.join(buf)

def read_string(f):
  buf = []
  c = f.getc()
  while c != '\"':
    buf.append(c)
    c = f.getc()
  return ''.join(buf)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == '-' and f.peek().isdigit() or f.peek() == '.')
      or (c == '.' and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  elif c == '#':
    c = f.getc()
    if c == 't':
      return True
    elif c == 'f':
      return False
    elif c == '\\':
      return read_character(f)
    else:
      return "Error boolean value must be #t or #f not #" + c
  elif is_initial(c):
    f.ungetc(c)
    return Symbol(read_symbol(f))
  elif c == '\"':
    return read_string(f)
  else:
    return "scheme_read: not implemented"

With string handling completed there is only one major feature left for scheme_read, pairs. Pairs are used to implement linked lists in Scheme. The linked list is a very important data structure as all Scheme code is written and parsed into linked list format. This code/data duality is one of the things that gives Scheme (and other Lisps) its great power. The next post will be about the implementation of linked lists and how to parse them.

Checkout this version on GitHub (v0.02)

GOTO Part 3 | Pairs and linked lists

GOTO | Table of Contents

Stage 1 of implementing Scheme in Python is to write the reader. The entire read layer is self-contained so we don’t need to worry about evaluating and printing values just yet.

To start with let’s take a look at the basic shape of the read function. We’re going to call it scheme_read to avoid any name clashes that might arise, but other than that it should be more-or-less equivalent to any other Scheme read function.

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if c:
    return "Representation of a Scheme type."

scheme_read(f) is called with a buffered file-like object f that must support getc, ungetc, peek and remove_whitespace methods. First we remove any leading whitespace as whitespace in Scheme is only significant as a delimiter.

Before we get started let’s implement a simple repl so that we can play with our new features as we add them. Create a new file named repl.py and insert the following code.

#!/usr/bin/env python
import sys
from scheme_read import scheme_read
from buffered_input import Buff

while True:
  print "> ",
  print scheme_read(Buff(sys.stdin))

The next step is to get the first non-whitespace character, which we store in the variable c. We will use this character (and possibly the next) to determine the type of the expression. For instance if c is a digit then the remaining characters (until the next delimiter) must form a number, since digits are not allowed as the first character of an identifier. We’ll do this check in the if c: portion of our code. Let’s look at that code.

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if c.isdigit():
    return "Representation of a Scheme number."

What if c is not a digit? In that case, our read function does not recognize the input and we should return something that states this fact. For now let’s just return a string that says “scheme_read: not implemented”. Later on you may wish to make this a proper error, but returning a string here keeps things simple.

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if c.isdigit():
    return "Representation of a Scheme number."
  else:
    return "scheme_read: not implemented"

Now obviously, “Representation of a Scheme number.” is not actually a representation of a Scheme number. To get that we have to read the number from f. Instead of putting all the logic of reading a number into the if block, we should use a helper function here called read_number.

def read_number(f):
  return "Representation of a Scheme number."

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if c.isdigit():
    return read_number(f)
  else:
    return "scheme_read: not implemented"

We pass the file object f to read_number so that it can read and return the rest of the number. But wait, we already read the first digit! Before we pass the work off to read_number, we have to put the first digit back onto f. We do this using the ungetc method.

def read_number(f):
  return "Representation of a Scheme number."

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if c.isdigit():
    f.ungetc(c)
    return read_number(f)
  else:
    return "scheme_read: not implemented"

Now to read a number we simply read a character at a time, placing each into a buffer, until we hit a delimiter. To make that job easier we’ll provide a function is_delimiter that takes a character and returns True if that character is a delimiter and False if not. Then we’ll use that to build a buffer and return an integer.

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  return int(buf)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if c.isdigit():
    f.ungetc(c)
    return read_number(f)
  else:
    return "scheme_read: not implemented"

That takes care of positive integers, however the positive integers are a very small subset of numbers. We are going to want to handle negative integers as well as positive and negative floating point numbers at least. Luckily, this is very simple to do in Python.

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if '.' in buf:
    return float(buf)
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if c.isdigit():
    f.ungetc(c)
    return read_number(f)
  else:
    return "scheme_read: not implemented"

However, we’re not quite done yet. Our implementation won’t handle negative numbers. To do that we’re going to have to go back to the reader because “-” is not a digit and our current implementation will skip right over it.

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if '.' in buf:
    return float(buf)
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if c.isdigit() or (c == '-' and f.peek().isdigit()):
    f.ungetc(c)
    return read_number(f)
  else:
    return "scheme_read: not implemented"

We’re still not quite done though. What about floating point values where 0 < n < 1 that start with a decimal point?

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (‘ ‘, ‘(‘, ‘)’, ‘\"’, ‘;’, ‘\n’) or not c

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if ‘.’ in buf:
    return float(buf)
  elif ‘/’ in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == ‘-’ and f.peek().isdigit())
      or (c == ‘.’ and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  else:
    return "scheme_read: not implemented"

But wait, there's more! What about negative floating point numbers where -1 < n < 0 which start with a "-."?

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if '.' in buf:
    return float(buf)
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == '-' and f.peek().isdigit() or f.peek() == '.')
      or (c == '.' and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  else:
    return "scheme_read: not implemented"

As you can see, a lot of the work of implementing a programming language goes into dealing with corner cases. In the next post we'll extend our reader to handle booleans, characters, strings and symbols.

Checkout this version on GitHub (v0.01)

GOTO Part 2 | Extending the read layer

GOTO | Table of Contents

1) Read Layer
2) Eval Layer
3) Language Layer

The read layer

The read layer is responsible for consuming source code and transforming it into a data structure so that it can be passed to the eval layer.

The eval layer

After the reader returns a data structure representing the source code, the eval layer evaluates it according to a set of rules and returns a value.

After an expression has been evaluated it’s value is often printed to the screen. Immediately after that the program usually loops back to reading the next expression, hence the term REPL (Read, Eval, Print, Loop).

The language layer

This interpreter is designed using the “languages as libraries” concept. This approach is currently being used by the Racket and PyPy projects (though in a much more advanced form).

A language is usually defined as the core components, or primitives, plus additional libraries written using those primitives. In most languages it is an error to attempt to redefine a primitive procedure and even in languages where this is permissible, there is usually no way to swap all of the primitives.

Using the languages as libraries approach, we implement as little as possible in the source code of the interpreter and allow a file to be loaded with definitions of the primitives in the language itself. This implementation only requires one primitive to be defined at the interpreter level¹, define-syntax, which allows the user to define new primitive syntax.

Before we begin

The read function presented here is a port of the read function in Bootstrap Scheme by Peter Michaux. His implementation relies on the C functions getc and ungetc. Unfortunately input in Python is line-buffered and not character-buffered and Python does not provide ungetc.

To accommodate for this I have created a class (Buff) which wraps stdin (or any other file-like object) and provides the functions getc, ungetc and peek. Reading from input in this manner is not in the least bit Pythonic, however this method of reading is very simple and clearly shows the steps involved in lexing and parsing input strings.

The operation of Buff is simple. getc reads a single character from the given file if there are no characters waiting in the buffer, otherwise it pops off the last character added to the buffer. ungetc simply adds a character to the buffer, and peek uses getc to read a character and store it in a variable, then ungetc to put the character back in the buffer.

class Buff:
  def __init__(self, f):
    self.f = f
    self.last = []

  def getc(self):
    if self.last:
      return self.last.pop(0)
    else:
      return self.f.read(1)

  def ungetc(self, c):
    self.last.insert(0, c)

  def peek(self):
    if self.last:
      return self.last[0]
    else:
      c = self.f.read(1)
      self.last.append(c)
      return c

  def remove_whitespace(self):
    c = self.getc()
    while c:
      if c == ' ':
        c = self.getc()
      elif c == '\n':
        c = self.getc()
      elif c == ';':
        c = self.getc()
        while c and c != '\n':
          c = self.getc()
        self.ungetc(c)
      else:
        self.ungetc(c)
        break

You may notice that there is also a function: remove_whitespace. This simply reads and discards any characters which are whitespace (space and newline) and discards comments ;starting at the semicolon and continuing until a newline is reached.

With that we have the tools we need to get started building the read level.

[1] In theory, in practice we also define “load” to avoid the necessity of redefining it every time we restart the interpreter.

Checkout this project on GitHub

GOTO Part 1 | Numbers

GOTO | Table of Contents

Pipe: Infix syntax for Python

Here’s some more links…

Python Package Index Page
Github Page

For now this is simply a reminder to myself to play around with this code a bit. At some point in the future I may write up a short post on this.

Here’s a short excerpt from Julien Palard’s post:

Compare the readability of the classical prefix syntax :

sum(select(where(take_while(fib(), lambda x: x < 1000000) lambda x: x % 2), lambda x: x * x))

And the infix syntax :

fib() | take_while(lambda x: x < 1000000)
      | where(lambda x: x % 2)
      | select(lambda x: x * x)
      | sum()

Isn’t the infix syntax more readable?

I want to create a program that can keep track of what episode of different shows im on.
I’m watching several and have trouble remembering which in’s im on. I know i could just make a word doc but this seems cooler. Id like each show to have a button that says what episode and when i click it, the number goes up. Also, it has to save that information. Thanks anybody for helping.

The first response was priceless. I’m only going to quote part of it, you can find the rest here.

geez dude, what you’re asking for isnt exactly a DOS kind output…(sic)

I had the same reaction to the question at first. After a minute or so I realized that I could probably use a program like this. I too watch a number of television shows, almost exclusively downloaded from the internet, and I find myself looking through my file system to figure out which episode I need to download next. Surely this is wasteful, and a quick program could save me precious seconds each day (which I would undoubtedly use to browse reddit or LtU).

About 20 minutes later I hacked together some of the ugliest code I’ve seen in a while and posted an answer to the question. Then I made some nachos. About an hour later I decided to spend another 10 minutes and fix a few bugs in my first version. After about 30 minutes of hacking I came up with this code, which is probably the most functionality I’ve ever created from scratch in 30 minutes of coding (though the code itself is crap).

You can also find a more current version of this as a gist at my github account.

#!/usr/bin/env python
##############################################################################
## If you would like to contribute, contact jacktradespublic@gmail.com
##
## tvtracker is free software: you can redistribute it and/or
## modify it under the terms of the GNU Affero General Public
## License version 3 as published by the Free Software Foundation.
##
## This program is distributed in the hope that it will be useful,
## but WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
## GNU Affero General Public License version 3 for more details.
##
## You should have received a copy of the GNU Affero General Public
## License version 3 along with this program. If not, see
## <http://www.gnu.org/licenses/>.
##############################################################################
## Known Bugs:
## - This needs some serious refactoring, even for this simple script
## - The current episode number just keeps growing and doesn't wrap
##   around seasons.
## - There is no way to undo an increment.
##
## Improvements
## - Next episode available: Input the day of the week that the
##   show is usually aired and the program will notify you when the
##   next episode is available. You should also be able to enter a
##   delay (for things like hulu or torrents).
##############################################################################

from Tkinter import *
import pickle

def loadShows():
  global shows
  f = open(filename, 'r')
  shows = pickle.load(f)
  f.close()

def saveShows():
  f = open(filename, 'wb')
  pickle.dump(shows, f)
  f.close()

class Add_Episode:
  def __init__(self, parent):
    self.add_frame = Frame(parent)

    self.name_label = Label(self.add_frame, text="Name: ")
    self.name_entry = Entry(self.add_frame)
    self.episode_label = Label(self.add_frame, text="Current Episode: ")
    self.episode_entry = Entry(self.add_frame)
    self.add_show_button = Button(self.add_frame, text="Add Show", command=self.add_show)

    self.name_label.pack(side=LEFT)
    self.name_entry.pack(side=LEFT)
    self.episode_label.pack(side=LEFT)
    self.episode_entry.pack(side=LEFT)
    self.add_show_button.pack(side=LEFT)
    self.add_frame.pack(side=TOP)

  def add_show(self):
    name = self.name_entry.get()
    episode = int(self.episode_entry.get())
    shows[name] = episode
    Show(root, name, episode)
    saveShows()

class Show:
  def __init__(self, parent, name, episode):
    self.name = name
    self.episode = episode

    f = Frame(parent)
    f.pack(side=LEFT)
    self.label = Label(f, text=name)
    self.entry = Label(f, text=episode)
    self.button = Button(f, text="Increment", command=self.increment)

    self.label.pack()
    self.entry.pack()
    self.button.pack()

  def increment(self):
    self.episode += 1
    self.entry.config(text = self.episode)
    shows[self.name] += 1
    saveShows()

if __name__ == "__main__":

  filename = "tvshows.pk"

  try:
    loadShows()
  except IOError:
    shows = {}

  root = Tk()
  root.title('Tv Shows')

  Add_Episode(root)

  for n, e in shows.items():
    Show(root, n, e)

  root.mainloop()

The reason I’m posting this here is not because the code is elegant (it definately isn’t) or because the app is great (again, not so much). I’m posting this because I think this is a good project for a beginner to bite into. This is a very simple application and adding features, fixing bugs and refactoring my poorly thought out code is a great exercise.

If anyone is interested in helping with this project I would be more than willing to give you help, hints, code reviews, ideas or whatever other assistance you want. You can also take this code and work on it yourself if you want, it’s really up to you. Though I haven’t started using it yet, it is quickly approaching “good enough for my use case” and I probably won’t be doing much more to it.

¹ Yahoo Answers is just about the worst service I’ve ever used. The interface is horrible and if you copy/paste the question into a search engine, the vast majority of the time the answer will be the #1 result. However, unlike stackoverflow or the mailing lists, I don’t feel like I have to spend all day watching for a new question, just to have a chance at answering it before someone else.

There are two schools of thought when it comes to writing code. The first is to sit down and throughly plan out your code and what it should do. The second is to sit down and start writing code, no matter how ugly and bug prone it may be, then hammer at the bugs until you have a beautifully exquisite piece of software. It is the latter approach that I recommend that you use, because it not only teaches you what works, but more importantly what doesn’t so that you can avoid that in the future.

This is a relatively short intro to the programming language Python. Python is a great beginner language as it hides many of the more advanced aspects of programming from the programmer. This doesn’t mean though, that Python is not a ‘real’ programming language, rather it takes care of things like memory management and object typing behind the scenes, making your code less buggy and error prone.

There are some terms you should know before setting out to learn any programming language. To those power-users some of these terms might be redundant.

Code: Code is a command, or commands, that are executed by the interpreter. Code can be saved in a file and run or typed into the interpreter.

Script: A script is simply a text file that contains code to be executed by an interpreter. You create a python script by putting .py at the end of the filename.

Program: A program can be a script or a collection of scripts that interact with each other.

Interpreter: An interpreter is a program that reads code and produces the output specified by the code. The Python interpreter will accept single lines of code, scripts or programs.

0) Using Python

Why does this tutorial start at zero? Computers usually start counting from zero. The simplest way to look at this is that zero is the first number in our number system, though there are some other reasons that may become clear to you later.

There are a number of ways to use Python, we will concentrate on the easiest method first (later on we will discuss some of the more advanced methods). First you must download the python executable from the Python website (www.python.org at the time of this writing). Once you have the program downloaded go ahead and set it up by double clicking on the icon and following along with the prompts (you will notice that it is no different than any windows program installation). Click ‘finished’ when you are done and you are now ready to start using Python. The program that we will be doing all our work in is called ‘Idle’ and it is Python’s default interactive interpreter (there are others, which we will get to toward the end of this tutorial). To start idle go to the Start menu, then All Programs, then Python. Click on Idle and the the Idle interpreter will pop up.

Python 2.4.3 (#1, Jun 13 2006, 14:23:21)
[GCC 4.1.1 20060612 (Red Hat 4.1.1-2)] on linux2
Type "copyright", "credits" or "license()" for more information.

 ****************************************************************
 Personal firewall software may warn about the connection IDLE
 makes to its subprocess using this computer's internal loopback
 interface. This connection is not visible on any external
 interface and no data is sent to or received from the Internet.
 ****************************************************************

IDLE 1.1.3
>>>

You should see something like the above when you open Idle. For now the only significant thing here is the >>>. This is the command prompt where you will enter all your commands. Just to make sure it works type 2+2 and then hit return. Did it print 4 on the line below 2+2? As you can see Python can do math too, but we’ll save that for a little bit later. Now you can move on to writing your first program.

1) Hello World

It is customary to start each introductory text with a hello world program. This is typically the simplest program you can write in any programming language. It is a very simple program which only does one thing, it prints “hello world” to the screen on the interpreter.

>>> print 'Hello World!'
'Hello World!'

Great, you have just created your very first program. Don’t feel special yet? Don’t worry there’s much more to come. Let’s break this simple program down to see what Python does with it.

print – print is the command used to display results on the screen. It doesn’t print anything to your printer so don’t go putting paper in.

‘Hello World!’ – This is a string. A string is any combination of letters, numbers and special symbols encased in either single quotes ‘ ‘, double quotes ” “, or triple quotes “”" “”" or ”’ ”’.

Let’s try something else:

>>> print "hello", 'world'
'hello world'

The comma separates objects to be printed and automatically inserts a space between them. You can separate letters, numbers and symbols with a comma. Note also that we can use different types of quoting in the same sentence and Python doesn’t care.

>>> print "I can't", 'run today'
I can't run today

This illustrates why we would want to use different quotes around different strings. If we used single quotes around ‘I can’t’ you see that the computer could get confused as to where to end the string (after n or t?).

>>> print 'Go see "The Rock"'
Go see "The Rock"

One of the best reasons to use single quotes is to preserve quotation within a string.

>>> print 'hello'+'world'
helloworld

Using the + sign between words performs an operation called concatenation, which is basically a big word for joining together strings without a space. Below you can see how to use concatenation and enter spaces between the words.

>>> print 'hello '+'world'
hello world

This prints a space between the words because there is a space after hello and before the ending quotation.

Print also can print numbers and the results of mathematical operators.

>>> print 5
5

>>> print 2435656
2435656

>>> print 5+5
10

Numbers with quotes around them are strings so the ‘+’ operator will concatenate instead of adding the two numbers.

>>> print '5'+'5'
'55'

2) Basic Math

Python as well as every other programming language is simply an overpowered calculator. Math in Python is very simple and the rules that you learned in math class will apply directly here. Here are a few math problems and their results.

>>> 5+5
10
>>> 5-4
1
>>> 5*5
25
>>> 5.0/2.0
2.5
>>> 5/2
2

The last problem is the only one here that seems strange. All whole numbers are automatically considered to be integers. A decimal is a special type of number called a floating point number, or float for short. You can declare a float by putting a .0 on the end of a whole number, so that 5 becomes 5.0.

To get the remainder of an expression you simply use the ‘%’ operator. The remainder of 5/2 is 1.

>>> 5 % 2
1

Python can also return a float.

>>> 5/2.0
2.5
>>> 5.0/2
2.5

Python will convert the number to float if one number in the expression is a float.

>>> 2**3
8
>>> 3**3
27

The ‘**’ operator represents raising the number to the x power, so that 2**x raises 2 to the x power.

Introducing Parenthesis

>>> (5*5)+3
28

The parenthesis in Python are equivalent to the parenthesis in math. Anything inside a parenthesis gets done before the things outside the parenthesis. So the expression above reads (25)+3.

>>> ((2*5)*5)+10
510

Parenthesis can be nested also. In this case the innermost parenthesis get evaluated first. So ((2*5)*5)+10 == (10*5)+10 == 500+10 == 510.

You might wonder why I used two equals signs (==). Python uses one equal sign to assign a variable. If you want to describe an equal to relationship you must use two equal signs ==.

Python has other comparison operators that you probably remember from grade school math class.

== equal to
!= not equal to
>= greater than or equal to
<= less than or equal to
> greater than
< less than

There are a few common ways to write Python code but I suggest that you stick with the method described in this tutorial. Other ways are as follows:

>>> 5+5
10
>>> 5 + 5
10
>>> ( 5+5 ) + 3
13
>>> (5 + 5) + 3
13
>>> (5+5)+ 3
13

Whitespace in Python is very important, however whitespace within expressions can be used however you want. It is best though, to pick a style and stick with it so that other developers don’t get confused when they read your code.

3) Variables

Variables come in very handy when programming. A variable is an identifier that holds a value. The best way to explain this is to show you.

>>> a = 10
>>> print a
10

The above example demonstrates how to create a variable (a = 10) and how to access that variable (print a). From now on when we enter the letter ‘a’ the interpreter will substitute the letter for the number 10. Let’s try some more.

>>> b = 5
>>> print b
5
>>> print a+b
15

As you can see we can perform mathematical operations on the variables.

>>> c = 5+15
>>> print c
20
>>> d = a+b
>>> print d
15

We can also assign the value of a mathematical expression to a variable.

>>> print d
15
>>> d = d+1
>>> print d
16

We can also change the variable by referring to itself.

>>> print d
16
>>> d = d/2
>>> print d
8

We can also assign multiple variables at once by using tuples (tuples are a type of list that we will discuss a little later)

>>> (i, j, k) = (1, 2, 3)
>>> print i
1
>>> print j
2
>>> print k
3

Tuples are always enclosed in parenthesis and each value is separated by a comma. The first variable (i) is assigned to the first value (1), the second variable (j) is assigned to the second value (2) and so on. You can have as many variables/values as you like, just make sure that you have equal numbers of each. Below you can see what happens if you don’t.

>>> (i, j, k) = (1, 2)
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
ValueError: need more than 2 values to unpack

>>> (i, j, k) = (1, 2, 3, 4)
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
ValueError: too many values to unpack

In both instances above the Python interpreter spits out a nasty error message. We’ll take this opportunity to talk a little about error messages. Let’s break down the error message into its three lines.

Traceback (most recent call last):

Line 1 of the error message simply tells you that the last instruction that the interpreter tried to do failed, and that the following messages are a result of this instruction failing.

File "<stdin>", line 1, in <module>

This line tells us exactly what instruction failed.

First it tells us what file the error was in. Since we aren’t using a file (we are typing directly into the interpreter) it tells us that the error was in “<stdin>” which is a cryptic way of saying standard input, or input from the keyboard.

Then it tells us what line the program failed on. Here we only have one line so it tells us the error occurred on line 1.

The final hint, in <module>, tells us what function or class the instruction was in. Since we haven’t covered functions or classes yet, just ignore this part.

ValueError: too many values to unpack

Finally this line tells us what type of error occurred. This time it tells us that we gave it too many values (before it told us that we gave it too few values). This line will give you a good hint as to what you need to change to make the instruction work, here we need to take away one value.

For more information on errors, check out What In The Hell Are Errors.

You can also store your values in a variable like the example below.

>>> w = (4, 5, 6)
>>> (x, y, z) = w
>>> print x
4
>>> print y
5
>>> print z
6

Remember here that a variable is simply a placeholder, so when the Python interpreter sees the command (x, y, z) = w it changes it to (x, y, z) = (4, 5, 6).

Variables aren’t just for numbers, you can store any data type in a variable. You can even store a variable in a variable.

>>> m = 'Hello'
>>> o = 'World'
>>> print m, o
Hello World

>>> p = m
>>> q = o
>>> print p, q
Hello World

One of the main reasons to store a variable in a variable is if you want to change the initial variable.

>>> s = 50
>>> t = s
>>> s = s/2
>>> print s
25
>>> print t
50

Notice that t keeps the initial value of s even after we change s by dividing it. This feature will come in very handy when you begin to learn about loops.

4) Functions

A function is a set of instructions.

def HelloWorld():
 print 'Hello World!'

Above is probably the simplest function that you can create. Let’s break it down.

def HelloWorld():

This is where we define the function. We do this by giving the def command followed by a name (like a variable name). We follow the name with a set of parenthesis and after the parenthesis we put a colon. The colon tells the interpreter that we are done defining the function.

 print 'Hello World'

You should remember this from chapter 1. Under the definition you can list any number of commands. These commands should be indented one level from the definition and there should only be one command per line.

When you want to use a function you simply type the name followed by parenthesis.

>>> HelloWorld()
Hello World!

Here’s some more functions and their output when you use them.

>>> def add():
 print 2 + 2

>>> add()
4

>>> def multiply():
 print 2 * 3

>>> multiply()
6

>>> def mu():
 print 2 * 3

>>> mu()
6

You can think of a function as a variable that stores multiple commands. Below we’ll see some more advanced functions that use some of the things we learned in previous chapters.

>>> def commands():
 print 'Hello. I will show you the multiples of 2'
 print 2**0
 print 2**1
 print 2**2
 print 2**3
 print 'There are ' , (2**4)*2 , 'possible combinations in a binary byte'

>>> commands()
Hello. I will show you the multiples of 2
0
2
4
8
There are 16 possible combinations in a binary byte

As you can see above each line in the function commands() is executed one by one until it reaches the end. So far we have only used the print statement in our functions. Let’s see what happens when we use a variable to store data.

>>> def twoPlusTwo():
 add2 = 2+2

>>> twoPlusTwo()

Notice that the function twoPlusTwo() did not print anything when we called it? You might have noticed that add2 = 2+2 will not print anything either when we type it on the command line. Let’s try to use the variable add2 by printing it.

>>> print add2
Traceback (most recent call last):
 File "", line 1, in ?
NameError: name 'add2' is not defined

The interpreter spits out a error message telling us that add2 has not been defined. How can that be? Any variable that you define in a function stays in that function, and is not accessible from outside of the function. The technical term for this is a local variable. Why does Python do this? There are many reasons which you will come to find are very useful, but the main reason is so you don’t have to keep track of all the variables that you used in your program, because redefining a global variable (or a variable accessible from anywhere, more on this below) will erase it’s current value and replace it with the new one. Yikes, that means that if we forget that we used a variable name we could crash the program, or make it spit out completely wrong data!

Functions do not return values by default, instead we must specify a return value if we want the function to produce something. In the above example of twoPlusTwo() when we called it it just gave us a new command line. The function is not broken (Python really did calculate 2+2 and assign it to add2) it merely has no return value. Let’s redefine twoPlusTwo() and give it a return value.

>>> def twoPlusTwo():
 add2 = 2+2
 return add2

>>> twoPlusTwo()
4

Notice that we added one more line, return add2. This line tells the interpreter to return (or give back when called) a value. Note that return and print are two completely different commands (even though they produce the same output here) print simply prints a message to the screen, while return gives us back a value that we can use. Let’s look at an example of this.

>>> def r():
 add2 = 2+2
 return add2

>>> r() + 2
6

Here we can see that we can use the function just like a variable. In this example r() + 2 the interpreter executes the function r() and get a return value of 4. Then it takes that value and adds 2 to it and returns the value of 6. Let’s see what happens if we try this using print.

>>> def p():
 add2 = 2+2
 print add2

>>> p() + 2
Traceback (most recent call last):
 File "", line 1, in ?
TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'

Python spits out another error message that tells us that we cannot add NoneType and int. This tells us that the function p() returned NoneType, which is a way of saying that p() returned nothing, and that we cannot add 2 to nothing (note here that NoneType is not equivalent to 0, we could add 0+2)

Earlier we talked about local variables. Let’s try a little experiment with the return statement.

>>> def ret():
 sub2 = 4 - 2
 return sub2

>>> ret()
2
>>> print sub2
Traceback (most recent call last):
 File "", line 1, in ?
NameError: name 'sub2' is not defined

Note here that the return statement did not give us a global variable of sub2. Instead it just returned the value of sub2.

4.1) Global and Local Variables

Close your Python interpreter and reopen it (this will get rid of all the variables and commands we entered already, and give us a clean slate to work with). Below is a session that shows us the difference between local and global variables.

>>> a = 10 ## a is a global variable because it is not inside a function
>>> b = 15 ## b is a global variable because it is not inside a function
>>> def add():
 c = 20 ## c is a local variable
 return a+b+c

Global variables are variables defined outside of any function (or class, but we haven’t learned about classes yet). Another way to think about it is that any variable that is declared and is not indented is a global variable (this will be a little more clear below)

We can use global variables anywhere, but we can only use local variables inside the function that declared them. When we are using more than a few lines to tell the interpreter what to do it is often easier to create a script, or a file containing commands for the interpreter. Every Python script must end in the ‘.py’ extension, you can do this simply by typing ‘.py’ at the end of the filename when you save it. Let’s rewrite the above code in a script.

First go to File then click New Window in the Python interpreter. This will give you a blank window. The first thing we’ll do is save the file (even thought it’s blank). Go to File –> Save. A window will pop up asking you where you would like to save the file and what you would like to name it. For now we’ll choose the default location and we’ll name the file ‘test.py’. Saving the file right away does a few things for us that make writing a script easier. First, and most important, it lets us save the document anytime we want by simply using the keyboard shortcut Ctrl – s. Second saving the file as a Python script enables a feature of the text editor called syntax highlighting that changes certain keywords to a different color, while this might seem frivolous or even confusing at first, when you get used to the colors it will help you identify your code.

OK now that we’ve got that out of the way let’s get down to writing our script.

#############
## test.py ##
#############

# This is a comment. Anything after the '#' until the next line will be ignored by the interpreter
# Use comments to explain your code so that it is easier for you to remember what does what

a = 10
b = 15

def add():
 c = 20
 return a+b+c

The code above is a script. Strictly speaking any comments are not necessary, but to help us remember or to help someone else understand the code they are useful. I use a comment box, which is simply text surrounded by comment symbols, to identify things like filename, date created, date revised, changes to the script, TODOs and special sections of code. You do not have to use the comment box, instead you can use just a single comment (#) to denote the same thing. I also often use two comments to signify a comment (##) instead of just one. Again this is just personal preference and you can choose to comment however you like.

Like I stated earlier, one of the easiest ways to identify a global variable is to look at the level of indentation. Global variables will have no spaces before their declaration. This means that a and b are both global variables, while c is not (because it has spaces before it).

This script right now will run, but it won’t output anything. This is because we haven’t yet used any of the variables/functions that we defined. Let’s add a little bit to the script so that it does something.

#############
## test.py ##
#############

a = 10
b = 15

def add():
 c = 20
 return a+b+c

add() ## This line uses the function add() just like we've done on the command line

One of the nice features of the IDLE text editor is the F5 key. To run your script you simply press the F5 key and a window will pop up that tells you that you must save the script before you can run it (if you haven’t saved it already). Hit OK and IDLE will save the script for you and then run it on the command line. You will see the interpreter pop up and then you will see it spit out the number 45 (10 + 15 + 20). Congratulations, you have written your first script.

Python executes a script one line at a time (like we did manually at the command line). The script above is executed as follows.

1) set a = 10
2) set b = 15
3) define the function add()
4) execute add() and return the value

If we were to change the script and put the definition and the command add() before we declared a and b we would get an error telling us that a and b haven’t been defined yet. The code below is an example of trying to define and use a function before declaring the variables that are used in the function.

#############
## test.py ##
#############

def add():
 c = 20
 return a+b+c

add()

a = 10
b = 15

One of the properties of functions is that they are not executed in order. Instead a function is only executed when it is called. The script above breaks when we try to run it. But is that because we defined the function before we defined the variables? Or because we tried to use the function before we defined the variables. Look at the code below for the answer.

#############
## test.py ##
#############

def add():
 c = 20
 return a+b+c

a = 10
b = 15

add()

This works! The only thing we’ve changed here is where we execute the function (after we declare the variables). This shows us that a function is not evaluated until it is used. So even though we used variables in the function that don’t yet exist (from a line-by-line execution viewpoint) we define the variables before we use the function, making the script work.

To learn more about functions check out What In The Hell Are Functions.

5) Conditionals

A conditional is an if/then statement. We use these all the time in everyday life. For instance we might say; if the temperature is over 80 degrees, then turn on the air conditioning. The code for this statement would look something like this:

if temperature over 80:
 turn on air conditioning

The above code is what we programmers call psudocode. Psudocode won’t actually run, its purpose is instead to give us an idea of how the code should look. It is also very easy to turn psudocode into real code.

if temp > 80:
 runAC()

This is the real code for the psudocode above. Notice the similarities? One of the new features here is the > sign. > is called an operator and is part of a group of operators.

== equal to
!= not equal to
>= greater than or equal to
<= less than or equal to
> greater than
< less than

Each of these operators returns a boolean value. A boolean value is either True or False so if we say a > b the boolean response would be True if a is greater than b, or False if a was less than or equal to b.

Operators are used extensively in conditional statements so it is best to fully understand them and their use. Operators return only a boolean value, you cannot for instance, get an integer or string from an operator.

You can execute as many commands as you would like in an if statement. The following code demonstrates this:

if number > 60:
 hours = number/60
 minutes = number % 60
 string = hours + ':' + minutes
 print string

This code takes a number of minutes and transforms it into hours:minutes, then prints the output.

To learn more about conditionals check out What In The Hell Are Conditionals.

Links

Pointless Programming – What In The Hell Series

Official Python Tutorial

How to Think Like a Computer Scientist

A Byte of Python

More books and tutorials for beginner programmers.

In Python for and while loops are not blocks. This means that variables defined within a loop still exist after that loop ends.

for i in range(3):
  pass
i
#===> 2

while True:
  a = 42
  break
a
#===> 42

Python’s for loop also accepts an else statement that only executes if the loop runs to completion.

for i in range(3):
  if i == 1:
    break
else:
  print "Else not executed here because the for loop ended with a break."

for i in range(3):
  pass
else:
  print "This will print because the loop completed successfully."

Booleans can act like integers, but only sometimes.

isinstance(False, bool)
#===> True
isinstance(False, int)
#===> True

a = True
b = True
a + b
#===> 2
a == 1
#===> True
a is 1
#===> False

Tuples always need commas.

a = (1)
type(a)
#===> integer

b = (1,)
type(b)
#===> tuple

+= vs + with mutable objects

a = [1, 2, 3]
id(a)
#===> 140415741190800
a += [4]
id(a)
#===> 140415741190800
a = a + [5]
id(a)
#===> 140415741191304

Default arguments are assigned when a function is defined, not when it is evaluated.

from time import time, sleep

def test(a=time()):
  print a

test()
#===> 1299590594.48
sleep(3)
test()
#===> 1299590594.48
# a is bound to time() at def, so will always return the same value

Tuple object does not support item assignment… or does it?

a = ([42],)
a[0] += [43, 44]
#===> TypeError: 'tuple' object does not support item assignment
a
#===> ([42, 43, 44],)

Links

Here are links to similar articles, in no particular order.

Python Pitfalls
Stack Overflow – Python Gotchas and Landmines
Python Warts
Charming Python: Python Elegance and Warts Part 1 and Part 2
Stack Overflow – Hidden Features of Python
When Pythons Attack
Python Gotchas
10 Python Quirks
Python Idioms and Efficiency
bool is int

* Most of these will only be “surprises” to newer Python programmers.
** These are not necessarily problems/warts/bugs in Python.

def reverse(s):
  r = ""
  for c in s:
    r = c + r
  return r
s = "String to reverse."
print reverse(s)

and…

s = "String to reverse."
print "".join(s1 for c in xrange(len(s) - 1, -1, -1))

Probably the easiest and close to the fastest way to reverse a string is to use Python’s extended slice syntax. This allows you to specify a start, stop and step value to use when creating a slice. The syntax is: [start:stop:step].

s = "String to reverse."
print s[::-1]

If start is omitted it defaults to 0 and if stop is omitted it defaults to the length of the string. A step of -1 tells Python to start counting by 1 from the stop until it reaches the start.

When working with large strings, or when you just don’t want to reverse the whole string at once, you can use the reversed() built-in. reversed() returns an iterator and is arguably the most Pythonic way to reverse a string.

s = "String to reverse."
print "".join(reversed(s))

#!/usr/bin/env python
from __future__ import division
import xmlrpclib

MAX_POSTS = 100

server = xmlrpclib.ServerProxy('https://YOUR_BLOG_ADDRESS.wordpress.com/xmlrpc.php')
result = server.metaWeblog.getRecentPosts('YOUR_BLOG_ADDRESS', 'YOUR_USERNAME', 'YOUR_PASSWORD', MAX_POSTS)

numPosts = len(result)
numWords = 0
for post in result:
  numWords += len(post['description'].split())
avgWords = numWords/numPosts

print """Total Posts: %d
Total Words: %d
Avg Words: %d""" % (numPosts, numWords, avgWords)

And the output I get is:

Total Posts: 21
Total Words: 9731
Avg Words: 463

Which is slightly wrong because this simple script does not differentiate between drafts and posts (and my word counting routine is probably too naive among other things). A more correct and robust version of this is left as an exercise to the reader. I’m more than satisfied with this simple, rough estimate.

This article will deal with retrieving and using input to your CGI programs.  There are two ways to get input from your CGI programs, encoded within the URL and submitted through forms.  This tutorial will go over both methods.

We’ll need to do a little setup to begin, if you’ve already read through the first tutorial you should be very familiar with this process.

First we need to create a directory to house our CGI server and scripts, for this tutorial we’ll name it ‘server’.  Create a new file, with the following program, in the ‘server’ directory and name it ‘server.py’.

#!/usr/bin/env python

import BaseHTTPServer
import CGIHTTPServer

server = BaseHTTPServer.HTTPServer
handler = CGIHTTPServer.CGIHTTPRequestHandler
server_address = ("", 8000)
handler.cgi_directories = [""]

httpd = server(server_address, handler)
httpd.serve_forever()

Give ‘server.py’ executable permissions ($ chmod +x server.py).  The next step is to create a page that will give us a link with an encoded URL.  To do this we’ll create a simple HTML page with nothing but the encoded link in it.  Create a new file named ‘link.py’ in the ‘server’ directory, with the following code and give it executable permissions ($ chmod +x link.py).

#!/usr/bin/env python

print """Content-type: text/html

<html><head><title>Test URL Encoding</title></head><body>
<a href="http://localhost:8000/test_urlencode.py?first=Jack&last=Trades">Link</a>
</body></html>'

Now we need to create the CGI script that will receive this data and output it into a new page.  Create a new file named ‘test_urlencode.py’ in the ‘server’ directory and input the following code.

#!/usr/bin/env python

import cgi

form = cgi.FieldStorage()

val1 = form.getvalue('first')
val2 = form.getvalue('last')

print """Content-type: text/html

<html><head><title>Test URL Encoding</title></head><body>
Hello my name is %s %s
</body></html>""" % (val1, val2)

Remember to give executable permissions to all your CGI scripts ($ chmod +x test_urlencode.py).  That’s it!  We now have a working CGI script.  To test it type the following into the location bar of your browser.

http://localhost:8000/link.py

Click on the link and the ‘test_urlencode.py’ script will output a page that says “Hello my name is Jack Trades”.  You can play around with the output by changing the URL of the link to whatever you want.  Below is a sample of URLs and their output, you can type/paste these directly into the location bar (no need to modify the ‘link.py’ script).

http://localhost:8000/test_urlencode.py?first=Jack&last=Trades
Hello my name is Jack Trades

http://localhost:8000/test_urlencode.py?first=Adam&last=Baum
Hello my name is Adam Baum

http://localhost:8000/test_urlencode.py?first=Anna&last=Grahm
Hello my name is Anna Grahm

Courtesy of Bart Simpson…
http://localhost:8000/test_urlencode.py?last=Tinkle&first=Ivana
Hello my name is Ivana Tinkle

OK, that’s enough fun for now.  Let’s move on to retrieving input from form data.  The first thing we must do is create a form to enter some data into.  We’ll do this by writing another CGI script to output the HTML form.  Create another file in the ‘server’ directory named ‘form.py’ with the code below.

#!/usr/bin/env python

print """Content-type: text/html

<form method="post" action="test_form.py">
<textarea name="comments" cols="40" rows="5">
Enter comments here...
</textarea>
<br/>
<input type="submit" value="Submit">
</form>"""

This is the form that will be presented to the user to allow them to enter their ‘comments’.  Remember to give this script executable permissions ($ chmod +x form.py).  Next create another file in the ‘server’ directory named ‘test_form.py’, this is the CGI script that will accept the form data and create a new page that includes the data.

#!/usr/bin/env python

import cgi

form = cgi.FieldStorage()

val1 = form.getvalue('comments')

print """Content-type: text/html

The form input is below...<br/>"""
print val1

Give this file executable permissions as well ($ chmod +x test_form.py) and then go back to your browser and enter the following line in the location bar.

http://localhost:8000/form.py

You should see a form with a single text area and a submit button.  Type whatever you want to in the text area, I chose “la de da de, we like to party”, and hit the submit button.  You should be presented by a page that says…

The form input is below…
la de da de, we like to party

Congratulations you have successfully handled form data with a CGI script.  You’ll notice that Python’s cgi module uses cgi.FieldStorage() to access both URL encoded input as well as form based input.  I neglected to go over the scripts line-by-line because there are very few concepts introduced in each of them.  If you have any questions, or would like me to clarify something, please leave a comment below.

To start with we need to create a directory for our server and CGI scripts to reside in. I simply created a directory named ‘server’ in my home directory, but you may name it anything you wish and place it anywhere in your filesystem.

The next step is to create a simple CGI server. To do this open up your favorite text editor and write the program below. (I’ll go through it line-by-line at the end of this tutorial.)

#!/usr/bin/env python

import BaseHTTPServer
import CGIHTTPServer
import cgitb; cgitb.enable()  ## This line enables CGI error reporting

server = BaseHTTPServer.HTTPServer
handler = CGIHTTPServer.CGIHTTPRequestHandler
server_address = ("", 8000)
handler.cgi_directories = [""]

httpd = server(server_address, handler)
httpd.serve_forever()

Save this file in the ‘server’ directory as ‘server.py’, or whatever name tickles your fancy, and give it executable permissions. On Linux systems this is achieved with the command below…

$ chmod +x server.py

That’s it for the server! Simple wasn’t that? Now that we have a server ready to go we need something for it to serve. For that we will create a simple “Hello World!” CGI script. Open up your text editor again and type in the following program.

#!/usr/bin/env python

print "Content-type: text/html"
print
print "<title>Test CGI</title>"
print "<p>Hello World!</p>"

Save this file in the ‘server’ directory as ‘test_cgi.py’ and give it executable permissions, just like the server.py file.

$ chmod +x test_cgi.py

Now we have all the components in place. To start your Python CGI server simply open up a terminal and cd into your ‘server’ directory. When you are there simply type the following command.

$ ./server.py

Your server is now fully operational! To see your first page fire up a browser and type the following into the location bar.

http://localhost:8000/test_cgi.py

You should be greeted by a webpage that proudly proclaims “Hello World!”. If you are not, make sure to check your file permissions (both the server and CGI script should be given executable permissions) and make sure you copied the programs above verbatim. To kill your server simply go back to the terminal and hit ‘ctrl-c’.

Server Code Line-By-Line

For those of you who want a more in-depth analysis of what this code is doing, here is a line-by-line explanation.

The first thing we need to do is make this script executable on the command line. The following ‘she-bang’ line simplifies this process so that we can type: ./server.py instead of the more verbose: python server.py

#!/usr/bin/env python

The next two lines import the necessary libraries. The first, BaseHTTPServer, provides us with a simple web server and CGIHTTPServer provides us with a request handler.

import BaseHTTPServer
import CGIHTTPServer

The following line enables in-browser CGI error reporting. This can be very handy as you develop CGI scripts, however if you don’t want to expose the internals of your program to users you may wish to log the output of this to a file instead.

import cgitb; cgitb.enable()  ## This line enables CGI error reporting

The next two lines create a server and a request handler.

server = BaseHTTPServer.HTTPServer
handler = CGIHTTPServer.CGIHTTPRequestHandler

The next line specifies an address for the server, in this case we want to use ‘localhost’ so simply specifing “” will give us this. This line also specifies a port number to associate with our server. You can choose whatever port number you wish.

server_address = ("", 8000)

The next line specifies where the CGI scripts will reside in relation to the ‘server’ directory. In this example we put our CGI script in the base directory by using “”, however you may perfer to place your scripts in a special ‘cgi’ or ‘cgi-bin’ directory.

handler.cgi_directories = [""]

The next line actually ‘creates’ our server, passing it the server address and port number, as well as the CGI handler. If you’re wondering why we name it httpd it stands for HTTP Daemon.

httpd = server(server_address, handler)

The final line puts our program into an infinite loop so that the server can ‘serve_forever’. Until you kill the server with ‘ctrl-c’ from the terminal, it will continue to listen for requests and serve the appropriate web pages.

httpd.serve_forever()

GOTO Part 2 | A Simple Python CGI Tutorial – CGI Input and Output