First install the readline-dev packages for your OS. For Ubuntu I installed the libreadline6-dev package using apt-get, but this may vary a bit depending on which OS you’re running.

sudo apt-get install libreadline6-dev

Then you need to get the readline egg from the egg repository and install it. Fortunately chicken-install makes this a snap!

chicken-install readline

Then create a new file called .csirc in your home directory and also a .csi.history file:

touch ~/.csirc
touch ~/.csi.history

And put the following in .csirc:

(use readline irregex)
(current-input-port (make-gnu-readline-port))
(gnu-history-install-file-manager
  (string-append (or (get-environment-variable "HOME") ".") "/.csi.history"))

That’s it! Readline and history should now work in Chicken!

Had I been more careful, or paid more attention to the numerous warnings during this process, I would have made backups of my wiki and issue tracker data. Unfortunately I did not and the next day when I went to make some updates to the wiki of one of my projects I was hit with the realization that the data was lost forever.

I’ve only just started using the Github wiki and issue trackers a few months ago, so the data loss was not catastrophic. In total I probably lost about 30 hours worth of work, however it will likely take me twice that to recreate the data and there will be some things that will never be recovered.

Overall I was happy with my (limited) experience of using Github issues and wiki. Being able to reference issues from commits and vice versa was a nice feature and the wiki served its purpose well enough. Previously I kept my issues/to-do list in the source code itself, which worked well enough, but didn’t provide as much flexibility as a dedicated issue tracker. Switching to Github’s issue tracker provided a centralized location for all my bugs and to-do items, as well as a history of completed tasks which had a positive influence on my workflow. For documentation I used README files, and I still do, however a wiki provides a much larger canvas and leads to writing more through documentation.

Having had a taste of these features and the resulting increase productivity and quality of my projects, I do not want to return to the old way of doing things. However my recent experience has enlightened me as to some of the downsides of using these tools.

Issues and documentation are part of your project

The overall insight this experience has given me is that issues and documentation are part of your project. Every file in your project is stored in the same repository, why isn’t the issue tracker and wiki data stored in that same repo?

Granted my mistake was a stupid one, stemming from carelessly performing operations which cannot be undone. However consider the case of moving your central repository from Github to another service or private host. The move isn’t as simple as changing the origin and pushing to the new repo, instead you have to backup the issues/wiki and transfer that in a second step. Sometimes transferring that data is difficult, especially if the tools provided by the new host are radically different.

Having all your issue/wiki data stored in the same repo as your source code means that there are (likely) multiple copies of that data on many different machines. Changing hosts for the central repository is a simple matter of pushing to a new origin. Developers can use their preferred tools to work with issues and documentation.

I’ve done a cursory search for issue trackers/wikis which are designed to be included in the main git repository and have found ticgit which takes care of issue tracking. I haven’t had a chance yet to evaluate it, but plan on doing so in the near future. I haven’t yet found a solution for a wiki.

Ideally I would like both an issue tracker and wiki with a similar interface (as much as that is possible). If I’m unable to find a suitable solution I will likely take this up as my next project. Here are some of the features that I’m looking for:

Version Control

Changes to issues or wiki should be tracked by version control.

Users

New issues or changes should be associated with a user (git name + email + key).

Editing

Editing should be allowed through a number of tools (web, command line, text editor/IDE) and should support markdown format.

Viewing

Viewing the tracker/wiki should be allowed through a number of tools (web, editor/IDE).

Alerts

A program can monitor the repo for changes and alert users via email when changes occur.

Unfortunately the speed at which these features have been added has ended the step-by-step nature of this series. I will continue to provide progress updates and hope to produce a number of articles based on architecture or challenges that arise during development. Though there is not much discussion, you can still follow along with the commit log of the project. Each commit is relatively small and, in most cases, addresses a single concern. There is also some information provided in the issue tracker, which I have been making heavy use of and almost every commit has an associated issue.

The project is not yet ready for release, however there is a preview version available. Criticisms, comments, bug reports and feature requests are more than welcome. You can leave comments below or file a bug report or feature request on the issue tracker.

Try the demo

Features:

• 5 chart types (candlestick, bar, ohlc, hlc, line)
• 15, 30, 60 and 90 day charts
• Change chart colors
• Advance time by day, week or month (animated)
• Buy/sell stocks
• Portfolio view
• Sell directly from portfolio view
• Market, limit and stop orders
• Pending order pane
• Cancel pending orders
• Change theme of site
• Previously viewed stock data is cached for fast switching
• Most features auto-hide when not necessary to reduce clutter
• Account balance and total portfolio value show gains/losses

getElementById sometimes returns elements by name in some versions of IE and Opera.

                                // HANDLE: $("#id")
                                } else {
                                        elem = document.getElementById( match[2] );

                                        // Check parentNode to catch when Blackberry 4.6 returns
                                        // nodes that are no longer in the document #6963
                                        if ( elem && elem.parentNode ) {
                                                // Handle the case where IE and Opera return items
                                                // by name instead of ID
                                                if ( elem.id !== match[2] ) {
                                                        return rootjQuery.find( selector );
                                                }

                                                // Otherwise, we inject the element directly into the jQuery object
                                                this.length = 1;
                                                this[0] = elem;
                                        }

                                        this.context = document;
                                        this.selector = selector;
                                        return this;
                                }

Even something as simple as getElementById can have edge cases you have to take into account. In IE 6 and 7 for Windows and some versions of Opera, dom elements can be returned by name instead of id. In addition, as seen in the comments of the code above, Blackberry 4.6 can return nodes that are not even in the document.

See references here and here.

A false positive on the domready event can cause rapid page refreshing.

        // Handle when the DOM is ready
        ready: function( wait ) {
                // Either a released hold or an DOMready/load event and not yet ready
                if ( (wait === true && !--jQuery.readyWait) || (wait !== true && !jQuery.isReady) ) {
                        // Make sure body exists, at least, in case IE gets a little overzealous (ticket #5443).
                        if ( !document.body ) {
                                return setTimeout( jQuery.ready, 1 );
                        }

The domready event, which is present in some form or another in many browsers can fire before the dom is actually loaded. This can cause continuous page reloads and other errors which are hard to reproduce and debug. The solution to this is to check that the document.body is not null before accepting the domready event.

Of course you can always fall back to the window.onload event, however this will force you to wait until the entire content of the page (including possibly large images) has loaded.

See references here and here.

We are finally going to implement lambda and procedures.

To begin implementing procedures we have to define a procedure type, just like we did with primitives. A procedure has 3 basic parts. A list of parameters, a body and an environment. So we’ll set up a new procedure type with these fields and call it Procedure.

scheme_types.py

class Symbol(str):
  """A symbol is an immutable string, but must be its own type."""
  pass

class The_Empty_List():
  """The empty list is used as a list terminator and should be a singleton"""
  def __repr__(self):
    return "()"

the_empty_list = The_Empty_List()

class Pair(object):
  """A pair is a classic Lisp cons cell used to implement lists"""
  def __init__(self, car, cdr):
    self.car = car
    self.cdr = cdr
  def __iter__(self):
    x = self
    while not isinstance(x, The_Empty_List):
      yield x.car
      x = x.cdr
  def __repr__(self):
    return "(" + ' '.join([str(i) for i in list(self)]) + ")"

class Primitive(object):
  def __init__(self, fn):
    self.fn = fn

class Procedure(object):
  def __init__(self, parameters, body, environment):
    self.parameters = parameters
    self.body = body
    self.environment = environment

Like primitives, procedures are handled by the scheme_apply procedure. We’ll temporarily add a placeholder in scheme_apply for procedures. In addition we need new syntax to create a procedure. The syntax for procedures in Scheme is (lambda (parameters) body). Most Schemes allow the syntax (define (name params …) body …) but this is really just shorthand for (define name (lambda (params …) body …)). For now we won’t overload the define keyword, choosing instead to define functions the more verbose way.

It is important to take note of this before moving on. In many programming languages defining a function is a single step. However logically, there are 2 different operations that take place. First a function object is created, then that function object is bound to a symbol. By requiring the (define name (lambda …)) syntax we are making these two steps explicit.

scheme_eval.py

from scheme_types import Symbol, Pair, Primitive, the_empty_list, Procedure
from buffered_input import Buff
from scheme_read import scheme_read

##############################################################################
## Environments
##############################################################################

def current_environment(env):
  """Return the frame of the current environment"""
  return env.car

def enclosing_environment(env):
  """Return the environment stack with the first frame removed"""
  return env.cdr

def extend_environment(bindings, base_environment):
  """Push a new frame onto the given base_environment"""
  return Pair(bindings, base_environment)

global_environment = extend_environment({}, the_empty_list)
special_forms = {}

def set_symbol(symbol, val, env):
  """Set the binding (symbol, val) in the current frame of env"""
  current_environment(env)[symbol] = val

def lookup_symbol_value(symbol, environment):
  """Return the value of symbol or Unbound Symbol error"""
  env = environment
  while env != the_empty_list:
    if symbol in current_environment(env):
      return current_environment(env)[symbol]
    else:
      env = enclosing_environment(env)
  return "Error: Unbound symbol: " + symbol

##############################################################################
## Eval and Apply
##############################################################################

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr, env):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr, env)
  elif type(expr) is Pair:
    if expr.car in special_forms:
      return special_forms[expr.car](expr.cdr, env)
    else:
      return scheme_apply(scheme_eval(expr.car, env), [scheme_eval(a, env) for a in expr.cdr])
  else:
    return "scheme_eval: not implemented"

def scheme_apply(proc, args):
  if type(proc) is Primitive:
    return apply(proc.fn, args)
  elif type(proc) is Procedure:
    return "Attempted to apply a procedure"
  else:
    return "Error: Undefined procedure"

##############################################################################
## Builtin Syntax
##############################################################################

def special_form_handler(expr, env):
  """Register a symbol with a Python function named "f" that implements a special form"""
  exec(expr.cdr.car)
  special_forms[expr.car] = f

def load(expr):
  """Given a filename, open it and eval each expression in the global_environment"""
  f = open(expr.car, 'r')
  b = Buff(f)
  while b.peek():
    scheme_eval(scheme_read(b), global_environment)
    b.remove_whitespace()
  f.close()

special_forms['scheme-syntax'] = special_form_handler
special_forms['load'] = load

syntax.scm

(scheme-syntax define-primitive "
def f(expr, env):
  set_symbol(expr.car, Primitive(eval(expr.cdr.car)), env)
")

(scheme-syntax define "
def f(expr, env):
  set_symbol(expr.car, scheme_eval(expr.cdr.car, env), env)
")

(scheme-syntax if "
def f(expr, env):
  if scheme_eval(expr.car, env):
    return scheme_eval(expr.cdr.car, env)
  else:
    return scheme_eval(expr.cdr.cdr.car, env)
")

(scheme-syntax lambda "
def f(expr, env):
  return Procedure(expr.car, expr.cdr.car, env)
")

(define a (lambda (x) x))
a
;===> <scheme_types.Procedure object at 0xb733b10c>
(a 42)
;===> Attempted to apply a procedure

Now that we have a basic outline for what our procedures will look like, we can focus on making them work!

To apply a procedure we have to evaluate the body. In the example above the body of the procedure is just x. However we can’t just evaluate x because x is not bound to anything yet.

First we need to create a new environment and bind the parameters to the arguments supplied in the procedure call. In this case we have to bind x to the value 42. Then we just evaluate each expression in the body within the new environment and return the result of evaluating the last expression.

from scheme_types import Symbol, Pair, Primitive, the_empty_list, Procedure
from buffered_input import Buff
from scheme_read import scheme_read

##############################################################################
## Environments
##############################################################################

def current_environment(env):
  """Return the frame of the current environment"""
  return env.car

def enclosing_environment(env):
  """Return the environment stack with the first frame removed"""
  return env.cdr

def extend_environment(bindings, base_environment):
  """Push a new frame onto the given base_environment"""
  return Pair(bindings, base_environment)

global_environment = extend_environment({}, the_empty_list)
special_forms = {}

def set_symbol(symbol, val, env):
  """Set the binding (symbol, val) in the current frame of env"""
  current_environment(env)[symbol] = val

def lookup_symbol_value(symbol, environment):
  """Return the value of symbol or Unbound Symbol error"""
  env = environment
  while env != the_empty_list:
    if symbol in current_environment(env):
      return current_environment(env)[symbol]
    else:
      env = enclosing_environment(env)
  return "Error: Unbound symbol: " + symbol

##############################################################################
## Eval and Apply
##############################################################################

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr, env):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr, env)
  elif type(expr) is Pair:
    if expr.car in special_forms:
      return special_forms[expr.car](expr.cdr, env)
    else:
      return scheme_apply(scheme_eval(expr.car, env), [scheme_eval(a, env) for a in expr.cdr])
  else:
    return "scheme_eval: not implemented"

def scheme_apply(proc, args):
  if type(proc) is Primitive:
    return apply(proc.fn, args)
  elif type(proc) is Procedure:
    new_env = extend_environment(dict(zip(proc.parameters, args)), proc.environment)
    return [scheme_eval(e,new_env) for e in proc.body][-1]
  else:
    return "Error: Undefined procedure"

##############################################################################
## Builtin Syntax
##############################################################################

def special_form_handler(expr, env):
  """Register a symbol with a Python function named "f" that implements a special form"""
  exec(expr.cdr.car)
  special_forms[expr.car] = f

def load(expr):
  """Given a filename, open it and eval each expression in the global_environment"""
  f = open(expr.car, 'r')
  b = Buff(f)
  while b.peek():
    scheme_eval(scheme_read(b), global_environment)
    b.remove_whitespace()
  f.close()

special_forms['scheme-syntax'] = special_form_handler
special_forms['load'] = load

(define test (pred conseq alt)
  (if pred conseq alt))
(test 1 2 3)
;===> 2
(define-primitive + "lambda x,y: x+y")
(define-primitive < "lambda x,y: x<y")
(define a (lambda (x y) (if (< x y) (a (+ x 1) y) x)))
(a 1 3)
;===> 3

With that, we have implemented lambda in Scheme! However there are some glaring omissions from our implementation. For example ((lambda () 42)) will result in an iteration over non-sequence error from trying to iterate over the_empty_list. Also, tail-call elimination is not implemented as required by the Scheme standard.

Also note that there is very little error-checking going on here. For instance calling a procedure with the wrong number of arguments results in the whole interpreter crashing. This would not be good in a production language. However including error checking in this code would probably triple its size at least. The point of this exercise is to learn the concepts, not write a bullet-proof language.

I will likely come back to this at some point in the future and address some of these concerns. The rest is left for you to experiment with. You can get more information on how and what to implement from R5RS.

I will be Jumping back into Javascript for the next few posts at least, so this series will not be updated for a while (with the exception of bug fixes). When I do get back to language design and implementation, I will probably pick back up with the Scheme version of this interpreter, or possibly with a new project on compiling.

If you have any questions about how to proceed from here, or just want to show off your own version, please leave a comment.

Checkout this version on GitHub (v0.10)

GOTO | Table of Contents

We’ve been dealing with an environment ever since part 5 in this series when we implemented define, except we’ve been calling it a frame. Until now our frame has served us well, it has provided a simple environment where we could set and retrieve the value of symbols. However implementing procedures requires us to add a few features to our frame. Consider the code below.

(define x 5)
(define (func x)
  x)
(func 100)
;===> 100

With our frame this wouldn’t work as expected. Because we have already defined x in the global environment, there will be one of two problems with the x in func. Our function could use the global value of x (5) instead of the supplied value (100) or when the supplied value (100) is mapped onto the function body it will overwrite the value of the global value of x (5) with the supplied value (100).

Currently we do not make any distinction between variables in a function body and in the global environment. To properly implement procedures we will have to figure out a way to do that. Consider the next bit of code.

(define x 5)
(define (func y)
  (+ x y))
(func 10)
;===> 15

Even though we need to make a distinction between environments they also need to be connected. In the example above we use the global variable x in func without ever declaring it. This is accomplished by a method called chaining environments.

(define x 1)
(define (outer-func y)
  (define (inner-func z)
    (+ x y z)
  (inner-func 3))
(outer-func 2)
;===> 6

The environments produced by the above code are best described by the diagram below.

So when x appears in inner-func we have to check inner-func’s environment for x, then outer-func’s environment for x and finally the global environment for x before we raise an undefined symbol error.

You may have noticed that the diagram above looks a lot like a linked list. We can take our frame and combine it with linked lists to form an environment. It is more informative (and closer to the actual design) to view environments from the point of view of eval. This diagram is a view of the environment structure when the code (+ x y z) is evaluated.

I used the names current-environment to refer to the inner-func environment and enclosing-environment to refer to the outer-func environment. We will use these same names in our source code. Note that the global-environment is just the enclosing-environment of outer-func, the only thing special about it is that its enclosing-environment is the empty list.

In addition we need to replace our global frame with global_environment, which we’ll do by providing a procedure extend_environment, and change lookup_symbol_value to check enclosing-environments if the symbol is not found.

scheme_eval.py

from scheme_types import Symbol, Pair, Primitive, the_empty_list
from buffered_input import Buff
from scheme_read import scheme_read

def current_environment(env):
  """Return the frame of the current environment"""
  return env.car

def enclosing_environment(env):
  """Return the environment stack with the first frame removed"""
  return env.cdr

def extend_environment(bindings, base_environment):
  """Push a new frame onto the given base_environment"""
  return Pair(bindings, base_environment)

global_environment = extend_environment({}, the_empty_list)
special_forms = {}

def set_symbol(symbol, val, env):
  """Set the binding (symbol, val) in the current frame of env"""
  current_environment(env)[symbol] = val

def lookup_symbol_value(symbol, environment):
  """Return the value of symbol or Unbound Symbol error"""
  env = environment
  while env != the_empty_list:
    if symbol in current_environment(env):
      return current_environment(env)[symbol]
    else:
      env = enclosing_environment(env)
  return "Error: Unbound symbol: " + symbol

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr, global_environment)
  elif type(expr) is Pair:
    if expr.car in special_forms:
      return special_forms[expr.car](expr.cdr)
    else:
      return scheme_apply(scheme_eval(expr.car), [scheme_eval(a) for a in expr.cdr])
  else:
    return "scheme_eval: not implemented"

def scheme_apply(proc, args):
  if type(proc) is Primitive:
    return apply(proc.fn, args)
  else:
    return "Error: Undefined procedure"

def special_form_handler(expr):
  """Register a symbol with a Python function named "f" that implements a special form"""
  exec(expr.cdr.car)
  special_forms[expr.car] = f

def load(expr):
  """Given a filename, open it and eval each expression in the global_environment"""
  f = open(expr.car, 'r')
  b = Buff(f)
  while b.peek():
    scheme_eval(scheme_read(b))
    b.remove_whitespace()
  f.close()

special_forms['scheme-syntax'] = special_form_handler
special_forms['load'] = load

We also need to make some changes to syntax.scm to use the new set_symbol function.

syntax.scm

(scheme-syntax define-primitive "
def f(expr):
  set_symbol(expr.car, Primitive(eval(expr.cdr.car)), global_environment)
")

(scheme-syntax define "
def f(expr):
  set_symbol(expr.car, scheme_eval(expr.cdr.car), global_environment)
")

(scheme-syntax if "
def f(expr):
  if scheme_eval(expr.car):
    return scheme_eval(expr.cdr.car)
  else:
    return scheme_eval(expr.cdr.cdr.car)
")

(define a 5)
;===> #<unspecified>
a
;===> 5

Before we go any further I’m going to add in some comments to the scheme_eval.py file, mainly to group code into logical sections. I won’t provide a highlighted diff here, but you will notice the sections in the next bit of code.

To use the full potential of our new environments we are going to have to rewrite our program a little bit to pass them around correctly. Since eval is the main method of dispatch for our interpreter we’ll supply the environment as an argument to eval, which means finding every use of scheme_eval and adding the env argument. We’ll also have to pass an environment variable to procedures like set_symbol and lookup_symbol_value. One final place we’ll need environments is in scheme-syntax, we’ll have to change the stored procedures there to use “def f(expr, env):” instead of just “def f(expr):”.

from scheme_types import Symbol, Pair, Primitive, the_empty_list
from buffered_input import Buff
from scheme_read import scheme_read

##############################################################################
## Environments
##############################################################################

def current_environment(env):
  """Return the frame of the current environment"""
  return env.car

def enclosing_environment(env):
  """Return the environment stack with the first frame removed"""
  return env.cdr

def extend_environment(bindings, base_environment):
  """Push a new frame onto the given base_environment"""
  return Pair(bindings, base_environment)

global_environment = extend_environment({}, the_empty_list)
special_forms = {}

def set_symbol(symbol, val, env):
  """Set the binding (symbol, val) in the current frame of env"""
  current_environment(env)[symbol] = val

def lookup_symbol_value(symbol, environment):
  """Return the value of symbol or Unbound Symbol error"""
  env = environment
  while env != the_empty_list:
    if symbol in current_environment(env):
      return current_environment(env)[symbol]
    else:
      env = enclosing_environment(env)
  return "Error: Unbound symbol: " + symbol

##############################################################################
## Eval and Apply
##############################################################################

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr, env):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr, env)
  elif type(expr) is Pair:
    if expr.car in special_forms:
      return special_forms[expr.car](expr.cdr, env)
    else:
      return scheme_apply(scheme_eval(expr.car, env), [scheme_eval(a, env) for a in expr.cdr])
  else:
    return "scheme_eval: not implemented"

def scheme_apply(proc, args):
  if type(proc) is Primitive:
    return apply(proc.fn, args)
  else:
    return "Error: Undefined procedure"

##############################################################################
## Builtin Syntax
##############################################################################

def special_form_handler(expr, env):
  """Register a symbol with a Python function named "f" that implements a special form"""
  exec(expr.cdr.car)
  special_forms[expr.car] = f

def load(expr):
  """Given a filename, open it and eval each expression in the global_environment"""
  f = open(expr.car, 'r')
  b = Buff(f)
  while b.peek():
    scheme_eval(scheme_read(b), global_environment)
    b.remove_whitespace()
  f.close()

special_forms['scheme-syntax'] = special_form_handler
special_forms['load'] = load

repl.py

import sys
from scheme_read import scheme_read
from scheme_eval import scheme_eval, special_forms, global_environment
from scheme_types import Pair
from buffered_input import Buff

special_forms['load'](Pair("syntax.scm", None))

while True:
  inp = scheme_eval(scheme_read(Buff(sys.stdin)), global_environment)
  if inp != None:
    print ';===>', inp

syntax.chicken

(scheme-syntax define-primitive "
def f(expr, env):
  set_symbol(expr.car, Primitive(eval(expr.cdr.car)), env)
")

(scheme-syntax define "
def f(expr, env):
  set_symbol(expr.car, scheme_eval(expr.cdr.car, env), env)
")

(scheme-syntax if "
def f(expr, env):
  if scheme_eval(expr.car, env):
    return scheme_eval(expr.cdr.car, env)
  else:
    return scheme_eval(expr.cdr.cdr.car, env)
")

(define a 5)
a
;===> 5
(if 1 2 3)
;===> 2
(if #f 1 2)
;===> 2
(define-primitive + "lambda x,y: x+y")
(+ 3 5)
;===> 8

If you still don’t fully understand environments, that’s OK. You’ll get a chance to play around with them in the next part where we’ll finally cover implementing procedures in Lispy!

Checkout this version on GitHub (v0.09)

GOTO Part 10 | lambda

GOTO | Table of Contents

The goal for this part is to implement primitive procedures, apply and a method of setting new primitives.

The first thing we need to do is figure out a way to represent our primitive procedures. SICP uses tagged lists, which are just regular lists with the first element designated as a tag. Python uses objects to define new types and since we’ve already been doing that, we’ll continue here.

class Symbol(str):
  """A symbol is an immutable string, but must be its own type."""
  pass

class The_Empty_List():
  """The empty list is used as a list terminator and should be a singleton"""
  def __repr__(self):
    return "()"

the_empty_list = The_Empty_List()

class Pair(object):
  """A pair is a classic Lisp cons cell used to implement lists"""
  def __init__(self, car, cdr):
    self.car = car
    self.cdr = cdr
  def __iter__(self):
    x = self
    while not isinstance(x, The_Empty_List):
      yield x.car
      x = x.cdr
  def __repr__(self):
    return "(" + ' '.join([str(i) for i in list(self)]) + ")"

class Primitive(object):
  def __init__(self, fn):
    self.fn = fn

If the symbol of an expression of the form (symbol args …) is not found in special_forms it must be a procedure application (or not exist). To apply a primitive procedure first we have to evaluate all the arguments using scheme_eval. Then we use that list as the args value to apply(primitive, args). The primitive returns a Scheme value and we continue on.

For now we’re going to hardcode one primitive + into our interpreter. We’ll come up with something a little more elegant next.

from scheme_types import Symbol, Pair, Primitive
from buffered_input import Buff
from scheme_read import scheme_read

frame = {'+':Primitive(lambda x,y: x+y)}
special_forms = {}

def lookup_symbol_value(symbol):
  try:
    return frame[symbol]
  except KeyError:
    return "Error: Unbound variable"

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr)
  elif type(expr) is Pair:
    if expr.car in special_forms:
      return special_forms[expr.car](expr.cdr)
    else:
      return scheme_apply(scheme_eval(expr.car), [scheme_eval(a) for a in expr.cdr])
  else:
    return "scheme_eval: not implemented"

def scheme_apply(proc, args):
  if type(proc) is Primitive:
    return apply(proc.fn, args)
  else:
    return "Error: Undefined procedure"

def special_form_handler(expr):
  """Register a symbol with a Python function named "f" that implements a special form"""
  exec(expr.cdr.car)
  special_forms[expr.car] = f

def load(expr):
  """Given a filename, open it and eval each expression in the global_environment"""
  f = open(expr.car, 'r')
  b = Buff(f)
  while b.peek():
    scheme_eval(scheme_read(b))
    b.remove_whitespace()
  f.close()

special_forms['scheme-syntax'] = special_form_handler
special_forms['load'] = load

(+ 3 4)
;===> 7
(- 3 4)
;===> Error: Undefined procedure

The next step is to allow defining primitive procedures from within Scheme, as we did with special forms. We’ll implement define-primitive as a scheme-syntax macro. First remove the line that sets the + symbol. We’re only going to be modifying the syntax.scm file for now so that’s all I’ll show you.

We need to create a scheme-syntax macro that makes a Primitive object and sets it as the value of a symbol. This macro is a lot like the define macro that we wrote earlier. The only difference is instead of calling scheme_eval on the cadr we call eval and pass that result to make-primitive.

syntax.scm

(scheme-syntax define-primitive "
def f(expr):
  frame[expr.car] = Primitive(eval(expr.cdr.car))
")

(scheme-syntax define "
def f(expr):
  frame[expr.car] = scheme_eval(expr.cdr.car)
")

(scheme-syntax if "
def f(expr):
  if scheme_eval(expr.car):
    return scheme_eval(expr.cdr.car)
  else:
    return scheme_eval(expr.cdr.cdr.car)
")

With that we can define all sorts of primitives for our language…

(define-primitive + "lambda x,y: x+y")
(+ 3 4)
;===> 7
(- 3 4)
;===> Error: Undefined procedure

(define-primitive - "lambda x,y: x-y")
(- 3 4)
;===> -1

(define-primitive square "lambda x: x*x")
(square 4)
;===> 16

(define-primitive len "len")
(len "hello")
;===> 5

You can start a new file and define a bunch of primitives, load it the same way you load syntax.scm. I’m not going to define any primitives just yet but you are more than welcome to. Instead in the next post we’ll implement environments and then it’s on to implementing lambda!

Checkout this version on GitHub (v0.08)

GOTO Part 9 | Environments

GOTO | Table of Contents

In this part we’re going to be doing a bit of refactoring, so let’s start with that and a recap. The first thing that we’re going to do is remove define and if from our language. We’ll put them back in later, but first let’s look at what our language looks like with only scheme-syntax defined.

from scheme_types import Symbol, Pair

frame = {}
special_forms = {}

def lookup_symbol_value(symbol):
  try:
    return frame[symbol]
  except KeyError:
    return "Error: Unbound variable"

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr)
  elif type(expr) is Pair:
    if expr.car in special_forms:
      return special_forms[expr.car](expr.cdr)
    else:
      return "Error: unbound special form"
  else:
    return "scheme_eval: not implemented"

def special_form_handler(expr):
  """Register a symbol with a Python function named "f" that implements a special form"""
  exec(expr.cdr.car)
  special_forms[expr.car] = f

special_forms['scheme-syntax'] = special_form_handler

Let’s take a moment to consider what we can do so far. We now have a language that can understand self-evaluating Scheme values and lookup symbol values. In addition to that it has exactly one primitive, a macro that defines other primitives by giving us access to Python.

You may consider this language lacking, and I admit it does not have many features. However it is now possible to define any language construct (including procedures, eval and apply) from within Scheme with the help of scheme-syntax and Python underneath. After writing this code we could define the rest of the language 100% from within Scheme (though it may not be easy).

However there are a few more things that conceptually belong in the core language. One of those things is the ability to load a Scheme file. This will let us break out our syntax definitions into another file and load it automatically before we start the repl.

from scheme_types import Symbol, Pair
from buffered_input import Buff
from scheme_read import scheme_read

frame = {}
special_forms = {}

def lookup_symbol_value(symbol):
  try:
    return frame[symbol]
  except KeyError:
    return "Error: Unbound variable"

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr)
  elif type(expr) is Pair:
    if expr.car in special_forms:
      return special_forms[expr.car](expr.cdr)
    else:
      return "Error: unbound special form"
  else:
    return "scheme_eval: not implemented"

def special_form_handler(expr):
  """Register a symbol with a Python function named "f" that implements a special form"""
  exec(expr.cdr.car)
  special_forms[expr.car] = f

def load(expr):
  """Given a filename, open it and eval each expression in the global_environment"""
  f = open(expr.car, 'r')
  b = Buff(f)
  while b.peek():
    scheme_eval(scheme_read(b))
    b.remove_whitespace()
  f.close()

special_forms['scheme-syntax'] = special_form_handler
special_forms['load'] = load

Now that we have load, we need something to load. Create a new file called syntax.scm, or whatever you prefer. This is where we will define our primitive syntax forms like define, if and lambda. For this moment we’re going to leave it blank, and make the repl load it by default so we don’t have to do it manually each time.

Also I’m going to make some changes to the repl to make our sessions look more like the examples I’ve been giving (so ;===> will come before the output and there will be no prompt for input). Later I may go back to the beginning of this series and change this, as this is how it should have been from the start. I’ll also take this opportunity to skip printing any None value which will make using expressions that don’t return values more aesthetically pleasing.

repl.py

import sys
from scheme_read import scheme_read
from scheme_eval import scheme_eval, special_forms
from scheme_types import Pair
from buffered_input import Buff

special_forms['load'](Pair("syntax.scm", None))

while True:
  inp = scheme_read(Buff(sys.stdin))
  if inp != None:
    print ';===>', scheme_eval(inp)

Now we can define some special forms in syntax.scm to load by default. For now we’ll just reimplement define and if.

syntax.scm

(scheme-syntax define "
def f(expr):
  special_forms[expr.car] = scheme_eval(expr.cdr.car)
")

(scheme-syntax if "
def f(expr):
  if scheme_eval(expr.car):
    return scheme_eval(expr.cdr.car)
  else:
    return scheme_eval(expr.cdr.cdr.car)
")

(define a 5)
a
;===> 5
(if 1 2 3)
;===> 2

Now that we have implemented load we are one step closer to having a platform that we can build almost any language we want with. To change languages all we would have to do is load a different syntax file.

I encourage you to play around with that concept for a little while. Write two different syntax.scm files and implement a couple of basic forms like define, or and if. Try to make your two implementations as different as possible even if making it different doesn’t make sense. For instance in one implementation if could take a mandatory keyword before the alternate branch (if pred consequent else alt) or it could act like and, taking two predicates and only evaluating the consequent branch if both predicates evaluate to true.

This article is a little short on code because we made some larger conceptual changes that I did not want to breeze past. In the next article we’ll implement primitive procedures and apply. The ability to define procedures from within Scheme is just around the corner.

Checkout this version on GitHub (v0.07)

GOTO Part 8 | Primitive procedures and apply

GOTO | Table of Contents

In the last part we implemented our first special form and the first expression that was not self-evaluating, define. The goal for this part is to implement (if predicate consequent alternate).

We’re going to start by implementing if the same way we implemented define. The first thing to do is test if the car of the pair equals if, then call a helper procedure we’ll name apply_if.

We know that if will have to evaluate the predicate argument to determine whether to evaluate the consequent or alternate arguments. If the predicate is true we will evaluate the consequent expression and not the alternate. If the predicate is false we will evaluate the alternate expression and not the consequent.

from scheme_types import Symbol, Pair

frame = {}

def lookup_symbol_value(symbol):
  try:
    return frame[symbol]
  except KeyError:
    return "Error: Unbound variable"

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr)
  elif type(expr) is Pair:
    if expr.car == "define":
      frame[expr.cdr.car] = scheme_eval(expr.cdr.cdr.car)
      return "set symbol -> value"
    elif expr.car == "if":
      return apply_if(expr.cdr.car, expr.cdr.cdr.car, expr.cdr.cdr.cdr.car)
    else:
      return "scheme_eval: not implemented"
  else:
    return "scheme_eval: not implemented"

def apply_if(predicate, consequent, alternate):
  if scheme_eval(predicate):
    return scheme_eval(consequent)
  else:
    return scheme_eval(alternate)

(if 1 2 3)
;===> 2
(if #f 2 3)
;===> 3

Dispatching special forms using if is fine while you have a small number of primitive syntax forms. However once you implement a large number of primitive syntax forms this method of dispatch becomes cumbersome. Also note that it is not possible to extend the primitive syntax forms except by modifying the original source by hand.

Reusing the Frame

On top of that we have already implemented a mechanism of associating and storing symbol and value pairs. Let’s try something like that first, we’ll start by making another global dictionary called special_forms. Then we’ll modify eval to check this dictionary instead of using if/elif.

from scheme_types import Symbol, Pair

frame = {}
special_forms = {}

def lookup_symbol_value(symbol):
  try:
    return frame[symbol]
  except KeyError:
    return "Error: Unbound variable"

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr)
  elif type(expr) is Pair:
    if expr.car in special_forms:
      return special_forms[expr.car](expr.cdr)
    else:
      return "Error: unbound special form"
  else:
    return "scheme_eval: not implemented"

def define(expr):
  frame[expr.car] = scheme_eval(expr.cdr.car)
  return "set symbol -> value"

def apply_if(expr):
  if scheme_eval(expr.car):
    return scheme_eval(expr.cdr.car)
  else:
    return scheme_eval(expr.cdr.cdr.car)

special_forms['define'] = define
special_forms['if'] = apply_if

(repl)
(define a 5)
;===> #<unspecified>
a
;===> 5
(if 1 2 3)
;===> 2
(hello)
;===> Not implemented

With that addition we are finished with the first level of the eval function. An expression can only be one of three things: self-evaluating, a symbol or an application (either syntax or procedure which we’ll cover later).

How does it work? special_forms stores (symbol, procedure) pairs where the symbol is a keyword (like define or if) and the procedure is the Python code that implements that special form. Every procedure in special_forms takes a single argument, an unevaluated pair representing the rest of the expression passed to scheme_eval. Because special forms have their own evaluation rules, it is the responsibility of this procedure to evaluate the expression and return a result.

In other words, when lispy-eval comes across a special form it transfers control to a Python procedure located in the special_forms dict. That Python procedure is responsible for evaluating the arguments and returning a Scheme value.

Now that we have a data structure to hold our special_forms it would be a good idea to supply an easy mechanism to set a new special form. For this purpose we’ll design a special form in Scheme that will set new special forms called scheme-syntax.

from scheme_types import Symbol, Pair

frame = {}
special_forms = {}

def lookup_symbol_value(symbol):
  try:
    return frame[symbol]
  except KeyError:
    return "Error: Unbound variable"

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr)
  elif type(expr) is Pair:
    if expr.car in special_forms:
      return special_forms[expr.car](expr.cdr)
    else:
      return "Error: unbound special form"
  else:
    return "scheme_eval: not implemented"

def special_form_handler(expr):
  """Register a symbol with a Python function named "f" that implements a special form"""
  exec(expr.cdr.car)
  special_forms[expr.car] = f

special_forms['scheme-syntax'] = special_form_handler

def define(expr):
  frame[expr.car] = scheme_eval(expr.cdr.car)
  return "set symbol -> value"

def apply_if(expr):
  if scheme_eval(expr.car):
    return scheme_eval(expr.cdr.car)
  else:
    return scheme_eval(expr.cdr.cdr.car)

special_forms['define'] = define
special_forms['if'] = apply_if

(scheme-syntax quote "def f(expr): return expr.car")
;===> None
(1 2 3)
;===> Error: unbound special form
(quote (1 2 3))
;===> (1 2 3)

With the addition of scheme-syntax it is now possible to write Python directly in our interpreter. Because of this we can now implement all of the primitive syntax forms in Scheme. Some might consider this cheating, but eventually we’re going to want a way to interface with Python. Why not implement it right away, use it, test it and make it solid by implementing everything else in it? We’ll still have to make some modifications to the core interpreter to implement procedures and a load form later.

Because of Python’s restrictions on lambda expressions, this solution is not as elegant as the scheme-syntax form in Scheme in Scheme. For this version you must define a named function, and it must be named “f” to make this implementation work. If you know of a more elegant way to approach this please let me know in the comments.

We haven’t even gotten to the mutually recursive yin-yang that is the eval/apply cycle yet. However, thanks to piggy-backing on Python, we already have a somewhat useful language. A language without procedures is a pretty bad one, so we’ll keep pressing on, but check out some of the things that are possible already…

(scheme-syntax print "
def f(expr):
  print scheme_eval(expr.car)
  return None
")
;===> None
(scheme-syntax loop "
def f(expr):
  for n in range(scheme_eval(expr.car)):
    scheme_eval(expr.cdr.car)
")
;===> None
(loop 3 (print "Hi"))
;===> Hi
;===> Hi
;===> Hi
;===> None

In the next article we’ll do a little refactoring, remove define and if from the core of the interpreter and implement a load function so we can remove our syntax definitions out of the main interpreter file.

Checkout this version on GitHub (v0.06)

GOTO Part 7 | Refactor and load

GOTO | Table of Contents

Our goal in this part is to implement assignment via the form (define symbol value). There are three challenges to this goal. First we need to come up with a system to store symbol names and their values. Then we need to display those values when the symbol is eval’d. Finally we need to recognize the define form and set the value of its symbol.

Starting with a Stub

To start with we need to modify our eval procedure to recognize symbols. We’ll call a procedure lookup_symbol_value that will eventually give the value of the symbol. Right now we’ll just have it return a string stating our intentions.

from scheme_types import Symbol

def lookup_symbol_value(symbol):
  return "Here's where we lookup the value of the symbol."

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr)
  else:
    return "scheme_eval: not implemented"

The Global Frame

To set the value of a symbol we need to have somewhere to set it. That somewhere is going to be a dictionary. The dictionary that stores a set of names and their associated values is referred to as a frame.

For now we’ll define our frame as a global variable named frame. Since we don’t have any method of assigning values to symbols yet, we define a symbol “test” when we make the frame so we can see if it works.

from scheme_types import Symbol

frame = {'test':"Value retrieved successfully"}

def lookup_symbol_value(symbol):
  try:
    return frame[symbol]
  except KeyError:
    return "Error: Unbound variable"

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr)
  else:
    return "scheme_eval: not implemented"

test
;===> Value retrieved successfully
hello
;===> Error: Unbound variable

(define symbol value)

Now that we can store and retrieve the value of a symbol, we need a way to do it from within Scheme. Scheme uses the form (define symbol value) to define a new symbol or set the value of an existing symbol.

The form (define symbol value) is a pair (also a list, but all lists are also pairs) so the first step to detecting its usage is to look for pairs. The next step is to check the symbol in the car of that pair (or the first item of the list), if that symbol is “define” we run the code to define a symbol. For now we’ll just return a string stating our intentions.

from scheme_types import Symbol, Pair

frame = {'test':"Value retrieved successfully"}

def lookup_symbol_value(symbol):
  try:
    return frame[symbol]
  except KeyError:
    return "Error: Unbound variable"

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr)
  elif type(expr) is Pair:
    if expr.car == "define":
      return "set the value of the symbol"
    else:
      return "scheme_eval: not implemented"
  else:
    return "scheme_eval: not implemented"

test
;===> Value retrieved successfully
(define a 1)
;===> Set the value of the symbol.
(set a 1)
;===> scheme_eval: not implemented

Setting the Value of Symbols

To actually set the value of the symbol we need to get the arguments to define. Getting the symbol argument is straightforward, however we need to eval the value argument as it might not be self-evaluating.

Usually (define symbol value) will return void, or nothing. I have chosen to return a string here to give some visual feedback that setting the symbol happened as well as avoid having to modify the repl to handle None. You can choose to take the more standard approach and return None here, then modify the repl to skip printing None values.

from scheme_types import Symbol, Pair

frame = {}

def lookup_symbol_value(symbol):
  try:
    return frame[symbol]
  except KeyError:
    return "Error: Unbound variable"

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) is Symbol:
    return lookup_symbol_value(expr)
  elif type(expr) is Pair:
    if expr.car == "define":
      frame[expr.cdr.car] = scheme_eval(expr.cdr.cdr.car)
      return "set symbol -> value"
    else:
      return "scheme_eval: not implemented"
  else:
    return "scheme_eval: not implemented"

a
;===> Error: Unbound variable
(define a 5)
;===> set symbol -> value
a
;===> 5

In addition to being able to set symbols, we now have a glimpse into how basic forms (like define) are handled. It really is no more complicated than checking the procedure name, evaluating the arguments that need to be, then passing the arguments to a Python procedure that implements the desired functionality.

* To any Schemers that may be reading this: I avoided implementing environments here for two reasons. I wanted to show the simplest implementation possible and in an upcoming post we will transform our global frame into a lookup table for primitive syntax.

Checkout this version on GitHub (v0.05)

GOTO Part 6 | scheme-syntax macro

GOTO | Table of Contents

In the last post we finished implementing the read layer. There is a lot of room for improvement, however it will serve our purposes for the moment. With the read layer complete we can move on to the eval layer.

The eval layer is where the Scheme in Scheme series starts and from here on there will be many similarities between these two series. In fact a sizable portion of this post was simply copy/pasted from the Scheme in Scheme series. With the exception of the code and language specific explanation of it, the rest of the series will also be a close replica.

Create a new file called scheme_eval.py, the rest of this series will focus mainly on this file. In it we will define a function scheme_eval. For now we’re simply going to echo the input back without processing it at all.

scheme_eval.py

def scheme_eval(expr):
  return expr

repl.py

import sys
from scheme_read import scheme_read
from scheme_eval import scheme_eval
from buffered_input import Buff

while True:
  print "> ",
  print scheme_eval(scheme_read(Buff(sys.stdin)))

With that we have the basic layout of the interpreter. If you read the last line backward you will see the layers of the interpreter print(eval(read(stdin))) all wrapped up in an infinite loop. From this point we can concentrate on evaluating the expressions that scheme_read returns.

A Language of Numbers

The simplest language is a language of numbers. Numbers are self-evaluating, so if the expression is a number we’ll just return it for now. If the expression is anything else, we’ll return an error string, just as we did in the read layer. If you are following along, you will probably want to add proper errors/error handling to your version.

def scheme_eval(expr):
  if type(expr) is int:
    return expr
  else:
    return "scheme_eval: not implemented"

A Language of Self-Evaluating Values

Continuing with our language of self-evaluating values is just as simple. Booleans, strings and characters are also self-evaluating so we’ll add them now. More complex data structures like hash tables are also self evaluating, however we’ll leave them for later.

def self_evaluating(expr):
  t = type(expr)
  return t is int or t is float or t is str or t is bool

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  else:
    return "scheme_eval: not implemented"

The first stage of the eval layer is complete. We now have a functional language of self-evaluating values. Though one could argue exactly how functional it really is. The next step along our path will be to introduce symbols and implement assignment.

Checkout this version on GitHub (v0.04)

GOTO Part 5 | Assignment and define

GOTO | Table of Contents

In the last post we added support for booleans, characters, symbols and strings. There is only one major data type left to add in the read layer, the pair/linked list.

A pair is a structure which contains 2 elements normally referred to as the car and cdr in Scheme. Lists are built on top of pairs where the car is some value and the cdr is another pair or the empty list. In a proper list the cdr of the last pair must be the empty list.

To illustrate this concept a bit more clearly here is a diagram of the list ‘(1 2 3) in Scheme:

This list is constructed out of 3 pairs. The car of the first pair is the value 1 and the cdr is the second pair. The car of the second pair has the value 2 and the cdr is the third pair. The car of the third pair has the value 3 and the cdr contains the empty list (Nil in the diagram).

class Pair:
  def __init__(self, car, cdr):
    self.car = car
    self.cdr = cdr

Above is the class definition for a very simple Pair object in Python. We can construct the above list using the Pair class as follows:

lst = Pair(1, Pair(2, Pair(3, None)))

This implementation of a list is called a linked list and you can find a more detailed description in my article WITH Linked Lists. Python does not have an implementation of a list in the core language, the list data type in Python is called an array in most other languages.

Scheme sourcecode is written using the linked list data type. This is one of the most powerful features of the language as the sourcecode is just a regular Scheme data type. This allows some really interesting features such as macros, but we won’t get into that just yet.

To start with we’ll implement the empty list. Since this is a distinct object in Scheme we’ll make a new type called The_Empty_List and since we only need one empty list we’ll also make a global variable to refer to it called the_empty_list. We’ll do this in the scheme_types.py file.

class Symbol(str):
  """A symbol is an immutable string, but must be its own type."""
  pass

class The_Empty_List():
  """The empty list is used as a list terminator and should be a singleton"""
  def __repr__(self):
    return "()"

the_empty_list = The_Empty_List()

Now that we have an empty list type we need to modify our reader to recognize it. When the “(” character is encountered, scheme_read will call read_pair. Since the only pair we’re going to allow for the moment is the empty list, we just need to check that the next character is “)” and return the_empty_list.

from scheme_types import Symbol, the_empty_list

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def is_initial(c):
  """Is c a valid initial character for an identifier?"""
  return c.isalpha() or c in '-+*/><=?!&'

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if '.' in buf:
    return float(buf)
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def read_expected_string(f, s):
  """Consume expected characters from the input buffer."""
  expected_chars = list(s)
  c = f.getc()
  while expected_chars:
    if c == expected_chars[0]:
      expected_chars.pop(0)
      c = f.getc()
    else:
      return "unexpected character"

def read_character(f):
  c = f.getc()
  if c == 's' and f.peek() == 'p':
    read_expected_string(f, "pace")
    return ' '
  elif c == 'n' and f.peek() == 'e':
    read_expected_string(f, "ewline")
    return '\n'
  else:
    return c

def read_symbol(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  return ''.join(buf)

def read_string(f):
  buf = []
  c = f.getc()
  while c != '\"':
    buf.append(c)
    c = f.getc()
  return ''.join(buf)

def read_pair(f):
  f.remove_whitespace()
  c = f.getc()
  if c == ')':
    return the_empty_list

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == '-' and f.peek().isdigit() or f.peek() == '.')
      or (c == '.' and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  elif c == '#':
    c = f.getc()
    if c == 't':
      return True
    elif c == 'f':
      return False
    elif c == '\\':
      return read_character(f)
    else:
      return "Error boolean value must be #t or #f not #" + c
  elif is_initial(c):
    f.ungetc(c)
    return Symbol(read_symbol(f))
  elif c == '\"':
    return read_string(f)
  elif c == '(':
    return read_pair(f)
  else:
    return "scheme_read: not implemented"

With that done, we’re only two steps away from being able to read linked lists. First we have to create a Pair type that will represent a Scheme pair. To do this, we’ll add the following to the scheme_types.py file.

class Symbol(str):
  """A symbol is an immutable string, but must be its own type."""
  pass

class The_Empty_List():
  """The empty list is used as a list terminator and should be a singleton"""
  def __repr__(self):
    return "()"

the_empty_list = The_Empty_List()

class Pair(object):
  """A pair is a classic Lisp cons cell used to implement lists"""
  def __init__(self, car, cdr):
    self.car = car
    self.cdr = cdr
  def __iter__(self):
    x = self
    while not isinstance(x, The_Empty_List):
      yield x.car
      x = x.cdr
  def __repr__(self):
    return "(" + ' '.join([str(i) for i in list(self)]) + ")"

In addition to a car and cdr field I have also implemented the __iter__ and __repr__ special methods. __iter__ will help us later on when we work with these pairs by allowing easy conversion to a Python list data type or allowing the use of for/in or map. For more information on __iter__ see the Python docs on iterators.

The only thing left to do is modify read_pair. The procedure to read a pair is as follows:

1. Get a value for the car slot by calling scheme_read on the input file
2. Remove any whitespace
3. Get a value for the cdr by calling read_pair on the input file
   • If the next character is “)” set the cdr to the_empty_list
   • If the next character is anything else set the cdr to a new pair by calling read_pair

from scheme_types import Symbol, the_empty_list, Pair

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def is_initial(c):
  """Is c a valid initial character for an identifier?"""
  return c.isalpha() or c in '-+*/><=?!&'

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if '.' in buf:
    return float(buf)
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def read_expected_string(f, s):
  """Consume expected characters from the input buffer."""
  expected_chars = list(s)
  c = f.getc()
  while expected_chars:
    if c == expected_chars[0]:
      expected_chars.pop(0)
      c = f.getc()
    else:
      return "unexpected character"

def read_character(f):
  c = f.getc()
  if c == 's' and f.peek() == 'p':
    read_expected_string(f, "pace")
    return ' '
  elif c == 'n' and f.peek() == 'e':
    read_expected_string(f, "ewline")
    return '\n'
  else:
    return c

def read_symbol(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  return ''.join(buf)

def read_string(f):
  buf = []
  c = f.getc()
  while c != '\"':
    buf.append(c)
    c = f.getc()
  return ''.join(buf)

def read_pair(f):
  f.remove_whitespace()
  c = f.getc()
  if c == ')':
    return the_empty_list
  f.ungetc(c)
  car = scheme_read(f)
  f.remove_whitespace()
  cdr = read_pair(f)
  return Pair(car, cdr)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == '-' and f.peek().isdigit() or f.peek() == '.')
      or (c == '.' and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  elif c == '#':
    c = f.getc()
    if c == 't':
      return True
    elif c == 'f':
      return False
    elif c == '\\':
      return read_character(f)
    else:
      return "Error boolean value must be #t or #f not #" + c
  elif is_initial(c):
    f.ungetc(c)
    return Symbol(read_symbol(f))
  elif c == '\"':
    return read_string(f)
  elif c == '(':
    return read_pair(f)
  else:
    return "scheme_read: not implemented"

There are two things to note about this implementation. First, () is matched as the_empty_list when it really shouldn’t be, the correct syntax is ‘(). Second, any literal list over 1,000 elements will hit Python’s recursion limit. This shouldn’t be a problem in the vast majority of cases, however it is something to be aware of.

With that, the read layer of our interpreter is complete (for the purposes of this series, there is a lot more that could be done). From this point on we will be concentrating on the eval layer of the interpreter.

Checkout this version on GitHub (v0.03)

GOTO Part 4 | Self-evaluating values

GOTO | Table of Contents

In the last post we created a read function that worked with integers and floating point numbers. We also covered the basic layout of the read function. In this post we will extend the read function to handle booleans, characters, strings and symbols.

We’ll start with booleans as they are the smallest set. There are 2 boolean values in Scheme #t and #f, which represent true and false respectively. You probably noticed that both start with a # character and that is how we will recognize them in the reader.

We will represent #t and #f by the Python values True and False. Since the code for this is so simple, we will handle this within the scheme_read function.

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if '.' in buf:
    return float(buf)
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == '-' and f.peek().isdigit() or f.peek() == '.')
      or (c == '.' and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  elif c == '#':
    c = f.getc()
    if c == 't':
      return True
    elif c == 'f':
      return False
    else:
      return "Error boolean value must be #t or #f not #" + c
  else:
    return "scheme_read: not implemented"

The code to handle booleans is pretty straightforward. If the first character is #, then we get the next character. If the next character is t or f, we return the appropriate boolean. Otherwise, we return an error string.

The next thing we’ll handle is characters. Characters in Scheme also start with the # symbol followed by a \, which gives us #\c where c is any character.

There are some special characters, but we’ll start by implementing the basics. We’ll check if an expression is a character in the same place as we check for booleans since they’re so similar.

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if '.' in buf:
    return float(buf)
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def read_character(f):
  c = f.getc()
  return c

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == '-' and f.peek().isdigit() or f.peek() == '.')
      or (c == '.' and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  elif c == '#':
    c = f.getc()
    if c == 't':
      return True
    elif c == 'f':
      return False
    elif c == '\\':
      return read_character(f)
    else:
      return "Error boolean value must be #t or #f not #" + c
  else:
    return "scheme_read: not implemented"

Scheme has a few special characters that we must deal with, namely #\space and #\newline. These will require a bit more work to process. To help, we’ll define a function read_expected_string which will consume characters from the input buffer, provided they match what’s expected (and return an error if they don’t).

I won’t go over the workings of read_expected_string any further than to say it compares characters from the expected string and the input buffer one-by-one until it reaches the end. We also need to add a check in read_character if the input buffer contains #\s or #\n. Both of these could be either the characters s or n or the special characters #\space or #\newline. To tell the difference we need to check the next character.

There are a few problems with this implementation, namely if you enter an invalid special character such as #\spxce it will leave unconsumed characters on the input buffer and will not return the error string. Handling these error conditions would add a lot of bloat to the code, though you would definitely want to do this in your own implementation.

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if '.' in buf:
    return float(buf)
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def read_expected_string(f, s):
  """Consume expected characters from the input buffer."""
  expected_chars = list(s)
  c = f.getc()
  while expected_chars:
    if c == expected_chars[0]:
      expected_chars.pop(0)
      c = f.getc()
    else:
      return "unexpected character"

def read_character(f):
  c = f.getc()
  if c == 's' and f.peek() == 'p':
    read_expected_string(f, "pace")
    return ' '
  elif c == 'n' and f.peek() == 'e':
    read_expected_string(f, "ewline")
    return '\n'
  else:
    return c

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == '-' and f.peek().isdigit() or f.peek() == '.')
      or (c == '.' and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  elif c == '#':
    c = f.getc()
    if c == 't':
      return True
    elif c == 'f':
      return False
    elif c == '\\':
      return read_character(f)
    else:
      return "Error boolean value must be #t or #f not #" + c
  else:
    return "scheme_read: not implemented"

Next up are symbols, which are the names of variables/constants. In most other languages symbols are not given their own type and you, as a user, are not allowed to manipulate them as first-class values. However this is not the case in Scheme.

We will represent symbols in our implementation as strings. However since we also need to implement strings, it is necessary to make symbols a distinct type. Since we will need to make other types as well as have access to them outside the read layer, make a new file called scheme_types.py and enter the following definition.

class Symbol(str):
  """A symbol is an immutable string, but must be its own type."""
  pass

How do we know if an expression is a symbol? There is a limited set of values that a symbol may start with. To test if a character is part of this set, we’ll provide a function is_initial.

Once we’ve identified that the expression is a symbol, the next step is to read each character into a buffer until we’ve hit a delimiter (just like reading a number). Then we turn the buffer into a string and pass it to Symbol to create a new symbol.

from scheme_types import Symbol

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def is_initial(c):
  """Is c a valid initial character for an identifier?"""
  return c.isalpha() or c in '-+*/><=?!&'

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if '.' in buf:
    return float(buf)
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def read_expected_string(f, s):
  """Consume expected characters from the input buffer."""
  expected_chars = list(s)
  c = f.getc()
  while expected_chars:
    if c == expected_chars[0]:
      expected_chars.pop(0)
      c = f.getc()
    else:
      return "unexpected character"

def read_character(f):
  c = f.getc()
  if c == 's' and f.peek() == 'p':
    read_expected_string(f, "pace")
    return ' '
  elif c == 'n' and f.peek() == 'e':
    read_expected_string(f, "ewline")
    return '\n'
  else:
    return c

def read_symbol(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  return ''.join(buf)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == '-' and f.peek().isdigit() or f.peek() == '.')
      or (c == '.' and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  elif c == '#':
    c = f.getc()
    if c == 't':
      return True
    elif c == 'f':
      return False
    elif c == '\\':
      return read_character(f)
    else:
      return "Error boolean value must be #t or #f not #" + c
  elif is_initial(c):
    f.ungetc(c)
    return Symbol(read_symbol(f))
  else:
    return "scheme_read: not implemented"

The reader does not lookup the value of symbols, that’s the job of the eval layer, so we’re all done for handling symbols.

With symbols done we only have strings left to implement for this part in the series. Luckily the implementation for strings is almost exactly the same as the one for symbols. Strings in Scheme start and end with “, so an expression is a string if the first character is a “.

The read_string function we’ll provide is a copy/paste of the read_symbol function with only one change. Instead of while not is_delimiter(c) we will use while c != ‘\”‘ since delimiter characters are allowed to appear in strings. Since the functionality is identical, you could abstract this into one function by adding a parameter that passes the stopping condition. However I feel that it’s more clear, for this series, as two.

from scheme_types import Symbol

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def is_initial(c):
  """Is c a valid initial character for an identifier?"""
  return c.isalpha() or c in '-+*/><=?!&'

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if '.' in buf:
    return float(buf)
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def read_expected_string(f, s):
  """Consume expected characters from the input buffer."""
  expected_chars = list(s)
  c = f.getc()
  while expected_chars:
    if c == expected_chars[0]:
      expected_chars.pop(0)
      c = f.getc()
    else:
      return "unexpected character"

def read_character(f):
  c = f.getc()
  if c == 's' and f.peek() == 'p':
    read_expected_string(f, "pace")
    return ' '
  elif c == 'n' and f.peek() == 'e':
    read_expected_string(f, "ewline")
    return '\n'
  else:
    return c

def read_symbol(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  return ''.join(buf)

def read_string(f):
  buf = []
  c = f.getc()
  while c != '\"':
    buf.append(c)
    c = f.getc()
  return ''.join(buf)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == '-' and f.peek().isdigit() or f.peek() == '.')
      or (c == '.' and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  elif c == '#':
    c = f.getc()
    if c == 't':
      return True
    elif c == 'f':
      return False
    elif c == '\\':
      return read_character(f)
    else:
      return "Error boolean value must be #t or #f not #" + c
  elif is_initial(c):
    f.ungetc(c)
    return Symbol(read_symbol(f))
  elif c == '\"':
    return read_string(f)
  else:
    return "scheme_read: not implemented"

With string handling completed there is only one major feature left for scheme_read, pairs. Pairs are used to implement linked lists in Scheme. The linked list is a very important data structure as all Scheme code is written and parsed into linked list format. This code/data duality is one of the things that gives Scheme (and other Lisps) its great power. The next post will be about the implementation of linked lists and how to parse them.

Checkout this version on GitHub (v0.02)

GOTO Part 3 | Pairs and linked lists

GOTO | Table of Contents

Stage 1 of implementing Scheme in Python is to write the reader. The entire read layer is self-contained so we don’t need to worry about evaluating and printing values just yet.

To start with let’s take a look at the basic shape of the read function. We’re going to call it scheme_read to avoid any name clashes that might arise, but other than that it should be more-or-less equivalent to any other Scheme read function.

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if c:
    return "Representation of a Scheme type."

scheme_read(f) is called with a buffered file-like object f that must support getc, ungetc, peek and remove_whitespace methods. First we remove any leading whitespace as whitespace in Scheme is only significant as a delimiter.

Before we get started let’s implement a simple repl so that we can play with our new features as we add them. Create a new file named repl.py and insert the following code.

#!/usr/bin/env python
import sys
from scheme_read import scheme_read
from buffered_input import Buff

while True:
  print "> ",
  print scheme_read(Buff(sys.stdin))

The next step is to get the first non-whitespace character, which we store in the variable c. We will use this character (and possibly the next) to determine the type of the expression. For instance if c is a digit then the remaining characters (until the next delimiter) must form a number, since digits are not allowed as the first character of an identifier. We’ll do this check in the if c: portion of our code. Let’s look at that code.

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if c.isdigit():
    return "Representation of a Scheme number."

What if c is not a digit? In that case, our read function does not recognize the input and we should return something that states this fact. For now let’s just return a string that says “scheme_read: not implemented”. Later on you may wish to make this a proper error, but returning a string here keeps things simple.

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if c.isdigit():
    return "Representation of a Scheme number."
  else:
    return "scheme_read: not implemented"

Now obviously, “Representation of a Scheme number.” is not actually a representation of a Scheme number. To get that we have to read the number from f. Instead of putting all the logic of reading a number into the if block, we should use a helper function here called read_number.

def read_number(f):
  return "Representation of a Scheme number."

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if c.isdigit():
    return read_number(f)
  else:
    return "scheme_read: not implemented"

We pass the file object f to read_number so that it can read and return the rest of the number. But wait, we already read the first digit! Before we pass the work off to read_number, we have to put the first digit back onto f. We do this using the ungetc method.

def read_number(f):
  return "Representation of a Scheme number."

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if c.isdigit():
    f.ungetc(c)
    return read_number(f)
  else:
    return "scheme_read: not implemented"

Now to read a number we simply read a character at a time, placing each into a buffer, until we hit a delimiter. To make that job easier we’ll provide a function is_delimiter that takes a character and returns True if that character is a delimiter and False if not. Then we’ll use that to build a buffer and return an integer.

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  return int(buf)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if c.isdigit():
    f.ungetc(c)
    return read_number(f)
  else:
    return "scheme_read: not implemented"

That takes care of positive integers, however the positive integers are a very small subset of numbers. We are going to want to handle negative integers as well as positive and negative floating point numbers at least. Luckily, this is very simple to do in Python.

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if '.' in buf:
    return float(buf)
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if c.isdigit():
    f.ungetc(c)
    return read_number(f)
  else:
    return "scheme_read: not implemented"

However, we’re not quite done yet. Our implementation won’t handle negative numbers. To do that we’re going to have to go back to the reader because “-” is not a digit and our current implementation will skip right over it.

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if '.' in buf:
    return float(buf)
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if c.isdigit() or (c == '-' and f.peek().isdigit()):
    f.ungetc(c)
    return read_number(f)
  else:
    return "scheme_read: not implemented"

We’re still not quite done though. What about floating point values where 0 < n < 1 that start with a decimal point?

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (‘ ‘, ‘(‘, ‘)’, ‘\"’, ‘;’, ‘\n’) or not c

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if ‘.’ in buf:
    return float(buf)
  elif ‘/’ in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == ‘-’ and f.peek().isdigit())
      or (c == ‘.’ and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  else:
    return "scheme_read: not implemented"

But wait, there's more! What about negative floating point numbers where -1 < n < 0 which start with a "-."?

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return c in (' ', '(', ')', '\"', ';', '\n') or not c

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  buf = "".join(buf)
  if '.' in buf:
    return float(buf)
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(buf)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == '-' and f.peek().isdigit() or f.peek() == '.')
      or (c == '.' and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  else:
    return "scheme_read: not implemented"

As you can see, a lot of the work of implementing a programming language goes into dealing with corner cases. In the next post we'll extend our reader to handle booleans, characters, strings and symbols.

Checkout this version on GitHub (v0.01)

GOTO Part 2 | Extending the read layer

GOTO | Table of Contents

1) Read Layer
2) Eval Layer
3) Language Layer

The read layer

The read layer is responsible for consuming source code and transforming it into a data structure so that it can be passed to the eval layer.

The eval layer

After the reader returns a data structure representing the source code, the eval layer evaluates it according to a set of rules and returns a value.

After an expression has been evaluated it’s value is often printed to the screen. Immediately after that the program usually loops back to reading the next expression, hence the term REPL (Read, Eval, Print, Loop).

The language layer

This interpreter is designed using the “languages as libraries” concept. This approach is currently being used by the Racket and PyPy projects (though in a much more advanced form).

A language is usually defined as the core components, or primitives, plus additional libraries written using those primitives. In most languages it is an error to attempt to redefine a primitive procedure and even in languages where this is permissible, there is usually no way to swap all of the primitives.

Using the languages as libraries approach, we implement as little as possible in the source code of the interpreter and allow a file to be loaded with definitions of the primitives in the language itself. This implementation only requires one primitive to be defined at the interpreter level¹, define-syntax, which allows the user to define new primitive syntax.

Before we begin

The read function presented here is a port of the read function in Bootstrap Scheme by Peter Michaux. His implementation relies on the C functions getc and ungetc. Unfortunately input in Python is line-buffered and not character-buffered and Python does not provide ungetc.

To accommodate for this I have created a class (Buff) which wraps stdin (or any other file-like object) and provides the functions getc, ungetc and peek. Reading from input in this manner is not in the least bit Pythonic, however this method of reading is very simple and clearly shows the steps involved in lexing and parsing input strings.

The operation of Buff is simple. getc reads a single character from the given file if there are no characters waiting in the buffer, otherwise it pops off the last character added to the buffer. ungetc simply adds a character to the buffer, and peek uses getc to read a character and store it in a variable, then ungetc to put the character back in the buffer.

class Buff:
  def __init__(self, f):
    self.f = f
    self.last = []

  def getc(self):
    if self.last:
      return self.last.pop(0)
    else:
      return self.f.read(1)

  def ungetc(self, c):
    self.last.insert(0, c)

  def peek(self):
    if self.last:
      return self.last[0]
    else:
      c = self.f.read(1)
      self.last.append(c)
      return c

  def remove_whitespace(self):
    c = self.getc()
    while c:
      if c == ' ':
        c = self.getc()
      elif c == '\n':
        c = self.getc()
      elif c == ';':
        c = self.getc()
        while c and c != '\n':
          c = self.getc()
        self.ungetc(c)
      else:
        self.ungetc(c)
        break

You may notice that there is also a function: remove_whitespace. This simply reads and discards any characters which are whitespace (space and newline) and discards comments ;starting at the semicolon and continuing until a newline is reached.

With that we have the tools we need to get started building the read level.

[1] In theory, in practice we also define “load” to avoid the necessity of redefining it every time we restart the interpreter.

Checkout this project on GitHub

GOTO Part 1 | Numbers

GOTO | Table of Contents

The scheme-syntax macro adds a lot of power to our language, however the way we’ve been defining special forms so far does not use that power. To take full advantage of scheme-syntax we need to implement a function that will allow us to load a file with syntax definitions into our interpreter.

The load function will work in the same way as all the other special forms, however this will be the only special form that is defined without using scheme-syntax. All the other special forms will use load combined with scheme-syntax.

First we’ll define a procedure called load that will take a filename and execute all the expressions in that file. We will also create a file named scheme_syntax.scm that will hold some test code to make sure the load function works.

"""
scheme.py
(c)2011 Nick Zarczynski
version 0.1
"""
import sys
from buffered_input import Buff

#############################################################################
## Data Types
#############################################################################

class Symbol(str):
  pass

class Pair(object):
  def __init__(self, car, cdr):
    self.car = car
    self.cdr = cdr

class The_Empty_List(object):
  def __repr__(self):
    return "()"
the_empty_list = The_Empty_List()

#############################################################################
## Environments
#############################################################################

global_environment = {"hello":"world", "number":42}

def lookup_symbol_value(symbol):
  try:
    return global_environment[symbol]
  except KeyError:
    return "symbol not found"

#############################################################################
## Read
#############################################################################

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return (c == ' ' or c == '(' or c == ')' or c == '\"'
          or c == ';' or c == '\n' or not c)

def is_initial(c):
  """Is c a valid initial character for an identifier?"""
  return (c.isalpha() or c == '*' or c == '/' or c == '>' or c == '<'
          or c == '=' or c == '?' or c == '!' or c == '-' or c == '&')

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  if '.' in buf:
    return float(''.join(buf))
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(''.join(buf))

def read_expected_string(f, s):
  """Consume expected characters from the input buffer."""
  expected_chars = list(s)
  c = f.getc()
  while expected_chars:
    if c == expected_chars[0]:
      expected_chars.pop(0)
      c = f.getc()
    else:
      return "unexpected character"

def read_character(f):
  """Read a character literal including #\space and #\newline"""
  c = f.getc()
  if c == 's':
    if f.peek() == 'p':
      read_expected_string(f, "pace")
      return ' '
    else:
      return 's'
  elif c == 'n':
    if f.peek() == 'e':
      read_expected_string(f, 'ewline')
      return '\n'
    else:
      return 'n'
  else:
    return c

def read_symbol(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  return ''.join(buf)

def read_string(f):
  buf = []
  c = f.getc()
  while c != '\"':
    buf.append(c)
    c = f.getc()
  return ''.join(buf)

def read_pair(f):
  f.remove_whitespace()
  c = f.getc()
  if c == ')':
    return the_empty_list
  f.ungetc(c)
  car = scheme_read(f)
  f.remove_whitespace()
  cdr = read_pair(f)
  return Pair(car, cdr)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == '-' and f.peek().isdigit() or f.peek() == '.')
      or (c == '.' and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  elif c == '#':
    c = f.getc()
    if c == 't':
      return True
    elif c == 'f':
      return False
    elif c == '\\':
      return read_character(f)
  elif is_initial(c) or ((c == '+' or c == '-') and is_delimiter(f.peek())):
    f.ungetc(c)
    return Symbol(read_symbol(f))
  elif c == '\"':
    return read_string(f)
  elif c == '(':
    return read_pair(f)
  else:
    return "not implemented (read)"

#############################################################################
## Eval
#############################################################################

def self_evaluating(expr):
  t = type(expr)
  return (t is int or t is float or t is bool
          or t is str or expr == the_empty_list)

special_forms = {}

def apply_define(expr):
  global_environment[expr.car] = scheme_eval(expr.cdr.car)

special_forms['define'] = apply_define

def apply_if(expr):
  pred = scheme_eval(expr.car)
  if pred:
    return scheme_eval(expr.cdr.car)
  else:
    return scheme_eval(expr.cdr.cdr.car)

special_forms['if'] = apply_if

def special_form_handler(expr):
  """Register a symbol with a Python function named "f" that implements a special form"""
  exec(expr.cdr.car)
  special_forms[expr.car] = f

special_forms['scheme-syntax'] = special_form_handler

def load(expr):
  f = open(expr.car, 'r')
  b = Buff(f)
  while b.peek():
    scheme_print(scheme_eval(scheme_read(b)))
    b.remove_whitespace()
  f.close()

special_forms['load'] = load

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) == Symbol:
    return lookup_symbol_value(expr)
  elif type(expr) == Pair:
    sym = expr.car
    func = special_forms[sym]
    return func(expr.cdr)
  else:
    return "not implemented (eval)"

#############################################################################
## Print
#############################################################################

def scheme_print(expr):
  if type(expr) == Pair:
    sys.stdout.write('(')
    p = expr
    while p.cdr != the_empty_list:
      scheme_print(p.car)
      sys.stdout.write(' ')
      p = p.cdr
    scheme_print(p.car)
    sys.stdout.write(')')
  elif expr is None:
    pass
  else:
    sys.stdout.write(str(expr))

#############################################################################
## REPL
#############################################################################

def repl():
  while 1:
    sys.stdout.write('> ')
    scheme_print(scheme_eval(scheme_read(Buff(sys.stdin))))
    print

if __name__ == "__main__":
  repl()

And here is the scheme_syntax.scm file with the test definitions add and sub.

(scheme-syntax add "
def f(expr):
  return expr.car + expr.cdr.car")

(scheme-syntax sub "
def f(expr):
  return expr.car - expr.cdr.car")

When you run the interpreter you can load scheme_syntax as follows and add and sub should both work.

> (load "scheme_syntax.scm")
> (add 3 4)
7
> (sub 3 4)
-1

Now that we can define special forms with scheme-syntax and load arbitrary files with the load form we can move our scheme-syntax macros over to the new scheme_syntax.scm file. This allows us the flexibility to play around with different methods of implementing language features from within our interpreter.

We’re going to move define and if over to scheme_syntax.scm for now. Because we still have to implement procedures and environments, I’ll wait to add more special forms for now, but you can add as many as you like (use R5RS for a guide if you need). The code for define and if was removed from scheme.py though I won’t paste the new file as there’s no changes to highlight. I also removed the add and sub forms as they’re no longer needed.

(scheme-syntax define "
def f(expr):
  global_environment[expr.car] = scheme_eval(expr.cdr.car)")

(scheme-syntax if "
def f(expr):
  pred = scheme_eval(expr.car)
  if pred:
    return scheme_eval(expr.cdr.car)
  else:
    return scheme_eval(expr.cdr.cdr.car)")

We’re not far from having all the basics of a Scheme implementation. The next few parts will focus on creating primitive procedures, environments and compound procedures.

GOTO Table of Contents

In the last post we briefly discussed how using if/elif/else in scheme_eval could become cumbersome. There is another problem with this approach as well. In order to add new special forms to the language we need to directly modify the source code of the interpreter.

Instead of dispatching with the if/else construct we can use another means of dispatching that will allow runtime modification by using a dictionary (or hash table). With this method we use the symbol (so far define and if) as the key for a dictionary lookup.

With this method the symbol (so far define and if) is used as the key and a function that takes the rest of the expression and returns a result is stored in the dictionary. Let’s take a look at how this is implemented.

"""
scheme.py
(c)2011 Nick Zarczynski
version 0.1
"""
import sys
from buffered_input import Buff

#############################################################################
## Data Types
#############################################################################

class Symbol(str):
  pass

class Pair(object):
  def __init__(self, car, cdr):
    self.car = car
    self.cdr = cdr

class The_Empty_List(object):
  def __repr__(self):
    return "()"
the_empty_list = The_Empty_List()

#############################################################################
## Environments
#############################################################################

global_environment = {"hello":"world", "number":42}

def lookup_symbol_value(symbol):
  try:
    return global_environment[symbol]
  except KeyError:
    return "symbol not found"

#############################################################################
## Read
#############################################################################

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return (c == ' ' or c == '(' or c == ')' or c == '\"'
          or c == ';' or c == '\n' or not c)

def is_initial(c):
  """Is c a valid initial character for an identifier?"""
  return (c.isalpha() or c == '*' or c == '/' or c == '>' or c == '<'
          or c == '=' or c == '?' or c == '!' or c == '-' or c == '&')

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  if '.' in buf:
    return float(''.join(buf))
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(''.join(buf))

def read_expected_string(f, s):
  """Consume expected characters from the input buffer."""
  expected_chars = list(s)
  c = f.getc()
  while expected_chars:
    if c == expected_chars[0]:
      expected_chars.pop(0)
      c = f.getc()
    else:
      return "unexpected character"

def read_character(f):
  """Read a character literal including #\space and #\newline"""
  c = f.getc()
  if c == 's':
    if f.peek() == 'p':
      read_expected_string(f, "pace")
      return ' '
    else:
      return 's'
  elif c == 'n':
    if f.peek() == 'e':
      read_expected_string(f, 'ewline')
      return '\n'
    else:
      return 'n'
  else:
    return c

def read_symbol(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  return ''.join(buf)

def read_string(f):
  buf = []
  c = f.getc()
  while c != '\"':
    buf.append(c)
    c = f.getc()
  return ''.join(buf)

def read_pair(f):
  f.remove_whitespace()
  c = f.getc()
  if c == ')':
    return the_empty_list
  f.ungetc(c)
  car = scheme_read(f)
  f.remove_whitespace()
  cdr = read_pair(f)
  return Pair(car, cdr)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == '-' and f.peek().isdigit() or f.peek() == '.')
      or (c == '.' and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  elif c == '#':
    c = f.getc()
    if c == 't':
      return True
    elif c == 'f':
      return False
    elif c == '\\':
      return read_character(f)
  elif is_initial(c) or ((c == '+' or c == '-') and is_delimiter(f.peek())):
    f.ungetc(c)
    return Symbol(read_symbol(f))
  elif c == '\"':
    return read_string(f)
  elif c == '(':
    return read_pair(f)
  else:
    return "not implemented (read)"

#############################################################################
## Eval
#############################################################################

def self_evaluating(expr):
  t = type(expr)
  return (t is int or t is float or t is bool
          or t is str or expr == the_empty_list)

special_forms = {}

def apply_define(expr):
  global_environment[expr.car] = scheme_eval(expr.cdr.car)

special_forms['define'] = apply_define

def apply_if(expr):
  pred = scheme_eval(expr.car)
  if pred:
    return scheme_eval(expr.cdr.car)
  else:
    return scheme_eval(expr.cdr.cdr.car)

special_forms['if'] = apply_if

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) == Symbol:
    return lookup_symbol_value(expr)
  elif type(expr) == Pair:
    sym = expr.car
    func = special_forms[sym]
    return func(expr.cdr)
  else:
    return "not implemented (eval)"

#############################################################################
## Print
#############################################################################

def scheme_print(expr):
  if type(expr) == Pair:
    sys.stdout.write('(')
    p = expr
    while p.cdr != the_empty_list:
      scheme_print(p.car)
      sys.stdout.write(' ')
      p = p.cdr
    scheme_print(p.car)
    sys.stdout.write(')')
  else:
    sys.stdout.write(str(expr))

#############################################################################
## REPL
#############################################################################

def repl():
  while 1:
    sys.stdout.write('> ')
    scheme_print(scheme_eval(scheme_read(Buff(sys.stdin))))
    print

if __name__ == "__main__":
  repl()

With this method dispatching is simply retrieving the correct function with special_forms[symbol] and calling it with the rest of the expression. This also allows us to add special forms directly from our interpreter without having to modify the source code. Before I get into the implementation in Python, here’s how this functionality is implemented in Lispy in Scheme’s scheme-syntax macro.

(define (scheme-syntax expr)
  (hash-table-set! global-syntax-definitions (car expr) (eval (cadr expr))))

;; And it is called like so
(scheme-syntax 'if
  (lambda (expr)
    ;; code to implement if

First the symbol if is used as the key in global-syntax-definitions which is Lispy’s version of global_environment. Then the lambda expression (written in Scheme) is evaluated and set as the value associated with the symbol.

Unfortunately this cannot be a straightforward port into Python for a couple of reasons. First lambdas in Python are very limited and are not equivalent to named function definition. For instance you are not allowed to use statements (such as if, for, etc.) and you are also not allowed to use more than one expression. This rules out using lambda and means that we have to allow the user to define a regular Python function.

To eval the code now we must use the exec statement, because function definition is a statement and not an expression. However the exec statement does not allow a return value from the executed code. For example:

a = exec("""
def double(x):
  return x * 2
double
""")

Will not set a to the function double, rather it will halt with a SyntaxError. To get around this limitation we will require that the supplied function be named “f” using code equivalent to this:

def exec_definition(s):
  exec(s)
  return f

This creates a new function from the string s. Hopefully the user follows the convention of naming the function f, otherwise this implementation will fail miserably.

"""
scheme.py
(c)2011 Nick Zarczynski
version 0.1
"""
import sys
from buffered_input import Buff

#############################################################################
## Data Types
#############################################################################

class Symbol(str):
  pass

class Pair(object):
  def __init__(self, car, cdr):
    self.car = car
    self.cdr = cdr

class The_Empty_List(object):
  def __repr__(self):
    return "()"
the_empty_list = The_Empty_List()

#############################################################################
## Environments
#############################################################################

global_environment = {"hello":"world", "number":42}

def lookup_symbol_value(symbol):
  try:
    return global_environment[symbol]
  except KeyError:
    return "symbol not found"

#############################################################################
## Read
#############################################################################

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return (c == ' ' or c == '(' or c == ')' or c == '\"'
          or c == ';' or c == '\n' or not c)

def is_initial(c):
  """Is c a valid initial character for an identifier?"""
  return (c.isalpha() or c == '*' or c == '/' or c == '>' or c == '<'
          or c == '=' or c == '?' or c == '!' or c == '-' or c == '&')

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  if '.' in buf:
    return float(''.join(buf))
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(''.join(buf))

def read_expected_string(f, s):
  """Consume expected characters from the input buffer."""
  expected_chars = list(s)
  c = f.getc()
  while expected_chars:
    if c == expected_chars[0]:
      expected_chars.pop(0)
      c = f.getc()
    else:
      return "unexpected character"

def read_character(f):
  """Read a character literal including #\space and #\newline"""
  c = f.getc()
  if c == 's':
    if f.peek() == 'p':
      read_expected_string(f, "pace")
      return ' '
    else:
      return 's'
  elif c == 'n':
    if f.peek() == 'e':
      read_expected_string(f, 'ewline')
      return '\n'
    else:
      return 'n'
  else:
    return c

def read_symbol(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  return ''.join(buf)

def read_string(f):
  buf = []
  c = f.getc()
  while c != '\"':
    buf.append(c)
    c = f.getc()
  return ''.join(buf)

def read_pair(f):
  f.remove_whitespace()
  c = f.getc()
  if c == ')':
    return the_empty_list
  f.ungetc(c)
  car = scheme_read(f)
  f.remove_whitespace()
  cdr = read_pair(f)
  return Pair(car, cdr)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == '-' and f.peek().isdigit() or f.peek() == '.')
      or (c == '.' and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  elif c == '#':
    c = f.getc()
    if c == 't':
      return True
    elif c == 'f':
      return False
    elif c == '\\':
      return read_character(f)
  elif is_initial(c) or ((c == '+' or c == '-') and is_delimiter(f.peek())):
    f.ungetc(c)
    return Symbol(read_symbol(f))
  elif c == '\"':
    return read_string(f)
  elif c == '(':
    return read_pair(f)
  else:
    return "not implemented (read)"

#############################################################################
## Eval
#############################################################################

def self_evaluating(expr):
  t = type(expr)
  return (t is int or t is float or t is bool
          or t is str or expr == the_empty_list)

special_forms = {}

def apply_define(expr):
  global_environment[expr.car] = scheme_eval(expr.cdr.car)

special_forms['define'] = apply_define

def apply_if(expr):
  pred = scheme_eval(expr.car)
  if pred:
    return scheme_eval(expr.cdr.car)
  else:
    return scheme_eval(expr.cdr.cdr.car)

special_forms['if'] = apply_if

def special_form_handler(expr):
  """Register a symbol with a Python function named "f" that implements a special form"""
  exec(expr.cdr.car)
  special_forms[expr.car] = f

special_forms['scheme-syntax'] = special_form_handler

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) == Symbol:
    return lookup_symbol_value(expr)
  elif type(expr) == Pair:
    sym = expr.car
    func = special_forms[sym]
    return func(expr.cdr)
  else:
    return "not implemented (eval)"

#############################################################################
## Print
#############################################################################

def scheme_print(expr):
  if type(expr) == Pair:
    sys.stdout.write('(')
    p = expr
    while p.cdr != the_empty_list:
      scheme_print(p.car)
      sys.stdout.write(' ')
      p = p.cdr
    scheme_print(p.car)
    sys.stdout.write(')')
  else:
    sys.stdout.write(str(expr))

#############################################################################
## REPL
#############################################################################

def repl():
  while 1:
    sys.stdout.write('> ')
    scheme_print(scheme_eval(scheme_read(Buff(sys.stdin))))
    print

if __name__ == "__main__":
  repl()

scheme-syntax has become a foreign function interface (or FFI) of sorts, allowing us to write Python functions directly within our interpreter. For instance, instead of writing define out as:

def apply_define(expr):
  global_environment[expr.car] = scheme_eval(expr.cdr.car)

special_forms['define'] = apply_define

We can instead write it within our interpreter as:

(scheme-syntax define "
def f(expr):
  global_environment[expr.car] = scheme_eval(expr.cdr.car)")

The scheme-syntax macro adds a considerable amount of power to our interpreter. Now we can define new primitives and special forms without touching the source code of the interpreter at all. In fact, we can use scheme-syntax to write both define and if in Scheme, but before we do that we should implement a way to load a Scheme source file so we’re not mixing Scheme code with Python. The next part will focus on implementing load, moving our current definitions out of the core interpreter and adding some new ones.

* This implementation of scheme-syntax leaves a lot to be desired, but it works for now. If you have any suggestions on how to implement this functionality more cleanly, please let me know.

Checkout this version on github – v0.07

GOTO Part 8 | Refactor and Load

GOTO Table of Contents

In the last part we did a pretty major refactoring of the read layer. With that out of the way we can move on to the eval layer. In this part we will implement define which has the syntax:

(define identifier value)

Since we have already implemented pairs we simply have to check if the expression is a pair in the eval layer and perform the appropriate action.

"""
scheme.py
(c)2011 Nick Zarczynski
version 0.1
"""
import sys
from buffered_input import Buff

#############################################################################
## Data Types
#############################################################################

class Symbol(str):
  pass

class Pair(object):
  def __init__(self, car, cdr):
    self.car = car
    self.cdr = cdr

class The_Empty_List(object):
  def __repr__(self):
    return "()"
the_empty_list = The_Empty_List()

#############################################################################
## Environments
#############################################################################

global_environment = {"hello":"world", "number":42}

def lookup_symbol_value(symbol):
  try:
    return global_environment[symbol]
  except KeyError:
    return "symbol not found"

#############################################################################
## Read
#############################################################################

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return (c == ' ' or c == '(' or c == ')' or c == '\"'
          or c == ';' or c == '\n' or not c)

def is_initial(c):
  """Is c a valid initial character for an identifier?"""
  return (c.isalpha() or c == '*' or c == '/' or c == '>' or c == '<'
          or c == '=' or c == '?' or c == '!' or c == '-' or c == '&')

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  if '.' in buf:
    return float(''.join(buf))
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(''.join(buf))

def read_expected_string(f, s):
  """Consume expected characters from the input buffer."""
  expected_chars = list(s)
  c = f.getc()
  while expected_chars:
    if c == expected_chars[0]:
      expected_chars.pop(0)
      c = f.getc()
    else:
      return "unexpected character"

def read_character(f):
  """Read a character literal including #\space and #\newline"""
  c = f.getc()
  if c == 's':
    if f.peek() == 'p':
      read_expected_string(f, "pace")
      return ' '
    else:
      return 's'
  elif c == 'n':
    if f.peek() == 'e':
      read_expected_string(f, 'ewline')
      return '\n'
    else:
      return 'n'
  else:
    return c

def read_symbol(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  return ''.join(buf)

def read_string(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  return ''.join(buf)

def read_pair(f):
  f.remove_whitespace()
  c = f.getc()
  if c == ')':
    return the_empty_list
  f.ungetc(c)
  car = scheme_read(f)
  f.remove_whitespace()
  cdr = read_pair(f)
  return Pair(car, cdr)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == '-' and f.peek().isdigit() or f.peek() == '.')
      or (c == '.' and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  elif c == '#':
    c = f.getc()
    if c == 't':
      return True
    elif c == 'f':
      return False
    elif c == '\\':
      return read_character(f)
  elif is_initial(c) or ((c == '+' or c == '-') and is_delimiter(f.peek())):
    f.ungetc(c)
    return Symbol(read_symbol(f))
  elif c == '\"':
    return read_string(f)
  elif c == '(':
    return read_pair(f)
  else:
    return "not implemented (read)"

#############################################################################
## Eval
#############################################################################

def self_evaluating(expr):
  t = type(expr)
  return (t is int or t is float or t is bool
          or t is str or expr == the_empty_list)

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) == Symbol:
    return lookup_symbol_value(expr)
  elif type(expr) == Pair:
    sym = expr.car
    if sym == 'define':
      global_environment[expr.cdr.car] = expr.cdr.cdr.car
      return None
  else:
    return "not implemented (eval)"

#############################################################################
## Print
#############################################################################

def scheme_print(expr):
  if type(expr) == Pair:
    sys.stdout.write('(')
    p = expr
    while p.cdr != the_empty_list:
      scheme_print(p.car)
      sys.stdout.write(' ')
      p = p.cdr
    scheme_print(p.car)
    sys.stdout.write(')')
  else:
    sys.stdout.write(str(expr))

#############################################################################
## REPL
#############################################################################

def repl():
  while 1:
    sys.stdout.write('> ')
    scheme_print(scheme_eval(scheme_read(Buff(sys.stdin))))
    print

if __name__ == "__main__":
  repl()

If an expression is a pair we check the value of the car and perform the appropriate action. In this case we add an entry into the global_environment that maps the symbol “a” to the value 42. We need to be sure to call scheme_eval on the value as it might be an expression such as (+ 40 2).

After we determine that this expression will be setting a variable we need to get that variable and evaluate the value. Then we create a new entry in the global_environment (or modify an existing entry) with the evaluated value.

The next primitive we will implement is if, the syntax for which is (if predicate consequent alternate). if has to be implemented as a special form because only one branch is evaluated based on the outcome of the predicate test. So here are the steps that must be taken with if.

1) Evaluate the predicate
2.a) If the predicate evaluates to true; evaluate the consequent branch
2.b) If the predicate evaluates to false; evaluate the alternate branch
3) Return the result of evaluating the appropriate expression.

Like define we will check the first value of the pair to see if the symbol is if. We’ll implement this in a similar way to define for the moment. In the next post we’ll look at a better way to do this kind of dispatching.

"""
scheme.py
(c)2011 Nick Zarczynski
version 0.1
"""
import sys
from buffered_input import Buff

#############################################################################
## Data Types
#############################################################################

class Symbol(str):
  pass

class Pair(object):
  def __init__(self, car, cdr):
    self.car = car
    self.cdr = cdr

class The_Empty_List(object):
  def __repr__(self):
    return "()"
the_empty_list = The_Empty_List()

#############################################################################
## Environments
#############################################################################

global_environment = {"hello":"world", "number":42}

def lookup_symbol_value(symbol):
  try:
    return global_environment[symbol]
  except KeyError:
    return "symbol not found"

#############################################################################
## Read
#############################################################################

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return (c == ' ' or c == '(' or c == ')' or c == '\"'
          or c == ';' or c == '\n' or not c)

def is_initial(c):
  """Is c a valid initial character for an identifier?"""
  return (c.isalpha() or c == '*' or c == '/' or c == '>' or c == '<'
          or c == '=' or c == '?' or c == '!' or c == '-' or c == '&')

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  if '.' in buf:
    return float(''.join(buf))
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(''.join(buf))

def read_expected_string(f, s):
  """Consume expected characters from the input buffer."""
  expected_chars = list(s)
  c = f.getc()
  while expected_chars:
    if c == expected_chars[0]:
      expected_chars.pop(0)
      c = f.getc()
    else:
      return "unexpected character"

def read_character(f):
  """Read a character literal including #\space and #\newline"""
  c = f.getc()
  if c == 's':
    if f.peek() == 'p':
      read_expected_string(f, "pace")
      return ' '
    else:
      return 's'
  elif c == 'n':
    if f.peek() == 'e':
      read_expected_string(f, 'ewline')
      return '\n'
    else:
      return 'n'
  else:
    return c

def read_symbol(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  return ''.join(buf)

def read_string(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  return ''.join(buf)

def read_pair(f):
  f.remove_whitespace()
  c = f.getc()
  if c == ')':
    return the_empty_list
  f.ungetc(c)
  car = scheme_read(f)
  f.remove_whitespace()
  cdr = read_pair(f)
  return Pair(car, cdr)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == '-' and f.peek().isdigit() or f.peek() == '.')
      or (c == '.' and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  elif c == '#':
    c = f.getc()
    if c == 't':
      return True
    elif c == 'f':
      return False
    elif c == '\\':
      return read_character(f)
  elif is_initial(c) or ((c == '+' or c == '-') and is_delimiter(f.peek())):
    f.ungetc(c)
    return Symbol(read_symbol(f))
  elif c == '\"':
    return read_string(f)
  elif c == '(':
    return read_pair(f)
  else:
    return "not implemented (read)"

#############################################################################
## Eval
#############################################################################

def self_evaluating(expr):
  t = type(expr)
  return (t is int or t is float or t is bool
          or t is str or expr == the_empty_list)

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) == Symbol:
    return lookup_symbol_value(expr)
  elif type(expr) == Pair:
    sym = expr.car
    if sym == 'define':
      global_environment[expr.cdr.car] = scheme_eval(expr.cdr.cdr.car)
      return None
    elif sym == 'if':
      pred = scheme_eval(expr.cdr.car)
      if pred:
        return scheme_eval(expr.cdr.cdr.car)
      else:
        return scheme_eval(expr.cdr.cdr.cdr.car)
  else:
    return "not implemented (eval)"

#############################################################################
## Print
#############################################################################

def scheme_print(expr):
  if type(expr) == Pair:
    sys.stdout.write('(')
    p = expr
    while p.cdr != the_empty_list:
      scheme_print(p.car)
      sys.stdout.write(' ')
      p = p.cdr
    scheme_print(p.car)
    sys.stdout.write(')')
  else:
    sys.stdout.write(str(expr))

#############################################################################
## REPL
#############################################################################

def repl():
  while 1:
    sys.stdout.write('> ')
    scheme_print(scheme_eval(scheme_read(Buff(sys.stdin))))
    print

if __name__ == "__main__":
  repl()

(if 1 1 2)
;===> 1
(if #f 1 2)
;===> 2

The only value in Scheme that evaluates to false is the symbol #f all other Scheme values evaluate to true. The test case above shows that this version of if works as expected. We’re pretty limited for now on what we can test because we can’t yet define procedures or use much beyond self-evaluating values here.

There are a number of special forms that need to be implemented for Scheme. You might be thinking that it will eventually become cumbersome to check all the values using if/elif/else. We could break out the code for handling if and define into separate functions, but in the next part we’ll take a different approach to dispatching that is much easier to extend.

GOTO Table of Contents

This article will describe some of the changes that have been made to the read layer of the Scheme in Python interpreter.

The biggest conceptual change is that the read layer no longer deals with strings. Instead the read layer now processes streams of characters. This change will not only simplify the implementation, it will also allow the read function to easily work on files as well.

The implementation of scheme_read is a port of the read function from Bootstrap Scheme by Peter Michaux. Peter’s implementation makes heavy use of the getc and ungetc functions in C. Python does not support these functions and using them is more than a bit unpythonic.

Writing a parser by hand is probably the wrong way to go about this from the beginning. However this is supposed to be a learning exercise and I think it aids in understanding the system from the bottom up.

This implementation depends on a class I wrote to do buffered input (the only way ungetc will work in Python). This is found in the file buffered_input.py and you use it as follows:

from buffered_input import Buff
f = Buff(file_like_object)

# To read a single character
c = f.getc()

# To peek at the next character without removing it from the buffer
c = f.peek()

# To place a character back on the buffer
f.ungetc(c)

# To remove leading whitespace
f.remove_whitespace()

# To read from stdin
import sys
f = Buff(sys.stdin)

And here’s the sourcecode that implements the Buff class.

class Buff:
  def __init__(self, f):
    self.f = f
    self.last = []

  def getc(self):
    if self.last:
      return self.last.pop(0)
    else:
      return self.f.read(1)

  def ungetc(self, c):
    self.last.insert(0, c)

  def peek(self):
    if self.last:
      return self.last[0]
    else:
      c = self.f.read(1)
      self.last.append(c)
      return c

  def remove_comment(self):
    c = self.getc()
    while c and c != '\n':
      c = self.getc()
    self.ungetc(c)  

  def remove_whitespace(self):
    c = self.getc()
    while c:
      if c == ' ':
        c = self.getc()
      elif c == '\n':
        c = self.getc()
      elif c == ';':
        self.remove_comment()
      else:
        self.ungetc(c)
        break

Even though a lot of the code has changed, most of the concepts have not. We still look at the first character in an expression to decide what to do with it. Some of the code has not changed at all.

Handling different types has been broken out into helper functions to keep the scheme_read function clean. There are 5 main helper functions read_number, read_character, read_string, read_symbol and read_pair. Each of the first 4 performs the same basic task. First it reads from the input file, character by character, accumulating each character into a buffer until a delimiter is reached. Then it joins the buffer and returns the correct type. read_pair is implemented close to the same way it was in v0.04 except it now handles many more valid lists.

I also added a few helper functions such as is_delimiter, is_initial and self_evaluating to make the code easier to read and extend. Finally I added some docstrings to functions which are not obvious and some block comments to visually separate sections of code. Without further ado here’s the new code for scheme.py.

import sys
from buffered_input import Buff

#############################################################################
## Data Types
#############################################################################

class Symbol(str):
  pass

class Pair(object):
  def __init__(self, car, cdr):
    self.car = car
    self.cdr = cdr

class The_Empty_List(object):
  def __repr__(self):
    return "()"
the_empty_list = The_Empty_List()

#############################################################################
## Environments
#############################################################################

global_environment = {"hello":"world", "number":42}

def lookup_symbol_value(symbol):
  try:
    return global_environment[symbol]
  except KeyError:
    return "symbol not found"

#############################################################################
## Read
#############################################################################

def is_delimiter(c):
  """Is c a valid expression delimiter?"""
  return (c == ' ' or c == '(' or c == ')' or c == '"' or c == '"'
          or c == ';' or c == '\n' or not c)

def is_initial(c):
  """Is c a valid initial character for an identifier?"""
  return (c.isalpha() or c == '*' or c == '/' or c == '>' or c == '<'
          or c == '=' or c == '?' or c == '!' or c == '-' or c == '&')

def read_number(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  if '.' in buf:
    return float(''.join(buf))
  elif '/' in buf:
    return "rationals not implemented"
  else:
    return int(''.join(buf))

def read_expected_string(f, s):
  """Consume expected characters from the input buffer."""
  expected_chars = list(s)
  c = f.getc()
  while expected_chars:
    if c == expected_chars[0]:
      expected_chars.pop(0)
      c = f.getc()
    else:
      return "unexpected character"

def read_character(f):
  """Read a character literal including #\space and #\newline"""
  c = f.getc()
  if c == 's':
    if f.peek() == 'p':
      read_expected_string(f, "pace")
      return ' '
    else:
      return 's'
  elif c == 'n':
    if f.peek() == 'e':
      read_expected_string(f, 'ewline')
      return '\n'
    else:
      return 'n'
  else:
    return c

def read_symbol(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  f.ungetc(c)
  return ''.join(buf)

def read_string(f):
  buf = []
  c = f.getc()
  while not is_delimiter(c):
    buf.append(c)
    c = f.getc()
  return ''.join(buf)

def read_pair(f):
  f.remove_whitespace()
  c = f.getc()
  if c == ')':
    return the_empty_list
  f.ungetc(c)
  car = scheme_read(f)
  f.remove_whitespace()
  cdr = read_pair(f)
  return Pair(car, cdr)

def scheme_read(f):
  f.remove_whitespace()
  c = f.getc()
  if (c.isdigit() or (c == '-' and f.peek().isdigit() or f.peek() == '.')
      or (c == '.' and f.peek().isdigit())):
    f.ungetc(c)
    return read_number(f)
  elif c == '#':
    c = f.getc()
    if c == 't':
      return True
    elif c == 'f':
      return False
    elif c == '\\':
      return read_character(f)
  elif is_initial(c) or ((c == '+' or c == '-') and is_delimiter(f.peek())):
    f.ungetc(c)
    return Symbol(read_symbol(f))
  elif c == '"' or c == '"': # single " breaks wordpress highlighting
    return read_string(f)
  elif c == '(':
    return read_pair(f)
  else:
    return "not implemented (read)"

#############################################################################
## Eval
#############################################################################

def self_evaluating(expr):
  t = type(expr)
  return (t is int or t is float or t is bool
          or t is str or expr == the_empty_list)

def scheme_eval(expr):
  if self_evaluating(expr):
    return expr
  elif type(expr) == Symbol:
    return lookup_symbol_value(expr)
  elif type(expr) == Pair:
    return expr
  else:
    return "not implemented (eval)"

#############################################################################
## Print
#############################################################################

def scheme_print(expr):
  if type(expr) == Pair:
    sys.stdout.write('(')
    p = expr
    while p.cdr != the_empty_list:
      scheme_print(p.car)
      sys.stdout.write(' ')
      p = p.cdr
    scheme_print(p.car)
    sys.stdout.write(')')
  else:
    sys.stdout.write(str(expr))

#############################################################################
## REPL
#############################################################################

def repl():
  while 1:
    sys.stdout.write('> ')
    scheme_print(scheme_eval(scheme_read(Buff(sys.stdin))))
    print

if __name__ == "__main__":
  repl()

* In lines 39 and 136 I had to repeat the ” because a single ” broke the wordpress highlighting.

In addition to those changes the scheme_test.py file also underwent some changes to accommodate the new buffered_input.

If you have any questions about the refactoring that I did not address, please post them in the comments here. Though the implementation of the read layer has changed pretty drastically, the output has not changed at all. Also, since the string type is so similar to the symbol type I added it in this version as well.

GOTO | Table of Contents

This article will focus on implementing lists. Lists in Scheme are an extension of the pair type.

A pair is a structure which contains 2 elements normally referred to as the car and cdr in Scheme. Lists are built on top of pairs where the car is some value and the cdr is another pair or the empty list. In a proper list the cdr of the last pair must be the empty list.

To illustrate this concept a bit more clearly here is a diagram of a list in Scheme:

This list is constructed out of 3 pairs. The car of the first pair is the value 1 and the cdr is the second pair. The car second pair has the value 2 and the cdr is the third pair. The car of the third pair has the value 3 and the cdr contains the empty list (Nil in the diagram).

class Pair:
  def __init__(self, car, cdr):
    self.car = car
    self.cdr = cdr

Above is the class definition for a very simple Pair object in Python. We can construct the above list using the Pair class as follows:

lst = Pair(1, Pair(2, Pair(3, None)))

This implementation of a list is called a linked list and you can find a more detailed description in my article WITH Linked Lists. Python does not have an implementation of a list in the core language, the list data type in Python is called an array in most other languages.

Scheme sourcecode is written using the list data type. This is one of the most powerful features of the language as the sourcecode is just a regular Scheme data type. This allows some really interesting features such as macros, but we won’t get into that just yet.

In Scheme sourcecode a pair is started by an open parenthesis and closed by a matching closing parenthesis. Let’s take a look at an expression that defines the identifier number to be 42 and its internal representation.

(define number 42)
Pair(Symbol("define"), Pair(Symbol("number"), Pair(42, None)))

To start with we’ll implement the empty list. Since this is a distinct object in Scheme we’ll make a new type called The_Empty_List and since we only need one empty list we’ll also make a global variable to refer to it called the_empty_list.

class Symbol(str):
  pass

class The_Empty_List(object):
  def __repr__(self):
    return "()"
the_empty_list = The_Empty_List()

global_environment = {"hello":"world", "number":42}

def lookup_symbol_value(symbol):
  try:
    return global_environment[symbol]
  except KeyError:
    return "symbol not found"

def is_number(s):
  c = s[0]
  if c.isdigit():
    return True
  elif c == '.' and s[1].isdigit():
    return True
  elif c == '-' and (s[1].isdigit() or (s[1] == '.' and s[2].isdigit())):
    return True
  else:
    return False

def read_pair(s):
  if s[0] == ')':
    return the_empty_list
  else:
    return "Pairs not implemented yet"

def scheme_read(s):
  s.lstrip()
  c = s[0]
  if is_number(s):
    if '.' in s:
      return float(s)
    elif '/' in s:
      return "not implemented"
    else:
      return int(s)
  elif c == '#':
    if s[1] == 't':
      return True
    elif s[1] == 'f':
      return False
    elif s[1] == '\\':
      if s == '#\space':
        return ' '
      elif s == r'#\newline':
        return '\n'
      else:
        return s[2]
  elif c.isalpha() or c in ('!', '-', '+', '*', '/', '@', '$'):
    return Symbol(s)
  elif c == '(':
    return read_pair(s[1:])
  else:
    return "not implemented"

def scheme_eval(expr):
  if type(expr) == int:
    return expr
  elif type(expr) == float:
    return expr
  elif type(expr) == bool:
    return expr
  elif type(expr) == str:
    return expr
  elif type(expr) == Symbol:
    return lookup_symbol_value(expr)
  elif type(expr) == The_Empty_List:
    return expr
  else:
    return "not implemented"

def scheme_print(expr):
  print expr

def repl():
  while 1:
    scheme_print(scheme_eval(scheme_read(raw_input("> "))))

As you can see when the first character of an expression is “(” read_pair is called with the remaining expression. When it is the empty list, read_pair simply returns the_empty_list. What happens when there is something else there?

To construct a list we need to evaluate the expressions within that list and build up a chain of Pairs. To do that we make scheme_read and read_pair mutually recursive. This means that we take the first expression up to a delimiter (we’ll only deal with spaces for now) and call scheme_read on it. Then we assign the value that scheme_read returns to the car of a new Pair and call read_pair on the remaining expression, assigning the return value from that to the cdr of the new Pair.

import sys

class Symbol(str):
  pass

class Pair(object):
  def __init__(self, car, cdr):
    self.car = car
    self.cdr = cdr

class The_Empty_List(object):
  def __repr__(self):
    return "()"
the_empty_list = The_Empty_List()

global_environment = {"hello":"world", "number":42}

def lookup_symbol_value(symbol):
  try:
    return global_environment[symbol]
  except KeyError:
    return "symbol not found"

def is_number(s):
  c = s[0]
  if c.isdigit():
    return True
  elif c == '.' and s[1].isdigit():
    return True
  elif c == '-' and (s[1].isdigit() or (s[1] == '.' and s[2].isdigit())):
    return True
  else:
    return False

def read_pair(s):
  s = s.lstrip()
  if s[0] == ')':
    return the_empty_list
  sp = s.split(' ', 1)
  car_obj = scheme_read(sp[0])
  cdr_obj = read_pair(sp[1])
  return Pair(car_obj, cdr_obj)

def scheme_read(s):
  s.lstrip()
  c = s[0]
  if is_number(s):
    if '.' in s:
      return float(s)
    elif '/' in s:
      return "not implemented"
    else:
      return int(s)
  elif c == '#':
    if s[1] == 't':
      return True
    elif s[1] == 'f':
      return False
    elif s[1] == '\\':
      if s == '#\space':
        return ' '
      elif s == r'#\newline':
        return '\n'
      else:
        return s[2]
  elif c.isalpha() or c in ('!', '-', '+', '*', '/', '@', '$'):
    return Symbol(s)
  elif c == '(':
    return read_pair(s[1:])
  else:
    return "not implemented (read)"

def scheme_eval(expr):
  if type(expr) == int:
    return expr
  elif type(expr) == float:
    return expr
  elif type(expr) == bool:
    return expr
  elif type(expr) == str:
    return expr
  elif type(expr) == Symbol:
    return lookup_symbol_value(expr)
  elif type(expr) == The_Empty_List:
    return expr
  elif type(expr) == Pair:
    return expr
  else:
    return "not implemented (eval)"

def scheme_print(expr):
  if type(expr) == Pair:
    sys.stdout.write('(')
    p = expr
    while p.cdr != the_empty_list:
      scheme_print(p.car)
      sys.stdout.write(' ')
      p = p.cdr
    scheme_print(p.car)
    sys.stdout.write(')')
  else:
    sys.stdout.write(str(expr))

def repl():
  while 1:
    scheme_print(scheme_eval(scheme_read(raw_input("> "))))
    print

This version of read_pair and scheme_read is incredibly naive. In fact it only works on a very limited subset of valid pairs. The reason for this is the use of strings as arguments to read_pair and scheme_read. For instance this version cannot read nested lists and it is also necessary to put a space before the closing ‘)’. I included this version only to demonstrate the mutually recursive nature of read_pair and scheme_read in the simplest way I could think of.

We could continue to use strings as we were doing, but eventually this will get too complicated. To fix this properly we need to treat the input expression as a stream of characters. This is going to require a decent amount of refactoring to the read and read_pair functions.

I probably won’t go into detail with the changes made to scheme_read and read_pair. The important part is that you understand that the purpose of the read function is to translate a series of characters into an abstract syntax tree (AST) of the appropriate data structures. From this point on we will be focusing mainly on the evaluation of this AST.

GOTO | Table of Contents

There are some self-evaluating values we didn’t implement in the last article. Vectors and hash tables are self-evaluating but were left out for simplicity. Strings have also been left out, though they are a pretty essential data type. The reason for leaving them out is that embedded newlines are allowed in Scheme strings and the current implementation makes it difficult to handle this.

This post will focus on symbols (usually known as variables in other languages) and assignment. For this we will need to start focusing on parts of the interpreter other than the reader.

According to R5RS a symbol (or identifier) is a sequence of letters, digits, and “extended alphabetic characters” that begins with a character but cannot begin with a number is an identifier. To make things simple we will only allow (!, -, +, *, /, @, $) as valid initial characters for symbols.

We’ll start by identifying that an expression is a symbol in the reader and we’ll return the string representing the symbol.

def is_number(s):
  c = s[0]
  if c.isdigit():
    return True
  elif c == '.' and s[1].isdigit():
    return True
  elif c == '-' and (s[1].isdigit() or (s[1] == '.' and s[2].isdigit())):
    return True
  else:
    return False

def scheme_read(s):
  s.lstrip()
  c = s[0]
  if is_number(s):
    if '.' in s:
      return float(s)
    elif '/' in s:
      return "not implemented"
    else:
      return int(s)
  elif c == '#':
    if s[1] == 't':
      return True
    elif s[1] == 'f':
      return False
    elif s[1] == '\\':
      if s == '#\space':
        return ' '
      elif s == r'#\newline':
        return '\n'
      else:
        return s[2]
  elif c.isalpha() or c in ('!', '-', '+', '*', '/', '@', '$'):
    return s
  else:
    return "not implemented"

def scheme_eval(expr):
  if type(expr) == int:
    return expr
  elif type(expr) == float:
    return expr
  elif type(expr) == bool:
    return expr
  elif type(expr) == str:
    return expr
  else:
    return "not implemented"

def scheme_print(expr):
  print expr

def repl():
  while 1:
    scheme_print(scheme_eval(scheme_read(raw_input("> "))))

You may have noticed that we didn’t need to modify scheme_eval to allow symbols. This is because both symbols and characters are represented by Python strings in this implementation. This does not conform to the Scheme standard which requires strings and symbols to be distinct data types.

Since Python does not have a symbol type, or even a comparable type we can use as a substitute, we will have to make one. To do this we’ll make a new class “Symbol” to represent symbols and modify eval to check for that. Since we don’t need any additional functionality over a basic string the implementation is very simple.

class Symbol(str):
  pass

def is_number(s):
  c = s[0]
  if c.isdigit():
    return True
  elif c == '.' and s[1].isdigit():
    return True
  elif c == '-' and (s[1].isdigit() or (s[1] == '.' and s[2].isdigit())):
    return True
  else:
    return False

def scheme_read(s):
  s.lstrip()
  c = s[0]
  if is_number(s):
    if '.' in s:
      return float(s)
    elif '/' in s:
      return "not implemented"
    else:
      return int(s)
  elif c == '#':
    if s[1] == 't':
      return True
    elif s[1] == 'f':
      return False
    elif s[1] == '\\':
      if s == '#\space':
        return ' '
      elif s == r'#\newline':
        return '\n'
      else:
        return s[2]
  elif c.isalpha() or c in ('!', '-', '+', '*', '/', '@', '$'):
    return Symbol(s)
  else:
    return "not implemented"

def scheme_eval(expr):
  if type(expr) == int:
    return expr
  elif type(expr) == float:
    return expr
  elif type(expr) == bool:
    return expr
  elif type(expr) == str:
    return expr
  elif type(expr) == Symbol:
    return expr
  else:
    return "not implemented"

def scheme_print(expr):
  print expr

def repl():
  while 1:
    scheme_print(scheme_eval(scheme_read(raw_input("> "))))

Symbols are different from the other types we have implemented so far, they are the first type that is not self-evaluating. To make symbols behave the way they should we have to introduce an environment that stores symbol names and their associated values, as well as modify eval to lookup those values.

We’ll start by creating a new global variable that will hold the environment, which will be called global_environment. This will be a dictionary that holds (name, value) pairs. We also need a function to lookup symbol names in that environment, which we’ll call lookup_symbol_value. Since we don’t have any mechanism for adding new symbols to the global_environment yet, we’ll populate it with a few symbols to use for testing purposes.

class Symbol(str):
  pass

global_environment = {"hello":"world", "number":42}

def lookup_symbol_value(symbol):
  try:
    return global_environment[symbol]
  except KeyError:
    return "symbol not found"

def is_number(s):
  c = s[0]
  if c.isdigit():
    return True
  elif c == '.' and s[1].isdigit():
    return True
  elif c == '-' and (s[1].isdigit() or (s[1] == '.' and s[2].isdigit())):
    return True
  else:
    return False

def scheme_read(s):
  s.lstrip()
  c = s[0]
  if is_number(s):
    if '.' in s:
      return float(s)
    elif '/' in s:
      return "not implemented"
    else:
      return int(s)
  elif c == '#':
    if s[1] == 't':
      return True
    elif s[1] == 'f':
      return False
    elif s[1] == '\\':
      if s == '#\space':
        return ' '
      elif s == r'#\newline':
        return '\n'
      else:
        return s[2]
  elif c.isalpha() or c in ('!', '-', '+', '*', '/', '@', '$'):
    return Symbol(s)
  else:
    return "not implemented"

def scheme_eval(expr):
  if type(expr) == int:
    return expr
  elif type(expr) == float:
    return expr
  elif type(expr) == bool:
    return expr
  elif type(expr) == str:
    return expr
  elif type(expr) == Symbol:
    return lookup_symbol_value(expr)
  else:
    return "not implemented"

def scheme_print(expr):
  print expr

def repl():
  while 1:
    scheme_print(scheme_eval(scheme_read(raw_input("> "))))

That will be all for this post. Setting a symbol in Scheme uses the syntax: (define symbol value). This requires that we have implemented lists, which we haven’t done yet. The next article will most likely be about implementing lists so we can add this functionality.

GOTO | Table of Contents

In this part we will implement a few more self-evaluating values, namely floating point numbers, booleans, and characters. To implement these we will mainly be modifying the read function.

In the previous post I mentioned that the logic behind recognizing integers was a bit too simple. First in Scheme the “-” character is a legal character in identifiers (unlike most other languages). For this reason we can’t assume that an expression that starts with a “-” is a number. So if the first character is “-” we have to check that the second character is a digit. But that’s still a bit too simple because -.3 is also a number.

After we identify that the expression is a number, all we have to do is figure out if it is a float or integer. This is simple to achieve by testing if “.” is in the number. The difficulty here comes in identifying the number in the first place. Here are some numbers that we have to identify.

42
-42
.42
42.
-.42
0.42
-0.42
4/2

To identify a number we’ll create a function is_number that takes an expression as a string and returns True if the expression is a number or False if it is not. We’ll remove some of the logic from scheme_read and move it into the is_number function. For now we won’t implement rational numbers (4/2) however we should recognize them and return “not implemented”.

Beyond that we only have to modify eval to recognize floats as well. Since they’re also self-evaluating we only have to return the value the same as an integer.

def is_number(s):
  c = s[0]
  if c.isdigit():
    return True
  elif c == '.' and s[1].isdigit():
    return True
  elif c == '-' and (s[1].isdigit() or (s[1] == '.' and s[2].isdigit())):
    return True
  else:
    return False

def scheme_read(s):
  s.lstrip()
  c = s[0]
  if is_number(s):
    if '.' in s:
      return float(s)
    elif '/' in s:
      return "not implemented"
    else:
      return int(s)
  else:
    return "not implemented"

def scheme_eval(expr):
  if type(expr) == int:
    return expr
  elif type(expr) == float:
    return expr
  else:
    return "not implemented"

def scheme_print(expr):
  print expr

def repl():
  while 1:
    scheme_print(scheme_eval(scheme_read(raw_input("> "))))

There are some things that are not covered by this implementation, for instance the expression “-.” will cause the interpreter to crash. This is caused by checking s[2] in line 7 without first verifying that this is a valid index (which, in the case of “-.” it isn’t). This should signal a syntax error, however since this project is focused on simplicity and not robustness I’ll leave that as an exercise to the reader.

Also you will notice that I have left out test cases in this post. Mostly this is because the number of tests are becoming quite large. I have created another file which I called scheme_test.py where the tests for the interpreter are located. Checkout the git repo for the test suite as well as the full sourcecode for the interpreter.

Booleans

Booleans are relatively straightforward to implement. The syntax for true and false is #t and #f respectively. Like integers and floats, we’ll use Python’s native boolean type to represent booleans for now.

def is_number(s):
  c = s[0]
  if c.isdigit():
    return True
  elif c == '.' and s[1].isdigit():
    return True
  elif c == '-' and (s[1].isdigit() or (s[1] == '.' and s[2].isdigit())):
    return True
  else:
    return False

def scheme_read(s):
  s.lstrip()
  c = s[0]
  if is_number(s):
    if '.' in s:
      return float(s)
    elif '/' in s:
      return "not implemented"
    else:
      return int(s)
  elif c == '#':
    if s[1] == 't':
      return True
    elif s[1] == 'f':
      return False
  else:
    return "not implemented"

def scheme_eval(expr):
  if type(expr) == int:
    return expr
  elif type(expr) == float:
    return expr
  elif type(expr) == bool:
    return expr
  else:
    return "not implemented"

def scheme_print(expr):
  print expr

def repl():
  while 1:
    scheme_print(scheme_eval(scheme_read(raw_input("> "))))

Characters

The syntax for characters is similar to booleans, characters start with #\ and are followed by, of all things, a character. Since we’re already testing for the # character for booleans we can recognize characters in the same conditional.

There are only a few things to note about characters in Scheme, for the most part characters are of the form #\x where x is any character a-Z or special characters. However there are two special characters that we also need to recognize #\space and #\newline.

The only thing left is to decide how to represent a character. There are two obvious choices here, we could create a character type or we could just use strings. Since Python does not have a separate character type we’ll go the easier route and use strings.

def is_number(s):
  c = s[0]
  if c.isdigit():
    return True
  elif c == '.' and s[1].isdigit():
    return True
  elif c == '-' and (s[1].isdigit() or (s[1] == '.' and s[2].isdigit())):
    return True
  else:
    return False

def scheme_read(s):
  s.lstrip()
  c = s[0]
  if is_number(s):
    if '.' in s:
      return float(s)
    elif '/' in s:
      return "not implemented"
    else:
      return int(s)
  elif c == '#':
    if s[1] == 't':
      return True
    elif s[1] == 'f':
      return False
    elif s[1] == '\\':
      if s == '#\space':
        return ' '
      elif s == r'#\newline':
        return '\n'
      else:
        return s[2]
  else:
    return "not implemented"

def scheme_eval(expr):
  if type(expr) == int:
    return expr
  elif type(expr) == float:
    return expr
  elif type(expr) == bool:
    return expr
  elif type(expr) == str:
    return expr
  else:
    return "not implemented"

def scheme_print(expr):
  print expr

def repl():
  while 1:
    scheme_print(scheme_eval(scheme_read(raw_input("> "))))

Once again there are some problems with this implementation, mostly the same as before. #\ will crash the interpreter, also #\spcae should produce a syntax error but instead gives ‘s’.

You have probably noticed by now that the hardest part of writing a parser is not recognizing valid expressions, but dealing with invalid ones. Most of the work in writing a language is dealing with corner cases.

GOTO | Table of Contents

Unlike Lispy in Scheme the first article of this series will cover only a language of integers. This is because we will have to implement quite a bit more to get the same features.

There’s 4 layers to a basic Scheme referred to as the REPL. REPL stands for Read, Evaluate, Print and Loop. The REPL is more commonly referred to as an interpreter outside the Lisp community.

Most Scheme expressions are evaluated in the same way*. First we read the input from stdin or a source file and construct the appropriate data. Then we evaluate the data to get a result and print that result to stdout. Finally we loop and do it all again until the EOF is reached or there is no more input.

We’ll start by implementing each of these layers one-by-one for a language of integers. This will allow us to understand the basic structure of the interpreter without complicating things too much.

We’ll start with the read layer, which we’ll call scheme_read to avoid any name clashes. The purpose of the scheme_read function will be to analyze a string and return a representation of an integer. One of the decisions we will have to make will be how to represent the integers. There are two obvious choices here, we could make an integer object or we can use regular Python integers. In this implementation we’ll use standard Python types wherever possible.

def scheme_read(s):
  s.lstrip()
  c = s[0]
  if c == '-' or c.isdigit() and '.' not in s:
    return int(s)
  else:
    return "not implemented"

scheme_read("42")
#===> 42
scheme_read("-35")
#===> -35
scheme_read("7.5")
#===> "not implemented"

Let’s take a moment to break down the scheme_read function line by line.

  s.lstrip()

The first thing we do is remove any whitespace on the left-hand side of the expression. This makes the expression ” 42″ equivalent to the expression “42″ as it should be since whitespace is only significant in Scheme as a delimiter.

  c = s[0]

Next we set a variable c to the first character in the string. This is not strictly necessary but it makes the code a bit cleaner.

  if c == '-' or c.isdigit() and '.' not in s:

Then we use the first character to decide what type the expression is. An integer can only start with “-” or a digit and cannot contain the “.” character. This logic is actually a bit too simple, but for now it works as expected.

    return int(s)

After recognizing that the input is an integer we construct a representation of that integer in Python and return it.

  else:
    return "not implemented"

If the expression is not an integer we will simply return the string “not implemented” for now. This is a very simplistic version of a syntax error and is good enough for now.

The read layer of our language of integers is now complete. Next we’ll move on to the eval layer.

def scheme_read(s):
  s.lstrip()
  c = s[0]
  if c == '-' or c.isdigit() and '.' not in s:
    return int(s)
  else:
    return "not implemented"

def scheme_eval(expr):
  if type(expr) == int:
    return expr
  else:
    return "not implemented"

scheme_eval(scheme_read("42"))
#===> 42
scheme_eval(scheme_read("-35"))
#===> -35
scheme_eval(scheme_read("7.5"))
#===> "not implemented"

Integers are self-evaluating in Scheme, therefore all we need to do is return the expression if the expression is an integer and return “not implemented” if it is not. At this point all eval does is echo the input back and I think it’s simple enough to warrant skipping the line-by-line explanation.

The print layer is next and since we’re using Python’s integer type to represent the integers it is even more simple than eval. All we have to do is print the expression to the screen.

def scheme_read(s):
  s.lstrip()
  c = s[0]
  if c == '-' or c.isdigit() and '.' not in s:
    return int(s)
  else:
    return "not implemented"

def scheme_eval(expr):
  if type(expr) == int:
    return expr
  else:
    return "not implemented"

def scheme_print(expr):
  print expr

scheme_print(scheme_eval(scheme_read("42")))
#===> 42
scheme_print(scheme_eval(scheme_read("-35")))
#===> -35
scheme_print(scheme_eval(scheme_read("7.5")))
#===> not implemented

Technically both scheme_eval and scheme_print don’t really add anything to this interpreter. However we will need them very soon and they form the basis for just about everything else, so it’s important to understand the layout of the interpreter now.

The only thing left for our extraordinarily simple interpreter is the loop layer. You’ve already seen what it will do on each iteration in the scheme_print tests so I won’t delay any further.

def scheme_read(s):
  s.lstrip()
  c = s[0]
  if c == '-' or c.isdigit() and '.' not in s:
    return int(s)
  else:
    return "not implemented"

def scheme_eval(expr):
  if type(expr) == int:
    return expr
  else:
    return "not implemented"

def scheme_print(expr):
  print expr

def repl():
  while 1:
    scheme_print(scheme_eval(scheme_read(raw_input("> "))))

repl()
> 42
42
> -35
-35
> 7.5
not implemented

With the REPL completed we have finished our language of integers. In the next post we’ll continue expanding our language with other self-evaluating values by adding floats, characters and booleans.

GOTO | Table of Contents

* With the exception of macros, which I’ll hopefully get into later on in this series. However the process for evaluating macros is very similar.

I’ll start with some of the most basic git commands for creating a repo, adding files and commiting changes to those files.

Create a new repo

To create a new repo, first create a new directory and name it whatever you like. Then from the command line cd into that directory and use the following command.

$ git init

This will create a new repo in that directory. If you already had files in that directory, the git init command will not automatically add those files to the repo. To add a file use the add command.

Add files to repo

$ git add filename

Where filename is the name of the file you want to add. However this is not enough to add the file to the repo, it simply stages the file to be added in the next commit. To commit your changes to the repo you need to use the commit command.

Commit changes

$ git commit -a -m "Type your commit message here"

The -m option allows you to write a commit message that describes the changes you made. If this option is not present the default editor will be opened for you to write a commit message, they are mandatory. The -a tells git to commit all the changes from every file. If you only want to commit changes from a specific file, you can specify that file instead of -a.

$ git commit filename -m "Only changes in filename are committed."

Push changes to remote repo

So far all of the changes and commits have been done locally. If you want to upload your changes to a server such as github you would use the push command. However before you can do this you need to tell git where that remote repository is. Your source code hosting site should give you instructions on how to do this. Here’s an example of the command I needed to run to upload to github for Scheme in Python.

git remote add origin git@github.com:jacktrades/Scheme-in-Python.git

You will only need to do this once. Afterward you can just use the push command.

$ git push -u origin master

This will upload the master branch to the server. If you don’t do any branching master will be the only branch you have.

Creating and changing branches

Creating a new branch in git is simple, just use the branch command.

$ git branch v0.01

Here a new branch was created and named v0.01. You could choose any name you want for the branch, though I’m not sure exactly what characters are allowed.

Even though the v0.01 branch was created, git does not automatically switch you into it. To find out what branch you are in you can use the branch command without any other arguments.

$ git branch
* master
  v0.01

The branch name with the asterisk(*) next to it is the branch you are currently in. To change to another branch use the checkout command.

$ git checkout v0.01

This changes the current branch to v0.01, to verify this you can use the branch command again.

$ git branch
  master
* v0.01

When you make changes and commit them you will be commiting to the current branch (in this instance the v0.01 branch). If you make some changes, commit them then switch back to the master branch, you’ll find that the commits are not there.

Pushing branches to a remote repo

To upload the new branch to the server you need to tell the push command which branch you want to push. So for the v0.01 branch you would need to do this.

$ git push origin v0.01

To push changes made in multiple branches you can leave off the branch at the end of the push command.

git push origin will push all changes on the local branches that have matching remote branches at origin.
- docgnome from stackoverflow

Merging

To merge one branch into another first checkout the branch that you want to merge into. For this example we’ll merge the v0.01 branch into master. So first we checkout the master branch then do a merge command.

$ git checkout master
$ git merge v0.01

Links

Git Book – Basic Branching and Merging

StackOverflow – How do I fix merge conflicts in Git

* That’s all for now, but I’ll come back and add commands for merging and how to resolve conflicts once I get the chance to use those features. This page will likely be updated a number of times as I learn more about git.

You are in the worst position to choose for yourself because you don’t have enough knowledge yet to assess the pros and cons.

If you ask people that do know, you will get tons of different responses.

Most important things:

• There are enough learning materials.
• The community is active and helpful to newcomers.
• The language is difficult enough to challenge you, but easy enough not to discourage you.

Why it doesn’t matter:

• Over the course of the next few years you will learn many languages (if you plan on being any good).
• People say that if you pick an easy language (Python or Javascript) you will not learn the important concepts. This can be true, but only if you stop learning a few months in.
• You’re going to do it wrong in the beginning no matter what language you pick.
• Read How Did Things Ever Get This Good? by James Hague. The language you pick really doesn’t matter one bit.

The most important thing when learning to program is staying motivated. The easiest way to stay motivated is to work on projects that you think are cool and actually complete them. If you pick a language that is too hard, you are likely to give up in frustration. Nothing is stopping you from learning the harder languages after you pick up the basics of the easier ones.

Pros and Cons of Specific Languages

In order of best to worst languages to learn as your first IMHO.

Python
Pros: Beginner friendly, lots of community support for newbies, lots of libraries to get you going quickly
Cons: If you don’t continually push yourself to learn, you can miss out on a lot of important computer science concepts that Python abstracts away for you.

Javascript
Pros: Very easy to start with, lots of libraries and code in the wild for you to study.
Cons: Community is full of non-programmers, you will pick up bad habits,

Scheme
Pros: Beautiful language, great books for intermediate level study
Cons: Small community, not very beginner friendly, few books that start at the ultra-noob level.

C
Pros: Will force you to learn all of the difficult concepts.
Cons: It will be hard. There are many traps and pitfalls that await you.

Java
Pros: Huge community and libraries
Cons: A sizable portion of people who only know Java (The perils of Java schools), forces you into a single way of thinking about programming. I strongly recommend staying away from Java as a first language.

More Tips

You should spend about a year getting to know your first language. After that you will want to start learning other languages. In my opinion you can’t call yourself a programmer until you are at least familiar with 3 languages. Meaning that you have written a small/medium sized program in each (~3,000 lines each).

But that’s not all. Programming isn’t really about languages at all! Programming is about being familiar with data structures, algorithms and other fundamental computer science concepts and how they apply to solving real problems. In the end it’s all about experience, build applications and solve problems and eventually you will look back on what you have done and say, “Now I think I’m a programmer”.

But wait! That’s not all! Next year you will look back on the year before and say, “I had absolutely no idea what I was doing last year.” The year after you’ll say the same, and the year after that, and the year after that, and so on. Learning to program is a never-ending process that will keep challenging you until you decide to hang up your hat.

Also read Tips for a New Programmer.

function Shape(x, y) {
    var self = {};

    self.x = x;
    self.y = y;

    self.toString = function() {
        return 'Shape at ' + self.x + ', ' + self.y;
    };

    return self;
}

function Circle(x, y, r) {
  var self = Shape(x, y);

  self.r = r;

  var _baseToString = self.toString;
  self.toString = function() {
    return 'Circular ' + _baseToString(self) + ' with radius ' + r;
  };

  return self;
};

var mycircle = Circle(2, 3, 5);
console.log(mycircle.x === 2);
console.log(mycircle.y === 3);
console.log(mycircle.r === 5);
console.log(mycircle.toString());

With this pattern you have to use the ‘self’ variable (which you can name anything you wish, the original was named ‘that’) to access and modify attributes of the class just like in Python. Unlike Python you must explicitly declare the self variable at the beginning of the class and return it at the end. Also if you are going to override an attribute in the parent class, but still need to access the parent attribute, you must rename it before overriding it, otherwise the reference to self.toString will end up in an infinite recursive loop.

When I ran my blog through the validator at w3c I noticed two problems. The first was with the Page Links To plugin using target=”_blank”. This was simple enough to fix, just uncheck the “Open this link in a new window” box. It’s probably better to let your visitors choose where they want their links to open, after all they can always middle click to open in a new tab.

The second problem was with SyntaxHighlighter Evolved’s use of <meta id=”syntaxhighlighteranchor” …> in the head tag. This is not valid html and was a bit harder to fix. I came across this post explaining the problem and rational behind the design while Googling for a solution. It seems that the id tag was used to provide a method for users to customize the plugin without editing the source files. Since I quite enjoy digging into the source files, this wasn’t a show stopper for me. In the end my solution was simple. Remove the feature and get on with my life.

The first thing to do is to locate this line…

		echo '<meta id="syntaxhighlighteranchor" name="syntaxhighlighter-version" content="' . esc_attr( $this->pluginver ) . '" />' . "\n";

and remove the id=”syntaxhighlighteranchor” tag.

The plugin uses document.getElementById(“syntaxhighlighteranchor”) twice so you have to make two changes. Instead of using the .insertBefore method to inject the css into the head element, use the .appendChild method.

		document.getElementsByTagName("head")[0].insertBefore( corecss, document.getElementById("syntaxhighlighteranchor") );
// becomes
		document.getElementsByTagName("head")[0].appendChild( corecss );

and

		document.getElementsByTagName("head")[0].insertBefore( themecss, document.getElementById("syntaxhighlighteranchor") );
// becomes
		document.getElementsByTagName("head")[0].appendChild( themecss );

You might want to put a comment in the sourcecode describing your change so you know what you did in the future in case it breaks. If you really think you’re going to forget, you can always write a blog post about it too.

Again I can’t say enough good things about SyntaxHighlighter Evolved. If you have an idea for a fix for this that preserves the original feature of being able to customize the plugin through css without delving into the sourcecode, please leave it in the comments.

Now that we have the data to draw and know how to move things on the canvas, we need to focus on making the chart look more like a normal chart. This post will focus on the drawing algorithms that will turn our data into a nice display and redrawing that same chart at different time periods/scales. Here’s the screenshot representing our progress from last time.

The first step is to find out the lowest value that will have to be drawn on the chart. We can align all the rest of the values with the bottom of the chart if we subtract the lowest value from each value on the chart. To do this we first have to traverse the sequence of data that will be shown and look for the lowest value.

var width = 600, height = 500
var chart = document.getElementById("chart");
var c = chart.getContext("2d");
chart.width = width; chart.height = height

//___________________________________________________________________________//
// Query and Init Procedures
//___________________________________________________________________________//

function getUrl(symbol) {
  return "http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20csv%20where%20url%3D'http%3A%2F%2Fichart.finance.yahoo.com%2Ftable.csv%3Fs%3D" + symbol + "%26d%3D2%26e%3D27%26f%3D2011%26g%3Dd%26a%3D0%26b%3D1%26c%3D2000%26ignore%3D.csv'&format=json&callback=?";
}

$.getJSON(getUrl("ibm"), init);

function init(result) {
  var data = result.query.results.row;
  drawRects(data, 1);
}

//___________________________________________________________________________//
// Chart Drawing Procedures
//___________________________________________________________________________//

function drawRects(d, s) {
  var e = s + 30;
  var low = d[s].col3;

  for (var i = s; i < e; i++) {
    if (d[i].col3 < low) { low = d[i].col3 }
  }

  for (var i = s; i < e; i++) {
    c.fillRect(i*30, height-(d[i].col1-low), 20, d[i].col1 - d[i].col4);
  }
}

The next thing to do is scale up the bar heights to match the height of the chart. For this we also have to determine the highest high. Then we take the difference between the highest high and the lowest low and divide the chart height by that number, this gives us our multiple. Then we take the multiple and use it to compute the lower starting position as well as the height of each bar.

Before we were using the open data, however this won’t always give the lowest price. On any day the stock price fell, the open will represent the highest price. We don’t have to compute if the open is lower or higher than the close because our data gives us the low and high prices for the day. This is also beneficial as the low or high from the day could be significantly lower/higher than the open or close.

Also we’ve been drawing our chart backward, to flip it we need to subtract the offset from the width of the chart. In addition we should put in some color to tell what days are up or down. Since code is probably the best description of this formula I’ll stop trying to explain it and just show you the algorithm. I’m sure there is a cleaner/simpler solution (or maybe this solution is even too simple) however this can be updated a bit later down the road.

var width = 600, height = 500
var chart = document.getElementById("chart");
var c = chart.getContext("2d");
chart.width = width; chart.height = height

//chart.onclick = moveRects;

//___________________________________________________________________________//
// Query and Init Procedures
//___________________________________________________________________________//

function getUrl(symbol) {
  return "http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20csv%20where%20url%3D'http%3A%2F%2Fichart.finance.yahoo.com%2Ftable.csv%3Fs%3D" + symbol + "%26d%3D2%26e%4D04%26f%3D2011%26g%3Dd%26a%3D0%26b%3D1%26c%3D2000%26ignore%3D.csv'&format=json&callback=?";
}

$.getJSON(getUrl("ibm"), init);

function init(result) {
  var data = result.query.results.row;
  console.log(data[1].col0);
  drawRects(data, 1);
}

//___________________________________________________________________________//
// Chart Drawing Procedures
//___________________________________________________________________________//

function drawRects(d, s) {
  var e = s + 15;
  var low = d[s].col3;
  var high = d[s].col2;

  for (var i = s; i < e; i++) {
    if (d[i].col3 < low) { low = d[i].col3 }
    if (d[i].col2 > high) { high = d[i].col2 }
  }

  var mul = height / (high - low);

  for (var i = e; i > s; i--) {
    if (d[i].col1 < d[i].col4) { c.fillStyle = "#00f"; }
    else { c.fillStyle = "#f00"; }
    c.fillRect(width - (i*30),
               height - (mul * (d[i].col1-low)),
               20,
               (mul * (d[i].col1 - d[i].col4)));
  }
}

And here’s the Yahoo Finance chart as comparison…

Finally here’s a snapshot of pyTrade, the father of JamochaTrade. Something similar to this is the ultimate goal of JamochaTrade.

As you can see our chart now looks a lot more like a stock chart! However, we still have a long way to go. Wicks, labeled axes, hover tags and much more need to be added, not to mention different styles of charts (line, mountain, dot, etc.). There are also a few bugs with our current implementation that need to be worked out.

The next article will concentrate on refactoring and debugging. Notably we’ll move our code into a namespace, as was discussed in the previous article. We’ll also rip out the drawing logic and put it into a function so we can make a more general drawing procedure that can handle different types of bar charts (Candlestick, OHLC, HLC).

GOTO Part 7 | Good news and bad news

Get JamochaTrade on GitHub

Here’s a snippet from the Zen of Python that I think should be included in some theoretical incantation of the Zen of Javascript:

Namespaces are one honking great idea — let’s do more of those!

Namespaces are an important concept in any programming language. Programming without them is inevitably an exercise in tracking down hard-to-find bugs that occur when a definition is clobbered by an unrelated assignment.

Here’s the problem: There is a single global namespace for all the scripts running in a page!

This means that defining: a = 42 in the file display.js will clobber the: a = 7 definition in database.js! Let me restate that. When you are creating global variables in a Javascript file, you must know about ALL the global variables created in ALL the other script files.

This may be possible for small apps that you write by yourself. However for large apps and/or when multiple programmers are working on the same project, this is a nightmare. Luckily there is a solution and it’s called the namespace pattern.

The namespace pattern is very similar to, but not quite the same as, the module pattern. The module pattern is designed to allow private/public attributes of an object, which has benefits and drawbacks that are outside the scope of this article.

Links that describe these patterns can be found at the bottom of this page. It is not within the scope of this series to describe all the aspects of the module pattern. I will, however, give a short overview of the two approaches that I think are worthy of consideration. The first is the module pattern and the second doesn’t have a name that I have found, so I’ll call it the Prototype Pattern.

Module Pattern

var MODULE = (function () {
  // all code here
  return {
    // return everything you want to be public
  }
}());

Prototype Pattern

var MODULE = {};
MODULE.variable = 42;
MODULE.fn = function (x) {return x*x};

With the module pattern you must declare everything you want to be public. With the prototype approach you must prefix every attribute creation/access with the namespace. I have read some articles that suggest that prototypes are more efficient than constructor functions. I simply don’t know enough about Javascript at the moment to make that claim. Please research this yourself if it is important to you, don’t take my word for it.

You may come across some “Module Pattern Considered Harmful” type articles when researching. To save you some time here’s the argument: the module pattern is not a solution for every conceivable problem, therefore it should be avoided. Some of these articles do raise valid use cases where the module pattern is not applicable, if you’re interested in that they are a good read. Just don’t drink the kool-aid and believe that they are always useless/harmful.

Most of the issues with the module pattern revolve around private attributes. I have spent most of my time programming in Python and Scheme and, in the general case, I haven’t had a need for private attributes. However the real benefit of the module pattern (IMHO) is not private attributes, but avoiding the global namespace problem. A problem for which the module pattern is an adequate solution.

GOTO Part 6 – Drawing JSONP Data on the Canvas

Get JamochaTrade on GitHub

Links

JavaScript: The Good Parts
A Google Tech Talk given by Doug Crockford. This talk is a great overview of the good and the bad of Javascript for experienced programmers.

Javascript Programming Patterns
A good article that covers a number of methods of tackling the namespace problem.

JavaScript Module Pattern: In-Depth
More information about the module pattern.

I use Kate and I’m proud of it.

Yes Kate, the KDE Advanced Text Editor. Kate is more than a text editor but not quite an IDE, yet is very versatile. From what I’ve read Kate is not used very much by developers, I personally love it and here’s a few reasons why…

Kate works with just about anything I’ve thrown at it:

I do a decent amount of Scheme development and Kate does a surprisingly good job at both highlighting and indenting Scheme. Parenthesis are colored at all times and are highlighted as a matching set when the cursor is over them. It is one of the few editors that I have used that not only works adequately well with Scheme but also has a modern interface and handles other languages very well.

I also do a lot of Python, and Kate has phenomenal support for that. When I did projects in C, Javascript, HTML and even plain text files, Kate supported them well. Kate might not be the best at any one thing, but since it’s usually close to the top, I don’t have to waste any time learning yet another editor/IDE just because I’m doing a project in a new language. Even though it doesn’t take that long, I’d rather focus on developing (and writing blog posts about my editor).

Sessions

Kate has really good support for sessions. You can choose to have your last session open automatically, or use a Kate launcher that lets you choose sessions. Any files you open will be automatically saved to the current session (unless you’re using the anonymous session feature). I rarely “use” sessions in Kate they just do their thing with little interference from me, except for the occasional “Sessions” -> “Quick Open Session”.

Tabs, Documents and the File Manager

Managing the documents in your current session is both easy and customizable. There is a toolbar on the left hand side that has both an open documents manager and a filesystem browser. Both are easy and intuitive to use, but if that’s not enough there are extensions that offer tabs as well. In fact there are two, an older extension “Tab Bar” and a newer and sleeker looking “Tabify”.

Terminal and External Tools

Most of the languages that I use have some sort of REPL. In almost all these cases using the REPL from the command line is not only easy, but is “the way to do it”. Kate offers an embedded terminal that allows quick testing of code snippets in a REPL*. In addition the terminal can be set to auto-sync with the current document, which comes in handy in a number of situations.

With External Tools, Kate can be set up to compile/execute your files just about any way you see fit. External tools can be used for lots of other scripting tasks too, though I haven’t played with it much and can’t comment further.

Find and Regular Expressions

Kate has a pretty powerful search/replace feature including highlighting of all found terms. It’s not just for the document you have open though, Kate can search through multiple files or the entire hard drive if you want. You can also type your query as a regular expression giving you that much more power to find what you’re looking for.

Code Folding, Bookmarking and Split Views

Common IDE features such as code folding and bookmarking are also present in Kate. You can also split the view pane of the editor, horizontally or vertically, allowing multiple documents to be viewed and edited at once.

Shortcuts and VI mode

Just about every command in Kate can be set as a keyboard shortcut. This makes things like the terminal plugin feel right at home. In addition there is also a VI mode that provides Vi(m) like editing. Though not 100% compatible with VI(m) commands, it is an interesting feature. Unfortunately not one that I have any experience using yet, so you can check here and here for more info.

Word/Code Completion and Auto-Braces

One of the things I really like in Kate is word completion. It’s a bit different from the code completion that is commonly found in IDEs. The word completion in Kate is based off other words that are present in the file. This means that you won’t be able to do someObject.met and see all the attributes/methods of that object that start with “met”. However you can use auto-completion when you’re writing anything from C to Scheme to HTML to README files.

Code Tools

Kate also comes with a number of tools for working with code. A comment/uncomment command that works with just about any language and style of comment. Indent/Dedent, support for Unix, Windows and Mac newlines and tons of character encodings, word wrap and spell checking are available and that just barely scratches the surface.

Plugins, Extensions and Scripting

Over 20 plugins and extensions are available (some for KatePart and some for Kate) that add more functionality. Plus you can write KatePart scripts in JavaScript that can add tons of features or just implement a quick hack to get things done.

There are many good things that coding in Kate brings.
These are but a few of my favorite things!

Links

The Kate Editor Homepage

Using Kate As a Web Editor