Race conditions can be notoriously difficult to find, reproduce and debug.

To start with we have a variable a that is set to 7. I’ll show you what can happen, step-by-step, when two pieces of code try to use and modify that variable.

Both a = a + 35 and a = a – 10 will execute “at the same time”. I have “at the same time” in scare quotes because we don’t really know what “at the same time” means. Computers process instructions sequentially, this means that one operation has to come before the other. Let’s see how that can play out…

From looking at the code in the beginning, you may have guessed that the value of a after these operations would be 42, and that was indeed the intent. However, because of the race condition the value of a ended up being -3. Not at all what we wanted!

Probably the most common way to handle race conditions is to use locks. Here’s how the above process would look using locks…

Locks prevent race conditions by making sure that only one piece of code can access a memory location at any time. Locks can become very complicated to manage and are themselves the source of many bugs. Consider what would happen if some piece of code did not unlock the memory it was modifying! Or worse, did not lock it in the first place, in which case we fall back to the race condition.

Locks are not the only way to handle race conditions (though they are the most common). Another way to handle race conditions is to use immutable variables. This means that no code is allowed to change the value of a. Immutable variables will be the subject of another WITH article, so I won’t explain them here.

GOTO Table of Contents

Are you done? How could you be? Whatever you wrote down I could make a better by adding another 3 at the end, or another trillion 3′s at the end. How about the biggest number you can think of? Any number you can write down I can make better by adding one more digit.

So you can see that we have to put some limits somewhere. The limits that we put on numbers in a computer correspond to the amount of memory it takes to represent those numbers.

To understand this, we need to know a little bit about computer memory. Computers store information in bits. A bit is a binary number (1 or 0) that represents the on or off state of the memory. Since a single bit is often too little to use effectively, computer memory is organized into blocks of 8 bits called bytes.

The 8 bit byte can represent the numbers 00000000 through 11111111 in binary, which is 0 to 255. Your computer’s memory is organized into a big row of bytes.

Each address above refers to one byte. A char is one byte. However, what do we do when we need a number bigger than 255? We have to use more than one byte. A number that is 2 bytes can represent a number from 0 to 65,535 and is called a short int.

A 4 byte number (32 bit) could represent 0 to 4,294,967,295 and is called an int or integer.

An 8 byte number (64 bit) could represent 0 to 18,446,744,073,709,551,615 and is called a double.

Another way to look at the basic numeric types is as names for sizes of memory.

What happens when we want negative numbers?

Let’s go back to the char or the single byte. A char can represent 256 values and only 256 values, so to get negative numbers we need to give up some of the positive ones. In fact we’ll give up exactly half of the positive ones and represent the numbers -128 through 127.

When we want to represent negative numbers we use a signed type. Unsigned types represent only positive numbers and 0. Any of char, short int, int or double can be signed or unsigned and each follows the split in half rule.

What about the decimal point?

Numbers with a decimal point in them are called floating point numbers. An entire book can probably be written about floating point numbers as they can become quite complicated. A future WITH article will cover floating point numbers in more detail.

* The size of each char, int, etc can vary depending upon the hardware. This article used common sizes for a 32 bit architecture.
* Teaching Tip: Write down numbers on a whiteboard/paper and measure them with a ruler. Imagine each inch is a block of memory. How many different numbers can you store in one block? How about two or four? What would you name those blocks so you could talk about them with other people? Bonus: how many blocks does it take to write a character?

GOTO Table of Contents

Recursion is often a feared and dreaded word among new computer scientists. Most see it as a mystical process that they have no hope of understanding. Others loudly proclaim that they’ve been programming for years and have never needed recursion, mostly because they see it as a mystical process that they have no hope of understanding.

However recursion is actually pretty straightforward. And that’s a good thing, because there are a lot of problems that have very elegant recursive solutions. We’ll cover some of those when we learn about tree algorithms and sorting in future WITH articles.

For now we’re going to focus on the basics of recursion. Recursive functions are most often used as a form of loop and for this article we’ll use one of the most basic loops; counting. We’re going to describe a process of printing all the numbers from 0 to infinity. Without recursion you would probably write something similar to the following.

def count(n):
  while 1:
    print n
    n = n + 1

count(0)
0
1
2
...

Next we’ll look at the recursive solution.

def count(n):
  print n
  count(n+1)

count(0)
0
1
2
...
998
RuntimeError: maximum recursion depth exceeded

The above code should be pretty self-explanatory, but I’ll explain it a little bit anyway. The function count has only two instructions. First print n to the screen, then call count with n + 1 as the argument. So when we call count(0) it should print 0 then call count(1) which will print 1 then call count(2)…

You probably noticed the error in the above output. We’re going to ignore that for the moment and move on from trying to count to infinity. Let’s just count to 5. To tell when we’ve hit 5 we’ll need to check the value of n each time count is called.

def count(n):
  if n < 5:
    print n
    count(n+1)

count(5)
0
1
2
3
4

As you can see from these examples, a recursive function is simply a function that calls itself. Let’s look at another recursive solution to counting. This time we’re going to use a list of the numbers 0 through 4 and print each number in the list.

def print_list(lst):
  if lst:
    print lst[0]
    print_list([1:])

print_list(range(5))
0
1
2
3
4

In this code we print the first element in the list, then call print_list with the rest of that list from the 2nd element on. Here’s a diagram of the process to make it a bit more clear.

When writing recursive code it is important to include a stopping condition, which was the empty list in the code above. If you don’t include a stopping condition, your code will enter an infinite loop.

Tail-Call Optimization

One problem with recursive code is that most programming languages translate recursive code into something very similar to the diagram above.

Let’s go back to trying to print to infinity again and look at the error when we run the code. Depending on how many lines your terminal logs you may not see this.

993
994
995
996
997
998
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in print_infinity
  File "<stdin>", line 3, in print_infinity
  File "<stdin>", line 3, in print_infinity
  File "<stdin>", line 3, in print_infinity
  ...
  RuntimeError: maximum recursion depth exceeded

Since most languages, Python included, see recursive code exactly like the diagram above, try to imagine how big our diagram would have to be to represent a 1,000! Eventually the language simply runs out of memory to store this huge diagram or it hits a set limit to prevent that from happening.

Let’s take a look at another example, countdown. This function simply counts down from a number, n to 1.

def countdown(n):
  if n:
    print n
    countdown(n-1)

countdown(5)
5
4
3
2
1

Each time Python sees countdown(n-1) it makes a new function call. Because Python is lexically scoped, the function call creates a new frame to hold the bindings and arguments. When the next countdown(n-1) is evaluated a new frame is created inside the last frame. Then again, and again, until it looks like the code below, each new level of indentation is a new frame within the previous frame.

countdown(5)
  print 5
  countdown(4)
    print 4
    countdown(3)
      print 3
      countdown(2)
        print 2
        countdown(1)
          print 1
          countdown(0)

From looking at the code above, it doesn’t make much sense why any language would do this. However if you switch the code for countdown around a little bit, it starts to make sense.

def countdown(n):
  if n:
    countdown(n-1)
    print n

All we did was switch the countdown and print lines, but let’s see how that changes the execution of this function.

countdown(5)
1
2
3
4
5

This version of countdown doesn’t countdown at all! In fact it counts up. To see why, let’s examine the execution of this function.

countdown(5)
  countdown(4)
    countdown(3)
      countdown(2)
        countdown(1)
          countdown(0)
          print 1
        print 2
      print 3
    print 4
  print 5

And a longer, but more descriptive version…

countdown(5)
  if 5:
    countdown(5 - 1)
      if 4:
        countdown(4 - 1)
          if 3:
            countdown(3 - 1)
              if 2:
                countdown(2 - 1)
                  if 1:
                    countdown(1 - 1)
                    print 1
                print 2
            print 3
        print 4
    print 5

In this scenario there is a print instruction after the call to countdown. This means that the language must evaluate countdown before printing the number. The only way to do this is to create a tree structure as depicted above and keep it all in memory until the function has finished executing.

Some languages like Scheme perform tail-call optimization (TCO), also known as tail-call elimination. If the recursive function call is the last thing you do in the function body, there cannot be anymore instructions to perform after the call. Because it is guaranteed that none of the information from the initial frame is needed, there is no reason to keep it around by nesting new frames in it. In other words tail-call optimization transforms a recursive definition into an iterative one.

Language support for tail-call optimization varies widely. Some languages, like Scheme, guarantee that recursive function calls in the tail position will run in constant space. While on the lower end of the spectrum is a compiler like gcc that may run tail calls in constant space, even though the C standard does not support TCO.

Languages that guarantee tail-call optimization generally use recursion as the main form of looping. Because of this function calls in these languages are usually highly optimized.

Here are tail-call optimized versions of count and countdown in Scheme. Notice how the recursive calls are the last calls that are executed in the body.

(define (count n)
  (print n)
  (count (+ n 1)))

(define (countdown n)
  (if (= n 0)
    (print n)
    (begin
      (print n)
      (countdown (- n 1)))))

(countdown 3)
3
2
1
0
(count 0)
1
2
3
...

Mutually Recursive Functions

This is the classic example of a mutually recursive function. These two functions together will determine whether a number is even or odd. Take a minute to look at the code and try to figure out for yourself how it works.

def even(n):
  if n == 0:
    return True
  else:
    return odd(n-1)

def odd(n):
  if n == 0:
    return False
  else:
    return even(n-1)

even(4)
#===> True
odd(4)
#===> False
odd(3)
#===> True

Because everyone likes a good sports analogy let’s imagine two kids playing catch. One kid is named Even and the other is named Odd. Even starts with the ball in his hand and they throw the ball back and forth until they lose count. If Even has the ball last, they threw the ball an even number of times. If Odd has the ball last, they threw the ball an odd number of times. When they finish playing, the kid with the ball screams “I got it!” and the game ends.

The mutually recursive function above works the same way. even subtracts 1 from the number and throws it to odd. odd also subtracts 1 from the number and throws it back to even. When the number reaches 0 the last function to have the number returns, #t if even is the last to have it and #f if odd is the last to have it.

We’ve already learned that a recursive function creates a loop by calling itself. Mutually recursive functions create a loop by calling each other. Like recursive functions, mutually recursive functions must check for some condition to prevent an infinite loop.

GOTO Table of Contents

Almost every program you will ever write will have errors.

As soon as we started programming, we found out to our surprise that it wasn’t as easy to get programs right as we had thought. Debugging had to be discovered. I can remember the exact instant when I realized that a large part of my life from then on was going to be spent in finding mistakes in my own programs.
-Maurice Wilkes

Syntax Errors

Syntax errors are probably the most common errors in programming. Luckily they are usually also the easiest to fix. Most languages will point out syntax errors as soon as you try to run or compile your program. Usually you will be provided with a file and line number of the offending syntax. Some IDEs check your syntax as you type, providing a sort of spell checker for syntax.

Python

def add(x, y)
  return x + y

File "./test.py", line 2
  def add(x, y)
              ^
SyntaxError: invalid syntax

Chicken Scheme

(define add(x y) (+ x y))

Error: during expansion of (define ...) - in `define' - too many arguments: (define add (x y) (+ x y))

        Call history:

        <syntax>          (define add (x y) (+ x y))

Type Errors

Type errors occur when your code tries to do things like adding an integer and a string together. Depending on the language you use you may be notified of type errors when compiling your program or while your program is running. Type errors are also very common and are a little bit harder to fix.

Python

def add(x, y):
  return x + y
add(1, "a")

Traceback (most recent call last):
  File "./test.py", line 4, in <module>
    add(1, "a")
  File "./test.py", line 3, in add
    return x + y
TypeError: unsupported operand type(s) for +: 'int' and 'str'

Chicken Scheme

(+ 1 "a")

Error: (+) bad argument type: "a"

        Call history:

        <syntax>          (+ 1 "a")
        <eval>    (+ 1 "a")     <--

Logical Errors

Logical errors occur when you write code that performs correctly, but does not give the output that you desire. Logical errors can be the worst kind of bugs. There is rarely any support for detecting them built into the language, as there is technically nothing wrong with the code. These bugs happen somewhat frequently and can range from minor inconvenience to major disruption.

Below is an example of a logical error that would fall into the minor inconvenience category. We’re trying to define a function named add that takes two arguments, adds them together and returns the result. The code we have below does not have any syntax or type errors in it and it is perfectly valid code. However instead of adding the two arguments it subtracts them and we get the answer 0 when we expected the answer to be 4.

Python

def add(x, y):
  return x - y
add(2, 2)
#===> 4

0

Chicken Scheme

(define (add x y)
  (- x y))
(add 2 2)
;===> 4

0

Errors and Error Handling

Each language will have its own way of representing errors. Some languages provide a mechanism called error handling which let’s you control what happens when an error occurs. We’ll get deeper into errors and error handling in an upcoming WITH article.

GOTO Table of Contents

Big O is a notation for describing the worst-case performance of an algorithm or procedure. This post will show real-world examples of big O notation.

O(1) – Constant Time

A constant time algorithm will always take the same amount of time to run.

Example:
Finding a book in a library.

Why:
There are many methods to finding a book in a library, but I’m going to stick with the simplest, ask the librarian.  Assuming the librarian is familiar with their library, it should take about the same amount of time to find a book about mathematics as it would about poetry.

For another example see indexing an array.

O(n) – Linear Time

A linear algorithm runs once for each item(n).

Example:
Finding a CD in a stack of CDs

Why:
The simplest method of finding a CD in a stack of CDs is to just look at the one on top.  If that CD is the one you’re looking for, you’ve found it.  If it is not, look at the next one.  Repeat until there are no CDs.  If the CD we’re looking for is the last one in the stack, we had to look at every CD to find it.

But what about all the times that you look for a CD and the CD you’re looking for is in the middle or on the top?  Big O notation describes the worst-case running time of your algorithm.  It assumes that every search will be for the last CD (or a CD that doesn’t exist).  Big O is not a statistical model of how often a given condition will occur in an algorithm!

O(log n) – Logarithmic Time

A logarithmic algorithm runs log₂(n) times.

Example:
Let’s play a guessing game.  I’m going to pick a number between 0 and 100 and you have to guess it.  You have to try to guess the number in as few guesses as possible.  Every time you guess I will tell you if my number is higher or lower.

Why:
You could devise a number of strategies for playing this game, however one strategy is particularly good at this type of game, binary search.  Each time we guess, we guess the number right in the middle.  By doing this we halve the number of possibilities every time, let’s play a little game…

The game above plays out the worst case performance of an O(log n) algorithm like binary search.  Instead of making 100 guesses at worst, we only make 7!

O(n²) – Quadratic Time

A quadratic time algorithm runs n² times, or n times for each n.

Example:
Check if there are any duplicates in a deck of cards.

Why:
The simplest way to check if there are any duplicate cards in a deck is to pick the first card from the deck. Then compare that card to every other card in the deck. If there are no matches take the second card from the deck. Then compare the second card to every other card in the deck. Continue until you have checked all the cards.

By the time we are done we have looked at each card in the deck 52 times for a total of (52 * 52) 2704 comparisons. There are ways to lessen this number, like not looking through cards that you have already compared. However the most important thing to remember with big O notation is that it always describes the worst-case time, not the average.

Here’s what the process looks like if you don’t re-check cards you’ve already compared.

I’ll come back later and add some more big O notations, so check back. For now I’ll leave you with a question to ponder. What is the big O notation of the diagram above?

Links

Sorting (this will be the topic of another WITH article, but for now…)

Visual Representation of Sorting Algorithms in Javascript

Visualizing Sorting Algorithms

sortvis

Parallel GPU Sorting (pdf)

What different sorting algorithms sound like

GOTO Table of Contents

A tree is a hierarchical data structure that contains a set of nodes. The node is the the basic building block of a tree and to understand trees you have to understand nodes.

A node is a data structure that has a parent, can contain a value and can have 0 or more child nodes. A node with no child nodes is called a leaf node, or more commonly just a leaf.

A node with no parent is referred to as a root node.

Each node can have a child node of its own.

A node can have more than two children too.

By now you should grasp the basic structure of a tree, but what are they good for and where are they used? Well the good news is that trees are all around you. Your operating system’s directory structure is a tree, HTML and XML are both trees. Your filing cabinet could even be considered a tree. A linked list is also a form of tree, where each node has exactly one child node. Let’s take a look at some more trees.

You belong to a tree too!

and of course…

Implementation of Trees

Programming languages have varying levels of support for tree-like data structures. The abstract idea of a tree and the implementation of one are often very different things. Pretty much any nestable data structure can suffice for an ad-hoc tree implementation. For example here’s one of the simplest trees you can implement in Python.

class Node:
  def __init__(self, value=None, *args):
    self.value = value
    self.children = args

# A tree can then be defined as
tree = Node('root', Node(1), Node(2))

Scheme makes it quite a bit easier, both code and data are represented using trees (actually lists).

(define tree (cons 1 2))
tree
;===> (1 . 2)

(define nested-tree (cons 1 (cons 2 3)))
nested-tree
;===> (1 2 . 3)

If you’re wondering why the nested-tree looks like a list, remember that lists are a subset of trees.

I didn’t get to heavily into how trees are implemented in any particular language. Mostly this is because the implementations of tree data structures varies widely from one language to the next and sometimes there are even multiple implementations of trees in a single language.

Many decisions go into a tree implementation and the tools that work with them. Because of this there are many different types of trees, each with its own strengths and weaknesses and tools for manipulating them. Future WITH articles will will go deeper into trees, their implementations and characteristics.

GOTO Table of Contents

A good way to think about functions is as machines. Machines are designed to repeat a process over and over again. Functions share a lot of characteristics with machines, so let’s look at some of the properties of machines we’re already familiar with.

Machines take input and produce output, let’s take a look at a few examples of that. Your dish washer takes dirty dishes as input and returns clean dishes as output, your washer does the same for clothes. Your radio takes radio waves as input and produces music on your speakers as output.

For the most part you don’t have to know how a machine works, you only need to know how to use it. For the moment we’re going to look at functions the same way, not thinking about how they work, just about how to use them. You’ll be happy to learn that you’re already familiar with a few functions, in fact here’s one you use every day.

Using the function above is simple. We don’t care how it works, whether it is carrying it’s ones or counting on its fingers. The only thing that matters to us is that we give it 2 numbers, 35 and 7 and it gives us back 42.

Here’s an example of another function. This one reverses any string that we give it.

Terminology

When we use a function it is referred to as calling that function. So in the above two examples we called the add function and we also called the reverse function.

The input to a function is referred to as the arguments or argument in the case of a single input. Adding arguments to the above we get, we called the add function with the arguments 35 and 7 and we also called the string function with Hello as an argument. Arguments are also commonly referred to as parameters and the two words can be used interchangeably.

The output of a function is most commonly referred to as the return value, though it is referred to as output as well.

Calling Functions

Now that we’ve explored some of the properties and terminology of functions let’s look at how they’re represented in code. Just like before, we are not going to worry about how the functions work just yet. We’re only going to call a few.

Function calls can be represented a number of ways, depending on the language you are using. However the vast majority of mainstream languages use this convention.

____________( ______, ______ )

Instead of a fill-in-the-blanks style as above, programmers usually use a shorthand for describing things that can be anything. Unfortunately there is no standardization for this shorthand, so to know what is a shorthand you need to know what words are never used as real names. The use of foo, bar and baz are commonplace and you will have to get accustomed to reading them eventually.

I will take another approach here. I’ll use prototypes to talk about the basic structure and I’ll use descriptive names in any code. This approach is used in many Scheme manuals and I have found it easy to grasp. For now a prototype will be a source code snippet with no line numbering and all the lines highlighted.

Lets start our introductions to prototypes with a prototype of the add function.

Prototype:

name(argument, argument)

Let’s look at the diagram for add and put these labels on it in red.

Function Arguments

Functions often take arguments of a specific type. For example the add function above only takes numbers. If you attempt to give it a string it will produce an error. In contrast the reverse function will only accept a string.

To make it easier for you to tell what type of argument to use, the prototypes will state the type.

add(number, number)
reverse(string)
length(argument)

Whenever the type of the argument does not matter or can be of multiple types the word argument will be used.

Let’s take a look at a few more functions and how calling them maps out onto our diagram. The first function we’ll look at is max it will take two numbers as input and give us back the largest one. First I’ll show you the prototype then the code used to call max followed by the diagram.

Prototype:

max(number, number)

max(35, 7)
#===> 35

You probably noticed the #===> symbol in the source code. # is a comment in Python, which is the language used for the psudo-code in this series. A comment is completely disregarded by the computer and is only meant for you to read. This series uses the ===> symbol to show you the output of the code that was just run.

Our next example is length which takes a list or string as input and produces the length of that list or string as output. Because it takes a list or a string, we’ll use argument in our prototype.

Prototype:

length(argument)

length("Hello")

Functions can take two different kinds of arguments too. Here’s a function that takes a string and a number as input and gives us back a letter. Its output is the same as finding the letter by starting at the beginning and counting each letter until you get to number then return that letter.

Prototype:

letter(string, number)

letter("Hello", 2)
#===> e

Writing Your Own Functions

I’m going to borrow from SICP here and start with a simple function called squared. This function takes one number as input and returns the result of multiplying that number by itself.

The first thing we’ll do is define what this function should do. You will most likely do this in your head, but here I’ll lay it out like I laid out the functions above. We know we need a number as input and the output should be that number multiplied by itself, so let’s start with a prototype, a fake call, and a diagram like before.

Prototype:

squared(number)

squared(4)
#===> 16

Now that we know what our function should look like when it’s finished, we have to define it. But before we can do that let’s define what a function is a little more precisely.

What is a Function

When we were using functions we could think of them as “black boxes” that we put input into and got output out of. Now that we’re writing our own we need to think about how our functions work. Remember that a machine is made to repeat a process over and over. Functions are exactly the same, they also repeat a process or series of processes over and over.

A function is both a set of directions for performing a task and a machine that performs that task.

The syntax for defining a function can vary widely between programming languages. Some languages require that you declare argument and return types. We’ll cover that a little bit later, for now we’re going to focus on the method used by Python.

To define a function in Python you must supply a function name and its arguments. Here’s the prototype of Python’s def form, followed by a definition of squared.

Prototype:

def name(argument, argument, ...):
  # body

def squared(number):
  number * number

There are two main concepts in the above code that we have not covered yet. The body of a function contains the code that describes the process of the function. It is in the body that the instructions for performing the function are held. When we call the function we will use the body, but for now it is just a description of the process.

squared(4)

When we call the function, the programming language looks up the instructions for the function, then replaces all the argument names with the arguments the function is called with. Here’s a simple view of what your programming language sees when you call squared(4).

def squared(4):
  4 * 4

As you can see the number argument is replaced by the 4 argument. Your programming language will do this behind the scenes for you, but you conceptually do this transformation in your head every time you write a function.

However there is one thing wrong with the above function. When we call it, it doesn’t give us back the answer. In fact it seems like it does nothing at all!

Returning Values

Most mainstream languages require the use of a return keyword to signal that you want to return a value. Python also requires a return statement so we’ll have to add one in to get the return value.

def squared(number):
  return number * number
squared(4)
#===> 16

Combining Functions

Now that we have a function squared that will square a number, let’s look at how to use that in another function.

We’re going to define a new function aoc which will calculate the area of a circle, given a radius as input. The formula for the area of a circle is πr2. We already have a way to square the radius with our squared function, so let’s just write the new function.

def aoc(radius):
  return 3.14 * squared(radius)
aoc(4)
#===> 50.240000000000002

When aoc(4) is called Python looks up the definition of aoc and replaces all occurrences of radius with the number 4. Then Python evaluates the instructions found in aoc. In the instructions Python finds a call to squared so it looks up the instructions to squared and replaces all occurrences of number with the value 4. It then evaluates the instructions in squared and returns 16. squared(4) is then replaced with 16 and Python returns 3.14 * 16 or roughly 50.24.

Examples of Functions in Other Languages

Javascript

function aoc(c) {
  var r = c/2;
  return 3.145*(r*r);
}
aoc(8)
//===> 50.264

Scheme

(define (aoc c)
  (define r (/ c 2))
  (* 3.1415 (* r r)))
(aoc 8)
;===> 50.264

Links

GOTO Table of Contents

A conditional is a mechanism for altering the control flow of a program based on the result of a test. That is a very technical and complicated description of something that is very simple in practice. Let’s look at a small example before continuing.

if temperature > 80:
  run_air_conditioning()

All humans are used to dealing with conditional statements, in fact we probably compute thousands of conditionals every day without even thinking about them. They’re so in-grained in our thinking that I didn’t even have to explain the example above, you just got it.

The most basic form of conditional, which is present in almost every language, is if/else. Each language has slightly different syntax which we’ll get to in a moment, first let’s look at some psudo-code that describes the basic workings of if/else.

if test:
  do_if_test_is_true()
else:
  do_if_test_is_false()

When our programming language comes across this block of code it will first evaluate test. If the evaluation of test equals true, the do_if_test_is_true() function will be called. If the evaluation of test equals false, the do_if_test_is_false() function is called.

Note that if test evaluates to true the do_if_test_is_false() function is never called. This behavior is called branching and a popular way of specifying control flow in a program.

if predicate:
  consequent
else:
  alternate

These are some common terms that refer to the parts of an if/else. Predicate refers to the expression or statement that is to be tested. Consequent refers to the action to be taken if the predicate is equal to true. Alternate refers to the action to be taken if the predicate is equal to false.

You’ve already seen examples from Python (even our psudo-code examples were valid Python), here’s some examples from other languages of if/else.

Javascript

if (2 > 1) 1
//===> 1

if (1 > 2) {
  1;}
else {
  2;}
//===> 2

Scheme

(if (> 2 1) 1)
;===> 1

(if (> 1 2)
    1
    2)
;===> 2

Else If

Another common pattern is if/else if/else. This pattern is equivalent to chaining if/else expressions together. Here’s some psudo-code of an if/else if/else in Python and then a diagram of how it is executed.

if temp > 80:
  run_ac()
elif temp < 60:
  run_heat()
else:
  do_nothing()

As you can see, Python first checks if the temperature is greater than 80. If the temperature is greater than 80, Python will call run_ac() and ignore the rest of the code. If the temperature is less than 80 it will call the elif branch and check if the temperature is less than 60. If the temperature is less than 60, Python will call run_heat() and if the temperature is between 60 and 80 Python will call do_nothing().

Instead of using the keyword elif, we could have written our Python code a different way that emphasizes the nesting.

if temp > 80:
  run_ac()
else:
  if temp < 60:
    run_heat()
  else:
    do_nothing()

Writing our program this way makes it clear that each ‘else if’ branch is just another if/else. However keywords like elif exist for a reason and if you are thinking about nesting your if/else expressions, don’t.

You’ve seen elif in Python, now I’ll show you how some other languages handle it.

Functions in Other Languages

Javascript

if (temp > 80) {
  1;}
else if (temp < 60) {
  2;}
else {
  3;}

Scheme

(cond ((> temp 80) (run-ac))
      ((< temp 60) (run-heat))
      (else (do-nothing)))

As you can see the syntax can vary quite considerably between languages. However the fundamental concept of the if/else is the same in most programming languages.

Switch

Switch is another form of conditional. Using the if/else if/else model above is sufficient for a small number of possible conditions. However as the number of conditions grows the limitations of if/else if/else start to show.

The main problem is that each condition must be checked every time the conditional is evaluated. If there were 100 conditions each one would have to be checked until one of them was true. If the condition that returns true is at the end of that list, it could take a lot of time to find it.

Switch statements are usually found in lower-level languages, however I’ll show you how to implement them in higher-level languages as well (should you need them). A switch in a lower-level language is often implemented as a branch table (or jump table) and can only be done on integers.

Switch in C:

switch(expression_returning_int) {
  case 1:
    printf("Expression returned 1.\n");
    break;
  case 2:
    printf("Expression returned 2.\n");
    /* since there is no break; here it will 'fall through' to default. */
  default:
    printf("Expression returned greater than 1.\n");
    break;
}

In the example above the expression expression_returning_int will be evaluated (it must return an integer). The value of the expression will then be used as an index into the branch table. Control is then transfered to the branch referred to by the expression or the default branch (if supplied).

In the example above a break statement was not supplied for the case 2. If expression_returning_int equals 2 it will print:

Expression returned 2.
Expression returned greater than 1.

The break statement normally stops the execution of that branch. When it is not present the control flow “falls through” to the next branch, and the next, and the next until it hits a break statement or the end of the switch. This is a common characteristic of switch statements, however it is not universal. A switch would still be a switch even if it didn’t allow execution to fall through branches.

Switch in Higher-Level Languages

Many high level languages do not provide a switch statement. However this is rarely a problem as they often do supply data structures which let us represent one easily.

A switch on integers can be done as an array lookup, if you want to learn more about arrays first go to What In The Hell Are Arrays.

If we think about the switch statement above it maps out a little bit like this…

You’ll notice a problem with the above representation. Where is the ‘default’ branch? The default branch would be executed by array_lookup if it did not find a value in the array. The implementation of this will be left as an exercise to the reader as the specifics vary from language to language.

Similarly a switch on strings, or other hash-able values, can be done as a hash lookup. If you want to learn more about hash tables go to What In The Hell Are Hash Tables.

Links

GOTO Table of Contents

A hash table is a combination of two parts; an array, to store the data, and a hash function that computes the index of the array from a key. The best analogy to a hash table is a dictionary. When you use a dictionary you lookup a word (the key) and find the definition (the value).

Hash tables are an improvement over arrays and linked lists for large data sets. With a hash table you don’t have to search through every element in the collection (O(n) access time). Instead you pay a one time cost (the hash function) to retrieve data (O(1) access time).

The Hash Function

The purpose of the hash function is to compute an index from a key. That index is then used to store and retrieve data from the array. Many higher-level languages include some type of hash table and hash function.

As you can see from the diagram above the hash(“string”) function transforms a string into an integer which is then used as the index into the array. This index is used to store or retrieve data in the array.

The amount of time it takes to set or retrieve a value from a hash table is the time it takes to run the hash function plus the time it takes to access an element of the array. For small data sets this could take longer than simply iterating over each element of the array. However, since it is a fixed cost, for larger data sets using a hash table could be much faster.

Assuming we have a function set that takes a key to be hashed as its first argument and a value to be stored as its second argument, we can create the following hash table.

As you can see from the above diagram we store both the key and the value in the table. The reason for this is collisions, which will be explained below.

Collisions

A collision occurs when two different keys hash to the same value. Collisions will occur in even the best designed hash tables, however there are a few reasons why they can occur more frequently. The first reason is a poor hash function that does not distribute the indices evenly throughout the table, this will be explained in more detail below. The second reason is that the array is too small. If you have a 10 element array and you are trying to store 100 key/values in the hash table, there will obviously be a lot of collisions.

In the diagram above “Puff” hashed to the same value as “Peter” causing a collision. Both “Peter” and “Puff” are now stored in the same index of the array. A certain amount of collisions are unavoidable in a hash table, so we’ll need a way to deal with them.

The simplest way to deal with collisions is to make each element of the array a linked list. Then, when a collision occurs, we simply search through the linked list to find the data we want. This way the hash function gives us a ‘starting point’ to begin our search.

With a good hash function and an appropriately sized array collisions are a rare occurrence. However it is possible, though very unlikely, that all the key/values will be stored in one index and we will end up doing a search on a linked list.

What is a good hash function?

The sole job of the hash function is to return a unique array index for each key. This is not always possible and collisions can be expected to occur when the number of elements in the hash table approaches the square root of the size of the hash table’s array. A hash function that does not provide a uniform distribution of keys will cause a lot of collisions, possibly breaking our hash table’s performance down from O(1) to O(n). The only workaround to an ineffective hash function is to make the hash table’s array much larger than the number of elements that it holds, which obviously wastes space.

Hash Table Implementations

Python

Python’s equivalent of a hash table is called a dictionary or dict.

# Creating a dictionary
d = {'a':5, 'b':10}

# Adding key/value pairs
d['c'] = 15

# Get value by key
d['c']
#===> 15

Chicken Scheme

;; Creating a hash-table from an alist
(define d (alist->hash-table '((a . 5) (b . 10))))

;; Adding key/value pairs
(hash-table-set d 'c 15)

;; Get value by key
(hash-table-ref d 'c)
;===> 15

Performance

Indexing: O(1)
Insertion/Removal: O(1)
Search: O(1)

The performance listed above is what is normally found in specific hash table implementations. There are actually a number of different implementations, each with their own advantages, disadvantages and performance guarantees. Specific implementations will have to be presented in future WITH articles.

Links

GOTO Table of Contents

Strings are a data type that consist of a sequence of characters. The simplest way to think about strings is that they are text. Strings are usually formed by “putting quotes around some text” or by using a string function.

Strings come in two varieties. Terminated strings use a special character to signal the end of the string. This character is not allowed to appear in the string, as it would cause the string to end, leaving out the rest. Strings can also be implemented using a length field. These strings do not need a terminating character as the length of the string is stored with the string. In many languages you will not be able to tell which type of string you are using and it is usually irrelevant. However some languages, like C, require you to place the termination character manually.

Strings can use a variety of encodings for the characters contained within. The two most common are ASCII, which can represent 128 different characters, and Unicode, which has support for over 100,000 different characters and allows strings to hold non-English characters. Both forms of encoding are popular today, however most applications are moving toward Unicode characters as the need for international software grows.

One common problem with strings is representing characters that would otherwise be interpreted by the language to mean something else. For instance the string “Have you read the book “1984″ by George Orwell?” would be interpreted as 3 expressions; (1) The string: “Have you read the book ” (2) The number: 1984 (3) The string: ” by George Orwell?”. This is obviously not what we want. In this case we would use an escape character to stop the quotes around “1984″ from terminating the string. Our new string would look something like this (this may vary from language to language): “Have you read the book “1984″ by George Orwell?”. In this string the ” character tells the language not to end the string when it sees the following “.

Terminology for Strings

Concatenation:
Joining two strings together to form one string is called concatenation.

Substring:
A substring is a part of a larger string. For example, in the string “Hello World” one possible substring would be “Hello”, and another could be “llo Wo”.

Literal:
A literal string is a string that appears directly in the source code. Strings formed by “putting quotes around them” are usually literal strings in a language. To build a regular string you must usually use a string function. In languages with mutable strings it is usually not possible to mutate a literal string, even where it is possible it is almost never a good idea. More information on string literals can be found here.

Code

C
Strings in C are represented as a null-terminated array of characters. To create a string you create an array, fill it with characters and place a null character directly after the last character in the string. If you are using string literals to create a string you do not have to supply the size or the null character, the compiler will do that for you.

/* Create a string */
char s[6] = {'H', 'e', 'l', 'l', 'o', '\0'}
char sl[] = "Hello World"

/* Escape Character */
char e[50] = "Have you read the book \"1984\" by George Orwell?"

Python
Strings in Python are enclosed in quotes. You can use one of three different quoting styles to create a string. Single quotes are useful when you have double quote characters in the string. Triple quotes allow you to place newlines in the string.

# Create a string
a = "Hello World"
b = 'Have you read "1984" by George Orwell?'
c = """Materials:
Pen
Paper"""
# Escape Character
d = "Have you read the book \"1984\" by George Orwell?"

Scheme
Strings in Scheme are enclosed in quotes or created with the ‘string’ function. Newlines may be embedded in any string.

;; Create a string
(define a "Hello World")
(define b (string #H #e #l #l #o))

;; Escape Character:
(define c "Have you read the book \"1984\" by George Orwell?")

Performance

The performance characteristics of a string will depend on the data type that is used to represent them. Most often arrays are used, however linked lists are somewhat common as well. Strings may also be either mutable or immutable which can have an impact on their algorithmic performance.

Because of these factors an accurate overview of the performance of strings cannot be given. However in the vast majority of cases strings will share the same characteristics as arrays.

In the case of terminated strings implemented as an array the indexing time can still be O(1) however supplying an index greater than the length will access memory outside the string and, hopefully, cause an error. You may need to determine the length first to prevent this, which would be an O(n) operation.

Links

What the heck is: a String
A great article on strings, much more in-depth than this one. Incidentally where I got the inspiration for this series.

GOTO Table of Contents

This is the first in a series of articles describing programming concepts. This series assumes you have some basic experience in programming, but attempts to describe these concepts as simplistically as possible. This series will mainly use Python as the language of choice for examples. This is not because of any perceived superiority of Python over other languages, rather Python is basically executable psudo-code so we can express the concepts simply and give you a chance to play with them inside an interpreter.

Singly-Linked Lists

A linked list is one of the simplest data structures, at its heart it is simply a two element node (or cell). The first slot contains the data and the second slot contains a pointer to the next node. A simple illustration demonstrates this concept more concretely…

The graphic above represents the list (1, 2, 3). Each node in the list has two parts, data and pointer. The last node has a special symbol ‘Nil’ to tell both us and the computer that we have reached the end of the list. The python code for each node in the list is below…

class Node:

  def __init__(self, data=None, next=None):
    self.data = data
    self.next  = next

  def __repr__(self):
    return repr(self.data)

To make a list out of these nodes we would need to make a node for each element in the list (1, 2 and 3) and link them together.

a = Node(1, Node(2, Node(3, None)))
print a
#===> 1
print a.next
#===> 2
print a.next.next
#===> 3
print a.next.next.next
#===> None

Notice the nesting of the calls to Node above. Linked lists are an inherently nested data structure. To construct the list without having to nest the calls to Node we would have to work backwards. We have to do this because if we try to create the first node (1) there is not currently a second node to which we can point to.

c = Node(3, None)
b = Node(2, c)
a = Node(1, b)

The graphic below shows another way to look at Nodes that reflects their nested nature.

Programming languages like Lisp and Scheme make heavy use of linked lists. Consequently they have a rich library of functions for dealing with them. The code below shows how you create a linked list in Scheme using the ‘cons’ function. ‘cons’ is short for construct and the nodes are called ‘cons cells’.

(define a (cons 1 (cons 2 (cons 3 '()))))
a
;===> (1 2 3)

Now that we have created a list with cons (note its similarity to the Python example of Node) we can use the built-in procedures car and cdr to access elements of that list.

(car a)
;===> 1
(cdr a)
;===> (2 3)
(car (cdr a))
;===> 2
(cdr (cdr a))
;===> (3)
(car (cdr (cdr a)))
;===> 3
(cdr (cdr (cdr a)))
;===> ()

Most Scheme implementations usually define shorthand functions for performing these operations, like cadr and caddr (and many more). Many also provide more descriptive names such as first, rest, second, third…

(cadr a)
;===> 2
(caddr a)
;===> 3
(cddr a)
;===> (3)
(first a)
;===> 1
(rest a)
;===> (2 3)
(second a)
;===> 2
(third a)
;===> 3

Constructing linked lists and accessing their elements is usually most elegantly accomplished using recursion. The Scheme code below shows how we would recursively construct a list of integers, we have to do it backward to avoid a call to (reverse).

(define (make-list-of-integers start stop)
  (let loop ((s stop) (aList '()))
    (if (< s start)
        aList
        (loop (- s 1) (cons s aList)))))

(make-list-of-integers 0 5)
;===> (0 1 2 3 4 5)

Circular Linked Lists

A circular list is a linked-list where the last element’s pointer is set to the first element of the list. The graphic below shows the nodes of a circular list.

Using the Node class that we defined above we can create a circular list in Python as follows…

a = Node(1)
a.next = Node(2, Node(3, a))
a.next
#===> 2
a.next.next
#===> 3
a.next.next.next
#===> 1
a.next.next.next.next
#===> 2

The Scheme code for defining a circular list is below…

(define a (cons 1 (cons 2 (cons 3 '()))))
(set-cdr! (cddr a) (car a))
a
;===> (1 2 3 . 1)

Doubly-Linked Lists

Usually the term linked list refers to singly-linked lists as we have implemented them above. However there are other ways of implementing linked lists as well. The next most common form of linked list is the doubly-linked list. This is very similar to singly-linked lists except that each node in the list also contains a pointer to the previous node as shown in the graphic below.

Below is the Python code to implement doubly-linked lists.

class dNode:
  def __init__(self, prev=None, data=None, next=None):
    self.prev = prev
    self.data = data
    self.next = next

  def __repr__(self):
    return str(self.data)

a = dNode(data=1)
b = dNode(data=2)
c = dNode(data=3)
d = dNode(data=4)
a.next = b
b.prev, b.next = a, c
c.prev, c.next = b, d
d.prev = c

a
#===> 1
a.next
#===> 2
a.next.next
#===> 3
a.next.next.prev
#===> 2
a.next.next.next.prev
#===> 3

To make the doubly-linked list easier to use we would have to add some helper functions. As you can see it is fairly difficult to create a doubly-linked list, so we’ll define an append function here that will make creating one a little easier.

def dAppend(dLinkedList, nodeToAppend):
  if dLinkedList.next == None:
    dLinkedList.next = nodeToAppend
    nodeToAppend.prev = dLinkedList
  else:
    dAppend(dLinkedList.next, nodeToAppend)

a = dNode(data=1)
dAppend(a, dNode(data=2)
dAppend(a, dNode(data=3)
a
#===> 1
a.next
#===> 2
a.next.next
#===> 3
a.next.next.prev
#===> 2
a.next.prev
#===> 1

The simplest way to achieve this is through a recursive function as defined above. However Python does not perform tail-call optimization and will either run out of memory or reach its recursion limit (default is 1000) on large lists. Below is a non-recursive version of the same function.

def dNodeAppend(dLinkedList, nodeToAppend):
  node = dLinkedList
  while True:
    if node.next == None:
      node.next = nodeToAppend
      nodeToAppend.prev = node
      break
    else:
      node = node.next

a = dNode(data=1)
dNodeAppend(a, dNode(data=2))
dNodeAppend(a, dNode(data=3))
dNodeAppend(a, dNode(data=4))
a
#===> 1
a.next
#===> 2
a.next.next
#===> 3
a.next.next.next
#===> 4
a.next.next.next.prev
#===> 3
a.next.next.prev
#===> 2
a.next.prev
#===> 1

Performance

Insertion/Removal at head: O(1)
Insertion/Removal at n: O(n)
Indexing: O(n)
Search (unsorted): O(n)

Links

How to Think Like A Computer Scientist – Chapter 17 – Linked Lists
This chapter describes a way of implementing linked lists in Python. It also covers circular lists.
http://greenteapress.com/thinkpython/html/chap17.html

An Introduction to Scheme and its Implementation – Lists
http://www-pu.informatik.uni-tuebingen.de/users/sperber/pfg-2001/scheme/schintro-v14/schintro_93.html

Wikipedia Article on Linked Lists
http://en.wikipedia.org/wiki/Linked_list

GOTO Table of Contents

Arrays are one of the two fundamental data structures in computer science (the other is the tree).  An array is a contiguous (uninterrupted or sequential) block of memory that stores a set of elements.

Elements in an array can be accessed by their index.  An array’s index usually starts at 0 and stops at the number of elements in the array minus 1. The diagram below shows an array of the characters in the string “HELLO”.

In the array above the element at index 0 is the character “H”, the element at index 1 is the character “E” and so on.

Arrays have constant time access to the elements stored within. This means that it takes the same amount of time to access any element. For example, it would take the same amount of time to access the last element of an array as it would the first, or even an element in the middle.

A multi-dimensional array is simply an array of arrays. A spreadsheet is a form of multi-dimensional array. The diagram above shows a 3 element array where each element is also a 3 element array. Multi-dimensional arrays can be used for a variety of tasks from tracking pieces on a chess board to drawing graphics on the screen.

A “true” multi-dimensional array is implemented as a contiguous block of memory, just like the regular array. However it is also common practice to implement multi-dimensional arrays as trees.

Code

C

An array in C can contain elements of only one type/size and is not resizable.

/* Create an array */
int a[3] = {5, 10, 15};

/* Access an array element */
a[1] /*==> 10 */

Python

An array in Python is called a list. It can contain elements of any type/size and is resizable.

# Create an array
a = [5, 10, 15]

# Access an array element

a[1]  #==>  10

Scheme

An array in Scheme is called a vector. It can contain elements of any type/size and may be resizable.

;; Create an array

(define a (vector 5 10 15))
(define b #(5 10 15))

;; Access an array element

(vector-ref a 1)  ;;==>  10

What is the index?

An index is a way of labeling elements in an array. The first element is usually located at index 0, the second at index 1, the third at 2 and so on. The index is used to calculate the address of the element.

To understand this we have to remember that an array is just a reserved block of memory. Let’s look at a simple diagram of how the memory in your computer is organized.

Each block of memory can hold one value (usually equal to an integer in a language like C). To create an array we reserve a block of memory big enough to hold all our elements. If we had 4 elements it would look like the diagram below.

To access our array we only need to keep around the address of the beginning of our reserved space or 102 in this example. This address is equal to the index 0. But our index values still don’t point us at the right addresses yet. To do that we multiply the index by the size of an element then add that value to the start of our array. Let’s see how that maps out on our array.

Sometimes elements in an array are larger than just one block of memory. Here is what an array looks like when each element is two blocks of memory large.

An array can only contain elements of one size. This is because we need to multiply the index by the size of the elements in the array. How then do some languages let you put elements in an array that are different sizes? One simple way to do that is to use the size of the largest element in the array for indexing. This leads to some wasted space when storing smaller elements but allows the array to work just like before.

How do we know when we’re at the end of the array? One way is to store the length of the array with the address to the first element. In that case we just check if the index is <= the length. Another way to signal the end of an array is a sentinel node. A sentinel node is a value that can only occur at the end of an array and signals the end.

Performance

Indexing: O(1)
Insertion/Removal: O(n)
Search: O(n)

Arrays are either static or dynamic. A static array is set at a fixed size, elements can be inserted and removed but the array cannot grow or shrink. A dynamic array, on the other hand, can grow or shrink. Dynamic arrays are often initialized to be larger than the set of elements it contains. When enough elements are added to the array, such that the number of elements is greater than the array size, the array must be resized. Resizing an array could be a costly operation if the array is large or the act of copying is slow.

Links

GOTO Table of Contents