Introducing SteelSeries GoLisp

One of the capabilities we needed in order to achieve our architectural goals was the ability to load code into our Go app at runtime. Go couldn’t do this. The logical solution was to use an embedded scripting language of some sort. Given my love of Lisp and it’s relative simplicity, I decided that what we needed was an embedded Lisp runtime.

What is GoLisp?

GoLisp is a simple Lisp language and runtime implemented in Google’s Go programming language. It’s main purpose is integration into a Go application to provide runtime extension and scripting. The core of a basic Lisp is provided, but with limited special forms and primitive functions. More of these will be added as required without having to modify the GoLisp core. Also, a REPL is provided and GoLisp can be used as-is to enter and evaluate code.

Work on GoLisp was started late April 2013 and we’ve been using it in production since September 2013 when Engine3 debuted at PAX Prime with the Siberia Elite.

GoLisp is a lexically scoped Lisp-1:

Lexically scoped:

The scope of definitions are determined by where they are made in the lexical structure of the code. Each function, lambda, let, and do creates a new lexical scope. From [2]:

Here references to the established entity can occur only within certain program portions that are lexically (that is, textually) contained within the establishing construct. Typically the construct will have a part designated the body, and the scope of all entities established will be (or include) the body.

Lisp-1:

A Lisp where functions and variables share a single namespace. This differs from a Lisp-2 in which functions and variables have separate namespaces.

A bit more about it

The language and runtime of GoLisp are largely inspired by Scheme [1].

One decision I made was to implement the core builtin functions in GO rather than in Lisp itself (which is common and valid) for performance reasons.

The GoLisp REPL supports readline and so provides history and command editing. History is local to the directory from which you execute GoLisp.

GoLisp supports all the major data types from Scheme, as well as objects and bytearrays. The types are described here:

Cons Cells are the central data type in classic Lisp used both as dotted pairs (a . b) and general lists (a b). For an overview of cons cells and how to use them see [1], [3], or http://cs.gmu.edu/~sean/lisp/cons/.

Symbols are simple identifiers, e.g. function-name. Symbols follow the follow 2 simple rules:

• starts with a letter, and

• contains alphanumeric, -, _, ?, !, >, and *.

Typically, - is used to separate words in a symbol, _ is used in special symbols (such as system use) to separate words and as a prefix and suffix. The characters ?, !, and * are typically used as the final character of a function name to denote:

? a predicate, e.g. nil?

! a mutating function (changes the argument rather than returning a modified copy), e.g. set-car!

* a variant of the primary function, e.g. flatten (which does a one level flattening of a list) and flatten* (which is a recursive flatten)

Some builtin function names violate these rules (e.g. arithmetic and relative functions). You can’t create symbols like this without adding support for them to the tokenizer and parser.

Strings are any sequence of characters other than " enclosed by a pair of ", e.g. "string". If you need to have " in a string, use \".

Integers are sixtyfour bit signed integers. Both decimal and hexadecimal formats are supported. E.g. 26, 0x1a, 0x1A.

Floats are Go float32 numbers. Accordingly they are signed. All arithmetic functions with the exception of modulus work as expected for both integers and floats. Numbers are coerced to floats as required, specifically if any arguments are float, all are converted to float and the result will be a float.

Booleans represent true and false. nil, (), and false are all logically false, everything else is logically true. Boolean literals are #t and #f for true and false, respectively.

Functions are user defined procedures created by lambda or define.

Objects allow you to encapsulate a Go object (struct) in a Lisp data object. There is no way to do this from Lisp itself, but is useful when writing primitive functions (see below).

Bytearrays are simply objects that encapsulate []byte objects. The difference is that there is syntactic support for them. Use square braces surrounding a list of numbers between 0 and 255, inclusive. For example: [1 2 3 4 5]. That format will parse to an Object containing a the Go bytearray (i.e. []byte). Bytearrays evaluate to themselves. There are also functions for doing Bytearray manipulation.

Primitives are just as they are in Lisp or Smalltalk: functions written in Go and exposed as functions in Lisp. The combination of primitives and objects allow you to integrate with the underlying Go program.

For full documentation see the language reference.

We will be opensourcing GoLisp very soon. Watch for the announcement here.

References

[1] Harold Abelson, Gerald Jay Sussman, and Julie Sussman. Structure and Interpretation of Computer Programs. MIT Press, Cambridge, Mass., 1985.

[2] Richard P. Gabriel and Kent M. Pitman. Endpaper: Technical issues of separation in function cells and value cells. Lisp and Symbolic Computation, 1(1):81–101, June 1988.

[3] Guy L. Steele. COMMON LISP: the language. Digital Press, 12 Crosby Drive, Bedford, MA 01730, USA, 1984. With contributions by Scott E. Fahlman and Richard P. Gabriel and David A. Moon and Daniel L. Weinreb.