Reading 19: Parsers

A parser generator is a good tool that you should make part of your toolbox. A parser generator takes a grammar as input and automatically generates a parser, which takes a sequence of characters and tries to match the sequence against the grammar.

The parser typically produces a parse tree, which shows how grammar productions are expanded into a sentence that matches the character sequence. The root of the parse tree is the root nonterminal of the grammar. Each node of the parse tree expands into one production of the grammar. We’ll see how a parse tree actually looks later in this reading.

The final step of parsing is to do something useful with this parse tree. We are going to translate it into a value of a recursive data type. Recursive abstract data types are often used to represent expressions in a language, like HTML, Markdown, TypeScript, or algebraic expressions (the symbolic language of algebra). A recursive abstract data type that represents a language expression is called an abstract syntax tree (AST).

In 6.031, we are going to use ParserLib, a parser generator for TypeScript developed by the course staff. The parser generator is similar in spirit to more widely used parser generators like Antlr, but it has a simpler interface and is generally easier to use.

Here is what our HTML grammar looks like as a ParserLib grammar:

html ::= ( italic | normal ) * ;
italic ::= '<i>' html '</i>' ;
normal ::= text ; 
text ::= [^<>]+ ;  /* represents a string of one or more characters that are not < or > */

Let’s break it down.

Each ParserLib rule consists of a name, followed by a ::=, followed by its definition, terminated by a semicolon. The ParserLib grammar can also include TypeScript-style comments, both single line and multiline.

By convention, we use lowercase for nonterminals: html, normal, italic, text. The ParserLib library is case-insensitive with respect to nonterminal names; it canonicalizes names to all-lowercase, so even if you don’t write all your names into lowercase, you will see them as lowercase when you print your grammar.

Terminals are quoted strings, like '<i>', or more generally regular expressions, like [^<>]+. Unlike normal regular expressions, however, ParserLib syntax requires literal characters to be either quoted or enclosed in character-class brackets. Thus the regex (alpha|beta|[c-z])* is written as ('alpha'|'beta'|[c-z])* in ParserLib syntax.

html ::= ( italic | normal ) * ;

This rule shows that ParserLib rules can have the alternation operator |, repetition operators like * (and also + and ?, even though they’re not shown in this rule), and parentheses for grouping, in the same way we’ve been using in the grammars reading.

Whitespace in a grammar is not significant (outside of quoted strings or [...] character classes). Here the operators have been surrounded by spaces to make them more visible. Writing the rule as html::=(italic|normal)*; would have the same effect, just with less readability.

The html nonterminal also happens to be the root of this grammar (also called starting symbol). The root is the nonterminal that the whole input needs to match. It’s good practice to put the rule for the root nonterminal first in the grammar, so that a human reader can take a top-down view, but it isn’t essential. When we create a parser from the grammar, we tell ParserLib which nonterminal the parse should use as the root.

italic ::= '<i>' html '</i>' ;
normal ::= text ;
text ::= [^<>]+ ;

Note that the text production uses the inverted character class [^<>], discussed in the grammars reading, to represent any character except < and >.

Consider the grammar shown below.

expr ::= sum ;
sum ::= primary ('+' primary)* ;
primary ::= constant | '(' sum ')' ;
constant ::= [0-9]+ ;

This grammar will accept an expression like 42+2+5, but will reject a similar expression that has any spaces between the constants and the + signs. We could modify the grammar to allow whitespace around the plus sign by modifying the production rule for sum like this:

sum ::= primary (whitespace* '+' whitespace* primary)* ;
whitespace ::= [ \t\r\n]+ ;

However, this can become cumbersome very quickly once the grammar becomes more complicated. ParserLib allows a shorthand to indicate that certain kinds of characters should be skipped.

// the IntegerExpression grammar
@skip whitespace {
    expr ::= sum ;
    sum ::= primary ('+' primary)* ;
    primary ::= constant | '(' sum ')' ;
}
whitespace ::= [ \t\r\n]+ ;
constant ::= [0-9]+ ;

The @skip whitespace notation indicates that zero or more matches to the whitespace nonterminal should be automatically ignored, before and after each terminal, nonterminal, and character class on the righthand side of expr, sum, and primary. So from the point of view of grammar matching, these three rules effectively become:

expr ::= whitespace* sum whitespace* ;
sum ::= whitespace* primary whitespace* (whitespace* '+' whitespace* primary whitespace*)* whitespace* ;
primary ::= whitespace* constant whitespace* | whitespace* '('  whitespace* sum  whitespace* ')' whitespace* ;

Several things are important to note about @skip. First, there is nothing special about whitespace. The @skip directive works with any nonterminal defined in the grammar – so you could @skip punctuation or @skip spacesAndComments instead, by defining appropriate rules for those nonterminals.

Second, note how the definition of constant was intentionally placed outside the @skip block. This is because we want to accept expressions with extra space around a constant, like 42 + 2, but we want to reject spaces within constants, like 4 2 + 2. Putting the constant rule inside the @skip block would cause it to effectively become constant ::= (whitespace* [0-9] whitespace*)+, so it would accept constants with spaces inside them. Putting the rule for primary inside the @skip block – so that its use of constant can be surrounded by whitespace – but keeping the rule for constant outside, provides the desired behavior.

@skip spaces {
  semester ::= season year ;
  season ::= 'Fall' | 'Spring' ;
  year ::= [0-9] [0-9] ;
}
spaces ::= ' '+ ;

@skip spaces {
  semester ::= season year ;
}
season ::= 'Fall' | 'Spring' ;
year ::= [0-9] [0-9] ;
spaces ::= ' '+ ;

@skip space {
  semester ::= season year ;
}
season ::= 'Fall' | 'Spring' ;
year ::= [0-9] [0-9] ;
space ::= ' ' ;

The rest of this reading will use as a running example the IntegerExpression grammar defined just above. In order to embed the grammar in TypeScript with its newlines intact, we will use backquotes (also called a template literal):

const grammar: string = `
  @skip whitespace {
      expr ::= sum;
      sum ::= primary ('+' primary)*;
      primary ::= constant | '(' sum ')';
  }
  constant ::= [0-9]+;
  whitespace ::= [ \\t\\r\\n]+;  // <-- note that backslashes must be escaped
`;

The ParserLib parser generator tool converts a grammar string like this into a parser. In order to do this, you need to follow three steps. First, you need to import the ParserLib library:

import { Parser, ParseTree, compile } from "parserlib";

The second step is to define an enum type that contains all the nonterminals used by your grammar. This will tell ParserLib which definitions to expect in the grammar and will allow it to check for any missing ones.

enum IntegerGrammar {
  Expr, Sum, Primary, Constant, Whitespace
}

Note that ParserLib itself is case insensitive, but by convention, the names of enum values are capitalized.

From within your code, you can create a Parser by calling its compile factory function.

const parser: Parser<IntegerGrammar> = compile(grammar, IntegerGrammar, IntegerGrammar.Expr);

The compile method takes a grammar string, the nonterminal enumeration, and the nonterminal to use as the root nonterminal of the grammar. In this case, the root nonterminal we want is expr, so we pass IntegerGrammar.Expr.

Assuming you don’t have any syntax errors in your grammar, the result will be a Parser object that can be used to parse text. Notice that the Parser is a generic type that is parameterized by the IntegerGrammar enum that you defined earlier.

Now that you’ve generated the parser object, you are ready to parse your own text. The parser has a method called parse that takes in the text to be parsed as a string, and returns a ParseTree. Calling it produces a parse tree:

const parseTree: ParseTree<IntegerGrammar> = parser.parse("54+(2+ 89)");

Note that the ParseTree is also a generic type that is parameterized by the enum type IntegerGrammar.

For debugging, we can then print this tree out:

console.log(parseTree.toString());

You can also try calling the function visualizeAsUrl(tree) which will provide you with a URL which you can copy and paste to your browser to view the visualization.

So we’ve used the parser to turn a stream of characters into a parse tree, which shows how the grammar matches the stream. Next we want to do something useful with this parse tree: translating it into a value of a recursive abstract data type.

Like the Parser itself, the ParseTree is parameterized by the type NT, an enum type that lists all the nonterminals in the grammar, like the IntegerGrammar enumeration we defined earlier.

The first step is to learn how to traverse the parse tree. The ParseTree object has four methods that you need to be most familiar with. Three of them are fundamental observers:

interface ParseTree<NT> {

  /**
   * The nonterminal corresponding to this node in the parse tree.
   */
  name: NT;

  /**
   * The children of this node, in order, excluding @skipped subtrees
   */
  children: Array<ParseTree<NT>>;

  /**
   * The substring of the original string that this subtree matched
   */
  text: string;

Additionally, you can query the ParseTree for all children that match a particular production rule:

  /**
   * Get the children that correspond to a particular production rule 
   * @param name Name of the nonterminal corresponding to the desired production rule.
   * @returns children that represent matches of name's production rule.
   */
  public childrenByName(name: NT): Array<ParseTree<NT>>;
}

A good way to visit the nodes in a parse tree is to write a recursive function. For example, the recursive function below prints all nodes in the parse tree with proper indentation.

/**
 * Traverse a parse tree, indenting to make it easier to read.
 * @param node   parse tree to print.
 * @param indent indentation to use.
 */
function printNodes(node: ParseTree<IntegerGrammar>, indent: string): void {
    console.log(indent + node.name + ":" + node.text);
    for (const child of node.children){
        printNodes(child, indent + "  ");
    }
}

Running this function on the parse tree for 54+(2+ 89) produces the output below. For reference, the grammar and the visualized parse tree are shown at right.

Expr:54+(2+ 89)
  Sum:54+(2+ 89)
    Primary:54
      Constant:54
    Primary:(2+ 89)
      Sum:2+ 89
        Primary:2
          Constant:2
        Primary:89
          Constant:89

// the IntegerExpression grammar
@skip whitespace {
    expr ::= sum ;
    sum ::= primary ('+' primary)* ;
    primary ::= constant | '(' sum ')' ;
}
whitespace ::= [ \t\r\n]+ ;
constant ::= [0-9]+ ;

reading exercises

Parse trees

Which of the following statements are true of a ParserLib parse tree, from careful examination of the example output just above?

the root node of the tree corresponds to the root nonterminal of the grammar(missing answer)

(missing explanation)

each node of the tree is named by a nonterminal in the grammar(missing answer)

(missing explanation)

terminals do not have their own separate nodes that can be retrieved by children(missing answer)

(missing explanation)

only a grammar with recursive productions can generate a parse tree(missing answer)

(missing explanation)

a node’s immediate children must correspond to nonterminals mentioned in the node’s production (missing answer)

(missing explanation)

@skip N means that the skipped nonterminal N never appears among the children of nodes whose rules are inside the @skip block(missing answer)

(missing explanation)

check explain

Snapshot diagram of a ParseTree

Let’s see how a Parser­Lib parse tree looks as a snapshot diagram, because it’s closer to how we will think about it and work with it in TypeScript code.

The partial snapshot diagram at the right corresponds to part of the Integer­Grammar parse tree for "2+ 89", also shown.

The snapshot diagram shows the three correspondingly-colored nodes along the left branch of the parse tree: sum, primary, and constant.

Fill in the gray rectangles in the snapshot diagram.

What should appear at location A?

Constant,Primary,Sum,“2”,“+”,“89”(missing answer)

(missing explanation)

What should appear at location B?

Constant,Primary,Sum,“2”,“+”,“89”(missing answer)

(missing explanation)

What should appear at all the locations marked C?

IntegerGrammar,Array<ParseTree<IntegerGrammar>>,ParseTree<IntegerGrammar>(missing answer)

(missing explanation)

check explain

We need to convert the parse tree into a recursive data type. Here’s the definition of the recursive data type that we’re going to use to represent integer arithmetic expressions:

IntegerExpression = Constant(n:number)
                    + Plus(left:IntegerExpression, right:IntegerExpression)

If this syntax is mysterious, review recursive data type definitions.

When a recursive data type represents a language this way, it is often called an abstract syntax tree. An IntegerExpression value captures the important features of the expression – its grouping and the integers in it – while omitting unnecessary details of the sequence of characters that created it.

By contrast, the parse tree that we just generated with the IntegerExpression parser is a concrete syntax tree. It’s called concrete, rather than abstract, because it contains more details about how the expression is represented in actual characters. For example, the strings 2+2, ((2)+(2)), and 0002+0002 would each produce a different concrete syntax tree, but these trees would all correspond to the same abstract IntegerExpression value: Plus(Constant(2), Constant(2)).

Now, we can create a recursive function that walks the ParseTree to produce an IntegerExpression as follows:

/**
 * Convert a parse tree into an abstract syntax tree.
 * 
 * @param parseTree constructed according to the IntegerExpression grammar
 * @returns abstract syntax tree corresponding to parseTree
 */
function makeAbstractSyntaxTree(parseTree: ParseTree<IntegerGrammar>): IntegerExpression {
    if (parseTree.name === IntegerGrammar.Expr) {
        // expr ::= sum;
        const child: ParseTree<IntegerGrammar>|undefined = parseTree.children[0];
        assert(child, 'Expr node missing expected child');
        return makeAbstractSyntaxTree(child);

    } else if (parseTree.name === IntegerGrammar.Sum) {
        // sum ::= primary ('+' primary)*;
        const children: Array<ParseTree<IntegerGrammar>> = parseTree.childrenByName(IntegerGrammar.Primary);
        assert(children.length, 'Sum node missing expected children');
        const subexprs: Array<IntegerExpression> = children.map(makeAbstractSyntaxTree);
        const expression: IntegerExpression = subexprs.reduce((result, subexpr) => new Plus(result, subexpr));
        return expression;

    } else if (parseTree.name === IntegerGrammar.Primary) {
        // primary ::= constant | '(' sum ')';
        const child: ParseTree<IntegerGrammar>|undefined = parseTree.children[0];
        assert(child, 'Primary node missing expected child');
        // check which alternative (constant or sum) was actually matched
        switch (child.name) {
        case IntegerGrammar.Constant:
            return makeAbstractSyntaxTree(child);
        case IntegerGrammar.Sum:
            return makeAbstractSyntaxTree(child); // for this parser, we do the same thing either way
        default:
            assert.fail(`Primary node unexpected child ${IntegerGrammar[child.name]}`);
        }

    } else if (parseTree.name === IntegerGrammar.Constant) {
        // constant ::= [0-9]+;
        const n: number = parseInt(parseTree.text);
        return new Constant(n);

    } else {
        assert.fail(`cannot make AST for ${IntegerGrammar[parseTree.name]} node`);
    }
}

The function follows the structure of the grammar, handling each rule from the grammar in turn: expr, sum, primary, and constant. The only rule we don’t need to handle here is whitespace, because the grammar uses whitespace only in a @skip block, and skipped subtrees are not returned by children so they should never appear in the traversal.

Note that this code is tied very closely to the grammar. If you change the rules of the grammar in a significant way, this code will likely need to change too.

For sum nodes, the code uses map and reduce to avoid writing loops (and notice that parseTree.childrenByName is identical to performing a filter on parseTree.children).

To handle primary nodes, this code uses a switch statement. Any time you use switch, remember: the execution of a switch statement starts at the matching case and falls through to subsequent cases. Our code here is careful to return (or raise an error) in every case, never falling through.

Plus(Plus(Constant(1), Constant(2)), 
     Plus(Constant(3), Constant(4)))

@skip spaces { 
  date ::= monthname day ',' year | day '/' monthnum '/' year ;
}
monthname ::= 'January' | 'February' ;
monthnum ::= [0-9]+ ; 
day ::= [0-9]+ ;
year ::= [0-9]+ ;
spaces ::= ' '+ ;

@skip spaces { 
  list ::= int (',' list)? ;
}
int ::= '-'? [0-9]+ ;
spaces ::= ' '+ ;

enum ListGrammar { List, Int, Spaces }
function makeList(parseTree: ParseTree<ListGrammar>): Array<number> {
    switch (parseTree.name) {
    case ListGrammar.List:
        const list: Array<number> = [];
        const intChild: ParseTree<ListGrammar>|undefined = parseTree.children[0];
        const listChild: ParseTree<ListGrammar>|undefined = parseTree.children[1];
        assert(intChild);
        list.push(parseInt(▶▶A◀◀));
        if (▶▶B◀◀) {
            list = list.concat(▶▶C◀◀);
        }
        return list;
    default:
        throw new Error("should never get here");
    }
}

@skip spaces {
  semester ::= season year ;
  season ::= 'Fall' | 'Spring' ;
  year ::= [0-9] [0-9] ;
}
spaces ::= ' '+ ;

// nonterminals in the grammar
enum SemesterGrammar { Semester, Season, Year, Spaces }

// abstract data type representing a season of the year
enum Season { Fall, Spring, Winter, Summer }

/**
 * @param node must be a match to the season rule
 * @returns corresponding Season value
 */
function convertToSeason(node: ParseTree<SemesterGrammar>): Season {
  assert.equal(node.name, SemesterGrammar.Season);
  return ▶▶A◀◀ ? Season.Fall : Season.Spring;
}

@skip spaces {
  semester ::= season year ;
}
season ::= 'Fall' | 'Spring' ;
year ::= [0-9] [0-9] ;
spaces ::= ' '+ ;

  return ▶▶A◀◀ ? Season.Fall : Season.Spring;

Several things can go wrong when parsing.

Your grammar may have a syntax error in it. In this case, compile will throw a ParseError.

The string you are trying to parse may not be parseable with your given grammar. This might happen because your grammar is incorrect, or because your string is incorrect. Either way, the problem will be signaled by the parse method throwing an ParseError.

The ParseError error contains some information about the possible location of the error, although parse errors are sometimes inherently difficult to localize, since the parser cannot know what string you intended to write, so you may need to search a little to find the true location of the error.

ParserLib works by generating a top-down Recursive Descent Parser. These kind of parsers have a few limitations in terms of the grammars that they can parse. There are two in particular that are worth pointing out.

Left recursion. A recursive descent parser can go into an infinite loop if the grammar involves left recursion. This is a case where a definition for a nonterminal involves that nonterminal as its leftmost symbol. For example, the grammar below includes left recursion because one of the possible definitions of sum is sum '+' number which has sum as its leftmost symbol.

sum ::= number | sum '+' number ;
number ::= [0-9]+ ;

This left-recursive definition is problematic because the recursive descent parser needs to try matching all alternatives for sum, both number and sum '+' number. But trying to match sum '+' number immediately requires matching sum as its first step. The parser keeps trying the rule recursively but never makes any progress through the string being parsed – not reducing to a smaller subproblem as correct recursion requires to terminate. By contrast, a rule like expr = number | '(' expr ')', which is recursive but not left-recursive, is able to make progress through the string by matching ( before recursively applying the expr rule again again.

Left recursion can also happen indirectly. For example, changing the grammar above to the one below does not address the problem because the definition of sum still indirectly involves a symbol that has sum as its first symbol.

sum ::= number | thing number ;
thing ::= sum '+' ;
number ::= [0-9]+ ;

If you give any of these grammars to ParserLib and then try to use them to parse a symbol, ParserLib will fail with a ParseError listing the offending nonterminal.

There are some general techniques to eliminate left recursion; for our purposes, the simplest approach will be to replace left recursion with repetition (*), so the grammar above becomes:

sum ::= (number '+')* number ;
number ::= [0-9]+ ;

Greediness. This is an issue that you may not run into in this class, but it is a limitation of ParserLib you should be aware of. The ParserLib parsers are greedy in that at every point they try to match a maximal string for any rule they are currently considering. For example, consider the following grammar:

g ::= ab threeb ;
ab ::= 'a'*'b'* ;
threeb ::= 'bbb' ;

The string 'aaaabbb' clearly should match g, but a greedy parser cannot parse it because it will try to parse a maximal substring that matches the ab symbol, and then it will find that it cannot parse threeb because it has already consumed the entire string. Unlike left recursion, which is easy to fix, this is a more fundamental limitation of the type of parser implemented by ParserLib.

The topics of today’s reading connect to our three properties of good software as follows:

• Safe from bugs. A grammar is a declarative specification for strings and streams, which can be implemented automatically by a parser generator. These specifications are often simpler, more direct, and less likely to be buggy than parsing code written by hand.

• Easy to understand. A grammar captures the shape of a sequence in a form that is compact and easier to understand than hand-written parsing code.

• Ready for change. A grammar can be easily edited, then run through a parser generator to regenerate the parsing code.