Now I know that the parser is LL(1) and the lexer also only
needs one-character lookahead, which allows me to dramatically
simplify the interface for IO input, and improve the interface
to the lexer. Even if I did want unbounded peek, I'd want
the interface to be `peek(off)`, not that awful buffer.
I intend to use the new lexer interface to make the parser
states more stateful, and potentially read multiple tokens
in a row. Then, states would only be needed for recursive
structures, without the awkward intermediate states like
ST_LABEL which exists only to let me burn a token.
I also removed the nonsense related to base 64 parsing,
because it was unclear how to handle it, and the advantages
of having it weren't clear. I kept up to base 36, but honestly
I might want to consider getting rid of everything but decimal,
hex, and binary anyway. I'm not sure if I'd want to keep using
the current syntax for the radix either.
Also no top-level declarations or operator precedence.
The syntax is LL(1). LL syntax seems necessary because
our codegen requires emitting certain code (e.g. entering control)
prior to any codegen inside that context, whereas something like
LR would presumably parse the inner expression before recognizing
the control structure. There may be some way to work around this;
I don't know, I'm not a parsing expert.
Certain parts of the syntax are wonky, e.g. juxtaposition as
function application means a missing semicolon can give confusing
results. I suspect indentation-sensitive syntax would work
more nicely, and intend to implement it some time in the future.
James T. Martin