2. Lexical Structure

2.1 Input Format

Culsma source text:

1. MUST be parsed as UTF-8.
2. MAY use LF or CRLF line endings.
3. MUST be tokenized left-to-right under these lexical rules.

Parsing errors in source format or token stream MUST surface as syntax errors.

2.2 Keywords

Keyword set:

1. protocol
2. include
3. let
4. repeat
5. if
6. else
7. break
8. continue
9. and
10. or
11. true
12. false

Parser-surface control keywords additionally include:

13. with
14. env
15. constraint

Surface notes:

1. include is reserved for source-level file loading and for the protocol-include statement surface.
2. Qualified protocol reference uses Module.Protocol() without introducing a new keyword.
3. Legacy on syntax is not part of the parser surface.

Rules:

1. Keywords MUST NOT be used as IDENTIFIER.
2. true and false are lexed as BOOLEAN, not identifier names.
3. Step-call names such as sep, frac, img, ecp, and phy are not keywords; they are identifiers validated in semantic stages.

2.3 Identifiers

An IDENTIFIER token matches:

[A-Za-z][A-Za-z0-9_]*

Constraints:

1. The first character MUST be an ASCII letter.
2. Remaining characters MAY include ASCII letters, digits, and underscore.
3. A token that matches a keyword in Chapter 2.2 MUST be tokenized as keyword/boolean, not as IDENTIFIER.

2.4 Comments

Culsma supports:

1. Line comments: // ... to end-of-line
2. Block comments: /* ... */

Rules:

1. Comments are ignored by parsing and have no runtime semantics.
2. Comment text inside string literals is not treated as comment syntax.
3. Block comments are not nestable and terminate at the first matching */.

2.5 Whitespace

Whitespace characters include (at minimum):

1. Space (U+0020)
2. Horizontal tab (U+0009)
3. Line feed (U+000A)
4. Carriage return (U+000D)

Rules:

1. Whitespace matching the grammar ignore rule %ignore /\\s+/ is ignored outside string literals, except as token separator.
2. Implementations MUST treat equivalent whitespace forms identically for lexical behavior.

2.6 Tokens

2.6.1 String Literal

A STRING literal:

1. MUST use double quotes (").
2. MAY include backslash escapes.
3. MUST NOT cross line boundaries unless represented by escapes.

Executable lexical behavior:

1. The lexer accepts any \\. escape sequence.
2. The transformer only interprets \\\" and \\\\; other escapes are preserved as raw text.

2.6.2 Boolean Literal

Boolean literals are:

1. true
2. false

Both are case-sensitive.

2.6.3 Quantity Literal

A QUANTITY literal is a decimal/integer numeric token with an optional unit suffix.

Examples:

1. 10
2. 12.5
3. 5mL
4. 95C

Accepted canonical unit suffixes:

1. ng_per_uL, ug_per_mL
2. ms, s, min, h, day
3. rpm, rcf, xg, Hz
4. uL, mL, L
5. mM, uM, nM, M
6. mg, ug, kg, g
7. nm, um
8. %
9. mW, W, mV, V
10. C, K
11. X

Unit compatibility checks are semantic/typecheck responsibilities.

Compatibility note:

1. sec is accepted as a legacy alias for s.
2. hr is accepted as a legacy alias for h.
3. ul is accepted as a legacy alias for uL.
4. ml is accepted as a legacy alias for mL.
5. pct is accepted as a legacy alias for %.
6. Legacy aliases remain valid source text but are not canonical spellings.

Rules:

1. A leading - is not part of the QUANTITY token; it is parsed as unary negation in expressions.
2. Unit suffixes MUST not be immediately followed by [A-Za-z0-9_] (prevents partial matches).
   • Example: 200Cool tokenizes as QUANTITY(200) then IDENTIFIER(Cool), not QUANTITY(200C).

2.6.4 Operators and Delimiters

Recognized operator and delimiter tokens include:

1. Comparison: ==, !=, <, <=, >, >=
2. Arithmetic: +, -, *, /
3. Assignment: =
4. Mutation operator: <<
5. Grouping and punctuation: (, ), {, }, [, ], ,, ;
6. Qualified protocol separator: .

Parser token notes:

1. Plate region selectors such as plate96[A1:B2, D1] are tokenized by the executable grammar as a dedicated selector token after the base identifier.
2. Source-level behavior treats that shape as parser surface, even though the selector internals are later decoded into AST regions by the transformer.

2.6.5 Tokenization Priority

When multiple token patterns could match at a position:

1. The lexer MUST prefer the longest valid token.
2. true and false MUST be tokenized as BOOLEAN.
3. Multi-character operators MUST be tokenized before their single-character prefixes.
4. Keyword literals (such as "protocol") take precedence over the generic IDENTIFIER token.

2.6.6 Ignored Tokens

The following are ignored by the grammar (outside string literals):

1. Whitespace: %ignore /\\s+/
2. Line comments: %ignore /\\/\\/[^\n]*/
3. Block comments: %ignore /\\/\\*[\\s\\S]*?\\*\\//s

2.7 Conformance Boundary

Parser implementations and parser tests must conform to these lexical rules; they do not define separate lexical behavior.

If conflicts are found, maintainers MUST reconcile the reference and executable behavior before treating either side as settled.