The SciLex public API (scilex::lexer, scilex::rule, scilex::token). More...

Namespaces
namespace	detail

Classes
struct	frame
	One entry on the per-scan mode stack: the active mode and where it was entered (the entry position feeds the unterminated/diagnostic messages). More...

struct	grammar
	A parsed grammar: the rules in order, and each rule's name (index = the rule's kind). More...

class	grammar_error
	Thrown for a malformed `.lex` grammar: where (origin, 1-based line, and 1-based byte column in the line when the cause has one) and why. More...

class	layout_error
	Thrown when a line's indentation matches no enclosing level. More...

class	lex_error
	Thrown when no rule matches at a position (a lexical error). More...

class	lexer
	A lexer built from an ordered list of rules. More...

struct	mode_action
	A mode transition, fired when its rule wins, acting on the scan's mode stack: enter a nested mode, leave the current one, or replace it. More...

struct	position
	A location in the source text. More...

struct	rule
	A token rule: a kind, the pattern that recognizes it, whether matches are discarded (whitespace, comments), and — for contextual lexing — the modes it is active in and an optional mode transition it fires when it wins. More...

struct	token
	One lexical token: a typed slice of the source. More...

class	token_iterator
	Forward (single-pass) iterator yielding one token at a time. More...

class	token_range
	A lazy range of tokens, returned by lexer::scan. More...

class	token_stream
	Text fed in pieces, lexed as it arrives, with exactly the tokens lexer::tokenize gives the whole text: the same munches, modes, positions and errors. More...

Enumerations
enum class	tab_policy : std::uint8_t { columns , python }
	How the overload of `layout()` that takes the source measures indentation. More...

enum class	eof_policy { omit , append }
	Whether tokenization appends a synthetic end-of-input token. More...

enum class	dfa_policy : std::uint8_t { automatic , requested }
	Which modes a lexer tries to accelerate with a `real::dfa`. More...

enum class	error_policy { raise , token }
	What a lexer does when it reaches a byte that no rule in the active mode can begin. More...

enum class	column_unit { bytes , codepoints , utf16 }
	The unit a token's `position::column` is counted in. More...

Functions
grammar	parse_grammar (std::string_view text, const std::string &origin="<string>")
	Parses a `.lex` grammar from `text` (see the file documentation for the format).

grammar	load_grammar (const std::string &path)
	Reads and parses the `.lex` grammar at `path`.

std::vector< token >	layout (std::span< const token > tokens, const std::vector< bool > &mode_significant={})
	Rewrites `tokens` with NEWLINE / INDENT / DEDENT inserted.

std::vector< token >	layout (std::span< const token > tokens, std::string_view source, tab_policy tabs, const std::vector< bool > &mode_significant={})
	Rewrites `tokens` with NEWLINE / INDENT / DEDENT inserted, measuring indentation in `source` under `tabs`.

void	apply_transition (const rule &r, position start, std::vector< frame > &stack)
	Applies rule `r's` mode transition (if any) to `stack` — the per-scan mode-stack mutation, kept pure so the lexer and the fuzz oracle share it verbatim.

Variables
constexpr int	newline {std::numeric_limits<int>::min() + 1}
	Reserved kind: end of a logical line.

constexpr int	indent {std::numeric_limits<int>::min() + 2}
	Reserved kind: indentation increased (start of a deeper block).

constexpr int	dedent {std::numeric_limits<int>::min() + 3}
	Reserved kind: indentation decreased (end of a block).

constexpr std::size_t	max_mode_depth {65536}
	The deepest the mode stack may grow. A push beyond it is a lex_error under either error policy: each frame costs memory, and without a bound an input of openers alone (16 MiB of `(` under the python grammar) grew the stack past a gigabyte. 65 536 frames is ~2 MiB, far beyond any nesting a real grammar reaches.

constexpr int	end_of_input {std::numeric_limits<int>::min()}
	Reserved token kind for the synthetic end-of-input token.

constexpr int	error {std::numeric_limits<int>::min() + 4}
	Reserved token kind for a lexical-error run under scilex::error_policy::token.

Detailed Description

The SciLex public API (scilex::lexer, scilex::rule, scilex::token).

Enumeration Type Documentation

◆ column_unit

enum class scilex::column_unit

strong

The unit a token's position::column is counted in.

The default bytes is the historical behaviour, bit-for-bit. codepoints counts Unicode scalar values (each valid UTF-8 codepoint is one column), and utf16 counts UTF-16 code units (a BMP codepoint is 1, an astral codepoint 2) — the unit an LSP client expects. A malformed byte (an orphan continuation, an overlong or out-of-range sequence) counts as one unit in every mode, so the column stays defined on the error runs error_policy::token emits. The chosen unit is not carried on position (one field, not self-describing) — the lexer declares it via lexer::columns, a named trade-off rather than a silent default.

Enumerator
bytes	One column per byte (the default; column == byte offset within the line + 1).
codepoints	One column per Unicode scalar value (a valid UTF-8 codepoint).
utf16	One column per UTF-16 code unit (BMP = 1, astral = 2) — the LSP unit.

◆ dfa_policy

enum class scilex::dfa_policy : std::uint8_t

strong

Which modes a lexer tries to accelerate with a real::dfa.

The token stream is the same either way: a mode keeps its DFA only where the lexer has decided that the DFA reproduces the per-rule munch on every input (see lexer::dfa_modes_active).

Enumerator
automatic	Every mode (the default). In each, the rules a DFA reproduces go on one and the others stay on the per-rule path; the cost is the DFA construction.
requested	Only the modes named in `dfa_modes`; an empty list keeps every mode on the per-rule path.

◆ eof_policy

enum class scilex::eof_policy

strong

Whether tokenization appends a synthetic end-of-input token.

eof_policy::append yields one final token of kind end_of_input at the end position once the input is exhausted — the parser-friendly mode, so a cursor always has a current token to match against.

Enumerator
omit	Stop at the last real token (default).
append	Append one end_of_input token at the end position.

◆ error_policy

enum class scilex::error_policy

strong

What a lexer does when it reaches a byte that no rule in the active mode can begin.

The default preserves the historical behaviour exactly; token is opt-in recovery.

Enumerator
raise	Throw a lex_error at the first unmatched byte (the default).
token	Recover: emit the maximal unmatched byte run as one scilex::error token and resume. The cost of an error run is the grammar's no-match cost: a first-byte pre-filter skips positions no rule can begin (usually O(1) per byte), so an unanchored, greedy rule that scans far before failing is what makes recovery expensive on a long run — prefer a definite leading byte.

◆ tab_policy

enum class scilex::tab_policy : std::uint8_t

strong

How the overload of layout() that takes the source measures indentation.

Enumerator

columns

A tab is one column, like a space; mixed tabs and spaces are not policed (the default pass).

python

CPython's rule. A line's indentation is measured twice over the blanks before its first token: with tabs advancing to the next multiple of 8, and with every tab counted as 1. Levels compare by the first measure; when the second does not order the same way — a deeper, equal or shallower line by one measure and not by the other — the line is refused with "inconsistent use of tabs and spaces in indentation", the message CPython's TabError carries. A form feed resets both measures, as in CPython.

Function Documentation

◆ apply_transition()

void scilex::apply_transition	(	const rule &	r,
		position	start,
		std::vector< frame > &	stack
	)

inline

Applies rule r's mode transition (if any) to stack — the per-scan mode-stack mutation, kept pure so the lexer and the fuzz oracle share it verbatim.

Depends only on r's action (its pre-resolved mode_action::target_id), the token start start, and the stack; it mutates only stack. push enters the target (remembering start), pop leaves the current mode, set replaces it in place. The target id is resolved once at build time, so this hot per-token pivot does no name→id map lookup.

Exceptions

lex_error On a pop while the stack is at its root (nothing to leave), or a push that would take it past max_mode_depth.

◆ layout() [1/2]

std::vector< token > scilex::layout	(	std::span< const token >	tokens,
		const std::vector< bool > &	mode_significant = `{}`
	)

inline

Rewrites tokens with NEWLINE / INDENT / DEDENT inserted.

Parameters

[in]	tokens	An end-of-input-terminated token sequence.
[in]	mode_significant	Per-mode-id significance policy (Layout Awareness Level A): index by a token's `mode_id`; `true` (or a mode-id beyond the vector) means the token shapes layout, `false` means it is passed through without affecting indentation. An empty vector (the default) means every token is significant — byte-for-byte the positional pass. (A `std::vector<bool>` rather than a `std::span<const bool>`: the bit-packed `vector<bool>` cannot be viewed as a contiguous span of `bool`.)

Returns: The layout-aware token sequence (still end-of-input-terminated).

Exceptions

layout_error If a line dedents to an indentation that no open block used.

◆ layout() [2/2]

std::vector< token > scilex::layout	(	std::span< const token >	tokens,
		std::string_view	source,
		tab_policy	tabs,
		const std::vector< bool > &	mode_significant = `{}`
	)

inline

Rewrites tokens with NEWLINE / INDENT / DEDENT inserted, measuring indentation in source under tabs.

Parameters

[in]	tokens	An end-of-input-terminated token sequence lexed from `source`.
[in]	source	The text `tokens` were lexed from; a token's `start.offset` indexes it.
[in]	tabs	How indentation is measured (tab_policy).
[in]	mode_significant	As in the overload without a source.

Returns: The layout-aware token sequence (still end-of-input-terminated).

Exceptions

layout_error	If a line dedents to an indentation that no open block used, or (under `tab_policy::python`) mixes tabs and spaces so that its level is ambiguous.
std::invalid_argument	If a token's offset lies beyond `source`.

◆ load_grammar()

grammar scilex::load_grammar ( const std::string & path )

inline

Reads and parses the .lex grammar at path.

Parameters

[in] path The grammar file.

Returns: The rules and their names.

Exceptions

grammar_error If the file cannot be read (line 0) or is malformed (see parse_grammar).

◆ parse_grammar()

grammar scilex::parse_grammar	(	std::string_view	text,
		const std::string &	origin = `"<string>"`
	)

inline

Parses a .lex grammar from text (see the file documentation for the format).

Never returns a half-built grammar: the first malformed line throws.

Parameters

[in]	text	The grammar.
[in]	origin	What errors name as the grammar's origin (a path, or a label).

Returns: The rules and their names.

Exceptions

grammar_error On a malformed line, an invalid pattern (at its column in the line), or a grammar with no rules.

Variable Documentation

◆ dedent

constexpr int scilex::dedent {std::numeric_limits<int>::min() + 3}

inlineconstexpr

Reserved kind: indentation decreased (end of a block).

◆ end_of_input

constexpr int scilex::end_of_input {std::numeric_limits<int>::min()}

inlineconstexpr

Reserved token kind for the synthetic end-of-input token.

Emitted at the end of the input when tokenizing with scilex::eof_policy::append (the parser-friendly mode: there is always a current token, including a terminal one to match). SciLex reserves this value; user-defined token kinds must not use it.

◆ error

constexpr int scilex::error {std::numeric_limits<int>::min() + 4}

inlineconstexpr

Reserved token kind for a lexical-error run under scilex::error_policy::token.

When error recovery is enabled, a maximal run of bytes that no rule in the active mode can begin is emitted as one token of this kind (its token::lexeme is the exact offending bytes), instead of throwing. Part of the reserved family: end_of_input is min(), the layout kinds (scilex::newline / indent / dedent in layout.hpp) are min()+1..+3, so this takes the next free slot, min()+4. User-defined token kinds must not use it.

◆ indent

constexpr int scilex::indent {std::numeric_limits<int>::min() + 2}

inlineconstexpr

Reserved kind: indentation increased (start of a deeper block).

◆ max_mode_depth

constexpr std::size_t scilex::max_mode_depth {65536}

inlineconstexpr

The deepest the mode stack may grow. A push beyond it is a lex_error under either error policy: each frame costs memory, and without a bound an input of openers alone (16 MiB of ( under the python grammar) grew the stack past a gigabyte. 65 536 frames is ~2 MiB, far beyond any nesting a real grammar reaches.

◆ newline

constexpr int scilex::newline {std::numeric_limits<int>::min() + 1}

inlineconstexpr

Reserved kind: end of a logical line.

Namespaces

Classes

Enumerations

Functions

Variables

Detailed Description

Enumeration Type Documentation

◆ column_unit

◆ dfa_policy

◆ eof_policy

◆ error_policy

◆ tab_policy

Function Documentation

◆ apply_transition()

◆ layout() [1/2]

◆ layout() [2/2]

◆ load_grammar()

◆ parse_grammar()

Variable Documentation

◆ dedent

◆ end_of_input

◆ error

◆ indent

◆ max_mode_depth

◆ newline