token.hpp File Reference

The token produced by the lexer and its source position. More...

#include <cstddef>
#include <limits>
#include <string_view>

Include dependency graph for token.hpp:

Go to the source code of this file.

Classes
struct	scilex::position
	A location in the source text. More...
struct	scilex::token
	One lexical token: a typed slice of the source. More...

Namespaces
namespace	scilex
	The SciLex public API (scilex::lexer, scilex::rule, scilex::token).

Variables
constexpr int	scilex::end_of_input {std::numeric_limits<int>::min()}
	Reserved token kind for the synthetic end-of-input token.
constexpr int	scilex::error {std::numeric_limits<int>::min() + 4}
	Reserved token kind for a lexical-error run under scilex::error_policy::token.

Detailed Description

The token produced by the lexer and its source position.

A token is a non-owning view into the source text (a scilex::token borrows its lexeme), keeping tokenization allocation-light: the only allocation is the token vector itself.

Definition in file token.hpp.