SciLex
A header-only C++20 lexer built on REAL
Loading...
Searching...
No Matches
token.hpp
Go to the documentation of this file.
1
9#ifndef SCILEX_TOKEN_HPP
10#define SCILEX_TOKEN_HPP
11
12#include <cstddef>
13#include <limits>
14#include <string_view>
15
16namespace scilex {
17
26 inline constexpr int end_of_input {std::numeric_limits<int>::min()};
27
37 inline constexpr int error {std::numeric_limits<int>::min() + 4};
38
47 struct position
48 {
49 std::size_t offset;
50 std::size_t line;
51 std::size_t column;
52 };
53
57 struct token
58 {
59 int kind;
60 std::string_view lexeme;
62 std::size_t mode_id {0};
63 };
64} // namespace scilex
65
66#endif // SCILEX_TOKEN_HPP
The SciLex public API (scilex::lexer, scilex::rule, scilex::token).
Definition layout.hpp:47
constexpr int error
Reserved token kind for a lexical-error run under scilex::error_policy::token.
Definition token.hpp:37
constexpr int end_of_input
Reserved token kind for the synthetic end-of-input token.
Definition token.hpp:26
A location in the source text.
Definition token.hpp:48
std::size_t offset
0-based byte offset from the start of the source.
Definition token.hpp:49
std::size_t column
1-based byte column within the line.
Definition token.hpp:51
std::size_t line
1-based line number.
Definition token.hpp:50
One lexical token: a typed slice of the source.
Definition token.hpp:58
int kind
Caller-defined token kind (e.g. an enum value).
Definition token.hpp:59
std::size_t mode_id
The mode this token was lexed in (0 = the default/root mode).
Definition token.hpp:62
std::string_view lexeme
The matched text, viewing into the source.
Definition token.hpp:60
position start
Position of the token's first byte.
Definition token.hpp:61