scilex::token_iterator Class Reference

Forward (single-pass) iterator yielding one token at a time. More...

#include <lexer.hpp>

Collaboration diagram for scilex::token_iterator:

Public Types
using	iterator_category = std::input_iterator_tag
	Single-pass.
using	value_type = token
	Yielded element.
using	difference_type = std::ptrdiff_t
	Required typedef.
using	pointer = const token *
	Pointer to current.
using	reference = const token &
	Reference to current.

Public Member Functions
	token_iterator ()=default
	Constructs the end sentinel.
	token_iterator (const lexer &owner, std::string_view source, eof_policy policy)
	Constructs a begin iterator over source for owner.
reference	operator* () const
	Returns the current token.
pointer	operator-> () const
	Member access to the current token.
token_iterator &	operator++ ()
	Advances to the next token.
void	operator++ (int)
	Post-increment (single-pass: no useful copy is returned).
bool	operator== (const token_iterator &other) const
	Equality: both exhausted, or both at the same offset.
bool	operator!= (const token_iterator &other) const
	Inequality.

Private Member Functions
void	advance ()
	Produces the next token, or marks the iterator exhausted.

Private Attributes
const lexer *	owner_ {nullptr}
	Rules provider (not owned).
std::string_view	source_
	Text being scanned.
position	cursor_ {0, 1, 1}
	Current scan position.
std::vector< frame >	stack_ {frame {.mode_id = 0, .entry_pos = position {0, 1, 1}}}
	Mode stack (top = active).
token	current_ {}
	The current token.
eof_policy	policy_ {eof_policy::omit}
	End-of-input policy.
bool	eof_done_ {false}
	End-of-input token already yielded.
bool	done_ {true}
	True once exhausted (end sentinel).

Detailed Description

Forward (single-pass) iterator yielding one token at a time.

A default-constructed iterator is the end sentinel. Each increment runs the lexer just far enough to produce the next non-skipped token; a lex_error thrown by the lexer propagates out of the increment.

Definition at line 970 of file lexer.hpp.

Member Typedef Documentation

difference_type

using scilex::token_iterator::difference_type = std::ptrdiff_t

Required typedef.

Definition at line 976 of file lexer.hpp.

iterator_category

using scilex::token_iterator::iterator_category = std::input_iterator_tag

Single-pass.

Definition at line 974 of file lexer.hpp.

pointer

using scilex::token_iterator::pointer = const token*

Pointer to current.

Definition at line 977 of file lexer.hpp.

reference

using scilex::token_iterator::reference = const token&

Reference to current.

Definition at line 978 of file lexer.hpp.

value_type

using scilex::token_iterator::value_type = token

Yielded element.

Definition at line 975 of file lexer.hpp.

Constructor & Destructor Documentation

token_iterator() [1/2]

scilex::token_iterator::token_iterator ( )

default

Constructs the end sentinel.

token_iterator() [2/2]

scilex::token_iterator::token_iterator ( const lexer &  owner, std::string_view  source, eof_policy  policy  )

inline

Constructs a begin iterator over source for owner.

Parameters

[in]	owner	The lexer providing the rules.
[in]	source	The text to scan.
[in]	policy	Whether to yield a terminal end_of_input token.

Definition at line 989 of file lexer.hpp.

Member Function Documentation

advance()

void scilex::token_iterator::advance ( )

inlineprivate

Produces the next token, or marks the iterator exhausted.

Definition at line 1057 of file lexer.hpp.

operator!=()

bool scilex::token_iterator::operator!= ( const token_iterator &  other ) const

inline

Inequality.

Parameters

[in]

other

Another iterator.

Returns

true if the two differ.

Definition at line 1040 of file lexer.hpp.

operator*()

reference scilex::token_iterator::operator* ( ) const

inline

Returns the current token.

Definition at line 1001 of file lexer.hpp.

operator++() [1/2]

token_iterator & scilex::token_iterator::operator++ ( )

inline

Advances to the next token.

Returns

*this.

Definition at line 1013 of file lexer.hpp.

operator++() [2/2]

void scilex::token_iterator::operator++ ( int  )

inline

Post-increment (single-pass: no useful copy is returned).

Definition at line 1020 of file lexer.hpp.

operator->()

pointer scilex::token_iterator::operator-> ( ) const

inline

Member access to the current token.

Definition at line 1007 of file lexer.hpp.

operator==()

bool scilex::token_iterator::operator== ( const token_iterator &  other ) const

inline

Equality: both exhausted, or both at the same offset.

Parameters

[in]

other

Another iterator.

Returns

true if the two denote the same position/end.

Definition at line 1030 of file lexer.hpp.

Member Data Documentation

current_

token scilex::token_iterator::current_ {}

private

The current token.

Definition at line 1051 of file lexer.hpp.

cursor_

position scilex::token_iterator::cursor_ {0, 1, 1}

private

Current scan position.

Definition at line 1049 of file lexer.hpp.

done_

bool scilex::token_iterator::done_ {true}

private

True once exhausted (end sentinel).

Definition at line 1054 of file lexer.hpp.

eof_done_

bool scilex::token_iterator::eof_done_ {false}

private

End-of-input token already yielded.

Definition at line 1053 of file lexer.hpp.

owner_

const lexer* scilex::token_iterator::owner_ {nullptr}

private

Rules provider (not owned).

Definition at line 1047 of file lexer.hpp.

policy_

eof_policy scilex::token_iterator::policy_ {eof_policy::omit}

private

End-of-input policy.

Definition at line 1052 of file lexer.hpp.

source_

std::string_view scilex::token_iterator::source_

private

Text being scanned.

Definition at line 1048 of file lexer.hpp.

stack_

std::vector<frame> scilex::token_iterator::stack_ {frame {.mode_id = 0, .entry_pos = position {0, 1, 1}}}

private

Mode stack (top = active).

Definition at line 1050 of file lexer.hpp.

---

The documentation for this class was generated from the following file:

• include/scilex/lexer.hpp