scilex::token_iterator Class Reference

Forward (single-pass) iterator yielding one token at a time. More...

#include <lexer.hpp>

Public Types
using	iterator_category = std::input_iterator_tag
	Single-pass.
using	value_type = token
	Yielded element.
using	difference_type = std::ptrdiff_t
	Required typedef.
using	pointer = const token *
	Pointer to current.
using	reference = const token &
	Reference to current.

Public Member Functions
	token_iterator ()=default
	Constructs the end sentinel.
	token_iterator (const lexer &owner, std::string_view source, eof_policy policy)
	Constructs a begin iterator over source for owner.
reference	operator* () const
	Returns the current token.
pointer	operator-> () const
	Member access to the current token.
token_iterator &	operator++ ()
	Advances to the next token.
void	operator++ (int)
	Post-increment (single-pass: no useful copy is returned).
bool	operator== (const token_iterator &other) const
	Equality: both exhausted, or both at the same offset.
bool	operator!= (const token_iterator &other) const
	Inequality.

Detailed Description

Forward (single-pass) iterator yielding one token at a time.

A default-constructed iterator is the end sentinel. Each increment runs the lexer just far enough to produce the next non-skipped token; a lex_error thrown by the lexer propagates out of the increment.

Member Typedef Documentation

difference_type

using scilex::token_iterator::difference_type = std::ptrdiff_t

Required typedef.

iterator_category

using scilex::token_iterator::iterator_category = std::input_iterator_tag

Single-pass.

pointer

using scilex::token_iterator::pointer = const token*

Pointer to current.

reference

using scilex::token_iterator::reference = const token&

Reference to current.

value_type

using scilex::token_iterator::value_type = token

Yielded element.

Constructor & Destructor Documentation

token_iterator() [1/2]

scilex::token_iterator::token_iterator ( )

default

Constructs the end sentinel.

token_iterator() [2/2]

scilex::token_iterator::token_iterator ( const lexer &  owner, std::string_view  source, eof_policy  policy  )

inline

Constructs a begin iterator over source for owner.

Parameters

[in]	owner	The lexer providing the rules.
[in]	source	The text to scan.
[in]	policy	Whether to yield a terminal end_of_input token.

Member Function Documentation

operator!=()

bool scilex::token_iterator::operator!= ( const token_iterator &  other ) const

inline

Inequality.

Parameters

[in]

other

Another iterator.

Returns

true if the two differ.

operator*()

reference scilex::token_iterator::operator* ( ) const

inline

Returns the current token.

operator++() [1/2]

token_iterator & scilex::token_iterator::operator++ ( )

inline

Advances to the next token.

Returns

*this.

operator++() [2/2]

void scilex::token_iterator::operator++ ( int  )

inline

Post-increment (single-pass: no useful copy is returned).

operator->()

pointer scilex::token_iterator::operator-> ( ) const

inline

Member access to the current token.

operator==()

bool scilex::token_iterator::operator== ( const token_iterator &  other ) const

inline

Equality: both exhausted, or both at the same offset.

Parameters

[in]

other

Another iterator.

Returns

true if the two denote the same position/end.

---

The documentation for this class was generated from the following file:

• include/scilex/lexer.hpp