REAL
Regular Expression Algorithmic Library — constexpr C++20 regex
Loading...
Searching...
No Matches
lazy_dfa.hpp File Reference

A lazy, priority-preserving forward DFA over the Pike program (the kFirstMatch forward pass + cache). More...

#include <algorithm>
#include <array>
#include <atomic>
#include <limits>
#include <ranges>
#include <bit>
#include <cstddef>
#include <cstdint>
#include <span>
#include <string>
#include <string_view>
#include <vector>
#include "real/core/config.hpp"
#include "real/core/program.hpp"
#include "real/automata/utf8_ranges.hpp"
Include dependency graph for lazy_dfa.hpp:

Classes

struct  real::detail::byte_program
 A byte-level view of a Pike program for the DFA passes: every klass_cp is expanded into UTF-8 byte-range split/klass chains, so a forward DFA can represent it; the Pike program is untouched. eligible is false when an op no DFA can represent is present — the caller keeps the Pike VM. More...
 
struct  real::detail::utf8_trie_node
 One node of a minimal deterministic UTF-8 trie for a code-point class. Its byte-range transitions are pairwise disjoint (at most one edge matches a byte), which makes the byte-program one-pass-friendly. A target >= 0 is a node id; -1 is accept (the run continues at the construct's successor). More...
 
struct  real::detail::utf8_trie
 A minimal deterministic UTF-8 trie for a code-point class. root == -1 means the class is empty. More...
 
struct  real::detail::range_intern_table
 Intern table for UTF-8 edge byte ranges, keyed by the exact 16-bit (lo << 8) | hi. More...
 
struct  real::detail::literal_alt_trie
 A literal alternation (every branch a run of byte ops converging on one exit) factored into a trie that keeps leftmost-first priority. More...
 
struct  real::detail::literal_alt_trie::item
 One alternative of a node: a byte leading to a child, or the END of a branch. More...
 
struct  real::detail::literal_alt_trie::node
 One trie node: its alternatives in priority order. More...
 
struct  real::detail::literal_alt_chain
 The literal alternation starting at a split: its exit and each branch's bytes. More...
 
struct  real::detail::lazy_byte_alphabet
 Byte-class alphabet over a Pike program: bytes satisfying exactly the same byte/klass predicates share a class, so the DFA transitions over classes instead of 256 raw bytes. More...
 
struct  real::detail::pc_set_cache
 A chained hash set of interned state ids keyed by their pc-set: maps a candidate pc-set to its state id, or not_found. All-std::vector, not std::unordered_map, so the DFAs stay literal types (a constexpr real::regex embeds one in its scratch state). More...
 
struct  real::detail::visit_marks
 The pcs one closure computation has entered, by generation: starting one bumps the generation instead of clearing per pc, so a cache miss costs its closure, not the program's size (a large alternation's byte program runs to hundreds of thousands of instructions). More...
 
class  real::detail::lazy_dfa
 A lazy priority-preserving forward DFA over a Pike program (the kFirstMatch forward pass). More...
 
struct  real::detail::lazy_dfa::counters
 Cache-behaviour counters, for the policy tests. More...
 
struct  real::detail::lazy_dfa::anchored_result
 anchored_end's result: the match end (or real::npos) and how far the walk got. More...
 
class  real::detail::reverse_dfa
 The start-finder companion to lazy_dfa. Given a match end, it finds the leftmost start (the design guide §7.6 contract). It runs the inverted program — the forward program's edges transposed, its consuming bytes kept — as a cached DFA over the text scanned right-to-left from the end, recording an accept each time it reaches the original start (reverse-kLongest: the furthest-back accept is the start). It needs no priority ordering — its states are plain unordered (sorted) PC sets and its rule is longest — so it is simpler than the forward pass. Dynamic only. More...
 

Namespaces

namespace  real
 REAL's public API: real::regex, real::static_regex, real::flags and the match/iterator types built on them.
 
namespace  real::detail
 DFA construction internals: subset construction over a flattened NFA. Not a stable API.
 
namespace  real::detail::byte_ctx
 The context bits both lazy DFAs keep for the byte beside a position (before it forward, after it backward): a newline, an ASCII word byte, or – under word_quit only – a non-ASCII byte, which sets both bits at once since no ASCII byte does.
 

Enumerations

enum class  real::detail::ac_verdict : std::uint8_t { real::detail::not_consulted = 0 , real::detail::cascade , real::detail::automaton }
 What the AC density gate last decided; ac_density_last_verdict() below reports it. More...
 

Functions

bool & real::detail::lazy_dfa_route_disabled ()
 Test seam: force the matcher off the lazy-DFA route onto the pure Pike VM, so a differential can assert routed and unrouted searches agree in one binary. Not for production use (applies to every *_disabled seam below).
 
std::size_t & real::detail::lazy_dfa_byte_budget ()
 Test seam: the byte budget the search DFAs are built with (read at each DFA's construction).
 
bool & real::detail::bounded_backtrack_route_disabled ()
 Test seam: force the general loop off the bounded backtracker onto the Pike VM, so a differential can assert both agree on every small subject.
 
bool & real::detail::inner_literal_route_disabled ()
 Test seam: force the matcher off the inner-literal search route onto the core search. The route cannot miss a leftmost match because its reverse bound never advances mid-search.
 
bool & real::detail::rare_disc_route_disabled ()
 Test seam: force off the rare-discriminant prefilter (https?:// memchr-: route) onto prefix/first-byte search.
 
bool & real::detail::inner_literal_guard_disabled ()
 Test seam: force the inner-literal small-haystack guard off, so the route fires on any size and tiny correctness inputs exercise it. The guard uses regex_immutables::il_min_haystack on the first candidate scan and il_warm_floor thereafter.
 
bool & real::detail::trailing_la_route_disabled ()
 Test seam: force the matcher off the trailing-lookaround class+ route onto the pure Pike VM.
 
bool & real::detail::fixed_shape_pair_route_disabled ()
 Test seam: force the matcher off the heterogeneous fixed-shape pair-filter route onto the ordinary run_fixed_shape walk. The route only filters; match_fixed_body_wb decides each candidate.
 
bool & real::detail::fixed_shape_route_disabled ()
 Test seam: force the matcher off the fixed-shape walk (run_fixed_shape) onto the general Pike loop. inner_literal_route_disabled does not reach a fixed_shape pattern (the inner-literal gate excludes it); a differential on one needs this seam.
 
bool & real::detail::class_fastpath_disabled ()
 Test/profile seam: skip the dedicated class-scan fast paths (byte class-loop, cp-class-loop, codepoint_class, negated-class ./[^,]+), so such a pattern falls through to lazy-DFA / general.
 
bool & real::detail::possessive_fastpath_disabled ()
 Test/profile seam: force the matcher off the possessive-loop fast paths (bare/suffixed/delimited X*+/X++) onto the general VM.
 
bool & real::detail::aho_corasick_route_disabled ()
 Test seam: force the matcher off the Aho-Corasick multi-literal route onto the pattern_hints::fixed_alternation run_alternation path.
 
bool & real::detail::alternation_pairs_disabled ()
 Test seam: keep an alternation's block scans on its first bytes whatever the subject's density, so a differential can compare the pair filter with the first-byte scan.
 
bool & real::detail::alternation_nibbles_disabled ()
 Test seam: mask a dense alternation's blocks by its byte pairs rather than by the nibble fingerprint, so a differential can compare both filters.
 
bool & real::detail::ac_density_gate_disabled ()
 Test seam: take the Aho-Corasick density gate out, so the route is chosen on branch count alone.
 
std::atomic< bool > & real::detail::il_density_last_abandoned ()
 Test observability: whether the inner-literal density gate last abandoned the route.
 
std::atomic< ac_verdict > & real::detail::ac_density_last_verdict ()
 Test observability: the AC density gate's most recent verdict.
 
constexpr utf8_trie real::detail::build_utf8_trie (const cp_class &cc, std::span< const code_range > cp_ranges)
 Builds the minimal deterministic trie recognising a code-point class's UTF-8 byte sequences.
 
constexpr std::size_t real::detail::utf8_trie_emit_size (const utf8_trie &trie)
 The instruction count emit_utf8_trie writes: an empty class is one dead klass; otherwise each node is a split-guarded chain of k byte ranges (3k - 1 instructions).
 
REAL_BUILD_COLD constexpr void real::detail::emit_utf8_trie (byte_program &bp, const utf8_trie &trie, std::int32_t after, range_intern_table &seen)
 Emits trie into bp as a deterministic split/klass/jump fragment, interning each edge's byte range through seen.
 
bool & real::detail::alternation_trie_disabled ()
 Test seam: build the byte program's literal alternations flat, so a differential can compare the trie against them.
 
constexpr bool real::detail::detect_literal_alt (std::span< const instr > code, std::size_t pc, literal_alt_chain &out)
 Whether [pc, exit) is a chain of splits whose branches are runs of byte ops jumping forward to one exit, the last branch falling through to it.
 
REAL_BUILD_COLD constexpr byte_program real::detail::build_byte_program (const program_view &prog, bool keep_assertions=false, std::size_t max_size=max_byte_program_size)
 Builds the byte-level DFA program for prog (see byte_program). A klass_cp at P (the op plus three utf8_cont slots) is replaced by its class's deterministic UTF-8 trie (build_utf8_trie), converging on the mapped P+4; every other op is copied with remapped targets. The first pass sizes each construct into the old→new pc map and enforces max_size; the second emits.
 
constexpr bool real::detail::is_cr_line_assert (const instr &in)
 Whether in is an ECMAScript line assertion, whose line ends at \r as well as \n.
 
REAL_BUILD_COLD constexpr lazy_byte_alphabet real::detail::compute_lazy_alphabet (std::span< const instr > code, std::span< const char_class > classes)
 Partition 0..255 by the program's consuming predicates (every klass test, every byte literal). Bytes with an identical signature collapse to one class.
 
constexpr bool real::detail::undecidable_word (assert_kind kind, bool prev_word, bool prev_nonascii, bool next_word, bool next_nonascii)
 Whether a word assertion needs a code point's word-ness that one byte does not give: a side it reads is a non-ASCII byte, and the ASCII side does not settle it alone.
 
constexpr bool real::detail::dfa_representable (std::span< const instr > code, bool ascii_word)
 Whether a lazy DFA, forward or reversed, can represent every op of code.
 
constexpr bool real::detail::byte_ctx::is_newline (std::uint8_t ctx)
 Whether context ctx says its byte is a newline.
 
constexpr bool real::detail::byte_ctx::is_word (std::uint8_t ctx)
 Whether context ctx says its byte is an ASCII word byte.
 
constexpr bool real::detail::byte_ctx::is_nonascii (std::uint8_t ctx)
 Whether context ctx says its byte is not ASCII.
 
constexpr std::uint8_t real::detail::byte_ctx::of (std::uint8_t b, bool cr)
 The context bits of byte b.
 

Variables

constexpr int real::detail::max_loop_hops {8}
 Cap on how far a jump chain is followed to a loop head (empty-iteration exit routing), shared by every closure walk; a loop join reaches its split in one hop, so eight is headroom, not a knob.
 
constexpr std::size_t real::detail::lazy_dfa_default_byte_budget {std::size_t {64} << 20U}
 
constexpr std::size_t real::detail::alternation_trie_min_branches {64}
 
constexpr std::size_t real::detail::max_byte_program_size {20000}
 
constexpr std::uint8_t real::detail::byte_ctx::newline {2}
 The byte is a newline.
 
constexpr std::uint8_t real::detail::byte_ctx::word {4}
 The byte is an ASCII word byte.
 
constexpr std::uint8_t real::detail::byte_ctx::nonascii {6}
 The byte is not ASCII (word_quit only).
 

Detailed Description

A lazy, priority-preserving forward DFA over the Pike program (the kFirstMatch forward pass + cache).

Not real::dfa (<real/dfa.hpp>), a maximal-munch recognizer over unordered NFA-state sets. Here a DFA state is an ordered NFA-state set memoizing the leftmost-first Pike closure, so the forward pass reports the boundary the Pike VM would. pike.hpp routes an eligible search through the forward end, the reverse start (reverse_dfa), then the VM on the located window. Dynamic only: the cache is mutable.