# Differences from Python re REAL targets parity with Python's `re`, and a randomized differential fuzzer plus a large parity corpus enforce it. A handful of behaviours diverge **on purpose**; each is listed here with its rationale, and each is pinned by a direct, version-independent assertion in `TestIntentionalDivergences` (bindings/python/tests/test_real.py) so the contract cannot drift silently. ```{note} **Excluded by design — a closed door, not a missing feature.** Backreferences, recursion, and callouts are **not** on a roadmap: each makes matching super-linear and would reopen the ReDoS door this engine exists to close. Refusing them is the point of a linear-time engine, so they raise a clear `real::regex_error` rather than being "not yet supported". This is the one promise REAL will not trade for feature coverage. ``` (div_ascii)= ## UTF-8 character classes (code-point mode) In code-point mode (the default), a character class carries specific non-ASCII code-point members and ranges: `[é]`, `[éàü]`, `[à-ÿ]`, `[a-zé]`, and their negations `[^é]` / `[^à-ÿ]` all match exactly like `re` (the code-point oracle). They compile to the canonical UTF-8-ranges automaton, so a class never matches an overlong or surrogate byte sequence — `[é]` matches only `C3 A9`, and `[^é]` matches every valid code point *except* é. `.` and an ASCII-only negated class such as `[^x]` still match *any* non-ASCII code point (the pre-existing sound superset, which also accepts a malformed byte sequence as "a character"). A malformed UTF-8 member in the pattern is a `real::regex_error`. In **bytes mode** a non-ASCII class member is rejected (raw byte semantics), which is what the `std::regex` compat layer relies on to fall back to `std`. (div_icase)= ## Unicode text-mode semantics: case folding, shorthands and boundaries Under `IGNORECASE` in text (code-point) mode, REAL does full **Unicode simple case folding** for literals, classes and ranges — identical to `re.IGNORECASE`: `é` matches `É`, `[é]`/`[à-ÿ]` fold, `k` matches Kelvin (U+212A), `ß` matches `ẞ` (but not `ss` — simple, not full, folding). A cased literal is promoted to a foldable code point; a \xHH or octal byte escape keeps byte provenance and **never folds** (\xe9 under `IGNORECASE` does not match `É`) — the same deliberate byte-vs-code-point split described in the hex-escape section below. The \w \W \d \D \s \S shorthands are **Unicode** in text mode (matching `re`): `\\w` matches any Unicode word code point (é, ٣, ², astral letters…), `\d` any `Nd` digit (Arabic ٣, fullwidth ９…), `\s` exactly Python `re`'s whitespace — Unicode White_Space (NBSP, U+2028…) **plus** the four separators `U+001C`–`U+001F` that `re` (via `str.isspace`) adds and Unicode does not. This holds in or out of a character class (`[\\w]`, `[\\d.]`, and `[\\W]`/`[\\D]`/`[\\S]` — accepted as their Unicode complements, with `[^\\W]` == `\\w` etc). The `\\b \\B` word boundaries (and the `\\< \\>` word-start/end extensions) use the same Unicode word-ness, so `\\bété\\b` matches. In **bytes mode**, and under `flags::ascii` (`re.A`), every shorthand and boundary stays ASCII (byte-for-byte `std::regex`), and `IGNORECASE` folds ASCII only, so the compat layer is unaffected. Case-insensitive matching follows **CPython's** equivalences (via `str.upper`/`lower`), so the Turkish **dotless/dotted I** fold with I/i — `(?i)I` matches ı (U+0131) and İ (U+0130), exactly as stdlib `re` does (`re.fullmatch("(?i)I", "ı")` is True) and as `(?i)\p{Lu}` therefore matches ı. Unicode **simple CaseFolding** — used by RE2 and the Rust `regex` crate — keeps ı apart instead. This is one code point (two, with İ) where the two conventions differ; **both engines are correct for their contract** (REAL for `re`-parity, the crate for UTS#18 folding). The Rust binding's differential masks exactly this set (its `ICASE_FOLD_DELTAS`, computed by asking both engines), the twin of the `\\w`/`\\s` mask above. (div_cpython_scoped_ascii_bug)= ## A known CPython 3.14 oracle bug (not a REAL divergence) A scoped ascii group — `(?a:...)`, with `a` among the *added* letters — behaves inconsistently in CPython 3.14 specifically when it is the **pattern's own first construct**, with *nothing* preceding it at all (not even a zero-width assertion or a single literal byte): a negated shorthand `\S`/`\D`/`\W` inside it wrongly fails to match `U+001C`–`U+001F`, the same four separators {ref}`div_icase` above documents as included in text-mode `\s` and excluded from ascii-mode `\s`. Concretely, on `'\x1c'`: `re.search(r"(?a:\s)", ...)` correctly returns no match (not ascii whitespace) — but `re.search(r"(?a:\S)", ...)` **also** returns no match, an outright partition violation (`\x1c` matches neither `\s` nor its own negation `\S`). Prepending literally anything before the group — `\B`, `^`, a lookahead, or a single literal byte — "fixes" it on `re`'s side; a runtime `search(text, pos, endpos)` offset does **not** (confirmed a compile-time artifact of `re`'s own opcode order, not a runtime one — `pos > 0` still fails). REAL has no such inconsistency: `(?a:\S)` and `\B(?a:\S)` agree, and every one of these four codepoints partitions correctly between `\s`/`\S` regardless of what, if anything, precedes the scoped group. This is filtered out of the differential fuzzer rather than "fixed" on REAL's side — there is nothing on REAL's side to fix, `re` is the one behaving inconsistently — and pinned directly against `re`'s own (buggy) behaviour in `TestIntentionalDivergences. test_scoped_ascii_negated_shorthand_leading_bug` (bindings/python/tests/test_real.py), so a future CPython release that fixes its own bug is caught (the pin asserts `re`'s buggy value too, not just REAL's correct one) rather than silently masked forever. (div_hex)= ## \xHH (HH ≥ 0x80) is byte-level \xHH matches the raw byte `0xHH`, not the codepoint `chr(HH)`. On a str pattern `re` reads \xe9 as U+00E9 (é); REAL reads it as the single byte `0xE9` — which a well-formed str's UTF-8 of é (`C3 A9`) never contains. Use `é` for the codepoint. One deliberate exception makes \xHH **context-dependent** in a str (code-point-mode) pattern: *inside a character class* [\xHH], a value ≥ 0x80 is the code point `U+00HH` (so [\xe9] == `[é]` == `[é]`), because a class member is a code point, whereas outside a class \xe9 stays the single byte `0xE9`. In `bytes` mode \xHH is always the raw byte, in a class or not (so a `bytes` class is byte-for-byte `std::regex`). (div_boundary)= ## \b / \B on an empty string or region By definition an empty string has no word boundary, so \B (not-a-boundary) matches at position 0 while \b does not. REAL implements this by-definition behaviour, which is also `re`'s since Python 3.11 (`re` < 3.11 had the opposite quirk for \B on the empty string). The differential fuzzer therefore cannot use `re` as an oracle on an empty region and skips it; the assertions pin REAL's behaviour directly. (div_empty_iteration_capture)= ## Capture of a nullable loop's final iteration **The rule.** For a `*` or `+` loop whose body can match the empty string, `re` runs one final **empty** iteration after the last consuming one and records *its* (zero-width) capture for the group; REAL records the last **consuming** iteration instead. The overall match — group 0, and every match **span** — is identical in both; only the inner group's captured span differs. This remains only for a loop whose body **consumes** on its last productive iteration (so the inner group captured that run). A loop whose body can only ever match empty (`()*`, `($)*`) now agrees with `re` — the greedy-loop empty-exit fix routes its empty iteration to the loop exit, applying that iteration's capture, exactly as `re` does. **The verified forms** (search; `re` value ↦ REAL value for group 1): | Pattern | Input | `re` group 1 | REAL group 1 | | | --- | --- | --- | --- | --- | | `(a*)*` | `"a"` | `(1, 1)` | `(0, 1)` | diverges (consuming body) | | `(a*)+` | `"a"` | `(1, 1)` | `(0, 1)` | diverges (consuming body) | | `(a\|)*` | `"a"` | `(1, 1)` | `(0, 1)` | diverges (consuming body) | | `()*` | `""` | `(0, 0)` | `(0, 0)` | **now agrees with `re`** | | `($)*` | `""` | `(0, 0)` | `(0, 0)` | **now agrees with `re`** | **Lineage.** This is the linear-engine convention: **RE2, the Rust `regex` crate, and Go's `regexp`** share REAL's behaviour (they do not re-enter a loop on an empty match). `re` and PCRE, being backtrackers, take the extra empty step. REAL sits with the linear engines by design — the same family whose linear-time guarantee it shares. **It also touches the compat contract.** The same group-capture difference shows up under `real::compat` against the local `std::regex` (ECMAScript is a backtracker too). The exhaustive compat check measures it — 4 548 cases out of 3.2 M in the tier-1 space — and tolerates *only* this exact signature (whole match identical, the differing group is `std`'s zero-width empty-final iteration), failing on any other divergence. It is documented, not routed to `std`, on purpose: routing nested nullable quantifiers to a backtracker would forfeit the linearity `real::compat` exists to keep. See `COMPATIBILITY.md`. **When to revisit.** Only if a future goal is *exact* `re`/`std` capture parity for these degenerate loops, which would mean adding a trailing empty-iteration step purely to update a capture — a cost with no matching benefit for a linear engine, and none that would justify handing catastrophic-backtracking patterns to a backtracking fallback. Pinned, both values, in `TestIntentionalDivergences`; the compat side is guarded by the exhaustive-compat gate's exact-signature discriminator. (div_empty_first_branch_loop)= ## Span of an empty-first-branch loop under a forced-non-empty retry **The rule.** An unbounded `*`/`+` loop whose body's **first** alternative is empty and a later one consumes (`(|a)*`, `(|a)+`) has, under a **forced-non-empty** match — the step `finditer` / `sub` take at a position where the whole match would otherwise be empty — a different **span** in `re` and REAL. `re` exits the loop through the empty branch after the first consuming iteration (the shortest non-empty match); REAL consumes maximally. On `"aa"`: | Pattern | `re` finditer | REAL finditer | | --- | --- | --- | | `(\|a)*` | `(0,0)(0,1)(1,1)(1,2)(2,2)` | `(0,0)(0,2)(2,2)` | | `(\|a)+` | same as `*` | `(0,0)(0,2)(2,2)` | | `(a\|\|b)*` on `"ab"` | `(0,1)(1,1)(1,2)(2,2)` | `(0,2)(2,2)` | **Green witnesses (parity).** `search` / `match` / `fullmatch` agree for these patterns — the divergence is **only** the forced-non-empty retry. An empty-**last** branch `(a\|)*` agrees (its empty branch is the loop's natural secondary exit, not a deduped back-edge). The **bounded** forms `(\|a){2}`, `(\|a){1,3}` agree (unrolled, no back-edge). This is distinct from {ref}`div_empty_iteration_capture`, where the match spans are identical and only a group **capture** differs. **The mechanism, both sides.** In REAL's star loop (Thompson `split → body → jump-back`), an empty-first body makes the jump-back land at the **same position**; REAL's per-position thread dedup — which is *how* it guards against an infinite empty loop — removes that empty back-edge, so the greedy loop falls through to consuming. `re`, a backtracker, takes the empty iteration as a loop exit (the empty branch's exit preference). The `regex` crate does a **third** thing (all-empty: `(0,0)(1,1)(2,2)`), so **`re` is the arbiter, never the crate** (the crate's own bugs on this family are in the Rust binding's known-bugs). **When to revisit.** A fix would have to distinguish an empty iteration that should **exit** the loop from one that should be **deduped**, at `re`'s exact priority, under the runtime forbid-empty flag — a change to the star-loop termination that underlies *every* quantifier. Judged not worth it for a class this rare and this constrained (the forced-non-empty retry only; `search`/`match` in parity); a weighed choice, not an oversight. Pinned in `TestIntentionalDivergences`. (div_rejected)= ## Rejected by design Each of these raises a clear `real::regex_error`, for a reason worth keeping: - **Backreferences** (`(a)\1`, `(?P=name)`) — a backreference makes the language non-regular; supporting it would forfeit the linear-time, ReDoS-safe guarantee that is REAL's reason to exist. - **Conditional groups** `(?(id)yes|no)`, **recursion**, and **callouts / subroutine calls** — non-regular control flow, each super-linear in the worst case. Excluded for the same ReDoS reason (see the note at the top of this page). **If you must run one of these anyway**, the opt-in is only where a *backtracking* engine is within reach, and it forfeits the linear-time guarantee for that pattern alone: in C++ a `real::compat::regex` (``) built with `real::compat::policy::fallback` delegates it to `std::regex`, and in Python `real.compile(..., fallback=True)` delegates it to `re`. **Rust's `fallback` feature does not reach this far** — it delegates to the `regex` crate, which is linear too and refuses a backreference just as REAL does; there it closes the `\\p{...}` namespace and folding gaps instead. A native `real::regex` has no opt-in at all: it is the linear engine or nothing, which is what makes the chosen backend (`Pattern.engine`, `Regex::engine()`, the compat policy) worth reporting. Unlike the above, **a possessive quantifier or atomic group over a compound body** (`(?:ab)*+`, `(?>ab|a)`) is rejected "not supported **yet**", not "by design" — see {ref}`div_possessive`. The *linear-time* argument for the general case is expected to hold (a bounded compound body looks like it can reuse the same priority-kill sub-VM technique lookaround already ships), the gap is VM-integration work not yet done, not a ReDoS concession. Recursion/callouts/conditionals above stay permanently excluded regardless — that door really is closed. (div_module_surface)= ## The module surface (Python binding) Everything above is about the pattern language. Four differences are in the *module* instead, and a `import real as re` drop-in meets them without writing a pattern at all: - **Flags are plain `int`, not a `RegexFlag` enum.** `real.I | real.M` is `10`, where `re.I | re.M` reprs as `re.IGNORECASE|re.MULTILINE`; there is no `real.RegexFlag`, so an annotation spelled `flags: re.RegexFlag` becomes `flags: int`. The values themselves are `re`'s, so any flag expression that works there works here. - **`Pattern.flags` reports what you passed.** For a `str` pattern `re` adds `re.UNICODE` — `re.compile('a', re.I).flags` is `34`, REAL's is `2` — so compare against what you passed, not against `re`'s echo. Bytes patterns agree on both. - **`re.L` / `re.LOCALE` and `re.DEBUG` raise instead of being ignored.** Locale-dependent matching is excluded by design (this engine is Unicode in text mode and raw bytes otherwise, never locale-dependent); `DEBUG` dumps CPython's own compiler state, which REAL does not have. An unrecognised flag bit is refused too, rather than silently dropped. - **`re.Scanner` is absent.** It is undocumented in CPython and has no REAL equivalent. `re.error` and `re.PatternError` are both spellings of one class here, as they are in `re` since CPython 3.13, so `except` on either catches what the other raises. (div_possessive)= ## Possessive quantifiers and atomic groups (Tier 1 — a capability beyond re? no: parity, deliberately narrower for now) `re` (3.11+) and PCRE2 support possessive quantifiers (`a*+`, `a++`, `a?+`, `a{n,m}+`) and atomic groups `(?>...)` generally, over any body. Neither RE2 nor the Rust `regex` crate support them at all — REAL closing part of this gap is the differentiator, in the same class as REAL's already-shipped bounded lookarounds. **What REAL accepts (Tier 1 — the dominant real-world shape):** - A possessive quantifier over a **bare single atom** — a literal byte, a character class, or `.` — e.g. `[^"]*+`, `\d++`, `x?+`, `[a-z]{2,4}+`. - The same, wrapped in **exactly one capturing group** — `(a)*+` — the group reflects the *last* successful repetition's span, matching `re`'s own semantics for a possessive loop. - An atomic group `(?>X*)` / `(?>X+)` / `(?>X?)` / `(?>X{n,m})` wrapping a bare or singly-captured atom — desugars to exactly the possessive-quantifier case above, regardless of whether the inner repeat's own written form used the `+` suffix (so `(?>[^"]*)` and `(?>\d+)`, the dominant real-world atomic-group shapes, are NOT rejected as "unbounded" — a compile-time detection-order requirement, not an accident). - An atomic group with **no repeat at all**, over any deterministic (split-free) body, however compound — `(?>ab)`, `(?>)` — nothing to give back regardless of body shape, since it never loops; compiled inline, at zero extra VM cost. **What REAL rejects, "not supported yet":** a possessive quantifier or atomic group over a **compound, repeated body** — `(?:ab)*+`, `(?:X++)*+`, `(?:a?+)*+` — and an atomic group over a **genuine alternation**, even with no repeat — `(?>ab|a)`, `(?>a|ab)b`. A design-first spike (D0) confirmed the linear-time argument for the general case is sound in principle (the same isolated, bounded sub-VM technique lookaround already ships could plausibly carry it); implementing it is separately-scoped future work (D2), not a ReDoS concession — see the note in {ref}`div_rejected` above. **A correctness note on the shipped Tier 1 itself, for anyone extending it:** the possessive-loop opcode's match/no-match decision is made at **epsilon-closure time** (`add_thread`, pike.hpp), not deferred to the per-byte step. An earlier design that decided it one round later, in the byte-stepping dispatch, shipped with a real bug: inside an alternation, a lower-priority sibling that resolves in a single round could claim the shared post-alternation convergence point before the (higher-priority, but multi-round) possessive thread's step()-time exit got a chance to compete for it — plain first-felt-this-generation dedup has no notion of true priority once insertion order like that is violated. The fix — evaluating the atom test at insertion time, in the same priority-ordered closure pass as everything else — is precedented directly above it in the same closure: `assert_lookaround` already runs a whole sub-VM decision at closure time, so testing one byte/class/codepoint there is a direct extension of an existing pattern, not new architecture. Regression-pinned in `possessive_alternation_priority_regression` (tests/frontend/test_possessive_atomic.cpp) and found originally by the differential fuzzer's own possessive-quantifier generator (test_differential_fuzz.py) before it ever shipped. **A superset of `re`:** **Unicode property classes** \p{...} are built in for **General_Category**, **Script**, **Script_Extensions** and the **standard binary properties** (see {ref}`div_property`), which standard `re` rejects outright. **Named characters** \N{NAME} are also supported — the Python binding resolves the name via `unicodedata` and the engine takes the resulting code point — so no table lives in C++. REAL additionally accepts the scalar form \N{U+XXXX} directly (a PCRE2-style **extension**; `re` does not). (div_named_scalar)= ## \N{U+XXXX} scalar escape (a capability beyond re) REAL accepts the **scalar** form \N{U+XXXX} (1–6 hex, a PCRE2-style spelling) as a code point, on **both** surfaces — the C++ engine and the Python binding — so the two are uniform. `re` knows \N{...} only as a character *name* and rejects the U+ form (`undefined character name`). The *name* form \N{NAME} is exact `re`-parity (the binding resolves it via `unicodedata`); only the scalar form is the extension. Pinned real-accepts / re-rejects in `TestIntentionalDivergences`. (div_property)= ## Unicode property classes \p{...} (a capability beyond re) REAL builds in **General_Category** (\p{L}, \p{Lu}, \p{Nd}, and the groups \p{L}..\p{C}), **Script** (\p{sc=Greek}, \p{Script=Latin}) and **Script_Extensions** (\p{scx=Cyrl}), with short codes, the UCD long names (\p{Letter}), `gc=` / `sc=` prefixes, loose matching (UAX44-LM3), the single-letter form \pL, and negation \P{...} — on **both** the C++ engine and the Python binding. Standard `re` rejects \p entirely (a `bad escape` error), so accepting it is a **superset**: it cannot break re-compatible code (which could never use \p). The class matches as a linear code-point predicate (the same `klass_cp` mechanism as `\w`), so it stays ReDoS-safe, and `flags::ascii` (`re.A`) does **not** restrict it — a Unicode property is always Unicode. The tables are pinned to a Unicode version and validated exhaustively against the UCD (the regen guards). The **standard binary properties** (\p{Alphabetic}, \p{White_Space}, \p{Emoji}, …) are built in the same way; other UAX44 namespaces (\p{Bidi_Class=L}, `Word_Break`, `Age`, …) raise `unsupported` (the Rust binding can delegate those via its `fallback` feature). Pinned in the property and parity suites. (div_lookbehind)= ## Variable-width lookbehind (a capability beyond re) REAL accepts any **bounded** lookbehind, including variable-width alternations such as `(?<=a|bb)`, which `re` and PCRE reject as non-fixed-width. The bound keeps the scan linear; see How REAL Works — a guided tour for how the lookbehind sub-VM works. **`\b` / `\B` inside lookbehind.** Word-boundary assertions in a lookbehind are first-class and evaluate at the lookbehind's match position (the same rule as outside a lookaround). REAL agrees with Python `re`, JavaScript (V8/node), and the usual ECMAScript reading on these cases. **PCRE2 has known mis-evaluations of `\b` in lookbehind** (a few failures per ~100k differential cases in third-party audits) — treat PCRE2 as a *benchmark competitor*, not an oracle for `\b`-in-lookbehind. There is no in-tree PCRE2 differential fuzzer (unlike `fuzz_compat` vs `std` and `fuzz_re2`); if one is added later, those shapes belong on an allowlist the same way `std`'s non-spec `\b` edges are, not as REAL bugs.