NumericAL
A header-only C++20 lexer built on REAL
Loading...
Searching...
No Matches
Classes | Concepts | Typedefs | Enumerations | Functions | Variables
numerical::detail Namespace Reference

Classes

struct  a_rows
 A as gemm_packed reads it by default: row-major, m × k, its rows lda apart, packed by the tile. More...
 
struct  a_transposed
 A given as Aᵀ: row-major, k × m, its rows lda apart (C = Aᵀᵀ·B, no copy of the transpose). More...
 
struct  arithmetic
 The element operations of T the tensor operators apply: the standard ones. More...
 
struct  arithmetic< T >
 A signed integer's operations through its unsigned type, so that an overflow wraps modulo 2ⁿ as NumPy's integers do (two's complement) where the signed operation is undefined. More...
 
struct  arrow_workspace
 The buffers merge_arrow works in, kept by each thread from one join to the next (thread_arrow_workspace): mapping their pages anew took 11–16 % of an svd of order 64 to 512 on one x86-64 core, 13–20 % on six. More...
 
struct  axis_walk
 The extents and strides that walk a tensor of shape with axis axis taken out: outer positions before it, inner after it, the axis itself extent long at stride inner. More...
 
struct  b_rows
 B as gemm_packed reads it by default: row-major, k × n, its rows ldb apart. More...
 
struct  b_transposed
 B given as Bᵀ: row-major, n × k, its rows ldb apart (C = A·Bᵀᵀ, no copy of the transpose). More...
 
struct  balancing
 How a matrix was balanced: swaps, in the order they were applied, then the scaling of the active block low … high − 1 by scale. More...
 
struct  band_storage
 A band's entries in LAPACK's column-major band storage: AB(i, j) (from 1) at (i − 1) + (j − 1) ldab. More...
 
class  band_work
 The lower band of a symmetric or Hermitian matrix of order n and half-bandwidth width, one diagonal wider for the bulge a band reduction chases: entry (i, k) of the lower triangle on diagonal i − k. More...
 
struct  bidiagonal_node
 One block of bidiagonal_divide's tree: the bidiagonal it decomposes (its own copies scaled by a power of two, or its parent's), the two halves it splits into, and its decomposition once solved. More...
 
struct  block_exchange
 An exchange of two diagonal blocks as exchange_real_blocks plans it: the blocks after it, and the transforms of rows and columns, ‘S’ = LIᵀ S IR(IR‘ the columns’ transform). More...
 
struct  bluestein_plan
 Bluestein's convolution for a length n with a prime factor past fft_largest_radix. More...
 
struct  cluster_circles
 The circles of Cauchy's integral about the clusters of the reordered Schur form t (each a run of its diagonal): the runs' starts (and n), and each run's centre and radius (0 for a single eigenvalue). More...
 
struct  complex_divisor
 A complex pivot as over_pivot takes it: its reciprocal, or, where that would overflow (a subnormal pivot), the pivot itself, divided by. More...
 
struct  complex_operand
 A complex vector that rows are summed against, kept as (re, −im) and (im, re) pairs: a complex row, read as its parts in order ([complex.numbers]), against each is the real or the imaginary part of the sum, one dot_product each. More...
 
struct  contraction_step
 One step of a contraction path: the operands at first and second of the current list, whose result goes to the list's end. More...
 
struct  cosine_line
 The cosine transforms of scipy.fft.dct, unscaled, on one line of n points by a Fourier transform: type 1 the real transform of the even extension (2(n − 1) points); type 2 the real transform of the points reordered evens then odds reversed, each frequency turned by roots (e^{−iπk/2n}), as Makhoul's; type 3 its inverse, by the inverse real transform; type 4 of an even n the complex transform of n/2 points, the even ones and the odd ones reversed paired and turned by roots (e^{−iπ(4j+1)/4n}), each frequency by turns (e^{−iπk/n}); of an odd n the complex transform of n points, 8n-th roots split as eighth roots times n-th ones (u n + 8 v = 1, as n is odd): the point 2j + 1 (mod n) the j-th times χ₈(2j + 1) + i χ₋₈(2j + 1), each value ±√2 (signs) times the real part of one frequency (picks), v (2k + 1) or its opposite (mod n), by those characters of u (2k + 1). More...
 
struct  decimal_rounding
 A real of type R rounded to decimals places, half to even, as numpy.round: nearbyint(x · 10ᵈ) / 10ᵈ (nearbyint(x / 10⁻ᵈ) · 10⁻ᵈ for a negative d), computed in C. More...
 
struct  default_init_allocator
 The allocator of numerical::buffer. An element constructed with no value is default-initialised, so a double keeps whatever its memory held, where std::allocator zeroes it; one constructed with a value is constructed from it. A result written in full is then written once, by the threads that compute it, not zeroed first on one of them (a product of order 512 on eight arm64 threads spent a fifth of its time in that zeroing). More...
 
struct  definite_pencil
 The pencil A − λ B of a Hermitian A and a Hermitian positive-definite B (each read in its lower triangle, of the diagonal its real part) in standard form: B = L Lᴴ and C = L⁻¹ A L⁻ᴴ, Hermitian, as LAPACK's sygst (type 1) forms it from pencil_reduction_least, by two triangular solves below (pencil_standard_form). C's eigenpairs (λ, y) are the pencil's (λ, L⁻ᴴ y), those vectors B-orthonormal (Xᴴ B X = I), as scipy.linalg.eigh(a, b)'s. More...
 
struct  deflation_count
 What early_deflation found: the eigenvalues left as shifts, and those deflated. More...
 
struct  divide_node
 One block of divide_and_conquer's tree: its rows [0, n) of the diagonal d its parent reads, the diagonal and subdiagonal it is solved on (its own copies scaled by a power of two, or d and the parent's), and its diagonal block of the eigenvector matrix, q, rows ldq apart. More...
 
struct  double_double
 A double-double number: hi + lo, hi the sum rounded to double. More...
 
struct  dqds_state
 What one dqds step hands the next: the least ds of the last transform and the shift's memory. More...
 
struct  fft_complex
 The complex type a transform of T computes in: T itself, or a real type's complex. More...
 
struct  fft_complex< C >
 fft_complex of a complex type: itself. More...
 
struct  fft_output
 The result of a transform along one axis, allocated before its plan, whose tables grow with the points: a result past memory refused at once, an empty one planned for nothing. More...
 
struct  fft_plan
 The factors and passes of a transform of one length, built once and shared. More...
 
class  fft_scratch
 A transform's working space of C: the calling thread's own, kept for the next transform, up to fft_scratch_kept elements; past it, its own allocation. Each Slot is a space of its own, for spaces of one type in use at once. More...
 
struct  fft_source
 The elements a transform reads where they lie, row-major, of shape shape: a tensor's, or an array's a caller lends (the Python binding's NumPy arrays). More...
 
struct  four_step_plan
 A transform of n = n1·n2 points in four steps (Bailey's), each a batch of lines the threads share: x as n2 rows of n1, x[j1 + n1·j2]; each of its n1 columns transformed in place (column_plan, n2 points, a line's transform as transform_axis's along a first axis); each (k2, j1) times w^{j1·k2} (twiddles); each of its n2 rows transformed (row_plan, n1 points) and written down a column of the output, X[k2 + n2·k1] row k2's k1. Each element's operations fixed by n alone: its bits on any threads. More...
 
struct  hermitian_similarity
 A Hermitian matrix A's eigenvectors V (hermitian_system) and Vᴴ A V: its Schur form Λ to the similarity's rounding, which hermitian_remainder_power keeps above the diagonal. More...
 
struct  hessenberg_panel
 One panel of hessenberg_blocked, Q_panel = I − V T Vᴴ. More...
 
struct  interleaved_components
 A complex matrix's eigenvector during its back-substitution, kept as (re, −im) and (im, re) pairs: a complex row, read as its parts in order ([complex.numbers]), against each is the real or the imaginary part of their product, one dot_product each. More...
 
class  kept_blocks
 Blocks freed and kept for the next buffer that fits one: a fresh block's pages are each faulted in and zeroed by the system at their first write, on the thread that writes it (glibc hands a large freed block back to the system unless the next request is of its very size; macOS's allocator likewise), where a kept block's are already in place. Places blocks at most, each under Largest bytes, the newest kept: a block freed while every place is taken frees the one kept longest. The first allocation one fits, with no more than half of it unused, takes it. Blocks aligned on Alignment. More...
 
struct  laid_axis
 A transpose laid out (tensor::laid_out) and the axis matching one of its view's. More...
 
struct  library_sine
 The C library's sin (Cosine false) or cos, call(x): one class each, so that each is covered whole. More...
 
struct  library_sine< false >
 library_sine's sin. More...
 
struct  library_sine< true >
 library_sine's cos. More...
 
struct  merge_workspace
 The buffers of merge_rank_one, kept from one join to the next: a fresh buffer per join is mapped anew, page by page. Each is written before it is read (updated zeroed row by row by the gathers). More...
 
struct  numpy_loops
 Which of NumPy's complex loops fuse their multiply-adds: its product's vector loop, its product's scalar loop (detail::numpy_loop_is_scalar), its quotient's and its product's reduction loop. They follow how its wheels were built, not only the processor, so the binding measures them on the NumPy it runs beside. More...
 
struct  octant_share
 e^{−2πi k/n}'s angle reduced to an eighth of a turn by integer arithmetic, 8k = octant·n + rest: the octant, and the angle's share of π/4 times n, rest (n − rest in an odd octant, measured back from its end). n is at most 2⁵⁰, so that 8k and the share are exact. More...
 
struct  one_based
 A row-major matrix read through indices from 1, (1, 1) its first entry. More...
 
struct  operand_span
 The elements of an operand of einsum_pair, row-major along its labels: a tensor's, or an array's read where it lies. More...
 
struct  packed_left
 A left operand packed once for several products by it (an LU's −L₂₁ against blocks of its U₁₂): the tile chosen and A packed for it (gemm_packed's layout of one panel), or A to read in place (plain). More...
 
class  packing_storage
 The storage of a product's packed operands, kept by the calling thread from one product to the next. More...
 
struct  pairwise_tree
 The shape of NumPy's pairwise summation tree over the values of type T: below lanes values summed in order; up to block in lanes interleaved partial sums, added pairwise, then the remainder in order; above, the first half and the rest summed apart. NumPy counts in reals: a real's 8 partial sums and blocks of 128 are a complex number's 4 (its parts side by side) and 64, its half a multiple of 4. More...
 
struct  pairwise_tree< C >
 pairwise_tree of complex numbers, as NumPy's sums their parts interleaved. More...
 
struct  pencil_bounds
 The bounds of dlag2's WSIZE (pencil_wsize), fixed by the pencil's scales. More...
 
struct  pencil_pair
 The eigenvalues of a 2 × 2 pencil, scaled (pencil_eigenvalues): wr1 / scale1 and wr2 / scale2, or the pair (wr1 ± i wi) / scale1. More...
 
struct  pencil_rotations
 The rotations dlagv2 applies to a 2 × 2 pencil: [csl snl; −snl csl] (A, B) [csr −snr; snr csr]. More...
 
struct  plane_rotation
 A plane rotation [c s; −s̄ c] and what it leaves of its first argument. More...
 
struct  plane_rotation_real
 A plane rotation [c s; −s c] and what it leaves of its first argument, as LAPACK's dlartg makes it. More...
 
struct  pool_job
 One parallel_for handed to the pool: its chunks, the next one to take, and how many are done. Each run has its own, so a worker still leaving one run never touches the next. More...
 
class  qd_array
 The qd array Z of 4n entries, read one-based: Z(i) is z[i − 1]. More...
 
struct  qr_reflections
 The reflections of a Householder QR, Q left as its blocks of reflectors. More...
 
struct  qz_form
 A complex pencil in generalized Schur form being formed: S and T (n × n, row-major) with A = Q S Zᴴ and B = Q T Zᴴ, Q and Z unitary (qt, zt). More...
 
struct  real_components
 A real eigenvector during its back-substitution, the sums of rows against it by dot_product. More...
 
struct  real_eigenvectors
 A real matrix's eigenvectors before their back-transformation: Xᵀ's real and imaginary parts. More...
 
struct  real_fft_plan
 What a transform of n real points (or to them) takes: the complex plan of n/2 points for an even n, of n for an odd one, and for an even n the roots e^{−2πi k/n} for k up to n/2. More...
 
struct  real_operand
 A real vector that rows are summed against by dot_product. More...
 
struct  real_pencil_value
 A generalized eigenvalue (αr + i αi) / β of a real pencil, as LAPACK's dgges gives it. More...
 
struct  real_qz_form
 A real pencil in generalized real Schur form being formed: S and T (n × n, row-major) with A = Q S Zᵀ and B = Q T Zᵀ, Q and Z orthogonal (qt, zt). More...
 
struct  reflector_block
 The reflectors of one panel of the blocked QR, I − V T Vᴴ as LAPACK's dlarft forms it. More...
 
struct  riccati_operands
 The operands of an algebraic Riccati equation in the arithmetic of T, e and s absent when null. More...
 
struct  rotation_run
 A run of recorded rotations in which each shares a row with the next, as a sweep of the QL or QR iteration gives them: rows (i, i+1) for i going down (down), or going up. More...
 
struct  row_operation
 One operation of tridiagonal_ql or bidiagonal_qr on the rows of a vector matrix, recorded to be applied later. More...
 
struct  scaled_quotient
 A quotient of divided and the factor its dividend was scaled by: a type of its own, where a std::pair of them made GCC on arm64 print, at every caller, a note on its ABI change of GCC 10.1. More...
 
struct  schur_target
 Where a QR sweep applies its reflections: the rows and columns it updates, and Z. More...
 
struct  secular_solution
 A root of a secular equation: the pole it was sought from, and its offset from that pole. More...
 
class  shifted_power_norms
 ‖(T − I)ᵖ‖₁^(1/p) of the upper triangular n × n t, p from 2, exactly (SciPy estimates them, onenormest): each power the one before times T − I, formed once whatever order the norms are asked in. More...
 
struct  shifted_tridiagonal
 The factors of T − λ I for a symmetric tridiagonal T, as LAPACK's dlagtf leaves them: P (T − λ I) = L U, U of diagonal a and two superdiagonals b and d, L's multipliers in c, the row interchanges in swapped. More...
 
struct  sine_terms
 The terms table_sine adds for sin x = a + b·r + …: a = S and b = C for the sine, a = C and b = −S for the cosine (S, C the table's sin(jπ/32) and cos(jπ/32), each a head and a tail), and the value where |x| < 2⁻²⁷: x for the sine, 1 for the cosine. More...
 
struct  small_radix_constants
 The real constants of the radix-3, -5 and -8 butterflies, rounded once from their double-double values. More...
 
struct  small_square
 A square of order 4 at most, row-major in a fixed stride: the blocks dtgex2 exchanges and its transforms. More...
 
struct  split_components
 A real matrix's complex eigenvector during its back-substitution, its real and imaginary parts apart: a row of the real Schur form against it is two dot_product. More...
 
struct  split_operands
 The real operands of gemm_split, kept by each thread from one product to the next (split_storage): mapping their pages anew took a third to a half of a complex product of order 64 to 256 on one x86-64 core (six AVX2 cores, all threads: order 128 0.52 → 0.17 ms), 15–20 % of one of order 256 on arm64. More...
 
struct  standard_block
 The rotation standardize_block found, and the eigenvalues of the block it standardised. More...
 
struct  symmetric_two_by_two
 The eigenvalues and the rotation of a symmetric 2 × 2 [a b; b c], as LAPACK's dlaev2. More...
 
class  tagged
 A standard exception Std that numerical::error tags too, its one what answering for both bases. More...
 
struct  taylor_bound
 Al-Mohy and Higham's θ_m (2011, table 3.1, double precision): the 1-norm of t A up to which m terms of the Taylor series of e^{tA} B, taken s times, stay within the unit roundoff, for the m the selection tries (SciPy's, from 1 to 30, then by fives to 55). More...
 
struct  taylor_plan
 The terms m and the steps s of expm_multiply. More...
 
struct  thin_qr
 A thin QR factorisation, row-major: q is rows × cols with orthonormal columns, r is cols × cols upper triangular with a real, non-negative diagonal. More...
 
class  thread_pool
 The threads every kernel shares, started once: a parallel_for costs a wake-up, not a thread start. More...
 
struct  thread_release_steps
 The calling thread's steps that empty the workspaces it keeps between calls, each registered by the workspace when the thread first makes it (on_release), and their count. More...
 
struct  through_doubles_piece
 A *_into kernel of doubles as a kernel of floats: each piece of 512 converted to doubles on the stack, the kernel of them, each rounded back once (through_doubles's, piece by piece). More...
 
struct  triangle_svd
 The singular value decomposition of the upper triangular [f g; 0 h] (svd_of_triangle). More...
 
struct  tridiagonal_reduction
 The unitary M = Q Dᴴ of a reduction to real tridiagonal form, A = M T Mᴴ, kept as its factors. More...
 
class  unit_roots
 The roots e^{−2πi k/n} of one n, each as unit_root rounds it, from two tables of about √n angles instead of a series a root: the angle's share within its octant (reduce_to_octant) split as a + b·m, m the first table's length, and its cosine and sine those of the sum of the two tables' angles in double-double arithmetic, then placed by the octant and rounded once. The share 0 is the product of two exact (1, 0), so the exact roots stay exact, and a root and its mirror image share their share, so they stay each other's conjugates to the bit. More...
 
struct  vector_tile_kernel
 The vector path's tile: gemm_mr × gemm_nr elements of C in registers (gemm_kernel), each product rounded before its sum unless the compiler contracts c + a·b into a fused multiply-add. More...
 
struct  worker_bell
 A worker's own wake-up: its core, whether it sleeps, and what wakes it. More...
 
struct  workspace_buffer
 One buffer of merge_workspace: its storage and how many elements it holds. More...
 

Concepts

concept  fft_input
 The types numerical, :fft transforms: float, double and their complex numbers.
 
concept  split_complex
 A complex number of float or double, which gemm_split multiplies through the real kernel and the kernels take apart into its real and imaginary parts.
 
concept  element_product
 The element types each_product takes: those each_rounded rounds as a product of depth one.
 
concept  radix_orderable
 The types whose indices order_run sorts by the elements' bits.
 

Typedefs

using kept_page_blocks = kept_blocks< std::size_t {4096}, 4, kept_pages_largest >
 The blocks of 4 KiB pages of default_init_allocator, from kept_pages_least bytes: four, each under kept_pages_largest.
 
using qd_index = std::ptrdiff_t
 An index into the qd array, one-based and signed as LAPACK's.
 
using lapack_index = std::ptrdiff_t
 A signed index from 1, as LAPACK's: the multishift sweep starts its bulges in phantom columns above the matrix, before its first row.
 
using eigenvalue_selection = std::function< bool(const std::complex< double > &)>
 A choice among eigenvalues, as scipy.linalg.schur's sort makes it: true where one is to lead.
 
template<typename T >
using magnitude_t = decltype(std::abs(std::declval< const T & >()))
 The type std::abs gives for a T: the real type of a complex one, T itself otherwise.
 
template<typename T >
using carried_t = std::conditional_t< std::is_same_v< magnitude_t< T >, long double >, long double, double >
 The real type a T's norms, reflector lengths and pivot magnitudes are carried in: long double for a long double scalar, real or complex, whose precision a double would cut to its own; double otherwise.
 
template<typename T >
using value_range_t = std::conditional_t< std::is_same_v< T, float >||std::is_same_v< T, std::complex< float > >, float, carried_t< T > >
 The real type whose range a T's values take: float for a single-precision scalar, real or complex; carried_t otherwise.
 
using symmetric_row_kernel = double(*)(const double *, const double *, double, double *, std::size_t)
 A kernel of symmetric_row's form: symmetric_row itself, or an x86-64 one.
 
using symmetric_rows_kernel = std::array< double, symmetric_row_group >(*)(const double *, std::size_t, const double *, double *, std::size_t)
 A kernel of symmetric_rows' form: symmetric_rows itself, or an x86-64 one.
 
template<typename C >
using fft_real_t = typename C::value_type
 The part type of a complex number (R), or of a register of lines' (fft_pack).
 
template<typename C >
using fft_root_t = std::complex< fft_real_t< C > >
 The complex number of C's part type: a pass' factors.
 
template<typename T >
using gemm_tile = std::array< T, gemm_mr *gemm_nr< T > >
 
template<term_rounding Mode>
using rounding_is = std::integral_constant< term_rounding, Mode >
 The rounding mode Mode as a type, which picks rounded_term's overload.
 
using riccati_terms = riccati_operands< std::complex< double > >
 The operands of an algebraic Riccati equation as complex matrices (the residuals and Newton's step).
 
using chunk_runner = void(*)(const void *body, std::size_t begin, std::size_t end)
 A chunk of a parallel_for, type-erased: runs body(i) for i in [begin, end).
 
using side_runner = void(*)(const void *side)
 A task the caller of a run does before it takes chunks (parallel_for_beside), type-erased.
 
using doubles_into = void(*)(const double *, double *, std::size_t)
 A function of count contiguous doubles into as many (the *_into kernels).
 
template<typename T >
using order_key_t = std::conditional_t< sizeof(T)==8, std::uint64_t, std::uint32_t >
 The unsigned integer of T's width, whose order order_key maps T's onto.
 
template<typename U >
using byte_counts = std::array< std::array< std::size_t, 256 >, sizeof(U)>
 How many keys of U hold each value of each byte: the byte at bit 8·b counted in [b].
 

Enumerations

enum class  trig_family : unsigned char { cosine , sine }
 The family of a real trigonometric transform: dct's or dst's. More...
 
enum class  term_rounding : unsigned char { compiler , fused , x86_fused }
 How numerical::detail::gemm_rows rounds each term sum + a·b, as the tile of the path in use does: as the compiler rounds it (the vector tile's way), one fused multiply-add (arm64's NEON and AMX tiles), or one fused multiply-add of AVX2's unit (the x86-64 tiles), in code built for any x86-64 processor. More...
 
enum class  c_start : bool { kept , zero }
 What a product finds in C: the sums it adds to (C += A·B), or nothing it reads (C = A·B). Then the blocks by rows start their sums at zero and the packed tasks zero their block of C before its first panel, on the thread that computes it, where the caller's zeroing had the threads take C's lines from its cache (six AVX2 cores, all threads: order 64 7.1 → 5.2 µs, 96 12.7 → 9.4, 128 32.7 → 28.8; one: 32 1.62 → 1.47). More...
 
enum class  complex_term : std::uint8_t { compiler , tiles , split }
 How a complex product rounds a term that sums nothing (0 + a·b) on the path in use, as gemm_large_complex runs it. More...
 
enum class  left_kernel : unsigned char {
  plain , vector , neon , amx ,
  avx512 , avx2
}
 The tile a product's left operand is packed for (numerical::detail::pack_left): plain where it is not packed. More...
 

Functions

bool amx_usable () noexcept
 Whether this chip's AMX can be used: known family and a passing self-test, decided once.
 
template<typename T >
void band_lu (band_storage< T > &ab, std::size_t n, std::size_t kl, std::size_t ku, std::vector< std::size_t > &pivots)
 The LU factorisation with partial pivoting of the n × n band of kl subdiagonals and ku superdiagonals in ab (2 kl + ku + 1 rows, the band from row kl + 1, the rows above for the fill the interchanges bring), in place, as LAPACK's gbtf2: each column's pivot the entry of largest pivot_magnitude (|Re| + |Im| for complex numbers, as izamax; the first of them) within the band below the diagonal, its row swapped to the reach of the rows it touches, the multipliers below and the rank-1 update right of it. pivots (from 1) the rows swapped.
 
template<typename T >
void band_lu_solve (band_storage< T > &ab, std::size_t n, std::size_t kl, std::size_t ku, const std::vector< std::size_t > &pivots, buffer< T > &b, std::size_t k)
 The k right-hand sides of b (n × k, row-major) solved in place against band_lu's factors, as LAPACK's gbtrs (no transpose): the interchanges and L column by column, then U, a band of kl + ku superdiagonals, by back substitution.
 
template<typename T >
void band_lu_solve_adjoint (band_storage< T > &ab, std::size_t n, std::size_t kl, std::size_t ku, const std::vector< std::size_t > &pivots, T *x)
 One right-hand side x (n entries) solved in place against the adjoint of band_lu's factors, Aᴴ x = b, as LAPACK's gbtrs (conjugate transpose): Uᴴ, a band of kl + ku subdiagonals, by forward substitution, then Lᴴ and the interchanges column by column from the last.
 
template<typename T >
void band_cholesky (band_storage< T > &ab, std::size_t n, std::size_t kd, const std::string &name)
 The Cholesky factor U (A = Uᴴ U) of the Hermitian positive-definite band of kd superdiagonals in ab (kd + 1 rows, AB(kd + 1 + i − j, j) = A(i, j)), in place, as LAPACK's pbtf2 (upper): each diagonal entry's square root, the row right of it scaled, the trailing band less its outer product.
 
template<typename T >
void band_cholesky_solve (band_storage< T > &ab, std::size_t n, std::size_t kd, buffer< T > &b, std::size_t k)
 The k right-hand sides of b (n × k, row-major) solved in place against band_cholesky's U, as LAPACK's pbtrs: Uᴴ y = b forward, then U x = y back.
 
template<typename T >
void tridiagonal_solve (buffer< T > &dl, buffer< T > &d, buffer< T > &du, buffer< T > &b, std::size_t n, std::size_t k)
 The k right-hand sides of b (n × k, row-major) solved in place against the tridiagonal matrix of subdiagonal dl, diagonal d and superdiagonal du (n − 1, n, n − 1 entries, overwritten), as LAPACK's gtsv, which scipy.linalg.solve_banded calls for one sub- and one superdiagonal: Gaussian elimination with partial pivoting, row i + 1 exchanged with row i where its subdiagonal entry is the larger (dl then holding the second superdiagonal the exchange brings), then back substitution.
 
double positive_pivot_step (double d, double &e, double next)
 pttrf's step for a real tridiagonal: e ← e / d,
 
double positive_pivot_step (double d, std::complex< double > &e, double next)
 zpttrf's step for a complex one: e ← e / d by parts,
 
void positive_back_substitution (const buffer< double > &d, const buffer< double > &e, buffer< double > &b, std::size_t n, std::size_t k)
 dptts2's last two passes, after L: D and Lᵀ in one, b ← b / d − b_next e from the end.
 
void positive_back_substitution (const buffer< double > &d, const buffer< std::complex< double > > &e, buffer< std::complex< double > > &b, std::size_t n, std::size_t k)
 zptts2's (lower): D, then Lᴴ, b ← b − b_next conj(e) from the end.
 
template<typename T >
void positive_tridiagonal_solve (buffer< double > &d, buffer< T > &e, buffer< T > &b, std::size_t n, std::size_t k)
 The k right-hand sides of b (n × k, row-major) solved in place against the Hermitian positive definite tridiagonal matrix of real diagonal d and subdiagonal e (n and n − 1 entries, overwritten), as LAPACK's ptsv, which scipy.linalg.solveh_banded calls for one off-diagonal: A = L D Lᴴ (pttrf, e becoming L's subdiagonal), then L, D and Lᴴ in turn (ptts2).
 
double taken_as (const std::complex< double > &z, std::type_identity< double >)
 A complex value as a real result takes it: its real part (the imaginary part rounding's alone).
 
std::complex< double > taken_as (const std::complex< double > &z, std::type_identity< std::complex< double > >)
 A complex value as a complex result takes it: whole.
 
template<typename T >
void require_band (const matrix< T > &ab, std::size_t rows, const matrix< T > &b, const std::string &name)
 Refuses a band that is not rows × n and right-hand sides not of n rows.
 
template<typename T >
band_storage< T > upper_band (const matrix< T > &ab, bool lower_form)
 The upper band of kd superdiagonals of the band ab in either of scipy.linalg's forms (solveh_banded), AB(kd + 1 + i − j, j) = A(i, j) for i ≤ j, as band_cholesky takes it.
 
thread_release_steps & thread_releases () noexcept
 The calling thread's thread_release_steps.
 
bool on_release (void(*step)()) noexcept
 Registers step among the calling thread's thread_releases, while a place is left (a workspace past them is not freed by release_thread_memory).
 
void release_thread_memory ()
 Empties every workspace the calling thread keeps between calls: its thread_releases, in turn.
 
template<typename T >
bool operator== (const std::vector< T, default_init_allocator< T > > &buffer, const std::vector< T > &other)
 Element-wise equality of a numerical::buffer and a std::vector, found by argument-dependent lookup through the allocator: code that compared a tensor's elements with a std::vector keeps doing so.
 
void sine_and_cosine (double x, double &sine, double &cosine)
 sin and cos of x together, as glibc's complex functions take them: its sincos under glibc (whose single-precision one strayed from sinf and cosf by a unit in the last place before 2.29), the system's __sincos on macOS (a compiler may fuse a sin and a cos into it where it inlines them, and it strays from them by a unit: the same call in every context), std::sin and std::cos elsewhere.
 
void sine_and_cosine (float x, float &sine, float &cosine)
 sine_and_cosine of a float.
 
template<std::floating_point R>
R separate_sine (R x)
 std::sin of x in a call of its own, as NumPy's formulas take it: no build fuses it with a cos of the same argument into a sincos, which strays from them by a unit in the last place (glibc's sincosf before 2.29, macOS's __sincos).
 
template<std::floating_point R>
R separate_cosine (R x)
 std::cos of x in a call of its own, as separate_sine.
 
bool libm_fuses () noexcept
 Whether the C library's complex functions fuse their sums of products, a·b + c·d as fma(a, b, c·d), off arm64 (where FMA is in every build's baseline, and they do): under glibc as its build does, read once from its ctan of a number whose cos² + sinh² the two round apart. glibc 2.43 fuses there on an AMD EPYC 7763 (GitHub's Ubuntu 26.04 runners) and not on an Intel Core i5-8500, the same library; NumPy's complex functions are glibc's. Elsewhere no.
 
template<std::floating_point R>
R libm_sum_of_products (R a, R b, R c, R d)
 a·b + c·d as the C library's build computes it: fma(a, b, c·d) where it fuses (libm_fuses), each product rounded elsewhere.
 
template<std::floating_point R>
R libm_add_product (R s, R a, R b)
 s + a·b as the C library's build computes s += a·b: fused where it fuses, as libm_sum_of_products.
 
template<std::floating_point R>
void small_sincos (R x, R &sine, R &cosine)
 sin and cos of x (sine_and_cosine), or x and 1 for one under the smallest normal value (an exact sine there).
 
template<std::floating_point R>
std::complex< R > complex_exp (const std::complex< R > &z)
 exp(z): exp(re) times cos(im) + i sin(im), a real part past exp_bound taken off in steps of it (each a factor exp(bound) on the sine and the cosine), so that a finite product does not overflow early.
 
template<std::floating_point R>
std::complex< R > complex_cosh (const std::complex< R > &z)
 cosh(z): cosh(re) cos(im) + i sinh(re) sin(im); past exp_bound, exp(|re|) / 2 times the cosine and the sine, taken off in steps as complex_exp does.
 
template<std::floating_point R>
std::complex< R > complex_sinh (const std::complex< R > &z)
 sinh(z): sinh(re) cos(im) + i cosh(re) sin(im) of |re|, the sign of re carried by the cosine; past exp_bound as complex_cosh.
 
template<std::floating_point R>
std::complex< R > complex_cos (const std::complex< R > &z)
 cos(z), as cosh(i z): complex_cosh of −im + i re.
 
template<std::floating_point R>
std::complex< R > complex_sin (const std::complex< R > &z)
 sin(z): sin(re) cosh(im) + i cos(re) sinh(im) of |re|, the sign of re carried by the sine; an imaginary part past exp_bound as complex_sinh's real part.
 
template<std::floating_point R>
std::complex< R > complex_tanh (const std::complex< R > &z)
 tanh(z): (sinh(re) cosh(re) + i sin(im) cos(im)) / (sinh(re)² + cos(im)²); past half_exp_bound the real part is ±1 and the imaginary one 4 sin(im) cos(im) / exp(2|re|), divided in steps.
 
template<std::floating_point R>
std::complex< R > complex_tan (const std::complex< R > &z)
 tan(z): (sin(re) cos(re) + i sinh(im) cosh(im)) / (cos(re)² + sinh(im)²); past half_exp_bound the imaginary part is ±1 and the real one 4 sin(re) cos(re) / exp(2|im|), divided in steps.
 
double unit_circle_offset (double x, double y)
 x² + y² − 1 for 0.5 ≤ x < 1, y ≤ x and x² + y² ≥ 0.5, almost exactly: each square split into its rounded value and its error (a fused multiply-add), the five terms summed by magnitude, each pair's error carried to the next.
 
float unit_circle_offset (float x, float y)
 unit_circle_offset of floats, in double: (x − 1)(x + 1) + y².
 
template<std::floating_point R>
bool unit_circle_squares_reach_half (R x, R y)
 Whether x² + y² ≥ 0.5, as glibc's build computes the sum (libm_sum_of_products).
 
template<std::floating_point R>
std::complex< R > complex_log (const std::complex< R > &z)
 log(z): log|z| + i arg(z), log|z| as log1p(|z|² − 1) / 2 near the unit circle (|z|² − 1 exact or almost, unit_circle_offset) and as log(hypot) elsewhere, the larger part scaled by 2 near overflow and both by 2^digits under the smallest normal value.
 
template<std::floating_point R>
std::complex< R > complex_sqrt (const std::complex< R > &z)
 The square root of a complex number, as the GNU C library's csqrt computes it (NumPy's on Linux, to the bit): the infinities and NaNs of C's Annex G; a real or an imaginary argument apart (the root of -1 is i exactly, that of y i has equal parts); otherwise sqrt((|z| ± re) / 2) for the larger part and the imaginary part over twice it for the other, scaled by 4 near overflow and by a power of 2 among subnormal parts. std::sqrt of libc++ goes through the polar form: sqrt(-1e20 + 1j) in single precision had a real part of 755 for 5e-11.
 
template<std::floating_point R>
std::complex< R > complex_atanh (const std::complex< R > &z)
 atanh(z), as glibc's catanh: C's Annex G for the infinities and NaNs; π/2 and the reciprocal past 16/ε; else the real part log(|1 + z|² / |1 − z|²) / 4 (by log1p(4x / |1 − z|²) where the quotient is at least ½), the imaginary part atan2(2y, 1 − |z|²) / 2, 1 − |z|² from unit_circle_offset where it cancels.
 
template<std::floating_point R>
std::complex< R > kernel_asinh (const std::complex< R > &z, bool adjust)
 asinh(z) of a finite nonzero z, as glibc's __kernel_casinh, its imaginary part taken from π/2 where adjust (acos and acosh through it): the first quadrant's |re| and |im|, by the region they lie in, log(2z) past 1/ε, log(z + √(1 + z²)) by log1p and atan2 of parts that do not cancel nearer 1, else that sum by complex_sqrt and complex_log; the signs of z's parts given back last. Each product whose every use is a sum fused where the C library's build fuses (libm_add_product, libm_sum_of_products): GCC's contraction of glibc on arm64.
 
template<std::floating_point R>
std::complex< R > complex_asinh (const std::complex< R > &z)
 asinh(z), as glibc's casinh: C's Annex G for the infinities and NaNs, z itself at ±0 ± 0i, else kernel_asinh.
 
template<std::floating_point R>
std::complex< R > complex_asin (const std::complex< R > &z)
 asin(z), as glibc's casin: −i asinh(iz); a NaN part as Annex G has it.
 
template<std::floating_point R>
std::complex< R > complex_acos (const std::complex< R > &z)
 acos(z), as glibc's cacos: π/2 − asin(z) for the infinities, NaNs and zeros, else kernel_asinh of iz adjusted.
 
template<std::floating_point R>
std::complex< R > complex_acosh (const std::complex< R > &z)
 acosh(z), as glibc's cacosh: Annex G for the infinities, NaNs and zeros, else kernel_asinh of iz adjusted, its sign by z's imaginary part's.
 
template<std::floating_point R>
std::complex< R > complex_atan (const std::complex< R > &z)
 atan(z), as glibc's catan: C's Annex G for the infinities and NaNs; π/2 and the reciprocal past 16/ε; else the real part atan2(2x, 1 − |z|²) / 2, 1 − |z|² from unit_circle_offset where it cancels, the imaginary part log(|z + i|² / |z − i|²) / 4 (by log1p(4y / |z − i|²) where the quotient is at least ½): complex_atanh's turned a quarter, each operation in glibc's order.
 
template<std::floating_point R>
std::complex< R > complex_log2 (const std::complex< R > &z)
 log2(z), as NumPy's: complex_log times log₂ e, each part.
 
template<std::floating_point R>
std::complex< R > complex_log10 (const std::complex< R > &z)
 log10(z), as NumPy's: complex_log times log₁₀ e, each part.
 
template<std::floating_point R>
std::complex< R > complex_exp2 (const std::complex< R > &z)
 2^z, as NumPy's: complex_exp of z times ln 2, each part.
 
template<std::floating_point R>
std::complex< R > complex_expm1 (const std::complex< R > &z)
 exp(z) − 1, as NumPy's: expm1(re) cos(im) − 2 sin²(im/2) + i exp(re) sin(im), the sine and the cosine of im in calls of their own as NumPy's (separate_sine); past exp_bound, a finite im's complex_exp less 1, where exp(re) overflows and NumPy's product with the sine with it (NaN at +∞ + 0i, an infinite 710.2 + 0.1i).
 
template<std::floating_point R>
std::complex< R > complex_log1p (const std::complex< R > &z)
 log(1 + z): the imaginary part atan2(im, 1 + re), as NumPy's; the real part log|1 + z|, by log1p(d) / 2 of d = |1 + z|² − 1 = 2re + re² + im² where |re| and |im| are at most 3 and d at least −½ (the squares exact by fused multiply-adds, the sum compensated: d keeps its digits where it cancels, about the circle |1 + z| = 1 and about 0, where NumPy's log(hypot(1 + re, im)) loses them all); else complex_log of 1 + z, |1 + z| there far from 1 or 1 + re exact, finite to the largest numbers (NumPy's hypot overflows past max / √2). A zero is its own, its signs kept, as log1p of a real −0.
 
template<std::floating_point R>
std::complex< R > complex_sign (const std::complex< R > &z)
 z / |z|, as NumPy 2's sign of a complex number and the array API's: 0 for 0, NaN for a NaN modulus, and for an infinite modulus ±1 along the infinite part (NaN where both are), 0 across it.
 
void qd_pair (double &a, double b, double &c) noexcept
 The eigenvalues of the 2 × 2 [[a, √b], [√b, c]]-like block of a qd array, a = q₁, b = e₁, c = q₂: the larger into a, the smaller into c, each to high relative accuracy. No change where b is negligible against c, nor where half the sum (a − c) + b underflows to zero.
 
void dqds_shift (qd_index i0, qd_index n0, const qd_array &z, qd_index pp, qd_index n0in, double dmin, dqds_state &state)
 The shift of the next transform of the block i0…n0 (LAPACK's dlasq4): an estimate of the least eigenvalue from the last transform's least ds, by the case state's ttype records. Some cases return with the shift as it was, where a ratio of the array exceeds one.
 
void dqds_transform (qd_index i0, qd_index n0, const qd_array &z, qd_index pp, double &tau, double sigma, double &dmin, dqds_state &state)
 One dqds transform of the block i0…n0 (at least three values) with the shift tau, from the half pp of the array into the other (LAPACK's dlasq5, its IEEE form): a zero shift where it is under half of ε (σ + τ), the ds it takes below that bound set to zero. dmin is NaN where the last d is not finite: a d + e of zero, or past overflow, makes every d after it NaN, which a least of them alone does not show.
 
void dqd_transform (qd_index i0, qd_index n0, const qd_array &z, qd_index pp, double &dmin, dqds_state &state)
 One dqd transform (no shift) of the block i0…n0 (at least three values), from the half pp of the array into the other, each quotient guarded against underflow and overflow (LAPACK's dlasq6): the transform a shifted one falls back to when its numbers are not finite.
 
void qd_flip (qd_index i0, qd_index n0, const qd_array &z) noexcept
 Turns the block i0…n0 of the array end for end, both halves (LAPACK's flip in dlasq2 and dlasq3).
 
void dqds_step (qd_index i0, qd_index &n0, const qd_array &z, qd_index &pp, double &dmin, double &sigma, double &desig, double &qmax, dqds_state &state)
 One step on the block i0…n0 (LAPACK's dlasq3): deflates the values its end has converged to (into Z(4k − 3), sigma added back), flips the block where its far end is the smaller, then one transform with the shift of dqds_shift. A shift that makes a d negative is lowered and the transform retried; one whose numbers are not finite is dropped, the transform then unshifted, guarded as dqd_transform.
 
bool dqds_squares (buffer< double > &z, qd_index n, qd_index sweeps)
 The eigenvalues of the positive semi-definite tridiagonal whose qd array is in z (LAPACK's dlasq2): q₁ e₁ q₂ e₂ … qₙ, non-negative, in its first 2n − 1 entries, n at least two.
 
bool dqds_singular_values (buffer< double > &d, const buffer< double > &e, std::size_t n, qd_index sweeps=dqds_sweeps)
 The singular values of the real upper bidiagonal (d, e) of order n by dqds (LAPACK's dlasq1), descending into d: its entries' magnitudes scaled so that the largest is √(ε / DBL_MIN), their squares' eigenvalues found by dqds_squares, the square roots scaled back.
 
double modulus_1 (double x)
 |x| of a real x: what modulus_1 of a complex one stands for.
 
double modulus_1 (const std::complex< double > &x)
 |re x| + |im x|, the cheap modulus LAPACK's complex routines judge with.
 
template<typename A , typename B >
auto times (const A &a, const B &b)
 a · b.
 
std::complex< double > times (const std::complex< double > &a, const std::complex< double > &b)
 a · b of two complex numbers by the schoolbook formula, without C Annex G's recovery of infinities, which a large translation unit leaves out of line (a call per product: complex eig of order 64 took 2.7 times as long in the Python binding as alone). The entries here are finite.
 
double largest_part (double x)
 |x| of a real x.
 
double largest_part (const std::complex< double > &x)
 max(|re x|, |im x|): within √2 of |x|, without its square root.
 
template<typename T >
void exchange (buffer< T > &a, std::size_t n, std::size_t i, std::size_t j)
 Exchanges rows i and j, then columns i and j, of the row-major n × n a: a similarity.
 
template<typename T >
balancing balance (buffer< T > &a, std::size_t n, bool permute=true, bool scale=true)
 Balances the row-major n × n a in place, as LAPACK's ‘dgebal('B’): rows with no off-diagonal entry in the active block go to its bottom and columns with none to its top, each isolating an eigenvalue on the diagonal; then each row and column of the remaining block is scaled by a power of two until their 2-norms are within a factor of 2, which leaves every entry exact. Without \p permute no row moves (the block is the whole matrix), without \p scale none is scaled:dgebal's'S'and'P'`.
 
template<typename T >
balancing isolate (buffer< T > &a, std::size_t n)
 Isolates eigenvalues of the row-major n × n a in place by permutations alone, as LAPACK's ‘dgebal('P’)`: balance without its scaling, a similarity by an orthogonal matrix.
 
template<typename T >
buffer< T > hessenberg_unblocked (buffer< T > &a, std::size_t n, std::size_t low, std::size_t high, bool form)
 Reduces the active block low … high − 1 of the balanced row-major n × n a to upper Hessenberg form by Householder reflections one column at a time, as LAPACK's dgehd2: A = Q H Qᴴ, the entries below the subdiagonal set to zero.
 
bool already_reduced (double, double rest)
 Whether (α, x), of tail norm rest, is already (β, 0) with β real: for a real one, its tail zero.
 
bool already_reduced (const std::complex< double > &alpha, double rest)
 The same for a complex one: its tail zero and α real.
 
template<typename T >
T reflector_tau (T &alpha, T *x, std::size_t count)
 The Householder reflection H = I − τ v vᴴ, v = (1, v₁…), with Hᴴ (α, x) = (β, 0) and β real, as LAPACK's zlarfg (dlarfg for a real T): alpha becomes β and the count entries at x become v₁….
 
real_operand operand_of (const double *v, std::size_t stride, std::size_t count)
 The count entries stride apart at v as a real_operand.
 
complex_operand operand_of (const std::complex< double > *v, std::size_t stride, std::size_t count)
 The same as a complex_operand.
 
template<typename T >
buffer< T > hessenberg_blocked (buffer< T > &a, std::size_t n, std::size_t low, std::size_t high, bool form)
 hessenberg_unblocked by panels of hessenberg_block columns, as LAPACK's dgehrd and dlahr2: a panel's reflectors are formed one column at a time, each column first updated by those before it through Y = A V T; the rest of the matrix then takes the panel from both sides through the matrix product kernel, A ← (I − V Tᴴ Vᴴ)(A − Y Vᴴ). Its arguments and answer are hessenberg_unblocked's.
 
template<typename T >
buffer< T > hessenberg (buffer< T > &a, std::size_t n, std::size_t low, std::size_t high, bool form)
 hessenberg_blocked from an active order of hessenberg_crossover, else hessenberg_unblocked.
 
double small_reflector (double(&v)[3], std::size_t count)
 The Householder reflection I − τ v vᵀ, v = (1, x…), that takes (α, x…) to (β, 0…), as LAPACK's dlarfg for two or three entries: on return v[0] = β and v[1…] holds x's part of v.
 
std::complex< double > complex_reflector (std::complex< double >(&v)[2])
 The Householder reflection I − τ v vᴴ, v = (1, x), whose adjoint takes (α, x) to (β, 0), β real, as LAPACK's zlarfg for two entries: on return v[0] = β and v[1] holds x's part of v.
 
standard_block standardize_block (double &a, double &b, double &c, double &d)
 The Schur factorisation of the real 2 × 2 block [a b; c d], as LAPACK's dlanv2: [a b; c d] ← [cs −sn; sn cs]ᵀ [a b; c d] [cs −sn; sn cs], upper triangular where its eigenvalues are real, else with equal diagonal entries and b c < 0.
 
template<typename T >
void rotate (T *x, T *y, std::size_t count, std::size_t stride, double cs, double sn)
 (x, y) ← (cs x + sn y, cs y − sn x) over count pairs stride apart: a plane rotation.
 
template<typename T >
void reflect_three_rows (T *r1, T *r2, T *r3, std::size_t from, std::size_t to, const T &tau, const T &v2, const T &v3, bool adjoint)
 A reflection I − τ v vᴴ, v = (1, v₂, v₃), on the entries from … to − 1 of three rows: its adjoint from the left with adjoint (rows of H), else from the right (rows of Zᵀ, which are columns of Z). A reflection of two passes r3 = r2 and v3 = 0. One loop over the entries, which vectorises.
 
template<typename T >
void reflect_two_rows (T *r1, T *r2, std::size_t from, std::size_t to, const T &tau, const T &v2, bool adjoint)
 reflect_three_rows for a reflection of two entries, v = (1, v₂), on two rows: its own loop, which vectorises where the third row aliasing the second would not.
 
template<typename T >
void reflect_three_columns (T *first, std::size_t rows, std::size_t ld, const T &tau, const T &v2, const T &v3)
 A reflection I − τ v vᴴ, v = (1, v₂, v₃), from the right on rows rows (ld apart) of the three columns at first: x ← x − τ (x₁ + v₂ x₂ + v₃ x₃) (1, v̄₂, v̄₃) on each. The rows are short and apart: one call for them all.
 
template<typename T >
void reflect_two_columns (T *first, std::size_t rows, std::size_t ld, const T &tau, const T &v2)
 The same for a reflection of two entries, v = (1, v₂), on two columns.
 
void real_schur (const schur_target< double > &t, buffer< double > &re, buffer< double > &im, std::size_t steps)
 The real Schur form of the upper Hessenberg t's active block, as LAPACK's dlahqr: Francis's implicit double-shift QR, with the deflation criterion of Ahues and Tisseur and exceptional shifts after 10 and 20 iterations without deflation. The eigenvalues are left in re and im, a complex pair's in two consecutive places, the positive imaginary part first.
 
void complex_schur (const schur_target< std::complex< double > > &t, buffer< std::complex< double > > &values, std::size_t steps)
 The complex Schur form of the upper Hessenberg t's active block, as LAPACK's zlahqr: the shifted QR algorithm with Wilkinson's shift, the deflation criterion of Ahues and Tisseur, and exceptional shifts after 10 and 20 iterations without deflation. The eigenvalues are left in values.
 
std::size_t count_of (lapack_index count)
 A count as an unsigned size (never negative where it is taken).
 
constexpr lapack_index multishift_from (double)
 The order from which a real Schur form is taken by multishift_schur, below which real_schur is faster (eight arm64 cores: order 128 1.85 against 2.60 ms; 224, 8.9 against 8.6; with the vectors, 192, 8.1 against 7.1).
 
constexpr lapack_index multishift_from (const std::complex< double > &)
 The same for a complex one (order 76 1.8 against 2.1 ms, 128 7.5 against 6.5; with the vectors, 100, 6.4 against 5.6).
 
plane_rotation< double > givens (double f, double g)
 The rotation that takes (f, g) to (r, 0), as LAPACK's dlartg: r of f's sign.
 
plane_rotation< std::complex< double > > givens (const std::complex< double > &f, const std::complex< double > &g)
 The rotation that takes (f, g) to (r, 0), as LAPACK's zlartg: c real, r of f's phase.
 
template<typename T >
void rotate_pair (T *x, T *y, std::size_t count, std::size_t stride, double c, const T &s)
 (x, y) ← (c x + s y, c y − s̄ x) over count pairs stride apart.
 
template<typename T >
void exchange_diagonal (one_based< T > t, one_based< T > qt, lapack_index n, lapack_index k)
 Exchanges the diagonal entries k and k + 1 of the upper triangular (or quasi-triangular, both 1 × 1 there) t by a plane rotation, as LAPACK's ztrexc step and dlaexc for two 1 × 1 blocks; the rows of qt (Qᵀ) turn with it.
 
void reflect_rows_3 (const std::array< double, 3 > &u, double tau, double *t, std::size_t ld, std::size_t count)
 H = I − τ u uᵀ of three entries applied from the left to the 3 × count rows at t (row stride ld).
 
void reflect_columns_3 (const std::array< double, 3 > &u, double tau, double *t, std::size_t ld, std::size_t count)
 The same from the right, to the count × 3 columns at t.
 
void small_sylvester (const one_based< double > &tl, const one_based< double > &tr, const one_based< double > &b, lapack_index n1, lapack_index n2, double(&x)[4])
 The solution X (n1 × n2, row-major in x) of TL X − X TR = B, n1 and n2 at most 2, by Gaussian elimination with complete pivoting on its Kronecker form, a pivot below ε times the largest coefficient taken as that, as LAPACK's dlasy2 (the blocks' entries near 1: no scaling against overflow).
 
void standardize_at (one_based< double > t, one_based< double > qt, lapack_index n, lapack_index j)
 The standard form of the 2 × 2 block at j of the quasi-triangular t, its rotation carried to the rest of T and to the rows of qt, as dlaexc ends.
 
bool exchange_blocks (one_based< double > t, one_based< double > qt, lapack_index n, lapack_index j1, lapack_index n1, lapack_index n2)
 Exchanges the adjacent diagonal blocks of the real quasi-triangular t at j1, of orders n1 then n2, by an orthogonal similarity carried to the rows of qt, as LAPACK's dlaexc.
 
lapack_index block_order (one_based< double > t, lapack_index n, lapack_index j)
 The order of the diagonal block of the quasi-triangular t that starts at j: 2 where a subdiagonal entry below it is nonzero.
 
bool reorder (one_based< double > t, one_based< double > qt, lapack_index n, lapack_index ifst, lapack_index &ilst)
 Moves the diagonal block of the real quasi-triangular t at ifst to ilst by exchanges of adjacent blocks, the rows of qt (Qᵀ) turning with it, as LAPACK's dtrexc. ilst becomes where the block ended: short of its goal where an exchange was refused (then.
 
bool reorder (one_based< std::complex< double > > t, one_based< std::complex< double > > qt, lapack_index n, lapack_index ifst, lapack_index &ilst)
 Moves the diagonal entry of the complex upper triangular t at ifst to ilst, as LAPACK's ztrexc.
 
void bulge_column (one_based< double > h, lapack_index count, const std::complex< double > &s1, const std::complex< double > &s2, double *v)
 A multiple of the first column of (H − s₁ I)(H − s₂ I) for the leading 2 × 2 or 3 × 3 (count) of the real Hessenberg h, the shifts a conjugate pair or two reals, as LAPACK's dlaqr1.
 
void bulge_column (one_based< std::complex< double > > h, lapack_index count, const std::complex< double > &s1, const std::complex< double > &s2, std::complex< double > *v)
 The same for a complex h and any two shifts, as LAPACK's zlaqr1.
 
void pair_shifts (std::complex< double > *s, lapack_index count, double)
 Arranges a real matrix's shifts so that each pair is a conjugate pair or two reals, as dlaqr5 begins (conjugate shifts already adjacent).
 
void pair_shifts (std::complex< double > *, lapack_index, const std::complex< double > &)
 A complex matrix's shifts pair as they come.
 
double as_scalar (const std::complex< double > &s, double)
 The shift s as the matrix's scalar: a real matrix's are real where they are used one by one.
 
std::complex< double > as_scalar (const std::complex< double > &s, const std::complex< double > &)
 A complex matrix's shift as it is.
 
template<typename T >
void vigilant_deflation (one_based< T > h, lapack_index k, lapack_index ktop, lapack_index kbot, double smallest)
 H(k + 1, k) set to zero where the deflation criterion of Ahues and Tisseur holds, as the sweep of dlaqr5 tests after each step of a bulge (vigilant deflation).
 
template<typename T >
void reflect_from_left (T *x, std::size_t stride, const T &tau, const T &v2, const T &v3)
 The adjoint I − τ̄ v vᴴ of a reflection of three entries from the left on one column (x three entries stride apart).
 
template<typename T >
void chase_bulges (const schur_target< T > &target, lapack_index ktop, lapack_index kbot, std::complex< double > *s, lapack_index count, bool accumulate)
 One small-bulge multishift QR sweep over the active block ktop … kbot of target's Hessenberg matrix with the count shifts at s, as LAPACK's dlaqr5 and zlaqr5: chains of 3 × 3 bulges, two shifts each, chased down the diagonal by slabs of 4 · count / 2 columns; with accumulate, each slab's reflections are accumulated in U and applied to the rest of the matrix and to Z through the matrix product kernel, else applied to them directly (LAPACK's kacc22, which accumulates from 14 shifts). A subdiagonal entry that becomes negligible on the way is set to zero (vigilant deflation).
 
template<typename T >
void multishift_schur (const schur_target< T > &target, std::complex< double > *w, bool recursive, lapack_index steps)
 The Schur form of target's active block by the multishift QR algorithm with aggressive early deflation, as LAPACK's dlaqr0 and zlaqr0 (dlaqr4 without recursive): each iteration deflates what it can from a window at the bottom of the active block (early_deflation) and, unless that deflated enough, sweeps the block with the window's other eigenvalues as shifts (chase_bulges). The eigenvalues are left in w (one per row; a real matrix's complex pairs adjacent, the positive imaginary part first).
 
void plain_schur (const schur_target< double > &t, std::complex< double > *w)
 The Schur form of t's active block by real_schur, its eigenvalues to w (one per row of t).
 
void plain_schur (const schur_target< std::complex< double > > &t, std::complex< double > *w)
 The same by complex_schur.
 
lapack_index undeflated (one_based< double > t, one_based< double > vt, lapack_index jw, double s, double smallest, std::complex< double > *w)
 The eigenvalues of the real Schur form t (jw × jw) of a deflation window that do not deflate, moved to its top, as dlaqr3: from the bottom, a block whose spike entries s v₁ⱼ are negligible deflates, any other is moved up; the undeflated blocks are then sorted by decreasing modulus (which helps graded matrices), and every eigenvalue of the window written to w.
 
lapack_index undeflated (one_based< std::complex< double > > t, one_based< std::complex< double > > vt, lapack_index jw, const std::complex< double > &s, double smallest, std::complex< double > *w)
 The same for a complex Schur form, as zlaqr3: 1 × 1 blocks only, sorted by selection.
 
template<typename T >
deflation_count early_deflation (const schur_target< T > &target, lapack_index ktop, lapack_index kbot, lapack_index nw, std::complex< double > *w, bool recursive)
 Aggressive early deflation, as LAPACK's dlaqr3 and zlaqr3: the trailing nw × nw window of the active block ktop … kbot is brought to Schur form; its eigenvalues whose coupling to the rest (the spike, s times the first row of the window's Schur vectors) is negligible deflate; the window is reduced back to Hessenberg form above them, and its transformation carried to the rest of H and to Z through the matrix product kernel. Every eigenvalue of the window is written to w: the deflated ones final, the others the next shifts. recursive: the window itself by multishift_schur past its crossover.
 
lapack_index recommended_shifts (lapack_index nh)
 The shifts LAPACK's iparmq recommends for an active block of order nh.
 
lapack_index recommended_window (lapack_index nh, lapack_index ns)
 The deflation window LAPACK's iparmq recommends for an active block of order nh and ns shifts.
 
void exceptional_shifts (one_based< double > h, std::complex< double > *w, lapack_index ks, lapack_index kbot, lapack_index ktop)
 Exceptional shifts for a real matrix, as dlaqr0's after kexsh iterations without deflation: from the subdiagonal entries, as conjugate pairs of the 2 × 2 blocks dlahqr's exceptional shifts build.
 
void exceptional_shifts (one_based< std::complex< double > > h, std::complex< double > *w, lapack_index ks, lapack_index kbot, lapack_index)
 Exceptional shifts for a complex matrix, as zlaqr0's.
 
void pair_for_sweep (std::complex< double > *w, lapack_index ks, lapack_index kbot, double)
 A real matrix's shifts brought into pairs, each a conjugate pair or two reals, as dlaqr0 does before its sweep (conjugate shifts already adjacent).
 
void pair_for_sweep (std::complex< double > *, lapack_index, lapack_index, const std::complex< double > &)
 A complex matrix's shifts need no pairing.
 
bool two_real_shifts (const std::complex< double > *w, lapack_index ks, lapack_index kbot, double)
 Whether a real matrix's sweep is down to two shifts, both real: then the one nearer the corner serves twice.
 
bool two_real_shifts (const std::complex< double > *, lapack_index ks, lapack_index kbot, const std::complex< double > &)
 The same for a complex matrix: two shifts of any kind.
 
void schur_form (const schur_target< double > &t, buffer< double > &re, buffer< double > &im, std::size_t steps)
 The real Schur form of t's active block: real_schur below multishift_from, else multishift_schur, as LAPACK's dhseqr chooses. steps bounds real_schur's iterations.
 
void schur_form (const schur_target< std::complex< double > > &t, buffer< std::complex< double > > &values, std::size_t steps)
 The complex Schur form likewise: complex_schur or multishift_schur.
 
template<typename S >
double solve_block (const S(&m)[4], S(&r)[2], double floor)
 Solves [a b; c d] x = r in place of r by Gaussian elimination with complete pivoting, a pivot below floor taken as floor, as LAPACK's dlaln2; r is scaled first, and.
 
template<typename S , typename P >
scaled_quotient< S > divided (S r, P pivot, double floor)
 r / pivot, a pivot below floor taken as floor, r scaled first where the quotient would pass back_substitution_limit.
 
void normalize_vector (std::complex< double > *v, std::size_t n, bool signed_real)
 Scales v to unit Euclidean norm and turns it so that its first component of largest modulus is real: positive for a complex matrix (zgeev), of its real part's sign for a real one's pair (dgeev).
 
template<typename T , typename X >
void solve_upward (const buffer< T > &t, std::size_t n, std::size_t k, std::size_t top, typename X::scalar lambda, double floor, X &x)
 Solves (T − λ I) x = 0 upward from the component k, rows k − 1 … 0 of the row-major quasi-triangular t (its 2 × 2 blocks where a subdiagonal entry is nonzero), x [k … top] given: as LAPACK's dtrevc and ztrevc, a pivot below floor taken as floor, the components scaled down where they would grow past back_substitution_limit. The sums of rows against x are X 's, vectorised.
 
std::size_t vector_work (std::size_t n) noexcept
 A triangular solve's work for parallel_for over the n eigenvectors: k²/2 products for vector k, n²/6 on average.
 
std::size_t balanced_index (std::size_t task, std::size_t n) noexcept
 The eigenvector task of n solves, the short ones and the long ones in turn (0, n − 1, 1, n − 2, …): each of parallel_for's runs of tasks then as long as another.
 
buffer< std::complex< double > > complex_vectors (const buffer< std::complex< double > > &t, std::size_t n)
 The eigenvectors of the complex upper triangular t (row-major n × n) as the rows of Xᵀ: row k is x with (T − t_kk I) x = 0, x_k = 1, zero past k, as LAPACK's ztrevc.
 
real_eigenvectors real_vectors (const buffer< double > &t, const buffer< double > &re, const buffer< double > &im, std::size_t n)
 The eigenvectors of the real quasi-triangular t (row-major n × n, its 2 × 2 blocks standard) as the rows of Xᵀ, as LAPACK's dtrevc: real for a real eigenvalue, solved in real arithmetic; for a pair re ± i im (im [k] > 0 the first) the first's vector, the second's its conjugate.
 
template<typename T >
buffer< T > back_transformed (const buffer< T > &xt, const buffer< T > &zt, std::size_t n)
 Vᵀ = Xᵀ Zᵀ, row-major n × n, through the matrix product kernel.
 
template<typename T >
buffer< T > reversed_adjoint (const buffer< T > &t, std::size_t n)
 S = J Tᴴ J of the upper (quasi-)triangular t (row-major n × n, J the reversal): upper (quasi-)triangular again, a 2 × 2 block of standard form one still, its eigenvalues t's conjugated in reverse order. A right eigenvector w of S for conj λ gives J w, a left one of t for λ (yᴴ t = λ yᴴ, LAPACK's trevc side L).
 
template<typename T >
buffer< T > reversed_rows (const buffer< T > &w, std::size_t n)
 Row j of the rows w (n × n) its row n − 1 − j, its entries in reverse order: J applied to each of reversed_adjoint's vectors, in the order of t's eigenvalues.
 
void whole_schur_form (const schur_target< double > &t, std::size_t steps)
 The real Schur form of the whole of t (its eigenvalues not kept), steps its iterations' bound.
 
void whole_schur_form (const schur_target< std::complex< double > > &t, std::size_t steps)
 The complex Schur form likewise.
 
std::vector< std::complex< double > > schur_values (const double *t, std::size_t n)
 The eigenvalues of the real quasi-triangular t (n × n, row-major), in its order: a 2 × 2 block of standard form [a b; c a] the pair a ± i √|b| √|c|, as LAPACK's dlanv2 gives them, the first positive.
 
std::vector< std::complex< double > > schur_values (const std::complex< double > *t, std::size_t n)
 The eigenvalues of the complex upper triangular t: its diagonal.
 
lapack_index block_order (one_based< std::complex< double > >, lapack_index, lapack_index)
 A complex upper triangular form's diagonal blocks: each of order 1.
 
std::size_t selected_leading (const std::vector< std::complex< double > > &, const eigenvalue_selection &, std::size_t moved, std::type_identity< std::complex< double > >)
 The count of a complex Schur form's sort, as zgees takes trsen's: the eigenvalues it moved.
 
std::size_t selected_leading (const std::vector< std::complex< double > > &after, const eigenvalue_selection &select, std::size_t, std::type_identity< double >)
 The count of a real Schur form's sort, as dgees checks the order the exchanges left, from the eigenvalues as they now stand (after): a pair selected where either of its two is, counted twice.
 
template<typename T >
std::size_t move_leading (buffer< T > &t, buffer< T > &qt, std::size_t n, const std::vector< bool > &chosen)
 Moves the eigenvalues of the Schur form t whose entries of chosen are set to its leading block, the rows of qt (Zᵀ) turning with it, in their order, as LAPACK's trsen (dtrexc or ztrexc): each chosen block in turn to the first place past those already moved; a real pair moves where either of its two is chosen.
 
template<typename T >
void rotate_band (band_work< T > &a, std::size_t p, double c, const T &s, buffer< T > &turned)
 A ← G A Gᴴ of the band a, G the rotation [c s; −s̄ c] on rows and columns p and p + 1: the two rows within the bulge's reach, then their 2 × 2 block; turned (Mᴴ's rows, M the product of the rotations, where not empty) turned with them.
 
double blend (bool condition, double if_true, double if_false) noexcept
 if_true where condition holds, else if_false, by a mask of their bits: GCC turns a ?: between a value computed only for it and an argument into a branch around that computation, its table reads among it, and then vectorizes no loop of the function ("unsupported control flow in loop").
 
double table_exp (double x) noexcept
 exp(x) for an x whose result is normal or infinite (exp_normal_bound or above): x = (128 e + j) ln 2/128 + r, |r| ≤ ln 2/256, and exp(x) = 2^e · 2^(j/128) · exp(r), exp(r) − 1 by its Taylor polynomial of degree 6. Every multiply-add std::fma, written out, so that no compiler fuses otherwise; inlined into the loop that calls it (exp_into). Below the bound, its answer is not exp's; for NaN, NaN.
 
void two_sum (double a, double b, double &sum, double &error) noexcept
 The sum of a and b as sum rounded and error exact (Knuth's two-sum: no order of magnitudes needed).
 
std::int64_t expm1_parts (double clamped, double &sum, double &small) noexcept
 expm1(clamped) in units of 2^e, as sum + small unrounded, and e: table_exp's reduction, x = (128 e + j) ln 2/128 + r, then 2^(j/128) − 2^−e + 2^(j/128) r_head, each sum and product of them exact (two-sums and a fused product's error), into sum, and the small rest, 2^(j/128) (r − r_head + exp(r) − 1 − r) and the table's tail times exp(r), with the two-sums' errors, into small. For clamped from −40 to 710.
 
double table_expm1 (double x) noexcept
 expm1(x) by table_exp's reduction, x = (128 e + j) ln 2/128 + r: in units of 2^e, 2^(j/128) − 2^−e
 
double table_tanh (double x) noexcept
 tanh(x) = t / (t + 2), t = expm1(2|x|) as a pair (expm1_parts, its two parts summed exactly), the sum t + 2 a pair too, the quotient by the reciprocal of its head, corrected by its exact residual: within 0.502 ulp of the exact value. Past 22, ±1 (1 − tanh under half an ulp of it); x itself where |x| < 2^−27 (tanh(x) − x under half an ulp of it, ±0 kept) and for NaN.
 
double hyperbolic_parts (double magnitude, double &d_high, double &d_low, double &u_high, double &u_low) noexcept
 The parts of sinh and cosh of magnitude (0 to 711, past which both overflow): t = expm1(magnitude) (expm1_parts, its two parts summed exactly) at most 2^64 in its place (past it, t · 2^(64 − e), whose 1 and 1/(t + 1) then weigh under 2^−63 of it), d = t + 1 and u = 1/d as pairs (d_high + d_low, u_high + u_low); sinh = (d − u)/2 and cosh = (d + u)/2, each times the power of two returned. Near 0, d − u loses no digit: d_high − u_high is exact and the pairs' tails carry the rest.
 
double table_sinh (double x) noexcept
 sinh(x) = (d − u)/2 (hyperbolic_parts), the sign of x: within 0.503 ulp of the exact value up to the overflow (2^(e − 1) a normal double, where e^x alone would overflow from 709.78). x itself where |x| < 2^−27 (sinh(x) − x under half an ulp of it, ±0 kept) and for NaN.
 
double table_cosh (double x) noexcept
 cosh(x) = (d + u)/2 (hyperbolic_parts): within 0.502 ulp of the exact value up to the overflow; NaN for NaN.
 
double table_exp2 (double x) noexcept
 2^x for an x whose result is normal or infinite (−1022 or above): x = (128 e + j)/128 + f, |f| ≤ 1/256, each step exact (the scaling by 128 and f's difference), and 2^x = 2^e · 2^(j/128) · exp(f ln 2), as in table_exp. Within 0.506 ulp of the exact value, and exact at every integer (f = 0, j = 0). Below the bound, its answer is not 2^x's; for NaN, NaN.
 
template<bool Cosine>
double reduced_sine (double x) noexcept
 sin(x), or cos(x) where Cosine, for |x| ≤ sine_reduction_bound (another x gives a value of no use): k the integer nearest x·2/π, by the sum with 1.5·2⁵² whose low bits are k's (its quadrant); r = x − k·π/2 by three multiply-adds of half_pi_parts, |r| ≤ π/4; sin r and cos r by fdlibm's polynomials (within 2⁻⁵⁸ there), the quadrant's taken and signed by masks. A float's value rounded to float is within half its ulp and 2⁻²⁸: the floats below 2²⁰ come no nearer a multiple of π/2 than 2⁻²⁷·⁸ (252.898…, near 161·π/2), where the reduction errs by some 2⁻⁸⁰. ±0 kept. Every multiply-add std::fma, so the bits are the same in vectors.
 
template<bool Cosine>
sine_terms sine_terms_of (std::uint64_t entry, double x) noexcept
 sine_terms of the table's row at entry (sin head, sin tail, cos head, cos tail) for x: the sine's where not Cosine, the cosine's otherwise; one specialization each, no choice at run time.
 
template<>
sine_terms sine_terms_of< false > (std::uint64_t entry, double x) noexcept
 The terms of the sine (sine_terms): a = S, b = C, x itself where tiny.
 
template<>
sine_terms sine_terms_of< true > (std::uint64_t entry, double) noexcept
 The terms of the cosine (sine_terms): a = C, b = −S, 1 where tiny.
 
template<bool Cosine>
double table_sine (double x) noexcept
 sin(x), or cos(x) where Cosine, of a double within half an ulp and a few thousandths, for |x| ≤ sine_reduction_bound (another x gives a value of no use): j the integer nearest x·32/π (its low six bits the table's entry), r = x − jπ/32 by sine_step_parts as a pair r_hi + r_lo (|r| ≤ π/64); sin r − r and cos r − 1 by their Taylor series to r¹¹ and r¹⁰; then sin x = S + (C·r + S·(cos r − 1) + C·(sin r − r)), cos x = C + (−S·r + …) for S, C the table's sin(jπ/32) and cos(jπ/32), the first sum exact (a two-sum and a two-product), the small rest added once. x itself where |x| < 2⁻²⁷ (±0 kept), 1 for the cosine. Every product beside a sum fused explicitly and every other operation its own, so the bits are the same in vectors and under every compiler with the FMA instructions. Within 0.502 ulp of the exact value on 80 000 points to 2²⁰, a third of them near multiples of π/32 (the C library's of macOS: 0.86).
 
double table_atan (double x) noexcept
 atan(x): y = |x|, or 1/|x| past 1 (its rounding error carried), c = k/64 the nearest point of atan_table, t = (y − c)/(1 + y c) as a pair, |t| ≤ 2^−7, atan(t) − t by its Taylor series to t^11; atan(c) + t summed exactly, π/2 less that past 1, the small rest added once. Within 0.501 ulp of the exact value; x itself where |x| < 2^−27 (±0 kept) and for NaN, ±π/2 past 2^54. Every product beside a sum fused explicitly, as table_expm1's.
 
void log_parts (double u, double u_low, double &sum, double &small) noexcept
 ln(u + u_low) as sum + small unrounded, for u at least 2^−54 and finite and |u_low| at most half an ulp of it: u = 2^e m, m in [0.703125, 1.40625), c = k/128 the nearest point, r = m RN(1/c) − 1 + u_low RN(1/c)/2^e as a normalized pair; e ln 2 + (−ln RN(1/c)) + r − r²/2 summed exactly (an exact sum of multiples of 2^−42, two-sums, a fused square's error) into sum, and the rest, r³/3 − … − r¹⁰/10, the low parts and r's low part by the derivative 1 − r + r², into small. Near 0, u_low is as large as r, under which small would round to an ulp of the result. Every product beside a sum fused explicitly, as table_expm1's.
 
double table_log1p (double x) noexcept
 ln(1 + x) by log_parts of 1 + x as a pair (its two-sum): near 0, where c = 1, r is x exactly. Within 0.501 ulp of the exact value; x itself where |x| < 2^−54 (±0 kept) and for NaN, −∞ at −1, +∞ at +∞, NaN below −1.
 
template<typename T >
T pivot_divisor (const T &pivot)
 The divisor over_pivot takes for the pivot pivot. A complex pivot's reciprocal, once, as LAPACK's zgetf2 takes it (a complex division calls the C library's robust one, a quarter of a complex LU of order 48 when it ran for every element); a real pivot itself, each element divided.
 
template<typename R >
complex_divisor< R > pivot_divisor (const std::complex< R > &pivot)
 pivot_divisor of a complex pivot: its reciprocal, as zgetf2 takes one at least the smallest normal number in modulus (sfmin), and one under it divided by, whose reciprocal would overflow. (A zero pivot never comes here: the factorisation leaves its column, and the solves refuse a singular factor.)
 
template<typename T >
T over_pivot (const T &x, const T &divisor)
 x over the pivot whose pivot_divisor is divisor: the same quotient wherever the LU takes it.
 
template<typename R >
std::complex< R > over_pivot (const std::complex< R > &x, const complex_divisor< R > &divisor)
 over_pivot of a complex x: times the pivot's reciprocal, the product in line; over a subnormal pivot itself.
 
template<typename T >
T over_real_diagonal (const T &x, const T &diagonal)
 x over a Cholesky factor's diagonal entry diagonal, real and positive whatever the scalar.
 
template<typename R >
std::complex< R > over_real_diagonal (const std::complex< R > &x, const std::complex< R > &diagonal)
 over_real_diagonal of a complex x: each part over the diagonal's real part, one rounding each, where a complex division took the C library's scaled one (several roundings, and other bits under another library).
 
template<typename T >
void subtract_term (T &sum, const T &a, const T &b)
 sum −= a · b in one statement, which the compiler may fuse (as it fuses the matrix solve's).
 
template<typename R >
void subtract_term (std::complex< R > &sum, const std::complex< R > &a, const std::complex< R > &b)
 subtract_term of complex numbers: the product in line (operator_times, the operator's rounding).
 
constexpr std::size_t lu_block_for (std::size_t n) noexcept
 The panel width of the LU of order n: a power of two, so a row's panel is a mask away, not a division.
 
double power_of_two_scale (double largest) noexcept
 The power of two a decomposition divides a matrix by whose largest magnitude is largest: its leading bit's, the largest entry then in [1, 2); 1 where largest is 0. Dividing and multiplying back by it are exact, where a division by the largest itself rounded every entry.
 
constexpr std::size_t cholesky_block_for (std::size_t n) noexcept
 The panel width of the Cholesky factorisation of order n: narrower up to order 256, where the panels, row by row on one thread, are most of the time; a power of two, which the product kernels' tiles divide (widths of 24, 40 or 56 left the AMX's 32 × 16 tiles part empty: order 320, 20 % longer).
 
template<typename T >
T conj_of (const T &x)
 The conjugate of a real (or general) scalar — the value itself.
 
template<typename U >
std::complex< U > conj_of (const std::complex< U > &x)
 The conjugate of a complex scalar.
 
template<typename T >
magnitude_t< T > pivot_magnitude (const T &x)
 The magnitude partial pivoting compares: |x| of a real x.
 
template<typename U >
U pivot_magnitude (const std::complex< U > &x)
 The magnitude partial pivoting compares of a complex x: |Re x| + |Im x|, LAPACK's (izamax, cabs1), so the same pivots as its zgetrf, with no square root.
 
template<typename T >
T from_carried (carried_t< T > x)
 x, a magnitude carried in carried_t, as the scalar T: converted to T's real type first, so a std::complex<float> is built from a float (a double argument narrows inside its constructor).
 
template<typename T >
carried_t< T > real_part (const T &x)
 The real part of a real (or general) scalar, in carried_t.
 
template<typename U >
carried_t< U > real_part (const std::complex< U > &x)
 The real part of a complex scalar.
 
template<typename T >
carried_t< T > rescue_factor (carried_t< T > largest)
 The exact power of two that brings a magnitude below about the square root of the smallest normal number of T's range (2⁻⁵⁰⁰ in double, 2⁻⁶⁰ in float) up among normal numbers, as LAPACK's dlarfg rescales; 1 for any larger magnitude. It is a T itself: a zero takes the factor of the smallest normal number, which a double's range holds and a float's does not.
 
template<std::floating_point R>
R times_power_of_two (R x, int e)
 x times 2ᵉ, rounded once (a power of 2 scales a normal number exactly).
 
template<std::floating_point R>
R part_size (R x)
 The magnitude of a real x, x's exponent read from it.
 
template<std::floating_point U>
U part_size (const std::complex< U > &x)
 The larger magnitude of a complex x's parts: no sum of them, which overflows near the largest double.
 
template<std::floating_point U>
std::complex< U > times_power_of_two (const std::complex< U > &x, int e)
 times_power_of_two of each part of a complex x.
 
template<typename T , typename Factor >
T scaled_product (T start, std::size_t count, const Factor &factor)
 The product of count factors, factor(i) for each i, times start, as a determinant multiplies its pivots: each factor, and the running product after it, brought near 1 by a power of 2 (exactly) and the powers kept apart, then applied once at the end. The same bits as the plain product wherever that stays normal; where a partial product would overflow or underflow (pivots of 10¹⁵⁰ and 10⁻¹⁵⁰), the representable answer.
 
template<typename T >
T phase_of (const T &x)
 The unit-modulus factor of x: x / |x| (its sign, for a real scalar), and 1 at 0. A subnormal x is scaled up first, exactly: its modulus would round to the coarse subnormal grid, and x / |x| miss unit modulus by as much.
 
template<typename T >
carried_t< T > rescale_tiny (buffer< T > &x)
 Scales x by rescue_factor of its largest magnitude, exactly, so that a Householder vector built from it is computed among normal numbers: a subnormal norm rounds to the coarse subnormal grid, and the reflector is then not orthogonal.
 
template<typename T >
carried_t< T > norm_at_scale (const T *x, std::size_t count)
 The Euclidean norm of the count elements at x, summed at the scale of their largest magnitude: a plain sum of squares overflows near 1e155 and underflows to 0 near 1e-155.
 
template<typename T >
double plain_or_at_scale (double plain, const T *x, std::size_t count)
 √plain, the plain sum of the squares of the count elements at x, where it neither overflowed nor underflowed (between plain_squares_low and plain_squares_high); else norm_at_scale.
 
template<typename T >
carried_t< T > scaled_norm (const T *x, std::size_t count)
 The Euclidean norm of the count elements at x, by norm_at_scale for this type.
 
double scaled_norm (const double *x, std::size_t count)
 The same for double: the plain sum of squares in one pass of dot_product where it holds.
 
double scaled_norm (const std::complex< double > *x, std::size_t count)
 The same for std::complex<double>: dot_product over the parts, which an array of them holds in order ([complex.numbers]).
 
template<typename T >
carried_t< T > scaled_norm (const buffer< T > &x)
 scaled_norm of a whole vector.
 
double rotation_length (double a, double b) noexcept
 √(a² + b²) for a plane rotation of the QR and QL iterations, as LAPACK's dlartg takes it: the plain sum of squares where the larger magnitude lies within 2^±500, so its square neither overflows nor underflows and the smaller one's underflow is below its rounding; std::hypot, which scales every time, outside. A graded matrix reaches both ends though scaled to its largest entry: a QL shift's ratio (d₁ − d₀) / 2e came to 5e199 on [[1e-200, 1e-200], [1e-200, 1]].
 
template<typename T >
bool is_finite_entry (const T &x)
 Whether x is finite.
 
template<typename U >
bool is_finite_entry (const std::complex< U > &x)
 Whether x is finite: both of its parts.
 
template<typename T >
void require_finite (const matrix< T > &a, const char *what)
 The one non-finite policy, checked where every decomposition starts.
 
template<typename T >
matrix< T > parallel_copy_of (const matrix< T > &a)
 A copy of a whose pages the threads fault in (parallel_copy), a decomposition's own storage: the matrix's copy faulted every one in on the calling thread (order 1000 on one arm64 core 0.64 ms, an eighth of the Cholesky factorisation; on eight cores 0.28 ms).
 
bool all_finite_shared (const double *x, std::size_t count)
 all_finite on the threads by chunks of finite_chunk, past two of them: a pass bound by the memory, an element read weighed as half one copied (six x86-64 AVX2 cores, a Cholesky factorisation of order 500 with its other passes on the threads too: 1.12 → 0.91 ms).
 
void require_finite (const matrix< double > &a, const char *what)
 require_finite for double, its elements contiguous: all_finite.
 
void require_finite (const matrix< std::complex< double > > &a, const char *what)
 The same for std::complex<double>, whose array holds its parts in order ([complex.numbers]).
 
template<typename T >
bool lower_is_finite (const matrix< T > &a)
 Whether the lower triangle of the square a, and its diagonal's real part, are finite: entry by entry, kept out of line: the rare road of require_finite_lower where the whole matrix is not finite.
 
template<typename T >
void require_finite_lower (const matrix< T > &a, const char *what)
 require_finite over what a Hermitian eigensolver reads: the lower triangle, and the diagonal's real part (as numpy.linalg.eigh reads them, whatever the rest holds).
 
void require_finite_lower (const matrix< double > &a, const char *what)
 require_finite_lower for double: all_finite over the whole matrix, the one vectorised pass require_finite takes, and the triangle entry by entry only where it finds a NaN or an infinity. Out of line, so that a decomposition's constructor stays small enough to be inlined: in it, the check kept GCC from inlining the Cholesky's (order 8, 0.204 → 0.220 µs), as an entry-by-entry check (6 % at order 64 on arm64) or a row-by-row one (21 % at order 8 under GCC) cost in time.
 
void require_finite_lower (const matrix< std::complex< double > > &a, const char *what)
 The same for std::complex<double>, whose array holds its parts in order ([complex.numbers]).
 
template<typename T >
void subtract_panel_columns (T *column, const T *columns, std::size_t height, const T *l, std::size_t terms)
 column −= Σₜ x_t · conj(l[t]) over the terms columns columns (height elements each, one after another), term by term in ascending order: an elimination step of a Cholesky panel's rows below (an overload per scalar, as conj_of). Each term subtract_product, the column read and written again for every one.
 
void subtract_panel_columns (double *column, const double *columns, std::size_t height, const double *l, std::size_t terms)
 subtract_panel_columns for double: subtract_products, which holds the column in registers through the terms, the same multiply-adds in the same order, so the same values (a complex one keeps the term-by-term calls, whose x · f the other would round as f · x).
 
std::pair< matrix< float >, int > scaled_to_float (const matrix< double > &m)
 The finite m times 2⁻ᵉ in single precision, 2ᵉ the least power of two above its largest magnitude: every entry below 1 in magnitude, so none overflows float, and the scaling exact.
 
double residual_ratio (const matrix< double > &x, const matrix< double > &r)
 The largest of ‖r‖∞ / ‖x‖∞ over the columns of the residual r and the solution x (LAPACK's dsgesv compares each with ‖A‖∞ ε √n); NaN, which no bound meets, where an entry of r is not finite.
 
std::optional< matrix< double > > mixed_precision_solve (const matrix< double > &a, const matrix< double > &b, double *reciprocal_condition=nullptr)
 X of A X = B (A square, B of its rows) as LAPACK's dsgesv: A factorised in single precision, its solution corrected in double precision by the residual R = B − A X (each correction a single-precision solve of A D = R) until every column meets ‖R‖∞ ≤ ‖X‖∞ ‖A‖∞ ε √n (ε double's unit roundoff). Each operand goes to single precision scaled by a power of two (scaled_to_float): none overflows, where dsgesv falls back. A correction must divide the largest ‖R‖∞ / ‖X‖∞ by 8 at least, which bounds their count (about a dozen from single precision to the bound) where dsgesv allows 30: an A too ill-conditioned for single precision converges slowly or not at all, and the double factorisation is then the faster. The single factorisation costs half the double one's; each correction a product and a solve of n² k.
 
template<typename T >
void triangular_halves (const T *a, std::size_t n, bool lower, bool transposed, bool unit, T *x, std::size_t ldx, std::size_t k, std::size_t first, std::size_t last, T *minus)
 A X = B in place in x (n × k, row-major), for the n × n row-major triangle of a, its lower one or its upper one (the other never read), its diagonal taken as ones where unit: by halves (triangular_halves), each half's terms out of the other by one matrix product as deep as it, as a recursive trsm; one column by a dot product of each row with the unknowns solved.
 
template<typename T >
void triangular_rows (const T *a, std::size_t n, bool lower, bool transposed, bool unit, T *x, std::size_t k)
 M X = B in place in x (n × k, row-major) for M the n × n row-major a or, transposed, its transpose, read in place; lower whether a (not M) is lower-triangular, the other triangle never read, its diagonal taken as ones where unit. By halves (triangular_halves), as a recursive trsm, the columns by blocks of triangular_slice on the threads; one column of M = a by a dot product of each row with the unknowns solved, of M = aᵀ by a sweep, each unknown solved taken from the others by its row of a.
 
template<typename T >
bool elements_finite (const T *x, std::size_t count)
 Whether the count elements at x are finite, one at a time: a type no vectorised pass takes.
 
bool elements_finite (const double *x, std::size_t count)
 elements_finite of double: all_finite_shared, vectorised.
 
bool elements_finite (const std::complex< double > *x, std::size_t count)
 The same for std::complex<double>, whose array holds its parts in order ([complex.numbers]).
 
template<typename T >
bool triangle_is_finite (const T *a, std::size_t n, bool lower)
 Whether the triangle of the row-major n × n a a solve reads (the diagonal with it) is finite, entry by entry: the rare road of require_finite_triangle, where the whole matrix is not.
 
template<typename T >
void require_finite_triangle (const T *a, std::size_t n, bool lower, const T *x, std::size_t count)
 Refuses an infinite or NaN entry of the triangle of a a triangular solve reads, or of its right-hand side x (count elements): one vectorised pass over the whole matrix, then the triangle entry by entry only where that pass finds one (the other triangle may hold anything); the right-hand side in one pass.
 
template<typename T >
void solve_finite_triangular_in_place (const T *a, std::size_t n, bool lower, triangular_op op, bool unit_diagonal, T *x, std::size_t k)
 solve_triangular_in_place of a triangle and a right-hand side known finite (require_finite_triangle, or a check that covered them): a zero on the diagonal still refused.
 
template<typename T >
void solve_triangular_in_place (const T *a, std::size_t n, bool lower, triangular_op op, bool unit_diagonal, T *x, std::size_t k)
 numerical::solve_triangular on the n × n row-major a, in place in x (n × k, row-major): the triangle checked, then read where it lies, transposed in place; the adjoint by the transpose on the conjugates (Aᴴ X = B is Aᵀ conj X = conj B).
 
template<std::floating_point R>
R sign_entry (R x, int &sign) noexcept
 A real entry's sign for one_norm_estimate's sign vectors, dlacn2's: 1 from zero up, else −1; sign set to it.
 
template<std::floating_point R>
std::complex< R > sign_entry (std::complex< R > x, int &sign) noexcept
 A complex entry's unit-modulus sign, zlacn2's: x / |x|, 1 where |x| is not past the least normal number; sign 0, a complex climb having no sign vector to repeat.
 
template<std::floating_point R>
bool same_sign (R x, int sign) noexcept
 Whether a real entry's sign is sign (sign_entry's), as dlacn2 finds a sign vector repeated.
 
template<std::floating_point R>
bool same_sign (std::complex< R >, int) noexcept
 A complex entry's sign never repeats (zlacn2 has no such test).
 
template<std::floating_point R>
bool climb_ends (R last, R largest) noexcept
 Whether the climb's last column last is as large as the new largest entry largest, so it stops: dlacn2 compares the real entry itself with the largest modulus.
 
template<std::floating_point R>
bool climb_ends (std::complex< R > last, std::complex< R > largest) noexcept
 climb_ends of complex entries, as zlacn2: their moduli.
 
template<typename T >
double one_norm_estimate (std::size_t n, const std::function< void(T *, bool)> &apply)
 An estimate of ‖B‖₁ for the n × n operator apply gives, as LAPACK's dlacn2 and zlacn2 (Hager's method, Higham's refinements): apply(x, false) overwrites the n entries at x with B x, apply(x, true) with Bᴴ x. Products on sign vectors (unit-modulus ones for complex T) climb to a column of large norm, at most five each way; one more on (−1)ⁱ (1 + i / (n − 1)) guards against the operators that fool the climb. A lower bound, rarely short of ‖B‖₁ by more than a factor of 3 (Higham, Accuracy and Stability of Numerical Algorithms, 2002, §15.3); infinity where a product overflows.
 
template<typename T >
double reciprocal_condition (std::size_t n, double one_norm, const std::function< void(T *, bool)> &apply)
 The reciprocal condition number 1 / (‖A‖₁ ‖A⁻¹‖₁) of A from one_norm, ‖A‖₁, and an estimate of ‖A⁻¹‖₁ by one_norm_estimate over apply, as LAPACK's *con routines end: 1 for an empty matrix, 0 for a zero one, and 0 where the estimate overflows.
 
template<typename T >
double lu_reciprocal_condition (const T *lu, std::size_t n, double one_norm)
 reciprocal_condition of A = P⁻¹ L U from the n × n row-major lu (L below the diagonal, its unit diagonal not stored, U on and above it) and one_norm, ‖A‖₁, as LAPACK's gecon: A⁻¹'s products U⁻¹ L⁻¹ and L⁻ᴴ U⁻ᴴ (the permutation leaves 1-norms alone). 0 where U has a zero on its diagonal.
 
template<typename T >
double cholesky_reciprocal_condition (const T *factor, std::size_t n, bool lower, double one_norm)
 reciprocal_condition of A = L Lᴴ or Uᴴ U from its Cholesky factor, L in the lower triangle of the n × n row-major factor or U in its upper one, and one_norm, ‖A‖₁, as LAPACK's pocon: A⁻¹ its own adjoint, the two products one.
 
template<typename T >
double triangular_reciprocal_condition (const T *a, std::size_t n, bool lower)
 reciprocal_condition of the triangular A, the lower triangle of the n × n row-major a or its upper one, as LAPACK's trcon (1-norm): ‖A‖₁ the triangle's largest column sum of moduli, A⁻¹'s products by substitution. 0 where the diagonal holds a zero.
 
template<typename T >
std::pair< matrix< T >, matrix< T > > split_lu (const T *lu, std::size_t rows, std::size_t cols, std::size_t k)
 The rows of lu (rows × cols row-major, L below its diagonal, U on and above) as L, its first k columns (unit diagonal), and U, its first k rows.
 
template<typename T >
void tall_lu_columns (T *panel, std::size_t m, std::size_t n, std::size_t first, std::size_t last, std::vector< std::size_t > &rows)
 Factorises the columns [first, last) of the m × n column-major panel (m ≥ n; column j at panel[j m]) by partial pivoting, in place, as LAPACK's dgetrf2: the left half factorised, the top of the right half solved against its unit-lower L, the bottom updated by one product, then the right half; a leaf of solve_leaf columns, or a panel too short for its products to pay, unblocked. Each pivot's row swap reaches every column and rows (row i of the factors is row rows[i] of the input); a zero pivot column is left as it is.
 
template<typename T >
void reflect_rows (buffer< T > &a, std::size_t cols, const buffer< T > &v, std::size_t first, std::size_t from, const T &s=T {2})
 Applies I − s v vᴴ from the left (s 2 for a unit v) to the rows first…first+|v|-1 of a row-major array, on its columns from…cols-1.
 
template<typename T >
reflector_block< T > gather_reflectors (const std::vector< buffer< T > > &reflectors, std::size_t k0, std::size_t k1, std::size_t rows, const std::vector< T > *taus=nullptr)
 Gathers the reflectors reflectors[k0…k1-1] (each I − τ_k v vᴴ: τ_k from taus, else 2 for a unit v; an empty one the identity) into I − V T Vᴴ = H_{k0} H_{k0+1} … H_{k1-1}, T as LAPACK's dlarft forms it.
 
template<typename T >
void apply_reflector_block (const reflector_block< T > &block, bool adjoint, buffer< T > &a, std::size_t cols, std::size_t from, std::size_t to)
 Applies I − V T Vᴴ (or its adjoint I − V Tᴴ Vᴴ) from the left to the block's rows of a row-major array, on its columns from…to-1: W = Vᴴ A, then W ← T W or Tᴴ W, then A −= V W, the two outer products through the matrix product kernel.
 
template<typename T >
void apply_transposed_reflector_block (const reflector_block< T > &block, buffer< T > &at, std::size_t cols, std::size_t from, std::size_t to, bool adjoint=false)
 apply_reflector_block to A held transposed: at is Aᵀ (row-major, cols columns), and A ← (I − V T Vᴴ) A on A's rows first… and columns from…to-1 is Aᵀ ← Aᵀ (I − conj(V) Tᵀ Vᵀ) on Aᵀ's columns first… and rows from…to-1 (with the adjoint (I − V Tᴴ Vᴴ) A, conj(T) for Tᵀ). Its three products have to − from rows, which the matrix product kernel runs well and on every core; apply_reflector_block's first has the block's few (a reflector block's 32 rows by 1000 deep: 54 GFLOP/s on twenty x86-64 threads).
 
template<typename T >
requires std::is_floating_point_v<T>
T conjugate_dot (const T *x, const T *y, std::size_t count)
 Σ conj(x[i]) · y[i] for a real T, where the conjugate is the value: dot_product.
 
template<typename T >
T conjugate_dot (const T *x, const T *y, std::size_t count)
 Σ conj(x[i]) · y[i] of complex numbers (conjugate_dot_product).
 
template<typename T >
std::pair< T, T > larfg_in_place (buffer< T > &w)
 The reflector LAPACK's dlarfg (zlarfg) forms of w, in place: H = I − τ w wᴴ with w₀ = 1 and Hᴴ x = β e₁, β real, of the sign opposite to Re x₀. Exact inputs stay exact where LAPACK's do (e₂ reflected onto e₁: w = (1, 1), τ = 1, where a unit v holds 1/√2, rounded). For an x with something below x₀, or x₀ complex (the identity otherwise, which the caller keeps).
 
template<typename T >
void reflect_columns (T *panel, std::size_t height, const buffer< T > &v, std::size_t first, std::size_t from, std::size_t to, const T &s=T {2})
 Applies I − s v vᴴ from the left (s 2 for a unit v) to the columns from…to-1 of the column-major panel (height rows per column), on its rows first…first+|v|-1: each column a dot product with v, then v scaled out of it, both along the column at unit stride.
 
template<typename T >
void factor_qr_panel (T *panel, std::size_t ld, std::size_t top, std::vector< buffer< T > > &reflectors, std::vector< T > &taus, std::size_t k0, std::size_t k1)
 Householder reflections of the columns k0…k1-1 of a column-major matrix, rows from k0, one by one, in place, into reflectors[k0…k1-1] and taus: panel holds column k0, ld elements per column from row top (0 in the whole matrix, k0 in a copy of the panel). A reflector meets each column at unit stride. Recursive halves, the left's reflectors applied to the right at once, took longer at these widths (32 columns).
 
template<typename T >
std::vector< reflector_block< T > > reflect_in_columns (buffer< T > &columns, std::size_t rows, std::size_t cols, bool reflectors=true)
 The blocks of reflectors of a Householder QR of the column-major columns (rows per column, cols columns), factorised in place as LAPACK's dgeqrf leaves it: R on and above the diagonal, each reflector's w₁… below it. The columns right of a panel take its block as (I − V T Vᴴ)ᴴ, held transposed (apply_transposed_reflector_block): the products then have as many rows as there are columns to update, where the row-major form's first product has the block's 32 (one x86-64 core: 39 ms against 46 for the updates of order 1000). From qr_look_ahead_threads threads, the next panel is updated and factorised by one task while the others update the columns past it, a slice each. Where reflectors is false (R alone asked), a block no update reads is not gathered.
 
template<typename T >
void transpose_into (const T *from, std::size_t rows, std::size_t cols, T *to)
 The row-major rows × cols from into to as its transpose, cols × rows row-major, by tiles, bands of them on the threads (a pass bound by the memory, its bounds captured by value).
 
template<typename T >
void factor_row_major_panel (buffer< T > &a, std::size_t rows, std::size_t cols, std::vector< buffer< T > > &reflectors, std::vector< T > &taus, std::size_t k0, std::size_t k1)
 factor_qr_panel on the columns k0…k1-1 of the row-major a, rows from k0, in a column-major copy of the panel, copied back. A function of its own: written into reflect_in_rows' loop, the same instructions took 5 % more cycles at order 128 (clang, arm64).
 
template<typename T >
std::vector< reflector_block< T > > reflect_in_rows (buffer< T > &a, std::size_t rows, std::size_t cols, buffer< T > *r=nullptr, bool reflectors=true)
 The blocks of reflectors of a Householder QR of the row-major a (rows × cols), factorised in place as reflect_in_columns leaves the column-major one (R on and above the diagonal, each reflector's w₁… below it): each panel in a column-major copy, the columns right of it updated by apply_reflector_block, whose projection reads A as the product's right operand. The AMX tile reads that in place, and NEON's packs it faster than the column-major form's left one (arm64, order 512: 6.3 ms against 7.2 under clang, 16.4 against 17.9 under GCC). Where r is given, it is R (min(rows, cols) × cols row-major, zeros below the diagonal).
 
template<typename T >
qr_reflections< T > householder_reflect (buffer< T > a, std::size_t rows, std::size_t cols, bool reflectors=true)
 The reflections of householder_qr, without forming Q: A = Q R with Q the product of the blocks' reflectors applied to the first k = min(rows, cols) columns of the identity, R's diagonal not yet made real (a product with Qᴴ, or with Q of a block padded with zeros, applies the blocks instead). a is row-major. On the x86-64 units it is transposed in by tiles and factorised column-major (reflect_in_columns, whose updates' products run faster there: one AVX2 core, order 1000, 39 ms against 46), elsewhere in place (reflect_in_rows). Where reflectors is false, R alone is asked: a block no update reads is not gathered, and the blocks are not all there.
 
template<typename T >
void form_q_into (buffer< T > &out, const std::vector< reflector_block< T > > &blocks, std::size_t rows, std::size_t first, std::size_t count, bool transposed)
 The columns [first, first + count) of Q = H₀ H₁ …, the blocks applied (the last first) to those of the identity, into out: rows × count row-major, or where transposed Qᵀ row-major (Q column-major). Formed as householder_reflect factorises, so that a Q is the same to the bit in either layout: as the rows of Qᵀ on the x86-64 units (apply_transposed_reflector_block), row-major elsewhere (apply_reflector_block), then transposed where the other layout is asked. The columns of the identity left of a block's rows are left as they are.
 
template<typename T >
void apply_q (buffer< T > &p, const std::vector< reflector_block< T > > &blocks, std::size_t rows, std::size_t count)
 P ← Q P for the rows × count row-major p, Q = H₀ H₁ … the product of the blocks' reflectors, the last applied first: a basis taken through Q without forming Q, as LAPACK's dormqr applies it where dorgqr forms it. As form_q_into applies the blocks: on the x86-64 units to the rows of Pᵀ (apply_transposed_reflector_block), transposed in and out, elsewhere in place (apply_reflector_block).
 
template<typename T >
thin_qr< T > householder_qr (buffer< T > a, std::size_t rows, std::size_t cols, bool complete=false)
 Factorises the rows × cols row-major a by Householder reflections, A = Q R with k = min(rows, cols): Q is rows × k with orthonormal columns and R k × cols upper trapezoidal. Each column below the diagonal is reflected onto it, so Q is orthonormal to rounding whatever the conditioning of A, and a column dependent on the earlier ones leaves a zero on R's diagonal.
 
template<typename T >
buffer< T > householder_r (buffer< T > a, std::size_t rows, std::size_t cols)
 householder_qr's R alone, k × cols, k = min(rows, cols): the same reflections and the same rotation of each row by its diagonal's phase, so the same bits, Q never formed (numpy.linalg.qr's mode="r"; forming Q is about as much work again).
 
double rank_tolerance (double largest, std::size_t rows, std::size_t cols)
 The magnitude under which a diagonal entry of R or a singular value counts as zero, as NumPy draws it: the largest times the machine epsilon times the larger dimension.
 
double adjoint_dot (const double *x, const double *y, std::size_t count)
 Σ x[i] y[i], vectorised (dot_product): xᴴ y for real numbers.
 
std::complex< double > adjoint_dot (const std::complex< double > *x, const std::complex< double > *y, std::size_t count)
 Σ conj(x[i]) y[i], vectorised (conjugate_dot_product): xᴴ y.
 
template<typename T >
double contiguous_norm (const T *x, std::size_t count)
 The 2-norm of the count entries at x, scaled by the largest so no square overflows or underflows.
 
template<typename T >
requires std::is_floating_point_v<T>
void add_conjugate_product (T *y, const T *x, const T &f, std::size_t count)
 y[i] += conj(x[i]) · f for a real T, where the conjugate is the value: add_product.
 
template<typename T >
void add_conjugate_product (T *y, const T *x, const T &f, std::size_t count)
 y[i] += conj(x[i]) · f, each product's statements spelled out as the complex operator's (four part products rounded apart, ac − bd and ad + bc, then their sums with y): the same bits for finite parts whose products stay finite, where the operator's rescue of a product whose parts are both NaN never acts, and a loop the compiler vectorises (the reduction's rows, scaled by their largest magnitude, are such).
 
template<typename T >
T hermitian_row (symmetric_row_kernel, const T *row, const T *v, T *out, std::size_t i)
 Row i's terms of a Hermitian product from the lower triangle: out[j] += conj(row[j]) v[i] for j < i, and the return Σⱼ≤ᵢ row[j] v[j], for a complex T.
 
std::complex< double > hermitian_row (symmetric_row_kernel kernel, const std::complex< double > *row, const std::complex< double > *v, std::complex< double > *out, std::size_t i)
 The same for a std::complex<double> row: in one pass on AVX-512 or AVX2 with FMA where the matrix product's kernel runs there (avx512_hermitian_row), else as the template.
 
double hermitian_row (symmetric_row_kernel kernel, const double *row, const double *v, double *out, std::size_t i)
 The same for a real row, in one pass through kernel.
 
template<typename T >
std::size_t symmetric_row_speedup ()
 The weight of a symmetric row's multiply-adds in hermitian_product: on the x86-64 units, where its rows are paced by one core's loads, vector_speedup, as more threads pay than their multiply-adds alone would pay for (an i9-9900K's reduction of order 512 took 17 % less, an i9-10900X's 7 %); 1 elsewhere (arm64's, weighed so, took as long or longer).
 
template<typename T >
void hermitian_product (const T *block, std::size_t n, std::size_t m, const T *v, T *w, buffer< T > &scratch)
 w = B v for the Hermitian m × m block (row stride n) from its lower triangle alone, as LAPACK's dsymv: each row's lower part is read once and serves twice, w[i] += Σⱼ≤ᵢ B[i][j] v[j] and w[j] += conj(B[i][j]) v[i] for j < i, so a reduction step reads half the block it would read whole. Rows go by tasks of fixed size, each into its own partial w, summed in a fixed order: the result does not depend on the number of threads. A real row is read once for both (symmetric_row), on the widest x86-64 units the matrix product's kernel may use (gemm_path_current).
 
template<typename T >
requires std::is_floating_point_v<T>
void subtract_corrections (T *y, const T *f, const T *x, std::size_t ldx, std::size_t terms, std::size_t count)
 subtract_products of a real T, where the reduction's corrections are those of its type.
 
template<split_complex T>
void subtract_corrections (T *y, const T *f, const T *x, std::size_t ldx, std::size_t terms, std::size_t count)
 subtract_products of complex numbers, each term y[i] −= f · x[i] spelled out as the complex operator's statements (four part products rounded apart, ac − bd and ad + bc, then their differences with y): the same bits for finite parts whose products stay finite, where the operator's rescue of a product whose parts are both NaN never acts, and loops the compiler vectorises (the reduction's rows, scaled by their largest magnitude, are such). Complex double on the x86-64 units keeps its fused kernel.
 
template<typename T >
void corrected_hermitian_product (const T *block, std::size_t n, std::size_t m, const T *v, T *w, buffer< T > &scratch, const T *corrections, std::size_t ldc, std::size_t terms, T *factors, T *next=nullptr, const T *next_factors=nullptr)
 hermitian_product corrected by a panel's earlier steps: w = B v − Σₜ fₜ cₜ, the coefficients f[2p] = (c[2p+1] · v) and f[2p+1] = (c[2p] · v) (conjugated) computed beside the product, among its tasks, and the partials summed and the corrections taken by bands of rows: each element the operations, in the order, hermitian_product, conjugate_dot and subtract_products give it apart.
 
double real_subdiagonal (double t, double, double &)
 A real subdiagonal entry of the tridiagonal: kept with its sign (the rotations take any sign).
 
double real_subdiagonal (std::complex< double > t, std::complex< double > delta, std::complex< double > &next)
 A complex subdiagonal entry made real by the next row's unit phase: next = delta · conj(phase(t)).
 
template<typename T >
tridiagonal_reduction< T > tridiagonalize (buffer< T > &a, std::size_t n, buffer< double > &d, buffer< double > &e)
 Reduces the Hermitian (real: symmetric) n × n row-major a, of which the lower triangle is read, to a real symmetric tridiagonal by Householder reflections, A = M T Mᴴ with M unitary.
 
template<typename T >
buffer< T > tridiagonal_basis (const tridiagonal_reduction< T > &reduction, std::size_t n)
 Mᵀ of a reduction, row-major (its rows are the columns of M): Qᵀ by blocks of reflectors from the last back, as LAPACK's dorgtr forms Q, each block on the rows it reaches of the identity, then row j times conj(δⱼ).
 
template<typename T >
void to_matrix_basis (const tridiagonal_reduction< T > &reduction, buffer< T > &at, std::size_t n, std::size_t rows)
 The rows rows of at (n columns, every row by default), vectors in the tridiagonal's basis, taken to the matrix's: at ← at Mᵀ = at Dᴴ Qᵀ, column j times conj(δⱼ), then the blocks of reflectors from the last back on every row, as LAPACK's dormtr: two thirds of the operations of forming M (tridiagonal_basis) and multiplying by it.
 
template<typename T >
void to_matrix_basis (const tridiagonal_reduction< T > &reduction, buffer< T > &at, std::size_t n)
 to_matrix_basis of all n rows.
 
std::vector< rotation_run > rotation_runs (const std::vector< row_operation > &operations)
 The operations split into runs of chained rotations, every other operation a run of one.
 
template<typename T >
void apply_row_operations (buffer< T > &m, std::size_t n, const std::vector< row_operation > &operations)
 Applies the recorded operations, in order, to the row-major m (n per row). Each column takes them independently of the others, so chunks of columns run in parallel, every element receiving the same operations in the same order as applying each in place would give it.
 
plane_rotation_real plane_rotation_to_zero (double f, double g) noexcept
 The rotation that takes (f, g) to (r, 0), as LAPACK's dlartg: r of f's sign, c 0 or more; none for g 0, a quarter turn for f 0. Its length by rotation_length.
 
symmetric_two_by_two symmetric_eigen_2x2 (double a, double b, double c) noexcept
 The eigensystem of [a b; b c] as LAPACK's dlaev2 computes it: the larger eigenvalue in magnitude from the sum and the root of the difference, the smaller from the determinant (no cancellation), and the rotation (c, s) whose first row is the larger's eigenvector.
 
void tridiagonal_root_free_ql (buffer< double > &d, buffer< double > &e, std::size_t n, std::size_t max_steps)
 The eigenvalues of the symmetric tridiagonal (d, e), no vector, as LAPACK's dsterf: the Pal–Walker–Kahan variant of implicit QL or QR on the squares of the off-diagonal entries, no root in its inner loop, each block scaled into a safe range and iterated from its smaller end, an entry dropped against its neighbours (e² ≤ ε² |dᵢ dᵢ₊₁|). Faster than tridiagonal_ql's dsteqr without rotations (one core: order 500 4.4 → 3.4 ms, 2000 66 → 52) and more accurate on a graded matrix (the smallest of dᵢ = 0.01ⁱ, n = 10: 3e-16 against 4e-14).
 
template<typename T >
void tridiagonal_ql (buffer< double > &d, buffer< double > &e, buffer< T > &zt, std::size_t n, std::size_t max_steps)
 The eigenvalues of the symmetric tridiagonal (d, e) by implicit QL or QR as LAPACK's dsteqr, the rotations accumulated into the rows of zt. Each unreduced block, scaled into a safe range, is iterated from the end of its smaller diagonal entry (QL down a block graded downwards, QR up one graded upwards), an off-diagonal entry dropped against its neighbours' diagonal entries (e² ≤ ε² |dᵢ dᵢ₊₁|) rather than the matrix's norm, and no shift is subtracted from the whole diagonal: the small eigenvalues of a graded matrix keep their relative accuracy. EISPACK's tql2, here before, tested against the largest entry seen and moved every diagonal entry by the accumulated shift: on d = gⁱ, e near √(dᵢ dᵢ₊₁), the smallest eigenvalues lost every digit (g = 0.01, n = 10; NumPy's 3e-16).
 
double quadratic_step (double a, double b, double c)
 The smaller root η of C η² − A η + B = 0, in the form that does not cancel: a step of solve_secular's fit (shared by its instantiations, whose data reach one form or the other).
 
template<typename Offset >
secular_solution solve_secular (const buffer< double > &z, double rho, std::size_t i, std::size_t max_iterations, double *delta, const Offset &offset)
 Root i of 1/ρ + Σⱼ zⱼ²/(Pⱼ − x) = 0, poles P strictly ascending, no zⱼ zero, ρ > 0, as LAPACK's dlaed4 and dlasd4 find it. The poles are given only as their offsets from one another, offset(o, j) = Pⱼ − P_o, which the caller computes accurately.
 
double secular_root (const buffer< double > &d, const buffer< double > &z, double rho, std::size_t i, std::size_t max_iterations, double *delta)
 Root i of 1/ρ + Σⱼ zⱼ²/(dⱼ − λ) = 0 (d strictly ascending, ‖z‖ = 1), the i-th eigenvalue of diag(d) + ρ z zᵀ, by solve_secular.
 
double * workspace_of (workspace_buffer &buffer, std::size_t size)
 buffer at least size elements long, its storage kept for the next join. A larger one is mapped fresh, and its pages are touched by every thread, chunk by chunk: on Linux a page costs about a microsecond the first time it is written, and the join that first wrote it waited on them one thread at a time (an eigh of order 1024 mapped some 30 MB of these buffers on one x86-64 core).
 
merge_workspace & thread_workspace ()
 The calling thread's merge_workspace, kept from one decomposition to the next: mapping its pages anew took 8–11 % of the eigenvectors' time (six AVX2 cores, an eigh of order 256 or 512). A join uses it whole on one thread, and nests no other join; release_large_workspace lets a large buffer go after.
 
void release_large_workspace (merge_workspace &work)
 Releases the buffers of work longer than kept_workspace.
 
void merge_rank_one (buffer< double > &d, buffer< double > z, double rho, double *q, std::size_t ldq, std::size_t n, std::size_t split, merge_workspace &work)
 The eigenvalues of diag(d) + ρ z zᵀ, ascending, into d, and its eigenvectors times q (the row-major n × n block of rows ldq apart whose columns span the halves) into q, as LAPACK's dlaed1.
 
void plan_divide (std::vector< divide_node > &nodes, double *d, const double *e, std::size_t n, double *q, std::size_t ldq, std::size_t depth)
 Appends the block (d, e, n) and, before it is solved, everything below it to nodes, parents before children: each scaled to work near unit size by a power of two (exactly), as the deflation tolerance is relative to the weights, which have unit norm, and so to the tridiagonal only at that size; each half has its own. A block past divide_leaf is split into two halves plus a rank-one coupling, as LAPACK's dlaed0: T = diag(T₁ − ρ e_a e_aᵀ, T₂ − ρ e_b e_bᵀ) + ρ u uᵀ, u = e_a + sign(β) e_b, ρ = |β|, β the coupling.
 
void solve_divide_leaf (const divide_node &node)
 Solves the leaf node by implicit QL: its eigenvalues ascending into its diagonal, their eigenvectors as the columns of its block of q.
 
void join_divide_node (const divide_node &node, merge_workspace &work)
 Joins the two solved halves of node by merge_rank_one, in work.
 
void unscale_divide_node (const divide_node &node)
 The eigenvalues of a solved node, scaled back, where its parent reads them.
 
void solve_divide_subtree (std::vector< divide_node > &nodes, std::size_t index, merge_workspace &work)
 Solves the block index of nodes and everything below it, each block right after its halves.
 
void divide_and_conquer (double *d, const double *e, std::size_t n, buffer< double > &q)
 The eigenvalues, ascending, of the symmetric tridiagonal with diagonal d[0…n-1] and subdiagonal e[0…n-2] into d, and its eigenvectors as the columns of the row-major q (n × n), by Cuppen's divide and conquer: the tridiagonal is two halves plus a rank-one coupling, each half is diagonalised the same way (by implicit QL up to divide_leaf), and merge_rank_one joins them, each half into its own diagonal block of q, so the joins copy no half and leave the blocks off the diagonal as they found them until the join of their two halves fills them.
 
template<typename T >
void multiply_real_left (const buffer< double > &left, buffer< T > &zt, std::size_t n)
 zt ← left · zt, n × n, row-major, for any other element type: the real factor converted.
 
void multiply_real_left (const buffer< double > &left, buffer< double > &zt, std::size_t n)
 zt ← left · zt, n × n, row-major, for a real left factor.
 
void multiply_real_left (const buffer< double > &left, buffer< std::complex< double > > &zt, std::size_t n)
 zt ← left · zt for a complex zt: the real and imaginary parts by a real product each.
 
template<typename T >
buffer< T > tridiagonal_divide_and_conquer (buffer< double > &d, buffer< double > &e, const tridiagonal_reduction< T > &reduction, std::size_t n)
 The eigenvalues of the tridiagonal (d, e) of reduction into d, and the matrix's eigenvectors as the rows of the result, by divide and conquer (LAPACK's dstedc, then dormtr): most of the work in matrix products. Up to divide_leaf, tridiagonal_ql on Mᵀ itself.
 
void jacobi_rotate (buffer< double > &a, buffer< double > &v, std::size_t n, std::size_t p, std::size_t q)
 One Jacobi rotation of the n × n a zeroing a[p][q], accumulated into v.
 
void cyclic_jacobi (buffer< double > &a, buffer< double > &v, std::size_t n, std::size_t max_sweeps)
 Cyclic Jacobi on the full symmetric a, the rotations accumulated into v.
 
std::vector< std::size_t > ascending_order (const buffer< double > &diagonal)
 The indices of diagonal by ascending value.
 
template<typename T >
buffer< T > columns_by_value (const buffer< T > &zt, const std::vector< std::size_t > &order, std::size_t n)
 The n × n row-major matrix whose column c is row order[c] of zt (the eigenvectors as rows): by tiles, the rows of each on its own core. Element by element, each write was a row apart, and the copy of order 1024 took a tenth of an eigh on six x86-64 cores.
 
template<typename T >
void reduce_pencil_columns (T *c, std::size_t ldc, const T *l, std::size_t ldl, std::size_t n)
 The lower triangle of the n × n c (row stride ldc) taken to L⁻¹ C L⁻ᴴ for the lower-triangular l (row stride ldl), one column after another, as LAPACK's sygs2 (type 1): each diagonal element divided by L's squared, the column below it scaled, corrected by half of it twice around the rank-2 update of the rest, then solved against the rest of L. c's other triangle is neither read nor written.
 
template<typename T >
bool lower_triangle_product (std::size_t, std::size_t, const T *, const T *, T *, std::size_t)
 C += A·B on the lower triangle of the m × m c (row stride ldc), a m × depth and b depth × m row-major, depth two blocks of pencil_block (past gemm_min_depth), where the matrix product kernel packs a product of that size: on the triangle's tiles alone (gemm_triangle).
 
bool lower_triangle_product (std::size_t m, std::size_t depth, const double *a, const double *b, double *c, std::size_t ldc)
 lower_triangle_product for double.
 
template<typename T >
void reduce_pencil (T *c, const T *l, std::size_t n)
 The lower triangle of the Hermitian n × n c (row-major) taken to L⁻¹ C L⁻ᴴ for the lower-triangular l (n × n, row-major), as LAPACK's sygst (type 1, lower) by blocks of pencil_block columns: a block's diagonal by reduce_pencil_columns; the rows below it solved against its L₁₁ᴴ from the right, corrected by half of L₂₁ C₁₁ twice around the trailing rank-2k update C₂₂ −= C₂₁ L₂₁ᴴ + L₂₁ C₂₁ᴴ (its lower triangle); once every block is through, each one's rows solved against its L₂₂, the blocks on the threads together. Some n³ operations where two whole triangular solves took 2 n³. c's other triangle is neither read nor written.
 
template<typename T >
buffer< T > pencil_standard_form (buffer< T > y, const T *lower, std::size_t n, bool blocks)
 C = L⁻¹ A L⁻ᴴ whole, y holding A's lower triangle (n × n, row-major, the diagonal real) and lower L: by blocks (reduce_pencil, C's lower triangle, then its upper from it) where blocks, else by two whole triangular solves, L⁻¹ A and L⁻¹ (L⁻¹ A)ᴴ.
 
double_double two_sum (double a, double b) noexcept
 a + b exactly, as a sum and its rounding error (Knuth's two-sum).
 
double_double fast_two_sum (double a, double b) noexcept
 a + b exactly where |a| ≥ |b| or a is 0 (Dekker's fast two-sum).
 
double_double dd_add (double_double x, double_double y) noexcept
 The sum of two double-double numbers.
 
double_double dd_mul (double_double x, double_double y) noexcept
 The product of two double-double numbers, its cross terms by explicit fused multiply-adds.
 
double_double dd_div (double_double x, double d) noexcept
 x divided by the integer d (exact in a double).
 
const std::array< double_double, 30 > & inverse_factorials ()
 The terms of dd_sin_cos' series: 1/k! for k = 0 to 29, each from the last by dd_div.
 
std::pair< double_double, double_double > dd_sin_cos (double_double t)
 The cosine and sine of t, in [0, π/4], as double-double numbers: their Taylor series in t² by Horner's rule to the term of degree 28 and 29, whose next terms are under 10⁻³² of the sum there.
 
double unsigned_zero (double x) noexcept
 x with a zero's sign made positive: the twiddle factors' zeros are +0, as their exact values.
 
double_double dd_negate (double_double x) noexcept
 x negated, both parts.
 
octant_share reduce_to_octant (std::size_t k, std::size_t n) noexcept
 The octant_share of e^{−2πi k/n}.
 
std::pair< double_double, double_double > place_by_octant (std::size_t octant, double_double c, double_double s) noexcept
 e^{−2πi k/n} from the cosine c and sine s of its angle within the octant octant (reduce_to_octant): as double-double parts, real then imaginary.
 
std::pair< double_double, double_double > octant_sin_cos (std::size_t share, std::size_t n)
 The cosine and sine of (π/4)·share/n by dd_sin_cos.
 
std::pair< double_double, double_double > unit_root_parts (std::size_t k, std::size_t n)
 e^{−2πi k/n} as double-double parts: reduce_to_octant, octant_sin_cos, place_by_octant.
 
std::complex< double > unit_root (std::size_t k, std::size_t n)
 unit_root_parts rounded once to double, a zero's sign positive as the exact value's.
 
template<std::floating_point R>
R rounded_alone (R x) noexcept
 x, held in a register as its rounded value: GCC on arm64 with the complex-arithmetic unit (ARMv8.3's FCMLA: -mcpu=apple-m1, Neoverse V1) recognizes a·b − c·d, a·d + c·b as a complex product and fuses one product into the other's sum whatever -ffp-contract says; elsewhere x as it is.
 
template<std::floating_point R>
std::complex< R > rounded_product (const std::complex< R > &p, const std::complex< R > &q) noexcept
 p · q of complex numbers, each product rounded apart from the sum it enters (statements of their own, rounded_alone).
 
template<std::floating_point R>
std::complex< R > fused_product (const std::complex< R > &w, const std::complex< R > &v) noexcept
 w · v of complex numbers on the vector paths: w.re·v.re − w.im·v.im and w.re·v.im + w.im·v.re, each the second product rounded, the first fused with the sum by std::fma.
 
template<bool Fused, std::floating_point R>
requires Fused
std::complex< R > twiddled (const std::complex< R > &w, const std::complex< R > &v) noexcept
 A butterfly's factor times v on the vector paths: fused_product.
 
template<bool Fused, std::floating_point R>
requires (!Fused)
std::complex< R > twiddled (const std::complex< R > &w, const std::complex< R > &v) noexcept
 A butterfly's factor times v on the portable path: rounded_product.
 
std::vector< std::size_t > fft_radices (std::size_t n)
 The radix of each Stockham pass of n, or none where n has a prime factor past fft_largest_radix, its powers of two first (4 first, its stride 1, then 16 while 16 points remain, then 8 or 4), then its factors 3, 5 and 7, then each other prime in ascending order, then a lone factor 2 last, where its pass has no factor.
 
template<std::floating_point R>
small_radix_constants< R > make_small_radix_constants ()
 small_radix_constants from the roots of order 3 and 5 (unit_root, whose imaginary parts are −sin).
 
template<std::floating_point R>
void forward_fft (const fft_plan< R > &plan, std::complex< R > *data, std::complex< R > *work, bool fused, bool in_work)
 The forward transform of the plan's length into data, work its fft_plan::work, its Stockham passes, or Bluestein's convolution, or its four steps, the products fused where fused (fft_fuses): the input in data, or in work where in_work (only where fft_starts_in_work).
 
template<std::floating_point R>
bool fft_starts_in_work (const fft_plan< R > &plan) noexcept
 Whether a transform under plan, its input placed in its working space (forward_fft's in_work), ends in its data without a copy: an odd number of Stockham passes, no Bluestein's convolution nor four steps.
 
std::size_t four_step_columns (std::size_t n, std::size_t bytes)
 The columns n1 of a transform of n points of bytes each in four steps: the largest divisor of n at most √n, n of at least fft_four_step_least bytes; else 0, one transform of passes. n is taken in passes (its prime factors at most fft_largest_radix: make_fft_plan settles Bluestein's lengths first). Such an n, 2¹⁸ and more (√n at least 512), has a divisor from 16 to √n, so that neither side is short: a prime factor past 15 (at most 61), or else the product of its smallest prime factors first past 15 (at most 15·13).
 
std::size_t bluestein_length (std::size_t n)
 The points of Bluestein's convolution of n points: the power of two at least 2n − 1, or the shortest length of factors 2, 3 and 5 at least 2n − 1 where it is at most 3/5 of that power and the power at least 1024 (passes of 3 and 5 cost more a point than those of 16: ten AVX-512 cores, 4099 points 156 µs over 16 384, 95 over 8640; 1009 points 18 µs over 2048, 30 over 2025; 67 points 1.3 µs over 256, 1.9 over 135).
 
std::size_t largest_prime_factor (std::size_t n) noexcept
 The largest prime factor of n, at least 2.
 
template<std::floating_point R>
fft_plan< R > make_fft_plan (std::size_t n)
 The fft_plan of length n, its roots computed on the threads (each the same on any).
 
template<std::floating_point R>
std::shared_ptr< const fft_plan< R > > shared_fft_plan (std::size_t n)
 The shared fft_plan of length n, built on first use and kept among the fft_plans_kept most recently used.
 
template<bool Fused, typename C >
void radix_four_pass_body (std::size_t current, std::size_t stride, const C *in, C *out, const fft_root_t< C > *twiddles) noexcept
 One Stockham pass of radix 4 over in into out: current points at stride stride, so current · stride is the length. Element q + stride · (p + j · current/4) of the input is the j-th input of butterfly (p, q), whose outputs go to q + stride · (4p + j), the last three times w^{jp} (twiddled), none for p = 0.
 
template<bool Fused, std::floating_point R>
void radix_four_pass (std::size_t current, std::size_t stride, const std::complex< R > *in, std::complex< R > *out, const std::complex< R > *twiddles) noexcept
 radix_four_pass_body of complex numbers, its products as Fused.
 
template<typename C >
void four_point (const C &v0, const C &v1, const C &v2, const C &v3, C *y) noexcept
 A 4-point transform of v0 to v3 into y: the radix-4 butterfly's sums, in its order.
 
template<bool Fused, typename C >
void radix_sixteen_pass_body (std::size_t current, std::size_t stride, const C *in, C *out, const fft_root_t< C > *twiddles, const std::array< fft_root_t< C >, 16 > &sixteenths) noexcept
 One Stockham pass of radix 16 over in into out, as radix_four_pass is of radix 4: each butterfly a 16-point transform as two of 4 points, x_{j1 + 4·j2} first transformed over j2, each result (j1, k2) then times e^{−2πi j1·k2/16} (sixteenths; −i exactly where j1·k2 is 4), then over j1 into output 4·k1 + k2; output k of butterfly p times w^{kp}, none for p = 0. Fused as twiddled.
 
template<bool Fused, std::floating_point R>
void radix_sixteen_pass (std::size_t current, std::size_t stride, const std::complex< R > *in, std::complex< R > *out, const std::complex< R > *twiddles, const std::array< std::complex< R >, 16 > &sixteenths) noexcept
 radix_sixteen_pass_body of complex numbers, its products as Fused.
 
template<std::floating_point R>
std::complex< R > scaled (R c, const std::complex< R > &v) noexcept
 c · v of a real c and a complex v: each part's product, rounded.
 
template<bool Fused, std::floating_point R>
requires Fused
std::complex< R > scaled_add (const std::complex< R > &s, R c, const std::complex< R > &v) noexcept
 s + c · v of a complex s, a real c and a complex v on the vector paths: each part one fused multiply-add.
 
template<bool Fused, std::floating_point R>
requires (!Fused)
std::complex< R > scaled_add (const std::complex< R > &s, R c, const std::complex< R > &v) noexcept
 scaled_add on the portable path: the product rounded first.
 
template<bool Fused, typename C >
void any_radix_pass_body (std::size_t radix, std::size_t current, std::size_t stride, const C *in, C *out, const fft_root_t< C > *twiddles, const fft_root_t< C > *roots) noexcept
 One Stockham pass of an odd radix radix, as radix_four_pass is of 4, by the symmetry of its roots (radix_five_pass_body's, any odd r): a_j = x_j + x_{r−j}, b_j = x_j − x_{r−j} (j from 1 to h = (r − 1)/2); X_0 = x_0 + a_1 + … + a_h, and for k from 1 to h, A_k = x_0 + Σ_j cos(2πjk/r)·a_j, B_k = Σ_j sin(2πjk/r)·b_j, each sum in ascending j, its products fused as Fused (scaled_add; the cosines and sines roots' parts, jk reduced mod r by additions), X_k = A_k − i·B_k, X_{r−k} = A_k + i·B_k; output k then times w^{kp}, none for p = 0. Radices 3 and 5 so computed give their own butterflies' bits. Inlined into each caller, so that the x86-64 fused one is built for FMA.
 
template<bool Fused, typename C >
void radix_three_pass_body (std::size_t current, std::size_t stride, const C *in, C *out, const fft_root_t< C > *twiddles, const small_radix_constants< fft_real_t< C > > &k) noexcept
 One Stockham pass of radix 3, as any_radix_pass_body is of any: t = x1 + x2, v = x0 − t/2 (exact halving, one rounding), u = (√3/2)·(x1 − x2); X0 = x0 + t, X1 = v − i·u, X2 = v + i·u, then times w^{kp} as Fused. Inlined into each caller.
 
template<bool Fused, typename C >
void radix_five_pass_body (std::size_t current, std::size_t stride, const C *in, C *out, const fft_root_t< C > *twiddles, const small_radix_constants< fft_real_t< C > > &k) noexcept
 One Stockham pass of radix 5: t1 = x1 + x4, t2 = x2 + x3, t3 = x1 − x4, t4 = x2 − x3; a1 = x0 + c1·t1 + c2·t2, a2 = x0 + c2·t1 + c1·t2, b1 = s1·t3 + s2·t4, b2 = s2·t3 − s1·t4 (each sum in that order, its products fused as Fused); X0 = x0 + t1 + t2, X1 = a1 − i·b1, X4 = a1 + i·b1, X2 = a2 − i·b2, X3 = a2 + i·b2, then times w^{kp}. Inlined into each caller.
 
template<bool Fused, typename C >
void radix_eight_pass_body (std::size_t current, std::size_t stride, const C *in, C *out, const fft_root_t< C > *twiddles, fft_real_t< C > half_root_two) noexcept
 One Stockham pass of radix 8, as radix_four_pass is of 4: a_j = x_j + x_{j+4}, b_j = x_j − x_{j+4} (j below 4), b_j then times e^{−2πi j/8} (√½·(re + im, im − re), (im, −re), √½·(im − re, −(re + im)): each sum then its product, no fused operation, so the same bits on every path); the even outputs a's four_point, the odd ones b's; output k then times w^{kp} as Fused, none for p = 0. half_root_two is √½.
 
template<bool Fused, std::floating_point R>
void any_radix_pass (std::size_t radix, std::size_t current, std::size_t stride, const std::complex< R > *in, std::complex< R > *out, const std::complex< R > *twiddles, const std::complex< R > *roots, const small_radix_constants< R > &small) noexcept
 any_radix_pass_body, its products as Fused.
 
template<typename C >
void radix_two_pass (std::size_t stride, const C *in, C *out) noexcept
 The last pass of an odd power of two: butterflies of radix 2 at stride stride, no factor.
 
template<std::floating_point R, typename C , typename Four , typename Sixteen , typename Other >
void stockham_passes (const fft_plan< R > &plan, C *data, C *work, Four four, Sixteen sixteen, Other other, bool from_work=false) noexcept
 The Stockham passes of the plan over data, work as long, the result in data: radix 4 by four (radix_four_pass's arguments), 16 by sixteen (radix_sixteen_pass's), 2 by radix_two_pass (always the last pass), any other by other (any_radix_pass's: 3 and 5 by their own butterflies). The input in data, or from_work in work (fft_starts_in_work: an odd number of passes then ends in data, where from data the last pass wrote work, copied back).
 
bool fft_fuses () noexcept
 Whether a transform's butterflies fuse their products (fused_product): on every path past the portable.
 
template<bool Fused, std::floating_point R>
void bluestein_body (const fft_plan< R > &plan, std::complex< R > *data, std::complex< R > *work)
 Bluestein's transform of the plan's length in place in data, work its fft_plan::work, the input times the chirp, padded to M; its transform times the chirp's spectrum; the inverse transform of that (conj(fft(conj ·)), then 1/M, exact for a power of two), times the chirp. Products as Fused. Inlined into each caller.
 
template<bool Fused, std::floating_point R>
void twiddle_run_body (std::complex< R > *t, const std::complex< R > *w, std::size_t count) noexcept
 t[i] = w[i] · t[i] for count elements, as Fused (twiddled). Inlined into each caller.
 
template<std::floating_point R>
void twiddle_run (std::complex< R > *t, const std::complex< R > *w, std::size_t count, bool fused) noexcept
 twiddle_run_body on the path in use.
 
template<std::floating_point R>
void four_step_fft (const fft_plan< R > &plan, const std::complex< R > *in, std::complex< R > *out, std::complex< R > *work, bool fused, bool inverse=false, R scale=R(1))
 The transform of the plan's four_step_plan of in into out (in itself or apart from it), work its fft_plan::work (the columns' transforms, then the rows'), the products fused where fused; the inverse's conj(fft(conj x)) and times scale as transform_axis's lines.
 
template<std::floating_point R>
R fft_scale (fft_norm norm, bool inverse, std::size_t n)
 The scale of a transform's output for norm, forward or not, of length n: 1 where none.
 
template<std::floating_point R>
std::size_t fft_lane_groups (const fft_plan< R > &, bool, std::size_t, bool) noexcept
 No registers of lines.
 
template<typename T , std::floating_point R>
void fft_lines_together (const fft_plan< R > &, const T *, std::size_t, std::size_t, std::size_t, std::complex< R > *, std::size_t, std::size_t, bool, R) noexcept
 Never called: no registers of lines.
 
template<std::floating_point R>
void columns_fft (const fft_plan< R > &plan, const std::complex< R > *data, std::size_t columns, std::complex< R > *to, std::size_t to_rows, bool fused, bool conjugated)
 The transforms of plan's length along each of the columns columns of data into those of to (data's rows of columns, to's to_rows apart, as many as the plan's points): registers of neighbouring columns where they fill one (fft_lane_groups), the rest up to fft_column_block of them at a time, each row read and written as that many neighbours; the threads share them. Where conjugated, conj(fft(conj x)).
 
template<std::floating_point R>
void twiddled_rows_fft (const fft_plan< R > &plan, std::complex< R > *data, std::size_t rows, std::size_t apart, const std::complex< R > *twiddles, bool fused, std::complex< R > *out, bool inverse, R scale)
 Each of the rows rows of data (apart from one another; conjugated first where inverse) times its run of twiddles (each as long as the rows), then transformed in place (plan's length), then written down a column of out, row r's element k to out[r + rows·k], conjugated where inverse and times scale: fft_four_step_band rows a task, still in the caches, each element of out written among that many neighbours.
 
template<typename T >
fft_source< T > source_of (const tensor< T > &a)
 a's elements, where they lie.
 
template<typename T , typename F >
auto transformed_along (const tensor< T > &a, std::ptrdiff_t axis, const F &f) -> decltype(f(source_of(a), axis))
 f (f(source, axis), a transform along axis, negative from the last) of a: a transpose's elements as they lie where as_laid_along reads them (each line transformed on its own, its bits the same); any other tensor's row-major.
 
template<typename C >
void fft_load_line (const C *in, std::size_t kept, std::size_t points, bool inverse, C *to) noexcept
 The first kept of points elements of a line, in, into to, conjugated where inverse, then zeros. Its bounds and flags by value: read through a caller's references, each would be loaded again at every element, the writes to to possibly aliasing them, and the loop would not vectorise.
 
template<typename C >
void fft_finish_line (C *line, std::size_t points, bool inverse, typename C::value_type scale) noexcept
 A transformed line of points elements, line, conjugated back where inverse, then times scale where it is not 1; by value for the reason fft_load_line gives.
 
template<split_complex T>
tensor< T > transform_axis (const fft_source< T > &a, const std::optional< std::size_t > &n, std::ptrdiff_t axis, fft_norm norm, bool inverse)
 The transform of complex a along axis, n points (the axis cut or padded with zeros to it, as NumPy's), forward or inverse (conj(fft(conj x)), exact), scaled for norm. Each line gathered, transformed and scattered on its own, the lines shared among the threads: a line's bits are its own. (A real input takes real_transform_of.)
 
template<std::floating_point R>
std::shared_ptr< const real_fft_plan< R > > shared_real_fft_plan (std::size_t n)
 The real_fft_plan of n, shared as shared_fft_plan shares the complex ones.
 
template<std::floating_point R>
std::size_t no_registers (std::size_t, const std::complex< R > *, const std::complex< R > *, std::complex< R > *) noexcept
 No recombination by registers: the scalar loop's from k = 1.
 
template<std::floating_point R>
void real_forward_whole (std::size_t n, const R *x, std::complex< R > *X, bool fused)
 The first n/2 + 1 frequencies of the n real points x into X by the complex transform of n points, the products fused where fused: an even n's line holding an infinity or a NaN, which the recombination of its pairs would turn into NaN (∞ − ∞) where the complex transform carries it.
 
template<std::floating_point R>
void real_inverse_whole (std::size_t n, const std::complex< R > *X, R *x, R factor, bool fused)
 The n real points x from the n/2 + 1 frequencies X by the complex transform of n points (conj(fft(conj ·)) of the Hermitian spectrum, the imaginary parts of X_0 and X_{n/2} read as 0), times factor: an even n's spectrum holding an infinity or a NaN (real_forward_whole).
 
template<bool Fused, std::floating_point R, typename Registers >
void real_forward_body (const real_fft_plan< R > &plan, const R *x, std::complex< R > *X, std::complex< R > *work, Registers registers)
 The transform of n real points x (together) into X (n/2 + 1 complex numbers), work holding n/2 + 1 complex numbers beside the plan's work. An even n = 2h: the h complex numbers x_{2j} + i·x_{2j+1} transformed, then X_k = E_k + w^k·O_k, E_k = (Z_k + conj Z_{h−k})/2, O_k = −i·(Z_k − conj Z_{h−k})/2 (exact halvings and products by ±i; w^k·O_k as Fused), registers (h, Z, roots, X) computing the first of them and returning the next k. An odd n: the complex transform of the points, its first n/2 + 1 outputs. Inlined into each caller.
 
template<bool Fused, std::floating_point R, typename Registers >
void real_inverse_body (const real_fft_plan< R > &plan, const std::complex< R > *X, R *x, std::complex< R > *work, R scale, Registers registers)
 The inverse of real_forward_body, unscaled save its own halvings: n real points x from the n/2 + 1 complex numbers X (the imaginary parts of X_0 and, for an even n, X_{n/2} read as 0, as NumPy's). An even n = 2h: Z_k = E_k + i·O_k, E_k = (X_k + conj X_{h−k})/2, O_k = conj(w^k)·(X_k − conj X_{h−k})/2, then Z's inverse transform (conj(fft(conj ·))), its parts x_{2j} and x_{2j+1}, times scale, registers (h, X, roots, conj Z) computing the first conj Z_k and returning the next k; an odd n: the Hermitian spectrum of n points, its inverse transform's real parts times scale. Inlined into each caller.
 
template<std::floating_point R>
void real_forward (const real_fft_plan< R > &plan, const R *x, std::complex< R > *X, std::complex< R > *work, bool fused)
 real_forward_body on the path in use.
 
template<std::floating_point R>
void real_inverse (const real_fft_plan< R > &plan, const std::complex< R > *X, R *x, std::complex< R > *work, R scale, bool fused)
 real_inverse_body on the path in use.
 
template<typename In , typename Out , typename C , typename Line >
tensor< Out > lines_along (const fft_source< In > &a, std::size_t index, std::size_t in_points, fft_output< Out > &&result, std::size_t work, Line line)
 A transform of each line of a along axis index into the lines of result: each line gathered (its first in_points elements, padded with zeros), passed with its output line, both together, and the thread's working space of work elements of C to line, then scattered; a line together and not padded passed where it lies, its output line too. The lines shared among the threads. result holds elements: its callers return an empty one before they plan.
 
std::size_t fft_axis_index (std::ptrdiff_t axis, std::size_t rank, const char *who)
 axis of a tensor of rank rank from 0, NumPy's negative axes read from the end.
 
template<std::floating_point R>
tensor< std::complex< R > > real_input_transform (const fft_source< R > &a, std::size_t index, std::size_t points, bool inverse, R scale)
 The transform of real a along the axis index into all n frequencies, or the inverse transform where inverse: each line's real transform (real_forward, its n/2 + 1 non-negative frequencies), the rest by Hermitian symmetry, X_{n−k} = conj X_k, exactly; a length whose complex transform takes four steps, which half of it does not, by that complex transform, on the threads. The inverse of a real line the conjugate of its transform. Then times scale.
 
template<std::floating_point T>
tensor< std::complex< T > > real_transform_of (const fft_source< T > &a, const std::optional< std::size_t > &n, std::ptrdiff_t axis, fft_norm norm, bool inverse)
 fft or ifft (inverse) of the real elements a: fft of a real tensor's computation, of elements a caller lends.
 
template<std::floating_point T>
tensor< std::complex< T > > rfft_of (const fft_source< T > &a, const std::optional< std::size_t > &n, std::ptrdiff_t axis, fft_norm norm)
 rfft's computation, of real elements a caller lends.
 
template<std::floating_point T>
tensor< T > irfft_of (const fft_source< std::complex< T > > &a, const std::optional< std::size_t > &n, std::ptrdiff_t axis, fft_norm norm)
 irfft's computation, of complex elements a caller lends.
 
template<typename T >
tensor< T > rolled (const tensor< T > &a, const std::vector< std::size_t > &shifts)
 a rolled along every axis, element j of axis d to (j + shifts[d]) mod n_d, in one copy: each row of the result along its last axis read from its source row, in two contiguous pieces; the rows shared among the threads.
 
std::size_t eighth_inverse (std::size_t n) noexcept
 The inverse of 8 modulo the odd n, in [0, n): by Euclid's algorithm extended.
 
template<std::floating_point R>
cosine_line< R > make_cosine_line (int type, std::size_t n)
 The cosine_line of type on n points, its plan and roots formed: each root e^{−iπa/d} (d of n, 2n or 4n) as unit_roots of 8n rounds e^{−2πi(4n·a/d)/8n}, in double-double arithmetic on the threads, so the same on every platform.
 
template<std::floating_point R>
std::shared_ptr< const cosine_line< R > > shared_cosine_line (int type, std::size_t n)
 The shared cosine_line of type on n points, built on first use and kept among the fft_plans_kept most recently used, as shared_fft_plan keeps its plans.
 
template<std::floating_point R>
tensor< R > trig_transform (const fft_source< R > &source, trig_family family, int type, const std::optional< std::size_t > &n, std::ptrdiff_t axis, fft_norm norm, const std::optional< bool > &orthogonalize, bool inverse)
 dct, idct, dst or idst (family, inverse) of source along axis, as scipy.fft's: the type computed (an inverse's of types 2 and 3 the other), on each line of n points (the axis cut or padded), the sines by the cosines: type 1 the real transform of the odd extension (2(n + 1) points), type 2 the cosines of the points of alternate signs, in reverse order; types 3 and 4 those of the points reversed, of alternate signs. Scaled as norm says, orthogonalize (by default where norm is ortho) the first or last points and values weighted by √2 as SciPy's. The elements of source are read where they lie.
 
template<std::floating_point R>
tensor< R > trig_transform (const tensor< R > &a, trig_family family, int type, const std::optional< std::size_t > &n, std::ptrdiff_t axis, fft_norm norm, const std::optional< bool > &orthogonalize, bool inverse)
 trig_transform of the tensor a.
 
bool gemm_path_usable (gemm_path path) noexcept
 Whether path can run here.
 
gemm_path best_gemm_path (gemm_path most) noexcept
 The most capable path that can run here, most at the highest.
 
std::atomic< gemm_path > & gemm_path_cap () noexcept
 The most capable path products may take, as numerical::limit_gemm_path set it: none is probed until a product (or a question) reaches it, so a cap below the AMX keeps its instructions from ever running.
 
std::atomic< std::uint8_t > & gemm_path_found () noexcept
 The path gemm_path_current found last, packed with the cap it was found under (cap · 8 + path); 0xFF before any.
 
gemm_path gemm_path_current () noexcept
 The path the next double product takes: the most capable one under the cap that runs here (each path's probe is made once). Kept with its cap, one load, where each call walked the paths down from the cap through their probes' guards: every small step of a decomposition asks (an LU of order 100 through dgetrf_ on one AVX2 core spent 5 % of its time here). A race stores the same value twice.
 
bool rows_fuse () noexcept
 Whether the tile a double or float product past the plain loop runs on fuses each term (gemm_rows rounds its own so): double on arm64's NEON and AMX paths, float on the NEON tile there. The x86-64 tiles' double takes term_rounding::x86_fused; the vector tile, and so the portable path, rounds as the compiler does, as the plain loop then does. On arm64 every cap but the portable one runs the NEON or the AMX path, so the cap alone answers: one load, where the path in use took an order-4 product on one arm64 core from 10.6 to 13.3 ns (the cap, 11.5).
 
void subtract_product (double *y, const double &f, const double *x, std::size_t count)
 y[i] −= f · x[i] for double on the widest x86-64 units the matrix product's kernel may use (gemm_path_current), each element one fused multiply-add there; elsewhere subtract_product's template. The decompositions' row updates take it: a wheel built for the baseline x86-64 holds two double per vector and fuses none.
 
void subtract_product (double *y, const double *x, const double &f, std::size_t count)
 y[i] −= x[i] · f for double, as the other (a real product is the same either way round).
 
void subtract_products (double *y, const double *f, const double *x, std::size_t ldx, std::size_t terms, std::size_t count)
 y[i] -= Σₜ f[t] · x[(t · ldx) + i] for double, the units chosen as subtract_product chooses them, the elements held in registers through the terms there.
 
void subtract_rows (double *y, std::size_t ldy, const double *f, std::size_t ldf, const double *x, std::size_t ldx, std::size_t terms, std::size_t count)
 subtract_rows for double, the units chosen as subtract_products chooses them: each element the value subtract_products gives it.
 
void subtract_products (std::complex< double > *y, const std::complex< double > *f, const std::complex< double > *x, std::size_t ldx, std::size_t terms, std::size_t count)
 subtract_products for std::complex<double>: on AVX-512 or AVX2 with FMA where the matrix product's kernel runs there, each part of each term one fused multiply-add (Re f · x, then Im f times the swapped parts), the elements in registers through the terms; elsewhere the template, whole complex products as the compiler rounds them.
 
void subtract_product (std::complex< double > *y, const std::complex< double > &f, const std::complex< double > *x, std::size_t count)
 y[i] −= f · x[i] for std::complex<double>: one term of subtract_products on AVX-512 or AVX2 where the matrix product's kernel runs there, else the template.
 
void subtract_product (std::complex< double > *y, const std::complex< double > *x, const std::complex< double > &f, std::size_t count)
 y[i] −= x[i] · f for std::complex<double>, as the other (the template keeps the expression's order).
 
void add_product (double *y, const double &f, const double *x, std::size_t count)
 y[i] += f · x[i] for double, as subtract_product.
 
void add_product (std::complex< float > *y, const std::complex< float > &f, const std::complex< float > *x, std::size_t count)
 y[i] += f · x[i] for complex float: on the x86-64 paths by AVX2 vectors of its arithmetic (avx2_scaled_add_c32), the same bits as add_product's template, which runs elsewhere.
 
void add_product (std::complex< double > *y, const std::complex< double > &f, const std::complex< double > *x, std::size_t count)
 add_product for complex double (avx2_scaled_add_c64 on the x86-64 paths).
 
double dot_product (const double *x, const double *y, std::size_t count)
 Σ x[i] · y[i] for double, as subtract_product chooses its units (in another order of summation there, as fixed).
 
float dot_product (const float *x, const float *y, std::size_t count)
 Σ x[i] · y[i] for float, on the x86-64 units with AVX2 or AVX-512 (avx2_dot, avx512_dot of float), elsewhere the portable vectors' (in another order of summation there, as fixed).
 
template<split_complex C>
C complex_dot_product (const C *x, const C *y, std::size_t count)
 Σ x[i] · y[i] of complex numbers: on the x86-64 units with AVX2 (the AVX-512 path too) their fused chains (avx2_complex_dot), elsewhere the native vectors' (in another order of summation there, as fixed).
 
std::complex< double > dot_product (const std::complex< double > *x, const std::complex< double > *y, std::size_t count)
 dot_product for std::complex<double>, through complex_dot_product.
 
std::complex< float > dot_product (const std::complex< float > *x, const std::complex< float > *y, std::size_t count)
 dot_product for std::complex<float>, through complex_dot_product.
 
template<split_complex C>
void complex_row_dots (const C *m, std::size_t ld, std::size_t rows, const C *x, std::size_t count, C *out)
 row_dots of complex numbers: each row by complex_dot_product, as chunked_dot sums a row.
 
void row_dots (const std::complex< double > *m, std::size_t ld, std::size_t rows, const std::complex< double > *x, std::size_t count, std::complex< double > *out)
 row_dots for std::complex<double>, through complex_row_dots.
 
void row_dots (const std::complex< float > *m, std::size_t ld, std::size_t rows, const std::complex< float > *x, std::size_t count, std::complex< float > *out)
 row_dots for std::complex<float>, through complex_row_dots.
 
void row_dots (const double *m, std::size_t ld, std::size_t rows, const double *x, std::size_t count, double *out)
 row_dots for double: on the x86-64 units each row by their dot_product (wider than four rows of the portable vectors), elsewhere four rows at a time.
 
void row_dots (const float *m, std::size_t ld, std::size_t rows, const float *x, std::size_t count, float *out)
 row_dots for float: on the x86-64 units each row by their float dot product (dot_product), elsewhere the portable vectors'.
 
template<typename T >
void gemm_kernel (std::size_t kc, const T *ap, const T *bp, std::size_t ldb, gemm_tile< T > &tile)
 tile += Aₚ·Bₚ over one panel, for any scalar.
 
template<std::size_t Mr, typename T >
void gemm_pack_a (const T *a, std::size_t lda, std::size_t rows, std::size_t kc, T *out)
 Copies rows × kc of A (from row-major, leading dimension lda) into slivers of Mr rows, each stored column by column; a last partial sliver is padded with zeros.
 
template<std::size_t Nr, typename T >
void gemm_pack_b (const T *b, std::size_t ldb, std::size_t kc, std::size_t cols, T *out)
 Copies kc × cols of B (from row-major, leading dimension ldb) into slivers of Nr columns, each stored row by row; a last partial sliver is padded with zeros.
 
template<std::size_t Nr, typename T >
void gemm_pack_bt (const T *bt, std::size_t ldbt, std::size_t kc, std::size_t cols, T *out)
 gemm_pack_b of B = Bᵀᵀ, from the row-major bt (cols × kc, leading dimension ldbt): each column of a sliver is a row of bt, read along it; a last partial sliver is padded with zeros.
 
template<typename T >
const T * rows_in_place (const T *x, std::size_t ld, std::size_t &ld_out) noexcept
 A row-major rows × cols operand for the plain loop, x itself (ld apart).
 
template<typename T >
const T * rows_from_transpose (const T *x, std::size_t ld, std::size_t rows, std::size_t cols, buffer< T > &scratch, std::size_t &ld_out)
 A row-major rows × cols operand for the plain loop, copied into scratch from its transpose x (cols × rows, ld apart).
 
template<std::size_t Mr, typename T >
void gemm_pack_at (const T *at, std::size_t ldat, std::size_t rows, std::size_t kc, T *out)
 The slivers of Mr rows of A (rows × kc), column by column, from its transpose at (kc × rows row-major, ldat apart): each column of a sliver is a run of a row of at; a last partial sliver is padded with zeros, as gemm_pack_a lays them out.
 
template<typename T >
T rounded_apart_add (T sum, T a, T b) noexcept
 sum + a·b as the vector tile rounds it (vector_tile_kernel), so that the plain loop's elements are its own: under Clang's -ffp-contract=on one expression, fused in both; under GCC's, whose vector extensions fuse nothing while a scalar statement's multiply-add fuses, the product apart, a statement of its own.
 
template<typename R >
std::complex< R > rounded_apart_add (std::complex< R > sum, std::complex< R > a, std::complex< R > b) noexcept
 rounded_apart_add of complex numbers: the product in line (operator_times, the operator's own rounding, unfused), where it was a call that took a sixth of a complex LU.
 
bool by_rows (std::size_t m, std::size_t n, std::size_t k)
 Whether gemm runs an m × k by k × n product by rows (gemm_rows): gemm_rows_columns columns or fewer; or, from narrow_group rows, gemm_rows_work multiply-adds or fewer, or gemm_rows_share a thread where its blocks of rows give at least half the threads one each (fewer, and the packed product's tasks, which split the columns too, are faster: 12 × 302 by 302 × 389 on ten AVX-512 cores, 1.6 times). Where the target's packed tiles do not keep a small product ahead.
 
template<typename T >
T rounded_term (rounding_is< term_rounding::compiler >, T sum, T a, T b) noexcept
 sum + a·b as the compiler rounds it (rounded_apart_add).
 
template<typename T >
T rounded_term (rounding_is< term_rounding::fused >, T sum, T a, T b) noexcept
 sum + a·b as one fused multiply-add.
 
template<std::size_t R, std::size_t N, term_rounding Mode, typename T >
void narrow_block (std::size_t k, const T *a, std::size_t lda, const T *b, std::size_t ldb, T *c, std::size_t ldc, c_start start) noexcept
 C += A·B for R rows and N columns of C (C = A·B as start says), each sum in ascending order of the depth, as the plain loop's: the block held in registers, of a size the compiler knows; each term rounded as Mode says. Out of line: its unrolled sizes, inlined, spent GCC's growth budget, and the LU factorisation, which runs none of them, lost 3 to 8 % at every order (one arm64 thread, GCC 14).
 
template<std::size_t R, term_rounding Mode, typename T >
void narrow_rows (std::size_t n, std::size_t k, const T *a, std::size_t lda, const T *b, std::size_t ldb, T *c, std::size_t ldc, c_start start) noexcept
 C += A·B for R whole rows of C of n columns (C = A·B as start says): blocks of narrow_columns, the last narrower.
 
template<typename T >
void zero_block (T *c, std::size_t ldc, std::size_t first, std::size_t last, std::size_t from, std::size_t to) noexcept
 Zeroes rows first to last (excluded) of C, columns from to to (excluded), its rows ldc apart.
 
template<typename T >
void zero_c (T *c, std::size_t ldc, std::size_t m, std::size_t n)
 Zeroes C's m × n elements, its rows ldc apart: on the threads where they lie together and are many (parallel_zero), a fresh C's pages then faulted in by several.
 
template<term_rounding Mode, typename T >
void gemm_rows_rounded (std::size_t m, std::size_t n, std::size_t k, const T *a, std::size_t lda, const T *b, std::size_t ldb, T *c, std::size_t ldc, c_start start=c_start::kept)
 C += A·B by the plain i-k-j loop, parallel over the rows of C. Past a depth of 1 (narrow_least_depth), the rows run narrow_group at a time by blocks (narrow_rows), the last rows one by one: the same sums, six to ten times as fast from order 4 to 16 (M1 Pro) as the loop of unknown width, whose sums the compiler cannot keep in registers. A depth of 1 streams as fast through that loop.
 
template<std::floating_point T>
bool rows_terms_fuse () noexcept
 Whether gemm_rows rounds each term c + a·b as one fused multiply-add on the path in use, as its choice of term_rounding below makes it (the x86-64 AVX paths' double and float, arm64's fused paths' floating types): the first term of a product that sums nothing (0 + a·b) rounded as gemm_rows rounds it elsewhere.
 
complex_term complex_terms () noexcept
 The complex_term of the path in use.
 
template<typename T >
void gemm_rows (std::size_t m, std::size_t n, std::size_t k, const T *a, std::size_t lda, const T *b, std::size_t ldb, T *c, std::size_t ldc, c_start start=c_start::kept)
 C += A·B by the plain i-k-j loop (gemm_rows_rounded), each element rounded as the tile of the path in use rounds it, so that it is the same whatever the shape of the product it falls in: a caller that splits its columns by the threads (an LU inverse) then has the same bits on any number of them.
 
template<typename T >
T * line_aligned (T *at) noexcept
 The first element of at's run that starts a cache line (or at's nearest element towards one, for an element whose size does not divide the allocator's alignment).
 
template<typename Kernel >
constexpr bool gemm_shares (std::size_t m, std::size_t n, std::size_t k) noexcept
 Whether the product's threads pay: below Kernel's serial_below multiply-adds, the product runs on the caller alone.
 
template<typename Kernel , typename T >
void prefetch_next_row (const T *, std::size_t) noexcept
 Nothing, for a tile that prefetches no C (prefetches_next_row_of_c).
 
template<typename Kernel , typename T >
requires prefetches_next_row_of_c<Kernel>
void prefetch_next_row (const T *c, std::size_t ldc) noexcept
 The C tile at c prefetched by Kernel's prefetch_rows.
 
template<typename Kernel , typename T >
requires (!reads_a_in_place<Kernel, T>)
constexpr bool packs_a (const T *a_packed, std::size_t) noexcept
 Whether gemm_packed packs A for Kernel: unless given packed (a_packed).
 
template<typename Kernel , typename T >
requires reads_a_in_place<Kernel, T>
constexpr bool packs_a (const T *a_packed, std::size_t n) noexcept
 packs_a for a kernel that reads A in place: nor where B is one sliver of columns (n), which reads each element of A once.
 
template<typename Kernel , typename T >
requires (!writes_fresh<Kernel, T>)
constexpr bool zeroes_c (bool empty) noexcept
 Whether gemm_packed zeroes a C that holds nothing (empty) for Kernel's tiles to add to.
 
template<typename Kernel , typename T >
requires writes_fresh<Kernel, T>
constexpr bool zeroes_c (bool) noexcept
 zeroes_c for tiles that write C fresh: never.
 
template<typename Kernel >
constexpr std::size_t task_rows_cap (std::size_t) noexcept
 The most rows of a task of Kernel over a panel of depth kc: its block of rows.
 
template<typename Kernel >
requires requires(std::size_t kc) { Kernel::task_rows_for(kc); }
constexpr std::size_t task_rows_cap (std::size_t kc) noexcept
 task_rows_cap for a tile whose task rows depend on the panel's depth (Kernel::task_rows_for).
 
template<typename Kernel , typename T , typename B = b_rows, typename A = a_rows>
void gemm_packed (std::size_t m, std::size_t n, std::size_t k, const T *a, std::size_t lda, const T *b, std::size_t ldb, T *c, std::size_t ldc, const T *a_packed=nullptr, c_start start=c_start::kept)
 The packed product with the tile Kernel computes: for each block of B's columns and each panel of the depth, both operands are packed (in parallel, by the blocks the product takes), then tasks of gemm_mc rows by gemm_slivers_per_task slivers of columns run in parallel, each writing a disjoint block of C, one tile at a time. Given a_packed (A as pack_left packs it for Kernel, a depth of one panel), A is read from there and only B is packed: the same tiles, the same bits. B is B's layout (b_rows, b_transposed), A A's (a_rows, a_transposed): the packing alone differs, so the bits do not. start says what C holds (c_start), k then not zero.
 
template<typename Kernel , typename T , typename B = b_rows, typename A = a_rows>
void gemm_packed_triangle (std::size_t n, std::size_t k, const T *a, std::size_t lda, const T *b, std::size_t ldb, T *c, std::size_t ldc, bool lower)
 gemm_packed restricted to a triangle of the square C = A·B (n × n): its lower one (j ≤ i) where lower, else its upper one (j ≥ i); the other is neither written nor depended on. The same packing (A's row blocks with no tile in the triangle left out) and the same tiles, so each element's sum is the product's; a tile across the diagonal runs on a copy of its block of C, the triangle's part copied back. A syrk takes the triangle in one pass, its operands packed once a panel, where products by halves packed them again for every block.
 
template<typename T >
bool gemm_elsewhere (std::size_t m, std::size_t n, std::size_t k, const T *a, std::size_t lda, const T *b, std::size_t ldb, T *c, std::size_t ldc, c_start start=c_start::kept)
 A product a faster path than the vector one takes, if any: none for this scalar.
 
template<typename B , typename A = a_rows>
bool gemm_double_elsewhere (std::size_t m, std::size_t n, std::size_t k, const double *a, std::size_t lda, const double *b, std::size_t ldb, double *c, std::size_t ldc, c_start start=c_start::kept)
 A double product on the path in use (numerical::gemm_path_in_use), unless it is the portable one, B and A laid out as B and A say (b_rows, a_rows and their transposes).
 
template<typename A , typename B >
void gemm_triangle (std::size_t n, std::size_t k, const double *a, std::size_t lda, const double *b, std::size_t ldb, double *c, std::size_t ldc, bool lower)
 The triangle of the double product C = A·B (gemm_packed_triangle) on the tile of the path in use, A and B laid out as A and B say.
 
template<typename A , typename B >
void gemm_triangle (std::size_t n, std::size_t k, const float *a, std::size_t lda, const float *b, std::size_t ldb, float *c, std::size_t ldc, bool lower)
 The triangle of the float product C = A·B (gemm_packed_triangle) on the float tile of the path in use, A and B laid out as A and B say: the single-precision factorisations' trailing updates.
 
bool gemm_elsewhere (std::size_t m, std::size_t n, std::size_t k, const double *a, std::size_t lda, const double *b, std::size_t ldb, double *c, std::size_t ldc, c_start start=c_start::kept)
 gemm_double_elsewhere of a row-major B.
 
template<typename B , typename A = a_rows>
bool gemm_float_elsewhere (std::size_t m, std::size_t n, std::size_t k, const float *a, std::size_t lda, const float *b, std::size_t ldb, float *c, std::size_t ldc, c_start start=c_start::kept)
 A float product on the AMX where it suits the product, on the NEON tile on arm64's fused paths, on the x86-64 tiles there; elsewhere the vector tile. B and A laid out as B and A say.
 
bool gemm_elsewhere (std::size_t m, std::size_t n, std::size_t k, const float *a, std::size_t lda, const float *b, std::size_t ldb, float *c, std::size_t ldc, c_start start=c_start::kept)
 gemm_float_elsewhere of row-major operands.
 
template<typename T >
void gemm (std::size_t m, std::size_t n, std::size_t k, const T *a, std::size_t lda, const T *b, std::size_t ldb, T *c, std::size_t ldc, c_start start)
 C += A·B, every element of C summed in ascending order of the depth.
 
template<std::size_t Width, term_rounding Mode, typename T >
void few_rows (std::size_t m, std::size_t n, std::size_t k, const T *a, std::size_t lda, const T *b, std::size_t ldb, T *c, std::size_t ldc, c_start start)
 C += A·B (C = A·B as start says) for m rows of A, by groups of at most narrow_group rows and blocks of Width columns of C, each block its sums in registers over the whole depth with B read where it lies (narrow_block), the blocks of every group shared among the threads; the last columns by narrow_rows. Each element the fused chain of the tile, rounded as Mode says, so its bits.
 
template<split_complex C>
void split_row_columns (const C *a, const C *b, std::size_t row_step, std::size_t column_step, std::size_t inner, std::size_t first, std::size_t last, C *y, bool kept=false)
 Columns first to last of the complex vector a (inner) times B (B[k][j] at b[(k · row_step) + (j · column_step)]) into y, each element summed as gemm_split sums it (complex_term::split): by blocks of split_depth of the depth, Re a · Re B then −Im a · Im B into its real part, Re a · Im B then Im a · Re B into its imaginary part, each one chain of fused multiply-adds from +0 through the blocks, or from y's own where kept (y += a · B, as gemm_split starts from C).
 
bool split_by_parts (std::size_t m) noexcept
 Whether gemm_split runs a product of m rows of A by four real products of each block's parts (split_parts_block), else by one of its expanded operands (split_expanded_block): the same bits, the parts' products faster where they read B's parts in place (gemm_packed). On the AMX path up to split_parts_rows; elsewhere (the NEON path) up to one row of the NEON tile, its b_in_place_rows: M1 Pro, all threads, by 400 × 500, 1 to 4 rows 2 to 18 % faster, 6 to 32 rows 5 to 52 % slower.
 
template<typename R >
split_operands< R > & split_storage ()
 The calling thread's split_operands. A product uses them whole on one thread, and the real product it runs on them nests no complex one.
 
template<split_complex Z, typename B = b_rows, typename A = a_rows>
void split_parts_block (std::size_t m, std::size_t nb, std::size_t kb, std::size_t pc, std::size_t jc, const Z *a, std::size_t lda, const Z *b, std::size_t ldb, typename Z::value_type *ar, typename Z::value_type *br, typename Z::value_type *cr)
 One block of gemm_split ([Re C | Im C] at cr, m × nb each, += A·B, A m × kb and B kb × nb) by four real products of its parts: Re C += Re A·Re B, then −Im A·Im B; Im C += Re A·Im B, then Im A·Re B, each continuing its elements' chains ((−x)·(−y) is x·y exactly: the bits of −Im A·−Re B). B's parts are a copy of its size, which a product of a few rows of A reads in place. The block is A's columns and B's rows from pc, B's columns from jc, A and B laid out as A and B say.
 
template<split_complex Z, typename B = b_rows, typename A = a_rows>
void split_expanded_block (std::size_t m, std::size_t nb, std::size_t kb, std::size_t pc, std::size_t jc, const Z *a, std::size_t lda, const Z *b, std::size_t ldb, typename Z::value_type *ar, typename Z::value_type *br, typename Z::value_type *cr)
 One block of gemm_split (split_parts_block's) by one real product of its expanded operands, [Re C | Im C] += [Re A | −Im A] · [[Re B, Im B], [Im B, −Re B]]: the same chains, B copied four times its size, the product as wide and as deep as the block's two.
 
template<split_complex Z, typename B = b_rows, typename A = a_rows>
void gemm_split (std::size_t m, std::size_t n, std::size_t k, const Z *a, std::size_t lda, const Z *b, std::size_t ldb, Z *c, std::size_t ldc)
 C += A·B for complex numbers through the real kernel, by blocks of split_columns columns and split_depth of depth, on the real path's units and threads (a scalar complex tile has none). Within a block, each element of C takes the products of the real parts of A in ascending order of the depth, then those of the imaginary parts: by four real products of the block's parts (split_parts_block) or one of its expanded operands (split_expanded_block), as split_by_parts chooses, the same bits either way. A and B laid out as A and B say, read where they lie as the parts are copied.
 
bool plain_loop_pays (std::size_t m, std::size_t n, std::size_t k) noexcept
 Whether a double product of m rows, n columns and depth k, past gemm_rows's threshold, still runs faster in the plain loop than packed on the path in use: packing B's slivers for so few rows of A, or so short a depth, did not pay. One core, the plain loop's time over the packed product's: the NEON tile at a depth of 16 or less whatever the shape (0.5 to 0.6 up to 2000 × 2000), at 64 or less for 64 rows or fewer (0.4 to 0.9; the same under GCC 14 and on eight threads); the AVX2 tile for 16 rows or fewer to a depth of 64, or 32 rows to 32 (0.2 to 0.8, six cores too), as wide as plain_loop_least_width. The AMX's packing pays from 16 rows; a product below that (amx_outruns_plain_loop) runs the NEON tile's rule, where it went packed on that tile (one M1 Pro core, 8 to 15 rows to a depth of 8 to 64: 0.34 to 0.74 of the time).
 
template<typename T >
bool plain_loop_suits (std::size_t, std::size_t, std::size_t) noexcept
 plain_loop_pays for a product of T: double's and float's, those it was measured for.
 
template<>
bool plain_loop_suits< double > (std::size_t m, std::size_t n, std::size_t k) noexcept
 plain_loop_pays for double.
 
template<>
bool plain_loop_suits< float > (std::size_t m, std::size_t n, std::size_t k) noexcept
 plain_loop_pays for float: double's rule, where the plain loop outran the packed tiles as for double (random products of its region: 140 on one M1 Pro core, 0.51 of the time, 0.52 on eight threads; 89 on eight AVX2 cores, 0.75 and 0.68 under GCC, 0.88 and 0.80 under Clang, one thread then eight). The same bits.
 
template<typename T >
void gemm_large (std::size_t m, std::size_t n, std::size_t k, const T *a, std::size_t lda, const T *b, std::size_t ldb, T *c, std::size_t ldc, c_start start=c_start::kept)
 A product past the plain loop: on the AMX or x86-64 tiles where gemm_elsewhere takes it, else on the vector tile.
 
template<typename B , typename A , split_complex T>
void gemm_large_complex (std::size_t m, std::size_t n, std::size_t k, const T *a, std::size_t lda, const T *b, std::size_t ldb, T *c, std::size_t ldc, c_start start=c_start::kept)
 A complex product past the plain loop: on the vector tile on the portable path, whole complex products summed in order; on the x86-64 complex tiles there; else (arm64) through the real kernel (gemm_split). B and A laid out as B and A say; start what C holds (c_start), the packed paths' to settle, the split's C zeroed first.
 
bool few_complex_rows (std::size_t m, std::size_t n, std::size_t k) noexcept
 Whether gemm runs a complex product of m rows, B k × n, a few rows at a time with B read by rows (few_rows_complex_x86, few_rows_complex_neon), on a path past the portable one.
 
template<typename B , typename A >
void gemm_large_layouts (std::size_t m, std::size_t n, std::size_t k, const double *a, std::size_t lda, const double *b, std::size_t ldb, double *c, std::size_t ldc, c_start start)
 A double product past the plain loop, B and A laid out as B and A say (gemm_layouts).
 
template<typename B , typename A >
void gemm_large_layouts (std::size_t m, std::size_t n, std::size_t k, const float *a, std::size_t lda, const float *b, std::size_t ldb, float *c, std::size_t ldc, c_start start)
 A float product past the plain loop, B and A laid out as B and A say (gemm_layouts).
 
template<typename B , typename A , split_complex T>
void gemm_large_layouts (std::size_t m, std::size_t n, std::size_t k, const T *a, std::size_t lda, const T *b, std::size_t ldb, T *c, std::size_t ldc, c_start start)
 A complex product past the plain loop, B and A laid out as B and A say (gemm_layouts).
 
template<typename A , typename B , typename T >
void gemm_layouts (std::size_t m, std::size_t n, std::size_t k, const T *a, std::size_t lda, const T *b, std::size_t ldb, T *c, std::size_t ldc, c_start start=c_start::kept)
 gemm with either operand given as its transpose (A: a_rows or a_transposed, B: b_rows or b_transposed), so that a product by a transpose (a syrk's P Pᵀ, BLAS's transa and transb, a NumPy array in Fortran order) needs no copy of it. The tiles are gemm's, the operands packed (or, through gemm_split, copied into their parts) from where they lie, and so are the bits; a product gemm runs in the plain loop copies a transposed operand row-major and runs there, small as it is. start says what C holds (c_start), passed on to the route as gemm passes it: C zeroed apart first cost a pass over it, as long as the product's own at a shallow depth (1000 × 1000 by depth 4, one arm64 thread: 307 µs copying the transpose for gemm, 402 read in place).
 
template<typename T >
void gemm_bt (std::size_t m, std::size_t n, std::size_t k, const T *a, std::size_t lda, const T *bt, std::size_t ldbt, T *c, std::size_t ldc, c_start start=c_start::kept)
 gemm_layouts with B given as Bᵀ (bt, n × k, ldbt apart).
 
template<typename T >
void gemm_at (std::size_t m, std::size_t n, std::size_t k, const T *at, std::size_t ldat, const T *b, std::size_t ldb, T *c, std::size_t ldc, c_start start=c_start::kept)
 gemm_layouts with A given as Aᵀ (at, k × m, ldat apart).
 
template<typename T >
void gemm_atbt (std::size_t m, std::size_t n, std::size_t k, const T *at, std::size_t ldat, const T *bt, std::size_t ldbt, T *c, std::size_t ldc, c_start start=c_start::kept)
 gemm_layouts with both operands given as their transposes.
 
template<typename T >
packed_left< T > read_in_place (std::size_t m, std::size_t k, const T *a, std::size_t lda) noexcept
 The m × k a read in place by gemm_with, unpacked: each product packs it as gemm does.
 
template<typename T >
packed_left< T > left_rows (const packed_left< T > &left, std::size_t r0, std::size_t rows) noexcept
 The rows [r0, r0 + rows) of left, r0 a multiple of left.granule: their slivers in its packing, or their rows of A in place.
 
template<typename T >
left_kernel left_kernel_for (std::size_t, std::size_t, std::size_t) noexcept
 The tile gemm runs a product of m × n by depth k on, for a left operand packed once for many products: plain for a scalar other than double (written as an overload per scalar, as conj_of).
 
template<>
left_kernel left_kernel_for< double > (std::size_t m, std::size_t n, std::size_t k) noexcept
 left_kernel_for a double product: past gemm_rows, as gemm_elsewhere chooses, of a depth of one panel; plain otherwise.
 
template<typename Kernel , typename T >
const T * pack_left_for (std::size_t m, std::size_t k, const T *a, std::size_t lda, buffer< T > &storage)
 A packed for Kernel into storage, in parallel by its blocks of rows, as gemm_packed packs it.
 
template<typename T >
packed_left< T > pack_left (std::size_t m, std::size_t, std::size_t k, const T *a, std::size_t lda, buffer< T > &)
 The m × k a packed once for products with it of about n columns each (gemm_with), in storage (grown as needed, kept by the caller from one call to the next): on the tile gemm would choose for m × n × k. A scalar other than double (here), a depth of more than one panel, or a product gemm_rows takes, leaves A to be read in place.
 
template<>
packed_left< double > pack_left< double > (std::size_t m, std::size_t n, std::size_t k, const double *a, std::size_t lda, buffer< double > &storage)
 pack_left for double, on the tile left_kernel_for chooses.
 
template<typename T >
void gemm_with (const packed_left< T > &left, std::size_t n, const T *b, std::size_t ldb, T *c, std::size_t ldc)
 C += A·B for the packed left (pack_left) and the k × n b, as gemm computes it, A read in place here, for a scalar other than double.
 
template<>
void gemm_with< double > (const packed_left< double > &left, std::size_t n, const double *b, std::size_t ldb, double *c, std::size_t ldc)
 gemm_with for double: on the tile A was packed for, A read from its packing, where gemm would choose that tile for this n; else as gemm. Either way gemm's tiles and order of summation, its bits.
 
constexpr std::size_t dot_chunk_weight (std::size_t chunks) noexcept
 The work chunked_dot weighs one of its chunks chunks, in multiply-adds for parallel_for's split: four an element, each a read of memory's pace, where one kept the product on one thread to 131 072 elements (M1 Pro, 10⁵ elements: 19.6 → 13.2 µs); one an element under three chunks, whose split cost more than it saved (20 000 elements: 4.5 → 5.6 µs).
 
template<typename T , typename Dot >
T chunked_dot (const T *a, const T *b, std::size_t count, const Dot &dot)
 Σ a[i] · b[i] for a plain dot product: chunks of dot_chunk elements in parallel, each by dot, dot_product below (several vector accumulators, on the widest x86-64 units for double), their sums added in the chunks' order. The same on any number of threads; in the last bits, not the ascending sum, which waits on each addition before the next (ten times slower at 100 000 elements).
 
template<typename T >
T chunked_dot (const T *a, const T *b, std::size_t count)
 Σ a[i] · b[i]: chunked_dot of each chunk by dot_product.
 
template<typename T >
T chunked_conjugate_dot (const T *a, const T *b, std::size_t count)
 Σ conj(a[i]) · b[i] (zdotc): chunked_dot of each chunk by conjugate_dot_product, without a conjugated copy of a.
 
template<typename T >
void matrix_times_vector (const T *a, std::size_t lda, std::size_t rows, std::size_t cols, const T *x, T *y)
 y = A x for the row-major rows × cols a, its rows lda apart: each element the dot product of its row with x, as chunked_dot sums it (a row of a chunk or less is one dot_product), the rows in parallel by groups (row_dots, which share each load of x). The same on any number of threads. A double matrix of amx_dot_least elements to fewer than amx_dot_most, on the AMX path, runs on the AMX (amx_dot_rows_f64): each element then its eight lanes of fused multiply-adds, added in order.
 
void columns_amx_lanes (const double *a, std::size_t lda, std::size_t rows, std::size_t cols, const double *x, double *y)
 amx_dot_rows_f64's sums of the rows rows of the column-major a (column j at a + j · lda) without the AMX: lane j mod 8 of each row its fused multiply-adds in ascending order of j, a column at a time over the rows together, then its lanes added from 0 to 7. The same bits, from a transpose where it lies.
 
template<typename T >
requires std::is_same_v<T, double> || std::is_same_v<T, float>
bool columns_times_vector (const T *a, std::size_t lda, std::size_t rows, std::size_t cols, const T *x, T *y)
 y = A x for the column-major rows × cols a (column j at a + j · lda; NumPy's a.T, a transposed operand) on the x86-64 units: each element the bits matrix_times_vector gives its C-ordered copy (avx2_dot's or avx512_dot's sum of its row), from the partial sums those kernels keep, updated a column at a time over the rows together (avx2_column_partials), then finished as the kernels finish a row; blocks of column_rows rows among the threads. The copy NumPy made of the transpose first cost more than the product (500 × 500: 170 µs of 213). float likewise, from the partial sums of its own kernels (32 on AVX2, 64 on AVX-512). Elsewhere (GCC and Clang) from the portable vectors' partials (columns_row_dots), as their row_dots sums a row; where the row-major product takes the AMX, from its lanes (columns_amx_lanes), in place of a copy of the transpose (400 × 400: 22 µs against 68, M1 Pro).
 
template<split_complex C>
bool columns_times_vector (const C *a, std::size_t lda, std::size_t rows, std::size_t cols, const C *x, C *y)
 columns_times_vector of complex numbers: each element the bits of its C-ordered copy's product, from the slots the portable vectors' complex_dot_sums keep for a row (columns_complex_row_dots), blocks of column_rows rows among the threads (GCC and Clang); on the x86-64 AVX paths from the slots their kernel keeps (avx2_complex_column_slots).
 
double modulus_one (const std::complex< double > &z)
 |Re z| + |Im z|, LAPACK's ABS1.
 
qz_form hessenberg_triangular (const matrix< std::complex< double > > &a, const matrix< std::complex< double > > &b)
 a and b (n × n, row-major) to Hessenberg-triangular form, as LAPACK's zgges begins: B = Q R by the QR factorisation, A ← Qᴴ A, then zgghrd: each column of A from the first cleared below its subdiagonal from the bottom by rotations of rows, the fill each leaves below B's diagonal taken out by a rotation of columns; Q and Z accumulating them.
 
std::pair< std::complex< double >, std::complex< double > > standardize (qz_form &f, std::size_t j)
 T(j, j) made real and non-negative, as zhgeqz standardises it: row j of T and S above the diagonal and column j of S, T and Z taking the phase.
 
std::vector< std::pair< std::complex< double >, std::complex< double > > > qz_iteration (qz_form &f, std::size_t steps)
 The Hessenberg-triangular f to generalized Schur form, as LAPACK's zhgeqz (Schur form, Q and Z updated): single-shift QZ sweeps on the active block, the shift the eigenvalue of the trailing 2 × 2 of S T⁻¹ nearer its corner (an exceptional one every tenth iteration), a negligible subdiagonal entry of S splitting the block, a negligible diagonal entry of T chased down (or up, after two small subdiagonal entries) to split off an infinite eigenvalue.
 
double frobenius_two (const std::array< std::complex< double >, 4 > &m)
 The Frobenius norm of the 2 × 2 block m.
 
bool exchange_pencil_diagonal (qz_form &f, std::size_t j)
 Exchanges the diagonal entries j and j + 1 (from 1) of the triangular pencil f, as LAPACK's ztgex2: columns j, j + 1 rotated onto the right eigenvector of the second eigenvalue, then rows j, j + 1 back to triangular, the rotation from S's new first column where |S₂₂ T₁₁| ≥ |S₁₁ T₂₂|, else T's; Z and Q turning with them. The diagonal of T is then complex: standardize takes it back. Tried on the 2 × 2 blocks first and refused, f unchanged, unless it passes ztgex2's weak test (the new subdiagonal entries within 20 ε of the blocks' norms) and strong test (the blocks rebuilt from the rotations within the same bounds).
 
triangle_svd svd_of_triangle (double f, double g, double h)
 The SVD of the upper triangular [f g; 0 h], as LAPACK's dlasv2: [csl snl; −snl csl] [f g; 0 h] [csr −snr; snr csr] = diag(ssmax, ssmin), |ssmax| ≥ |ssmin|, each accurate to a few ulps.
 
double pencil_wsize (const pencil_bounds &b, double w) noexcept
 dlag2's WSIZE: the divisor that keeps an eigenvalue of magnitude w in range.
 
pencil_pair pencil_eigenvalues (double a11, double a12, double a21, double a22, double b11, double b12, double b22, double safmin)
 The eigenvalues of the pencil A − w B, A (a11 a12 / a21 a22) and the upper triangular B (b11 b12 / 0 b22), as LAPACK's dlag2, with scalings that keep scale · w from overflowing or underflowing; a diagonal entry of B below safmin's square root times its largest perturbed up to it.
 
double hypot3 (double x, double y, double z)
 √(x² + y² + z²) without overflow, LAPACK's dlapy3.
 
std::pair< real_pencil_value, real_pencil_value > pair_values (real_qz_form &f, std::size_t ilast, const pencil_pair &pair, double b11, double b22)
 The conjugate pair of the standardized 2 × 2 block ending at ilast of f (T diagonal, b11 and b22), as dhgeqz computes α and β from pair (dlag2's): β the norms of the diagonal of T after the complex rotations that would triangularise the block, α = (wr ± i wi) β / scale.
 
void double_shift_sweep (real_qz_form &f, std::size_t ifirst, std::size_t ilast, double ascale, double bscale, double safmin)
 One double-shift QZ sweep (Moler and Stewart) on rows and columns ifirst … ilast of f, as dhgeqz's: the shifts the eigenvalues of the trailing 2 × 2 of S T⁻¹ and its leading one's first column, the bulge chased by reflections of three rows (dlarfg) and of three columns (from the null vector of T's 2 × 2 below the bulge), a rotation pair for the last step.
 
real_qz_form real_hessenberg_triangular (const matrix< double > &a, const matrix< double > &b)
 The real a and b to Hessenberg-triangular form, as LAPACK's dgges begins: B = Q R by the QR factorisation, A ← Qᵀ A, then dgghrd: each column of A from the first cleared below its subdiagonal from the bottom by rotations of rows, the fill each leaves below B's diagonal taken out by a rotation of columns; Q and Z accumulating them.
 
std::vector< real_pencil_value > real_qz_iteration (real_qz_form &f, std::size_t steps)
 The Hessenberg-triangular f to generalized real Schur form, as LAPACK's dhgeqz (Schur form, Q and Z updated): double-shift QZ sweeps (Moler and Stewart) on the active block, the shifts the eigenvalues of its trailing 2 × 2 (dlag2; a single shift where they are real, an exceptional one every tenth iteration), a negligible subdiagonal entry of S splitting it, a negligible diagonal entry of T chased to split off an infinite eigenvalue; a 2 × 2 block of a conjugate pair standardized with T diagonal and positive (dlasv2), every T(j, j) non-negative. steps bounds the iterations (dhgeqz's 30 n).
 
small_square small_product (const small_square &x, bool tx, const small_square &y, bool ty, std::size_t m)
 op(x) op(y) of order m, op the transpose where tx (ty).
 
double small_norm (const small_square &x, std::size_t r0, std::size_t r1, std::size_t c0, std::size_t c1)
 The Frobenius norm of rows r0 … r1 − 1 and columns c0 … c1 − 1 of x.
 
small_square small_qr (small_square x, std::size_t m, std::size_t k)
 The orthogonal Q (order m) of the Householder QR factorisation of the first k columns of x (m × k, k ≤ m), as LAPACK's dgeqr2 then dorg2r: its first k columns span theirs, Qᵀ x upper trapezoidal.
 
pencil_rotations standardize_pencil (std::array< std::array< double, 2 >, 2 > &a, std::array< std::array< double, 2 >, 2 > &b)
 The 2 × 2 pencil (a, b), b upper triangular, standardized, as LAPACK's dlagv2: two real eigenvalues made triangular, a complex pair left with b diagonal and positive; both changed in place.
 
std::optional< std::pair< small_square, small_square > > generalized_sylvester (const small_square &s, const small_square &t, std::size_t n1, std::size_t n2)
 The solution (R, L) (n1 × n2 each) of the generalized Sylvester equations S₁₁ R − L S₂₂ = S₁₂, T₁₁ R − L T₂₂ = T₁₂ of the m × m blocks of s and t (m = n1 + n2), as LAPACK's dtgsy2 solves them: by elimination with complete pivoting of their Kronecker form; nothing where a pivot falls below ε times the largest entry (dgetc2 would perturb it, and dtgex2 then refuses the exchange).
 
std::optional< block_exchange > plan_single_exchange (small_square s, small_square t, double thresha, double threshb)
 Two 1 × 1 blocks' exchange, as LAPACK's dtgex2 makes it: columns turned onto the second eigenvalue's right eigenvector, then rows back to triangular, the rotation from S's new first column where |S₂₂ T₁₁| ≥ |S₁₁ T₂₂|, else T's. Nothing where the new subdiagonal entries pass thresha and threshb (the weak test).
 
std::optional< block_exchange > plan_block_exchange (small_square s, small_square t, std::size_t n1, std::size_t n2, double thresha)
 The exchange of blocks of orders n1 and n2 (one of them 2), as LAPACK's dtgex2 makes it: orthogonal bases of the deflating subspaces the generalized Sylvester equations give (generalized_sylvester), T made triangular again by an RQ or a QR factorisation (the one that leaves the smaller block below S's diagonal), each 2 × 2 block standardized (standardize_pencil). Nothing where the equations have no solution or that block passes thresha (the weak test).
 
bool exchange_real_blocks (real_qz_form &f, std::size_t j1, std::size_t n1, std::size_t n2)
 Exchanges the adjacent diagonal blocks of orders n1 and n2 (1 or 2) at j1 (from 1) of the real generalized Schur form f, as LAPACK's dtgex2: planned by plan_single_exchange or plan_block_exchange, then refused unless the blocks rebuilt from the transforms, ‘LI S’ IRᵀ, are within 20 ε‘ of the originals’ norms (the strong test, after the 2 × 2 blocks' standardization where dtgex2 makes it before: their rotations' rounding more); else the entries below the new blocks set to 0 and the rows and columns of S, T, Q and Z turned.
 
void exchange_or_refuse (real_qz_form &f, std::size_t j1, std::size_t n1, std::size_t n2)
 exchange_real_blocks, a refusal thrown as real_generalized_schur::reorder reports it.
 
void move_real_block_up (real_qz_form &f, std::size_t ifst, std::size_t ilst)
 Moves the diagonal block at ifst (from 1) of the real generalized Schur form f up to ilst, as LAPACK's dtgexc does it upwards: exchanged with each block above in turn (exchange_real_blocks), a 2 × 2 block that splits on the way moved as its two 1 × 1 ones; ifst and ilst taken to the first row of their blocks.
 
std::vector< real_pencil_value > real_values (real_qz_form &f)
 The eigenvalues of the real generalized Schur form f after a reordering, as LAPACK's dtgsen gives them: a 2 × 2 block's pair by dlag2 (β its scale), a 1 × 1 block's from its diagonal, T(k, k) made non-negative (row k of S and T and column k of Q negated).
 
template<typename T >
std::pair< matrix< T >, double > pencil_operand_in_range (const matrix< T > &m)
 m brought into the range [√safmin / ε, ε / √safmin] of its largest magnitude where it lies outside, as LAPACK's xgges scales A and B before the QZ iterations (their norms' squares, the shifts and the rotations stay in range): the matrix, and the factor that takes the scaled one back (1 where none).
 
template<typename T >
matrix< T > scaled_back (const buffer< T > &m, std::size_t n, double factor)
 The entries of m multiplied by factor, n × n row-major (the scaled form taken back).
 
matrix< std::complex< double > > triangular_pencil_vectors (const matrix< std::complex< double > > &s, const matrix< std::complex< double > > &t, const std::vector< std::pair< std::complex< double >, std::complex< double > > > &values, const matrix< std::complex< double > > &basis, bool left)
 The right (or, where left, the left) eigenvectors of the triangular pencil (s, t) whose diagonal is values, as LAPACK's ztgevc: for each (α, β), x with (β S − α T) x = 0 and x_k = 1 by back substitution (yᴴ (β S − α T) = 0 forward for a left one), then basis x made of unit 2-norm; a diagonal entry of β S − α T below ε (|β| ‖S‖ + |α| ‖T‖), or the smallest normal number, taken as that bound (SMIN), and α and β both zero (a singular pencil) given the unit vector of its place. The substitutions on the threads, each on its own; the basis applied to their nonzero parts by one product.
 
buffer< double > scaling_balance (buffer< double > &m, std::size_t n)
 The scaling of LAPACK's ‘dgebal('S’)for the non-negativen × n\p m (row-major, scaled in place): rows and columns by powers of two until their 2-norms are within a factor of 2, a row or column of zeros left alone, the factors kept within the safe range. \return The scalesDofD⁻¹ M D`.
 
template<typename T >
double norm_one (const matrix< T > &a)
 The 1-norm of a: its largest column sum of moduli.
 
void refuse_boundary (bool continuous)
 Refuses a Riccati equation whose pencil has eigenvalues on (or too close to) the boundary.
 
template<typename T >
matrix< T > stabilising_solution (const matrix< T > &top, const matrix< T > &bottom, bool continuous)
 X = U₂₁ U₁₁⁻¹ from the m leading Schur vectors' halves top (U₁₁) and bottom (U₂₁) of a Riccati equation's Hamiltonian (continuous) or symplectic matrix or pencil, as SciPy's checks have it.
 
template<typename T >
std::pair< buffer< T >, buffer< T > > riccati_pencil (const riccati_operands< T > &terms, bool continuous)
 The extended pencil (H, J) of order 2m + n of SciPy's solve_continuous_are (continuous) or solve_discrete_are, row-major.
 
template<typename T >
buffer< double > balance_pencil (buffer< T > &h, buffer< T > &j, std::size_t m, std::size_t order)
 Balances the pencil (H, J) of order 2m + n as SciPy does: ‘dgebal('S’)scaling of|H| + |J|off the diagonal, the first two blocks ofmthen givenDandD⁻¹from half the difference of their exponents (Benner's symplectic constraint), the pencil scaled toD⁻¹ (H, J) D. \return The scalesD`, all 1 when the scaling found none.
 
template<typename T >
matrix< T > riccati (const riccati_operands< T > &terms, bool continuous, bool balanced)
 The stabilising solution of the continuous (continuous) or discrete algebraic Riccati equation by SciPy's algorithm: the extended pencil, balanced unless balanced is false, deflated by the QR factorisation of its last n columns, its generalized Schur form sorted with the eigenvalues in the left half-plane (inside the unit circle) first, and X = U₁₀ U₀₀⁻¹ from the leading m columns of Z (with E, of the QR factorisation of [Ẽ U₀₀; U₁₀]), returned as (X + Xᴴ)/2 in the original scaling. Ẽ is E as the balanced pencil holds it, D₁ E D₁⁻¹: the basis U is of the balanced pencil, so X̃ Ẽ U₀₀ = U₁₀ for X̃ = D₁⁻¹ X D₁⁻¹, and the unscaled E there (as SciPy has it) answers a wrong X or none.
 
matrix< std::complex< double > > continuous_residual (const riccati_terms &terms, const matrix< std::complex< double > > &x, matrix< std::complex< double > > &closed)
 The residual Aᴴ X E + Eᴴ X A − (Eᴴ X B + S) R⁻¹ (Bᴴ X E + Sᴴ) + Q of the continuous equation at the Hermitian x, and the closed loop A − B K, K = R⁻¹ (Bᴴ X E + Sᴴ), into closed.
 
bool entries_finite (std::initializer_list< const matrix< std::complex< double > > * > xs)
 Whether every entry of the matrices xs is finite.
 
double continuous_backward_error (const riccati_terms &terms, const matrix< std::complex< double > > &x)
 The residual at x of the continuous equation without E (the Hamiltonian's), relative to its terms: ‖Res(X)‖₁ / (2 ‖A‖₁ ‖X‖₁ + ‖Bᴴ X + Sᴴ‖₁ ‖R⁻¹(Bᴴ X + Sᴴ)‖₁ + ‖Q‖₁) (continuous_residual); infinite where it is not finite.
 
matrix< std::complex< double > > newton_refined (const riccati_terms &terms, matrix< std::complex< double > > &&x)
 One step of Newton's method on the continuous equation from the Schur method's x (Kleinman, 1968), in its correction form: A_Kᴴ Δ E + Eᴴ Δ A_K = −Res(X) for the closed loop A_K (continuous_residual), a Lyapunov equation solved for Y = Eᴴ Δ E as Fᴴ Y + Y F = −Res(X), F = E⁻¹ A_K; X + Δ kept only where its residual's 1-norm is smaller. The generalized Schur form of a badly scaled equation (a graded A) loses digits that a step from a solution so near the stabilising one gains back.
 
matrix< std::complex< double > > answer_like (matrix< std::complex< double > > &&x, const matrix< std::complex< double > > &)
 x itself, for complex operands like.
 
matrix< double > answer_like (const matrix< std::complex< double > > &x, const matrix< double > &)
 x's real part, for real operands like: the solution of a real equation is real.
 
void require_hermitian (const matrix< std::complex< double > > &x, const char *name)
 Refuses x (named name in the message) unless Hermitian, as SciPy's _are_validate_args: ‖X − Xᴴ‖₁ within 100 spacings of ‖X‖₁.
 
template<typename T >
buffer< double > balance_hamiltonian (buffer< T > &h, std::size_t m)
 The diagonal D of a symplectic balancing of the Hamiltonian h of order 2m (row-major, in place): h ← T⁻¹ h T, T = diag(D, D⁻¹), powers of 2 (each scaling exact). Each dᵢ scales column i and row m + i by f (their common entry by f²), row i and column m + i by 1/f (theirs by 1/f²): the power of 2 whose off-diagonal sum g₁ f + g₂ f² + s₁/f + s₂/f² is least (a convex function of log f), taken where it lowers the sum by a twentieth, swept until none does: each step lowers the matrix's off-diagonal sum, so the sweeps end.
 
template<typename T >
matrix< T > hamiltonian_riccati (const matrix< T > &a, const matrix< T > &b, const matrix< T > &q, const matrix< T > &r, const matrix< T > *s, bool balanced)
 The stabilising solution of the continuous equation without E from the Schur form of the Hamiltonian H = [[Ã, −G], [−Q̃, −Ãᴴ]] (Laub, 1979), G = B R⁻¹ Bᴴ, Ã = A − B R⁻¹ Sᴴ and Q̃ = Q − S R⁻¹ Sᴴ in the arithmetic of T (a real H's real Schur form): its eigenvalues in the left half-plane moved first, X = U₂₁ U₁₁⁻¹ from their m Schur vectors, made Hermitian. A Schur form of order 2m by multishift QR where the pencil's QZ takes order 2m + n. Where balanced, H is first T⁻¹ H T, T = diag(D, D⁻¹) (balance_hamiltonian), still Hamiltonian, and X = D⁻¹ X̃ D⁻¹: a Q of 10¹⁰ beside a G of 1 left the unbalanced form's X wrong at 10⁻³.
 
template<typename T >
matrix< T > solve_are (const matrix< T > &a, const matrix< T > &b, const matrix< T > &q, const matrix< T > &r, const matrix< T > *e, const matrix< T > *s, bool balanced, bool continuous)
 The checks of SciPy's _are_validate_args, then riccati on complex operands, the answer real for real ones.
 
double pivot_size (double x)
 LAPACK's measure of a real entry, |x|.
 
double pivot_size (const std::complex< double > &z)
 LAPACK's measure of a complex entry, CABS1: |Re z| + |Im z|.
 
double diagonal_size (double x, bool)
 A real diagonal entry's measure, |x|, whatever hermitian.
 
double diagonal_size (const std::complex< double > &z, bool hermitian)
 A complex diagonal entry's measure: |Re z| for a Hermitian matrix (hetf2), CABS1 for a symmetric one (sytf2).
 
double conjugated_if (double x, bool)
 A real x as it is.
 
std::complex< double > conjugated_if (const std::complex< double > &z, bool hermitian)
 z's conjugate for a Hermitian matrix, z for a symmetric one.
 
double diagonal_kept (double x, bool)
 A real x as it is.
 
std::complex< double > diagonal_kept (const std::complex< double > &z, bool hermitian)
 A Hermitian matrix's diagonal entry z taken as real (hetf2 ignores its imaginary part), a symmetric one's as it is.
 
template<typename T >
void symmetric_rank_one (buffer< T > &a, std::size_t n, std::size_t k, buffer< T > &w)
 The trailing matrix's update by a pivot of order 1 at k, as sytf2 (syr, then L's column scaled): A ← A − x xᵀ / d on and below the diagonal, x / d into column k; the arithmetic of T. Row by row (a is row-major), each entry's one operation as syr's column by column: the factors −x_j / d first, in w (n entries).
 
void hermitian_rank_one (buffer< std::complex< double > > &a, std::size_t n, std::size_t k, buffer< std::complex< double > > &w)
 As symmetric_rank_one for a Hermitian matrix, as hetf2 (her): d real, A ← A − x xᴴ / d, the diagonal kept real.
 
template<typename T >
void symmetric_rank_two (buffer< T > &a, std::size_t n, std::size_t k, buffer< T > &w)
 The trailing matrix's update by a pivot of order 2 at rows k and k + 1, as sytf2: D21 scaled out, W = [x y] D⁻¹ from D11 / D21, D22 / D21 and t = 1 / (D11 D22 / D21² − 1), A ← A − [x y] Wᵀ, W into the two columns; the arithmetic of T. Row by row, W first, in w (2n entries), each row's x and y read before its two entries of W replace them.
 
void hermitian_rank_two (buffer< std::complex< double > > &a, std::size_t n, std::size_t k, buffer< std::complex< double > > &w)
 As symmetric_rank_two for a Hermitian matrix, as hetf2: |D21| scaled out in real arithmetic, the conjugates where Aᴴ has them, the diagonal kept real.
 
void eliminate_pivot (buffer< double > &a, std::size_t n, std::size_t k, std::size_t step, bool, buffer< double > &w)
 A real matrix's update by a pivot of order 1 or 2 (step), sytf2's whatever hermitian.
 
void eliminate_pivot (buffer< std::complex< double > > &a, std::size_t n, std::size_t k, std::size_t step, bool hermitian, buffer< std::complex< double > > &w)
 A complex matrix's update by a pivot of order 1 or 2 (step): hetf2's if hermitian, else sytf2's.
 
template<typename T >
std::vector< std::ptrdiff_t > bunch_kaufman_lower (buffer< T > &a, std::size_t n, bool hermitian, bool last_on_ties, std::size_t from=0)
 Bunch and Kaufman's factorisation of the lower triangle of a (n × n, row-major; entries (i, j) with i ≥ j read and written), as LAPACK's sytf2 and hetf2 on the lower triangle: at each step a pivot of order 1 or 2 chosen with α = (1 + √17) / 8, the rows and columns exchanged, the trailing matrix updated by the pivot's rank-1 or rank-2 correction, L's columns left below the pivot and D on and next to the diagonal. The largest entry of a column is its first (idamax) or, where last_on_ties, its last of equal size: the order of the upper triangle's search when a holds it reversed. From column from, the columns before it factorised already (bunch_kaufman_blocked_lower's last columns, as sytrf leaves them to sytf2).
 
template<typename T >
std::size_t factor_ldl_panel (buffer< T > &a, std::size_t n, std::size_t k0, bool hermitian, bool last_on_ties, buffer< T > &w, std::vector< std::ptrdiff_t > &pivots)
 One panel of bunch_kaufman_blocked_lower from column k0, as LAPACK's lasyf and lahef on the lower triangle: each column brought up to date by the panel's earlier ones (A − L Wᴴ, W = L D in w, n × ldl_block row-major, conjugated for a Hermitian matrix) before its pivot is chosen as bunch_kaufman_lower chooses it, the trailing block left as it was but for the exchanges of its rows and columns, its update deferred. ldl_block − 1 columns, or ldl_block where the last is a block of order 2; more than ldl_block columns remain.
 
template<typename T >
std::vector< std::ptrdiff_t > bunch_kaufman_blocked_lower (buffer< T > &a, std::size_t n, bool hermitian, bool last_on_ties)
 bunch_kaufman_lower by panels, as LAPACK's sytrf and hetrf on the lower triangle: each panel of ldl_block columns factorised with its updates deferred (factor_ldl_panel), then the trailing block's lower triangle less L₂₁ W₂₁ᴴ in one product, a real one of the kernel's size on the triangle's tiles alone, otherwise by bands of rows each to the diagonal; the last ldl_block columns or fewer by bunch_kaufman_lower's steps.
 
buffer< double > entry_moduli (const buffer< double > &a)
 The moduli of a real matrix's entries.
 
buffer< double > entry_moduli (const buffer< std::complex< double > > &a)
 The moduli of a complex matrix's entries: √(x² + y²), which vectorises, where the largest part lies in [2⁻⁵⁰⁰, 2⁵⁰⁰] (no square overflows; one that underflows is of an entry below 2⁻⁵⁰⁰ times the largest's, nothing to a norm); hypot otherwise, as std::abs takes it.
 
double column_sum_norm (const buffer< double > &moduli, std::size_t n)
 The 1-norm of the n × n row-major matrix whose entries' moduli are moduli (n from 1): its largest column sum.
 
template<typename T >
double one_norm (const buffer< T > &a, std::size_t n)
 The 1-norm of the n × n row-major a (n from 1): its largest column sum of moduli.
 
double modulus_power_norm (const buffer< double > &moduli, std::size_t n, std::size_t p, double scale)
 ‖ |A|ᵖ ‖₁ of the n × n row-major moduli |A| (n from 1) times scale, exactly: the row 1ᵀ |scale A|ᵖ by p products of a vector and |A|, a matrix of moduli's norm its largest entry.
 
template<typename T >
void square_product_into (const buffer< T > &a, const buffer< T > &b, std::size_t n, buffer< T > &c)
 The n × n product A B of row-major operands into c, whatever it held (its elements not read, as many made as needed): C = A·B, each part of it zeroed by the thread that computes it (c_start::zero), where a zeroed buffer was filled on the calling thread first. c is neither operand.
 
template<typename T >
buffer< T > square_product (const buffer< T > &a, const buffer< T > &b, std::size_t n)
 The n × n product A B of row-major operands (square_product_into, into storage of its own).
 
template<typename T >
void square_in_place (buffer< T > &x, buffer< T > &spare, std::size_t n)
 x squared count times, the products alternating between its storage and spare's (resized as needed): no fresh n × n mapped and faulted in per squaring.
 
const double * pade_coefficients (std::size_t m)
 The coefficients b₀ … b_m of the [m/m] Padé approximant of the exponential, as Higham's (2005).
 
std::size_t extra_squarings (const buffer< double > &moduli, std::size_t n, std::size_t m, double scale, double norm)
 Al-Mohy and Higham's ℓ(A, m) (2009): the squarings to add so that the [m/m] approximant's backward error bound, from the leading term of its series |c₂ₘ₊₁| ‖ |A|²ᵐ⁺¹ ‖₁ / ‖A‖₁, falls below the unit roundoff; A times scale (a power of two), its norms exact, from its moduli (|A|, formed once for every degree tried: a complex entry's modulus is a hypot).
 
template<typename T >
buffer< T > pade_approximant (const buffer< T > &a, const std::array< const buffer< T > *, 4 > &powers, std::size_t n, std::size_t m)
 The [m/m] Padé approximant (V − U)⁻¹ (V + U) of the exponential of a, its even powers powers (A², A⁴, A⁶, and A⁸ for m = 9), as Higham's (2005): U the odd terms, V the even, A¹³'s terms by A⁶ (A⁶ (b₁₃ A⁶ + b₁₁ A⁴ + b₉ A²) + …).
 
template<typename T >
T sinch (const T &x)
 sinh(x) / x, 1 at 0, by its series where x is small.
 
template<typename T >
void exact_triangular_bands (buffer< T > &x, const buffer< T > &t, std::size_t n, double scale)
 The diagonal and first superdiagonal of exp(scale · T) written exactly into x for the upper triangular t, as Al-Mohy and Higham's code fragment 2.1: e^{λᵢ}, and t_{i,i+1} e^{(λᵢ + λᵢ₊₁)/2} sinch((λᵢ − λᵢ₊₁)/2); past divided_exponentials_past, t_{i,i+1} (e^{λᵢ} − e^{λᵢ₊₁}) / (λᵢ − λᵢ₊₁), where that difference overflows each exponential divided by the gap's modulus within it (e^{λ − log|g|}), so that the band overflows only where it is past the largest double.
 
template<typename T >
bool exactly_hermitian (const matrix< T > &a)
 Whether a is Hermitian to the bit (symmetric where real), its diagonal real: a function of it is then V f(Λ) Vᴴ from one eigendecomposition (Armadillo's expmat_sym, logmat_sympd, sqrtmat_sympd).
 
std::pair< vector< double >, matrix< double > > hermitian_system (const matrix< double > &a)
 The eigenvalues (ascending) and orthonormal eigenvectors of the real symmetric a.
 
std::pair< vector< double >, matrix< std::complex< double > > > hermitian_system (const matrix< std::complex< double > > &a)
 The eigenvalues (ascending) and orthonormal eigenvectors of the complex Hermitian a.
 
template<typename R , typename T >
matrix< R > spectral (const matrix< T > &v, const std::vector< R > &f, bool hermitian)
 V diag(f) Vᴴ in the arithmetic of R (double only for real v and f), the eigenvectors v orthonormal: one product. Made Hermitian to the bit where hermitian (every f real).
 
taylor_plan plan_taylor (double norm, std::size_t columns, const std::function< double(std::size_t)> &power_norm)
 Al-Mohy and Higham's code fragment 3.1 (2011), as SciPy's _fragment_3_1 (m_max 55, ℓ 2): the m and s of least cost m s for t A of 1-norm norm and columns columns of B. Where condition 3.13 holds (‖tA‖₁ small for the columns), s = ⌈‖tA‖₁ / θ_m⌉; else s = ⌈α_p / θ_m⌉ over the p from 2 to 8 and the m from p (p − 1) − 1, α_p = max(d_p, d_{p+1}), d_p power_norm(p) (an estimate of ‖(tA)ᵖ‖₁^{1/p}), at least one step.
 
template<typename T >
double row_sum_norm (const buffer< T > &x, std::size_t n, std::size_t k)
 The largest row sum of moduli of the n × k row-major x (‖X‖∞).
 
template<typename T >
matrix< T > adjoint_of (const matrix< T > &m)
 Mᴴ: the transpose, conjugated.
 
std::size_t schur_block (const buffer< double > &t, std::size_t n, std::size_t k)
 The order of the diagonal block of the upper quasi-triangular t (n × n, row-major) that starts at k: 2 where the entry below its diagonal is nonzero (a real Schur form's pair).
 
std::size_t schur_block (const buffer< std::complex< double > > &, std::size_t, std::size_t)
 The same for a complex Schur form, triangular: 1.
 
template<typename T >
void small_sylvester_block (const T *r, std::size_t ldr, const T *s, std::size_t lds, std::size_t rows, std::size_t cols, T *c, std::size_t ldc, double smin)
 The rows × cols y of R y + y S = c (each at most 2, row-major in c, overwritten) for the diagonal blocks r (row stride ldr) and s (row stride lds): the Kronecker system of order rows · cols by Gaussian elimination with complete pivoting, a pivot under smin taken as smin, as LAPACK's dlasy2 perturbs one (the eigenvalues of R and −S close).
 
template<typename T >
void triangular_sylvester (const buffer< T > &r, std::size_t m, const buffer< T > &s, std::size_t n, buffer< T > &f)
 Y of R Y + Y S = F in place in f (m × n, row-major) for the upper quasi-triangular (real) or triangular (complex) r (m × m) and s (n × n), as LAPACK's trsyl (no transposes, +): by blocks of columns of S from the first, each first relieved of the columns solved before it through the matrix product, then by blocks of rows of R from the last, each a system of at most four unknowns (small_sylvester_block), R and −S close in an eigenvalue perturbing its pivot as trsyl does.
 
double frechet_bound (std::size_t m)
 ‖A‖₁ bounds of the [m/m] Padé approximants of the Fréchet derivative of the exponential, for m from 1 to 20, as Al-Mohy and Higham's table 6.1 (2009) and SciPy's ell_table_61.
 
template<typename T >
void add_scaled (buffer< T > &into, const buffer< T > &term, double coefficient)
 into += coefficient · term over the n × n buffers.
 
template<typename T >
buffer< T > both_products (const buffer< T > &p, const buffer< T > &q, std::size_t n)
 P Q + Q P of n × n buffers.
 
template<typename T >
std::pair< buffer< T >, buffer< T > > frechet_pade (const buffer< T > &a, const buffer< T > &e, std::size_t n, std::size_t m)
 e^A and its Fréchet derivative L(A, E) from the [m/m] Padé approximant, as Al-Mohy and Higham's algorithm 6.4 (2009) and SciPy's expm_frechet_algo_64: U, V and their derivatives Lu, Lv along e (M₂ = A E + E A, M₄ = A² M₂ + M₂ A², M₆ = A⁴ M₂ + M₄ A², M₈ = A⁴ M₄ + M₄ A⁴), each sum from its highest term, then R = (V − U)⁻¹ (U + V) and L = (V − U)⁻¹ (Lu + Lv + (Lu − Lv) R) by one LU factorisation; for m = 13, a and e already scaled, A⁶'s nesting as pade_approximant's.
 
template<typename T >
void square_root_columns (const buffer< T > &t, buffer< T > &r, std::size_t n, const std::vector< std::size_t > &starts)
 The off-diagonal blocks of the square root r (n × n, row-major) of the upper (quasi-)triangular t, its diagonal blocks already in r, column of blocks by column, each block row from the last: R_ii X + X R_jj = T_ij − Σ R_ik R_kj (Higham, 1987), between two scalars x = c / (r_ii + r_jj), a zero over a zero taken as 0 and a nonzero over a zero as infinite (no root there, as SciPy answers), else the small Sylvester system of the blocks (small_sylvester_block; principal roots of eigenvalues off the negative axis never sum to 0 there).
 
std::optional< matrix< double > > real_square_root (const matrix< double > &form, const matrix< double > *z)
 The real principal square root of the real a from its real Schur form (Higham, 1987), or none where a has a real eigenvalue below 0 (its principal root complex): a 1 × 1 block's root √t, a 2 × 2 block [θ b; c θ] of eigenvalues θ ± iμ the root α I + (T − θ I) / 2α for α + iβ = √(θ + iμ).
 
double scalar_sqrt (double x)
 √x of a real x at least 0.
 
std::complex< double > scalar_sqrt (const std::complex< double > &z)
 csqrt's √z, exact on the axes as NumPy's and SciPy's.
 
template<typename T >
buffer< T > pointwise_square_root (const buffer< T > &t, std::size_t n)
 The principal square root of the upper triangular n × n row-major t (no real eigenvalue below 0 where T is real), entry by entry: √t_ii on the diagonal, the rest by square_root_columns.
 
template<typename T >
buffer< T > triangular_square_root (const buffer< T > &t, std::size_t n)
 The principal square root of the upper triangular n × n row-major t (no real eigenvalue below 0 where T is real): pointwise_square_root up to square_root_block or where a diagonal entry is 0 (a zero over a zero taken as 0 there); else by blocks (Deadman, Higham and Ralha, 2013, as SciPy's _sqrtm_triu): each diagonal block's root entry by entry, then each block column from the diagonal up, R_ii X + X R_jj = T_ij − R_i,(i+1…j−1) R_(i+1…j−1),j, the product by gemm and the Sylvester equation by triangular_sylvester (R_ii + R_jj has no eigenvalue at 0: principal roots of nonzero eigenvalues).
 
matrix< std::complex< double > > complex_square_root (const matrix< std::complex< double > > &form, const matrix< std::complex< double > > *z)
 The principal square root of a complex matrix from its Schur form (triangular_square_root).
 
std::pair< matrix< std::complex< double > >, matrix< std::complex< double > > > complex_schur_form (const matrix< double > &t, const matrix< double > &z)
 The complex Schur form of a real one, as SciPy's rsf2csf: each 2 × 2 block of t (a conjugate pair) triangularised by a unitary rotation of its two rows and columns, the columns of z turning with it; a real eigenvalue's diagonal entry kept exactly real.
 
template<typename T >
bool upper_triangular (const matrix< T > &a)
 Whether the square a is upper triangular: its Schur form as it stands, Z = I (as SciPy takes it, so an infinite entry of the root reaches no other).
 
template<typename T >
matrix< std::complex< double > > hermitian_root (const matrix< T > &a)
 The principal square root of the Hermitian a, V √Λ Vᴴ: Hermitian where no eigenvalue is below 0.
 
std::complex< double > rounding_direction (std::size_t place)
 A direction of unit modulus for the rounding of entry place of an estimate's model, fixed and varying from one entry to the next.
 
void solve_upper_blocks (const matrix< std::complex< double > > &t, const std::vector< std::size_t > &starts, buffer< std::complex< double > > &x, const std::complex< double > *sources)
 
double recurrence_error (const matrix< std::complex< double > > &t, const std::vector< std::size_t > &starts, const buffer< std::complex< double > > &fx, buffer< std::complex< double > > error)
 The error of f(T), fx, from the block recurrence (solve_upper_blocks) over the runs starts of t's diagonal, relative to its largest entry: the recurrence run again on the error alone, from its diagonal blocks' errors (error, n × n, its diagonal blocks set: circle_error for a cluster, f's rounding for one eigenvalue), each entry above them with its rounding as a source, ε (Σ |F_ik| |T_kj| + |T_ik| |F_kj|) over the products that form it (the moduli as |re| + |im|), in a direction of its own (rounding_direction). Where clusters are far from normal, the separation of their blocks falls far below their eigenvalues' gaps and the recurrence grows the diagonal blocks' errors and its own roundings by its inverse: eight clusters of 32 at order 256 (spread 0.01, T's strict upper triangle 0.3 N(0, 1)), integrals right to 10⁻¹¹ each and f(T) wrong by 10⁻⁴ above them (by 10⁻⁶ from the roundings alone, a recurrence in 30 digits from the same blocks shows), where SciPy's estimate, ε ‖strict upper T‖₁ over the smallest gap, said 7·10⁻¹³; this one says 2·10⁻⁴.
 
double parlett_estimate (const matrix< std::complex< double > > &t, double smallest)
 SciPy's estimate for Parlett's recurrence on t: min(1, max(ε, ε ‖strict upper T‖₁ / δ)), δ smallest (ε where 0).
 
std::pair< matrix< std::complex< double > >, matrix< std::complex< double > > > function_schur_form (const matrix< double > &a)
 A real matrix's complex Schur form as scipy.linalg.funm takes it: the real form made complex (rsf2csf).
 
std::pair< matrix< std::complex< double > >, matrix< std::complex< double > > > function_schur_form (const matrix< std::complex< double > > &a)
 A complex matrix's Schur form.
 
std::pair< funm_result, double > parlett_function (const matrix< std::complex< double > > &t, const matrix< std::complex< double > > &z, const std::vector< std::complex< double > > &values)
 funm_from_schur with the recurrence's own estimate beside (recurrence_error, without SciPy's floor, which reads 1 wherever T(0, 0) is 0): what funm weighs its attempts by.
 
std::vector< std::size_t > eigenvalue_clusters (const std::vector< std::complex< double > > &values, double delta)
 The clusters of values joined where two lie within delta (transitively), numbered by their first member's place.
 
double cluster_bound (const std::vector< std::complex< double > > &values, double share)
 The clusters' bound: eigenvalues within share of the largest modulus (or of 1) gather (cluster_share, Davies and Higham's bound, relative here). Parlett's recurrence divides by the gaps between clusters: under it, gaps of 0.003 to 0.03 and an evenly spread spectrum lost 10⁶ to 10¹³ ε; a cluster's Cauchy integral takes as many points as its estimate asks (contour_points_most), a random matrix's whole spectrum among them.
 
bool clusters_separate (const std::vector< std::complex< double > > &values, double share)
 Whether the clusters of values within share are sound for circles_of, each cluster's farthest eigenvalue nearer its centre than every other one (no circle about it could keep the others out else), read from the values alone, before gather_clusters reorders a Schur form for them: a random matrix's spectrum chains into clusters no circle separates, whose reordering funm then gave up for the finer share (order 256: 12.7 of 41 ms). values not empty.
 
std::vector< std::complex< double > > diagonal_of (const matrix< std::complex< double > > &t)
 The diagonal of the square t.
 
cluster_circles circles_of (const matrix< std::complex< double > > &t, double share)
 cluster_circles of the Schur form t whose clusters are runs of its diagonal: a circle about each cluster of two or more, its radius the cluster's spread plus a margin as wide as the spread, the cluster bound, or the block's coupling (its largest entry above the diagonal: the resolvent's terms ‖N‖ᵏ / rᵏ⁺¹ then stay near 1/r and cancel without loss), half the way to the nearest other eigenvalue at most.
 
std::vector< std::size_t > starting_points (const cluster_circles &circles)
 The points to start each circle of circles about a cluster of two or more with: contour_points, funm doubling those its integral's estimate asks more of.
 
std::vector< std::complex< double > > circle_points (const cluster_circles &circles, const std::vector< std::size_t > &counts)
 The points of circles, counts on the circles about clusters of two or more in turn, after one another.
 
bool gather_clusters (matrix< std::complex< double > > &t, matrix< std::complex< double > > &z, double share)
 Reorders the complex Schur form t, z so each cluster of eigenvalues (eigenvalue_clusters, within cluster_bound) is a run of the diagonal, each in turn moved to the first place past those placed.
 
void add_circle_terms (const matrix< std::complex< double > > &t, std::size_t first, std::size_t p, std::complex< double > centre, double radius, std::size_t count, std::size_t from, std::size_t step, const std::complex< double > *images, buffer< std::complex< double > > &sum, buffer< double > &moduli)
 Adds to sum, for the points k = from, from + step, … < count of the grid of count points on the circle (centre, radius), f(z_k) (z_k − c) (z_k I − T_b)⁻¹, T_b the p × p block of t from first and f(z_k) the next of images in turn: the trapezoidal rule's terms of Cauchy's integral (1/2πi) ∮ f(z) (zI − T_b)⁻¹ dz (Hale, Higham and Trefethen, 2008), times the count.
 
std::vector< std::complex< double > > grid_points (std::complex< double > centre, double radius, std::size_t count, std::size_t from, std::size_t step)
 The points k = from, from + step, … < count of the grid of count points on the circle (centre, radius), in turn.
 
double circle_integral (buffer< std::complex< double > > &whole, buffer< std::complex< double > > &half, const buffer< double > &moduli, std::size_t p, std::size_t count, const std::complex< double > *diagonal)
 A circle's integral from its rule's sum over count points, whole, and over the even ones, half (each divided by its count here), and its own estimate, relative to the integral's largest entry: the two rules' largest difference, the diagonal's against diagonal (f at the cluster's eigenvalues: a singularity of f in the disk breaks it), and the sum's rounding, ε times its terms' moduli (moduli, summed over the count points). A cluster far from normal has resolvents on its circle whose terms cancel to an integral orders below them (eight clusters of 32 at order 256: an integral wrong by 2·10⁻⁴ where the rules agreed to 4·10⁻¹¹); more points do not bring those digits back.
 
buffer< std::complex< double > > circle_error (const buffer< std::complex< double > > &whole, const buffer< std::complex< double > > &half, const buffer< double > &moduli, std::size_t count)
 The error of a circle's integral whole as recurrence_error carries it above the diagonal: the two rules' difference (whole − half, a function of the block as its integral's error is), and its sum's rounding, ε times its terms' moduli (moduli, summed over the count points), each entry in a direction of its own. whole and half as circle_integral leaves them (p × p).
 
std::tuple< buffer< std::complex< double > >, double, double > assembled_function (const matrix< std::complex< double > > &t, const cluster_circles &circles, const std::vector< buffer< std::complex< double > > > &blocks, const std::vector< buffer< std::complex< double > > > &errors, const std::vector< std::complex< double > > &at_eigenvalues)
 The blocks of x above its diagonal (n × n, row-major, its diagonal blocks given) by the block recurrence of Davies and Higham's Schur–Parlett algorithm, a block superdiagonal at a time: T_ii X_ij − X_ij T_jj = X_ii T_ij − T_ij X_jj + Σ (X_ik T_kj − T_ik X_kj) + S_ij, a triangular Sylvester equation each, the clusters as runs of t's diagonal (starts, each run's first place, then n), and S sources (n × n), or none where null.
 
std::tuple< buffer< std::complex< double > >, double, std::vector< double > > clustered_function (const matrix< std::complex< double > > &t, const std::vector< std::complex< double > > &images, double share, const std::vector< std::size_t > &counts)
 f(T) of the Schur form t with clusters within share as runs of its diagonal, from images, f at the eigenvalues then at each cluster's circle (circles_of), counts points each: each cluster's block by its integral (add_circle_terms, its even points' rule beside for the estimate, circle_integral), the rest by assembled_function.
 
funm_result funm_answer (matrix< std::complex< double > > &&x, double error, std::vector< double > &&contours, const std::vector< std::complex< double > > &at_eigenvalues)
 x with its estimate error, ∞ where an entry of x or f at an eigenvalue (at_eigenvalues) is not finite (no f(A) then, whatever the circles gave), and the circles' estimates contours.
 
double unwinding (const std::complex< double > &z)
 ceil((Im z − π) / 2π), the unwinding number of z (SciPy's _unwindk).
 
std::complex< double > briggs (std::complex< double > a, std::size_t k)
 a^(1/2^k) − 1 without the cancellation of the subtraction, as SciPy's _briggs_helper_function (Al-Mohy and Higham's algorithm 2, 2012): (a − 1) / Π (1 + a^(1/2^j)), a first square root taken where arg a ≥ π/2.
 
std::complex< double > fractional_power_superdiagonal (const std::complex< double > &l1, const std::complex< double > &l2, const std::complex< double > &t12, double p)
 The superdiagonal entry of T^p for the 2 × 2 upper triangular [[l1, t12], [0, l2]], as SciPy's _fractional_power_superdiag_entry (Higham and Lin's equation 5.5, 2011).
 
std::complex< double > log_superdiagonal (const std::complex< double > &l1, const std::complex< double > &l2, const std::complex< double > &t12)
 The superdiagonal entry of log T for the 2 × 2 upper triangular [[l1, t12], [0, l2]], as SciPy's _logm_superdiag_entry.
 
double briggs (double a, std::size_t k)
 briggs of a real a above 0: (a − 1) / Π (1 + a^(1/2^j)), for k = 1 too (where √a − 1 would cancel).
 
double fractional_power_superdiagonal (double l1, double l2, double t12, double p)
 fractional_power_superdiagonal of real l1 and l2 above 0 (the unwinding number 0).
 
double log_superdiagonal (double l1, double l2, double t12)
 log_superdiagonal of real l1 and l2 above 0 (the unwinding number 0).
 
double scalar_log (double x)
 log x of a real x above 0.
 
std::complex< double > scalar_log (const std::complex< double > &z)
 complex_log of z.
 
bool principal_branch (const buffer< double > &, std::size_t)
 log has its principal branch at every diagonal entry of a real triangular form logm keeps real: none below 0, and its zeros taken as 10⁻²⁰ (force_nonsingular).
 
bool principal_branch (const buffer< std::complex< double > > &t, std::size_t n)
 Whether log has its principal branch at every diagonal entry of t (none real and at most 0).
 
template<typename T >
std::tuple< buffer< T >, std::size_t, std::size_t > inverse_squaring (const buffer< T > &t0, std::size_t n, const std::array< double, 8 > &theta=log_pade_bounds, const char *name="logm")
 Al-Mohy and Higham's inverse scaling and squaring (2012, algorithm 4.1) on the upper triangular t0, as SciPy's _inverse_squaring_helper: s square roots of T until T − I is small enough for the [m/m] Padé approximant whose bounds are theta (log(I + X)'s, or (I − X)ᵗ's; m from 1 to 7), the norms ‖(T − I)ᵖ‖₁^(1/p) exact; R = T − I with its diagonal and superdiagonal recomputed without cancellation where the principal branch exists.
 
template<typename T >
buffer< T > triangular_log (const buffer< T > &t0, std::size_t n)
 The logarithm of the upper triangular t0 (no zero on its diagonal), as SciPy's _logm_triu: R, s and m by inverse_squaring, U = 2ˢ Σ αₖ (I + βₖ R)⁻¹ R (the [m/m] Padé approximant as the partial fractions of m-point Gauss–Legendre quadrature on [0, 1], SciPy's nodes and weights), then the diagonal log λᵢ and the superdiagonal recomputed exactly where the principal branch exists.
 
template<typename T >
std::pair< bool, bool > force_nonsingular (buffer< T > &t, std::size_t n)
 The triangular form t made nonsingular as SciPy's _logm_force_nonsingular_triangular_matrix: a zero diagonal entry replaced by 10⁻²⁰.
 
const matrix< std::complex< double > > & as_complex (const matrix< std::complex< double > > &a)
 a itself: already complex.
 
matrix< std::complex< double > > as_complex (const matrix< double > &a)
 a's entries as complex numbers.
 
std::complex< double > principal_power (const std::complex< double > &z, double p)
 z to the real power p, principal: exp(p log z) (0 for z = 0 and p > 0).
 
double power_pade_constant (std::size_t i, double t)
 The coefficient cᵢ (i ≥ 1) of the continued fraction of (1 − x)ᵗ, as SciPy's _fractional_power_pade_constant (Higham and Lin, 2011, equation 4.1).
 
buffer< std::complex< double > > power_pade (const buffer< std::complex< double > > &x, std::size_t n, double t, std::size_t m)
 The [m/m] Padé approximant of (I − X)ᵗ for the upper triangular x, as SciPy's _fractional_power_pade: its continued fraction from the bottom, Y = c₂ₘ X, then Y = (I + Y)⁻¹ cⱼ X for j = 2m − 1, …, 1, and I + Y.
 
buffer< std::complex< double > > triangular_power (const buffer< std::complex< double > > &t0, std::size_t n, double t)
 T^t (−1 < t < 1) of the upper triangular t0 (no zero on its diagonal), as SciPy's _remainder_matrix_power_triu (Higham and Lin's Schur–Padé algorithm, 2013): a diagonal T entry by entry; else R = T^(1/2ˢ) − I and m by inverse_squaring with the bounds of (I − X)ᵗ, the [m/m] Padé approximant at −R, then s squarings. Where the principal branch exists, the diagonal λᵢ^(t/2ⁱ) and the superdiagonal (fractional_power_superdiagonal) are written exactly after each squaring, as Higham and Lin's algorithm 5.1 does (SciPy writes them before the first squaring only).
 
buffer< std::complex< double > > nearly_diagonal_power (const buffer< std::complex< double > > &t, std::size_t n, double p)
 T^t of a triangle t of order n whose strict upper part is a similarity's rounding (Zᴴ A Z, Z a Hermitian A's eigenvectors): its diagonal's principal powers, above them each entry times the divided difference of x^t between its diagonal entries (fractional_power_superdiagonal), Daleckii and Krein's first order, exact to the square of the rounding.
 
buffer< double > nearly_diagonal_power (const matrix< double > &similar, double p)
 nearly_diagonal_power of the upper triangle of a real similar whose diagonal is positive: real.
 
template<typename T >
hermitian_similarity< T > similarity_of (const matrix< T > &a)
 hermitian_similarity of the Hermitian a.
 
buffer< std::complex< double > > similar_power (const matrix< std::complex< double > > &similar, double t)
 T^t for T the upper triangle of similar (Zᴴ A Z, Z a Hermitian A's eigenvectors): the similarity's rounding above the diagonal kept as a Schur form keeps it (Λ alone erred two to five times more), its diagonal taken real (a negative eigenvalue's power the principal value's), T^t to first order in that rounding (nearly_diagonal_power, in place of the Padé steps: order 200 35 → 9 ms, the same errors).
 
matrix< std::complex< double > > hermitian_remainder_power (const hermitian_similarity< double > &s, double t)
 A^t (−1 < t < 1) of a real symmetric A not triangular from s: V T^t Vᵀ, T the upper triangle of Vᵀ A V (similar_power), the products real: T^t real where T's diagonal is positive (nearly_diagonal_power's in real arithmetic), else its real and imaginary parts apart.
 
matrix< std::complex< double > > hermitian_remainder_power (const hermitian_similarity< std::complex< double > > &s, double t)
 A^t (−1 < t < 1) of a complex Hermitian A not triangular from s: Z T^t Zᴴ, T the upper triangle of Zᴴ A Z (similar_power).
 
template<typename T >
std::optional< matrix< std::complex< double > > > remainder_power (const matrix< T > &a, double t, const std::optional< hermitian_similarity< T > > &similarity)
 A^t (−1 < t < 1) by the Schur form, as SciPy's _remainder_matrix_power: an upper triangular A directly, else Z T^t Zᴴ; none where the triangular form has a zero on its diagonal. An exactly Hermitian A not triangular by its eigenvectors (hermitian_remainder_power); a diagonal one's diagonal taken real, so that a negative eigenvalue's power is the principal value's.
 
template<typename T >
matrix< T > integer_power (const matrix< T > &a, double e)
 a to the integer power e (a finite whole number) by repeated squaring, as numpy.linalg.matrix_power: the identity for 0, the inverse's powers for e < 0.
 
template<typename T >
logm_result schur_logm (buffer< T > t, std::size_t n, const matrix< T > *z)
 log A from the upper triangular n × n t of A = Z T Zᴴ (z, or none where T is A), its zero diagonal entries taken as 10⁻²⁰ (force_nonsingular).
 
bool nonnegative_diagonal (const matrix< double > &t)
 Whether no diagonal entry of t is below 0.
 
logm_result general_logm (const matrix< std::complex< double > > &a)
 logm of a complex matrix past its Hermitian path: an upper triangular one directly, else its Schur form's.
 
logm_result general_logm (const matrix< double > &a)
 logm of a real matrix past its Hermitian path: in real arithmetic where its real Schur form (or the matrix, upper triangular) is triangular with no diagonal entry below 0, as SciPy's (_logm, _logm_triu), else in the complex form (rsf2csf).
 
void release_and_trim ()
 release_thread_memory, then the allocator's free pages to the system, on the calling thread: glibc keeps an arena for each thread that allocates, whose free pages the thread trims best itself.
 
void release_memory_on (thread_pool *pool)
 numerical::release_memory with pool's workers, none where it is nullptr (a pool never started holds nothing, and starting it would cost threads, not free memory).
 
long current_process () noexcept
 This process's identifier: a pool started before a fork belongs to the parent, whose threads the child does not have.
 
std::vector< std::size_t > worker_cores (std::size_t cores, std::size_t threads)
 The core of each worker the pool starts beside its callers, from the cores known (0: none) and its threads (at least 1): one worker per core, pinned there, where the cores are known (each run then leaves out the one on the caller's core, sits_out); else threads − 1 workers free to move.
 
constexpr bool sits_out (std::size_t worker, std::size_t caller) noexcept
 Whether the worker pinned to core worker sits out a run whose caller is on core caller: one thread per core, as two on one core's hyperthreads share its multiply-add units. A free worker never does.
 
std::size_t physical_cores () noexcept
 The physical cores this process may run on, or 0 where the system does not say. Linux: core_sets; macOS: hw.physicalcpu.
 
constexpr std::size_t pool_threads (std::size_t logical, std::size_t physical, std::size_t quota=0) noexcept
 The pool's threads from the logical hardware threads, the physical cores and the processors a CPU quota allows (each 0: unknown, or none): the cores where known, no more than the quota, at least 1.
 
constexpr std::size_t quota_processors (long long quota, long long period) noexcept
 The processors a CPU quota of quota microseconds a period allows, rounded up (cgroup v1's cpu.cfs_quota_us, -1 for none, and cpu.cfs_period_us): 0 for none.
 
std::size_t cpu_max_processors (std::string_view text) noexcept
 quota_processors of cgroup v2's cpu.max text, "<quota> <period>" ("max" for none): 0 for none or a text it does not read.
 
std::size_t quota_limit () noexcept
 The processors the CPU quotas over this process allow (cgroup_processors on Linux), 0 for none.
 
std::size_t hardware_threads () noexcept
 The threads the pool runs, the caller's included (at least 1), known without starting the pool: one per physical core where the system tells them (else one per hardware thread), no more than a CPU quota allows (quota_limit). Two threads on one core's hyperthreads share its multiply-add units: a product of order 128 on eight AVX2 cores with sixteen hyperthreads took 140 µs on sixteen threads, 71 on eight; order 2048, the same either way. Asked once, as the pool's workers are started once (glibc reads a file for the hardware threads on every call).
 
template<typename Ready >
void spin_until (const Ready &ready)
 Polls ready, yielding the core between polls, until it holds or spin_time has passed.
 
constexpr bool wants (std::size_t core, std::size_t caller, std::size_t helpers) noexcept
 Whether the worker on core joins a run of helpers workers whose caller is on caller: by rank among the cores, the caller's aside, so a run of few threads takes the same cores from one run to the next.
 
std::atomic< std::size_t > & thread_cap () noexcept
 The most threads a parallel_for uses (0: as many as the pool has), set by numerical::set_max_threads.
 
std::size_t threads_for (std::size_t count, std::size_t work_per_index, std::size_t hardware)
 How many threads count indices of work_per_index each pay for: one per work_per_thread, at most one per index and per hardware thread.
 
std::size_t parallel_threads (std::size_t count, std::size_t work_per_index)
 The threads parallel_for shares count indices of work_per_index out to (threads_for): the hardware's, at most the cap numerical::set_max_threads set. Fewer than two runs them on the caller.
 
template<typename Body >
void parallel_for (std::size_t count, std::size_t work_per_index, const Body &body)
 Runs body(i) for every i in [0, count).
 
template<typename Body , typename Side >
void parallel_for_beside (std::size_t count, std::size_t work_per_index, const Body &body, const Side &side)
 Runs side on the caller while the pool's workers run body(i) for every i in [0, count), then lets the caller take the chunks left: one task beside a parallel loop, each touching what the other does not (a factorisation's next panel beside its trailing update). side's own parallel loops find the pool busy and run in its thread. Where the work pays for fewer than two threads, or the pool is busy, side runs, then the loop, on the caller. An exception of side comes back after the loop's chunks have joined, before theirs.
 
template<typename T >
constexpr std::size_t fault_chunk (std::size_t count) noexcept
 The elements one index of parallel_zero or parallel_copy takes of count: zero_chunk, or, from the size at which a buffer takes Linux's 2 MiB pages (16 MiB), one such page's worth, so that no two threads fault in the same page: they waited on each other's fault there (six x86-64 AVX2 cores, 32 MiB zeroed into fresh memory 3.6 → 2.5 ms; the LU and Cholesky of order 2000, their operand copied by eight 4 KiB pages a thread, took 2 to 4 % longer than with one thread's copy).
 
template<typename T >
void parallel_zero (T *out, std::size_t count)
 Sets the count elements at out to zero, split across the threads the work pays for: a fresh result's pages are first touched, and faulted in, by several threads, where a sequential fill pays every fault on one (a product on eight arm64 threads: 16 % faster at order 512, 7 % at 1024). The zero is a constant in each chunk's fill, which the compiler turns into the C library's zeroing (on arm64 it clears whole cache lines without reading them; a value passed in, stored element by element, was 4 % slower at order 1024 on one thread).
 
template<typename T >
void parallel_fill (T *out, std::size_t count, const T &value)
 Sets the count elements at out to value, split across the threads as parallel_zero is, a fresh result's pages faulted in by several threads.
 
template<typename T >
void parallel_copy (const T *in, std::size_t count, T *out)
 Copies the count elements at in to out, split across the threads the work pays for, as parallel_zero does: a fresh copy's pages are faulted in by several threads, where a sequential copy pays every fault on one (a 2 MiB copy into fresh memory took 0.85 ms on one x86-64 core in a container, a third of the Cholesky factorisation of order 512).
 
void copy_few_bytes (const unsigned char *from, std::size_t bytes, unsigned char *to) noexcept
 Copies bytes bytes, 4 to 128, from from to to (apart): by two or four copies of one fixed size that overlap where the count is not a multiple of it, the last ending at the last byte. A few elements copied by a loop went to the library's memmove, a call a row of a tile's edge (one M1 Pro core, float products of order 100 to 200 on the AMX: 4 to 5 % of their time).
 
template<typename T >
void copy_few (const T *from, std::size_t count, T *to) noexcept
 copy_few_bytes of count elements of T, one at least and at most 128 bytes.
 
std::size_t transpose_piece (std::size_t from_bytes, std::size_t to_bytes) noexcept
 The side of the square pieces a transposing copy walks a tile by, from the bytes between rows of its source and of its result: a tile's 32 rows a multiple of a power of two of bytes apart fall in a few of the L1 cache's sets and evict one another, where 8 rows fit them. Pieces of 8 where either stride is a multiple of 1 KiB on x86-64 (sets every 4 KiB: four AVX2 cores, 256 × 256 doubles 165 → 51 µs, 128 × 128 15.8 → 10.3, 2048 × 2048 15.4 → 9.2 ms; ten AVX-512 cores 256 × 256 142 → 51 µs), of 8 KiB on arm64 (M1 Pro, 1024 × 1024 3.8 → 0.95 ms, 2048 × 2048 19.7 → 7.2); elsewhere the whole tile, which pieces slowed (100 × 100 on the AVX2 cores 10.5 → 11.6 µs, most orders on the M1 Pro 10 to 40 %).
 
template<std::size_t Piece, typename Copy >
void transpose_pieces (std::size_t i0, std::size_t i1, std::size_t j0, std::size_t j1, const Copy &copy)
 copy(i, j) for every i in [i0, i1) and j in [j0, j1), by square pieces of side Piece, j the faster within a piece.
 
template<typename Copy >
void transpose_walk (std::size_t i0, std::size_t i1, std::size_t j0, std::size_t j1, std::size_t piece, const Copy &copy)
 transpose_pieces of side piece (transpose_piece's 8 or the whole tile), each side its own instance: a side known only at run time slowed the whole tile's copy (100 × 100 doubles, four AVX2 cores, 10.4 → 11.2 µs).
 
template<typename T >
std::array< T, symmetric_row_group > symmetric_rows_tail (const T *rows, std::size_t n, const T *v, T *out, std::size_t i, std::size_t done, std::array< T, symmetric_row_group > sums)
 The columns done…i+r of symmetric_rows, one by one: for each row r in order, out[k] += row_r[k] · v[i+r] for done ≤ k < i + r and row_r[k] · v[k] added to sums[r], its diagonal term last.
 
template<std::size_t Rows, std::size_t Cols, typename T , typename Full >
void through_edge (T *c, std::size_t ldc, std::size_t rows, std::size_t cols, Full full) noexcept
 One tile of a matrix product kernel, rows × cols of C (at most Rows × Cols, leading dimension ldc), through full, which computes a whole tile in place given its first element and leading dimension: a whole tile in C itself, a partial one in a copy whose outside is zero. Only that copy is zeroed: zeroing it for every tile, whole ones included, took a twelfth of an x86-64 product's time.
 
template<std::size_t R, typename T , typename Full >
void rows_of_tile (std::size_t rows, T *c, std::size_t ldc, Full &full) noexcept
 full's tile of its first R rows, or of rows of them where fewer (through_rows).
 
template<std::size_t Rows, std::size_t Cols, typename T , typename Full >
void through_rows (T *c, std::size_t ldc, std::size_t rows, std::size_t cols, Full full) noexcept
 through_edge for a tile full computes for any count of its rows up to Rows, its template argument: a tile of its whole Cols columns and fewer rows computed in C itself, those rows alone (one row of complex products had six rows' multiply-adds computed through a whole tile); any other through a copy, as through_edge, its rows alone computed there too. Each element of C has the same operations either way.
 
template<typename T >
std::size_t subtract_scaled_vectors (T *, const T *, const T &, std::size_t)
 No leading elements in vectors, for a type no native vector holds.
 
template<typename T , typename Magnitude >
std::size_t largest_magnitude_vectors (const T *, std::size_t, Magnitude &, std::size_t &)
 Only x[0], the one best and at come in with, for a type no native vector holds.
 
template<typename T >
std::size_t subtract_combined_vectors (T *, const T *, const T *, std::size_t, std::size_t, std::size_t)
 No leading blocks in vectors, for a type no native vector holds or a compiler that vectorises the scalar block itself (Clang).
 
template<typename T >
std::size_t subtract_rows_vectors (T *, std::size_t, const T *, std::size_t, const T *, std::size_t, std::size_t, std::size_t)
 No leading columns in vectors, for a type no native vector holds or a compiler that vectorises the scalar block itself (Clang).
 
template<typename T >
std::size_t add_scaled_vectors (T *, const T *, const T &, std::size_t)
 No leading elements in vectors, for a type no native vector holds.
 
template<typename T >
T symmetric_row (const T *row, const T *v, const T &f, T *out, std::size_t count)
 Σ row[i] · v[i] and out[i] += row[i] · f, in order, for a type no native vector holds.
 
template<typename T >
std::array< T, symmetric_row_group > symmetric_rows (const T *rows, std::size_t n, const T *v, T *out, std::size_t i)
 symmetric_rows one column at a time, for a type no native vector holds.
 
template<typename T >
T dot_product (const T *x, const T *y, std::size_t count)
 Σ x[i] · y[i], in order, for a type no native vector holds.
 
template<typename T >
T conjugate_dot_product (const T *x, const T *y, std::size_t count)
 Σ conj(x[i]) · y[i], in order, where no native vector holds the parts.
 
template<typename T >
void row_dots (const T *m, std::size_t ld, std::size_t rows, const T *x, std::size_t count, T *out)
 dot_product of each row of m with x, for a type no native vector holds.
 
template<typename T >
T operator_times (const T &a, const T &b)
 a · b, as std::operator* computes it, in line. libc++'s complex product (the four products and their sums as statements of their own, so unfused) is copied here, its rescue of a product whose parts are both NaN left to it: where it was a call, out of line in a translation unit as large as the Python extension, it took 45 % of a complex LU of order 48. Elsewhere (libstdc++, whose product the compiler expands) and for a real T, the operator itself.
 
template<typename T >
void subtract_product (T *y, const T &f, const T *x, std::size_t count)
 y[i] -= f · x[i] for count elements of y and x, which do not overlap. Each element is computed exactly as that scalar statement computes it (its product rounded before its difference, or fused with it where the compiler contracts the statement); the arguments follow the expression, since a compiler may contract f · x and x · f of complex numbers differently. Under GCC, over native vectors where T has them: at -O2 GCC leaves the scalar loop scalar, unable to rule out an overlap without a check it will not add. Clang vectorises the scalar loop itself, with that check, and better than these vectors (an LU solve of order 128 on arm64 took 19 % longer through them). For double, gemm.hpp's overload chooses the x86-64 units at run time, a fused multiply-add per element there.
 
template<typename T >
T less_product (T held, const T &factor, const T &x)
 held − factor · x for the scalar blocks beside GCC's vectors, rounded as those vectors round it: under GCC the product a statement of its own, since GCC fuses no multiply-add of its vectors but contracts a scalar expression where its target has the instruction (an x86-64-v3 baseline, the default of Ubuntu 26.04's GCC on GitHub's runners, or arm64), and a row would then be rounded two ways, split by the count each call gets (an inverse of order 129 by column slices on two threads: 2700 of its entries changed). Elsewhere the expression as written, as the compiler rounds it (Clang vectorises these blocks itself, with no vectors of ours beside them).
 
template<typename T >
void subtract_products (T *y, const T *f, const T *x, std::size_t ldx, std::size_t terms, std::size_t count)
 y[i] -= Σₜ f[t] · x[(t · ldx) + i] for count elements of y, terms rows of x apart by ldx: subtract_product for each term in ascending order, so each element is computed as that sequence of calls computes it. For double, gemm.hpp's overload holds the elements in registers through the terms.
 
template<typename T >
requires std::is_floating_point_v<T>
void subtract_products (T *y, const T *f, const T *x, std::size_t ldx, std::size_t terms, std::size_t count)
 subtract_products for a real T: eight elements of y held in a local array through the terms, which the compiler keeps in registers, where each term was a call through the whole row. Each element takes its terms in the same order as a sequence of subtract_product gives them (an LU solve of order 64 and four columns on arm64: 18 → 8.5 µs), and the same value under Clang, and under GCC on x86-64, which fuses neither; under GCC on arm64 the vectors of subtract_combined_vectors and subtract_scaled_vectors round apart and the scalar statements fused, and their splits differ.
 
template<typename T >
requires std::is_floating_point_v<T>
void subtract_rows (T *y, std::size_t ldy, const T *f, std::size_t ldf, const T *x, std::size_t ldx, std::size_t terms, std::size_t count)
 subtract_products for product_rows rows of y at once, ldy apart, each with its own factors (ldf apart in f) on the same rows of x: y_r[i] -= Σₜ f_r[t] · x[(t · ldx) + i]. Each element takes the statement of subtract_products, a term at a time in ascending order, so the same value; each element of x read serves every row, and the rows' chains of multiply-adds advance together, where a row alone waits on its last one through each term.
 
template<typename T >
void subtract_product (T *y, const T *x, const T &f, std::size_t count)
 y[i] -= x[i] · f, as the other subtract_product (a real product is the same either way round).
 
template<split_complex C>
void complex_scaled_add (C &y, const C &f, const C &x)
 y += f · x of one complex number, its part products rounded apart as libc++'s operator computes them (add_scaled_vectors's arithmetic, for the elements past its vectors).
 
template<split_complex C>
void add_product (C *y, const C &f, const C *x, std::size_t count)
 add_product of complex numbers: vectors of their parts, then complex_scaled_add, every element the same arithmetic.
 
template<typename T >
void add_one_product (T &y, const T &f, const T &x)
 y += f · x of one element, as add_product computes each of its elements.
 
template<split_complex C>
void add_one_product (C &y, const C &f, const C &x)
 add_one_product of complex numbers: complex_scaled_add, as add_product computes them.
 
template<typename T >
void add_product (T *y, const T &f, const T *x, std::size_t count)
 y[i] += f · x[i], as subtract_product.
 
template<typename R >
requires std::is_same_v<R, double> || std::is_same_v<R, float>
bool all_finite (const R *x, std::size_t count) noexcept
 Whether the count values at x are all finite: a NaN or an infinity has every exponent bit set, the one exponent that one more carries into the sign bit. One pass with no early exit and no comparison, an integer AND, add and OR (a check per element and an exit at the first was a tenth of an LU of order 128; the comparison per element, one M1 Pro core: 6.4 → 8.5 values a nanosecond). Under GCC and Clang four vectors of 16 bytes a step, whatever the compiler would vectorise: GCC at -O2 (the arm64 wheels') vectorised the plain loop not at all, at -O3 by one vector (1089 doubles, one arm64 core: GCC -O2 541 → 132 ns, -O3 299 → 132, Clang level at 132; one x86-64 core, GCC 13 -O3 271 → 163 ns, Clang 18 level); a Cholesky factorisation of order 33 lost a fifth of its time to the plain loop under GCC -O2.
 
template<typename T >
void one_sided_jacobi (buffer< T > &w, buffer< T > &v, std::size_t n, std::size_t max_sweeps)
 One-sided (Hestenes) Jacobi on the columns of the n × n row-major w: plane rotations, accumulated into v, until every pair of columns is orthogonal to rounding. The column norms are then the singular values of the input, each to high relative accuracy.
 
template<typename T >
void form_bidiagonal_bases (const std::vector< buffer< T > > &left, std::vector< buffer< T > > right, const std::vector< T > &left_tau, const std::vector< T > &right_tau, const buffer< T > &left_phase, const buffer< T > &right_phase, std::size_t n, buffer< T > &ut, buffer< T > &vt)
 The bases of a bidiagonal reduction from its reflectors: U_b = (H₀ H₁ …) diag(conj left phases) and V_b = (H'₀ H'₁ …) diag(right phases), each formed by blocks of reflectors through the matrix product kernel, as QR's Q is, or a reflector at a time below bidiagonal_basis_crossover.
 
template<typename T >
void bidiagonalize_unblocked (buffer< T > &b, std::size_t n, buffer< double > &d, buffer< double > &e, buffer< T > &ut, buffer< T > &vt, bool form)
 bidiagonalize_unblocked of the square n × n row-major b.
 
template<typename T >
requires std::is_floating_point_v<T> || split_complex<T>
void bidiagonalize (buffer< T > &b, std::size_t n, buffer< double > &d, buffer< double > &e, buffer< T > &ut, buffer< T > &vt, bool form=true)
 Reduces the square n × n row-major b to upper-bidiagonal form, B = U_b D V_bᴴ, by panels of bidiagonal_block reflections from each side, as LAPACK's dgebrd and dlabrd (zgebrd, zlabrd): the matrix a panel's later steps see is B − V Ȳᵀ − X Ūᵀ (V, U the reflectors, Y, X what they move, Ȳ and Ū held conjugated, so a complex matrix's update is a plain product), applied to the one column and the one row each step needs; the trailing block takes it after the panel, through the matrix product kernel. A real bidiagonal keeps its signs, which the iterations that follow take; a complex one is turned real as bidiagonalize_unblocked turns it, a unit phase on row k before its right reflection and on column k + 1 once that column comes to be reflected.
 
template<typename T >
void bidiagonalize_unblocked (buffer< T > &b, std::size_t rows, std::size_t n, buffer< double > &d, buffer< double > &e, buffer< T > &ut, buffer< T > &vt, bool form)
 Reduces the rows × n row-major b (rows ≥ n) to real upper-bidiagonal form, B = U_b D V_bᴴ with D real and n × n, by Householder reflections alternately from the left and the right (Golub–Kahan, LAPACK's dgebd2), the left ones rows − k long; the bases formed for a square b alone.
 
template<typename T >
requires (!std::is_floating_point_v<T> && !split_complex<T>)
void bidiagonalize (buffer< T > &b, std::size_t n, buffer< double > &d, buffer< double > &e, buffer< T > &ut, buffer< T > &vt, bool form=true)
 bidiagonalize_unblocked, for any T the panels do not take (a complex long double).
 
template<typename T >
void bidiagonal_qr (buffer< double > &d, buffer< double > &e, buffer< T > &ut, buffer< T > &vt, std::size_t n, std::size_t max_steps)
 The singular values of the real upper bidiagonal (d, e) by implicit-shift QR with deflation (the Golub–Kahan–Reinsch iteration, LINPACK's dsvdc), the rotations accumulated into the rows of ut and vt. Each block's bulge is chased from its larger end, as LAPACK's dbdsqr chases it: from the top only, a block graded up (its small entries first) never converged.
 
template<typename Entry >
double largest_magnitude (std::size_t rows, std::size_t cols, const Entry &entry)
 The power_of_two_scale of the largest magnitude of entry over rows rows of cols: by bands of rows on the threads, each band's largest then the largest of those (a maximum, the same whatever the order).
 
template<typename T >
matrix< T > adjoint (const matrix< T > &a)
 The conjugate transpose of a, as numerical::transpose copies (by blocks, on the threads).
 
double singular_secular_root (const buffer< double > &d, const buffer< double > &z, std::size_t i, std::size_t max_iterations, double *below, double *above)
 Singular value i of the arrow M whose first row is z and whose diagonal is d (d₀ = 0, d strictly ascending): root σ of 1 + Σⱼ zⱼ²/(dⱼ² − σ²) = 0, as LAPACK's dlasd4.
 
void rotate_columns (buffer< double > &m, std::size_t cols, std::size_t a, std::size_t b, double c, double s)
 Columns a and b of the row-major m (cols per row) rotated: a ← c a + s b, b ← c b − s a.
 
arrow_workspace & thread_arrow_workspace ()
 The calling thread's arrow_workspace, kept from one decomposition to the next. A join uses it whole on one thread and nests no other join; release_large_arrow_workspace lets a large buffer go after.
 
void release_large_arrow_workspace (arrow_workspace &work)
 Releases the buffers of work longer than kept_workspace.
 
void merge_arrow (buffer< double > &d, buffer< double > z, double alpha, double beta, buffer< double > &u, buffer< double > &v, std::size_t n, std::size_t m, std::size_t split, arrow_workspace &work)
 The singular value decomposition of the arrow M (first row z, diagonal d, d₀ = 0), carried into the bases u (n × n) and v (m × m, its first n columns the arrow's; a last, null column when m = n + 1), as LAPACK's dlasd1–dlasd3: on return d holds the singular values ascending and u, v the singular vectors, v's null column last.
 
std::size_t plan_bidiagonal (std::vector< bidiagonal_node > &nodes, const double *d, const double *e, std::size_t n, std::size_t extra, std::size_t depth)
 Appends the block (d, e, n, extra) and everything below it to nodes, parents before children, each scaled to work near unit size by a power of two (exactly): the secular equation works on squares, which overflow past 1e154.
 
void start_bidiagonal_node (bidiagonal_node &node)
 Its singular values and vectors into node's, from scratch: u zero, v the identity.
 
void solve_bidiagonal_leaf (bidiagonal_node &node)
 Decomposes the leaf node by implicit-shift QR.
 
void join_bidiagonal_node (bidiagonal_node &node, const bidiagonal_node &one, const bidiagonal_node &two, arrow_workspace &work)
 Joins the two decomposed halves of node, one and two, by merge_arrow, node started (start_bidiagonal_node).
 
void unscale_bidiagonal_node (bidiagonal_node &node)
 The singular values of a solved node scaled back.
 
void join_bidiagonal_halves (std::vector< bidiagonal_node > &nodes, std::size_t index, arrow_workspace &work)
 Joins the halves of nodes' block index in work, then lets them go.
 
void solve_bidiagonal_subtree (std::vector< bidiagonal_node > &nodes, std::size_t index, arrow_workspace &work)
 Decomposes the block index of nodes and everything below it, each block right after its halves.
 
void bidiagonal_divide (const double *d, const double *e, std::size_t n, std::size_t extra, buffer< double > &sigma, buffer< double > &u, buffer< double > &v)
 The singular value decomposition of the upper bidiagonal n × (n + extra) (extra 0 or 1) with diagonal d[0…n-1] and superdiagonal e[0…n+extra-2] (e[n-1] the last column's entry when extra is 1), by divide and conquer, as LAPACK's dbdsdc: sigma ascending, u (n × n) and v (m × m, m = n + extra, the null vector last) row-major, their columns the singular vectors.
 
template<typename T >
void bidiagonal_divide_and_conquer (buffer< double > &d, buffer< double > &e, buffer< T > &ut, buffer< T > &vt, std::size_t n)
 The singular values of the upper bidiagonal (d, e) into d, descending, the singular vectors accumulated into the rows of ut and vt as bidiagonal_qr accumulates them, by divide and conquer (bidiagonal_divide): most of the work in matrix products.
 
double singular_value_in_double (double x) noexcept
 A singular value of a decomposition carried in double: x itself.
 
double singular_value_in_double (long double x)
 A singular value of a long double matrix, computed in double at the matrix's scale and scaled back, as the double its type holds. Where long double's range is wider than double's (x86-64's 80 bits), one past double's normal numbers cannot be held, and the decomposition refuses the matrix rather than answer 0 or ∞.
 
std::size_t numerical_rank (const vector< double > &sigma, std::size_t rows, std::size_t cols, rank_cutoff cutoff={})
 How many of the singular values sigma of an rows × cols matrix exceed cutoff: its numerical rank (the first that many, where sigma is descending).
 
template<typename T >
T scaled_by_two (const T &x, int exponent)
 x times two to the exponent, exactly where the result is a normal number (no power of two is formed, so none overflows).
 
template<typename U >
std::complex< U > scaled_by_two (const std::complex< U > &x, int exponent)
 scaled_by_two for a complex number, part by part.
 
template<typename T >
buffer< T > upper_inverse (const buffer< T > &r, std::size_t n)
 R⁻¹, upper triangular, of the n × n upper-triangular r (row-major): back substitution against the identity in parallel chunks of columns, each row's terms in registers (subtract_products). A chunk's rows past its last column stay zero; a zero on the diagonal leaves infinities or NaN.
 
template<typename T >
buffer< double > square_singular_values (buffer< T > b, std::size_t n, qd_index sweeps=dqds_sweeps)
 
template<typename T >
std::optional< matrix< T > > lstsq_by_qr (const matrix< T > &a, const matrix< T > &b, rank_cutoff cutoff, buffer< double > &values, std::pair< buffer< T >, int > *const r=nullptr)
 The least-squares solution of A X ≈ B by a QR factorisation, of A, or of Aᴴ for a wide A (the minimum-norm solution), where it is the one the singular value decomposition gives: with A divided by a power of two near its largest magnitude, the bound 1 / ‖R⁻¹‖_F under its smallest singular value must stay qr_rank_margin times above cutoff of the bound ‖R‖_F over its largest, so that every singular value is far above the cutoff and the rank full (a zero on R's diagonal makes the first bound zero, or NaN, which fails). Else nothing. A fraction of the decomposition's work, the rest in matrix products.
 
template<typename T >
buffer< double > tall_singular_values (buffer< T > b, std::size_t rows, std::size_t n, qd_index sweeps=dqds_sweeps)
 The singular values of the square n × n row-major b, descending, without its vectors: those of its Golub–Kahan bidiagonal by dqds (dqds_singular_values), no basis formed; by the implicit-shift QR iteration (bidiagonal_qr) where dqds does not converge within sweeps steps per value.
 
template<typename T >
matrix< T > lstsq_from_svd (const svd_decomposition< T > &s, std::size_t rank, const matrix< T > &b)
 The minimum-norm least-squares solution of A X ≈ B from the singular value decomposition s of A (m × n), its first rank triplets: X = V Σ⁺ Uᴴ B.
 
double scaled_tridiagonal (const vector< double > &d, const vector< double > &e, buffer< double > &diagonal, buffer< double > &off)
 The diagonal d and off-diagonal e of a symmetric tridiagonal matrix divided by the power of two of their largest magnitude (returned), e padded with a 0 to d's length, as the QL iteration takes them.
 
std::size_t sturm_count (const double *d, const double *e2, std::size_t n, double x, double pivmin)
 The number of eigenvalues below x of the symmetric tridiagonal of diagonal d and squared off-diagonal e2 (n and n − 1 entries), by Sturm's sequence: the negative pivots of T − x I, a pivot smaller than pivmin taken as −pivmin, as LAPACK's dlaebz.
 
void factor_shifted_tridiagonal (const double *diagonal, const double *off, std::size_t n, double lambda, shifted_tridiagonal &f)
 Factorises T − λ I, n from 2, T of diagonal diagonal and off-diagonal off, with partial pivoting as LAPACK's dlagtf: at each step the row of the larger entry relative to its row's size (the first where they tie).
 
void solve_shifted_tridiagonal (const shifted_tridiagonal &f, std::size_t n, double *y, double &tol)
 Solves (T − λ I) x = y in place in y from factor_shifted_tridiagonal's factors, as LAPACK's dlagts (job −1): a pivot that would overflow the quotient, or is zero, moved away from zero by tol, then by twice as much, until it does not; tol, where 0, set first to ε times the factors' largest magnitude (ε itself if that is 0), for the next solves.
 
void start_vector (std::size_t index, double *x, std::size_t n) noexcept
 A start vector of n entries in [−1, 1) for tridiagonal_eigenvectors' eigenvalue index, its own draw (splitmix64 from the index), so that each vector is the same on any number of threads.
 
bool inverse_iteration (const buffer< double > &diagonal, const buffer< double > &off, std::size_t n, const buffer< double > &values, buffer< double > &z, std::size_t most_solves=5)
 The eigenvectors (rows of z, k × n) of the symmetric tridiagonal of diagonal diagonal and off-diagonal off (n from 2) for its eigenvalues values (k, ascending), as LAPACK's dstein: inverse iteration from a random vector, each solve of T − λ I scaled so that the iterate's largest entry reaches √(0.1 / n) once the eigenvector dominates, then two more; an eigenvalue within 10 ε |λ| of the one before moved up to that distance; the vectors of eigenvalues within 10⁻³ ‖T‖₁ of one another (a cluster) orthogonalised against the cluster's earlier ones at each step. The clusters on the threads.
 
template<typename T >
eigen_subset< T > hermitian_subset (const matrix< T > &a, const std::function< std::pair< std::ptrdiff_t, std::ptrdiff_t >(const vector< double > &, const vector< double > &, double)> &select, bool vectors, const std::string &name)
 hermitian_eigen_subset of the indices select gives from the tridiagonal form's diagonal and off-diagonal and the matrix's scale ([first, last], last < first for none), its vectors where vectors. select behind one type: one instance per T.
 
template<typename T >
matrix< T > hermitian_from_lower (const matrix< T > &a)
 The real symmetric or complex Hermitian matrix whose lower triangle is the square a's, and of its diagonal the real part.
 
template<typename T >
matrix< T > inverse_on_kept (const vector< double > &sigma, const matrix< T > &right, const matrix< T > &left, rank_cutoff cutoff)
 Σ_d right_d (1 / σ_d) left_dᴴ over the columns d of right (cols × k) and left (rows × k) whose sigma is above cutoff (of the largest magnitude in sigma): a pseudo-inverse from a singular value decomposition (right = V, left = U) or from an eigendecomposition (V both, σ the eigenvalues, signed). Through the matrix product kernel: summed term by term it ran five times as long as NumPy's at order 512.
 
template<typename T >
vector< double > hermitian_singular_values (const matrix< T > &a)
 The singular values of the real symmetric or complex Hermitian matrix whose lower triangle is a's: of the matrix made whole, for a scalar the eigendecompositions do not take (they compute in double).
 
template<typename T >
vector< double > eigenvalue_magnitudes (const matrix< T > &a)
 The magnitudes of the eigenvalues of the real symmetric or complex Hermitian a, its lower triangle read: its singular values, unordered.
 
vector< double > hermitian_singular_values (const matrix< double > &a)
 hermitian_singular_values in double: the magnitudes of the eigenvalues, cheaper.
 
vector< double > hermitian_singular_values (const matrix< std::complex< double > > &a)
 hermitian_singular_values in std::complex<double>: the magnitudes of the eigenvalues, cheaper.
 
template<typename T >
matrix< T > hermitian_pseudo_inverse (const matrix< T > &a, rank_cutoff cutoff)
 The pseudo-inverse of the real symmetric or complex Hermitian matrix whose lower triangle is a's: from the singular value decomposition of the matrix made whole, for a scalar the eigendecompositions do not take.
 
matrix< double > hermitian_pseudo_inverse (const matrix< double > &a, rank_cutoff cutoff)
 hermitian_pseudo_inverse in double: V Λ⁺ Vᵀ from the eigendecomposition, cheaper.
 
matrix< std::complex< double > > hermitian_pseudo_inverse (const matrix< std::complex< double > > &a, rank_cutoff cutoff)
 hermitian_pseudo_inverse in std::complex<double>: V Λ⁺ Vᴴ from the eigendecomposition, cheaper.
 
template<typename T >
buffer< T > permuted (const T *data, std::size_t count, const std::string &from, const std::string &to, const std::map< char, std::size_t > &sizes, std::size_t ld)
 The elements of data, whose axes carry the labels from (extents in sizes), laid out row-major along the labels to, a permutation of from that differs from it (so at least one label): a transpose of the last two labels by tiles, each slice of the leading ones; any other row by row of to's last label, a strided sweep. The threads share the work. data holds count elements, the product of the extents; a matrix (two labels) may have its rows ld elements apart (a NumPy array of every other row), 0 for contiguous ones.
 
std::string shape_text (const std::vector< std::size_t > &shape)
 A shape as NumPy prints it, (2, 3), for a message.
 
std::vector< std::size_t > broadcast_strides (const std::vector< std::size_t > &shape, std::size_t rank)
 The row-major strides of shape, with a stride of 0 on each axis of extent 1 (it broadcasts), aligned on the last axis of a result of rank rank (missing leading axes stride 0 too).
 
template<typename Body >
void run_fused (const Body &body)
 body(0), where std::fma is the instruction (with_fma_instruction) on an x86-64 processor with AVX2 and FMA: a body whose multiply-adds are all std::fma's, NUMERICAL_INLINE_LAMBDA, as what it calls must be.
 
std::vector< std::ptrdiff_t > row_major_strides (const std::vector< std::size_t > &shape)
 The row-major strides of shape, in elements.
 
bool lies_row_major (const std::vector< std::size_t > &shape, const std::vector< std::ptrdiff_t > &strides)
 Whether elements strides apart along the axes of shape lie row-major from the first (an axis of fewer than two elements stepping anything).
 
template<typename T , typename Op , typename R = std::invoke_result_t<const Op&, const T&, const T&>>
tensor< R > pairs_into (const T *a, const T *b, const std::vector< std::size_t > &shape, const Op &op)
 op(a, b) element by element of the row-major a and b, both of shape shape, in op's result type R, by chunks of elementwise_chunk among the threads.
 
template<typename T , typename Op , typename R = std::invoke_result_t<const Op&, const T&, const T&>>
tensor< R > broadcast_into (const T *a, const std::vector< std::size_t > &a_shape, const std::vector< std::ptrdiff_t > &a_strides, const T *b, const std::vector< std::size_t > &b_shape, const std::vector< std::ptrdiff_t > &b_strides, const Op &op)
 op(a, b) element by element of a, of shape a_shape, its elements a_strides apart along each axis from a (negative along an axis it reverses), and of b likewise, broadcast against each other (broadcast_apply's, its operands read where they lie), in op's result type R, row-major. Each element alone: the bits of the same operands laid row-major.
 
template<typename T , typename Op , typename R = std::invoke_result_t<const Op&, const T&, const T&>>
tensor< R > broadcast_into (const T *a, const std::vector< std::size_t > &a_shape, const T *b, const std::vector< std::size_t > &b_shape, const Op &op)
 op(a, b) element by element of the row-major a (a_shape) and b (b_shape), broadcast against each other, in op's result type R.
 
template<typename T >
T pairwise_quad (const T *p, std::size_t width)
 Four partial sums at p, width apart, added pairwise: (p₀ + p₁) + (p₂ + p₃).
 
template<typename T >
T pairwise_sum (const T *first, std::size_t count, std::size_t stride)
 Sum of count values from first at stride stride, by NumPy's pairwise summation (pairwise_tree).
 
template<typename T >
void pairwise_leaves (std::size_t count, std::vector< std::size_t > &out)
 The lengths of pairwise_sum's subtrees of at most pairwise_leaf values of type T over count, in order.
 
template<typename T >
T pairwise_combine (std::size_t count, const T *&next)
 The subtrees' sums from next on (pairwise_leaves' order) added as pairwise_sum's tree over count values adds its halves.
 
template<typename T >
T parallel_pairwise_sum (const T *first, std::size_t count)
 pairwise_sum of count contiguous values on the threads, to the bit: the tree's subtrees of at most pairwise_leaf values summed in parallel, each as pairwise_sum sums it, then added in the tree's order. The subtrees follow from the count alone. Four subtrees or fewer: on the caller (2¹⁶ doubles, ten AVX-512 cores: 11.7 µs alone, 14.5 shared; 2²⁰ 247 → 44 µs, 2²⁴ 12.5 → 2.2 ms).
 
template<typename T >
void pairwise_sum_rows (const T *first, std::size_t count, std::size_t stride, std::size_t width, T *out)
 pairwise_sum of width neighbouring columns at once, into out: column q's sum that of pairwise_sum(first + q, count, stride), to the bit, each step adding a whole row of the columns (contiguous, stride apart) where a column was walked down stride at a time.
 
template<typename T >
void pairwise_sums (const T *data, std::size_t outer, std::size_t count, std::size_t inner, T *out)
 The pairwise sums of outer blocks of count rows of inner elements (row-major), each column of a block summed down its rows, into out (outer × inner): one column at a time where inner is 1, else by rows (pairwise_sum_rows), the same bits. The blocks, and their runs of pairwise_columns columns, shared among the threads: each sum computed alike by whichever thread.
 
bool is_label (char ch)
 Is ch an ASCII letter (a valid index label)?
 
std::vector< std::string > split_inputs (const std::string &lhs)
 Splits an einsum left-hand side into per-operand subscripts.
 
template<typename T >
tensor< T > tile_gemm (const T *a, std::size_t lda, const T *b, std::size_t ldb, std::size_t rows, std::size_t inner, std::size_t cols, std::vector< std::size_t > out_shape)
 Matrix product C(M×N) = A(M×K) · B(K×N), row-major.
 
template<typename T >
void add_column_term (std::true_type, T &sum, const T &f, const T &x)
 sum += f · x of strided_column_group, one fused multiply-add.
 
template<typename T >
void add_column_term (std::false_type, T &sum, const T &f, const T &x)
 sum += f · x of strided_column_group, as gemm_rows' plain loop rounds a term where it fuses none (rounded_apart_add, a complex product's infinities recovered as the library's).
 
template<std::size_t Width, bool Fused = false, typename T >
void strided_column_group (const T *a, const T *b, std::size_t row_step, std::size_t column_step, std::size_t inner, std::size_t j, T *y)
 Columns j to j + Width of vector_times_columns into y, their sums held through the rows: each term one fused multiply-add where Fused (as add_product adds each row on the arm64 fused paths), else add_one_product.
 
template<typename T , bool Fused = false>
buffer< T > vector_times_columns (const T *a, const T *b, std::size_t row_step, std::size_t column_step, std::size_t inner, std::size_t cols)
 canonical_product's vector a (inner) times a matrix b whose element B[k][j] is b[(k · row_step) + (j · column_step)]: each element of the product the sum canonical_product forms, a[k] · B[k][j] added in ascending order of k with add_product's arithmetic (add_one_product), so the same bits; strided_columns columns' sums held at once, where add_product stores and reloads them for every row of B.
 
buffer< float > vector_times_columns (const float *a, const float *b, std::size_t row_step, std::size_t column_step, std::size_t inner, std::size_t cols)
 vector_times_columns for float: each term as gemm_rows rounds it, fused on the x86-64 AVX paths (avx2_strided_columns) and on arm64's NEON and AMX paths, elsewhere the template's arithmetic.
 
buffer< double > vector_times_columns (const double *a, const double *b, std::size_t row_step, std::size_t column_step, std::size_t inner, std::size_t cols)
 vector_times_columns for double: on the x86-64 fused paths, where add_product adds each row by fused multiply-adds, by avx2_strided_columns; on the arm64 NEON and AMX paths, where it does too, by fused terms; elsewhere the template's arithmetic.
 
template<split_complex C>
buffer< C > complex_vector_times_columns (const C *a, const C *b, std::size_t row_step, std::size_t column_step, std::size_t inner, std::size_t cols)
 The complex vector a (inner) times a matrix b whose element B[k][j] is b[(k · row_step) + (j · column_step)], each element of the product rounded as the path's complex product rounds it (complex_terms, past complex_term::compiler): so the bits of its row in any product of a matrix by B. On the x86-64 complex tiles by their panels (avx2_complex_row_panels), on arm64 by gemm_split's blocks (split_row_columns).
 
template<typename T >
buffer< T > vector_times_transposed (const T *a, const T *bt, std::size_t ldbt, std::size_t inner, std::size_t cols)
 vector_times_columns of a matrix given as its transpose bt (cols × inner, its rows ldbt apart, B's columns); a complex one past complex_term::compiler by complex_vector_times_columns.
 
template<typename T >
void add_row_terms (T *y, const T *a, const T *b, std::size_t ldb, std::size_t inner, std::size_t count)
 y[j] += Σₖ a[k] · b[(k · ldb) + j] for count columns, B's inner rows added in order to a vector times a matrix by add_product, an integer's exact.
 
template<split_complex C>
bool parts_finite (const C *x, std::size_t count) noexcept
 Whether every part of count complex numbers from x is finite.
 
template<split_complex C>
void add_row_terms (C *y, const C *a, const C *b, std::size_t ldb, std::size_t inner, std::size_t count)
 add_row_terms of complex numbers as gemm_rows' plain loop rounds them (rounded_apart_add): a row by add_product's vectors where its factor and itself are finite, whose products are then the library's (both of a product's parts are NaN only from an infinite or NaN operand); else term by term, the library recovering a product of infinities.
 
template<typename T >
tensor< T > canonical_product (const T *a, std::size_t lda, const T *b, std::size_t ldb, std::size_t rows, std::size_t inner, std::size_t cols, std::vector< std::size_t > out_shape)
 The product of the row-major rows × inner a and inner × cols b, shaped out_shape: the canonical two-operand contraction of einsum_impl (a matrix times a vector, a vector times a matrix, a matrix product, a dot product). A matrix's rows lie lda (a) and ldb (b) apart, each row's elements together; a vector's elements together (b of one column: ldb then 1).
 
template<typename T >
std::optional< tensor< T > > diagonal_labels (const std::string &in, const std::string &output, std::size_t distinct, const std::vector< std::size_t > &shape, const T *data)
 einsum of one row-major operand data (shape, labelled in, distinct labels in all) whose last two labels are one (…ii), the others each once and output those in order, with that label (…ii->…i, each matrix's diagonal) or without (…ii->…, its sum by pairwise halving, as pairwise_sums adds a run: the same bits as trace's sum of the diagonal). None for any other spec.
 
template<typename T >
std::optional< tensor< T > > sum_over_run (const std::string &in, const std::string &output, const std::vector< std::size_t > &shape, const T *data)
 einsum of one row-major operand data (shape, its axes labelled in, each label once) into the labels output, where those it sums are one run of its axes and output the others in order (ij->i, ij->, ij->j, ijk->ik): each output element the sum of evenly spaced elements, by pairwise halving as sum() adds them (pairwise_sums), where einsum's general loop added them one after another. None for any other spec (nothing summed, the summed labels apart, the output reordered).
 
template<typename T >
buffer< T > permuted (const buffer< T > &data, const std::string &from, const std::string &to, const std::map< char, std::size_t > &sizes)
 permuted of a buffer's elements.
 
template<std::floating_point T>
T fused_from_zero (T p, T q) noexcept
 std::fma(p, q, +0) without the instruction (in code built for any x86-64, std::fma is a call to the library, 40 000 elements of an outer product 18 → 114 µs on four AVX2 cores): the rounded product, which keeps the sign of a product that underflows to zero, plus +0 where an operand is zero, as the fused sum of an exact zero product and +0 is.
 
template<std::floating_point R>
std::complex< R > tile_product (const std::complex< R > &p, const std::complex< R > &q) noexcept
 0 + p·q of complex numbers as the x86-64 complex tiles round it (complex_term::tiles): each part's product from zero, their difference and their sum, each added to 0.
 
template<std::floating_point R>
std::complex< R > split_product (const std::complex< R > &p, const std::complex< R > &q) noexcept
 0 + p·q of complex numbers as arm64's real kernel rounds it (complex_term::split), a product of [Re p | −Im p] by [[Re q, Im q], [Im q, −Re q]]: each part's first product from zero, the second fused into it.
 
template<typename T >
requires std::is_arithmetic_v<T>
tensor< T > each_rounded (const T *x, const std::vector< std::size_t > &x_shape, const T *y, const std::vector< std::size_t > &y_shape)
 The products that sum nothing of each_product, x and y broadcast against each other (broadcast_into), for a real or integer T: a fused first term where gemm_rows fuses one (rows_terms_fuse), else as the compiler rounds 0 + x·y.
 
template<split_complex T>
tensor< T > each_rounded (const T *x, const std::vector< std::size_t > &x_shape, const T *y, const std::vector< std::size_t > &y_shape)
 each_rounded for complex numbers: as the path's complex product rounds a first term (complex_terms).
 
template<typename T >
tensor< T > each_product (const std::string &sa, operand_span< T > a, const std::string &sb, operand_span< T > b, const std::string &output, const std::map< char, std::size_t > &sizes)
 A batch of two-operand products that sum nothing (ij,j->ij, ijk,ik->ijk), each element of the output its operands' product added to 0 as gemm_rows rounds a product's first term (rows_terms_fuse), a complex one as the path's complex product rounds it (complex_terms): the bits of contract_pair's products of depth one. Each operand's axes in the output's order (a copy only where they lie otherwise), broadcast against the other (broadcast_into), where those products went one matrix product of depth one per element of the batch (1000 × 1000 times a row of 1000, six AVX2 cores: 19.7 → 0.5 ms).
 
template<typename T >
tensor< T > contract_pair (const std::string &sa, operand_span< T > a, const std::string &sb, operand_span< T > b, const std::string &output, const std::string &summed, const std::map< char, std::size_t > &sizes)
 A two-operand contraction as matrix products: the labels both operands and the output carry (batch), those one operand and the output carry (free), and those both operands carry alone (summed, in ascending order, the order the general loop sums them in, so each element takes the same terms in the same order) regroup each operand as a stack of matrices, A as batch × free_a × summed and B as batch × summed × free_b, multiplied batch by batch through the packed kernel, batches in parallel; the result is laid out in the output's order; a dot product per batch by dot_product. Each label of an operand once, and each in the other operand or the output: einsum_impl sums an operand's own labels and reads a repeated one's diagonal first.
 
std::string pair_keeps (const std::vector< std::string > &labels, std::size_t i, std::size_t j, const std::string &output)
 The labels of labels[i] and labels[j] that the output or another operand carries, in their order of appearance, once each: what their contraction keeps.
 
double extent_product (const std::string &labels, const std::map< char, std::size_t > &sizes)
 The product of the extents of labels' distinct labels, as a double (a cost: it may pass size_t).
 
std::vector< std::string > after_step (const std::vector< std::string > &labels, contraction_step step, const std::string &output)
 The list after a contraction step: the two operands' labels removed, their result's appended.
 
void cheapest_path (const std::vector< std::string > &labels, const std::string &output, const std::map< char, std::size_t > &sizes, std::vector< contraction_step > &path, double &cost, double &largest)
 The cheapest order of contraction from labels, and its cost and largest intermediate: every order tried, the first found kept among equals.
 
std::vector< contraction_step > contraction_path (std::vector< std::string > labels, const std::string &output, const std::map< char, std::size_t > &sizes)
 The order in which to contract operands labelled labels into output, as opt_einsum chooses it: the cheapest in multiply-adds (the product of the extents a pair's labels span), the smaller largest intermediate among equals, over every order up to exhaustive_path_operands operands; past it, the cheapest pair at each step among those sharing a label (an outer product, cheap in itself, would leave its labels to every later step), the smaller result, then the first pair, among equals. It depends on the shapes alone, so a result is the same on any number of threads.
 
std::size_t ellipsis_in (const std::string &subscripts)
 Evaluates a parsed einsum over its operands.
 
template<typename T >
tensor< T > einsum_pair (const std::string &s0, operand_span< T > a, const std::string &s1, operand_span< T > b, const std::string &output, const std::string &summed, const std::map< char, std::size_t > &sizes)
 einsum of two operands a (labels s0) and b (s1) into output, summed the labels neither the output carries (ascending), sizes each label's extent: each label of an operand once, and each in the other operand or the output (einsum_impl sums an operand's own labels and reads a repeated one's diagonal first). A tensor's elements or an array's read where it lies, the same bits either way.
 
template<typename T >
tensor< T > einsum_impl (const std::string &spec_in, const std::vector< const tensor< T > * > &operands)
 Einstein summation of operands as spec_in reads it (einsum): einsum_spec.
 
template<typename T >
tensor< T > einsum_ellipsis (const std::string &spec, const std::vector< const tensor< T > * > &operands)
 einsum_impl of a spec with ellipses, as NumPy reads them: each ... stands for an operand's axes its letters leave, aligned from the right across the operands and broadcast (an extent of 1 repeated), named by letters the spec does not use. An operand's axis of extent 1 under the ellipsis where another's is longer is dropped with its letter, so that operand repeats along it. The output: its ... the broadcast axes (an output without one, where an operand has such axes, is refused, as NumPy refuses it); with no ->, the broadcast axes then the letters that appear once, in order.
 
template<typename To , typename From >
tensor< To > converted (const tensor< From > &t)
 A copy of t, each element converted to To (between a signed integer type and its unsigned one, the same bits, modulo 2ⁿ), by chunks of elementwise_chunk among the threads: one thread's copy and its pages' first touches were most of an int64 dot (10⁶ elements, M1 Pro: 0.49 of 0.61 ms).
 
template<typename T >
tensor< T > einsum_spec (const std::string &spec_in, const std::vector< const tensor< T > * > &operands)
 einsum_impl of a type whose sums and products are defined past its range: a float, a complex, an unsigned integer.
 
template<typename T >
tensor< T > stacked_matmul (const tensor< T > &a, const tensor< T > &b)
 numerical::matmul of a type whose sums and products are defined past its range: a float, a complex, an unsigned integer.
 
axis_walk walk_along (const std::vector< std::size_t > &shape, std::size_t axis)
 axis_walk of shape along axis; outer and inner 0 where the result is empty.
 
template<std::floating_point R>
R numpy_muladd (R x, R y, R z, bool fused)
 x · y + z: one fused multiply-add where fused, else the product rounded first (statements apart, so no compiler contracts them).
 
template<split_complex C>
C numpy_multiply (const C &a, const C &b, bool fused)
 a · b of complex numbers as NumPy's array loop computes it: ai·bi and ai·br rounded, then ar·br − ai·bi and ar·bi + ai·br, each by a fused multiply-add where fused (NumPy's vectors on arm64 and on x86-64 with FMA), else each product rounded first. No recovery of an infinity from a NaN, as std::complex's (C Annex G) makes: 1j · (inf + inf·j) is NaN, (1e308 + 1e308j) · (1e308 − 1e308j) inf + inf·j when fused.
 
template<split_complex C>
C numpy_multiply_late (const C &a, const C &b)
 a · b of complex numbers as NumPy's reduction loop computes it where MSVC contracts it (its Windows wheels from 2.5): ai·bi and ar·bi rounded, then ar·br − ai·bi and ai·br + ar·bi each one fused multiply-add (the imaginary part's other product fused than in numpy_multiply).
 
template<std::floating_point R>
R divided_by_zero (R x) noexcept
 x divided by +0, as IEEE divides it, spelled out: a NaN for a zero or a NaN, else an infinity of x's sign (a division by a zero the compiler sees is MSVC's C4723 under /WX).
 
template<split_complex C>
C numpy_divide (const C &a, const C &b, bool fused)
 a / b of complex numbers as NumPy's array loop computes it: Smith's division by the larger part of b (an exact zero divided part by part, to an infinity or a NaN), each x + y·z of it one fused multiply-add where fused (NumPy's build contracts them on arm64, not on x86-64).
 
template<split_complex C>
C numpy_reciprocal (const C &x, bool fused)
 NumPy's complex reciprocal 1 / x (its reciprocal, the loop x ** -1 takes): the ratio of the smaller part to the larger, d the larger plus the smaller times it (one fused multiply-add where fused), then 1 / d and −ratio / d — not numpy_divide's quotient of 1.
 
template<split_complex C>
C library_power (const C &a, const C &b)
 a ** b by std::pow: libstdc++'s calls the C library's cpow, which NumPy calls (the same bits under glibc); MSVC's is its own.
 
template<split_complex C>
C numpy_power (const C &a, const C &b, bool fused)
 NumPy's complex power a ** b (npymath's npy_cpow): 1 for a zero exponent, 0 or NaN for a zero base; an integer exponent below 100 in magnitude by products (numpy_multiply, from 1 past the cubes) and, when negative, the quotient of 1 by them (numpy_divide), each fused where fused; any other by the C library's cpow, as NumPy (library_power).
 
template<typename T >
T root (const T &x)
 The square root of a scalar, as std::sqrt gives it.
 
template<typename Into >
tensor< float > through_doubles (const float *data, const std::vector< std::size_t > &shape, Into into)
 The function into (one of the *_into of doubles) of every one of the row-major floats at data, of shape shape: each piece of 512 converted to doubles on the stack, into of them, each rounded back once. The double within 0.51 of its ulp, the float is within half of its own and 2⁻²⁸ more, where the C library's float functions are within one or two; and the doubles' vectors are faster than its scalar calls (glibc's sinhf 4.4 times the doubles' sinh, atanf 3.5, log1pf 2.9).
 
template<typename T >
T elementary_exp (const T &x)
 exp of a scalar, as std::exp gives it.
 
double elementary_exp (double x)
 exp of a double, as exp_into gives it: table_exp, or std::exp below exp_normal_bound (and throughout where exp_by_table says no).
 
void exp_into (const double *in, double *out, std::size_t count)
 exp of the count doubles at in into out, each elementary_exp's: table_exp in vectors (with the FMA instructions on an x86-64 processor, run_fused), then std::exp where the result is subnormal or zero, below exp_normal_bound, which the table's two scalings would round twice; std::exp throughout where exp_by_table says no.
 
template<typename T >
T elementary_expm1 (const T &x)
 expm1 of a scalar, as std::expm1 gives it.
 
double elementary_expm1 (double x)
 expm1 of a double, as expm1_into gives it: table_expm1, or std::expm1 where exp_by_table says no.
 
void expm1_into (const double *in, double *out, std::size_t count)
 expm1 of the count doubles at in into out, each elementary_expm1's: table_expm1 in vectors (with the FMA instructions on an x86-64 processor, run_fused), std::expm1 throughout where exp_by_table says no.
 
template<typename T >
tensor< T > expm1_of (const T *data, const std::vector< std::size_t > &shape)
 expm1 of every one of the row-major scalars at data, of shape shape: std::expm1 of each.
 
tensor< double > expm1_of (const double *data, const std::vector< std::size_t > &shape)
 expm1 of every one of the row-major doubles at data, of shape shape: expm1_into by chunks of elementwise_chunk among the threads.
 
float elementary_expm1 (float x)
 expm1 of a float, as expm1_of gives it: elementary_expm1 of the double, rounded once.
 
tensor< float > expm1_of (const float *data, const std::vector< std::size_t > &shape)
 expm1 of every one of the row-major floats at data, of shape shape: expm1_into of them as doubles (through_doubles).
 
template<typename T >
T elementary_atan (const T &x)
 atan of a scalar, as std::atan gives it.
 
double elementary_atan (double x)
 atan of a double, as atan_into gives it: table_atan, or std::atan where exp_by_table says no.
 
void atan_into (const double *in, double *out, std::size_t count)
 atan of the count doubles at in into out, each elementary_atan's: table_atan in vectors (with the FMA instructions on an x86-64 processor, run_fused), std::atan throughout where exp_by_table says no.
 
template<typename T >
tensor< T > atan_of (const T *data, const std::vector< std::size_t > &shape)
 atan of every one of the row-major scalars at data, of shape shape: std::atan of each.
 
tensor< double > atan_of (const double *data, const std::vector< std::size_t > &shape)
 atan of every one of the row-major doubles at data, of shape shape: atan_into by chunks of elementwise_chunk among the threads.
 
float elementary_atan (float x)
 atan of a float, as atan_of gives it: elementary_atan of the double, rounded once.
 
tensor< float > atan_of (const float *data, const std::vector< std::size_t > &shape)
 atan of every one of the row-major floats at data, of shape shape: atan_into of them as doubles (through_doubles).
 
template<typename T >
T elementary_sinh (const T &x)
 sinh of a scalar, as std::sinh gives it.
 
double elementary_sinh (double x)
 sinh of a double, as sinh_into gives it: table_sinh, or std::sinh where exp_by_table says no.
 
void sinh_into (const double *in, double *out, std::size_t count)
 sinh of the count doubles at in into out, each elementary_sinh's: table_sinh in vectors (with the FMA instructions on an x86-64 processor, run_fused), std::sinh throughout where exp_by_table says no.
 
template<typename T >
tensor< T > sinh_of (const T *data, const std::vector< std::size_t > &shape)
 sinh of every one of the row-major scalars at data, of shape shape: std::sinh of each.
 
tensor< double > sinh_of (const double *data, const std::vector< std::size_t > &shape)
 sinh of every one of the row-major doubles at data, of shape shape: sinh_into by chunks of elementwise_chunk among the threads.
 
float elementary_sinh (float x)
 sinh of a float, as sinh_of gives it: elementary_sinh of the double, rounded once.
 
tensor< float > sinh_of (const float *data, const std::vector< std::size_t > &shape)
 sinh of every one of the row-major floats at data, of shape shape: sinh_into of them as doubles (through_doubles).
 
template<typename T >
T elementary_cosh (const T &x)
 cosh of a scalar, as std::cosh gives it.
 
double elementary_cosh (double x)
 cosh of a double, as cosh_into gives it: table_cosh, or std::cosh where exp_by_table says no.
 
void cosh_into (const double *in, double *out, std::size_t count)
 cosh of the count doubles at in into out, each elementary_cosh's: table_cosh in vectors (with the FMA instructions on an x86-64 processor, run_fused), std::cosh throughout where exp_by_table says no.
 
template<typename T >
tensor< T > cosh_of (const T *data, const std::vector< std::size_t > &shape)
 cosh of every one of the row-major scalars at data, of shape shape: std::cosh of each.
 
tensor< double > cosh_of (const double *data, const std::vector< std::size_t > &shape)
 cosh of every one of the row-major doubles at data, of shape shape: cosh_into by chunks of elementwise_chunk among the threads.
 
float elementary_cosh (float x)
 cosh of a float, as cosh_of gives it: elementary_cosh of the double, rounded once.
 
tensor< float > cosh_of (const float *data, const std::vector< std::size_t > &shape)
 cosh of every one of the row-major floats at data, of shape shape: cosh_into of them as doubles (through_doubles).
 
template<typename T >
T elementary_exp2 (const T &x)
 exp2 of a scalar, as std::exp2 gives it.
 
double elementary_exp2 (double x)
 exp2 of a double, as exp2_into gives it: table_exp2, or std::exp2 below exp2_normal_bound (and throughout where exp_by_table says no).
 
void exp2_into (const double *in, double *out, std::size_t count)
 exp2 of the count doubles at in into out, each elementary_exp2's: table_exp2 in vectors (with the FMA instructions on an x86-64 processor, run_fused), then std::exp2 below exp2_normal_bound, as exp_into; std::exp2 throughout where exp_by_table says no.
 
template<typename T >
tensor< T > exp2_of (const T *data, const std::vector< std::size_t > &shape)
 exp2 of every one of the row-major scalars at data, of shape shape: std::exp2 of each.
 
tensor< double > exp2_of (const double *data, const std::vector< std::size_t > &shape)
 exp2 of every one of the row-major doubles at data, of shape shape: exp2_into by chunks of elementwise_chunk among the threads.
 
template<typename T >
T elementary_log (const T &x)
 log of a scalar, as std::log gives it.
 
template<typename T >
T elementary_log1p (const T &x)
 log1p of a scalar, as std::log1p gives it.
 
double elementary_log1p (double x)
 log1p of a double, as log1p_into gives it: table_log1p, or std::log1p where exp_by_table says no.
 
void log1p_into (const double *in, double *out, std::size_t count)
 log1p of the count doubles at in into out, each elementary_log1p's: table_log1p in vectors (with the FMA instructions on an x86-64 processor, run_fused), std::log1p throughout where exp_by_table says no.
 
template<typename T >
tensor< T > log1p_of (const T *data, const std::vector< std::size_t > &shape)
 log1p of every one of the row-major scalars at data, of shape shape: std::log1p of each.
 
tensor< double > log1p_of (const double *data, const std::vector< std::size_t > &shape)
 log1p of every one of the row-major doubles at data, of shape shape: log1p_into by chunks of elementwise_chunk among the threads.
 
float elementary_log1p (float x)
 log1p of a float, as log1p_of gives it: elementary_log1p of the double, rounded once.
 
tensor< float > log1p_of (const float *data, const std::vector< std::size_t > &shape)
 log1p of every one of the row-major floats at data, of shape shape: log1p_into of them as doubles (through_doubles).
 
template<typename T >
T elementary_sin (const T &x)
 sin of a scalar, as std::sin gives it.
 
template<typename T >
T elementary_cos (const T &x)
 cos of a scalar, as std::cos gives it.
 
template<bool Cosine>
double float_sine (double x)
 sin (Cosine false) or cos of a double that a float holds: reduced_sine to sine_reduction_bound, the C library's past it (and for an infinity or a NaN).
 
template<bool Cosine>
void float_sines_into (const double *in, double *out, std::size_t count)
 sin (Cosine false) or cos of the count doubles at in, each a float's, into out, each float_sine's: reduced_sine in vectors (with the FMA instructions on an x86-64 processor, run_fused), then the C library's past sine_reduction_bound, behind a branch; the C library's throughout where exp_by_table says no.
 
template<bool Cosine>
double double_sine (double x)
 sin (Cosine false) or cos of a double, as double_sines_into gives it: table_sine to sine_reduction_bound, the C library's past it (and for an infinity or a NaN) and where exp_by_table says no.
 
template<bool Cosine>
void double_sines_into (const double *in, double *out, std::size_t count)
 sin (Cosine false) or cos of the count doubles at in into out, each double_sine's: table_sine to sine_reduction_bound (with the FMA instructions on an x86-64 processor, run_fused), the C library's past it, element by element (the table's then the library's over every element past it took 1.4 times the library's alone: 10⁶ values to ±10⁷, 12 → 17 ms); the C library's throughout where exp_by_table says no.
 
double elementary_sin (double x)
 sin of a double, as double_sines_into gives it (double_sine).
 
double elementary_cos (double x)
 cos of a double, as double_sines_into gives it (double_sine).
 
float elementary_sin (float x)
 sin of a float, as float_sines_into gives it: float_sine of the double, rounded once.
 
float elementary_cos (float x)
 cos of a float, as float_sines_into gives it: float_sine of the double, rounded once.
 
template<typename T >
tensor< T > sin_of (const T *data, const std::vector< std::size_t > &shape)
 sin of every one of the row-major scalars at data, of shape shape: std::sin of each.
 
tensor< float > sin_of (const float *data, const std::vector< std::size_t > &shape)
 sin of every one of the row-major floats at data, of shape shape: float_sines_into of them as doubles (through_doubles).
 
tensor< double > sin_of (const double *data, const std::vector< std::size_t > &shape)
 sin of every one of the row-major doubles at data, of shape shape: double_sines_into by chunks of elementwise_chunk on the threads.
 
template<typename T >
tensor< T > cos_of (const T *data, const std::vector< std::size_t > &shape)
 cos of every one of the row-major scalars at data, of shape shape: std::cos of each.
 
tensor< double > cos_of (const double *data, const std::vector< std::size_t > &shape)
 cos of every one of the row-major doubles at data, of shape shape: double_sines_into by chunks of elementwise_chunk on the threads.
 
tensor< float > cos_of (const float *data, const std::vector< std::size_t > &shape)
 cos of every one of the row-major floats at data, of shape shape: float_sines_into of them as doubles (through_doubles).
 
template<typename T >
T elementary_tan (const T &x)
 tan of a scalar, as std::tan gives it.
 
template<typename T >
T elementary_tanh (const T &x)
 tanh of a scalar, as std::tanh gives it.
 
double elementary_tanh (double x)
 tanh of a double, as tanh_into gives it: table_tanh, or std::tanh where exp_by_table says no.
 
void tanh_into (const double *in, double *out, std::size_t count)
 tanh of the count doubles at in into out, each elementary_tanh's: table_tanh in vectors (with the FMA instructions on an x86-64 processor, run_fused), std::tanh throughout where exp_by_table says no.
 
template<typename T >
tensor< T > tanh_of (const T *data, const std::vector< std::size_t > &shape)
 tanh of every one of the row-major scalars at data, of shape shape: elementary_tanh of each.
 
tensor< double > tanh_of (const double *data, const std::vector< std::size_t > &shape)
 tanh of every one of the row-major doubles at data, of shape shape: tanh_into by chunks of elementwise_chunk among the threads.
 
float elementary_tanh (float x)
 tanh of a float, as tanh_of gives it: elementary_tanh of the double, rounded once.
 
tensor< float > tanh_of (const float *data, const std::vector< std::size_t > &shape)
 tanh of every one of the row-major floats at data, of shape shape: tanh_into of them as doubles (through_doubles).
 
template<split_complex C>
C::value_type numpy_absolute (const C &x, bool fused)
 NumPy's complex magnitude (its absolute's vector loop): the parts' magnitudes, an infinite one making both infinite and then a NaN both NaN, then the larger times sqrt(ratio² + 1) for the ratio of the smaller to it (0 where the larger is 0 or the smaller infinite), ratio² + 1 one fused multiply-add where fused.
 
bool numpy_fuses_products () noexcept
 Whether NumPy's complex product fuses its multiply-adds on this processor: on arm64, and on x86-64 where its AVX2 loop runs (FMA with it); elsewhere its loop rounds each product.
 
bool numpy_fuses_scalar_products () noexcept
 Whether NumPy's complex product fuses its multiply-adds in its scalar loop (where numpy_loop_is_scalar holds), as its PyPI wheels are built: that loop sits beside the vector one, and clang (macOS) and MSVC (Windows) contract it where the vector loop has FMA; GCC (Linux, ISO C) does not.
 
bool numpy_fuses_reductions () noexcept
 Whether NumPy's complex product fuses its multiply-adds in its reduction loop (the product's loop whose output is its own first operand: numpy.prod along an axis nothing follows, numpy.cumprod past two elements), as its PyPI wheels are built: a scalar loop, contracted as numpy_fuses_scalar_products says (GCC's on Linux rounds each product, though its vector loop fuses).
 
bool numpy_fuses_quotients () noexcept
 Whether NumPy's complex quotient fuses its multiply-adds, as its PyPI wheels are built: on macOS arm64, whose clang contracts them (FMA is arm64's baseline); not on Linux, whose GCC does not, nor on x86-64, whose baseline has no FMA.
 
bool numpy_loop_is_scalar (const std::vector< std::size_t > &a_shape, const std::vector< std::size_t > &b_shape) noexcept
 Whether NumPy multiplies an operand of shape a_shape by one of b_shape in its scalar loop rather than its vector one: when the result is one element and an operand of rank above 0 is broadcast to a higher rank, NumPy's iterator hands its loop an output step of 0, which the vector loop refuses.
 
template<typename T >
bool numpy_loop_is_scalar (const tensor< T > &a, const tensor< T > &b) noexcept
 numpy_loop_is_scalar of the shapes of a and b.
 
template<std::floating_point T>
T divided_by_count (T value, std::size_t n)
 A mean: value divided by the count n, in the wider of T and double and back to T, the count exact (to 2^53), as NumPy divides by its integer count; a float's count rounded to a float past 2^24 (2^24 + 1 ones averaged to 1, NumPy's 0.99999994).
 
template<std::floating_point T, typename Order >
T settled_zero (T value, const T *first, std::size_t count, std::size_t stride, const Order &larger)
 The extreme value of count elements stride apart from first, its zero's sign settled as IEEE 754-2019's maximum and minimum settle it, and NumPy's on arm64: no zero is larger than the other, so the first met stayed; max gives +0 where one is among them, min −0, whatever their order. A NaN, or any other value, as it is.
 
std::vector< std::size_t > without_axis (std::vector< std::size_t > shape, std::size_t axis)
 The shape with axis axis removed.
 
template<typename T , typename Reduce >
tensor< T > reduce_along (const tensor< T > &t, std::size_t axis, const Reduce &reduce)
 Reduces each line of t along axis by reduce, which takes the line's first element, its length and its stride.
 
template<typename R , typename T , typename Step >
tensor< R > fold_along (const tensor< T > &t, std::size_t axis, const R &init, const Step &step)
 Folds each line of t along axis in order, step(running, x) from init: a tensor of the result type R, the axis taken out. Where the lines are columns (something follows the axis), each task takes a band of up to band_columns of them and walks it row by row, contiguously; each line's steps are the same, in the same order, either way.
 
template<typename T >
T square (const T &x)
 x · x, by the type's own product (a complex one's C Annex G, as the library's *).
 
template<typename T >
bool is_nan (const T &)
 A NaN? Never for a scalar without one.
 
bool is_nan (double x)
 A NaN?
 
template<typename T , typename Order >
T contiguous_extreme (const T *first, std::size_t count, const Order &larger)
 The largest (larger is std::greater) or smallest of count contiguous elements, or a NaN if one is among them, as NumPy's: the comparisons run without a branch over native vectors (or eight scalar lanes), each lane its own running best, and the NaN is tracked apart.
 
template<typename T , typename Order >
T parallel_extreme (const T *first, std::size_t count, const Order &larger)
 contiguous_extreme of count elements on the threads: runs of pairwise_leaf each its own, then their extremes'. Exact whatever the runs (a maximum rounds nothing), so the same answer on any number of threads. Four runs or fewer: on the caller, as parallel_pairwise_sum.
 
template<typename T , typename Order >
T walked_extreme (const T *first, std::size_t count, std::size_t stride, const Order &larger)
 extreme of a line walked element by element: a NaN wins, as in NumPy (every comparison with it is false, so a NaN candidate is taken apart, and once taken nothing compares larger than it).
 
template<std::floating_point T, typename Order >
T extreme (const T *first, std::size_t count, std::size_t stride, const Order &larger)
 The largest (larger is std::greater) or smallest element of a line of reals: a contiguous one by the comparison without a branch (so it vectorises) and the NaN tracked apart, a NaN anywhere the answer, as NumPy's; a strided one walked (walked_extreme).
 
template<typename T , typename Order >
requires (!std::floating_point<T>)
T extreme (const T *first, std::size_t count, std::size_t stride, const Order &larger)
 The largest (larger is std::greater) or smallest element of a line of another type (walked_extreme).
 
template<typename T >
constexpr bool by_reduction (std::size_t inner) noexcept
 Whether extremes_along takes the reduction's path: an exact type, or an axis of single elements. A function, where the test written in the condition joined a constant to the axis's, which MSVC 19.44 (Visual Studio 2022's) rejects (C4127, an error under /WX).
 
template<typename T , typename Order >
tensor< T > extremes_along (const tensor< T > &t, std::size_t axis, const Order &larger)
 The largest (larger is std::greater) or smallest elements along axis. Along an inner axis each line is contiguous (extreme); along an outer one the slabs are compared element by element, so the loop runs over contiguous memory rather than down a column.
 
std::vector< std::size_t > view_axes (const std::vector< std::size_t > &order)
 The axes that put the elements of a tensor laid out in order (tensor::laid_out, its view's axes in the order they lie) back in the view's order: the permutation to transpose a result of the same axes by.
 
std::vector< std::size_t > reduced_axes (const std::vector< std::size_t > &order, std::size_t axis)
 The axes that put a result along axis of a tensor laid out in order (view_axes), that axis taken out, back in the view's order.
 
template<typename T >
std::optional< laid_axis< T > > laid_along (const tensor< T > &t, std::size_t axis)
 t laid out and the axis matching axis: nothing for any tensor but a transpose, or an axis past the rank (the operation's own to refuse).
 
template<typename T , typename F >
auto as_laid_along (const tensor< T > &t, std::size_t axis, const F &f) -> std::optional< decltype(f(t, axis))>
 Along axis of a transpose (laid_along): f (f(elements, axis)) of its elements as they lie along the matching axis, none copied, the result (of the same axes) put back in the view's order. Each line the same elements in the same order: what is computed line by line, the same. Nothing for any other tensor, an axis past the rank, or the view's last axis where its elements lie apart (a transpose's columns): copied out as rows, those lines are sorted, scanned and transformed faster than in place as columns (2000 of 2000 reals sorted, 8.0 ms against 12.4, M1 Pro), where along any other axis they are columns either way.
 
template<typename T , typename F >
auto as_laid_reduced (const tensor< T > &t, std::size_t axis, const F &f) -> std::optional< decltype(f(t, axis))>
 as_laid_along of an f whose result takes the axis out, along any axis: a reduction of columns runs across them as fast as along rows (2000 sums of 2000, 0.74 ms against 1.19 copied out first, M1 Pro).
 
template<typename T >
bool lies_close (const tensor< T > &t) noexcept
 Whether an element-wise operation reads the view t's elements where they lie: stepping within a cache line along its last axis (a slice).
 
template<typename T , typename F >
auto parallel_map (const T *data, const std::vector< std::size_t > &shape, const F &f) -> tensor< std::decay_t< decltype(f(std::declval< const T & >()))> >
 f(x) for every one of the row-major elements at data, of shape shape (another array's storage, read where it lies).
 
template<typename T , typename F >
auto parallel_map (const tensor< T > &t, const F &f) -> tensor< std::decay_t< decltype(f(std::declval< const T & >()))> >
 f(x) for every element of t, by chunks of elementwise_chunk among the threads (as broadcast_apply's same-shape pairs): for a stateless f, the library's element-wise functions.
 
template<typename T , typename Op >
auto broadcast_rows (const T *a, const std::vector< std::size_t > &a_shape, const T *b, const std::vector< std::size_t > &b_shape, const Op &op)
 op(a, b) element by element of the row-major a (a_shape) and b (b_shape), broadcast against each other, in op's result type: broadcast_apply's paths for two contiguous tensors, so their bits, for elements no tensor holds (the binding's NumPy arrays). One element against as many axes or more is a number.
 
template<split_complex C>
C numpy_reduction_multiply (const C &a, const C &b, const numpy_loops &loops)
 a · b as NumPy's reduction loop computes it, fused as loops says.
 
numpy_loops numpy_wheel_loops () noexcept
 NumPy's loops as its current PyPI wheels build them on this processor (detail::numpy_fuses_products, detail::numpy_fuses_scalar_products, detail::numpy_fuses_quotients).
 
template<detail::split_complex C>
tensor< C > numpy_multiply (const tensor< C > &a, const tensor< C > &b, const numpy_loops &loops=numpy_wheel_loops())
 a · b of complex tensors, broadcast, each product as NumPy's array loop computes it (detail::numpy_multiply, fused as loops says): the binding's *, where std::complex's operator (this library's *) recovers infinities NumPy's makes NaN.
 
template<detail::split_complex C>
tensor< C > numpy_divide (const tensor< C > &a, const tensor< C > &b, const numpy_loops &loops=numpy_wheel_loops())
 a / b of complex tensors, broadcast, each quotient as NumPy's array loop computes it (detail::numpy_divide, fused as loops says): the binding's /.
 
template<detail::split_complex C>
tensor< C > numpy_prod (const tensor< C > &t, std::size_t axis, const numpy_loops &loops=numpy_wheel_loops())
 The products of a complex t along axis, each line in order from 1 (NumPy's identity, which turns an infinite element alone into a NaN part), each product as NumPy's loops compute it (detail::numpy_multiply): its reduction loop's where nothing follows the axis, its vector loop's (between whole slices) where something does; fused as loops says. The binding's prod, where std::complex's operator (prod) recovers infinities NumPy's makes NaN.
 
template<detail::split_complex C>
C numpy_prod (const tensor< C > &t, const numpy_loops &loops=numpy_wheel_loops())
 The product of every element of a complex t, in order from 1 (NumPy's identity), as NumPy's reduction loop computes it (detail::numpy_multiply, fused as loops says).
 
template<detail::split_complex C>
tensor< typename C::value_type > numpy_absolute (const tensor< C > &a, const numpy_loops &loops=numpy_wheel_loops())
 |x| of every element of a complex tensor as NumPy's absolute computes it (detail::numpy_absolute), fused as loops says of its vector loops.
 
template<detail::split_complex C>
tensor< C > numpy_square (const tensor< C > &a, const numpy_loops &loops=numpy_wheel_loops())
 x ** 2 of a complex tensor as NumPy's square computes it (the loop of an array to the Python integer 2): its product's vector loop, fused as loops says of it.
 
template<detail::split_complex C>
tensor< C > numpy_reciprocal (const tensor< C > &a, const numpy_loops &loops=numpy_wheel_loops())
 x ** -1 of a complex tensor as NumPy's reciprocal computes it (the loop of an array to the Python integer -1, detail::numpy_reciprocal), fused as loops says of its scalar loops.
 
template<detail::split_complex C>
tensor< C > numpy_power (const tensor< C > &a, const tensor< C > &b, const numpy_loops &loops=numpy_wheel_loops())
 a ** b of complex tensors, broadcast, as NumPy's power computes each (detail::numpy_power), fused as loops says of its scalar loops.
 
template<typename T >
tensor< T > exp_of (const T *data, const std::vector< std::size_t > &shape)
 exp of every one of the row-major elements at data, of shape shape: elementary_exp of each.
 
tensor< double > exp_of (const double *data, const std::vector< std::size_t > &shape)
 exp of every one of the row-major doubles at data, of shape shape: exp_into by chunks of elementwise_chunk among the threads.
 
template<typename T , typename Kernel >
tensor< T > runs_into (const tensor< T > &t, const Kernel &kernel)
 kernel (kernel(in, out, count), of contiguous elements) of every element of t: a contiguous tensor at once by chunks of elementwise_chunk among the threads; a view lying close (lies_close; a slice, its rows apart) read where it lies, each run of its last axis passed where it is contiguous and gathered by pieces of 512 on the stack where its elements are apart; any other copied first. Each element alone: the bits of the same elements laid row-major (a slice of 200 000 doubles' exp copied first took a third more time).
 
tensor< double > by_runs (const tensor< double > &a, doubles_into into)
 A function of every element of a, its kernel of doubles (runs_into).
 
tensor< float > by_runs (const tensor< float > &a, doubles_into into)
 A function of every element of a, its kernel of doubles through doubles (through_doubles_piece).
 
template<typename T >
tensor< T > exp_viewed (const tensor< T > &a)
 exp of every element of a: of a contiguous copy (exp_of).
 
tensor< double > exp_viewed (const tensor< double > &a)
 exp of every double of a, a view read where it lies (by_runs).
 
template<typename T >
tensor< T > expm1_viewed (const tensor< T > &a)
 expm1 of every element of a: of a contiguous copy (expm1_of).
 
tensor< double > expm1_viewed (const tensor< double > &a)
 expm1 of every double of a, a view read where it lies (by_runs).
 
tensor< float > expm1_viewed (const tensor< float > &a)
 expm1 of every float of a through doubles, a view read where it lies (by_runs).
 
template<typename T >
tensor< T > atan_viewed (const tensor< T > &a)
 atan of every element of a: of a contiguous copy (atan_of).
 
tensor< double > atan_viewed (const tensor< double > &a)
 atan of every double of a, a view read where it lies (by_runs).
 
tensor< float > atan_viewed (const tensor< float > &a)
 atan of every float of a through doubles, a view read where it lies (by_runs).
 
template<typename T >
tensor< T > sinh_viewed (const tensor< T > &a)
 sinh of every element of a: of a contiguous copy (sinh_of).
 
tensor< double > sinh_viewed (const tensor< double > &a)
 sinh of every double of a, a view read where it lies (by_runs).
 
tensor< float > sinh_viewed (const tensor< float > &a)
 sinh of every float of a through doubles, a view read where it lies (by_runs).
 
template<typename T >
tensor< T > cosh_viewed (const tensor< T > &a)
 cosh of every element of a: of a contiguous copy (cosh_of).
 
tensor< double > cosh_viewed (const tensor< double > &a)
 cosh of every double of a, a view read where it lies (by_runs).
 
tensor< float > cosh_viewed (const tensor< float > &a)
 cosh of every float of a through doubles, a view read where it lies (by_runs).
 
template<typename T >
tensor< T > exp2_viewed (const tensor< T > &a)
 exp2 of every element of a: of a contiguous copy (exp2_of).
 
tensor< double > exp2_viewed (const tensor< double > &a)
 exp2 of every double of a, a view read where it lies (by_runs).
 
template<typename T >
tensor< T > log1p_viewed (const tensor< T > &a)
 log1p of every element of a: of a contiguous copy (log1p_of).
 
tensor< double > log1p_viewed (const tensor< double > &a)
 log1p of every double of a, a view read where it lies (by_runs).
 
tensor< float > log1p_viewed (const tensor< float > &a)
 log1p of every float of a through doubles, a view read where it lies (by_runs).
 
template<typename T >
tensor< T > sin_viewed (const tensor< T > &a)
 sin of every element of a: of a contiguous copy (sin_of).
 
tensor< double > sin_viewed (const tensor< double > &a)
 sin of every double of a, a view read where it lies (by_runs).
 
tensor< float > sin_viewed (const tensor< float > &a)
 sin of every float of a through doubles, a view read where it lies (by_runs).
 
template<typename T >
tensor< T > cos_viewed (const tensor< T > &a)
 cos of every element of a: of a contiguous copy (cos_of).
 
tensor< double > cos_viewed (const tensor< double > &a)
 cos of every double of a, a view read where it lies (by_runs).
 
tensor< float > cos_viewed (const tensor< float > &a)
 cos of every float of a through doubles, a view read where it lies (by_runs).
 
template<typename T >
tensor< T > tanh_viewed (const tensor< T > &a)
 tanh of every element of a: of a contiguous copy (tanh_of).
 
tensor< double > tanh_viewed (const tensor< double > &a)
 tanh of every double of a, a view read where it lies (by_runs).
 
tensor< float > tanh_viewed (const tensor< float > &a)
 tanh of every float of a through doubles, a view read where it lies (by_runs).
 
template<std::floating_point T>
T ieee_maximum (T a, T b) noexcept
 The larger of a and b as IEEE 754-2019's maximum orders them: a NaN wins (a's where both are), and +0 is over −0 whatever their order (a + b of two zeros: −0 only where both are).
 
template<std::floating_point T>
T ieee_minimum (T a, T b) noexcept
 The smaller of a and b as IEEE 754-2019's minimum orders them: a NaN wins (a's where both are), and −0 is under +0 whatever their order (−(−a − b) of two zeros: +0 only where both are).
 
template<std::floating_point T>
T masked_select (bool condition, T if_true, T if_false) noexcept
 if_true where condition holds, else if_false, by a mask of their bits: no branch, so a loop of it vectorises (GCC 13 makes a ?: of reals a branch).
 
template<std::floating_point T>
T clipped (T x, T low, T high) noexcept
 ieee_minimum of ieee_maximum of x and low, and high, the bounds no NaN: the same bits (a NaN x kept, zeros of the two signs ordered alike), by selections, so that clip's loop vectorises.
 
template<std::floating_point R, typename F >
requires (!std::same_as<R, float>)
auto with_decimal_rounding (int decimals, const F &f)
 f(rounding) with the decimal_rounding of R to decimals places: in R, as NumPy's, but in double for a float whose 10ᵈ overflows it (a float under 2²⁴ · 10⁻ᵈ still has places to round there).
 
template<std::same_as< float > R, typename F >
auto with_decimal_rounding (int decimals, const F &f)
 with_decimal_rounding of a float: in double where its 10ᵈ overflows a float.
 
template<std::floating_point T>
std::pair< T, T > floor_divmod (T a, T b) noexcept
 a // b and a % b of reals as NumPy's divmod computes them (Python's: the quotient floored, the remainder of b's sign): fmod first, the quotient (a − mod) / b adjusted by the remainder's sign and snapped to the nearest integer, a zero of the right sign. b zero: a / b and fmod(a, b).
 
template<std::signed_integral T>
T integer_power (T base, T exponent) noexcept
 base to the power exponent (not negative) by repeated squaring, each product wrapped modulo 2ⁿ as NumPy's integer power (arithmetic).
 
template<typename T , typename Step >
tensor< T > scan_along (const tensor< T > &t, std::size_t axis, const Step &step)
 Each line of t along axis replaced by its running step (x₀, step(x₀, x₁), …), in order.
 
template<detail::split_complex C>
tensor< C > numpy_cumprod (const tensor< C > &t, std::size_t axis, const numpy_loops &loops=numpy_wheel_loops())
 The running products of a complex t along axis, in order, each as NumPy's loops compute it (detail::numpy_multiply): its vector loop's for a line of two (one product), its reduction loop's past it; fused as loops says. The binding's cumprod.
 
template<std::floating_point T>
bool sorts_before (T a, T b) noexcept
 Whether a sorts before b as NumPy sorts reals: ascending, NaNs last.
 
template<split_complex C>
bool sorts_before (const C &a, const C &b) noexcept
 Whether a sorts before b as NumPy sorts complex numbers (npysort's CLT): those with no NaN by the real parts then the imaginary parts, then those with a NaN imaginary part alone by the real parts, then a NaN real part alone by the imaginary parts, then both NaN (R + Rj < R + nanj < nan + Rj < nan + nanj).
 
template<std::floating_point T>
order_key_t< T > order_key (T x) noexcept
 x's bits as an unsigned integer ordered as x is by <: the sign bit set for a positive real, every bit flipped for a negative one. A NaN and the zeros' signs are the caller's: −0 orders below +0.
 
template<std::floating_point T>
T from_order_key (order_key_t< T > key) noexcept
 The real whose order_key is key.
 
template<typename U >
void count_bytes (byte_counts< U > &counts, U key) noexcept
 Counts key's bytes in counts. Every byte is counted in the one read that makes the keys, where a read per byte cost a fifth of the sort (10⁶ float64: 22.8 → 18.5 ms on x86-64, 12.1 → 9.5 on arm64).
 
std::size_t part_begin (std::size_t count, std::size_t parts, std::size_t p) noexcept
 Where part p of parts sharing count elements begins: the first count % parts one longer.
 
template<typename U >
bool radix_passes (U *keys, U *key_spare, std::int64_t *indices, std::int64_t *index_spare, std::size_t count, std::vector< byte_counts< U > > &counts)
 Sorts the count keys at keys stably by their bytes from the least significant, the indices at indices with them unless null, in counts.size() parts among the threads. Each pass places a part's keys of one byte value after the earlier parts' and moves each part's in its order, so equal keys keep their order whatever the parts: the bits of one counting sort. counts holds each part's counts of every byte of the keys as they lie, counted again for a part once a pass has moved them (one part's are the whole's, whatever their order); a byte that every key shares is skipped.
 
template<typename T >
void radix_sort (T *first, T *last, std::size_t parts=1)
 Sorts [first, last) by order_key, a byte at a time from the least significant (by radix_passes, a stable counting sort each, so equal keys keep their order; a byte that every key shares skipped), in parts among the threads.
 
template<std::floating_point T>
void sort_run (T *first, T *last, std::size_t parts=1)
 Sorts [first, last) stably as sorts_before orders reals: the NaNs moved behind the rest in their order first, the rest by <, which tests for no NaN. A long run by radix_sort, its zeros then rewritten with their signs in their order (the bits order −0 below +0, where < holds them equal); a short one by std::stable_sort with a lambda of its own, not the default std::less, which libc++ 22 sorts by the bits from 1024 doubles on, −0 first. The radix sort in parts among the threads.
 
template<std::floating_point T>
order_key_t< T > argsort_key (T x) noexcept
 A real's order_key for an argsort: both zeros the same key, as < holds them equal (their indices then order them).
 
order_key_t< std::int64_t > argsort_key (std::int64_t x) noexcept
 An integer's order_key for an argsort.
 
template<std::floating_point T>
std::int64_t * numbers_end (std::int64_t *first, std::int64_t *last, const T *values, std::size_t stride)
 Where a real's NaNs begin among the indices [first, last), moved behind the rest in their order.
 
std::int64_t * numbers_end (std::int64_t *, std::int64_t *last, const std::int64_t *, std::size_t)
 An integer has no NaN: every index is a number's.
 
template<radix_orderable T, typename Less >
void order_run (std::int64_t *first, std::int64_t *last, const T *values, std::size_t stride, const Less &less, std::size_t parts=1)
 Sorts the indices [first, last) stably by their elements (values[k · stride]), as sorts_before orders them: comparisons for a run shorter than radix_sort_least, else the NaNs set apart in their order and the rest by argsort_key, a byte at a time (radix_passes, in parts among the threads: a stable counting sort of the keys and their indices together; a byte that every key shares skipped): equal elements by index either way.
 
template<std::integral T>
void sort_run (T *first, T *last)
 Sorts [first, last) stably as integers sort.
 
void sort_run (std::int64_t *first, std::int64_t *last, std::size_t parts=1)
 Sorts [first, last) as int64 sort: a long run by radix_sort in parts among the threads (equal integers are the same bits, so any order of them is the stable one).
 
template<split_complex C>
void sort_run (C *first, C *last)
 Sorts [first, last) stably as sorts_before orders complex numbers.
 
std::size_t merged_pieces (std::size_t threads) noexcept
 The pieces parallel_sort_run sorts a line in for threads: the least power of four that many.
 
template<typename T , typename SortRun , typename Less >
void parallel_sort_run (T *data, std::size_t count, std::size_t pieces, const SortRun &sort_piece, const Less &less)
 Sorts the count elements at data stably in pieces, a power of four, among the threads: each piece sorted apart by sort_piece, then the pieces merged in pairs by less (the order sort_piece sorts by), round after round, between data and a scratch buffer (std::merge takes the first piece's element of two equal ones, so the merge keeps their order). The rounds are even in number, so the last writes data.
 
template<radix_orderable T>
void sort_shared (T *run, std::size_t count, std::size_t threads)
 Sorts one long line of count elements at run among threads: each pass of its radix sort shared out (radix_passes), the one-thread sort's bits with no merge.
 
template<radix_orderable T, typename Less >
void order_shared (std::int64_t *order, std::size_t count, const T *values, std::size_t stride, const Less &less, std::size_t threads)
 Sorts the indices of one long line among threads: each pass of its radix sort shared out.
 
std::vector< std::size_t > partition_positions (std::vector< std::size_t > kths, std::size_t extent)
 kths ascending and each once, every one checked against extent.
 
template<typename It , typename Less >
void select_positions (It first, std::size_t count, const std::vector< std::size_t > &kths, const Less &less)
 Rearranges [first, first + count) by less so that each of the ascending kths holds the element a sort puts there: std::nth_element over what lies past the previous one.
 
template<typename T >
T conjugated_chunks (const T *a, const T *b, std::size_t count)
 Σ a[i] · b[i] of reals, their own conjugates: chunked_dot.
 
template<split_complex C>
C conjugated_chunks (const C *a, const C *b, std::size_t count)
 Σ conj(a[i]) · b[i] of complex numbers: chunked_conjugate_dot.
 
template<std::floating_point T>
bool wins_any_order (T x) noexcept
 A real's NaN, which wins numpy.argmax and numpy.argmin.
 
template<typename T , typename Match >
bool holds (const T *first, std::size_t count, const Match &match) noexcept
 Whether one of the count contiguous elements at first passes match: tested without a branch, so the loop vectorises.
 
template<typename T , typename Match >
std::size_t first_held (const T *first, std::size_t count, const Match &match) noexcept
 The index of the first of the contiguous elements at first that passes match, which one of them does: blocks of 64 tested by holds, then the block that holds it walked.
 
template<std::floating_point T, typename Order >
std::int64_t extreme_index (const T *first, std::size_t count, const Order &larger)
 The index of the first largest (larger std::greater) or smallest of count contiguous elements, as NumPy's: the first NaN where there is one. Reals in two passes without a branch each, as fast as NumPy's vectors: their extreme (contiguous_extreme, a NaN where there is one), then the first element that is it (zeros of both signs equal, so the first of either, as NumPy's).
 
template<typename T , typename Order >
tensor< std::int64_t > extreme_indices (const tensor< T > &t, std::size_t axis, const Order &larger)
 argmax or argmin along axis by larger.
 
template<typename T , typename Order >
std::int64_t parallel_extreme_index (const T *first, std::size_t count, const Order &larger)
 extreme_index of the count contiguous elements at first, by chunks on the threads: each chunk's own, then the first of the chunks' (a NaN's chunk wins at once, the earliest of equal ones otherwise).
 
template<std::floating_point T>
T stable_choice (const T *row, std::size_t count, std::size_t k, T chosen)
 Of the count elements at row, the one a stable sort (sort_run) puts at k, from the one chosen that a selection put there, equal to it under sorts_before. A zero (either sign) or a NaN (any payload) is not alone in its class, and the class's member of the same rank in row's order is the stable sort's.
 
template<typename T >
requires std::floating_point<T> || std::integral<T>
void select_row (const T *row, std::size_t count, const std::int64_t *at, std::size_t m, T *work, T *out)
 Into out, the elements of the count at row that a stable sort puts at the m positions at, work holding count elements of scratch: std::nth_element at each distinct position, smallest first, each over the part above the one before (the least or the largest of that part by a pass alone), then stable_choice; more positions than select_positions_sorted, the row sorted whole.
 
template<split_complex C>
void select_row (const C *row, std::size_t count, const std::int64_t *at, std::size_t m, C *work, C *out)
 A complex row is sorted whole: its classes under sorts_before are zeros and NaNs in either part.
 
template<typename T >
T correlation_step (const T &sum, const T &x, const T &y)
 sum + x · y of a real or an integer (wrapped) correlation's terms.
 
template<split_complex C>
C correlation_step (const C &sum, const C &x, const C &y)
 sum + x · conj(y) of a complex correlation's terms, NumPy's product rounded.
 
std::size_t triangle_split (std::size_t i, std::size_t rows, std::size_t cols, std::ptrdiff_t k, bool lower)
 The first column of row i past the part to and below its k diagonal (lower), or the first column of the part to and above it (else), within [0, cols]; k clamped first, so i + k cannot overflow.
 
constexpr bool range_within (std::size_t first, std::size_t count, std::ptrdiff_t step, std::size_t extent) noexcept
 Whether count indices from first, step apart, all lie in [0, extent): none is out for a count of 0. In unsigned arithmetic, the last index's distance from first never formed where it would pass the axis, so that no step and no count overflows ((count - 1) · step in std::ptrdiff_t did, a step of PTRDIFF_MIN and a count of 3 among them, and could then pass the test).
 
std::size_t checked_mul (std::size_t a, std::size_t b)
 a * b, throwing numerical::dimension_error if the product overflows std::size_t. Guards dimension/shape products (element counts, allocation sizes) against silent wraparound on absurd inputs. The a != 0 guard also keeps the max / a division well-defined.
 
template<typename It >
std::size_t element_count (It first, It last)
 The elements a shape's extents [first, last) hold: 0 where one of them is 0, whatever the others, else their product (checked_mul). The extents a tensor keeps, past one it holds 0 of, may multiply past std::size_t: a result of them is empty, not of a wrapped size.
 

Variables

constexpr std::size_t thread_release_places {256}
 The most workspaces a thread registers (on_release): one per kind, scalar type and slot, a few dozen where every type is used (the C++ tests' thread: 34), so that registering allocates nothing (their accessors throw nothing).
 
constexpr std::size_t kept_pages_largest {std::size_t {1} << 25U}
 The bytes under which a buffer's block is a kept one, no huge pages asked: 32 MiB, four of them at most 128 MiB kept.
 
constexpr std::size_t kept_pages_least {std::size_t {1} << 20U}
 The fewest bytes a buffer takes a kept block of 4 KiB pages for (kept_page_blocks): below, the allocator's own free lists reuse what is freed.
 
template<typename T >
constexpr bool is_complex_number {false}
 Whether T is a std::complex of a floating-point type.
 
template<std::floating_point R>
constexpr R exp_bound {static_cast<R>(static_cast<int>(static_cast<R>(std::numeric_limits<R>::max_exponent - 1) * std::numbers::ln2_v<R>))}
 The real part past which exp of it overflows, rounded down: 709 for double, 88 for float.
 
template<std::floating_point R>
constexpr R half_exp_bound {static_cast<R>(static_cast<int>(static_cast<R>(std::numeric_limits<R>::max_exponent - 1) * std::numbers::ln2_v<R> / R {2}))}
 Half of it, past which tanh's parts are ±1 and a quotient by exp(2x): 354 for double, 44 for float.
 
constexpr double dqds_tolerance_squared
 LAPACK's bound of the off-diagonal against the diagonal under which a value deflates: (100 ε)².
 
constexpr qd_index dqds_sweeps {100}
 The steps of the inner iteration per value of its block before dqds gives up (LAPACK's 100).
 
constexpr double schur_ulp {std::numeric_limits<double>::epsilon()}
 The relative spacing LAPACK's ‘dlamch('P’)` names: the roundoff a QR sweep judges a subdiagonal by.
 
constexpr std::size_t schur_steps_per_order {30}
 The iterations one eigenvalue (or pair) may take per unit of the active order, as dlahqr's 30.
 
constexpr double balance_limit {0x1p256}
 The largest power of two a balancing scales a row or a column by, either way: the matrix enters at a largest magnitude near 1, so no entry overflows, and the scaling stops where LAPACK's would be one factor among dozens.
 
constexpr double back_substitution_limit {0x1p400}
 The modulus an eigenvector's components are kept under during the back-substitution: the sums stay finite for any order a matrix can have, and a component that would grow past it scales the vector down.
 
constexpr std::size_t hessenberg_block {32}
 The columns of a panel of hessenberg_blocked.
 
constexpr std::size_t hessenberg_crossover {128}
 The active order from which hessenberg reduces by panels.
 
constexpr lapack_index multishift_crossover {75}
 LAPACK's nmin: the active order below which dlaqr0's iterations skip a sweep after enough deflation.
 
constexpr lapack_index window_growth {5}
 The iterations without deflation after which the window grows (kexnw).
 
constexpr lapack_index exceptional_after {6}
 The iterations without deflation after which the shifts are exceptional (kexsh).
 
constexpr lapack_index nibble {14}
 The share of the window, in percent, whose deflation makes the sweep unnecessary (nibble).
 
constexpr double exp_inv_ln2_128 {0x1.71547652b82fep+7}
 128/ln 2 (tools/exp_table.py derives each constant here).
 
constexpr double exp_ln2_128_high {0x1.62e42fee00000p-8}
 ln 2/128 to 32 significant bits, toward zero: its product by an integer of 18 bits or fewer is exact, and so is that product's difference from an argument it reduces.
 
constexpr double exp_ln2_128_low {0x1.a39ef35793c76p-40}
 What exp_ln2_128_high leaves out of ln 2/128, to the nearest double.
 
constexpr double exp_normal_bound {-0x1.6232bdd7abcd2p+9}
 ln 2^-1022 toward zero: below it, exp is subnormal, where table_exp's two scalings would round twice.
 
constexpr std::array< double, 256 > exp_table
 2^(j/128) for j = 0…127, at 2j the double nearest it and at 2j + 1 the double nearest what that one leaves out. Not inline: a copy in each translation unit, whose loads Clang's vectoriser bounds, where it kept a loop of table_exp scalar over an inline variable's ("cannot identify array bounds").
 
constexpr double exp2_ln2 {0x1.62e42fefa39efp-1}
 ln 2 to the nearest double.
 
constexpr double atan_half_pi_high {0x1.921fb54442d18p+0}
 π/2, the double nearest it (tools/atan_table.py derives each constant here).
 
constexpr double atan_half_pi_low {0x1.1a62633145c07p-54}
 What atan_half_pi_high leaves out of π/2, to the nearest double.
 
constexpr std::array< double, 130 > atan_table
 atan(k/64) for k = 0…64, at 2k the double nearest it and at 2k + 1 the double nearest what that one leaves out. Not inline, as exp_table.
 
constexpr std::array< double, 3 > half_pi_parts {0x1.921fb54442d18p+0, 0x1.1a62633145c07p-54, -0x1.f1976b7ed8fbcp-110}
 π/2 in three parts: its double, then the double of the rest, then the double of what is left (their sum within 6·10⁻⁵⁰ of π/2).
 
constexpr double sine_reduction_bound {0x1p20}
 The magnitude to which reduced_sine reduces its argument: k below 2²⁰, k·π/2's parts exact enough.
 
constexpr double sine_inverse_step {0x1.45f306dc9c883p+3}
 32/π, the double nearest it (tools/sine_table.py derives each constant here).
 
constexpr std::array< double, 3 > sine_step_parts {0x1.921fb54442d18p-4, 0x1.1a62633145c07p-58, -0x1.f1976b7ed8fbcp-114}
 π/32 in three parts: its double, then the double of the rest, then the double of what is left.
 
constexpr std::array< double, 256 > sine_table
 For j = 0…63, at 4j sin(jπ/32) and at 4j + 1 what that double leaves out, at 4j + 2 cos(jπ/32) and at 4j + 3 what that one leaves out, each the double nearest. Not inline, as exp_table.
 
constexpr double log_ln2_high {0x1.62e42fefa3800p-1}
 ln 2 to 42 significant bits, toward zero: its product by an exponent of 11 bits or fewer is exact (tools/log_table.py derives each constant here).
 
constexpr double log_ln2_low {0x1.ef35793c76730p-45}
 What log_ln2_high leaves out of ln 2, to the nearest double.
 
constexpr std::array< double, 91 > log_inverse
 For k = 90…180, at k − 90 the double nearest 128/k. Not inline, as exp_table.
 
constexpr std::array< double, 182 > log_table
 For k = 90…180, at 2(k − 90) −ln of log_inverse's entry to the nearest multiple of 2^−42 (its sum with an exponent times log_ln2_high exact) and at 2(k − 90) + 1 the double nearest what that leaves out. Not inline, as exp_table.
 
constexpr std::size_t lu_block {64}
 The columns of one panel of the blocked LU: the trailing update, a product of this depth, is deep enough for the packed kernel while the unblocked panel stays a small part of the work.
 
constexpr std::size_t lu_small_order {256}
 The order up to which the LU takes panels of lu_small_block columns: there the panels, a column at a time on one thread, are most of the time, and narrower ones halve their work (arm64, order 300 on every core: 14 % slower at 32 columns than at 64).
 
constexpr std::size_t lu_small_block {32}
 The columns of one panel of the LU of an order up to lu_small_order.
 
constexpr std::size_t lu_wide_order {3072}
 The order from which the LU takes panels of lu_wide_block columns: each trailing update rereads the whole trailing block, and from there the memory traffic of panels of 64, not their arithmetic, sets the time; 128 halves it. Order 4096: ten AVX-512 cores 322 → 273 ms, six AVX2 cores 416 → 330, eight arm64 cores 285 → 252; order 3072 10 to 17 % faster; 2560 the same; 2048 10 % slower, the wider panel's own work then the longer. The factors are the same to the bit at either width.
 
constexpr std::size_t lu_wide_block {128}
 The columns of one panel of the LU from lu_wide_order.
 
constexpr std::size_t lu_lookahead_columns {1024}
 The trailing columns at most past which the LU factorises its next panel beside the update of the rest (a look-ahead): a wider update runs well as one product, and cut in blocks beside the panel it ran longer (order 2048 on six x86-64 cores, 9 %); a narrower one leaves threads idle while the panel runs.
 
constexpr std::size_t lu_fused_columns {512}
 The trailing columns at most past which each block of the update beside the panel solves its own rows of U₁₂: a wider update keeps every thread busy past the panel, and its rows solved on all of them first ran faster than on the workers alone (order 1000 on four AVX2 cores, 2 %).
 
constexpr std::size_t lu_update_speedup {gemm_kernel_speedup}
 The multiply-adds of an LU's trailing update one of the tile's dependent ones stands for, as threads are weighed (gemm_kernel_speedup): the AMX tile's, on arm64; the AVX tiles' quarter of it on x86-64, where the update of order 256 waited on one thread (six cores: 690 → 622 µs, 1024: 12.5 → 12.1 ms).
 
constexpr std::size_t narrow_solve {16}
 The most right-hand sides an LU solve takes row by row rather than by blocks through the product kernel.
 
constexpr std::size_t slice_columns {64}
 The fewest columns an LU solve gives a thread of its own: a narrower slice starves the product kernel (arm64, order 256 and 64 right-hand sides in eight slices: 112 → 194 µs; 48 columns, order 300 and 300 right-hand sides: 8 % slower than 64).
 
constexpr std::size_t solve_leaf {16}
 The columns of one leaf of the recursive LU panel.
 
constexpr std::size_t inverse_band {16}
 The rows an LU inverse writes its identity into, and permutes the columns of, on one thread at a time.
 
constexpr std::size_t svd_band {32}
 The rows of an SVD's input one thread scales and copies at a time.
 
constexpr std::size_t zeroed_rows {32}
 The rows of a Cholesky factor whose upper part one thread zeroes at a time.
 
constexpr std::size_t cholesky_block {64}
 The columns of one panel of the blocked Cholesky factorisation (as lu_block).
 
constexpr std::size_t cholesky_diagonal_rows {4}
 The rows of a Cholesky diagonal block factorised together, one chain of sums each (one x86-64 AVX2 core, order 300: the diagonal blocks 128 → 65 µs; eight rows did no better).
 
constexpr std::size_t cholesky_together_width {16}
 The narrowest Cholesky diagonal block whose rows go together: in a block of 8, the groups' loops cost more than the chains save (order 8, one arm64 core: 0.21 → 0.23 µs).
 
constexpr std::size_t cholesky_one_panel {48}
 The order below which the Cholesky factorisation is one panel: no trailing product, whose packing the few panels of a small order do not repay (order 17 on one thread: 0.8 µs whole, 1.3 µs in panels of 16). Panels of 16 win from order 40 on x86-64 and from 48 on arm64.
 
constexpr std::size_t cholesky_rows {64}
 The rows of one chunk of a Cholesky panel's rows below it, solved together along each column.
 
constexpr std::size_t cholesky_update_columns {64}
 The fewest columns of the trailing block one Cholesky update's product takes: each product packs every row below its columns again, a panel's width apart the packing outweighed the products on the AMX (M1 Pro, order 128: 61 → 53 µs), and the triangle a wider block computes past the diagonal is wasted.
 
constexpr std::size_t cholesky_update_rows {256}
 The rows of one tile of a Cholesky update: the tiles of a panel's update, each one product on one thread, share the threads in one parallel loop, where a parallel product for each block of columns started and joined the threads some thirty times a panel.
 
template<typename T >
constexpr carried_t< T > rescue_below {std::is_same_v<value_range_t<T>, float> ? 0x1p-60 : 0x1p-500}
 The magnitude under which rescue_factor scales a T up: about the square root of the smallest normal number of its range, or below it (2⁻⁵⁰⁰ for a long double, whose range is the platform's).
 
constexpr double plain_squares_low {0x1p-900}
 The smallest plain sum of squares plain_or_at_scale keeps: a term that underflowed is below 2⁻¹⁰²², so a sum this large loses at most count · 2⁻¹²² of itself to them.
 
constexpr double plain_squares_high {0x1p900}
 The largest plain sum of squares plain_or_at_scale keeps: no term of it overflowed.
 
constexpr std::size_t finite_chunk {std::size_t {1} << 15U}
 The elements all_finite_shared gives a thread at a time.
 
template<typename T >
constexpr bool real_rows {std::is_same_v<T, double> || std::is_same_v<T, float>}
 Whether a solve of T goes by rows of real terms (lu_decomposition's solve_narrow): double and float.
 
constexpr std::size_t triangular_block {32}
 The rows triangular_halves solves one by one; past them, by halves of a multiple of them.
 
constexpr std::size_t triangular_slice {64}
 The columns of the right-hand sides a thread takes in triangular_halves' rows one by one.
 
constexpr std::size_t lu_tall_ratio {2}
 The ratio of rows to columns up to which lu_factors factorises a tall matrix as the square of it beside zero columns (m³ / 3, by the blocked LU), past which by tall_lu_columns (m n² on narrow products): the two even near twice as tall as wide (arm64, 1200 rows).
 
constexpr std::size_t qr_block {32}
 The reflectors of one panel of the blocked QR.
 
constexpr std::size_t qr_look_ahead_threads {3}
 The fewest threads for which reflect_in_columns looks ahead, a panel factorised while the threads update the columns past it: on two, one thread would update the rest alone (ten AVX-512 cores, order 1024: 29.4 ms as it is, 36.3 looking ahead), from three a panel no longer waits for the update (four threads 22.1 → 16.5 ms, eight 19.8 → 13.9).
 
constexpr std::size_t qr_update_slice {32}
 The fewest columns a task of reflect_in_columns' look-ahead updates: each task repacks the block, which narrow slices pay for too often (fixed slices of 64 columns on one thread, order 1024: 41 → 61 ms; with a slice per thread, a least of 32 or of 64 alike).
 
constexpr std::size_t qr_transpose_tile {32}
 The side of the tiles in which the QR transposes (transpose_into): a tile's rows and columns both stay in the cache, where a whole column read across rows misses on every element.
 
constexpr std::size_t jacobi_sweeps {60}
 The sweeps a Jacobi method may take: it converges quadratically, in under 15 on every matrix measured; the bound turns a non-convergence into an error instead of a hang.
 
constexpr std::size_t corrected_product_least {512}
 The rows from which corrected_hermitian_product shares its coefficients and bands with the threads: on six x86-64 AVX2 cores from 256 (below, eigh of order 128 took 13 % longer), on an M1 Pro from 512 (from 256, eigh of order 512 on eight cores took 3 to 9 % longer).
 
constexpr std::size_t tridiagonal_block {32}
 The reflections of one panel of the blocked tridiagonal reduction.
 
constexpr std::size_t tridiagonal_band {512}
 The rows of each band of the tridiagonal reduction's trailing update: a band reaches its diagonal, so a wider one computes more of the upper triangle the reduction never reads, a narrower one runs the product kernel on smaller products.
 
constexpr std::size_t ql_steps_per_value {30}
 The implicit QL steps the tridiagonal eigensolver may take per eigenvalue, as LAPACK's dsteqr bounds them (30 n in all): it converges cubically, in two or three per eigenvalue.
 
constexpr std::size_t divide_leaf {25}
 The order up to which tridiagonal_divide_and_conquer diagonalises by implicit QL.
 
constexpr std::size_t secular_iterations {100}
 The iterations one root of the secular equation may take: it converges in a few; the bound turns a non-convergence into an error instead of a hang.
 
constexpr std::size_t lowner_block {32}
 The poles a task of merge_rank_one's Löwner products takes, and the side of its tiles of u.
 
constexpr std::size_t first_touch_chunk {std::size_t {1} << 15}
 The elements a task of workspace_of touches first: 64 pages of 4 KiB, so a buffer of a few megabytes spreads over every core.
 
constexpr std::size_t kept_workspace {std::size_t {1} << 18}
 The most elements a buffer of a thread's merge_workspace keeps from one decomposition to the next: 2 MiB, every buffer an eigh of order 512 needs, so a thread keeps at most 8 MiB.
 
constexpr std::size_t pencil_block {64}
 The columns reduce_pencil reduces a block at a time, as LAPACK's sygst by its block size.
 
template<typename T >
constexpr std::size_t pencil_reduction_least {is_complex_number<T> ? 3072 : 2048}
 The order from which a pencil's standard form is reduced by blocks (reduce_pencil): half the operations of two whole triangular solves, but a block at a time, where the two solves share their columns among the threads whole. Below it the solves ran faster: ten AVX-512 cores up to order 1100 or so, six AVX2 cores up to 500, the M1 Pro (its products on AMX) up to some 2000 real and 2500 complex. From it: order 3072, six AVX2 cores 683 → 335 ms (complex 1749 → 884), ten AVX-512 cores 423 → 224 ms (complex 900 → 528), M1 Pro 204 → 161 ms (complex 724 → 684). One order whatever the threads: the bits of C do not depend on their number.
 
constexpr double_double quarter_pi {0x1.921fb54442d18p-1, 0x1.1a62633145c07p-55}
 π/4 as a double-double number.
 
constexpr std::size_t fft_largest_radix {61}
 The largest prime a Stockham pass takes as its radix, by its own r-point sum: a length with a larger prime factor is transformed by Bluestein's convolution.
 
constexpr std::size_t fft_four_step_least {std::size_t {1} << 22}
 The fewest bytes of complex numbers a transform takes in four steps (four_step_plan), whose columns and rows the threads share: fewer stay in the caches through the passes of one transform, faster on a thread alone (ten AVX-512 cores: complex64 2¹⁸ points 1.5 ms in passes, 2.0 in four steps). From 4 MiB (complex128 2¹⁸ points: 5.0 ms in passes, 3.8 in four steps on all threads, 5.4 on one; M1 Pro, one thread, 3.6 against 2.1 ms). By the length and the type alone, so that a transform's bits are its own on any threads.
 
constexpr std::size_t fft_four_step_pad {8}
 The complex numbers past each row of the four steps' working space: rows a power of two of bytes apart fall in a few sets of the first-level cache, a column's elements evicting one another.
 
template<std::floating_point R>
constexpr std::size_t fft_four_step_band {256 / sizeof(std::complex<R>)}
 The rows of the four steps' working space a task transforms together, then writes down the columns of the output: 256 bytes of neighbours (four cache lines) for each of a row's elements.
 
constexpr std::size_t fft_split_least {8192}
 The fewest points of a length with a prime factor p past fft_largest_radix, not prime, that takes Bluestein's convolution for p alone, in two steps (four_step_plan: p points one way, n/p the other), n/p at least 16: fewer, or fewer than 16 transforms of p points, run the convolution over the whole length faster (ten AVX-512 cores: 1009·16 points 521 µs whole, 372 in two steps; 67·64 98 against 109; 67·1024 3.7 against 2.6 ms).
 
constexpr std::size_t fft_root_cost {100}
 A root of unity's cost in parallel_for's units, about a nanosecond each: some 95 ns in double-double (unit_roots::parts).
 
constexpr std::size_t fft_plans_kept {32}
 The plans kept, the least recently used dropped past it: a plan in use lives on in its holders.
 
constexpr std::size_t fft_scratch_kept {std::size_t {1} << 16}
 The most elements fft_scratch keeps on a thread between transforms: 1 MiB of complex double.
 
template<std::floating_point R>
constexpr std::size_t fft_lines_width {1}
 One line at a time where no register holds lines.
 
constexpr std::size_t fft_column_block {8}
 The lines a task of transform_axis takes together along an axis whose lines lie apart: each row read and written as that many neighbours, where one line at a time took one element of each row.
 
constexpr bool neon_compiled {false}
 Whether the NEON path's tile is compiled in: on every arm64 target GCC or Clang builds.
 
constexpr std::size_t gemm_mr {4}
 The rows of C one register tile covers.
 
template<typename T >
constexpr std::size_t gemm_nr {sizeof(T) >= 16 ? std::size_t {6} : std::size_t {96} / sizeof(T)}
 The columns of C one register tile covers: 96 bytes of elements (12 double, 24 float), which with gemm_mr rows fills the vector registers of both arm64 and x86-64 with AVX.
 
constexpr std::size_t gemm_kc {256}
 The depth of one panel: a B sliver (gemm_kc × gemm_nr) stays in the L1 cache.
 
constexpr std::size_t gemm_mc {64}
 The rows of one task: an A block (gemm_mc × gemm_kc) stays in the L2 cache.
 
constexpr std::size_t gemm_slivers_per_task {4}
 The B slivers of one task.
 
constexpr std::size_t tasks_per_thread {4}
 The tasks a product's compute pass hands each thread at least, while its rows allow: a slower core then holds back no one.
 
constexpr std::size_t gemm_kernel_speedup {16}
 The multiply-adds the register tile does in the time of one in work_per_thread's dependent chain (about 20 G/s on one arm64 or x86-64 AVX2 core): a task weighs its work by it, so a product of 0.1 ms does not start eight threads.
 
template<typename T >
constexpr std::size_t gemm_nc {gemm_nr<T> * 128}
 The columns of B packed at once, a whole number of slivers.
 
constexpr std::size_t gemm_packed_threshold {std::size_t {1} << 13U}
 Below this many multiply-adds the plain loop (gemm_rows, by blocks of rows held in registers) runs instead of the packed kernel, both summing in the same order. Against the packed kernel on 400 random shapes from 512 multiply-adds, the plain loop took 0.25 to 0.41 times as long below 2^13 on arm64 (M1 Pro, geometric means by octave, the slowest shape 0.91); above, it lost up to 2.5 times on shapes the AMX suits. On x86-64 the plain loop has the baseline's vectors and the packed one the AVX2 or AVX-512 units: there it took 0.43 to 0.55 times as long below 2^11, 0.86 above with shapes up to 2 times slower (six AVX2 cores, GCC 13).
 
constexpr std::size_t gemm_min_depth {8}
 Below this depth the plain loop runs too: each panel reads and writes its tiles of C once, which a shallow product does not amortise (1000×1000×1 ran six times slower packed, ×4 still lost 10 %).
 
constexpr std::size_t narrow_least_depth {2}
 The least depth at which rows of C wider than one block (narrow_columns) run by blocks in gemm_rows; one block or narrower takes any depth past 1. Against the plain loop of unknown width, on 500 random shapes below the packed kernel, the blocks took 0.11 to 0.63 times as long from a depth of 2 on arm64 (M1 Pro, geometric means by width); on x86-64, where the loop has the baseline's vectors, 0.35 to 0.90 from a depth of 8 but 1.08 to 1.31 below it past one block (six AVX2 cores, GCC 13).
 
constexpr std::size_t narrow_columns {8}
 The columns of C one block of gemm_rows keeps in registers over the whole depth.
 
constexpr std::size_t gemm_rows_columns {24}
 The most columns of C for which gemm runs any product by gemm_rows, whose blocks keep C's rows in registers over the whole depth: the packed kernel's tile spans 16 or more columns, so a narrower C left most of it idle (x86-64, 169 × 579 by 579 × 2: 82 µs, OpenBLAS 13). Both round each term as the path's tile does, so the same bits. Against the packed kernel on 200 random products of 1 to 24 columns: x86-64 0.42 to 0.64 times as long by width (ten AVX-512 cores, six AVX2 cores, GCC); arm64 under GCC 0.44 to 0.54; under Clang 0.60 up to 8 columns, while from 9 its packed tile kept pace and some products took twice as long by rows (M1 Pro).
 
constexpr std::size_t gemm_rows_work {std::size_t {1} << 20U}
 The most multiply-adds of a product gemm runs by rows (gemm_rows) whatever its width: up to there its rows' blocks, on the threads (gemm_rows_weight), finished before the packed kernel had packed and shared its work. Six AVX2 cores: order 64 14.5 → 7.2 µs, 80 20.3 → 8.9, 96 24.5 → 12.7, 40 × 60 by 60 × 100 14.2 → 7.5; arm64 under GCC order 80 32 → 15, 96 49 → 18. Under Clang on arm64 the packed tiles (the AMX's) kept small products ahead, order 64 3.3 µs against 12 by rows: none there (by_rows). Fewer rows than one of gemm_rows' blocks (narrow_group) stay packed, B then read in place (arm64 under GCC, 3 × 137 by 137 × 138: 1.48 times as long by rows). The same bits.
 
constexpr std::size_t gemm_rows_weight {2}
 The divisor of a block of gemm_rows' rows' multiply-adds as its threads are weighed: on x86-64 its work itself, so a product of order 64 takes the threads (six AVX2 cores: 11.5 → 7.4 µs; 1000 × 64 by 64 × 5 28.3 → 10.8); on arm64 half of it, where weighing the whole slowed narrow products a fifth (M1 Pro, 300 × 300 by 300 × 8 under Clang, 128 × 128 by 128 × 8 5.9 → 7.3 µs) and a quarter kept tall ones on one thread (eight threads: 4000 × 16 by 16 × 8 28 → 13 µs, 4000 × 64 by 64 × 2 58 → 26, 200 × 200 by 200 × 8 14 → 10; GCC 14 alike).
 
constexpr std::size_t narrow_group {4}
 The rows of C one block of gemm_rows takes together: each element a chain of multiply-adds that waits on its last, four rows give the units four times the chains (M1 Pro, order 8 to 24: 1.6 to 1.9 times as fast as one row; two rows on six x86-64 AVX2 cores, the fastest there, were within 7 %).
 
constexpr std::size_t gemm_rows_share {std::size_t {1} << 19U}
 The multiply-adds by rows a thread takes at most past gemm_rows_work, the blocks of rows shared out: the packed product's pass of packing and its tasks scale less well there (all threads, order 128: six AVX2 cores 31 → 24 µs, ten AVX-512 cores 32 → 21, eight arm64 cores under GCC 72 → 30; of 150 drawn products, those it moves 0.64, 0.58 and 0.38 times as long). On one thread the packed product stays, 2.3 times as fast at order 128 on AVX-512.
 
constexpr std::size_t packed_line {64}
 The cache line a packed operand starts.
 
template<typename Kernel >
constexpr bool prefetches_next_row_of_c
 Whether Kernel has the C tile of the next row of tiles prefetched while a tile computes (Kernel::prefetch_rows): the tiles within a row walk one page's lines, which the processor's own prefetch follows, while the next row lands on other pages.
 
template<typename Kernel , typename T >
constexpr bool writes_fresh
 Whether Kernel's tiles write C as it holds nothing, their sums from zero (Kernel::tile's fresh): C then not zeroed first.
 
template<typename Kernel , typename T >
constexpr bool reads_a_in_place
 Whether Kernel reads A where it lies (Kernel::tile_a_in_place).
 
template<typename Kernel >
constexpr bool reads_partial_b_in_place {requires { requires Kernel::partial_b_in_place; }}
 Whether Kernel reads B's last partial sliver in place (Kernel::partial_b_in_place): a tile that stores only its columns inside C, its loads past them landing in accumulators it never stores (the AMX's).
 
constexpr std::size_t split_columns {512}
 The columns of B and C, and the depth, of one real product of gemm_split: its real operands stay a few megabytes whatever the shapes.
 
constexpr std::size_t split_depth {256}
 The depth of one real product of gemm_split (split_columns).
 
constexpr std::size_t split_row_width {32}
 The columns split_row_columns holds at once: a block of B's rows read twice stays in the caches.
 
constexpr std::size_t split_kept {std::size_t {1} << 18}
 The most elements an operand of gemm_split keeps from one product to the next on a thread: 2 MiB of double, every operand of a product of order 512; a larger one is let go after its product.
 
constexpr std::size_t split_parts_rows {64}
 The most rows of A for which gemm_split runs four real products of a block's parts on the AMX path (split_by_parts). M1 Pro, all threads, A of 16 to 64 rows by B of 256 to 1024: complex float 1.1 to 2.4 times faster, complex double 0.97 to 1.6; 96 to 128 rows by 512, complex double 1.2 to 1.4 times slower, where one wide product suited the AMX.
 
constexpr std::size_t split_narrow_area {1024}
 The largest n·k of a complex product gemm_split shares out by blocks of rows: the real product of its expanded parts then cannot share its own threads. Past it, a product of depth 64 by 64 columns, which that product shares, ran slower so on eight GCC threads (300 rows: 86 → 124 µs).
 
constexpr std::size_t plain_loop_least_width {64}
 The fewest columns of a product plain_loop_pays sends to the plain loop on the x86-64 tiles: there an order-24 SVD, whose two products of 24 × 24 × 24 it took, ran 9 % slower (GCC 13, the same bits), though such a product alone ran faster in the plain loop. On the NEON tile the small square products gain most (GCC 14, one core: a product of order 24 to 64 −32 to −50 %, a QR −5 to −21 %).
 
constexpr std::size_t dot_chunk {std::size_t {1} << 14U}
 The elements of one partial sum of chunked_dot: fixed, so the sum does not depend on the threads.
 
template<typename T >
constexpr std::size_t multiply_add_weight {1}
 The real multiply-adds one multiply-add of T weighs: 1 for a real number.
 
constexpr std::size_t row_group {64}
 The rows a thread takes of a matrix times a vector (matrix_times_vector).
 
constexpr std::size_t matrix_vector_weight {1}
 The weight of a matrix element in matrix_times_vector's share of the threads, beside its multiply-add: each element read once, the product the memory's, so more threads read it. On x86-64, 4: six AVX2 cores, a double matrix of 1000 × 100 times a vector 19.1 → 6.6 µs, 256 × 256 10.6 → 5.2, a complex one of 671 × 36 16.4 → 6.1, 539 × 68 14.3 → 9.8, the smallest the same. On arm64, 1: the M1 Pro's complex products took longer on more threads (100 × 100 3.1 → 7.0 µs), its double ones the same.
 
constexpr std::size_t column_rows {128}
 The rows of a column-major matrix columns_times_vector takes at once: their partial sums (32 KB on any path) stay in the L1 cache, and a matrix of a few hundred rows still makes a block for each core (six AVX2 cores, 500 × 500 by a vector: 33 µs in two blocks of 256 rows, 19.6 in four of 128).
 
constexpr std::size_t ldl_block {64}
 The columns of one panel of the blocked factorisation (bunch_kaufman_blocked_lower), LAPACK's nb for sytrf.
 
constexpr std::size_t ldl_blocked_least {384}
 The order from which bunch_kaufman_lower runs by panels (bunch_kaufman_blocked_lower): below it the panels' columns, each brought up to date one dot product at a time, cost about what the unblocked steps' updates do (arm64, real: order 256 1.08× their time under Clang and 1.25× under GCC, 384 0.74× and 0.78×; complex 0.55× and 0.76× at 256).
 
constexpr double divided_exponentials_past {64.0}
 The half gap past which exact_triangular_bands divides the difference of the exponentials by the gap: one exponential then outweighs the other by e¹²⁸, and the difference loses nothing, where the product e^{half sum} sinch(half gap) overflows sinh while the exponential underflows (a gap past about 1420).
 
constexpr std::array< taylor_bound, 35 > taylor_bounds
 The table of taylor_bound, m ascending.
 
constexpr std::size_t square_root_block {64}
 The order of the diagonal blocks of triangular_square_root (SciPy's _sqrtm_triu's).
 
constexpr std::size_t upper_block_product_least {64}
 The blocks' size (rows × columns) from which solve_upper_blocks forms a block's terms by gemm: below it, a call costs more than its sums (a random matrix's clusters, most of one eigenvalue).
 
constexpr double cluster_share {0.1}
 The share of the largest modulus (or of 1) within which eigenvalues gather (cluster_bound): Davies and Higham's 0.1, relative here.
 
constexpr double cluster_share_fine {0.01}
 The finer share funm falls back to where clusters at cluster_share leave its estimate past funm_fallback_error, or where, chained into irregular shapes (a random matrix's spectrum), a cluster's circle about its centre reaches another's eigenvalues, and no count of points helps (order 192: error 1.4).
 
constexpr double funm_fallback_error {1e3 * std::numeric_limits<double>::epsilon()}
 The estimate past which funm tries the finer clusters (SciPy reports a result past it inaccurate).
 
constexpr std::size_t contour_points {64}
 The trapezoidal rule's points on each circle of Cauchy's integral.
 
constexpr std::size_t contour_points_most {4096}
 The most points on one cluster's circle: funm doubles them from contour_points while the integrals' estimate passes contour_tolerance and halves at least (the trapezoidal rule converges geometrically on a circle: a cluster of 128 eigenvalues of a random matrix took 512 to 25 ε).
 
constexpr double contour_tolerance {32.0 * std::numeric_limits<double>::epsilon()}
 The integrals' estimate below which funm takes no more points.
 
constexpr std::size_t funm_batch_elements {std::size_t {1} << 20}
 The elements of one batch of add_circle_terms's points (their p × p inverses, and as many of their shifted blocks): 2²⁰, 16 MiB each, and at least one point.
 
constexpr std::array< double, 8 > log_pade_bounds {0.0, 1.59e-5, 2.31e-3, 1.94e-2, 6.21e-2, 1.28e-1, 2.06e-1, 2.88e-1}
 The bounds θ_m of the [m/m] Padé approximant of log(I + X) (Al-Mohy and Higham's table 2.1, 2012), m from 1 to 7.
 
constexpr std::array< double, 8 > power_pade_bounds {0.0, 1.51e-5, 2.24e-3, 1.88e-2, 6.04e-2, 1.24e-1, 2.00e-1, 2.79e-1}
 The bounds θ_m of the [m/m] Padé approximant of (I − X)ᵗ (Higham and Lin's table 3.1, 2013, as SciPy's _remainder_matrix_power_triu), m from 1 to 7.
 
constexpr std::size_t no_core {std::numeric_limits<std::size_t>::max()}
 No core: a worker free to run anywhere, or a caller on a processor the pool does not know.
 
constexpr std::chrono::microseconds spin_time {500}
 How long a thread polls before it sleeps: kernels run in quick succession (a matrix product packs, then computes, panel after panel), and a thread woken from sleep starts tens of microseconds late, its wake-up paid by the caller of the next run, about 10 µs a sleeping worker on six x86-64 cores. A factorisation's steps last about 100 µs at order 256, so at 100 µs the workers one step left out were asleep for the next (an LU of order 256: six x86-64 cores 560 → 500 µs at 500; eight arm64 cores 358 → 324).
 
constexpr std::size_t work_per_thread {std::size_t {1} << 16}
 The work one thread must have to pay for starting it: about 60 µs of multiply-adds, six times the cost of a start (measured on arm64: 16 indices of 4096 ran 80 µs sequentially, 92 µs on eight threads; of 16384, 324 µs and 128 µs).
 
constexpr std::size_t elementwise_chunk {work_per_thread / 4}
 The elements one task of an element-wise operation takes, a quarter of a thread's work: a tensor of a million shares out (M1 Pro, 2000 × 2000 doubles added: 3.1 → 1.4 ms), one of a few hundred thousand stays within its time (a sixteenth: 300 × 300 plus a row 13 % slower, each task setting up its odometer).
 
constexpr std::size_t vector_speedup {4}
 The multiply-adds a vectorised inner loop does in the time of one in work_per_thread's dependent chain: a product weighs its work by it (at order 64, a matrix product on eight threads took 95 µs, on one 35 µs).
 
constexpr std::size_t memory_element_weight {4}
 The weight of an element a pass bound by the memory writes or moves, against a multiply-add, as the BLAS routines' transposing copies (weighed as one, an LU inverse of order 300 permuted its columns on one thread).
 
constexpr std::size_t zero_chunk {std::size_t {1} << 12}
 The elements one index of parallel_zero writes: eight 4 KiB pages of double.
 
constexpr std::size_t zero_work {1}
 The work of zeroing one element, in multiply-adds: a second thread from 1 MiB of double. Below, the allocator mostly hands back memory already touched, and zeroing 512 KiB took 4.5 µs on one arm64 thread, 9 µs split across eight.
 
constexpr std::size_t transpose_tile {32}
 The side of the tiles a transposing copy goes by (an einsum permutation of two axes, the eigenvectors' columns): a tile read and a tile written fit the L1 cache, where element by element each write, a row apart, missed it (order 1024 on arm64: 4.7 → 1.5 ms; the eigenvectors of order 1024, a tenth of an eigh on six x86-64 cores).
 
constexpr std::size_t symmetric_row_group {4}
 The rows of a lower triangle symmetric_rows takes at once.
 
constexpr std::size_t product_rows {8}
 The rows of y subtract_rows takes at once.
 
constexpr double jacobi_tiny {std::numeric_limits<double>::min() / std::numeric_limits<double>::epsilon()}
 The sum of squares under which one-sided Jacobi judges a pair of columns over their norms: below it a square has lost digits to underflow (DBL_MIN / ε).
 
constexpr double jacobi_large {1e150}
 The |ζ| past which one-sided Jacobi takes the rotation's tangent as 1 / (2|ζ|): ζ² stays finite under it.
 
constexpr double cancelled {64.0}
 The fall of a column's norm, in units of √n ε, past which one_sided_jacobi takes what a rotation left of it for rounding alone.
 
constexpr std::size_t svd_qr_steps_per_value {30}
 The QR steps the bidiagonal singular value iteration may take per value: it converges in two or three; the bound turns a non-convergence into an error instead of a hang.
 
constexpr std::size_t bidiagonal_crossover {128}
 The order below which a real bidiagonal reduction goes a reflection at a time, as LAPACK's dgebrd falls back to dgebd2: the panels' products are too small to pay for packing. Order 64: 33 % shorter on six x86-64 cores, 7 % on arm64; order 160: 30 % longer on arm64, where x86-64 still gains 8 %.
 
constexpr std::size_t bidiagonal_basis_crossover {64}
 The order below which the bases of a bidiagonal reduction are formed a reflector at a time: the blocks' packing and triangular factor cost more than they save. Order 32: 11 % shorter on six x86-64 cores, 12 % on arm64; order 96: 7 % longer on both.
 
constexpr std::size_t bidiagonal_block {32}
 The rows from which a step of the blocked bidiagonal reduction shares its coefficients and corrections with the threads: on x86-64 (six AVX2 cores, the reduction of order 512 −5 to −14 %, 1024 −12 %); never on arm64, where its serial part is smaller and the shared step gained nothing (M1 Pro, eight cores, order 1024 0.97 to 1.05 times as long).
 
constexpr std::size_t bidiagonal_product_weight {1}
 The weight of a multiply-add of the bidiagonal reduction's matrix-vector products, which read the trailing block once for each column: on x86-64 a thread shares them from a quarter of the work (six cores, singular values of order 256: 8.1 → 7.3 ms, 512: 35 → 30); on arm64 that slowed order 256 by 10 %.
 
constexpr std::size_t sort_rows {64}
 The rows a task of merge_arrow reorders in place, one row's buffer between them.
 
constexpr std::size_t bidiagonal_leaf {32}
 The order up to which bidiagonal_divide decomposes by implicit-shift QR.
 
constexpr double qr_rank_margin {100.0}
 How far above the rank cutoff a bound on the smallest singular value must stay for lstsq_by_qr to answer: nearer, the singular value decomposition decides the rank.
 
constexpr std::size_t sturm_bisection_work {256}
 One eigenvalue's bisection in parallel_for's units per order of the matrix: some 60 halvings, each a Sturm count whose step divides (some four multiply-adds).
 
constexpr std::size_t inverse_iteration_cluster_most {64}
 The most eigenvalues tridiagonal_eigenvectors reorthogonalises together: a larger cluster's vectors come from the whole spectrum's divide and conquer, n k² reorthogonalisation passing its n³'s share.
 
constexpr std::size_t pairwise_lanes {8}
 The partial sums pairwise_sum interleaves, and the multiple of which its first half is long.
 
constexpr std::size_t pairwise_leaf {std::size_t {1} << 15U}
 The longest subtree of pairwise_sum's tree parallel_pairwise_sum sums on one thread: 256 KiB of double.
 
constexpr std::size_t pairwise_columns {512}
 The columns pairwise_sum_rows sums at once: its partial sums of eight rows, 32 KiB of doubles, stay in the L1 cache.
 
constexpr std::size_t vector_chunk {128}
 The columns a thread takes of a vector times a matrix: each of its rows added to them in order.
 
template<typename T >
constexpr std::size_t term_work {is_complex_number<T> ? 4 : 1}
 The real multiply-adds of a term a · b of T, as parallel_for counts work: four for a complex one.
 
constexpr std::size_t strided_columns {8}
 The columns of B one task of vector_times_columns sums together: their sums' chains interleaved, each waiting on its own last addition.
 
constexpr std::size_t complex_row_columns {32}
 The columns of a task of complex_vector_times_columns divide: the most its kernels hold in registers at once (sixteen NEON registers of complex float).
 
constexpr std::size_t column_sums_below {9}
 The columns under which canonical_product sums a vector times a matrix by vector_times_columns, the same bits as its rows added in order: below it faster for every type from a depth of a hundred (a few nanoseconds either way under it), from it slower for a real type. On x86-64 the rows' vectors of eight float (AVX2) make the rows' loop the faster sooner.
 
constexpr std::size_t exhaustive_path_operands {4}
 The operands at most whose contraction order contraction_path searches exhaustively; more take the cheapest pair at each step.
 
constexpr double exp2_normal_bound {-1022.0}
 2^x below which exp2 is subnormal, where table_exp2's two scalings would round twice.
 
constexpr std::size_t band_columns {256}
 The columns of a band that one task of fold_along and scan_along walks row by row.
 
constexpr std::size_t radix_sort_least {256}
 Runs this long or longer are sorted by their bits (radix_sort); shorter ones by comparisons.
 
template<typename U >
constexpr U sign_bit {U {1} << ((sizeof(U) * 8) - 1)}
 The highest bit of U, a real's or an integer's sign.
 
constexpr std::size_t select_positions_sorted {8}
 A line with more positions than these is sorted whole rather than selected position by position.
 
constexpr std::size_t correlation_block {16}
 The lags numerical::correlate sums together: a block's sums held in registers (four AVX2 vectors of doubles) through the terms of v.
 

Typedef Documentation

◆ byte_counts

template<typename U >
using numerical::detail::byte_counts = typedef std::array<std::array<std::size_t, 256>, sizeof(U)>

How many keys of U hold each value of each byte: the byte at bit 8·b counted in [b].

Definition at line 3545 of file tensor_ops.hpp.

◆ carried_t

template<typename T >
using numerical::detail::carried_t = typedef std::conditional_t<std::is_same_v<magnitude_t<T>, long double>, long double, double>

The real type a T's norms, reflector lengths and pivot magnitudes are carried in: long double for a long double scalar, real or complex, whose precision a double would cut to its own; double otherwise.

Definition at line 350 of file factorizations.hpp.

◆ chunk_runner

using numerical::detail::chunk_runner = typedef void (*)(const void* body, std::size_t begin, std::size_t end)

A chunk of a parallel_for, type-erased: runs body(i) for i in [begin, end).

Definition at line 330 of file parallel.hpp.

◆ doubles_into

A function of count contiguous doubles into as many (the *_into kernels).

Definition at line 2351 of file tensor_ops.hpp.

◆ eigenvalue_selection

using numerical::detail::eigenvalue_selection = typedef std::function<bool (const std::complex<double>&)>
private

A choice among eigenvalues, as scipy.linalg.schur's sort makes it: true where one is to lead.

Definition at line 3407 of file eigen.hpp.

◆ fft_real_t

The part type of a complex number (R), or of a register of lines' (fft_pack).

Definition at line 1150 of file fft.hpp.

◆ fft_root_t

The complex number of C's part type: a pass' factors.

Definition at line 1154 of file fft.hpp.

◆ gemm_tile

Definition at line 918 of file gemm.hpp.

◆ kept_page_blocks

The blocks of 4 KiB pages of default_init_allocator, from kept_pages_least bytes: four, each under kept_pages_largest.

Definition at line 229 of file buffer.hpp.

◆ lapack_index

A signed index from 1, as LAPACK's: the multishift sweep starts its bulges in phantom columns above the matrix, before its first row.

Definition at line 1248 of file eigen.hpp.

◆ magnitude_t

template<typename T >
using numerical::detail::magnitude_t = typedef decltype(std::abs(std::declval<const T&>()))

The type std::abs gives for a T: the real type of a complex one, T itself otherwise.

Definition at line 330 of file factorizations.hpp.

◆ order_key_t

template<typename T >
using numerical::detail::order_key_t = typedef std::conditional_t<sizeof(T) == 8, std::uint64_t, std::uint32_t>

The unsigned integer of T's width, whose order order_key maps T's onto.

Definition at line 3504 of file tensor_ops.hpp.

◆ qd_index

An index into the qd array, one-based and signed as LAPACK's.

Definition at line 23 of file dqds.hpp.

◆ riccati_terms

The operands of an algebraic Riccati equation as complex matrices (the residuals and Newton's step).

Definition at line 2762 of file generalized.hpp.

◆ rounding_is

template<term_rounding Mode>
using numerical::detail::rounding_is = typedef std::integral_constant<term_rounding, Mode>

The rounding mode Mode as a type, which picks rounded_term's overload.

Definition at line 1787 of file gemm.hpp.

◆ side_runner

A task the caller of a run does before it takes chunks (parallel_for_beside), type-erased.

Definition at line 333 of file parallel.hpp.

◆ symmetric_row_kernel

A kernel of symmetric_row's form: symmetric_row itself, or an x86-64 one.

Definition at line 4365 of file factorizations.hpp.

◆ symmetric_rows_kernel

A kernel of symmetric_rows' form: symmetric_rows itself, or an x86-64 one.

Definition at line 4368 of file factorizations.hpp.

◆ value_range_t

template<typename T >
using numerical::detail::value_range_t = typedef std::conditional_t<std::is_same_v<T, float> || std::is_same_v<T, std::complex<float> >, float, carried_t<T> >

The real type whose range a T's values take: float for a single-precision scalar, real or complex; carried_t otherwise.

Definition at line 377 of file factorizations.hpp.

Enumeration Type Documentation

◆ c_start

What a product finds in C: the sums it adds to (C += A·B), or nothing it reads (C = A·B). Then the blocks by rows start their sums at zero and the packed tasks zero their block of C before its first panel, on the thread that computes it, where the caller's zeroing had the threads take C's lines from its cache (six AVX2 cores, all threads: order 64 7.1 → 5.2 µs, 96 12.7 → 9.4, 128 32.7 → 28.8; one: 32 1.62 → 1.47).

Enumerator
kept 

C += A·B.

zero 

C = A·B: C's elements before the product are not read.

Definition at line 1813 of file gemm.hpp.

◆ complex_term

How a complex product rounds a term that sums nothing (0 + a·b) on the path in use, as gemm_large_complex runs it.

Enumerator
compiler 

The vector tile's and the plain loop's: 0 + a·b as the compiler rounds it (rounded_apart_add).

tiles 

The x86-64 complex tiles': each part's product from zero, re·re − im·im and re·im + im·re, then added to 0.

split 

arm64's real kernel's (gemm_split): Re a·Re b from zero, −Im a·Im b fused into it; Re a·Im b, Im a·Re b.

Definition at line 2310 of file gemm.hpp.

◆ left_kernel

The tile a product's left operand is packed for (numerical::detail::pack_left): plain where it is not packed.

Enumerator
plain 

gemm_rows, or a scalar whose products are not shared: A is read in place.

vector 

The portable path's vector tile.

neon 

The NEON tile.

amx 

The AMX tile.

avx512 

The AVX-512 tile.

avx2 

The AVX2 tile.

Definition at line 4818 of file gemm.hpp.

◆ term_rounding

How numerical::detail::gemm_rows rounds each term sum + a·b, as the tile of the path in use does: as the compiler rounds it (the vector tile's way), one fused multiply-add (arm64's NEON and AMX tiles), or one fused multiply-add of AVX2's unit (the x86-64 tiles), in code built for any x86-64 processor.

Enumerator
compiler 

As the compiler rounds sum + a·b (rounded_apart_add).

fused 

std::fma, where every target has the instruction.

x86_fused 

std::fma in functions built for AVX2 with FMA.

Definition at line 1778 of file gemm.hpp.

◆ trig_family

The family of a real trigonometric transform: dct's or dst's.

Enumerator
cosine 
sine 

Definition at line 3222 of file fft.hpp.

Function Documentation

◆ add_circle_terms()

void numerical::detail::add_circle_terms ( const matrix< std::complex< double > > &  t,
std::size_t  first,
std::size_t  p,
std::complex< double >  centre,
double  radius,
std::size_t  count,
std::size_t  from,
std::size_t  step,
const std::complex< double > *  images,
buffer< std::complex< double > > &  sum,
buffer< double > &  moduli 
)
inline

Adds to sum, for the points k = from, from + step, … < count of the grid of count points on the circle (centre, radius), f(z_k) (z_k − c) (z_k I − T_b)⁻¹, T_b the p × p block of t from first and f(z_k) the next of images in turn: the trapezoidal rule's terms of Cauchy's integral (1/2πi) ∮ f(z) (zI − T_b)⁻¹ dz (Hale, Higham and Trefethen, 2008), times the count.

Definition at line 2128 of file matrix_functions.hpp.

◆ add_column_term() [1/2]

template<typename T >
void numerical::detail::add_column_term ( std::false_type  ,
T &  sum,
const T &  f,
const T &  x 
)

sum += f · x of strided_column_group, as gemm_rows' plain loop rounds a term where it fuses none (rounded_apart_add, a complex product's infinities recovered as the library's).

Definition at line 1284 of file tensor.hpp.

◆ add_column_term() [2/2]

template<typename T >
void numerical::detail::add_column_term ( std::true_type  ,
T &  sum,
const T &  f,
const T &  x 
)

sum += f · x of strided_column_group, one fused multiply-add.

Definition at line 1273 of file tensor.hpp.

◆ add_conjugate_product() [1/2]

template<typename T >
requires std::is_floating_point_v<T>
void numerical::detail::add_conjugate_product ( T *  y,
const T *  x,
const T &  f,
std::size_t  count 
)

y[i] += conj(x[i]) · f for a real T, where the conjugate is the value: add_product.

Definition at line 4328 of file factorizations.hpp.

◆ add_conjugate_product() [2/2]

template<typename T >
void numerical::detail::add_conjugate_product ( T *  y,
const T *  x,
const T &  f,
std::size_t  count 
)

y[i] += conj(x[i]) · f, each product's statements spelled out as the complex operator's (four part products rounded apart, ac − bd and ad + bc, then their sums with y): the same bits for finite parts whose products stay finite, where the operator's rescue of a product whose parts are both NaN never acts, and a loop the compiler vectorises (the reduction's rows, scaled by their largest magnitude, are such).

Definition at line 4343 of file factorizations.hpp.

◆ add_one_product() [1/2]

template<split_complex C>
void numerical::detail::add_one_product ( C &  y,
const C &  f,
const C &  x 
)

add_one_product of complex numbers: complex_scaled_add, as add_product computes them.

Definition at line 1352 of file simd.hpp.

◆ add_one_product() [2/2]

template<typename T >
void numerical::detail::add_one_product ( T &  y,
const T &  f,
const T &  x 
)

y += f · x of one element, as add_product computes each of its elements.

Definition at line 1343 of file simd.hpp.

◆ add_product() [1/5]

template<split_complex C>
void numerical::detail::add_product ( C *  y,
const C &  f,
const C *  x,
std::size_t  count 
)

add_product of complex numbers: vectors of their parts, then complex_scaled_add, every element the same arithmetic.

Definition at line 1330 of file simd.hpp.

◆ add_product() [2/5]

void numerical::detail::add_product ( double *  y,
const double &  f,
const double *  x,
std::size_t  count 
)
inline

y[i] += f · x[i] for double, as subtract_product.

Definition at line 648 of file gemm.hpp.

◆ add_product() [3/5]

void numerical::detail::add_product ( std::complex< double > *  y,
const std::complex< double > &  f,
const std::complex< double > *  x,
std::size_t  count 
)
inline

add_product for complex double (avx2_scaled_add_c64 on the x86-64 paths).

Definition at line 695 of file gemm.hpp.

◆ add_product() [4/5]

void numerical::detail::add_product ( std::complex< float > *  y,
const std::complex< float > &  f,
const std::complex< float > *  x,
std::size_t  count 
)
inline

y[i] += f · x[i] for complex float: on the x86-64 paths by AVX2 vectors of its arithmetic (avx2_scaled_add_c32), the same bits as add_product's template, which runs elsewhere.

Definition at line 676 of file gemm.hpp.

◆ add_product() [5/5]

template<typename T >
void numerical::detail::add_product ( T *  y,
const T &  f,
const T *  x,
std::size_t  count 
)

y[i] += f · x[i], as subtract_product.

Definition at line 1361 of file simd.hpp.

◆ add_row_terms() [1/2]

template<split_complex C>
void numerical::detail::add_row_terms ( C *  y,
const C *  a,
const C *  b,
std::size_t  ldb,
std::size_t  inner,
std::size_t  count 
)

add_row_terms of complex numbers as gemm_rows' plain loop rounds them (rounded_apart_add): a row by add_product's vectors where its factor and itself are finite, whose products are then the library's (both of a product's parts are NaN only from an infinite or NaN operand); else term by term, the library recovering a product of infinities.

Definition at line 1594 of file tensor.hpp.

◆ add_row_terms() [2/2]

template<typename T >
void numerical::detail::add_row_terms ( T *  y,
const T *  a,
const T *  b,
std::size_t  ldb,
std::size_t  inner,
std::size_t  count 
)

y[j] += Σₖ a[k] · b[(k · ldb) + j] for count columns, B's inner rows added in order to a vector times a matrix by add_product, an integer's exact.

add_row_terms of a floating-point T: each term as gemm_rows rounds it (rows_terms_fuse), so a vector times a matrix has the bits of its row in a product of a matrix by it; the path read once.

Definition at line 1488 of file tensor.hpp.

◆ add_scaled()

template<typename T >
void numerical::detail::add_scaled ( buffer< T > &  into,
const buffer< T > &  term,
double  coefficient 
)

into += coefficient · term over the n × n buffers.

Definition at line 962 of file matrix_functions.hpp.

◆ add_scaled_vectors()

template<typename T >
std::size_t numerical::detail::add_scaled_vectors ( T *  ,
const T *  ,
const T &  ,
std::size_t   
)

No leading elements in vectors, for a type no native vector holds.

Definition at line 1038 of file simd.hpp.

◆ adjoint()

template<typename T >
matrix< T > numerical::detail::adjoint ( const matrix< T > &  a)

The conjugate transpose of a, as numerical::transpose copies (by blocks, on the threads).

Definition at line 860 of file svd.hpp.

◆ adjoint_dot() [1/2]

double numerical::detail::adjoint_dot ( const double *  x,
const double *  y,
std::size_t  count 
)
inlineprivate

Σ x[i] y[i], vectorised (dot_product): xᴴ y for real numbers.

Definition at line 3971 of file factorizations.hpp.

◆ adjoint_dot() [2/2]

std::complex< double > numerical::detail::adjoint_dot ( const std::complex< double > *  x,
const std::complex< double > *  y,
std::size_t  count 
)
inlineprivate

Σ conj(x[i]) y[i], vectorised (conjugate_dot_product): xᴴ y.

Definition at line 3979 of file factorizations.hpp.

◆ adjoint_of()

template<typename T >
matrix< T > numerical::detail::adjoint_of ( const matrix< T > &  m)

Mᴴ: the transpose, conjugated.

Definition at line 699 of file matrix_functions.hpp.

◆ after_step()

std::vector< std::string > numerical::detail::after_step ( const std::vector< std::string > &  labels,
contraction_step  step,
const std::string &  output 
)
inline

The list after a contraction step: the two operands' labels removed, their result's appended.

Definition at line 2108 of file tensor.hpp.

◆ all_finite()

template<typename R >
requires std::is_same_v<R, double> || std::is_same_v<R, float>
bool numerical::detail::all_finite ( const R *  x,
std::size_t  count 
)
noexcept

Whether the count values at x are all finite: a NaN or an infinity has every exponent bit set, the one exponent that one more carries into the sign bit. One pass with no early exit and no comparison, an integer AND, add and OR (a check per element and an exit at the first was a tenth of an LU of order 128; the comparison per element, one M1 Pro core: 6.4 → 8.5 values a nanosecond). Under GCC and Clang four vectors of 16 bytes a step, whatever the compiler would vectorise: GCC at -O2 (the arm64 wheels') vectorised the plain loop not at all, at -O3 by one vector (1089 doubles, one arm64 core: GCC -O2 541 → 132 ns, -O3 299 → 132, Clang level at 132; one x86-64 core, GCC 13 -O3 271 → 163 ns, Clang 18 level); a Cholesky factorisation of order 33 lost a fifth of its time to the plain loop under GCC -O2.

Definition at line 1392 of file simd.hpp.

◆ all_finite_shared()

bool numerical::detail::all_finite_shared ( const double *  x,
std::size_t  count 
)
inline

all_finite on the threads by chunks of finite_chunk, past two of them: a pass bound by the memory, an element read weighed as half one copied (six x86-64 AVX2 cores, a Cholesky factorisation of order 500 with its other passes on the threads too: 1.12 → 0.91 ms).

Definition at line 661 of file factorizations.hpp.

◆ already_reduced() [1/2]

bool numerical::detail::already_reduced ( const std::complex< double > &  alpha,
double  rest 
)
inline

The same for a complex one: its tail zero and α real.

Definition at line 338 of file eigen.hpp.

◆ already_reduced() [2/2]

bool numerical::detail::already_reduced ( double  ,
double  rest 
)
inline

Whether (α, x), of tail norm rest, is already (β, 0) with β real: for a real one, its tail zero.

Definition at line 331 of file eigen.hpp.

◆ amx_usable()

bool numerical::detail::amx_usable ( )
inlinenoexcept

Whether this chip's AMX can be used: known family and a passing self-test, decided once.

Definition at line 964 of file amx.hpp.

◆ answer_like() [1/2]

matrix< double > numerical::detail::answer_like ( const matrix< std::complex< double > > &  x,
const matrix< double > &   
)
inline

x's real part, for real operands like: the solution of a real equation is real.

Definition at line 3072 of file generalized.hpp.

◆ answer_like() [2/2]

matrix< std::complex< double > > numerical::detail::answer_like ( matrix< std::complex< double > > &&  x,
const matrix< std::complex< double > > &   
)
inline

x itself, for complex operands like.

Definition at line 3065 of file generalized.hpp.

◆ any_radix_pass()

template<bool Fused, std::floating_point R>
void numerical::detail::any_radix_pass ( std::size_t  radix,
std::size_t  current,
std::size_t  stride,
const std::complex< R > *  in,
std::complex< R > *  out,
const std::complex< R > *  twiddles,
const std::complex< R > *  roots,
const small_radix_constants< R > &  small 
)
noexcept

any_radix_pass_body, its products as Fused.

Definition at line 1536 of file fft.hpp.

◆ any_radix_pass_body()

template<bool Fused, typename C >
void numerical::detail::any_radix_pass_body ( std::size_t  radix,
std::size_t  current,
std::size_t  stride,
const C *  in,
C *  out,
const fft_root_t< C > *  twiddles,
const fft_root_t< C > *  roots 
)
inlinenoexcept

One Stockham pass of an odd radix radix, as radix_four_pass is of 4, by the symmetry of its roots (radix_five_pass_body's, any odd r): a_j = x_j + x_{r−j}, b_j = x_j − x_{r−j} (j from 1 to h = (r − 1)/2); X_0 = x_0 + a_1 + … + a_h, and for k from 1 to h, A_k = x_0 + Σ_j cos(2πjk/r)·a_j, B_k = Σ_j sin(2πjk/r)·b_j, each sum in ascending j, its products fused as Fused (scaled_add; the cosines and sines roots' parts, jk reduced mod r by additions), X_k = A_k − i·B_k, X_{r−k} = A_k + i·B_k; output k then times w^{kp}, none for p = 0. Radices 3 and 5 so computed give their own butterflies' bits. Inlined into each caller, so that the x86-64 fused one is built for FMA.

Definition at line 1351 of file fft.hpp.

◆ apply_q()

template<typename T >
void numerical::detail::apply_q ( buffer< T > &  p,
const std::vector< reflector_block< T > > &  blocks,
std::size_t  rows,
std::size_t  count 
)
private

P ← Q P for the rows × count row-major p, Q = H₀ H₁ … the product of the blocks' reflectors, the last applied first: a basis taken through Q without forming Q, as LAPACK's dormqr applies it where dorgqr forms it. As form_q_into applies the blocks: on the x86-64 units to the rows of Pᵀ (apply_transposed_reflector_block), transposed in and out, elsewhere in place (apply_reflector_block).

Definition at line 3601 of file factorizations.hpp.

◆ apply_reflector_block()

template<typename T >
void numerical::detail::apply_reflector_block ( const reflector_block< T > &  block,
bool  adjoint,
buffer< T > &  a,
std::size_t  cols,
std::size_t  from,
std::size_t  to 
)
private

Applies I − V T Vᴴ (or its adjoint I − V Tᴴ Vᴴ) from the left to the block's rows of a row-major array, on its columns from…to-1: W = Vᴴ A, then W ← T W or Tᴴ W, then A −= V W, the two outer products through the matrix product kernel.

Definition at line 3112 of file factorizations.hpp.

◆ apply_row_operations()

template<typename T >
void numerical::detail::apply_row_operations ( buffer< T > &  m,
std::size_t  n,
const std::vector< row_operation > &  operations 
)

Applies the recorded operations, in order, to the row-major m (n per row). Each column takes them independently of the others, so chunks of columns run in parallel, every element receiving the same operations in the same order as applying each in place would give it.

A run of chained rotations goes as LAPACK's dlasr: the row each rotation hands the next stays in a buffer the size of the chunk, so every row of the run is read and written once rather than twice per rotation.

Definition at line 5059 of file factorizations.hpp.

◆ apply_transposed_reflector_block()

template<typename T >
void numerical::detail::apply_transposed_reflector_block ( const reflector_block< T > &  block,
buffer< T > &  at,
std::size_t  cols,
std::size_t  from,
std::size_t  to,
bool  adjoint = false 
)
private

apply_reflector_block to A held transposed: at is Aᵀ (row-major, cols columns), and A ← (I − V T Vᴴ) A on A's rows first… and columns from…to-1 is Aᵀ ← Aᵀ (I − conj(V) Tᵀ Vᵀ) on Aᵀ's columns first… and rows from…to-1 (with the adjoint (I − V Tᴴ Vᴴ) A, conj(T) for Tᵀ). Its three products have to − from rows, which the matrix product kernel runs well and on every core; apply_reflector_block's first has the block's few (a reflector block's 32 rows by 1000 deep: 54 GFLOP/s on twenty x86-64 threads).

Definition at line 3155 of file factorizations.hpp.

◆ argsort_key() [1/2]

order_key_t< std::int64_t > numerical::detail::argsort_key ( std::int64_t  x)
inlinenoexcept

An integer's order_key for an argsort.

Definition at line 3718 of file tensor_ops.hpp.

◆ argsort_key() [2/2]

template<std::floating_point T>
order_key_t< T > numerical::detail::argsort_key ( T  x)
noexcept

A real's order_key for an argsort: both zeros the same key, as < holds them equal (their indices then order them).

Definition at line 3712 of file tensor_ops.hpp.

◆ as_complex() [1/2]

matrix< std::complex< double > > numerical::detail::as_complex ( const matrix< double > &  a)
inlineprivate

a's entries as complex numbers.

Definition at line 3061 of file matrix_functions.hpp.

◆ as_complex() [2/2]

const matrix< std::complex< double > > & numerical::detail::as_complex ( const matrix< std::complex< double > > &  a)
inlineprivate

a itself: already complex.

Definition at line 3055 of file matrix_functions.hpp.

◆ as_laid_along()

template<typename T , typename F >
auto numerical::detail::as_laid_along ( const tensor< T > &  t,
std::size_t  axis,
const F &  f 
) -> std::optional<decltype(f(t, axis))>

Along axis of a transpose (laid_along): f (f(elements, axis)) of its elements as they lie along the matching axis, none copied, the result (of the same axes) put back in the view's order. Each line the same elements in the same order: what is computed line by line, the same. Nothing for any other tensor, an axis past the rank, or the view's last axis where its elements lie apart (a transpose's columns): copied out as rows, those lines are sorted, scanned and transformed faster than in place as columns (2000 of 2000 reals sorted, 8.0 ms against 12.4, M1 Pro), where along any other axis they are columns either way.

Definition at line 1986 of file tensor_ops.hpp.

◆ as_laid_reduced()

template<typename T , typename F >
auto numerical::detail::as_laid_reduced ( const tensor< T > &  t,
std::size_t  axis,
const F &  f 
) -> std::optional<decltype(f(t, axis))>

as_laid_along of an f whose result takes the axis out, along any axis: a reduction of columns runs across them as fast as along rows (2000 sums of 2000, 0.74 ms against 1.19 copied out first, M1 Pro).

Definition at line 2001 of file tensor_ops.hpp.

◆ as_scalar() [1/2]

std::complex< double > numerical::detail::as_scalar ( const std::complex< double > &  s,
const std::complex< double > &   
)
inline

A complex matrix's shift as it is.

Definition at line 1857 of file eigen.hpp.

◆ as_scalar() [2/2]

double numerical::detail::as_scalar ( const std::complex< double > &  s,
double   
)
inline

The shift s as the matrix's scalar: a real matrix's are real where they are used one by one.

Definition at line 1850 of file eigen.hpp.

◆ ascending_order()

std::vector< std::size_t > numerical::detail::ascending_order ( const buffer< double > &  diagonal)
inline

The indices of diagonal by ascending value.

Definition at line 6339 of file factorizations.hpp.

◆ assembled_function()

std::tuple< buffer< std::complex< double > >, double, double > numerical::detail::assembled_function ( const matrix< std::complex< double > > &  t,
const cluster_circles &  circles,
const std::vector< buffer< std::complex< double > > > &  blocks,
const std::vector< buffer< std::complex< double > > > &  errors,
const std::vector< std::complex< double > > &  at_eigenvalues 
)
inline

The blocks of x above its diagonal (n × n, row-major, its diagonal blocks given) by the block recurrence of Davies and Higham's Schur–Parlett algorithm, a block superdiagonal at a time: T_ii X_ij − X_ij T_jj = X_ii T_ij − T_ij X_jj + Σ (X_ik T_kj − T_ik X_kj) + S_ij, a triangular Sylvester equation each, the clusters as runs of t's diagonal (starts, each run's first place, then n), and S sources (n × n), or none where null.

f(T) of the Schur form t with clusters as runs of its diagonal (circles), each cluster's block given in blocks (its circle's integral, the clusters of one eigenvalue none) with its estimate in contours (in the circles' order), and f at the eigenvalues in at_eigenvalues: Davies and Higham's Schur–Parlett algorithm (2003), the blocks above the diagonal by the block recurrence (solve_upper_blocks).

Returns
f(T), its estimate (recurrence_error, or SciPy's where larger), and the recurrence's alone.

Definition at line 2261 of file matrix_functions.hpp.

◆ atan_into()

void numerical::detail::atan_into ( const double *  in,
double *  out,
std::size_t  count 
)
inline

atan of the count doubles at in into out, each elementary_atan's: table_atan in vectors (with the FMA instructions on an x86-64 processor, run_fused), std::atan throughout where exp_by_table says no.

Definition at line 518 of file tensor_ops.hpp.

◆ atan_of() [1/3]

tensor< double > numerical::detail::atan_of ( const double *  data,
const std::vector< std::size_t > &  shape 
)
inline

atan of every one of the row-major doubles at data, of shape shape: atan_into by chunks of elementwise_chunk among the threads.

Definition at line 555 of file tensor_ops.hpp.

◆ atan_of() [2/3]

tensor< float > numerical::detail::atan_of ( const float *  data,
const std::vector< std::size_t > &  shape 
)
inline

atan of every one of the row-major floats at data, of shape shape: atan_into of them as doubles (through_doubles).

Definition at line 576 of file tensor_ops.hpp.

◆ atan_of() [3/3]

template<typename T >
tensor< T > numerical::detail::atan_of ( const T *  data,
const std::vector< std::size_t > &  shape 
)

atan of every one of the row-major scalars at data, of shape shape: std::atan of each.

Definition at line 547 of file tensor_ops.hpp.

◆ atan_viewed() [1/3]

tensor< double > numerical::detail::atan_viewed ( const tensor< double > &  a)
inline

atan of every double of a, a view read where it lies (by_runs).

Definition at line 2510 of file tensor_ops.hpp.

◆ atan_viewed() [2/3]

tensor< float > numerical::detail::atan_viewed ( const tensor< float > &  a)
inline

atan of every float of a through doubles, a view read where it lies (by_runs).

Definition at line 2516 of file tensor_ops.hpp.

◆ atan_viewed() [3/3]

template<typename T >
tensor< T > numerical::detail::atan_viewed ( const tensor< T > &  a)

atan of every element of a: of a contiguous copy (atan_of).

Definition at line 2504 of file tensor_ops.hpp.

◆ back_transformed()

template<typename T >
buffer< T > numerical::detail::back_transformed ( const buffer< T > &  xt,
const buffer< T > &  zt,
std::size_t  n 
)

Vᵀ = Xᵀ Zᵀ, row-major n × n, through the matrix product kernel.

Definition at line 3018 of file eigen.hpp.

◆ balance()

template<typename T >
balancing numerical::detail::balance ( buffer< T > &  a,
std::size_t  n,
bool  permute = true,
bool  scale = true 
)

Balances the row-major n × n a in place, as LAPACK's ‘dgebal('B’): rows with no off-diagonal entry in the active block go to its bottom and columns with none to its top, each isolating an eigenvalue on the diagonal; then each row and column of the remaining block is scaled by a power of two until their 2-norms are within a factor of 2, which leaves every entry exact. Without \p permute no row moves (the block is the whole matrix), without \p scale none is scaled:dgebal's'S'and'P'`.

Definition at line 130 of file eigen.hpp.

◆ balance_hamiltonian()

template<typename T >
buffer< double > numerical::detail::balance_hamiltonian ( buffer< T > &  h,
std::size_t  m 
)

The diagonal D of a symplectic balancing of the Hamiltonian h of order 2m (row-major, in place): h ← T⁻¹ h T, T = diag(D, D⁻¹), powers of 2 (each scaling exact). Each dᵢ scales column i and row m + i by f (their common entry by f²), row i and column m + i by 1/f (theirs by 1/f²): the power of 2 whose off-diagonal sum g₁ f + g₂ f² + s₁/f + s₂/f² is least (a convex function of log f), taken where it lowers the sum by a twentieth, swept until none does: each step lowers the matrix's off-diagonal sum, so the sweeps end.

Definition at line 3103 of file generalized.hpp.

◆ balance_pencil()

template<typename T >
buffer< double > numerical::detail::balance_pencil ( buffer< T > &  h,
buffer< T > &  j,
std::size_t  m,
std::size_t  order 
)
private

Balances the pencil (H, J) of order 2m + n as SciPy does: ‘dgebal('S’)scaling of|H| + |J|off the diagonal, the first two blocks ofmthen givenDandD⁻¹from half the difference of their exponents (Benner's symplectic constraint), the pencil scaled toD⁻¹ (H, J) D. \return The scalesD`, all 1 when the scaling found none.

Definition at line 2858 of file generalized.hpp.

◆ balanced_index()

std::size_t numerical::detail::balanced_index ( std::size_t  task,
std::size_t  n 
)
inlinenoexcept

The eigenvector task of n solves, the short ones and the long ones in turn (0, n − 1, 1, n − 2, …): each of parallel_for's runs of tasks then as long as another.

Definition at line 2938 of file eigen.hpp.

◆ band_cholesky()

template<typename T >
void numerical::detail::band_cholesky ( band_storage< T > &  ab,
std::size_t  n,
std::size_t  kd,
const std::string &  name 
)

The Cholesky factor U (A = Uᴴ U) of the Hermitian positive-definite band of kd superdiagonals in ab (kd + 1 rows, AB(kd + 1 + i − j, j) = A(i, j)), in place, as LAPACK's pbtf2 (upper): each diagonal entry's square root, the row right of it scaled, the trailing band less its outer product.

Exceptions
not_positive_definite_errorIf a diagonal entry is not positive when reached.

Definition at line 180 of file banded.hpp.

◆ band_cholesky_solve()

template<typename T >
void numerical::detail::band_cholesky_solve ( band_storage< T > &  ab,
std::size_t  n,
std::size_t  kd,
buffer< T > &  b,
std::size_t  k 
)

The k right-hand sides of b (n × k, row-major) solved in place against band_cholesky's U, as LAPACK's pbtrs: Uᴴ y = b forward, then U x = y back.

Definition at line 210 of file banded.hpp.

◆ band_lu()

template<typename T >
void numerical::detail::band_lu ( band_storage< T > &  ab,
std::size_t  n,
std::size_t  kl,
std::size_t  ku,
std::vector< std::size_t > &  pivots 
)

The LU factorisation with partial pivoting of the n × n band of kl subdiagonals and ku superdiagonals in ab (2 kl + ku + 1 rows, the band from row kl + 1, the rows above for the fill the interchanges bring), in place, as LAPACK's gbtf2: each column's pivot the entry of largest pivot_magnitude (|Re| + |Im| for complex numbers, as izamax; the first of them) within the band below the diagonal, its row swapped to the reach of the rows it touches, the multipliers below and the rank-1 update right of it. pivots (from 1) the rows swapped.

Exceptions
singular_matrix_errorIf a pivot is zero (gbtf2's info; LAPACK goes on, SciPy then raises).

Definition at line 55 of file banded.hpp.

◆ band_lu_solve()

template<typename T >
void numerical::detail::band_lu_solve ( band_storage< T > &  ab,
std::size_t  n,
std::size_t  kl,
std::size_t  ku,
const std::vector< std::size_t > &  pivots,
buffer< T > &  b,
std::size_t  k 
)

The k right-hand sides of b (n × k, row-major) solved in place against band_lu's factors, as LAPACK's gbtrs (no transpose): the interchanges and L column by column, then U, a band of kl + ku superdiagonals, by back substitution.

Definition at line 108 of file banded.hpp.

◆ band_lu_solve_adjoint()

template<typename T >
void numerical::detail::band_lu_solve_adjoint ( band_storage< T > &  ab,
std::size_t  n,
std::size_t  kl,
std::size_t  ku,
const std::vector< std::size_t > &  pivots,
T *  x 
)

One right-hand side x (n entries) solved in place against the adjoint of band_lu's factors, Aᴴ x = b, as LAPACK's gbtrs (conjugate transpose): Uᴴ, a band of kl + ku subdiagonals, by forward substitution, then Lᴴ and the interchanges column by column from the last.

Definition at line 146 of file banded.hpp.

◆ best_gemm_path()

gemm_path numerical::detail::best_gemm_path ( gemm_path  most)
inlinenoexcept

The most capable path that can run here, most at the highest.

Definition at line 116 of file gemm.hpp.

◆ bidiagonal_divide()

void numerical::detail::bidiagonal_divide ( const double *  d,
const double *  e,
std::size_t  n,
std::size_t  extra,
buffer< double > &  sigma,
buffer< double > &  u,
buffer< double > &  v 
)
inline

The singular value decomposition of the upper bidiagonal n × (n + extra) (extra 0 or 1) with diagonal d[0…n-1] and superdiagonal e[0…n+extra-2] (e[n-1] the last column's entry when extra is 1), by divide and conquer, as LAPACK's dbdsdc: sigma ascending, u (n × n) and v (m × m, m = n + extra, the null vector last) row-major, their columns the singular vectors.

The rows split at a middle row k: above it an k × (k + 1) bidiagonal, below it an (n − k − 1) × (…) one, each decomposed the same way (by implicit-shift QR up to bidiagonal_leaf); row k's two entries, in their right bases, with the two null vectors rotated into one, make an arrow, which merge_arrow decomposes.

The tree is laid out first (plan_bidiagonal). The blocks at the depth that gives each thread one, and the leaves above it, are independent of one another and go to the threads together, each decomposed on one thread, a block right after its halves while they are still in its caches; the joins above them then come a level at a time, those of one level together, a level of one join sharing it among the threads. Every block takes the same inputs either way, so the result is the same to the bit; on one thread the order is the recursion's.

Definition at line 1474 of file svd.hpp.

◆ bidiagonal_divide_and_conquer()

template<typename T >
void numerical::detail::bidiagonal_divide_and_conquer ( buffer< double > &  d,
buffer< double > &  e,
buffer< T > &  ut,
buffer< T > &  vt,
std::size_t  n 
)

The singular values of the upper bidiagonal (d, e) into d, descending, the singular vectors accumulated into the rows of ut and vt as bidiagonal_qr accumulates them, by divide and conquer (bidiagonal_divide): most of the work in matrix products.

Exceptions
convergence_errorIf an iteration did not converge within its bound.

Definition at line 1522 of file svd.hpp.

◆ bidiagonal_qr()

template<typename T >
void numerical::detail::bidiagonal_qr ( buffer< double > &  d,
buffer< double > &  e,
buffer< T > &  ut,
buffer< T > &  vt,
std::size_t  n,
std::size_t  max_steps 
)

The singular values of the real upper bidiagonal (d, e) by implicit-shift QR with deflation (the Golub–Kahan–Reinsch iteration, LINPACK's dsvdc), the rotations accumulated into the rows of ut and vt. Each block's bulge is chased from its larger end, as LAPACK's dbdsqr chases it: from the top only, a block graded up (its small entries first) never converged.

Parameters
[in,out]dThe diagonal; on return, the singular values, descending and non-negative.
[in,out]eThe superdiagonal in e[0…n-2], e[n-1] = 0; destroyed.
[in,out]utRow-major n × n; on return, row j is the left vector of d[j]. Empty, with vt: the singular values alone.
[in,out]vtRow-major n × n; on return, row j is the right vector of d[j].
[in]nThe order.
[in]max_stepsThe bound of QR steps in all.
Exceptions
convergence_errorIf the iteration has not converged after max_steps steps.

Definition at line 663 of file svd.hpp.

◆ bidiagonalize() [1/2]

template<typename T >
requires std::is_floating_point_v<T> || split_complex<T>
void numerical::detail::bidiagonalize ( buffer< T > &  b,
std::size_t  n,
buffer< double > &  d,
buffer< double > &  e,
buffer< T > &  ut,
buffer< T > &  vt,
bool  form = true 
)

Reduces the square n × n row-major b to upper-bidiagonal form, B = U_b D V_bᴴ, by panels of bidiagonal_block reflections from each side, as LAPACK's dgebrd and dlabrd (zgebrd, zlabrd): the matrix a panel's later steps see is B − V Ȳᵀ − X Ūᵀ (V, U the reflectors, Y, X what they move, Ȳ and Ū held conjugated, so a complex matrix's update is a plain product), applied to the one column and the one row each step needs; the trailing block takes it after the panel, through the matrix product kernel. A real bidiagonal keeps its signs, which the iterations that follow take; a complex one is turned real as bidiagonalize_unblocked turns it, a unit phase on row k before its right reflection and on column k + 1 once that column comes to be reflected.

Parameters
[in,out]bThe matrix; destroyed.
[in]nIts order.
[out]dThe diagonal (size n).
[out]eThe superdiagonal in e[0…n-2], and e[n-1] = 0 (size n).
[out]utU_bᵀ, row-major: its rows are the columns of U_b (left as it is without form).
[out]vtV_bᵀ, row-major: its rows are the columns of V_b (likewise).
[in]formWhether to form ut and vt; the bidiagonal alone gives the singular values.

Definition at line 302 of file svd.hpp.

◆ bidiagonalize() [2/2]

template<typename T >
requires (!std::is_floating_point_v<T> && !split_complex<T>)
void numerical::detail::bidiagonalize ( buffer< T > &  b,
std::size_t  n,
buffer< double > &  d,
buffer< double > &  e,
buffer< T > &  ut,
buffer< T > &  vt,
bool  form = true 
)

bidiagonalize_unblocked, for any T the panels do not take (a complex long double).

Definition at line 637 of file svd.hpp.

◆ bidiagonalize_unblocked() [1/2]

template<typename T >
void numerical::detail::bidiagonalize_unblocked ( buffer< T > &  b,
std::size_t  n,
buffer< double > &  d,
buffer< double > &  e,
buffer< T > &  ut,
buffer< T > &  vt,
bool  form = true 
)

bidiagonalize_unblocked of the square n × n row-major b.

Definition at line 623 of file svd.hpp.

◆ bidiagonalize_unblocked() [2/2]

template<typename T >
void numerical::detail::bidiagonalize_unblocked ( buffer< T > &  b,
std::size_t  rows,
std::size_t  n,
buffer< double > &  d,
buffer< double > &  e,
buffer< T > &  ut,
buffer< T > &  vt,
bool  form 
)

Reduces the rows × n row-major b (rows ≥ n) to real upper-bidiagonal form, B = U_b D V_bᴴ with D real and n × n, by Householder reflections alternately from the left and the right (Golub–Kahan, LAPACK's dgebd2), the left ones rows − k long; the bases formed for a square b alone.

A complex entry left on the bidiagonal is turned real by a unit phase on its row or column, as LAPACK does, so the iteration that follows needs only real rotations.

Parameters
[in,out]bThe matrix; destroyed.
[in]rowsIts rows, at least n; n where form.
[in]nIts columns.
[out]dThe diagonal (size n).
[out]eThe superdiagonal in e[0…n-2], and e[n-1] = 0 (size n).
[out]utU_bᵀ, row-major: its rows are the columns of U_b (left as it is without form).
[out]vtV_bᵀ, row-major: its rows are the columns of V_b (likewise).
[in]formWhether to form ut and vt.

Definition at line 555 of file svd.hpp.

◆ blend()

double numerical::detail::blend ( bool  condition,
double  if_true,
double  if_false 
)
inlinenoexcept

if_true where condition holds, else if_false, by a mask of their bits: GCC turns a ?: between a value computed only for it and an argument into a branch around that computation, its table reads among it, and then vectorizes no loop of the function ("unsupported control flow in loop").

Definition at line 180 of file elementary.hpp.

◆ block_order() [1/2]

lapack_index numerical::detail::block_order ( one_based< double >  t,
lapack_index  n,
lapack_index  j 
)
inline

The order of the diagonal block of the quasi-triangular t that starts at j: 2 where a subdiagonal entry below it is nonzero.

Definition at line 1632 of file eigen.hpp.

◆ block_order() [2/2]

lapack_index numerical::detail::block_order ( one_based< std::complex< double > >  ,
lapack_index  ,
lapack_index   
)
inlineprivate

A complex upper triangular form's diagonal blocks: each of order 1.

Definition at line 3410 of file eigen.hpp.

◆ bluestein_body()

template<bool Fused, std::floating_point R>
void numerical::detail::bluestein_body ( const fft_plan< R > &  plan,
std::complex< R > *  data,
std::complex< R > *  work 
)
inline

Bluestein's transform of the plan's length in place in data, work its fft_plan::work, the input times the chirp, padded to M; its transform times the chirp's spectrum; the inverse transform of that (conj(fft(conj ·)), then 1/M, exact for a power of two), times the chirp. Products as Fused. Inlined into each caller.

Definition at line 1723 of file fft.hpp.

◆ bluestein_length()

std::size_t numerical::detail::bluestein_length ( std::size_t  n)
inline

The points of Bluestein's convolution of n points: the power of two at least 2n − 1, or the shortest length of factors 2, 3 and 5 at least 2n − 1 where it is at most 3/5 of that power and the power at least 1024 (passes of 3 and 5 cost more a point than those of 16: ten AVX-512 cores, 4099 points 156 µs over 16 384, 95 over 8640; 1009 points 18 µs over 2048, 30 over 2025; 67 points 1.3 µs over 256, 1.9 over 135).

Definition at line 982 of file fft.hpp.

◆ both_products()

template<typename T >
buffer< T > numerical::detail::both_products ( const buffer< T > &  p,
const buffer< T > &  q,
std::size_t  n 
)

P Q + Q P of n × n buffers.

Definition at line 973 of file matrix_functions.hpp.

◆ briggs() [1/2]

double numerical::detail::briggs ( double  a,
std::size_t  k 
)
inline

briggs of a real a above 0: (a − 1) / Π (1 + a^(1/2^j)), for k = 1 too (where √a − 1 would cancel).

Definition at line 2762 of file matrix_functions.hpp.

◆ briggs() [2/2]

std::complex< double > numerical::detail::briggs ( std::complex< double >  a,
std::size_t  k 
)
inline

a^(1/2^k) − 1 without the cancellation of the subtraction, as SciPy's _briggs_helper_function (Al-Mohy and Higham's algorithm 2, 2012): (a − 1) / Π (1 + a^(1/2^j)), a first square root taken where arg a ≥ π/2.

Definition at line 2694 of file matrix_functions.hpp.

◆ broadcast_into() [1/2]

template<typename T , typename Op , typename R = std::invoke_result_t<const Op&, const T&, const T&>>
tensor< R > numerical::detail::broadcast_into ( const T *  a,
const std::vector< std::size_t > &  a_shape,
const std::vector< std::ptrdiff_t > &  a_strides,
const T *  b,
const std::vector< std::size_t > &  b_shape,
const std::vector< std::ptrdiff_t > &  b_strides,
const Op &  op 
)

op(a, b) element by element of a, of shape a_shape, its elements a_strides apart along each axis from a (negative along an axis it reverses), and of b likewise, broadcast against each other (broadcast_apply's, its operands read where they lie), in op's result type R, row-major. Each element alone: the bits of the same operands laid row-major.

Exceptions
dimension_errorIf the shapes do not broadcast.

Definition at line 769 of file tensor.hpp.

◆ broadcast_into() [2/2]

template<typename T , typename Op , typename R = std::invoke_result_t<const Op&, const T&, const T&>>
tensor< R > numerical::detail::broadcast_into ( const T *  a,
const std::vector< std::size_t > &  a_shape,
const T *  b,
const std::vector< std::size_t > &  b_shape,
const Op &  op 
)

op(a, b) element by element of the row-major a (a_shape) and b (b_shape), broadcast against each other, in op's result type R.

Exceptions
dimension_errorIf the shapes do not broadcast.

Definition at line 957 of file tensor.hpp.

◆ broadcast_rows()

template<typename T , typename Op >
auto numerical::detail::broadcast_rows ( const T *  a,
const std::vector< std::size_t > &  a_shape,
const T *  b,
const std::vector< std::size_t > &  b_shape,
const Op &  op 
)

op(a, b) element by element of the row-major a (a_shape) and b (b_shape), broadcast against each other, in op's result type: broadcast_apply's paths for two contiguous tensors, so their bits, for elements no tensor holds (the binding's NumPy arrays). One element against as many axes or more is a number.

Exceptions
dimension_errorIf the shapes do not broadcast.

Definition at line 2125 of file tensor_ops.hpp.

◆ broadcast_strides()

std::vector< std::size_t > numerical::detail::broadcast_strides ( const std::vector< std::size_t > &  shape,
std::size_t  rank 
)
inline

The row-major strides of shape, with a stride of 0 on each axis of extent 1 (it broadcasts), aligned on the last axis of a result of rank rank (missing leading axes stride 0 too).

Definition at line 660 of file tensor.hpp.

◆ bulge_column() [1/2]

void numerical::detail::bulge_column ( one_based< double >  h,
lapack_index  count,
const std::complex< double > &  s1,
const std::complex< double > &  s2,
double *  v 
)
inline

A multiple of the first column of (H − s₁ I)(H − s₂ I) for the leading 2 × 2 or 3 × 3 (count) of the real Hessenberg h, the shifts a conjugate pair or two reals, as LAPACK's dlaqr1.

Definition at line 1756 of file eigen.hpp.

◆ bulge_column() [2/2]

void numerical::detail::bulge_column ( one_based< std::complex< double > >  h,
lapack_index  count,
const std::complex< double > &  s1,
const std::complex< double > &  s2,
std::complex< double > *  v 
)
inline

The same for a complex h and any two shifts, as LAPACK's zlaqr1.

Definition at line 1793 of file eigen.hpp.

◆ bunch_kaufman_blocked_lower()

template<typename T >
std::vector< std::ptrdiff_t > numerical::detail::bunch_kaufman_blocked_lower ( buffer< T > &  a,
std::size_t  n,
bool  hermitian,
bool  last_on_ties 
)

bunch_kaufman_lower by panels, as LAPACK's sytrf and hetrf on the lower triangle: each panel of ldl_block columns factorised with its updates deferred (factor_ldl_panel), then the trailing block's lower triangle less L₂₁ W₂₁ᴴ in one product, a real one of the kernel's size on the triangle's tiles alone, otherwise by bands of rows each to the diagonal; the last ldl_block columns or fewer by bunch_kaufman_lower's steps.

Returns
As bunch_kaufman_lower.

Definition at line 476 of file ldl.hpp.

◆ bunch_kaufman_lower()

template<typename T >
std::vector< std::ptrdiff_t > numerical::detail::bunch_kaufman_lower ( buffer< T > &  a,
std::size_t  n,
bool  hermitian,
bool  last_on_ties,
std::size_t  from = 0 
)

Bunch and Kaufman's factorisation of the lower triangle of a (n × n, row-major; entries (i, j) with i ≥ j read and written), as LAPACK's sytf2 and hetf2 on the lower triangle: at each step a pivot of order 1 or 2 chosen with α = (1 + √17) / 8, the rows and columns exchanged, the trailing matrix updated by the pivot's rank-1 or rank-2 correction, L's columns left below the pivot and D on and next to the diagonal. The largest entry of a column is its first (idamax) or, where last_on_ties, its last of equal size: the order of the upper triangle's search when a holds it reversed. From column from, the columns before it factorised already (bunch_kaufman_blocked_lower's last columns, as sytrf leaves them to sytf2).

Returns
LAPACK's ipiv (from 1): k the row exchanged with row k for a block of order 1, −k on both rows of a block of order 2; those before from 0.

Definition at line 246 of file ldl.hpp.

◆ by_reduction()

template<typename T >
constexpr bool numerical::detail::by_reduction ( std::size_t  inner)
constexprnoexcept

Whether extremes_along takes the reduction's path: an exact type, or an axis of single elements. A function, where the test written in the condition joined a constant to the axis's, which MSVC 19.44 (Visual Studio 2022's) rejects (C4127, an error under /WX).

Definition at line 1867 of file tensor_ops.hpp.

◆ by_rows()

bool numerical::detail::by_rows ( std::size_t  m,
std::size_t  n,
std::size_t  k 
)
inline

Whether gemm runs an m × k by k × n product by rows (gemm_rows): gemm_rows_columns columns or fewer; or, from narrow_group rows, gemm_rows_work multiply-adds or fewer, or gemm_rows_share a thread where its blocks of rows give at least half the threads one each (fewer, and the packed product's tasks, which split the columns too, are faster: 12 × 302 by 302 × 389 on ten AVX-512 cores, 1.6 times). Where the target's packed tiles do not keep a small product ahead.

Definition at line 1758 of file gemm.hpp.

◆ by_runs() [1/2]

tensor< double > numerical::detail::by_runs ( const tensor< double > &  a,
doubles_into  into 
)
inline

A function of every element of a, its kernel of doubles (runs_into).

Definition at line 2457 of file tensor_ops.hpp.

◆ by_runs() [2/2]

tensor< float > numerical::detail::by_runs ( const tensor< float > &  a,
doubles_into  into 
)
inline

A function of every element of a, its kernel of doubles through doubles (through_doubles_piece).

Definition at line 2464 of file tensor_ops.hpp.

◆ canonical_product()

template<typename T >
tensor< T > numerical::detail::canonical_product ( const T *  a,
std::size_t  lda,
const T *  b,
std::size_t  ldb,
std::size_t  rows,
std::size_t  inner,
std::size_t  cols,
std::vector< std::size_t >  out_shape 
)

The product of the row-major rows × inner a and inner × cols b, shaped out_shape: the canonical two-operand contraction of einsum_impl (a matrix times a vector, a vector times a matrix, a matrix product, a dot product). A matrix's rows lie lda (a) and ldb (b) apart, each row's elements together; a vector's elements together (b of one column: ldb then 1).

Definition at line 1633 of file tensor.hpp.

◆ chase_bulges()

template<typename T >
void numerical::detail::chase_bulges ( const schur_target< T > &  target,
lapack_index  ktop,
lapack_index  kbot,
std::complex< double > *  s,
lapack_index  count,
bool  accumulate 
)

One small-bulge multishift QR sweep over the active block ktop … kbot of target's Hessenberg matrix with the count shifts at s, as LAPACK's dlaqr5 and zlaqr5: chains of 3 × 3 bulges, two shifts each, chased down the diagonal by slabs of 4 · count / 2 columns; with accumulate, each slab's reflections are accumulated in U and applied to the rest of the matrix and to Z through the matrix product kernel, else applied to them directly (LAPACK's kacc22, which accumulates from 14 shifts). A subdiagonal entry that becomes negligible on the way is set to zero (vigilant deflation).

Definition at line 1924 of file eigen.hpp.

◆ cheapest_path()

void numerical::detail::cheapest_path ( const std::vector< std::string > &  labels,
const std::string &  output,
const std::map< char, std::size_t > &  sizes,
std::vector< contraction_step > &  path,
double &  cost,
double &  largest 
)
inline

The cheapest order of contraction from labels, and its cost and largest intermediate: every order tried, the first found kept among equals.

Definition at line 2124 of file tensor.hpp.

◆ checked_mul()

std::size_t numerical::detail::checked_mul ( std::size_t  a,
std::size_t  b 
)
inline

a * b, throwing numerical::dimension_error if the product overflows std::size_t. Guards dimension/shape products (element counts, allocation sizes) against silent wraparound on absurd inputs. The a != 0 guard also keeps the max / a division well-defined.

Definition at line 135 of file vector.hpp.

◆ cholesky_block_for()

constexpr std::size_t numerical::detail::cholesky_block_for ( std::size_t  n)
constexprnoexcept

The panel width of the Cholesky factorisation of order n: narrower up to order 256, where the panels, row by row on one thread, are most of the time; a power of two, which the product kernels' tiles divide (widths of 24, 40 or 56 left the AMX's 32 × 16 tiles part empty: order 320, 20 % longer).

Definition at line 285 of file factorizations.hpp.

◆ cholesky_reciprocal_condition()

template<typename T >
double numerical::detail::cholesky_reciprocal_condition ( const T *  factor,
std::size_t  n,
bool  lower,
double  one_norm 
)

reciprocal_condition of A = L Lᴴ or Uᴴ U from its Cholesky factor, L in the lower triangle of the n × n row-major factor or U in its upper one, and one_norm, ‖A‖₁, as LAPACK's pocon: A⁻¹ its own adjoint, the two products one.

Definition at line 2273 of file factorizations.hpp.

◆ chunked_conjugate_dot()

template<typename T >
T numerical::detail::chunked_conjugate_dot ( const T *  a,
const T *  b,
std::size_t  count 
)

Σ conj(a[i]) · b[i] (zdotc): chunked_dot of each chunk by conjugate_dot_product, without a conjugated copy of a.

Definition at line 5099 of file gemm.hpp.

◆ chunked_dot() [1/2]

template<typename T >
T numerical::detail::chunked_dot ( const T *  a,
const T *  b,
std::size_t  count 
)

Σ a[i] · b[i]: chunked_dot of each chunk by dot_product.

Definition at line 5089 of file gemm.hpp.

◆ chunked_dot() [2/2]

template<typename T , typename Dot >
T numerical::detail::chunked_dot ( const T *  a,
const T *  b,
std::size_t  count,
const Dot &  dot 
)

Σ a[i] · b[i] for a plain dot product: chunks of dot_chunk elements in parallel, each by dot, dot_product below (several vector accumulators, on the widest x86-64 units for double), their sums added in the chunks' order. The same on any number of threads; in the last bits, not the ascending sum, which waits on each addition before the next (ten times slower at 100 000 elements).

Definition at line 5066 of file gemm.hpp.

◆ circle_error()

buffer< std::complex< double > > numerical::detail::circle_error ( const buffer< std::complex< double > > &  whole,
const buffer< std::complex< double > > &  half,
const buffer< double > &  moduli,
std::size_t  count 
)
inline

The error of a circle's integral whole as recurrence_error carries it above the diagonal: the two rules' difference (whole − half, a function of the block as its integral's error is), and its sum's rounding, ε times its terms' moduli (moduli, summed over the count points), each entry in a direction of its own. whole and half as circle_integral leaves them (p × p).

Definition at line 2234 of file matrix_functions.hpp.

◆ circle_integral()

double numerical::detail::circle_integral ( buffer< std::complex< double > > &  whole,
buffer< std::complex< double > > &  half,
const buffer< double > &  moduli,
std::size_t  p,
std::size_t  count,
const std::complex< double > *  diagonal 
)
inline

A circle's integral from its rule's sum over count points, whole, and over the even ones, half (each divided by its count here), and its own estimate, relative to the integral's largest entry: the two rules' largest difference, the diagonal's against diagonal (f at the cluster's eigenvalues: a singularity of f in the disk breaks it), and the sum's rounding, ε times its terms' moduli (moduli, summed over the count points). A cluster far from normal has resolvents on its circle whose terms cancel to an integral orders below them (eight clusters of 32 at order 256: an integral wrong by 2·10⁻⁴ where the rules agreed to 4·10⁻¹¹); more points do not bring those digits back.

Definition at line 2204 of file matrix_functions.hpp.

◆ circle_points()

std::vector< std::complex< double > > numerical::detail::circle_points ( const cluster_circles &  circles,
const std::vector< std::size_t > &  counts 
)
inline

The points of circles, counts on the circles about clusters of two or more in turn, after one another.

Definition at line 2066 of file matrix_functions.hpp.

◆ circles_of()

cluster_circles numerical::detail::circles_of ( const matrix< std::complex< double > > &  t,
double  share 
)
inline

cluster_circles of the Schur form t whose clusters are runs of its diagonal: a circle about each cluster of two or more, its radius the cluster's spread plus a margin as wide as the spread, the cluster bound, or the block's coupling (its largest entry above the diagonal: the resolvent's terms ‖N‖ᵏ / rᵏ⁺¹ then stay near 1/r and cancel without loss), half the way to the nearest other eigenvalue at most.

Definition at line 2010 of file matrix_functions.hpp.

◆ climb_ends() [1/2]

template<std::floating_point R>
bool numerical::detail::climb_ends ( R  last,
R  largest 
)
noexcept

Whether the climb's last column last is as large as the new largest entry largest, so it stops: dlacn2 compares the real entry itself with the largest modulus.

Definition at line 2123 of file factorizations.hpp.

◆ climb_ends() [2/2]

template<std::floating_point R>
bool numerical::detail::climb_ends ( std::complex< R >  last,
std::complex< R >  largest 
)
noexcept

climb_ends of complex entries, as zlacn2: their moduli.

Definition at line 2131 of file factorizations.hpp.

◆ clipped()

template<std::floating_point T>
T numerical::detail::clipped ( T  x,
T  low,
T  high 
)
noexcept

ieee_minimum of ieee_maximum of x and low, and high, the bounds no NaN: the same bits (a NaN x kept, zeros of the two signs ordered alike), by selections, so that clip's loop vectorises.

Definition at line 2876 of file tensor_ops.hpp.

◆ cluster_bound()

double numerical::detail::cluster_bound ( const std::vector< std::complex< double > > &  values,
double  share 
)
inline

The clusters' bound: eigenvalues within share of the largest modulus (or of 1) gather (cluster_share, Davies and Higham's bound, relative here). Parlett's recurrence divides by the gaps between clusters: under it, gaps of 0.003 to 0.03 and an evenly spread spectrum lost 10⁶ to 10¹³ ε; a cluster's Cauchy integral takes as many points as its estimate asks (contour_points_most), a random matrix's whole spectrum among them.

Definition at line 1922 of file matrix_functions.hpp.

◆ clustered_function()

std::tuple< buffer< std::complex< double > >, double, std::vector< double > > numerical::detail::clustered_function ( const matrix< std::complex< double > > &  t,
const std::vector< std::complex< double > > &  images,
double  share,
const std::vector< std::size_t > &  counts 
)
inline

f(T) of the Schur form t with clusters within share as runs of its diagonal, from images, f at the eigenvalues then at each cluster's circle (circles_of), counts points each: each cluster's block by its integral (add_circle_terms, its even points' rule beside for the estimate, circle_integral), the rest by assembled_function.

Returns
f(T), the estimate (the larger of the recurrence's and the integrals'), and each circle's.
Exceptions
dimension_errorIf images does not hold one value per eigenvalue and counts per circle, even.

Definition at line 2306 of file matrix_functions.hpp.

◆ clusters_separate()

bool numerical::detail::clusters_separate ( const std::vector< std::complex< double > > &  values,
double  share 
)
inline

Whether the clusters of values within share are sound for circles_of, each cluster's farthest eigenvalue nearer its centre than every other one (no circle about it could keep the others out else), read from the values alone, before gather_clusters reorders a Schur form for them: a random matrix's spectrum chains into clusters no circle separates, whose reordering funm then gave up for the finer share (order 256: 12.7 of 41 ms). values not empty.

Definition at line 1939 of file matrix_functions.hpp.

◆ column_sum_norm()

double numerical::detail::column_sum_norm ( const buffer< double > &  moduli,
std::size_t  n 
)
inline

The 1-norm of the n × n row-major matrix whose entries' moduli are moduli (n from 1): its largest column sum.

Definition at line 68 of file matrix_functions.hpp.

◆ columns_amx_lanes()

void numerical::detail::columns_amx_lanes ( const double *  a,
std::size_t  lda,
std::size_t  rows,
std::size_t  cols,
const double *  x,
double *  y 
)
inline

amx_dot_rows_f64's sums of the rows rows of the column-major a (column j at a + j · lda) without the AMX: lane j mod 8 of each row its fused multiply-adds in ascending order of j, a column at a time over the rows together, then its lanes added from 0 to 7. The same bits, from a transpose where it lies.

Definition at line 5196 of file gemm.hpp.

◆ columns_by_value()

template<typename T >
buffer< T > numerical::detail::columns_by_value ( const buffer< T > &  zt,
const std::vector< std::size_t > &  order,
std::size_t  n 
)

The n × n row-major matrix whose column c is row order[c] of zt (the eigenvectors as rows): by tiles, the rows of each on its own core. Element by element, each write was a row apart, and the copy of order 1024 took a tenth of an eigh on six x86-64 cores.

Definition at line 6355 of file factorizations.hpp.

◆ columns_fft()

template<std::floating_point R>
void numerical::detail::columns_fft ( const fft_plan< R > &  plan,
const std::complex< R > *  data,
std::size_t  columns,
std::complex< R > *  to,
std::size_t  to_rows,
bool  fused,
bool  conjugated 
)

The transforms of plan's length along each of the columns columns of data into those of to (data's rows of columns, to's to_rows apart, as many as the plan's points): registers of neighbouring columns where they fill one (fft_lane_groups), the rest up to fft_column_block of them at a time, each row read and written as that many neighbours; the threads share them. Where conjugated, conj(fft(conj x)).

Definition at line 2133 of file fft.hpp.

◆ columns_times_vector() [1/2]

template<split_complex C>
bool numerical::detail::columns_times_vector ( const C *  a,
std::size_t  lda,
std::size_t  rows,
std::size_t  cols,
const C *  x,
C *  y 
)

columns_times_vector of complex numbers: each element the bits of its C-ordered copy's product, from the slots the portable vectors' complex_dot_sums keep for a row (columns_complex_row_dots), blocks of column_rows rows among the threads (GCC and Clang); on the x86-64 AVX paths from the slots their kernel keeps (avx2_complex_column_slots).

Returns
False, nothing written, under another compiler, or for rows longer than a chunk.

Definition at line 5297 of file gemm.hpp.

◆ columns_times_vector() [2/2]

template<typename T >
requires std::is_same_v<T, double> || std::is_same_v<T, float>
bool numerical::detail::columns_times_vector ( const T *  a,
std::size_t  lda,
std::size_t  rows,
std::size_t  cols,
const T *  x,
T *  y 
)

y = A x for the column-major rows × cols a (column j at a + j · lda; NumPy's a.T, a transposed operand) on the x86-64 units: each element the bits matrix_times_vector gives its C-ordered copy (avx2_dot's or avx512_dot's sum of its row), from the partial sums those kernels keep, updated a column at a time over the rows together (avx2_column_partials), then finished as the kernels finish a row; blocks of column_rows rows among the threads. The copy NumPy made of the transpose first cost more than the product (500 × 500: 170 µs of 213). float likewise, from the partial sums of its own kernels (32 on AVX2, 64 on AVX-512). Elsewhere (GCC and Clang) from the portable vectors' partials (columns_row_dots), as their row_dots sums a row; where the row-major product takes the AMX, from its lanes (columns_amx_lanes), in place of a copy of the transpose (400 × 400: 22 µs against 68, M1 Pro).

Returns
False, nothing written, on x86-64's portable path, under another compiler, or for rows longer than a chunk (chunked_dot sums them otherwise).

Definition at line 5237 of file gemm.hpp.

◆ complex_acos()

template<std::floating_point R>
std::complex< R > numerical::detail::complex_acos ( const std::complex< R > &  z)

acos(z), as glibc's cacos: π/2 − asin(z) for the infinities, NaNs and zeros, else kernel_asinh of iz adjusted.

Definition at line 964 of file complex_math.hpp.

◆ complex_acosh()

template<std::floating_point R>
std::complex< R > numerical::detail::complex_acosh ( const std::complex< R > &  z)

acosh(z), as glibc's cacosh: Annex G for the infinities, NaNs and zeros, else kernel_asinh of iz adjusted, its sign by z's imaginary part's.

Definition at line 978 of file complex_math.hpp.

◆ complex_asin()

template<std::floating_point R>
std::complex< R > numerical::detail::complex_asin ( const std::complex< R > &  z)

asin(z), as glibc's casin: −i asinh(iz); a NaN part as Annex G has it.

Definition at line 945 of file complex_math.hpp.

◆ complex_asinh()

template<std::floating_point R>
std::complex< R > numerical::detail::complex_asinh ( const std::complex< R > &  z)

asinh(z), as glibc's casinh: C's Annex G for the infinities and NaNs, z itself at ±0 ± 0i, else kernel_asinh.

Definition at line 922 of file complex_math.hpp.

◆ complex_atan()

template<std::floating_point R>
std::complex< R > numerical::detail::complex_atan ( const std::complex< R > &  z)

atan(z), as glibc's catan: C's Annex G for the infinities and NaNs; π/2 and the reciprocal past 16/ε; else the real part atan2(2x, 1 − |z|²) / 2, 1 − |z|² from unit_circle_offset where it cancels, the imaginary part log(|z + i|² / |z − i|²) / 4 (by log1p(4y / |z − i|²) where the quotient is at least ½): complex_atanh's turned a quarter, each operation in glibc's order.

Definition at line 1010 of file complex_math.hpp.

◆ complex_atanh()

template<std::floating_point R>
std::complex< R > numerical::detail::complex_atanh ( const std::complex< R > &  z)

atanh(z), as glibc's catanh: C's Annex G for the infinities and NaNs; π/2 and the reciprocal past 16/ε; else the real part log(|1 + z|² / |1 − z|²) / 4 (by log1p(4x / |1 − z|²) where the quotient is at least ½), the imaginary part atan2(2y, 1 − |z|²) / 2, 1 − |z|² from unit_circle_offset where it cancels.

Definition at line 739 of file complex_math.hpp.

◆ complex_cos()

template<std::floating_point R>
std::complex< R > numerical::detail::complex_cos ( const std::complex< R > &  z)

cos(z), as cosh(i z): complex_cosh of −im + i re.

Definition at line 345 of file complex_math.hpp.

◆ complex_cosh()

template<std::floating_point R>
std::complex< R > numerical::detail::complex_cosh ( const std::complex< R > &  z)

cosh(z): cosh(re) cos(im) + i sinh(re) sin(im); past exp_bound, exp(|re|) / 2 times the cosine and the sine, taken off in steps as complex_exp does.

Definition at line 235 of file complex_math.hpp.

◆ complex_dot_product()

template<split_complex C>
C numerical::detail::complex_dot_product ( const C *  x,
const C *  y,
std::size_t  count 
)

Σ x[i] · y[i] of complex numbers: on the x86-64 units with AVX2 (the AVX-512 path too) their fused chains (avx2_complex_dot), elsewhere the native vectors' (in another order of summation there, as fixed).

Definition at line 753 of file gemm.hpp.

◆ complex_exp()

template<std::floating_point R>
std::complex< R > numerical::detail::complex_exp ( const std::complex< R > &  z)

exp(z): exp(re) times cos(im) + i sin(im), a real part past exp_bound taken off in steps of it (each a factor exp(bound) on the sine and the cosine), so that a finite product does not overflow early.

Definition at line 180 of file complex_math.hpp.

◆ complex_exp2()

template<std::floating_point R>
std::complex< R > numerical::detail::complex_exp2 ( const std::complex< R > &  z)

2^z, as NumPy's: complex_exp of z times ln 2, each part.

Definition at line 1098 of file complex_math.hpp.

◆ complex_expm1()

template<std::floating_point R>
std::complex< R > numerical::detail::complex_expm1 ( const std::complex< R > &  z)

exp(z) − 1, as NumPy's: expm1(re) cos(im) − 2 sin²(im/2) + i exp(re) sin(im), the sine and the cosine of im in calls of their own as NumPy's (separate_sine); past exp_bound, a finite im's complex_exp less 1, where exp(re) overflows and NumPy's product with the sine with it (NaN at +∞ + 0i, an infinite 710.2 + 0.1i).

Definition at line 1110 of file complex_math.hpp.

◆ complex_log()

template<std::floating_point R>
std::complex< R > numerical::detail::complex_log ( const std::complex< R > &  z)

log(z): log|z| + i arg(z), log|z| as log1p(|z|² − 1) / 2 near the unit circle (|z|² − 1 exact or almost, unit_circle_offset) and as log(hypot) elsewhere, the larger part scaled by 2 near overflow and both by 2^digits under the smallest normal value.

Definition at line 589 of file complex_math.hpp.

◆ complex_log10()

template<std::floating_point R>
std::complex< R > numerical::detail::complex_log10 ( const std::complex< R > &  z)

log10(z), as NumPy's: complex_log times log₁₀ e, each part.

Definition at line 1090 of file complex_math.hpp.

◆ complex_log1p()

template<std::floating_point R>
std::complex< R > numerical::detail::complex_log1p ( const std::complex< R > &  z)

log(1 + z): the imaginary part atan2(im, 1 + re), as NumPy's; the real part log|1 + z|, by log1p(d) / 2 of d = |1 + z|² − 1 = 2re + re² + im² where |re| and |im| are at most 3 and d at least −½ (the squares exact by fused multiply-adds, the sum compensated: d keeps its digits where it cancels, about the circle |1 + z| = 1 and about 0, where NumPy's log(hypot(1 + re, im)) loses them all); else complex_log of 1 + z, |1 + z| there far from 1 or 1 + re exact, finite to the largest numbers (NumPy's hypot overflows past max / √2). A zero is its own, its signs kept, as log1p of a real −0.

Definition at line 1131 of file complex_math.hpp.

◆ complex_log2()

template<std::floating_point R>
std::complex< R > numerical::detail::complex_log2 ( const std::complex< R > &  z)

log2(z), as NumPy's: complex_log times log₂ e, each part.

Definition at line 1082 of file complex_math.hpp.

◆ complex_reflector()

std::complex< double > numerical::detail::complex_reflector ( std::complex< double >(&)  v[2])
inline

The Householder reflection I − τ v vᴴ, v = (1, x), whose adjoint takes (α, x) to (β, 0), β real, as LAPACK's zlarfg for two entries: on return v[0] = β and v[1] holds x's part of v.

Returns
τ, 0 where x is zero and α real (the identity).

Definition at line 678 of file eigen.hpp.

◆ complex_row_dots()

template<split_complex C>
void numerical::detail::complex_row_dots ( const C *  m,
std::size_t  ld,
std::size_t  rows,
const C *  x,
std::size_t  count,
C *  out 
)

row_dots of complex numbers: each row by complex_dot_product, as chunked_dot sums a row.

Definition at line 783 of file gemm.hpp.

◆ complex_scaled_add()

template<split_complex C>
void numerical::detail::complex_scaled_add ( C &  y,
const C &  f,
const C &  x 
)

y += f · x of one complex number, its part products rounded apart as libc++'s operator computes them (add_scaled_vectors's arithmetic, for the elements past its vectors).

Definition at line 1313 of file simd.hpp.

◆ complex_schur()

void numerical::detail::complex_schur ( const schur_target< std::complex< double > > &  t,
buffer< std::complex< double > > &  values,
std::size_t  steps 
)
inline

The complex Schur form of the upper Hessenberg t's active block, as LAPACK's zlahqr: the shifted QR algorithm with Wilkinson's shift, the deflation criterion of Ahues and Tisseur, and exceptional shifts after 10 and 20 iterations without deflation. The eigenvalues are left in values.

Parameters
[in]tThe Hessenberg matrix, its active block, and the Schur vectors to update.
[in,out]valuesThe eigenvalues (the isolated ones already in place).
[in]stepsThe iterations allowed between two deflations.
Exceptions
convergence_errorIf an eigenvalue has not converged within steps.

Definition at line 1126 of file eigen.hpp.

◆ complex_schur_form()

std::pair< matrix< std::complex< double > >, matrix< std::complex< double > > > numerical::detail::complex_schur_form ( const matrix< double > &  t,
const matrix< double > &  z 
)
inline

The complex Schur form of a real one, as SciPy's rsf2csf: each 2 × 2 block of t (a conjugate pair) triangularised by a unitary rotation of its two rows and columns, the columns of z turning with it; a real eigenvalue's diagonal entry kept exactly real.

Returns
(T, Z) complex.

Definition at line 1401 of file matrix_functions.hpp.

◆ complex_sign()

template<std::floating_point R>
std::complex< R > numerical::detail::complex_sign ( const std::complex< R > &  z)

z / |z|, as NumPy 2's sign of a complex number and the array API's: 0 for 0, NaN for a NaN modulus, and for an infinite modulus ±1 along the infinite part (NaN where both are), 0 across it.

Definition at line 1164 of file complex_math.hpp.

◆ complex_sin()

template<std::floating_point R>
std::complex< R > numerical::detail::complex_sin ( const std::complex< R > &  z)

sin(z): sin(re) cosh(im) + i cos(re) sinh(im) of |re|, the sign of re carried by the sine; an imaginary part past exp_bound as complex_sinh's real part.

Definition at line 355 of file complex_math.hpp.

◆ complex_sinh()

template<std::floating_point R>
std::complex< R > numerical::detail::complex_sinh ( const std::complex< R > &  z)

sinh(z): sinh(re) cos(im) + i cosh(re) sin(im) of |re|, the sign of re carried by the cosine; past exp_bound as complex_cosh.

Definition at line 288 of file complex_math.hpp.

◆ complex_sqrt()

template<std::floating_point R>
std::complex< R > numerical::detail::complex_sqrt ( const std::complex< R > &  z)

The square root of a complex number, as the GNU C library's csqrt computes it (NumPy's on Linux, to the bit): the infinities and NaNs of C's Annex G; a real or an imaginary argument apart (the root of -1 is i exactly, that of y i has equal parts); otherwise sqrt((|z| ± re) / 2) for the larger part and the imaginary part over twice it for the other, scaled by 4 near overflow and by a power of 2 among subnormal parts. std::sqrt of libc++ goes through the polar form: sqrt(-1e20 + 1j) in single precision had a real part of 755 for 5e-11.

Definition at line 654 of file complex_math.hpp.

◆ complex_square_root()

matrix< std::complex< double > > numerical::detail::complex_square_root ( const matrix< std::complex< double > > &  form,
const matrix< std::complex< double > > *  z 
)
inline

The principal square root of a complex matrix from its Schur form (triangular_square_root).

Definition at line 1387 of file matrix_functions.hpp.

◆ complex_tan()

template<std::floating_point R>
std::complex< R > numerical::detail::complex_tan ( const std::complex< R > &  z)

tan(z): (sin(re) cos(re) + i sinh(im) cosh(im)) / (cos(re)² + sinh(im)²); past half_exp_bound the imaginary part is ±1 and the real one 4 sin(re) cos(re) / exp(2|im|), divided in steps.

Definition at line 475 of file complex_math.hpp.

◆ complex_tanh()

template<std::floating_point R>
std::complex< R > numerical::detail::complex_tanh ( const std::complex< R > &  z)

tanh(z): (sinh(re) cosh(re) + i sin(im) cos(im)) / (sinh(re)² + cos(im)²); past half_exp_bound the real part is ±1 and the imaginary one 4 sin(im) cos(im) / exp(2|re|), divided in steps.

Definition at line 419 of file complex_math.hpp.

◆ complex_terms()

complex_term numerical::detail::complex_terms ( )
inlinenoexcept

The complex_term of the path in use.

Definition at line 2318 of file gemm.hpp.

◆ complex_vector_times_columns()

template<split_complex C>
buffer< C > numerical::detail::complex_vector_times_columns ( const C *  a,
const C *  b,
std::size_t  row_step,
std::size_t  column_step,
std::size_t  inner,
std::size_t  cols 
)

The complex vector a (inner) times a matrix b whose element B[k][j] is b[(k · row_step) + (j · column_step)], each element of the product rounded as the path's complex product rounds it (complex_terms, past complex_term::compiler): so the bits of its row in any product of a matrix by B. On the x86-64 complex tiles by their panels (avx2_complex_row_panels), on arm64 by gemm_split's blocks (split_row_columns).

Definition at line 1431 of file tensor.hpp.

◆ complex_vectors()

buffer< std::complex< double > > numerical::detail::complex_vectors ( const buffer< std::complex< double > > &  t,
std::size_t  n 
)
inline

The eigenvectors of the complex upper triangular t (row-major n × n) as the rows of Xᵀ: row k is x with (T − t_kk I) x = 0, x_k = 1, zero past k, as LAPACK's ztrevc.

Definition at line 2948 of file eigen.hpp.

◆ conj_of() [1/2]

template<typename U >
std::complex< U > numerical::detail::conj_of ( const std::complex< U > &  x)

The conjugate of a complex scalar.

Definition at line 323 of file factorizations.hpp.

◆ conj_of() [2/2]

template<typename T >
T numerical::detail::conj_of ( const T &  x)

The conjugate of a real (or general) scalar — the value itself.

Definition at line 316 of file factorizations.hpp.

◆ conjugate_dot() [1/2]

template<typename T >
requires std::is_floating_point_v<T>
T numerical::detail::conjugate_dot ( const T *  x,
const T *  y,
std::size_t  count 
)
private

Σ conj(x[i]) · y[i] for a real T, where the conjugate is the value: dot_product.

Definition at line 3190 of file factorizations.hpp.

◆ conjugate_dot() [2/2]

template<typename T >
T numerical::detail::conjugate_dot ( const T *  x,
const T *  y,
std::size_t  count 
)
private

Σ conj(x[i]) · y[i] of complex numbers (conjugate_dot_product).

Definition at line 3199 of file factorizations.hpp.

◆ conjugate_dot_product()

template<typename T >
T numerical::detail::conjugate_dot_product ( const T *  x,
const T *  y,
std::size_t  count 
)

Σ conj(x[i]) · y[i], in order, where no native vector holds the parts.

Definition at line 1089 of file simd.hpp.

◆ conjugated_chunks() [1/2]

template<split_complex C>
C numerical::detail::conjugated_chunks ( const C *  a,
const C *  b,
std::size_t  count 
)

Σ conj(a[i]) · b[i] of complex numbers: chunked_conjugate_dot.

Definition at line 4142 of file tensor_ops.hpp.

◆ conjugated_chunks() [2/2]

template<typename T >
T numerical::detail::conjugated_chunks ( const T *  a,
const T *  b,
std::size_t  count 
)

Σ a[i] · b[i] of reals, their own conjugates: chunked_dot.

Definition at line 4133 of file tensor_ops.hpp.

◆ conjugated_if() [1/2]

std::complex< double > numerical::detail::conjugated_if ( const std::complex< double > &  z,
bool  hermitian 
)
inline

z's conjugate for a Hermitian matrix, z for a symmetric one.

Definition at line 60 of file ldl.hpp.

◆ conjugated_if() [2/2]

double numerical::detail::conjugated_if ( double  x,
bool   
)
inline

A real x as it is.

Definition at line 53 of file ldl.hpp.

◆ contiguous_extreme()

template<typename T , typename Order >
T numerical::detail::contiguous_extreme ( const T *  first,
std::size_t  count,
const Order &  larger 
)

The largest (larger is std::greater) or smallest of count contiguous elements, or a NaN if one is among them, as NumPy's: the comparisons run without a branch over native vectors (or eight scalar lanes), each lane its own running best, and the NaN is tracked apart.

Definition at line 1713 of file tensor_ops.hpp.

◆ contiguous_norm()

template<typename T >
double numerical::detail::contiguous_norm ( const T *  x,
std::size_t  count 
)
private

The 2-norm of the count entries at x, scaled by the largest so no square overflows or underflows.

Definition at line 3988 of file factorizations.hpp.

◆ continuous_backward_error()

double numerical::detail::continuous_backward_error ( const riccati_terms &  terms,
const matrix< std::complex< double > > &  x 
)
inlineprivate

The residual at x of the continuous equation without E (the Hamiltonian's), relative to its terms: ‖Res(X)‖₁ / (2 ‖A‖₁ ‖X‖₁ + ‖Bᴴ X + Sᴴ‖₁ ‖R⁻¹(Bᴴ X + Sᴴ)‖₁ + ‖Q‖₁) (continuous_residual); infinite where it is not finite.

Definition at line 3011 of file generalized.hpp.

◆ continuous_residual()

matrix< std::complex< double > > numerical::detail::continuous_residual ( const riccati_terms &  terms,
const matrix< std::complex< double > > &  x,
matrix< std::complex< double > > &  closed 
)
inlineprivate

The residual Aᴴ X E + Eᴴ X A − (Eᴴ X B + S) R⁻¹ (Bᴴ X E + Sᴴ) + Q of the continuous equation at the Hermitian x, and the closed loop A − B K, K = R⁻¹ (Bᴴ X E + Sᴴ), into closed.

Definition at line 2982 of file generalized.hpp.

◆ contract_pair()

template<typename T >
tensor< T > numerical::detail::contract_pair ( const std::string &  sa,
operand_span< T >  a,
const std::string &  sb,
operand_span< T >  b,
const std::string &  output,
const std::string &  summed,
const std::map< char, std::size_t > &  sizes 
)

A two-operand contraction as matrix products: the labels both operands and the output carry (batch), those one operand and the output carry (free), and those both operands carry alone (summed, in ascending order, the order the general loop sums them in, so each element takes the same terms in the same order) regroup each operand as a stack of matrices, A as batch × free_a × summed and B as batch × summed × free_b, multiplied batch by batch through the packed kernel, batches in parallel; the result is laid out in the output's order; a dot product per batch by dot_product. Each label of an operand once, and each in the other operand or the output: einsum_impl sums an operand's own labels and reads a repeated one's diagonal first.

Definition at line 1982 of file tensor.hpp.

◆ contraction_path()

std::vector< contraction_step > numerical::detail::contraction_path ( std::vector< std::string >  labels,
const std::string &  output,
const std::map< char, std::size_t > &  sizes 
)
inline

The order in which to contract operands labelled labels into output, as opt_einsum chooses it: the cheapest in multiply-adds (the product of the extents a pair's labels span), the smaller largest intermediate among equals, over every order up to exhaustive_path_operands operands; past it, the cheapest pair at each step among those sharing a label (an outer product, cheap in itself, would leave its labels to every later step), the smaller result, then the first pair, among equals. It depends on the shapes alone, so a result is the same on any number of threads.

Definition at line 2168 of file tensor.hpp.

◆ converted()

template<typename To , typename From >
tensor< To > numerical::detail::converted ( const tensor< From > &  t)

A copy of t, each element converted to To (between a signed integer type and its unsigned one, the same bits, modulo 2ⁿ), by chunks of elementwise_chunk among the threads: one thread's copy and its pages' first touches were most of an int64 dot (10⁶ elements, M1 Pro: 0.49 of 0.61 ms).

Definition at line 2413 of file tensor.hpp.

◆ copy_few()

template<typename T >
void numerical::detail::copy_few ( const T *  from,
std::size_t  count,
T *  to 
)
inlinenoexcept

copy_few_bytes of count elements of T, one at least and at most 128 bytes.

Definition at line 77 of file simd.hpp.

◆ copy_few_bytes()

void numerical::detail::copy_few_bytes ( const unsigned char *  from,
std::size_t  bytes,
unsigned char *  to 
)
inlinenoexcept

Copies bytes bytes, 4 to 128, from from to to (apart): by two or four copies of one fixed size that overlap where the count is not a multiple of it, the last ending at the last byte. A few elements copied by a loop went to the library's memmove, a call a row of a tile's edge (one M1 Pro core, float products of order 100 to 200 on the AMX: 4 to 5 % of their time).

Definition at line 44 of file simd.hpp.

◆ corrected_hermitian_product()

template<typename T >
void numerical::detail::corrected_hermitian_product ( const T *  block,
std::size_t  n,
std::size_t  m,
const T *  v,
T *  w,
buffer< T > &  scratch,
const T *  corrections,
std::size_t  ldc,
std::size_t  terms,
T *  factors,
T *  next = nullptr,
const T *  next_factors = nullptr 
)

hermitian_product corrected by a panel's earlier steps: w = B v − Σₜ fₜ cₜ, the coefficients f[2p] = (c[2p+1] · v) and f[2p+1] = (c[2p] · v) (conjugated) computed beside the product, among its tasks, and the partials summed and the corrections taken by bands of rows: each element the operations, in the order, hermitian_product, conjugate_dot and subtract_products give it apart.

Parameters
[in]blockThe block's first element; its rows n apart.
[in]nThe row stride.
[in]mThe block's order.
[in]vThe vector (m elements).
[out]wB v corrected (m elements).
[in,out]scratchhermitian_product's partials, kept by the caller from one step to the next.
[in]correctionsThe rows c₀ … c_{terms−1} (ldc apart), over the block's rows.
[in]ldcThe distance between those rows.
[in]termsTheir count, even (two a step: its v and its w).
[out]factorsThe coefficients (terms).
[in,out]nextThe panel's next column over the block's rows, which takes the terms of these corrections with next_factors in the same pass; none past a panel's last step.
[in]next_factorsIts coefficients of the rows (terms).

Definition at line 4657 of file factorizations.hpp.

◆ correlation_step() [1/2]

template<split_complex C>
C numerical::detail::correlation_step ( const C &  sum,
const C &  x,
const C &  y 
)

sum + x · conj(y) of a complex correlation's terms, NumPy's product rounded.

Definition at line 4837 of file tensor_ops.hpp.

◆ correlation_step() [2/2]

template<typename T >
T numerical::detail::correlation_step ( const T &  sum,
const T &  x,
const T &  y 
)

sum + x · y of a real or an integer (wrapped) correlation's terms.

Definition at line 4828 of file tensor_ops.hpp.

◆ cos_of() [1/3]

tensor< double > numerical::detail::cos_of ( const double *  data,
const std::vector< std::size_t > &  shape 
)
inline

cos of every one of the row-major doubles at data, of shape shape: double_sines_into by chunks of elementwise_chunk on the threads.

Definition at line 1158 of file tensor_ops.hpp.

◆ cos_of() [2/3]

tensor< float > numerical::detail::cos_of ( const float *  data,
const std::vector< std::size_t > &  shape 
)
inline

cos of every one of the row-major floats at data, of shape shape: float_sines_into of them as doubles (through_doubles).

Definition at line 1172 of file tensor_ops.hpp.

◆ cos_of() [3/3]

template<typename T >
tensor< T > numerical::detail::cos_of ( const T *  data,
const std::vector< std::size_t > &  shape 
)

cos of every one of the row-major scalars at data, of shape shape: std::cos of each.

Definition at line 1150 of file tensor_ops.hpp.

◆ cos_viewed() [1/3]

tensor< double > numerical::detail::cos_viewed ( const tensor< double > &  a)
inline

cos of every double of a, a view read where it lies (by_runs).

Definition at line 2618 of file tensor_ops.hpp.

◆ cos_viewed() [2/3]

tensor< float > numerical::detail::cos_viewed ( const tensor< float > &  a)
inline

cos of every float of a through doubles, a view read where it lies (by_runs).

Definition at line 2624 of file tensor_ops.hpp.

◆ cos_viewed() [3/3]

template<typename T >
tensor< T > numerical::detail::cos_viewed ( const tensor< T > &  a)

cos of every element of a: of a contiguous copy (cos_of).

Definition at line 2612 of file tensor_ops.hpp.

◆ cosh_into()

void numerical::detail::cosh_into ( const double *  in,
double *  out,
std::size_t  count 
)
inline

cosh of the count doubles at in into out, each elementary_cosh's: table_cosh in vectors (with the FMA instructions on an x86-64 processor, run_fused), std::cosh throughout where exp_by_table says no.

Definition at line 690 of file tensor_ops.hpp.

◆ cosh_of() [1/3]

tensor< double > numerical::detail::cosh_of ( const double *  data,
const std::vector< std::size_t > &  shape 
)
inline

cosh of every one of the row-major doubles at data, of shape shape: cosh_into by chunks of elementwise_chunk among the threads.

Definition at line 727 of file tensor_ops.hpp.

◆ cosh_of() [2/3]

tensor< float > numerical::detail::cosh_of ( const float *  data,
const std::vector< std::size_t > &  shape 
)
inline

cosh of every one of the row-major floats at data, of shape shape: cosh_into of them as doubles (through_doubles).

Definition at line 748 of file tensor_ops.hpp.

◆ cosh_of() [3/3]

template<typename T >
tensor< T > numerical::detail::cosh_of ( const T *  data,
const std::vector< std::size_t > &  shape 
)

cosh of every one of the row-major scalars at data, of shape shape: std::cosh of each.

Definition at line 719 of file tensor_ops.hpp.

◆ cosh_viewed() [1/3]

tensor< double > numerical::detail::cosh_viewed ( const tensor< double > &  a)
inline

cosh of every double of a, a view read where it lies (by_runs).

Definition at line 2548 of file tensor_ops.hpp.

◆ cosh_viewed() [2/3]

tensor< float > numerical::detail::cosh_viewed ( const tensor< float > &  a)
inline

cosh of every float of a through doubles, a view read where it lies (by_runs).

Definition at line 2554 of file tensor_ops.hpp.

◆ cosh_viewed() [3/3]

template<typename T >
tensor< T > numerical::detail::cosh_viewed ( const tensor< T > &  a)

cosh of every element of a: of a contiguous copy (cosh_of).

Definition at line 2542 of file tensor_ops.hpp.

◆ count_bytes()

template<typename U >
void numerical::detail::count_bytes ( byte_counts< U > &  counts,
U  key 
)
noexcept

Counts key's bytes in counts. Every byte is counted in the one read that makes the keys, where a read per byte cost a fifth of the sort (10⁶ float64: 22.8 → 18.5 ms on x86-64, 12.1 → 9.5 on arm64).

Definition at line 3550 of file tensor_ops.hpp.

◆ count_of()

std::size_t numerical::detail::count_of ( lapack_index  count)
inline

A count as an unsigned size (never negative where it is taken).

Definition at line 1272 of file eigen.hpp.

◆ cpu_max_processors()

std::size_t numerical::detail::cpu_max_processors ( std::string_view  text)
inlinenoexcept

quota_processors of cgroup v2's cpu.max text, "<quota> <period>" ("max" for none): 0 for none or a text it does not read.

Definition at line 220 of file parallel.hpp.

◆ current_process()

long numerical::detail::current_process ( )
inlinenoexcept

This process's identifier: a pool started before a fork belongs to the parent, whose threads the child does not have.

Definition at line 54 of file parallel.hpp.

◆ cyclic_jacobi()

void numerical::detail::cyclic_jacobi ( buffer< double > &  a,
buffer< double > &  v,
std::size_t  n,
std::size_t  max_sweeps 
)
inline

Cyclic Jacobi on the full symmetric a, the rotations accumulated into v.

Parameters
[in,out]aThe matrix; on return, diagonal to rounding, each element off it below ε times the geometric mean of its row's and column's diagonal elements.
[in,out]vThe rotations (start from the identity).
[in]nThe order.
[in]max_sweepsThe bound of sweeps.
Exceptions
convergence_errorIf a sweep past max_sweeps would still rotate.

Definition at line 6310 of file factorizations.hpp.

◆ dd_add()

double_double numerical::detail::dd_add ( double_double  x,
double_double  y 
)
inlinenoexcept

The sum of two double-double numbers.

Definition at line 90 of file fft.hpp.

◆ dd_div()

double_double numerical::detail::dd_div ( double_double  x,
double  d 
)
inlinenoexcept

x divided by the integer d (exact in a double).

Definition at line 113 of file fft.hpp.

◆ dd_mul()

double_double numerical::detail::dd_mul ( double_double  x,
double_double  y 
)
inlinenoexcept

The product of two double-double numbers, its cross terms by explicit fused multiply-adds.

Definition at line 102 of file fft.hpp.

◆ dd_negate()

double_double numerical::detail::dd_negate ( double_double  x)
inlinenoexcept

x negated, both parts.

Definition at line 167 of file fft.hpp.

◆ dd_sin_cos()

std::pair< double_double, double_double > numerical::detail::dd_sin_cos ( double_double  t)
inline

The cosine and sine of t, in [0, π/4], as double-double numbers: their Taylor series in t² by Horner's rule to the term of degree 28 and 29, whose next terms are under 10⁻³² of the sum there.

Definition at line 143 of file fft.hpp.

◆ diagonal_kept() [1/2]

std::complex< double > numerical::detail::diagonal_kept ( const std::complex< double > &  z,
bool  hermitian 
)
inline

A Hermitian matrix's diagonal entry z taken as real (hetf2 ignores its imaginary part), a symmetric one's as it is.

Definition at line 75 of file ldl.hpp.

◆ diagonal_kept() [2/2]

double numerical::detail::diagonal_kept ( double  x,
bool   
)
inline

A real x as it is.

Definition at line 67 of file ldl.hpp.

◆ diagonal_labels()

template<typename T >
std::optional< tensor< T > > numerical::detail::diagonal_labels ( const std::string &  in,
const std::string &  output,
std::size_t  distinct,
const std::vector< std::size_t > &  shape,
const T *  data 
)

einsum of one row-major operand data (shape, labelled in, distinct labels in all) whose last two labels are one (…ii), the others each once and output those in order, with that label (…ii->…i, each matrix's diagonal) or without (…ii->…, its sum by pairwise halving, as pairwise_sums adds a run: the same bits as trace's sum of the diagonal). None for any other spec.

Definition at line 1677 of file tensor.hpp.

◆ diagonal_of()

std::vector< std::complex< double > > numerical::detail::diagonal_of ( const matrix< std::complex< double > > &  t)
inline

The diagonal of the square t.

Definition at line 1975 of file matrix_functions.hpp.

◆ diagonal_size() [1/2]

double numerical::detail::diagonal_size ( const std::complex< double > &  z,
bool  hermitian 
)
inline

A complex diagonal entry's measure: |Re z| for a Hermitian matrix (hetf2), CABS1 for a symmetric one (sytf2).

Definition at line 46 of file ldl.hpp.

◆ diagonal_size() [2/2]

double numerical::detail::diagonal_size ( double  x,
bool   
)
inline

A real diagonal entry's measure, |x|, whatever hermitian.

Definition at line 38 of file ldl.hpp.

◆ divide_and_conquer()

void numerical::detail::divide_and_conquer ( double *  d,
const double *  e,
std::size_t  n,
buffer< double > &  q 
)
inline

The eigenvalues, ascending, of the symmetric tridiagonal with diagonal d[0…n-1] and subdiagonal e[0…n-2] into d, and its eigenvectors as the columns of the row-major q (n × n), by Cuppen's divide and conquer: the tridiagonal is two halves plus a rank-one coupling, each half is diagonalised the same way (by implicit QL up to divide_leaf), and merge_rank_one joins them, each half into its own diagonal block of q, so the joins copy no half and leave the blocks off the diagonal as they found them until the join of their two halves fills them.

The tree is laid out first (plan_divide). The blocks at the depth that gives each thread one, and the leaves above it, are independent of one another and go to the threads together, each solved on one thread with its own buffers, a block right after its halves while they are still in its caches; the joins above them then come a level at a time, those of one level together, a level of one join sharing it among the threads. One after another each block had run alone (six AVX2 cores, an eigh of order 256: the leaves and the joins below the last, 1.5 ms of the 4 ms the eigenvectors took, no faster than on one). Every block takes the same inputs either way, so the result is the same to the bit; on one thread the order is the recursion's.

Definition at line 6147 of file factorizations.hpp.

◆ divided()

template<typename S , typename P >
scaled_quotient< S > numerical::detail::divided ( S  r,
P  pivot,
double  floor 
)

r / pivot, a pivot below floor taken as floor, r scaled first where the quotient would pass back_substitution_limit.

Returns
The quotient and the factor (1 where none was needed).

Definition at line 2733 of file eigen.hpp.

◆ divided_by_count()

template<std::floating_point T>
T numerical::detail::divided_by_count ( T  value,
std::size_t  n 
)

A mean: value divided by the count n, in the wider of T and double and back to T, the count exact (to 2^53), as NumPy divides by its integer count; a float's count rounded to a float past 2^24 (2^24 + 1 ones averaged to 1, NumPy's 0.99999994).

divided_by_count of any other scalar: divided by the count as a T.

divided_by_count of a complex number of float or double, as NumPy's mean divides it: by the complex count n + 0j, in complex double, by its division (numpy_divide), so a NaN part makes both NaN and each part is multiplied by the reciprocal of the count.

Exceptions
dimension_errorIf the count is 0: an integer has no NaN for the mean of nothing.

Definition at line 1403 of file tensor_ops.hpp.

◆ divided_by_zero()

template<std::floating_point R>
R numerical::detail::divided_by_zero ( R  x)
noexcept

x divided by +0, as IEEE divides it, spelled out: a NaN for a zero or a NaN, else an infinity of x's sign (a division by a zero the compiler sees is MSVC's C4723 under /WX).

Definition at line 124 of file tensor_ops.hpp.

◆ dot_chunk_weight()

constexpr std::size_t numerical::detail::dot_chunk_weight ( std::size_t  chunks)
constexprnoexcept

The work chunked_dot weighs one of its chunks chunks, in multiply-adds for parallel_for's split: four an element, each a read of memory's pace, where one kept the product on one thread to 131 072 elements (M1 Pro, 10⁵ elements: 19.6 → 13.2 µs); one an element under three chunks, whose split cost more than it saved (20 000 elements: 4.5 → 5.6 µs).

Definition at line 5053 of file gemm.hpp.

◆ dot_product() [1/5]

double numerical::detail::dot_product ( const double *  x,
const double *  y,
std::size_t  count 
)
inline

Σ x[i] · y[i] for double, as subtract_product chooses its units (in another order of summation there, as fixed).

Definition at line 715 of file gemm.hpp.

◆ dot_product() [2/5]

float numerical::detail::dot_product ( const float *  x,
const float *  y,
std::size_t  count 
)
inline

Σ x[i] · y[i] for float, on the x86-64 units with AVX2 or AVX-512 (avx2_dot, avx512_dot of float), elsewhere the portable vectors' (in another order of summation there, as fixed).

Definition at line 733 of file gemm.hpp.

◆ dot_product() [3/5]

std::complex< double > numerical::detail::dot_product ( const std::complex< double > *  x,
const std::complex< double > *  y,
std::size_t  count 
)
inline

dot_product for std::complex<double>, through complex_dot_product.

Definition at line 766 of file gemm.hpp.

◆ dot_product() [4/5]

std::complex< float > numerical::detail::dot_product ( const std::complex< float > *  x,
const std::complex< float > *  y,
std::size_t  count 
)
inline

dot_product for std::complex<float>, through complex_dot_product.

Definition at line 774 of file gemm.hpp.

◆ dot_product() [5/5]

template<typename T >
T numerical::detail::dot_product ( const T *  x,
const T *  y,
std::size_t  count 
)

Σ x[i] · y[i], in order, for a type no native vector holds.

Definition at line 1076 of file simd.hpp.

◆ double_shift_sweep()

void numerical::detail::double_shift_sweep ( real_qz_form &  f,
std::size_t  ifirst,
std::size_t  ilast,
double  ascale,
double  bscale,
double  safmin 
)
inline

One double-shift QZ sweep (Moler and Stewart) on rows and columns ifirst … ilast of f, as dhgeqz's: the shifts the eigenvalues of the trailing 2 × 2 of S T⁻¹ and its leading one's first column, the bulge chased by reflections of three rows (dlarfg) and of three columns (from the null vector of T's 2 × 2 below the bulge), a rotation pair for the last step.

Definition at line 930 of file generalized.hpp.

◆ double_sine()

template<bool Cosine>
double numerical::detail::double_sine ( double  x)

sin (Cosine false) or cos of a double, as double_sines_into gives it: table_sine to sine_reduction_bound, the C library's past it (and for an infinity or a NaN) and where exp_by_table says no.

Definition at line 1056 of file tensor_ops.hpp.

◆ double_sines_into()

template<bool Cosine>
void numerical::detail::double_sines_into ( const double *  in,
double *  out,
std::size_t  count 
)

sin (Cosine false) or cos of the count doubles at in into out, each double_sine's: table_sine to sine_reduction_bound (with the FMA instructions on an x86-64 processor, run_fused), the C library's past it, element by element (the table's then the library's over every element past it took 1.4 times the library's alone: 10⁶ values to ±10⁷, 12 → 17 ms); the C library's throughout where exp_by_table says no.

Definition at line 1077 of file tensor_ops.hpp.

◆ dqd_transform()

void numerical::detail::dqd_transform ( qd_index  i0,
qd_index  n0,
const qd_array &  z,
qd_index  pp,
double &  dmin,
dqds_state &  state 
)
inline

One dqd transform (no shift) of the block i0…n0 (at least three values), from the half pp of the array into the other, each quotient guarded against underflow and overflow (LAPACK's dlasq6): the transform a shifted one falls back to when its numbers are not finite.

Definition at line 422 of file dqds.hpp.

◆ dqds_shift()

void numerical::detail::dqds_shift ( qd_index  i0,
qd_index  n0,
const qd_array &  z,
qd_index  pp,
qd_index  n0in,
double  dmin,
dqds_state &  state 
)
inline

The shift of the next transform of the block i0…n0 (LAPACK's dlasq4): an estimate of the least eigenvalue from the last transform's least ds, by the case state's ttype records. Some cases return with the shift as it was, where a ratio of the array exceeds one.

Parameters
[in]i0The block's first index.
[in]n0Its last.
[in]zThe qd array.
[in]ppWhich of the ping-pong halves holds the array (0 or 1).
[in]n0inThe last index before this step's deflations.
[in]dminThe last transform's least d.
[in,out]stateThe other least ds; on return, the shift, its type and g.

Definition at line 105 of file dqds.hpp.

◆ dqds_singular_values()

bool numerical::detail::dqds_singular_values ( buffer< double > &  d,
const buffer< double > &  e,
std::size_t  n,
qd_index  sweeps = dqds_sweeps 
)
inline

The singular values of the real upper bidiagonal (d, e) of order n by dqds (LAPACK's dlasq1), descending into d: its entries' magnitudes scaled so that the largest is √(ε / DBL_MIN), their squares' eigenvalues found by dqds_squares, the square roots scaled back.

Parameters
[in,out]dThe diagonal; on success, the singular values, descending. Untouched on failure.
[in]eThe superdiagonal in e[0…n-2].
[in]nThe order, at least one.
[in]sweepsAs dqds_squares takes it.
Returns
Whether dqds converged within that bound.

Definition at line 831 of file dqds.hpp.

◆ dqds_squares()

bool numerical::detail::dqds_squares ( buffer< double > &  z,
qd_index  n,
qd_index  sweeps 
)
inline

The eigenvalues of the positive semi-definite tridiagonal whose qd array is in z (LAPACK's dlasq2): q₁ e₁ q₂ e₂ … qₙ, non-negative, in its first 2n − 1 entries, n at least two.

Parameters
[in,out]zThe array, of at least 4n entries; on success, the eigenvalues in its first n, descending.
[in]nThe order.
[in]sweepsThe steps of the inner iteration per value of its block before giving up.
Returns
Whether the iteration converged within that bound.

Definition at line 660 of file dqds.hpp.

◆ dqds_step()

void numerical::detail::dqds_step ( qd_index  i0,
qd_index &  n0,
const qd_array &  z,
qd_index &  pp,
double &  dmin,
double &  sigma,
double &  desig,
double &  qmax,
dqds_state &  state 
)
inline

One step on the block i0…n0 (LAPACK's dlasq3): deflates the values its end has converged to (into Z(4k − 3), sigma added back), flips the block where its far end is the smaller, then one transform with the shift of dqds_shift. A shift that makes a d negative is lowered and the transform retried; one whose numbers are not finite is dropped, the transform then unshifted, guarded as dqd_transform.

Parameters
[in]i0The block's first index.
[in,out]n0Its last; on return, less the values deflated.
[in]zThe qd array.
[in,out]ppThe half that holds the array (2 on the first step after a flip in dqds_squares).
[in,out]dminThe last transform's least d.
[in,out]sigmaThe shifts so far, summed.
[in,out]desigThe rounding error of that sum.
[in,out]qmaxThe largest q of the block.
[in,out]stateThe rest of the last transform's results and the shift's memory.

Definition at line 545 of file dqds.hpp.

◆ dqds_transform()

void numerical::detail::dqds_transform ( qd_index  i0,
qd_index  n0,
const qd_array &  z,
qd_index  pp,
double &  tau,
double  sigma,
double &  dmin,
dqds_state &  state 
)
inline

One dqds transform of the block i0…n0 (at least three values) with the shift tau, from the half pp of the array into the other (LAPACK's dlasq5, its IEEE form): a zero shift where it is under half of ε (σ + τ), the ds it takes below that bound set to zero. dmin is NaN where the last d is not finite: a d + e of zero, or past overflow, makes every d after it NaN, which a least of them alone does not show.

Definition at line 350 of file dqds.hpp.

◆ each_product()

template<typename T >
tensor< T > numerical::detail::each_product ( const std::string &  sa,
operand_span< T >  a,
const std::string &  sb,
operand_span< T >  b,
const std::string &  output,
const std::map< char, std::size_t > &  sizes 
)

A batch of two-operand products that sum nothing (ij,j->ij, ijk,ik->ijk), each element of the output its operands' product added to 0 as gemm_rows rounds a product's first term (rows_terms_fuse), a complex one as the path's complex product rounds it (complex_terms): the bits of contract_pair's products of depth one. Each operand's axes in the output's order (a copy only where they lie otherwise), broadcast against the other (broadcast_into), where those products went one matrix product of depth one per element of the batch (1000 × 1000 times a row of 1000, six AVX2 cores: 19.7 → 0.5 ms).

Definition at line 1940 of file tensor.hpp.

◆ each_rounded() [1/2]

template<typename T >
requires std::is_arithmetic_v<T>
tensor< T > numerical::detail::each_rounded ( const T *  x,
const std::vector< std::size_t > &  x_shape,
const T *  y,
const std::vector< std::size_t > &  y_shape 
)

The products that sum nothing of each_product, x and y broadcast against each other (broadcast_into), for a real or integer T: a fused first term where gemm_rows fuses one (rows_terms_fuse), else as the compiler rounds 0 + x·y.

Definition at line 1897 of file tensor.hpp.

◆ each_rounded() [2/2]

template<split_complex T>
tensor< T > numerical::detail::each_rounded ( const T *  x,
const std::vector< std::size_t > &  x_shape,
const T *  y,
const std::vector< std::size_t > &  y_shape 
)

each_rounded for complex numbers: as the path's complex product rounds a first term (complex_terms).

Definition at line 1912 of file tensor.hpp.

◆ early_deflation()

template<typename T >
deflation_count numerical::detail::early_deflation ( const schur_target< T > &  target,
lapack_index  ktop,
lapack_index  kbot,
lapack_index  nw,
std::complex< double > *  w,
bool  recursive 
)

Aggressive early deflation, as LAPACK's dlaqr3 and zlaqr3: the trailing nw × nw window of the active block ktop … kbot is brought to Schur form; its eigenvalues whose coupling to the rest (the spike, s times the first row of the window's Schur vectors) is negligible deflate; the window is reduced back to Hessenberg form above them, and its transformation carried to the rest of H and to Z through the matrix product kernel. Every eigenvalue of the window is written to w: the deflated ones final, the others the next shifts. recursive: the window itself by multishift_schur past its crossover.

Definition at line 2279 of file eigen.hpp.

◆ eigenvalue_clusters()

std::vector< std::size_t > numerical::detail::eigenvalue_clusters ( const std::vector< std::complex< double > > &  values,
double  delta 
)
inline

The clusters of values joined where two lie within delta (transitively), numbered by their first member's place.

Definition at line 1875 of file matrix_functions.hpp.

◆ eigenvalue_magnitudes()

template<typename T >
vector< double > numerical::detail::eigenvalue_magnitudes ( const matrix< T > &  a)

The magnitudes of the eigenvalues of the real symmetric or complex Hermitian a, its lower triangle read: its singular values, unordered.

Definition at line 3150 of file svd.hpp.

◆ eighth_inverse()

std::size_t numerical::detail::eighth_inverse ( std::size_t  n)
inlinenoexcept

The inverse of 8 modulo the odd n, in [0, n): by Euclid's algorithm extended.

Definition at line 3329 of file fft.hpp.

◆ einsum_ellipsis()

template<typename T >
tensor< T > numerical::detail::einsum_ellipsis ( const std::string &  spec,
const std::vector< const tensor< T > * > &  operands 
)

einsum_impl of a spec with ellipses, as NumPy reads them: each ... stands for an operand's axes its letters leave, aligned from the right across the operands and broadcast (an extent of 1 repeated), named by letters the spec does not use. An operand's axis of extent 1 under the ellipsis where another's is longer is dropped with its letter, so that operand repeats along it. The output: its ... the broadcast axes (an output without one, where an operand has such axes, is refused, as NumPy refuses it); with no ->, the broadcast axes then the letters that appear once, in order.

Definition at line 2297 of file tensor.hpp.

◆ einsum_impl()

template<typename T >
tensor< T > numerical::detail::einsum_impl ( const std::string &  spec_in,
const std::vector< const tensor< T > * > &  operands 
)

Einstein summation of operands as spec_in reads it (einsum): einsum_spec.

einsum_impl of a signed integer type: through its unsigned type, modulo 2ⁿ the same bits, so that an overflow wraps as NumPy's integers wrap it, where the signed operation is undefined.

Definition at line 2717 of file tensor.hpp.

◆ einsum_pair()

template<typename T >
tensor< T > numerical::detail::einsum_pair ( const std::string &  s0,
operand_span< T >  a,
const std::string &  s1,
operand_span< T >  b,
const std::string &  output,
const std::string &  summed,
const std::map< char, std::size_t > &  sizes 
)

einsum of two operands a (labels s0) and b (s1) into output, summed the labels neither the output carries (ascending), sizes each label's extent: each label of an operand once, and each in the other operand or the output (einsum_impl sums an operand's own labels and reads a repeated one's diagonal first). A tensor's elements or an array's read where it lies, the same bits either way.

Definition at line 2242 of file tensor.hpp.

◆ einsum_spec()

template<typename T >
tensor< T > numerical::detail::einsum_spec ( const std::string &  spec_in,
const std::vector< const tensor< T > * > &  operands 
)

einsum_impl of a type whose sums and products are defined past its range: a float, a complex, an unsigned integer.

Definition at line 2429 of file tensor.hpp.

◆ element_count()

template<typename It >
std::size_t numerical::detail::element_count ( It  first,
It  last 
)

The elements a shape's extents [first, last) hold: 0 where one of them is 0, whatever the others, else their product (checked_mul). The extents a tensor keeps, past one it holds 0 of, may multiply past std::size_t: a result of them is empty, not of a wrapped size.

Exceptions
dimension_errorIf none is 0 and their product overflows.

Definition at line 151 of file vector.hpp.

◆ elementary_atan() [1/3]

template<typename T >
T numerical::detail::elementary_atan ( const T &  x)

atan of a scalar, as std::atan gives it.

Definition at line 498 of file tensor_ops.hpp.

◆ elementary_atan() [2/3]

double numerical::detail::elementary_atan ( double  x)
inline

atan of a double, as atan_into gives it: table_atan, or std::atan where exp_by_table says no.

Definition at line 505 of file tensor_ops.hpp.

◆ elementary_atan() [3/3]

float numerical::detail::elementary_atan ( float  x)
inline

atan of a float, as atan_of gives it: elementary_atan of the double, rounded once.

Definition at line 569 of file tensor_ops.hpp.

◆ elementary_cos() [1/3]

template<typename T >
T numerical::detail::elementary_cos ( const T &  x)

cos of a scalar, as std::cos gives it.

cos of a complex number of float or double: complex_cos.

Definition at line 970 of file tensor_ops.hpp.

◆ elementary_cos() [2/3]

double numerical::detail::elementary_cos ( double  x)
inline

cos of a double, as double_sines_into gives it (double_sine).

Definition at line 1101 of file tensor_ops.hpp.

◆ elementary_cos() [3/3]

float numerical::detail::elementary_cos ( float  x)
inline

cos of a float, as float_sines_into gives it: float_sine of the double, rounded once.

Definition at line 1113 of file tensor_ops.hpp.

◆ elementary_cosh() [1/3]

template<typename T >
T numerical::detail::elementary_cosh ( const T &  x)

cosh of a scalar, as std::cosh gives it.

Definition at line 670 of file tensor_ops.hpp.

◆ elementary_cosh() [2/3]

double numerical::detail::elementary_cosh ( double  x)
inline

cosh of a double, as cosh_into gives it: table_cosh, or std::cosh where exp_by_table says no.

Definition at line 677 of file tensor_ops.hpp.

◆ elementary_cosh() [3/3]

float numerical::detail::elementary_cosh ( float  x)
inline

cosh of a float, as cosh_of gives it: elementary_cosh of the double, rounded once.

Definition at line 741 of file tensor_ops.hpp.

◆ elementary_exp() [1/2]

template<typename T >
T numerical::detail::elementary_exp ( const T &  x)

exp of a scalar, as std::exp gives it.

exp of a complex number of float or double: complex_exp.

Definition at line 325 of file tensor_ops.hpp.

◆ elementary_exp() [2/2]

double numerical::detail::elementary_exp ( double  x)
inline

exp of a double, as exp_into gives it: table_exp, or std::exp below exp_normal_bound (and throughout where exp_by_table says no).

Definition at line 349 of file tensor_ops.hpp.

◆ elementary_exp2() [1/2]

template<typename T >
T numerical::detail::elementary_exp2 ( const T &  x)

exp2 of a scalar, as std::exp2 gives it.

Definition at line 759 of file tensor_ops.hpp.

◆ elementary_exp2() [2/2]

double numerical::detail::elementary_exp2 ( double  x)
inline

exp2 of a double, as exp2_into gives it: table_exp2, or std::exp2 below exp2_normal_bound (and throughout where exp_by_table says no).

Definition at line 766 of file tensor_ops.hpp.

◆ elementary_expm1() [1/3]

template<typename T >
T numerical::detail::elementary_expm1 ( const T &  x)

expm1 of a scalar, as std::expm1 gives it.

Definition at line 412 of file tensor_ops.hpp.

◆ elementary_expm1() [2/3]

double numerical::detail::elementary_expm1 ( double  x)
inline

expm1 of a double, as expm1_into gives it: table_expm1, or std::expm1 where exp_by_table says no.

Definition at line 419 of file tensor_ops.hpp.

◆ elementary_expm1() [3/3]

float numerical::detail::elementary_expm1 ( float  x)
inline

expm1 of a float, as expm1_of gives it: elementary_expm1 of the double, rounded once.

Definition at line 483 of file tensor_ops.hpp.

◆ elementary_log()

template<typename T >
T numerical::detail::elementary_log ( const T &  x)

log of a scalar, as std::log gives it.

log of a complex number of float or double: complex_log.

Definition at line 856 of file tensor_ops.hpp.

◆ elementary_log1p() [1/3]

template<typename T >
T numerical::detail::elementary_log1p ( const T &  x)

log1p of a scalar, as std::log1p gives it.

Definition at line 870 of file tensor_ops.hpp.

◆ elementary_log1p() [2/3]

double numerical::detail::elementary_log1p ( double  x)
inline

log1p of a double, as log1p_into gives it: table_log1p, or std::log1p where exp_by_table says no.

Definition at line 877 of file tensor_ops.hpp.

◆ elementary_log1p() [3/3]

float numerical::detail::elementary_log1p ( float  x)
inline

log1p of a float, as log1p_of gives it: elementary_log1p of the double, rounded once.

Definition at line 941 of file tensor_ops.hpp.

◆ elementary_sin() [1/3]

template<typename T >
T numerical::detail::elementary_sin ( const T &  x)

sin of a scalar, as std::sin gives it.

sin of a complex number of float or double: complex_sin.

Definition at line 956 of file tensor_ops.hpp.

◆ elementary_sin() [2/3]

double numerical::detail::elementary_sin ( double  x)
inline

sin of a double, as double_sines_into gives it (double_sine).

Definition at line 1095 of file tensor_ops.hpp.

◆ elementary_sin() [3/3]

float numerical::detail::elementary_sin ( float  x)
inline

sin of a float, as float_sines_into gives it: float_sine of the double, rounded once.

Definition at line 1107 of file tensor_ops.hpp.

◆ elementary_sinh() [1/3]

template<typename T >
T numerical::detail::elementary_sinh ( const T &  x)

sinh of a scalar, as std::sinh gives it.

Definition at line 584 of file tensor_ops.hpp.

◆ elementary_sinh() [2/3]

double numerical::detail::elementary_sinh ( double  x)
inline

sinh of a double, as sinh_into gives it: table_sinh, or std::sinh where exp_by_table says no.

Definition at line 591 of file tensor_ops.hpp.

◆ elementary_sinh() [3/3]

float numerical::detail::elementary_sinh ( float  x)
inline

sinh of a float, as sinh_of gives it: elementary_sinh of the double, rounded once.

Definition at line 655 of file tensor_ops.hpp.

◆ elementary_tan()

template<typename T >
T numerical::detail::elementary_tan ( const T &  x)

tan of a scalar, as std::tan gives it.

tan of a complex number of float or double: complex_tan.

Definition at line 1180 of file tensor_ops.hpp.

◆ elementary_tanh() [1/3]

template<typename T >
T numerical::detail::elementary_tanh ( const T &  x)

tanh of a scalar, as std::tanh gives it.

tanh of a complex number of float or double: complex_tanh.

Definition at line 1194 of file tensor_ops.hpp.

◆ elementary_tanh() [2/3]

double numerical::detail::elementary_tanh ( double  x)
inline

tanh of a double, as tanh_into gives it: table_tanh, or std::tanh where exp_by_table says no.

Definition at line 1208 of file tensor_ops.hpp.

◆ elementary_tanh() [3/3]

float numerical::detail::elementary_tanh ( float  x)
inline

tanh of a float, as tanh_of gives it: elementary_tanh of the double, rounded once.

Definition at line 1272 of file tensor_ops.hpp.

◆ elements_finite() [1/3]

bool numerical::detail::elements_finite ( const double *  x,
std::size_t  count 
)
inline

elements_finite of double: all_finite_shared, vectorised.

Definition at line 1987 of file factorizations.hpp.

◆ elements_finite() [2/3]

bool numerical::detail::elements_finite ( const std::complex< double > *  x,
std::size_t  count 
)
inline

The same for std::complex<double>, whose array holds its parts in order ([complex.numbers]).

Definition at line 1994 of file factorizations.hpp.

◆ elements_finite() [3/3]

template<typename T >
bool numerical::detail::elements_finite ( const T *  x,
std::size_t  count 
)

Whether the count elements at x are finite, one at a time: a type no vectorised pass takes.

Definition at line 1980 of file factorizations.hpp.

◆ eliminate_pivot() [1/2]

void numerical::detail::eliminate_pivot ( buffer< double > &  a,
std::size_t  n,
std::size_t  k,
std::size_t  step,
bool  ,
buffer< double > &  w 
)
inline

A real matrix's update by a pivot of order 1 or 2 (step), sytf2's whatever hermitian.

Definition at line 194 of file ldl.hpp.

◆ eliminate_pivot() [2/2]

void numerical::detail::eliminate_pivot ( buffer< std::complex< double > > &  a,
std::size_t  n,
std::size_t  k,
std::size_t  step,
bool  hermitian,
buffer< std::complex< double > > &  w 
)
inline

A complex matrix's update by a pivot of order 1 or 2 (step): hetf2's if hermitian, else sytf2's.

Definition at line 210 of file ldl.hpp.

◆ ellipsis_in()

std::size_t numerical::detail::ellipsis_in ( const std::string &  subscripts)
inline

Evaluates a parsed einsum over its operands.

See also
numerical::einsum

After parsing and the consistency checks, the contraction runs over a single integer index space — the output (free) labels first, then the summed labels. Each operand's flat offset is carried incrementally by a mixed-radix odometer using a per-label stride coefficient computed once, so the hot loop allocates nothing and performs no map lookups; its innermost reduction is a strided accumulation the compiler can auto-vectorise. (Pairwise contraction-path optimisation, as in opt_einsum, and a BLAS reduction are deliberately not included — they would trade the library's generic simplicity and zero dependencies for raw throughput.)

The position of the ellipsis in subscripts, or npos where there is none: ‘’.'only as one.... \throws einsum_error If a'.'` stands otherwise.

Definition at line 2225 of file tensor.hpp.

◆ entries_finite()

bool numerical::detail::entries_finite ( std::initializer_list< const matrix< std::complex< double > > * >  xs)
inlineprivate

Whether every entry of the matrices xs is finite.

Definition at line 2999 of file generalized.hpp.

◆ entry_moduli() [1/2]

buffer< double > numerical::detail::entry_moduli ( const buffer< double > &  a)
inline

The moduli of a real matrix's entries.

Definition at line 35 of file matrix_functions.hpp.

◆ entry_moduli() [2/2]

buffer< double > numerical::detail::entry_moduli ( const buffer< std::complex< double > > &  a)
inline

The moduli of a complex matrix's entries: √(x² + y²), which vectorises, where the largest part lies in [2⁻⁵⁰⁰, 2⁵⁰⁰] (no square overflows; one that underflows is of an entry below 2⁻⁵⁰⁰ times the largest's, nothing to a norm); hypot otherwise, as std::abs takes it.

Definition at line 47 of file matrix_functions.hpp.

◆ exact_triangular_bands()

template<typename T >
void numerical::detail::exact_triangular_bands ( buffer< T > &  x,
const buffer< T > &  t,
std::size_t  n,
double  scale 
)

The diagonal and first superdiagonal of exp(scale · T) written exactly into x for the upper triangular t, as Al-Mohy and Higham's code fragment 2.1: e^{λᵢ}, and t_{i,i+1} e^{(λᵢ + λᵢ₊₁)/2} sinch((λᵢ − λᵢ₊₁)/2); past divided_exponentials_past, t_{i,i+1} (e^{λᵢ} − e^{λᵢ₊₁}) / (λᵢ − λᵢ₊₁), where that difference overflows each exponential divided by the gap's modulus within it (e^{λ − log|g|}), so that the band overflows only where it is past the largest double.

Definition at line 282 of file matrix_functions.hpp.

◆ exactly_hermitian()

template<typename T >
bool numerical::detail::exactly_hermitian ( const matrix< T > &  a)

Whether a is Hermitian to the bit (symmetric where real), its diagonal real: a function of it is then V f(Λ) Vᴴ from one eigendecomposition (Armadillo's expmat_sym, logmat_sympd, sqrtmat_sympd).

Definition at line 314 of file matrix_functions.hpp.

◆ exceptional_shifts() [1/2]

void numerical::detail::exceptional_shifts ( one_based< double >  h,
std::complex< double > *  w,
lapack_index  ks,
lapack_index  kbot,
lapack_index  ktop 
)
inline

Exceptional shifts for a real matrix, as dlaqr0's after kexsh iterations without deflation: from the subdiagonal entries, as conjugate pairs of the 2 × 2 blocks dlahqr's exceptional shifts build.

Definition at line 2454 of file eigen.hpp.

◆ exceptional_shifts() [2/2]

void numerical::detail::exceptional_shifts ( one_based< std::complex< double > >  h,
std::complex< double > *  w,
lapack_index  ks,
lapack_index  kbot,
lapack_index   
)
inline

Exceptional shifts for a complex matrix, as zlaqr0's.

Definition at line 2475 of file eigen.hpp.

◆ exchange()

template<typename T >
void numerical::detail::exchange ( buffer< T > &  a,
std::size_t  n,
std::size_t  i,
std::size_t  j 
)

Exchanges rows i and j, then columns i and j, of the row-major n × n a: a similarity.

Definition at line 109 of file eigen.hpp.

◆ exchange_blocks()

bool numerical::detail::exchange_blocks ( one_based< double >  t,
one_based< double >  qt,
lapack_index  n,
lapack_index  j1,
lapack_index  n1,
lapack_index  n2 
)
inline

Exchanges the adjacent diagonal blocks of the real quasi-triangular t at j1, of orders n1 then n2, by an orthogonal similarity carried to the rows of qt, as LAPACK's dlaexc.

Returns
Whether it did: a swap that would perturb the blocks past 10 ε ‖D‖ is refused and nothing changes.

Definition at line 1512 of file eigen.hpp.

◆ exchange_diagonal()

template<typename T >
void numerical::detail::exchange_diagonal ( one_based< T >  t,
one_based< T >  qt,
lapack_index  n,
lapack_index  k 
)

Exchanges the diagonal entries k and k + 1 of the upper triangular (or quasi-triangular, both 1 × 1 there) t by a plane rotation, as LAPACK's ztrexc step and dlaexc for two 1 × 1 blocks; the rows of qt (Qᵀ) turn with it.

Definition at line 1368 of file eigen.hpp.

◆ exchange_or_refuse()

void numerical::detail::exchange_or_refuse ( real_qz_form &  f,
std::size_t  j1,
std::size_t  n1,
std::size_t  n2 
)
inline

exchange_real_blocks, a refusal thrown as real_generalized_schur::reorder reports it.

Definition at line 2035 of file generalized.hpp.

◆ exchange_pencil_diagonal()

bool numerical::detail::exchange_pencil_diagonal ( qz_form &  f,
std::size_t  j 
)
inline

Exchanges the diagonal entries j and j + 1 (from 1) of the triangular pencil f, as LAPACK's ztgex2: columns j, j + 1 rotated onto the right eigenvector of the second eigenvalue, then rows j, j + 1 back to triangular, the rotation from S's new first column where |S₂₂ T₁₁| ≥ |S₁₁ T₂₂|, else T's; Z and Q turning with them. The diagonal of T is then complex: standardize takes it back. Tried on the 2 × 2 blocks first and refused, f unchanged, unless it passes ztgex2's weak test (the new subdiagonal entries within 20 ε of the blocks' norms) and strong test (the blocks rebuilt from the rotations within the same bounds).

Returns
Whether the exchange was made.

Definition at line 423 of file generalized.hpp.

◆ exchange_real_blocks()

bool numerical::detail::exchange_real_blocks ( real_qz_form &  f,
std::size_t  j1,
std::size_t  n1,
std::size_t  n2 
)
inline

Exchanges the adjacent diagonal blocks of orders n1 and n2 (1 or 2) at j1 (from 1) of the real generalized Schur form f, as LAPACK's dtgex2: planned by plan_single_exchange or plan_block_exchange, then refused unless the blocks rebuilt from the transforms, ‘LI S’ IRᵀ, are within 20 ε‘ of the originals’ norms (the strong test, after the 2 × 2 blocks' standardization where dtgex2 makes it before: their rotations' rounding more); else the entries below the new blocks set to 0 and the rows and columns of S, T, Q and Z turned.

Returns
Whether the exchange was made, f unchanged where not.

Definition at line 1941 of file generalized.hpp.

◆ exp2_into()

void numerical::detail::exp2_into ( const double *  in,
double *  out,
std::size_t  count 
)
inline

exp2 of the count doubles at in into out, each elementary_exp2's: table_exp2 in vectors (with the FMA instructions on an x86-64 processor, run_fused), then std::exp2 below exp2_normal_bound, as exp_into; std::exp2 throughout where exp_by_table says no.

Definition at line 781 of file tensor_ops.hpp.

◆ exp2_of() [1/2]

tensor< double > numerical::detail::exp2_of ( const double *  data,
const std::vector< std::size_t > &  shape 
)
inline

exp2 of every one of the row-major doubles at data, of shape shape: exp2_into by chunks of elementwise_chunk among the threads.

Definition at line 834 of file tensor_ops.hpp.

◆ exp2_of() [2/2]

template<typename T >
tensor< T > numerical::detail::exp2_of ( const T *  data,
const std::vector< std::size_t > &  shape 
)

exp2 of every one of the row-major scalars at data, of shape shape: std::exp2 of each.

Definition at line 826 of file tensor_ops.hpp.

◆ exp2_viewed() [1/2]

tensor< double > numerical::detail::exp2_viewed ( const tensor< double > &  a)
inline

exp2 of every double of a, a view read where it lies (by_runs).

Definition at line 2567 of file tensor_ops.hpp.

◆ exp2_viewed() [2/2]

template<typename T >
tensor< T > numerical::detail::exp2_viewed ( const tensor< T > &  a)

exp2 of every element of a: of a contiguous copy (exp2_of).

Definition at line 2561 of file tensor_ops.hpp.

◆ exp_into()

void numerical::detail::exp_into ( const double *  in,
double *  out,
std::size_t  count 
)
inline

exp of the count doubles at in into out, each elementary_exp's: table_exp in vectors (with the FMA instructions on an x86-64 processor, run_fused), then std::exp where the result is subnormal or zero, below exp_normal_bound, which the table's two scalings would round twice; std::exp throughout where exp_by_table says no.

Definition at line 365 of file tensor_ops.hpp.

◆ exp_of() [1/2]

tensor< double > numerical::detail::exp_of ( const double *  data,
const std::vector< std::size_t > &  shape 
)
inline

exp of every one of the row-major doubles at data, of shape shape: exp_into by chunks of elementwise_chunk among the threads.

Definition at line 2337 of file tensor_ops.hpp.

◆ exp_of() [2/2]

template<typename T >
tensor< T > numerical::detail::exp_of ( const T *  data,
const std::vector< std::size_t > &  shape 
)

exp of every one of the row-major elements at data, of shape shape: elementary_exp of each.

Definition at line 2329 of file tensor_ops.hpp.

◆ exp_viewed() [1/2]

tensor< double > numerical::detail::exp_viewed ( const tensor< double > &  a)
inline

exp of every double of a, a view read where it lies (by_runs).

Definition at line 2478 of file tensor_ops.hpp.

◆ exp_viewed() [2/2]

template<typename T >
tensor< T > numerical::detail::exp_viewed ( const tensor< T > &  a)

exp of every element of a: of a contiguous copy (exp_of).

Definition at line 2472 of file tensor_ops.hpp.

◆ expm1_into()

void numerical::detail::expm1_into ( const double *  in,
double *  out,
std::size_t  count 
)
inline

expm1 of the count doubles at in into out, each elementary_expm1's: table_expm1 in vectors (with the FMA instructions on an x86-64 processor, run_fused), std::expm1 throughout where exp_by_table says no.

Definition at line 432 of file tensor_ops.hpp.

◆ expm1_of() [1/3]

tensor< double > numerical::detail::expm1_of ( const double *  data,
const std::vector< std::size_t > &  shape 
)
inline

expm1 of every one of the row-major doubles at data, of shape shape: expm1_into by chunks of elementwise_chunk among the threads.

Definition at line 469 of file tensor_ops.hpp.

◆ expm1_of() [2/3]

tensor< float > numerical::detail::expm1_of ( const float *  data,
const std::vector< std::size_t > &  shape 
)
inline

expm1 of every one of the row-major floats at data, of shape shape: expm1_into of them as doubles (through_doubles).

Definition at line 490 of file tensor_ops.hpp.

◆ expm1_of() [3/3]

template<typename T >
tensor< T > numerical::detail::expm1_of ( const T *  data,
const std::vector< std::size_t > &  shape 
)

expm1 of every one of the row-major scalars at data, of shape shape: std::expm1 of each.

Definition at line 461 of file tensor_ops.hpp.

◆ expm1_parts()

std::int64_t numerical::detail::expm1_parts ( double  clamped,
double &  sum,
double &  small 
)
inlinenoexcept

expm1(clamped) in units of 2^e, as sum + small unrounded, and e: table_exp's reduction, x = (128 e + j) ln 2/128 + r, then 2^(j/128) − 2^−e + 2^(j/128) r_head, each sum and product of them exact (two-sums and a fused product's error), into sum, and the small rest, 2^(j/128) (r − r_head + exp(r) − 1 − r) and the table's tail times exp(r), with the two-sums' errors, into small. For clamped from −40 to 710.

Definition at line 239 of file elementary.hpp.

◆ expm1_viewed() [1/3]

tensor< double > numerical::detail::expm1_viewed ( const tensor< double > &  a)
inline

expm1 of every double of a, a view read where it lies (by_runs).

Definition at line 2491 of file tensor_ops.hpp.

◆ expm1_viewed() [2/3]

tensor< float > numerical::detail::expm1_viewed ( const tensor< float > &  a)
inline

expm1 of every float of a through doubles, a view read where it lies (by_runs).

Definition at line 2497 of file tensor_ops.hpp.

◆ expm1_viewed() [3/3]

template<typename T >
tensor< T > numerical::detail::expm1_viewed ( const tensor< T > &  a)

expm1 of every element of a: of a contiguous copy (expm1_of).

Definition at line 2485 of file tensor_ops.hpp.

◆ extent_product()

double numerical::detail::extent_product ( const std::string &  labels,
const std::map< char, std::size_t > &  sizes 
)
inline

The product of the extents of labels' distinct labels, as a double (a cost: it may pass size_t).

Definition at line 2093 of file tensor.hpp.

◆ extra_squarings()

std::size_t numerical::detail::extra_squarings ( const buffer< double > &  moduli,
std::size_t  n,
std::size_t  m,
double  scale,
double  norm 
)
inline

Al-Mohy and Higham's ℓ(A, m) (2009): the squarings to add so that the [m/m] approximant's backward error bound, from the leading term of its series |c₂ₘ₊₁| ‖ |A|²ᵐ⁺¹ ‖₁ / ‖A‖₁, falls below the unit roundoff; A times scale (a power of two), its norms exact, from its moduli (|A|, formed once for every degree tried: a complex entry's modulus is a hypot).

Definition at line 171 of file matrix_functions.hpp.

◆ extreme() [1/2]

template<std::floating_point T, typename Order >
T numerical::detail::extreme ( const T *  first,
std::size_t  count,
std::size_t  stride,
const Order &  larger 
)

The largest (larger is std::greater) or smallest element of a line of reals: a contiguous one by the comparison without a branch (so it vectorises) and the NaN tracked apart, a NaN anywhere the answer, as NumPy's; a strided one walked (walked_extreme).

Definition at line 1833 of file tensor_ops.hpp.

◆ extreme() [2/2]

template<typename T , typename Order >
requires (!std::floating_point<T>)
T numerical::detail::extreme ( const T *  first,
std::size_t  count,
std::size_t  stride,
const Order &  larger 
)

The largest (larger is std::greater) or smallest element of a line of another type (walked_extreme).

Definition at line 1852 of file tensor_ops.hpp.

◆ extreme_index()

template<std::floating_point T, typename Order >
std::int64_t numerical::detail::extreme_index ( const T *  first,
std::size_t  count,
const Order &  larger 
)

The index of the first largest (larger std::greater) or smallest of count contiguous elements, as NumPy's: the first NaN where there is one. Reals in two passes without a branch each, as fast as NumPy's vectors: their extreme (contiguous_extreme, a NaN where there is one), then the first element that is it (zeros of both signs equal, so the first of either, as NumPy's).

extreme_index of integers: the first that none after it passes.

Definition at line 4279 of file tensor_ops.hpp.

◆ extreme_indices()

template<typename T , typename Order >
tensor< std::int64_t > numerical::detail::extreme_indices ( const tensor< T > &  t,
std::size_t  axis,
const Order &  larger 
)

argmax or argmin along axis by larger.

Definition at line 4309 of file tensor_ops.hpp.

◆ extremes_along()

template<typename T , typename Order >
tensor< T > numerical::detail::extremes_along ( const tensor< T > &  t,
std::size_t  axis,
const Order &  larger 
)

The largest (larger is std::greater) or smallest elements along axis. Along an inner axis each line is contiguous (extreme); along an outer one the slabs are compared element by element, so the loop runs over contiguous memory rather than down a column.

Definition at line 1879 of file tensor_ops.hpp.

◆ factor_ldl_panel()

template<typename T >
std::size_t numerical::detail::factor_ldl_panel ( buffer< T > &  a,
std::size_t  n,
std::size_t  k0,
bool  hermitian,
bool  last_on_ties,
buffer< T > &  w,
std::vector< std::ptrdiff_t > &  pivots 
)

One panel of bunch_kaufman_blocked_lower from column k0, as LAPACK's lasyf and lahef on the lower triangle: each column brought up to date by the panel's earlier ones (A − L Wᴴ, W = L D in w, n × ldl_block row-major, conjugated for a Hermitian matrix) before its pivot is chosen as bunch_kaufman_lower chooses it, the trailing block left as it was but for the exchanges of its rows and columns, its update deferred. ldl_block − 1 columns, or ldl_block where the last is a block of order 2; more than ldl_block columns remain.

Returns
The column after the panel's last.

Definition at line 341 of file ldl.hpp.

◆ factor_qr_panel()

template<typename T >
void numerical::detail::factor_qr_panel ( T *  panel,
std::size_t  ld,
std::size_t  top,
std::vector< buffer< T > > &  reflectors,
std::vector< T > &  taus,
std::size_t  k0,
std::size_t  k1 
)
private

Householder reflections of the columns k0…k1-1 of a column-major matrix, rows from k0, one by one, in place, into reflectors[k0…k1-1] and taus: panel holds column k0, ld elements per column from row top (0 in the whole matrix, k0 in a copy of the panel). A reflector meets each column at unit stride. Recursive halves, the left's reflectors applied to the right at once, took longer at these widths (32 columns).

Each as LAPACK's dlarfg (zlarfg) forms it: H = I − τ w wᴴ, w₀ = 1, with Hᴴ x = β e₁ for a real β of the sign opposite to Re x₀; the identity (τ = 0, an empty reflector) where nothing lies below the diagonal and x₀ is real. Exact inputs stay exact where LAPACK's do (e₃ reflected onto e₁: w = (1, 0, 1), τ = 1; a unit v there holds 1/√2, rounded), and the factors are LAPACK's to rounding. β goes on the diagonal and w₁… below it, where dgeqrf leaves them.

Definition at line 3266 of file factorizations.hpp.

◆ factor_row_major_panel()

template<typename T >
void numerical::detail::factor_row_major_panel ( buffer< T > &  a,
std::size_t  rows,
std::size_t  cols,
std::vector< buffer< T > > &  reflectors,
std::vector< T > &  taus,
std::size_t  k0,
std::size_t  k1 
)
private

factor_qr_panel on the columns k0…k1-1 of the row-major a, rows from k0, in a column-major copy of the panel, copied back. A function of its own: written into reflect_in_rows' loop, the same instructions took 5 % more cycles at order 128 (clang, arm64).

Definition at line 3412 of file factorizations.hpp.

◆ factor_shifted_tridiagonal()

void numerical::detail::factor_shifted_tridiagonal ( const double *  diagonal,
const double *  off,
std::size_t  n,
double  lambda,
shifted_tridiagonal &  f 
)
inlineprivate

Factorises T − λ I, n from 2, T of diagonal diagonal and off-diagonal off, with partial pivoting as LAPACK's dlagtf: at each step the row of the larger entry relative to its row's size (the first where they tie).

Definition at line 2560 of file svd.hpp.

◆ fast_two_sum()

double_double numerical::detail::fast_two_sum ( double  a,
double  b 
)
inlinenoexcept

a + b exactly where |a| ≥ |b| or a is 0 (Dekker's fast two-sum).

Definition at line 81 of file fft.hpp.

◆ fault_chunk()

template<typename T >
constexpr std::size_t numerical::detail::fault_chunk ( std::size_t  count)
constexprnoexcept

The elements one index of parallel_zero or parallel_copy takes of count: zero_chunk, or, from the size at which a buffer takes Linux's 2 MiB pages (16 MiB), one such page's worth, so that no two threads fault in the same page: they waited on each other's fault there (six x86-64 AVX2 cores, 32 MiB zeroed into fresh memory 3.6 → 2.5 ms; the LU and Cholesky of order 2000, their operand copied by eight 4 KiB pages a thread, took 2 to 4 % longer than with one thread's copy).

Definition at line 863 of file parallel.hpp.

◆ few_complex_rows()

bool numerical::detail::few_complex_rows ( std::size_t  m,
std::size_t  n,
std::size_t  k 
)
inlinenoexcept

Whether gemm runs a complex product of m rows, B k × n, a few rows at a time with B read by rows (few_rows_complex_x86, few_rows_complex_neon), on a path past the portable one.

Definition at line 4539 of file gemm.hpp.

◆ few_rows()

template<std::size_t Width, term_rounding Mode, typename T >
void numerical::detail::few_rows ( std::size_t  m,
std::size_t  n,
std::size_t  k,
const T *  a,
std::size_t  lda,
const T *  b,
std::size_t  ldb,
T *  c,
std::size_t  ldc,
c_start  start 
)

C += A·B (C = A·B as start says) for m rows of A, by groups of at most narrow_group rows and blocks of Width columns of C, each block its sums in registers over the whole depth with B read where it lies (narrow_block), the blocks of every group shared among the threads; the last columns by narrow_rows. Each element the fused chain of the tile, rounded as Mode says, so its bits.

Definition at line 3516 of file gemm.hpp.

◆ fft_axis_index()

std::size_t numerical::detail::fft_axis_index ( std::ptrdiff_t  axis,
std::size_t  rank,
const char *  who 
)
inline

axis of a tensor of rank rank from 0, NumPy's negative axes read from the end.

Exceptions
dimension_errorIf axis is not an axis.

Definition at line 2819 of file fft.hpp.

◆ fft_finish_line()

template<typename C >
void numerical::detail::fft_finish_line ( C *  line,
std::size_t  points,
bool  inverse,
typename C::value_type  scale 
)
noexcept

A transformed line of points elements, line, conjugated back where inverse, then times scale where it is not 1; by value for the reason fft_load_line gives.

Definition at line 2297 of file fft.hpp.

◆ fft_fuses()

bool numerical::detail::fft_fuses ( )
inlinenoexcept

Whether a transform's butterflies fuse their products (fused_product): on every path past the portable.

Definition at line 1711 of file fft.hpp.

◆ fft_lane_groups()

template<std::floating_point R>
std::size_t numerical::detail::fft_lane_groups ( const fft_plan< R > &  ,
bool  ,
std::size_t  ,
bool   
)
noexcept

No registers of lines.

Definition at line 2096 of file fft.hpp.

◆ fft_lines_together()

template<typename T , std::floating_point R>
void numerical::detail::fft_lines_together ( const fft_plan< R > &  ,
const T *  ,
std::size_t  ,
std::size_t  ,
std::size_t  ,
std::complex< R > *  ,
std::size_t  ,
std::size_t  ,
bool  ,
R   
)
noexcept

Never called: no registers of lines.

Definition at line 2107 of file fft.hpp.

◆ fft_load_line()

template<typename C >
void numerical::detail::fft_load_line ( const C *  in,
std::size_t  kept,
std::size_t  points,
bool  inverse,
C *  to 
)
noexcept

The first kept of points elements of a line, in, into to, conjugated where inverse, then zeros. Its bounds and flags by value: read through a caller's references, each would be loaded again at every element, the writes to to possibly aliasing them, and the loop would not vectorise.

Definition at line 2279 of file fft.hpp.

◆ fft_radices()

std::vector< std::size_t > numerical::detail::fft_radices ( std::size_t  n)
inline

The radix of each Stockham pass of n, or none where n has a prime factor past fft_largest_radix, its powers of two first (4 first, its stride 1, then 16 while 16 points remain, then 8 or 4), then its factors 3, 5 and 7, then each other prime in ascending order, then a lone factor 2 last, where its pass has no factor.

Definition at line 814 of file fft.hpp.

◆ fft_scale()

template<std::floating_point R>
R numerical::detail::fft_scale ( fft_norm  norm,
bool  inverse,
std::size_t  n 
)

The scale of a transform's output for norm, forward or not, of length n: 1 where none.

Definition at line 1986 of file fft.hpp.

◆ fft_starts_in_work()

template<std::floating_point R>
bool numerical::detail::fft_starts_in_work ( const fft_plan< R > &  plan)
noexcept

Whether a transform under plan, its input placed in its working space (forward_fft's in_work), ends in its data without a copy: an odd number of Stockham passes, no Bluestein's convolution nor four steps.

Definition at line 949 of file fft.hpp.

◆ first_held()

template<typename T , typename Match >
std::size_t numerical::detail::first_held ( const T *  first,
std::size_t  count,
const Match &  match 
)
noexcept

The index of the first of the contiguous elements at first that passes match, which one of them does: blocks of 64 tested by holds, then the block that holds it walked.

Definition at line 4257 of file tensor_ops.hpp.

◆ float_sine()

template<bool Cosine>
double numerical::detail::float_sine ( double  x)

sin (Cosine false) or cos of a double that a float holds: reduced_sine to sine_reduction_bound, the C library's past it (and for an infinity or a NaN).

Definition at line 1011 of file tensor_ops.hpp.

◆ float_sines_into()

template<bool Cosine>
void numerical::detail::float_sines_into ( const double *  in,
double *  out,
std::size_t  count 
)

sin (Cosine false) or cos of the count doubles at in, each a float's, into out, each float_sine's: reduced_sine in vectors (with the FMA instructions on an x86-64 processor, run_fused), then the C library's past sine_reduction_bound, behind a branch; the C library's throughout where exp_by_table says no.

Definition at line 1026 of file tensor_ops.hpp.

◆ floor_divmod()

template<std::floating_point T>
std::pair< T, T > numerical::detail::floor_divmod ( T  a,
T  b 
)
noexcept

a // b and a % b of reals as NumPy's divmod computes them (Python's: the quotient floored, the remainder of b's sign): fmod first, the quotient (a − mod) / b adjusted by the remainder's sign and snapped to the nearest integer, a zero of the right sign. b zero: a / b and fmod(a, b).

a // b and a % b of signed integers as NumPy's: the quotient floored, the remainder of b's sign; by zero both 0 (NumPy's answer, beside its warning), and the most negative value by −1 its own negation (0 remainder), wrapped where C++'s would be undefined.

Definition at line 3064 of file tensor_ops.hpp.

◆ fold_along()

template<typename R , typename T , typename Step >
tensor< R > numerical::detail::fold_along ( const tensor< T > &  t,
std::size_t  axis,
const R &  init,
const Step &  step 
)

Folds each line of t along axis in order, step(running, x) from init: a tensor of the result type R, the axis taken out. Where the lines are columns (something follows the axis), each task takes a band of up to band_columns of them and walks it row by row, contiguously; each line's steps are the same, in the same order, either way.

Definition at line 1518 of file tensor_ops.hpp.

◆ force_nonsingular()

template<typename T >
std::pair< bool, bool > numerical::detail::force_nonsingular ( buffer< T > &  t,
std::size_t  n 
)

The triangular form t made nonsingular as SciPy's _logm_force_nonsingular_triangular_matrix: a zero diagonal entry replaced by 10⁻²⁰.

Returns
Whether one was (exactly singular), and whether one was below 10⁻²⁰ otherwise (nearly singular).

Definition at line 3032 of file matrix_functions.hpp.

◆ form_bidiagonal_bases()

template<typename T >
void numerical::detail::form_bidiagonal_bases ( const std::vector< buffer< T > > &  left,
std::vector< buffer< T > >  right,
const std::vector< T > &  left_tau,
const std::vector< T > &  right_tau,
const buffer< T > &  left_phase,
const buffer< T > &  right_phase,
std::size_t  n,
buffer< T > &  ut,
buffer< T > &  vt 
)

The bases of a bidiagonal reduction from its reflectors: U_b = (H₀ H₁ …) diag(conj left phases) and V_b = (H'₀ H'₁ …) diag(right phases), each formed by blocks of reflectors through the matrix product kernel, as QR's Q is, or a reflector at a time below bidiagonal_basis_crossover.

Parameters
[in]leftThe left reflector of each step (acting from its row), empty where none.
[in]rightThe right reflector of each step (acting from the column past it), likewise.
[in]left_tauEach left reflector's τ (I − τ w wᴴ, larfg_in_place).
[in]right_tauEach right reflector's.
[in]left_phaseThe unit phase each row took.
[in]right_phaseThe unit phase each column past the diagonal took.
[in]nThe order.
[out]utU_bᵀ, row-major: its rows are the columns of U_b.
[out]vtV_bᵀ, likewise.

Definition at line 213 of file svd.hpp.

◆ form_q_into()

template<typename T >
void numerical::detail::form_q_into ( buffer< T > &  out,
const std::vector< reflector_block< T > > &  blocks,
std::size_t  rows,
std::size_t  first,
std::size_t  count,
bool  transposed 
)
private

The columns [first, first + count) of Q = H₀ H₁ …, the blocks applied (the last first) to those of the identity, into out: rows × count row-major, or where transposed Qᵀ row-major (Q column-major). Formed as householder_reflect factorises, so that a Q is the same to the bit in either layout: as the rows of Qᵀ on the x86-64 units (apply_transposed_reflector_block), row-major elsewhere (apply_reflector_block), then transposed where the other layout is asked. The columns of the identity left of a block's rows are left as they are.

Definition at line 3556 of file factorizations.hpp.

◆ forward_fft()

template<std::floating_point R>
void numerical::detail::forward_fft ( const fft_plan< R > &  plan,
std::complex< R > *  data,
std::complex< R > *  work,
bool  fused,
bool  in_work = false 
)

The forward transform of the plan's length into data, work its fft_plan::work, its Stockham passes, or Bluestein's convolution, or its four steps, the products fused where fused (fft_fuses): the input in data, or in work where in_work (only where fft_starts_in_work).

Definition at line 1832 of file fft.hpp.

◆ four_point()

template<typename C >
void numerical::detail::four_point ( const C &  v0,
const C &  v1,
const C &  v2,
const C &  v3,
C *  y 
)
inlinenoexcept

A 4-point transform of v0 to v3 into y: the radix-4 butterfly's sums, in its order.

Definition at line 1211 of file fft.hpp.

◆ four_step_columns()

std::size_t numerical::detail::four_step_columns ( std::size_t  n,
std::size_t  bytes 
)
inline

The columns n1 of a transform of n points of bytes each in four steps: the largest divisor of n at most √n, n of at least fft_four_step_least bytes; else 0, one transform of passes. n is taken in passes (its prime factors at most fft_largest_radix: make_fft_plan settles Bluestein's lengths first). Such an n, 2¹⁸ and more (√n at least 512), has a divisor from 16 to √n, so that neither side is short: a prime factor past 15 (at most 61), or else the product of its smallest prime factors first past 15 (at most 15·13).

Definition at line 962 of file fft.hpp.

◆ four_step_fft()

template<std::floating_point R>
void numerical::detail::four_step_fft ( const fft_plan< R > &  plan,
const std::complex< R > *  in,
std::complex< R > *  out,
std::complex< R > *  work,
bool  fused,
bool  inverse = false,
R  scale = R(1) 
)

The transform of the plan's four_step_plan of in into out (in itself or apart from it), work its fft_plan::work (the columns' transforms, then the rows'), the products fused where fused; the inverse's conj(fft(conj x)) and times scale as transform_axis's lines.

Definition at line 2220 of file fft.hpp.

◆ fractional_power_superdiagonal() [1/2]

std::complex< double > numerical::detail::fractional_power_superdiagonal ( const std::complex< double > &  l1,
const std::complex< double > &  l2,
const std::complex< double > &  t12,
double  p 
)
inline

The superdiagonal entry of T^p for the 2 × 2 upper triangular [[l1, t12], [0, l2]], as SciPy's _fractional_power_superdiag_entry (Higham and Lin's equation 5.5, 2011).

Definition at line 2721 of file matrix_functions.hpp.

◆ fractional_power_superdiagonal() [2/2]

double numerical::detail::fractional_power_superdiagonal ( double  l1,
double  l2,
double  t12,
double  p 
)
inline

fractional_power_superdiagonal of real l1 and l2 above 0 (the unwinding number 0).

Definition at line 2779 of file matrix_functions.hpp.

◆ frechet_bound()

double numerical::detail::frechet_bound ( std::size_t  m)
inline

‖A‖₁ bounds of the [m/m] Padé approximants of the Fréchet derivative of the exponential, for m from 1 to 20, as Al-Mohy and Higham's table 6.1 (2009) and SciPy's ell_table_61.

Definition at line 953 of file matrix_functions.hpp.

◆ frechet_pade()

template<typename T >
std::pair< buffer< T >, buffer< T > > numerical::detail::frechet_pade ( const buffer< T > &  a,
const buffer< T > &  e,
std::size_t  n,
std::size_t  m 
)

e^A and its Fréchet derivative L(A, E) from the [m/m] Padé approximant, as Al-Mohy and Higham's algorithm 6.4 (2009) and SciPy's expm_frechet_algo_64: U, V and their derivatives Lu, Lv along e (M₂ = A E + E A, M₄ = A² M₂ + M₂ A², M₆ = A⁴ M₂ + M₄ A², M₈ = A⁴ M₄ + M₄ A⁴), each sum from its highest term, then R = (V − U)⁻¹ (U + V) and L = (V − U)⁻¹ (Lu + Lv + (Lu − Lv) R) by one LU factorisation; for m = 13, a and e already scaled, A⁶'s nesting as pade_approximant's.

Definition at line 993 of file matrix_functions.hpp.

◆ frobenius_two()

double numerical::detail::frobenius_two ( const std::array< std::complex< double >, 4 > &  m)
inline

The Frobenius norm of the 2 × 2 block m.

Definition at line 409 of file generalized.hpp.

◆ from_carried()

template<typename T >
T numerical::detail::from_carried ( carried_t< T >  x)

x, a magnitude carried in carried_t, as the scalar T: converted to T's real type first, so a std::complex<float> is built from a float (a double argument narrows inside its constructor).

Definition at line 355 of file factorizations.hpp.

◆ from_order_key()

template<std::floating_point T>
T numerical::detail::from_order_key ( order_key_t< T >  key)
noexcept

The real whose order_key is key.

The integer whose order_key is key.

Definition at line 3530 of file tensor_ops.hpp.

◆ function_schur_form() [1/2]

std::pair< matrix< std::complex< double > >, matrix< std::complex< double > > > numerical::detail::function_schur_form ( const matrix< double > &  a)
inline

A real matrix's complex Schur form as scipy.linalg.funm takes it: the real form made complex (rsf2csf).

Definition at line 1764 of file matrix_functions.hpp.

◆ function_schur_form() [2/2]

std::pair< matrix< std::complex< double > >, matrix< std::complex< double > > > numerical::detail::function_schur_form ( const matrix< std::complex< double > > &  a)
inline

A complex matrix's Schur form.

Definition at line 1771 of file matrix_functions.hpp.

◆ funm_answer()

funm_result numerical::detail::funm_answer ( matrix< std::complex< double > > &&  x,
double  error,
std::vector< double > &&  contours,
const std::vector< std::complex< double > > &  at_eigenvalues 
)
inline

x with its estimate error, ∞ where an entry of x or f at an eigenvalue (at_eigenvalues) is not finite (no f(A) then, whatever the circles gave), and the circles' estimates contours.

Definition at line 2358 of file matrix_functions.hpp.

◆ fused_from_zero()

template<std::floating_point T>
T numerical::detail::fused_from_zero ( T  p,
T  q 
)
inlinenoexcept

std::fma(p, q, +0) without the instruction (in code built for any x86-64, std::fma is a call to the library, 40 000 elements of an outer product 18 → 114 µs on four AVX2 cores): the rounded product, which keeps the sign of a product that underflows to zero, plus +0 where an operand is zero, as the fused sum of an exact zero product and +0 is.

Definition at line 1855 of file tensor.hpp.

◆ fused_product()

template<std::floating_point R>
std::complex< R > numerical::detail::fused_product ( const std::complex< R > &  w,
const std::complex< R > &  v 
)
inlinenoexcept

w · v of complex numbers on the vector paths: w.re·v.re − w.im·v.im and w.re·v.im + w.im·v.re, each the second product rounded, the first fused with the sum by std::fma.

Definition at line 315 of file fft.hpp.

◆ gather_clusters()

bool numerical::detail::gather_clusters ( matrix< std::complex< double > > &  t,
matrix< std::complex< double > > &  z,
double  share 
)
inline

Reorders the complex Schur form t, z so each cluster of eigenvalues (eigenvalue_clusters, within cluster_bound) is a run of the diagonal, each in turn moved to the first place past those placed.

Returns
Whether any cluster holds two or more eigenvalues (none moved otherwise).

Definition at line 2087 of file matrix_functions.hpp.

◆ gather_reflectors()

template<typename T >
reflector_block< T > numerical::detail::gather_reflectors ( const std::vector< buffer< T > > &  reflectors,
std::size_t  k0,
std::size_t  k1,
std::size_t  rows,
const std::vector< T > *  taus = nullptr 
)
private

Gathers the reflectors reflectors[k0…k1-1] (each I − τ_k v vᴴ: τ_k from taus, else 2 for a unit v; an empty one the identity) into I − V T Vᴴ = H_{k0} H_{k0+1} … H_{k1-1}, T as LAPACK's dlarft forms it.

Definition at line 3064 of file factorizations.hpp.

◆ gemm()

template<typename T >
void numerical::detail::gemm ( std::size_t  m,
std::size_t  n,
std::size_t  k,
const T *  a,
std::size_t  lda,
const T *  b,
std::size_t  ldb,
T *  c,
std::size_t  ldc,
c_start  start 
)

C += A·B, every element of C summed in ascending order of the depth.

Products of fewer than gemm_packed_threshold multiply-adds, or shallower than gemm_min_depth, run gemm_rows. The rest run packed (gemm_packed). A double product runs on the AMX where it is in use and suits the product (amx_suits), else on the NEON or AVX tiles: each element a chain of fused multiply-adds; a float one on the NEON tile on arm64's NEON and AMX paths. On the portable path either runs on the vector tile, which rounds c + a·b as the compiler does: each product before its sum with contraction off, as the plain loop then does; fused under Clang's -ffp-contract=on, as the plain loop then is; under GCC's, apart, where the plain loop's scalar statement is fused. Every element is summed in ascending order of the depth on any number of threads. Complex numbers on the x86-64 paths run on their complex tiles (avx2_full_tile_c64, avx512_full_tile_c64 and their _c32 for float); past the portable path on arm64, through the real kernel (gemm_split), one or two rows of A by its chains row by row (few_rows_complex_neon). Either is the same on any number of threads too, but no longer the plain loop's to the bit: so past the portable path a complex product of any shape runs there, and each element of C is the same whatever the shape of the product it falls in. A caller that splits its columns by the threads (an LU inverse's slices) then has the same bits on any number of them, where a slice narrow enough for the plain loop rounded its complex products otherwise.

Parameters
[in]m,n,kThe shapes: A is m×k, B k×n, C m×n, all row-major.
[in]a,lda,b,ldbThe operands and the distance between their rows.
[in,out]c,ldcThe result and the distance between its rows (C may be a block of a larger matrix that A and B do not overlap).
[in]startWhat C holds (c_start): with c_start::zero, C = A·B, the same bits as C zeroed then this product, each part of C zeroed by the thread that computes it. A zero m or n leaves C as it is; a zero k too, or zeroes it with c_start::zero.

Definition at line 4583 of file gemm.hpp.

◆ gemm_at()

template<typename T >
void numerical::detail::gemm_at ( std::size_t  m,
std::size_t  n,
std::size_t  k,
const T *  at,
std::size_t  ldat,
const T *  b,
std::size_t  ldb,
T *  c,
std::size_t  ldc,
c_start  start = c_start::kept 
)

gemm_layouts with A given as Aᵀ (at, k × m, ldat apart).

Definition at line 4787 of file gemm.hpp.

◆ gemm_atbt()

template<typename T >
void numerical::detail::gemm_atbt ( std::size_t  m,
std::size_t  n,
std::size_t  k,
const T *  at,
std::size_t  ldat,
const T *  bt,
std::size_t  ldbt,
T *  c,
std::size_t  ldc,
c_start  start = c_start::kept 
)

gemm_layouts with both operands given as their transposes.

Definition at line 4803 of file gemm.hpp.

◆ gemm_bt()

template<typename T >
void numerical::detail::gemm_bt ( std::size_t  m,
std::size_t  n,
std::size_t  k,
const T *  a,
std::size_t  lda,
const T *  bt,
std::size_t  ldbt,
T *  c,
std::size_t  ldc,
c_start  start = c_start::kept 
)

gemm_layouts with B given as Bᵀ (bt, n × k, ldbt apart).

Definition at line 4771 of file gemm.hpp.

◆ gemm_double_elsewhere()

template<typename B , typename A = a_rows>
bool numerical::detail::gemm_double_elsewhere ( std::size_t  m,
std::size_t  n,
std::size_t  k,
const double *  a,
std::size_t  lda,
const double *  b,
std::size_t  ldb,
double *  c,
std::size_t  ldc,
c_start  start = c_start::kept 
)

A double product on the path in use (numerical::gemm_path_in_use), unless it is the portable one, B and A laid out as B and A say (b_rows, a_rows and their transposes).

Returns
Whether it ran there.

Definition at line 3311 of file gemm.hpp.

◆ gemm_elsewhere() [1/3]

bool numerical::detail::gemm_elsewhere ( std::size_t  m,
std::size_t  n,
std::size_t  k,
const double *  a,
std::size_t  lda,
const double *  b,
std::size_t  ldb,
double *  c,
std::size_t  ldc,
c_start  start = c_start::kept 
)
inline

gemm_double_elsewhere of a row-major B.

Definition at line 3424 of file gemm.hpp.

◆ gemm_elsewhere() [2/3]

bool numerical::detail::gemm_elsewhere ( std::size_t  m,
std::size_t  n,
std::size_t  k,
const float *  a,
std::size_t  lda,
const float *  b,
std::size_t  ldb,
float *  c,
std::size_t  ldc,
c_start  start = c_start::kept 
)
inline

gemm_float_elsewhere of row-major operands.

Definition at line 3483 of file gemm.hpp.

◆ gemm_elsewhere() [3/3]

template<typename T >
bool numerical::detail::gemm_elsewhere ( std::size_t  m,
std::size_t  n,
std::size_t  k,
const T *  a,
std::size_t  lda,
const T *  b,
std::size_t  ldb,
T *  c,
std::size_t  ldc,
c_start  start = c_start::kept 
)

A product a faster path than the vector one takes, if any: none for this scalar.

Definition at line 3291 of file gemm.hpp.

◆ gemm_float_elsewhere()

template<typename B , typename A = a_rows>
bool numerical::detail::gemm_float_elsewhere ( std::size_t  m,
std::size_t  n,
std::size_t  k,
const float *  a,
std::size_t  lda,
const float *  b,
std::size_t  ldb,
float *  c,
std::size_t  ldc,
c_start  start = c_start::kept 
)

A float product on the AMX where it suits the product, on the NEON tile on arm64's fused paths, on the x86-64 tiles there; elsewhere the vector tile. B and A laid out as B and A say.

Returns
Whether it ran on a tile of its path.

Definition at line 3443 of file gemm.hpp.

◆ gemm_kernel()

template<typename T >
void numerical::detail::gemm_kernel ( std::size_t  kc,
const T *  ap,
const T *  bp,
std::size_t  ldb,
gemm_tile< T > &  tile 
)

tile += Aₚ·Bₚ over one panel, for any scalar.

Parameters
[in]kcThe panel depth.
[in]apThe A sliver, kc columns of gemm_mr elements.
[in]bpThe B sliver, kc rows of gemm_nr elements.
[in]ldbThe distance between its rows.
[in,out]tileThe accumulators, row-major.

Definition at line 929 of file gemm.hpp.

◆ gemm_large()

template<typename T >
void numerical::detail::gemm_large ( std::size_t  m,
std::size_t  n,
std::size_t  k,
const T *  a,
std::size_t  lda,
const T *  b,
std::size_t  ldb,
T *  c,
std::size_t  ldc,
c_start  start = c_start::kept 
)

A product past the plain loop: on the AMX or x86-64 tiles where gemm_elsewhere takes it, else on the vector tile.

gemm_large_complex of row-major operands.

Definition at line 4430 of file gemm.hpp.

◆ gemm_large_complex()

template<typename B , typename A , split_complex T>
void numerical::detail::gemm_large_complex ( std::size_t  m,
std::size_t  n,
std::size_t  k,
const T *  a,
std::size_t  lda,
const T *  b,
std::size_t  ldb,
T *  c,
std::size_t  ldc,
c_start  start = c_start::kept 
)

A complex product past the plain loop: on the vector tile on the portable path, whole complex products summed in order; on the x86-64 complex tiles there; else (arm64) through the real kernel (gemm_split). B and A laid out as B and A say; start what C holds (c_start), the packed paths' to settle, the split's C zeroed first.

Definition at line 4470 of file gemm.hpp.

◆ gemm_large_layouts() [1/3]

template<typename B , typename A >
void numerical::detail::gemm_large_layouts ( std::size_t  m,
std::size_t  n,
std::size_t  k,
const double *  a,
std::size_t  lda,
const double *  b,
std::size_t  ldb,
double *  c,
std::size_t  ldc,
c_start  start 
)

A double product past the plain loop, B and A laid out as B and A say (gemm_layouts).

Definition at line 4650 of file gemm.hpp.

◆ gemm_large_layouts() [2/3]

template<typename B , typename A >
void numerical::detail::gemm_large_layouts ( std::size_t  m,
std::size_t  n,
std::size_t  k,
const float *  a,
std::size_t  lda,
const float *  b,
std::size_t  ldb,
float *  c,
std::size_t  ldc,
c_start  start 
)

A float product past the plain loop, B and A laid out as B and A say (gemm_layouts).

Definition at line 4669 of file gemm.hpp.

◆ gemm_large_layouts() [3/3]

template<typename B , typename A , split_complex T>
void numerical::detail::gemm_large_layouts ( std::size_t  m,
std::size_t  n,
std::size_t  k,
const T *  a,
std::size_t  lda,
const T *  b,
std::size_t  ldb,
T *  c,
std::size_t  ldc,
c_start  start 
)

A complex product past the plain loop, B and A laid out as B and A say (gemm_layouts).

Definition at line 4689 of file gemm.hpp.

◆ gemm_layouts()

template<typename A , typename B , typename T >
void numerical::detail::gemm_layouts ( std::size_t  m,
std::size_t  n,
std::size_t  k,
const T *  a,
std::size_t  lda,
const T *  b,
std::size_t  ldb,
T *  c,
std::size_t  ldc,
c_start  start = c_start::kept 
)

gemm with either operand given as its transpose (A: a_rows or a_transposed, B: b_rows or b_transposed), so that a product by a transpose (a syrk's P Pᵀ, BLAS's transa and transb, a NumPy array in Fortran order) needs no copy of it. The tiles are gemm's, the operands packed (or, through gemm_split, copied into their parts) from where they lie, and so are the bits; a product gemm runs in the plain loop copies a transposed operand row-major and runs there, small as it is. start says what C holds (c_start), passed on to the route as gemm passes it: C zeroed apart first cost a pass over it, as long as the product's own at a shallow depth (1000 × 1000 by depth 4, one arm64 thread: 307 µs copying the transpose for gemm, 402 read in place).

Definition at line 4716 of file gemm.hpp.

◆ gemm_pack_a()

template<std::size_t Mr, typename T >
void numerical::detail::gemm_pack_a ( const T *  a,
std::size_t  lda,
std::size_t  rows,
std::size_t  kc,
T *  out 
)

Copies rows × kc of A (from row-major, leading dimension lda) into slivers of Mr rows, each stored column by column; a last partial sliver is padded with zeros.

Definition at line 1076 of file gemm.hpp.

◆ gemm_pack_at()

template<std::size_t Mr, typename T >
void numerical::detail::gemm_pack_at ( const T *  at,
std::size_t  ldat,
std::size_t  rows,
std::size_t  kc,
T *  out 
)

The slivers of Mr rows of A (rows × kc), column by column, from its transpose at (kc × rows row-major, ldat apart): each column of a sliver is a run of a row of at; a last partial sliver is padded with zeros, as gemm_pack_a lays them out.

Definition at line 1317 of file gemm.hpp.

◆ gemm_pack_b()

template<std::size_t Nr, typename T >
void numerical::detail::gemm_pack_b ( const T *  b,
std::size_t  ldb,
std::size_t  kc,
std::size_t  cols,
T *  out 
)

Copies kc × cols of B (from row-major, leading dimension ldb) into slivers of Nr columns, each stored row by row; a last partial sliver is padded with zeros.

Definition at line 1107 of file gemm.hpp.

◆ gemm_pack_bt()

template<std::size_t Nr, typename T >
void numerical::detail::gemm_pack_bt ( const T *  bt,
std::size_t  ldbt,
std::size_t  kc,
std::size_t  cols,
T *  out 
)

gemm_pack_b of B = Bᵀᵀ, from the row-major bt (cols × kc, leading dimension ldbt): each column of a sliver is a row of bt, read along it; a last partial sliver is padded with zeros.

Definition at line 1132 of file gemm.hpp.

◆ gemm_packed()

template<typename Kernel , typename T , typename B = b_rows, typename A = a_rows>
void numerical::detail::gemm_packed ( std::size_t  m,
std::size_t  n,
std::size_t  k,
const T *  a,
std::size_t  lda,
const T *  b,
std::size_t  ldb,
T *  c,
std::size_t  ldc,
const T *  a_packed = nullptr,
c_start  start = c_start::kept 
)

The packed product with the tile Kernel computes: for each block of B's columns and each panel of the depth, both operands are packed (in parallel, by the blocks the product takes), then tasks of gemm_mc rows by gemm_slivers_per_task slivers of columns run in parallel, each writing a disjoint block of C, one tile at a time. Given a_packed (A as pack_left packs it for Kernel, a depth of one panel), A is read from there and only B is packed: the same tiles, the same bits. B is B's layout (b_rows, b_transposed), A A's (a_rows, a_transposed): the packing alone differs, so the bits do not. start says what C holds (c_start), k then not zero.

Definition at line 2583 of file gemm.hpp.

◆ gemm_packed_triangle()

template<typename Kernel , typename T , typename B = b_rows, typename A = a_rows>
void numerical::detail::gemm_packed_triangle ( std::size_t  n,
std::size_t  k,
const T *  a,
std::size_t  lda,
const T *  b,
std::size_t  ldb,
T *  c,
std::size_t  ldc,
bool  lower 
)

gemm_packed restricted to a triangle of the square C = A·B (n × n): its lower one (j ≤ i) where lower, else its upper one (j ≥ i); the other is neither written nor depended on. The same packing (A's row blocks with no tile in the triangle left out) and the same tiles, so each element's sum is the product's; a tile across the diagonal runs on a copy of its block of C, the triangle's part copied back. A syrk takes the triangle in one pass, its operands packed once a panel, where products by halves packed them again for every block.

Definition at line 2732 of file gemm.hpp.

◆ gemm_path_cap()

std::atomic< gemm_path > & numerical::detail::gemm_path_cap ( )
inlinenoexcept

The most capable path products may take, as numerical::limit_gemm_path set it: none is probed until a product (or a question) reaches it, so a cap below the AMX keeps its instructions from ever running.

Definition at line 128 of file gemm.hpp.

◆ gemm_path_current()

gemm_path numerical::detail::gemm_path_current ( )
inlinenoexcept

The path the next double product takes: the most capable one under the cap that runs here (each path's probe is made once). Kept with its cap, one load, where each call walked the paths down from the cap through their probes' guards: every small step of a decomposition asks (an LU of order 100 through dgetrf_ on one AVX2 core spent 5 % of its time here). A race stores the same value twice.

Definition at line 148 of file gemm.hpp.

◆ gemm_path_found()

std::atomic< std::uint8_t > & numerical::detail::gemm_path_found ( )
inlinenoexcept

The path gemm_path_current found last, packed with the cap it was found under (cap · 8 + path); 0xFF before any.

Definition at line 136 of file gemm.hpp.

◆ gemm_path_usable()

bool numerical::detail::gemm_path_usable ( gemm_path  path)
inlinenoexcept

Whether path can run here.

Definition at line 102 of file gemm.hpp.

◆ gemm_rows()

template<typename T >
void numerical::detail::gemm_rows ( std::size_t  m,
std::size_t  n,
std::size_t  k,
const T *  a,
std::size_t  lda,
const T *  b,
std::size_t  ldb,
T *  c,
std::size_t  ldc,
c_start  start = c_start::kept 
)

C += A·B by the plain i-k-j loop (gemm_rows_rounded), each element rounded as the tile of the path in use rounds it, so that it is the same whatever the shape of the product it falls in: a caller that splits its columns by the threads (an LU inverse) then has the same bits on any number of them.

Definition at line 2336 of file gemm.hpp.

◆ gemm_rows_rounded()

template<term_rounding Mode, typename T >
void numerical::detail::gemm_rows_rounded ( std::size_t  m,
std::size_t  n,
std::size_t  k,
const T *  a,
std::size_t  lda,
const T *  b,
std::size_t  ldb,
T *  c,
std::size_t  ldc,
c_start  start = c_start::kept 
)

C += A·B by the plain i-k-j loop, parallel over the rows of C. Past a depth of 1 (narrow_least_depth), the rows run narrow_group at a time by blocks (narrow_rows), the last rows one by one: the same sums, six to ten times as fast from order 4 to 16 (M1 Pro) as the loop of unknown width, whose sums the compiler cannot keep in registers. A depth of 1 streams as fast through that loop.

Parameters
[in]m,n,kThe shapes: A is m×k, B k×n, C m×n, all row-major.
[in]a,lda,b,ldbThe operands and the distance between their rows.
[in,out]c,ldcThe result and the distance between its rows.
[in]startWhat C holds (c_start): by blocks, their sums from zero; one by one, each row zeroed by the thread that computes it. Each term rounded as Mode says (term_rounding).

Definition at line 2244 of file gemm.hpp.

◆ gemm_shares()

template<typename Kernel >
constexpr bool numerical::detail::gemm_shares ( std::size_t  m,
std::size_t  n,
std::size_t  k 
)
constexprnoexcept

Whether the product's threads pay: below Kernel's serial_below multiply-adds, the product runs on the caller alone.

Definition at line 2459 of file gemm.hpp.

◆ gemm_split()

template<split_complex Z, typename B = b_rows, typename A = a_rows>
void numerical::detail::gemm_split ( std::size_t  m,
std::size_t  n,
std::size_t  k,
const Z *  a,
std::size_t  lda,
const Z *  b,
std::size_t  ldb,
Z *  c,
std::size_t  ldc 
)

C += A·B for complex numbers through the real kernel, by blocks of split_columns columns and split_depth of depth, on the real path's units and threads (a scalar complex tile has none). Within a block, each element of C takes the products of the real parts of A in ascending order of the depth, then those of the imaginary parts: by four real products of the block's parts (split_parts_block) or one of its expanded operands (split_expanded_block), as split_by_parts chooses, the same bits either way. A and B laid out as A and B say, read where they lie as the parts are copied.

Definition at line 4305 of file gemm.hpp.

◆ gemm_triangle() [1/2]

template<typename A , typename B >
void numerical::detail::gemm_triangle ( std::size_t  n,
std::size_t  k,
const double *  a,
std::size_t  lda,
const double *  b,
std::size_t  ldb,
double *  c,
std::size_t  ldc,
bool  lower 
)

The triangle of the double product C = A·B (gemm_packed_triangle) on the tile of the path in use, A and B laid out as A and B say.

Definition at line 3356 of file gemm.hpp.

◆ gemm_triangle() [2/2]

template<typename A , typename B >
void numerical::detail::gemm_triangle ( std::size_t  n,
std::size_t  k,
const float *  a,
std::size_t  lda,
const float *  b,
std::size_t  ldb,
float *  c,
std::size_t  ldc,
bool  lower 
)

The triangle of the float product C = A·B (gemm_packed_triangle) on the float tile of the path in use, A and B laid out as A and B say: the single-precision factorisations' trailing updates.

Definition at line 3399 of file gemm.hpp.

◆ gemm_with()

template<typename T >
void numerical::detail::gemm_with ( const packed_left< T > &  left,
std::size_t  n,
const T *  b,
std::size_t  ldb,
T *  c,
std::size_t  ldc 
)

C += A·B for the packed left (pack_left) and the k × n b, as gemm computes it, A read in place here, for a scalar other than double.

Definition at line 4992 of file gemm.hpp.

◆ gemm_with< double >()

template<>
void numerical::detail::gemm_with< double > ( const packed_left< double > &  left,
std::size_t  n,
const double *  b,
std::size_t  ldb,
double *  c,
std::size_t  ldc 
)
inline

gemm_with for double: on the tile A was packed for, A read from its packing, where gemm would choose that tile for this n; else as gemm. Either way gemm's tiles and order of summation, its bits.

Definition at line 5008 of file gemm.hpp.

◆ general_logm() [1/2]

logm_result numerical::detail::general_logm ( const matrix< double > &  a)
inline

logm of a real matrix past its Hermitian path: in real arithmetic where its real Schur form (or the matrix, upper triangular) is triangular with no diagonal entry below 0, as SciPy's (_logm, _logm_triu), else in the complex form (rsf2csf).

Definition at line 3489 of file matrix_functions.hpp.

◆ general_logm() [2/2]

logm_result numerical::detail::general_logm ( const matrix< std::complex< double > > &  a)
inline

logm of a complex matrix past its Hermitian path: an upper triangular one directly, else its Schur form's.

Definition at line 3476 of file matrix_functions.hpp.

◆ generalized_sylvester()

std::optional< std::pair< small_square, small_square > > numerical::detail::generalized_sylvester ( const small_square &  s,
const small_square &  t,
std::size_t  n1,
std::size_t  n2 
)
inline

The solution (R, L) (n1 × n2 each) of the generalized Sylvester equations S₁₁ R − L S₂₂ = S₁₂, T₁₁ R − L T₂₂ = T₁₂ of the m × m blocks of s and t (m = n1 + n2), as LAPACK's dtgsy2 solves them: by elimination with complete pivoting of their Kronecker form; nothing where a pivot falls below ε times the largest entry (dgetc2 would perturb it, and dtgex2 then refuses the exchange).

Definition at line 1676 of file generalized.hpp.

◆ givens() [1/2]

plane_rotation< std::complex< double > > numerical::detail::givens ( const std::complex< double > &  f,
const std::complex< double > &  g 
)
inline

The rotation that takes (f, g) to (r, 0), as LAPACK's zlartg: c real, r of f's phase.

Definition at line 1329 of file eigen.hpp.

◆ givens() [2/2]

plane_rotation< double > numerical::detail::givens ( double  f,
double  g 
)
inline

The rotation that takes (f, g) to (r, 0), as LAPACK's dlartg: r of f's sign.

Definition at line 1315 of file eigen.hpp.

◆ grid_points()

std::vector< std::complex< double > > numerical::detail::grid_points ( std::complex< double >  centre,
double  radius,
std::size_t  count,
std::size_t  from,
std::size_t  step 
)
inline

The points k = from, from + step, … < count of the grid of count points on the circle (centre, radius), in turn.

Definition at line 2182 of file matrix_functions.hpp.

◆ hamiltonian_riccati()

template<typename T >
matrix< T > numerical::detail::hamiltonian_riccati ( const matrix< T > &  a,
const matrix< T > &  b,
const matrix< T > &  q,
const matrix< T > &  r,
const matrix< T > *  s,
bool  balanced 
)

The stabilising solution of the continuous equation without E from the Schur form of the Hamiltonian H = [[Ã, −G], [−Q̃, −Ãᴴ]] (Laub, 1979), G = B R⁻¹ Bᴴ, Ã = A − B R⁻¹ Sᴴ and Q̃ = Q − S R⁻¹ Sᴴ in the arithmetic of T (a real H's real Schur form): its eigenvalues in the left half-plane moved first, X = U₂₁ U₁₁⁻¹ from their m Schur vectors, made Hermitian. A Schur form of order 2m by multishift QR where the pencil's QZ takes order 2m + n. Where balanced, H is first T⁻¹ H T, T = diag(D, D⁻¹) (balance_hamiltonian), still Hamiltonian, and X = D⁻¹ X̃ D⁻¹: a Q of 10¹⁰ beside a G of 1 left the unbalanced form's X wrong at 10⁻³.

Exceptions
singular_matrix_errorIf U₁₁ is numerically singular (stabilising_solution).
convergence_errorIf the left half-plane holds other than m eigenvalues, or as stabilising_solution says (eigenvalues on or too close to the imaginary axis).

Definition at line 3165 of file generalized.hpp.

◆ hardware_threads()

std::size_t numerical::detail::hardware_threads ( )
inlinenoexcept

The threads the pool runs, the caller's included (at least 1), known without starting the pool: one per physical core where the system tells them (else one per hardware thread), no more than a CPU quota allows (quota_limit). Two threads on one core's hyperthreads share its multiply-add units: a product of order 128 on eight AVX2 cores with sixteen hyperthreads took 140 µs on sixteen threads, 71 on eight; order 2048, the same either way. Asked once, as the pool's workers are started once (glibc reads a file for the hardware threads on every call).

Definition at line 306 of file parallel.hpp.

◆ hermitian_from_lower()

template<typename T >
matrix< T > numerical::detail::hermitian_from_lower ( const matrix< T > &  a)

The real symmetric or complex Hermitian matrix whose lower triangle is the square a's, and of its diagonal the real part.

Definition at line 3079 of file svd.hpp.

◆ hermitian_product()

template<typename T >
void numerical::detail::hermitian_product ( const T *  block,
std::size_t  n,
std::size_t  m,
const T *  v,
T *  w,
buffer< T > &  scratch 
)

w = B v for the Hermitian m × m block (row stride n) from its lower triangle alone, as LAPACK's dsymv: each row's lower part is read once and serves twice, w[i] += Σⱼ≤ᵢ B[i][j] v[j] and w[j] += conj(B[i][j]) v[i] for j < i, so a reduction step reads half the block it would read whole. Rows go by tasks of fixed size, each into its own partial w, summed in a fixed order: the result does not depend on the number of threads. A real row is read once for both (symmetric_row), on the widest x86-64 units the matrix product's kernel may use (gemm_path_current).

Parameters
[in]blockThe block's first element; its rows n apart.
[in]nThe row stride.
[in]mThe block's order.
[in]vThe vector (m elements).
[out]wB v (m elements).
[in,out]scratchThe partials, kept by the caller from one step of a reduction to the next: a fresh buffer of their size each step is mapped anew, page by page.

Definition at line 4515 of file factorizations.hpp.

◆ hermitian_pseudo_inverse() [1/3]

matrix< double > numerical::detail::hermitian_pseudo_inverse ( const matrix< double > &  a,
rank_cutoff  cutoff 
)
inline

hermitian_pseudo_inverse in double: V Λ⁺ Vᵀ from the eigendecomposition, cheaper.

Definition at line 3185 of file svd.hpp.

◆ hermitian_pseudo_inverse() [2/3]

matrix< std::complex< double > > numerical::detail::hermitian_pseudo_inverse ( const matrix< std::complex< double > > &  a,
rank_cutoff  cutoff 
)
inline

hermitian_pseudo_inverse in std::complex<double>: V Λ⁺ Vᴴ from the eigendecomposition, cheaper.

Definition at line 3194 of file svd.hpp.

◆ hermitian_pseudo_inverse() [3/3]

template<typename T >
matrix< T > numerical::detail::hermitian_pseudo_inverse ( const matrix< T > &  a,
rank_cutoff  cutoff 
)

The pseudo-inverse of the real symmetric or complex Hermitian matrix whose lower triangle is a's: from the singular value decomposition of the matrix made whole, for a scalar the eigendecompositions do not take.

Definition at line 3177 of file svd.hpp.

◆ hermitian_rank_one()

void numerical::detail::hermitian_rank_one ( buffer< std::complex< double > > &  a,
std::size_t  n,
std::size_t  k,
buffer< std::complex< double > > &  w 
)
inline

As symmetric_rank_one for a Hermitian matrix, as hetf2 (her): d real, A ← A − x xᴴ / d, the diagonal kept real.

Definition at line 109 of file ldl.hpp.

◆ hermitian_rank_two()

void numerical::detail::hermitian_rank_two ( buffer< std::complex< double > > &  a,
std::size_t  n,
std::size_t  k,
buffer< std::complex< double > > &  w 
)
inline

As symmetric_rank_two for a Hermitian matrix, as hetf2: |D21| scaled out in real arithmetic, the conjugates where Aᴴ has them, the diagonal kept real.

Definition at line 164 of file ldl.hpp.

◆ hermitian_remainder_power() [1/2]

matrix< std::complex< double > > numerical::detail::hermitian_remainder_power ( const hermitian_similarity< double > &  s,
double  t 
)
inlineprivate

A^t (−1 < t < 1) of a real symmetric A not triangular from s: V T^t Vᵀ, T the upper triangle of Vᵀ A V (similar_power), the products real: T^t real where T's diagonal is positive (nearly_diagonal_power's in real arithmetic), else its real and imaginary parts apart.

Precondition
No diagonal entry of T is 0.

Definition at line 3252 of file matrix_functions.hpp.

◆ hermitian_remainder_power() [2/2]

matrix< std::complex< double > > numerical::detail::hermitian_remainder_power ( const hermitian_similarity< std::complex< double > > &  s,
double  t 
)
inlineprivate

A^t (−1 < t < 1) of a complex Hermitian A not triangular from s: Z T^t Zᴴ, T the upper triangle of Zᴴ A Z (similar_power).

Precondition
No real part on T's diagonal is 0.

Definition at line 3285 of file matrix_functions.hpp.

◆ hermitian_root()

template<typename T >
matrix< std::complex< double > > numerical::detail::hermitian_root ( const matrix< T > &  a)

The principal square root of the Hermitian a, V √Λ Vᴴ: Hermitian where no eigenvalue is below 0.

Definition at line 1476 of file matrix_functions.hpp.

◆ hermitian_row() [1/3]

double numerical::detail::hermitian_row ( symmetric_row_kernel  kernel,
const double *  row,
const double *  v,
double *  out,
std::size_t  i 
)
inline

The same for a real row, in one pass through kernel.

Definition at line 4471 of file factorizations.hpp.

◆ hermitian_row() [2/3]

std::complex< double > numerical::detail::hermitian_row ( symmetric_row_kernel  kernel,
const std::complex< double > *  row,
const std::complex< double > *  v,
std::complex< double > *  out,
std::size_t  i 
)
inline

The same for a std::complex<double> row: in one pass on AVX-512 or AVX2 with FMA where the matrix product's kernel runs there (avx512_hermitian_row), else as the template.

Definition at line 4439 of file factorizations.hpp.

◆ hermitian_row() [3/3]

template<typename T >
T numerical::detail::hermitian_row ( symmetric_row_kernel  ,
const T *  row,
const T *  v,
T *  out,
std::size_t  i 
)

Row i's terms of a Hermitian product from the lower triangle: out[j] += conj(row[j]) v[i] for j < i, and the return Σⱼ≤ᵢ row[j] v[j], for a complex T.

Definition at line 4376 of file factorizations.hpp.

◆ hermitian_singular_values() [1/3]

vector< double > numerical::detail::hermitian_singular_values ( const matrix< double > &  a)
inline

hermitian_singular_values in double: the magnitudes of the eigenvalues, cheaper.

Definition at line 3161 of file svd.hpp.

◆ hermitian_singular_values() [2/3]

vector< double > numerical::detail::hermitian_singular_values ( const matrix< std::complex< double > > &  a)
inline

hermitian_singular_values in std::complex<double>: the magnitudes of the eigenvalues, cheaper.

Definition at line 3167 of file svd.hpp.

◆ hermitian_singular_values() [3/3]

template<typename T >
vector< double > numerical::detail::hermitian_singular_values ( const matrix< T > &  a)

The singular values of the real symmetric or complex Hermitian matrix whose lower triangle is a's: of the matrix made whole, for a scalar the eigendecompositions do not take (they compute in double).

Definition at line 3142 of file svd.hpp.

◆ hermitian_subset()

template<typename T >
eigen_subset< T > numerical::detail::hermitian_subset ( const matrix< T > &  a,
const std::function< std::pair< std::ptrdiff_t, std::ptrdiff_t >(const vector< double > &, const vector< double > &, double)> &  select,
bool  vectors,
const std::string &  name 
)

hermitian_eigen_subset of the indices select gives from the tridiagonal form's diagonal and off-diagonal and the matrix's scale ([first, last], last < first for none), its vectors where vectors. select behind one type: one instance per T.

Definition at line 2895 of file svd.hpp.

◆ hermitian_system() [1/2]

std::pair< vector< double >, matrix< double > > numerical::detail::hermitian_system ( const matrix< double > &  a)
inline

The eigenvalues (ascending) and orthonormal eigenvectors of the real symmetric a.

Definition at line 331 of file matrix_functions.hpp.

◆ hermitian_system() [2/2]

std::pair< vector< double >, matrix< std::complex< double > > > numerical::detail::hermitian_system ( const matrix< std::complex< double > > &  a)
inline

The eigenvalues (ascending) and orthonormal eigenvectors of the complex Hermitian a.

Definition at line 339 of file matrix_functions.hpp.

◆ hessenberg()

template<typename T >
buffer< T > numerical::detail::hessenberg ( buffer< T > &  a,
std::size_t  n,
std::size_t  low,
std::size_t  high,
bool  form 
)

hessenberg_blocked from an active order of hessenberg_crossover, else hessenberg_unblocked.

Definition at line 630 of file eigen.hpp.

◆ hessenberg_blocked()

template<typename T >
buffer< T > numerical::detail::hessenberg_blocked ( buffer< T > &  a,
std::size_t  n,
std::size_t  low,
std::size_t  high,
bool  form 
)

hessenberg_unblocked by panels of hessenberg_block columns, as LAPACK's dgehrd and dlahr2: a panel's reflectors are formed one column at a time, each column first updated by those before it through Y = A V T; the rest of the matrix then takes the panel from both sides through the matrix product kernel, A ← (I − V Tᴴ Vᴴ)(A − Y Vᴴ). Its arguments and answer are hessenberg_unblocked's.

Definition at line 451 of file eigen.hpp.

◆ hessenberg_triangular()

qz_form numerical::detail::hessenberg_triangular ( const matrix< std::complex< double > > &  a,
const matrix< std::complex< double > > &  b 
)
inline

a and b (n × n, row-major) to Hessenberg-triangular form, as LAPACK's zgges begins: B = Q R by the QR factorisation, A ← Qᴴ A, then zgghrd: each column of A from the first cleared below its subdiagonal from the bottom by rotations of rows, the fill each leaves below B's diagonal taken out by a rotation of columns; Q and Z accumulating them.

Definition at line 113 of file generalized.hpp.

◆ hessenberg_unblocked()

template<typename T >
buffer< T > numerical::detail::hessenberg_unblocked ( buffer< T > &  a,
std::size_t  n,
std::size_t  low,
std::size_t  high,
bool  form 
)

Reduces the active block low … high − 1 of the balanced row-major n × n a to upper Hessenberg form by Householder reflections one column at a time, as LAPACK's dgehd2: A = Q H Qᴴ, the entries below the subdiagonal set to zero.

Parameters
[in,out]aThe matrix; on return H.
[in]nIts order.
[in]lowThe active block's first row and column.
[in]highOne past its last.
[in]formWhether to form Q.
Returns
Qᵀ, row-major n × n (the identity outside the active block), or empty without form: a QR sweep updates columns of Q, which are then contiguous.

Definition at line 234 of file eigen.hpp.

◆ holds()

template<typename T , typename Match >
bool numerical::detail::holds ( const T *  first,
std::size_t  count,
const Match &  match 
)
noexcept

Whether one of the count contiguous elements at first passes match: tested without a branch, so the loop vectorises.

Definition at line 4242 of file tensor_ops.hpp.

◆ householder_qr()

template<typename T >
thin_qr< T > numerical::detail::householder_qr ( buffer< T >  a,
std::size_t  rows,
std::size_t  cols,
bool  complete = false 
)
private

Factorises the rows × cols row-major a by Householder reflections, A = Q R with k = min(rows, cols): Q is rows × k with orthonormal columns and R k × cols upper trapezoidal. Each column below the diagonal is reflected onto it, so Q is orthonormal to rounding whatever the conditioning of A, and a column dependent on the earlier ones leaves a zero on R's diagonal.

Blocked as LAPACK's dgeqrf: the reflectors of a panel of qr_block columns act on the panel one by one, then on the columns right of it at once, as I − V T Vᴴ through the matrix product kernel; Q is formed block by block the same way.

Parameters
[in]aThe matrix, row-major.
[in]rowsIts row count.
[in]colsIts column count.
[in]completeWhether Q is the whole rows × rows product of the reflections, its columns past k an orthonormal basis of the complement of A's column space (numpy.linalg.qr(mode="complete")).
Returns
Q and R.

Definition at line 3638 of file factorizations.hpp.

◆ householder_r()

template<typename T >
buffer< T > numerical::detail::householder_r ( buffer< T >  a,
std::size_t  rows,
std::size_t  cols 
)
private

householder_qr's R alone, k × cols, k = min(rows, cols): the same reflections and the same rotation of each row by its diagonal's phase, so the same bits, Q never formed (numpy.linalg.qr's mode="r"; forming Q is about as much work again).

Parameters
[in]aThe matrix, row-major.
[in]rowsIts row count.
[in]colsIts column count.
Returns
R, row-major, its diagonal real and non-negative.

Definition at line 3686 of file factorizations.hpp.

◆ householder_reflect()

template<typename T >
qr_reflections< T > numerical::detail::householder_reflect ( buffer< T >  a,
std::size_t  rows,
std::size_t  cols,
bool  reflectors = true 
)
private

The reflections of householder_qr, without forming Q: A = Q R with Q the product of the blocks' reflectors applied to the first k = min(rows, cols) columns of the identity, R's diagonal not yet made real (a product with Qᴴ, or with Q of a block padded with zeros, applies the blocks instead). a is row-major. On the x86-64 units it is transposed in by tiles and factorised column-major (reflect_in_columns, whose updates' products run faster there: one AVX2 core, order 1000, 39 ms against 46), elsewhere in place (reflect_in_rows). Where reflectors is false, R alone is asked: a block no update reads is not gathered, and the blocks are not all there.

Definition at line 3521 of file factorizations.hpp.

◆ hyperbolic_parts()

double numerical::detail::hyperbolic_parts ( double  magnitude,
double &  d_high,
double &  d_low,
double &  u_high,
double &  u_low 
)
inlinenoexcept

The parts of sinh and cosh of magnitude (0 to 711, past which both overflow): t = expm1(magnitude) (expm1_parts, its two parts summed exactly) at most 2^64 in its place (past it, t · 2^(64 − e), whose 1 and 1/(t + 1) then weigh under 2^−63 of it), d = t + 1 and u = 1/d as pairs (d_high + d_low, u_high + u_low); sinh = (d − u)/2 and cosh = (d + u)/2, each times the power of two returned. Near 0, d − u loses no digit: d_high − u_high is exact and the pairs' tails carry the rest.

Definition at line 335 of file elementary.hpp.

◆ hypot3()

double numerical::detail::hypot3 ( double  x,
double  y,
double  z 
)
inline

√(x² + y² + z²) without overflow, LAPACK's dlapy3.

Definition at line 826 of file generalized.hpp.

◆ ieee_maximum()

template<std::floating_point T>
T numerical::detail::ieee_maximum ( T  a,
T  b 
)
noexcept

The larger of a and b as IEEE 754-2019's maximum orders them: a NaN wins (a's where both are), and +0 is over −0 whatever their order (a + b of two zeros: −0 only where both are).

Definition at line 2833 of file tensor_ops.hpp.

◆ ieee_minimum()

template<std::floating_point T>
T numerical::detail::ieee_minimum ( T  a,
T  b 
)
noexcept

The smaller of a and b as IEEE 754-2019's minimum orders them: a NaN wins (a's where both are), and −0 is under +0 whatever their order (−(−a − b) of two zeros: +0 only where both are).

Definition at line 2848 of file tensor_ops.hpp.

◆ integer_power() [1/2]

template<typename T >
matrix< T > numerical::detail::integer_power ( const matrix< T > &  a,
double  e 
)
private

a to the integer power e (a finite whole number) by repeated squaring, as numpy.linalg.matrix_power: the identity for 0, the inverse's powers for e < 0.

Definition at line 3335 of file matrix_functions.hpp.

◆ integer_power() [2/2]

template<std::signed_integral T>
T numerical::detail::integer_power ( T  base,
T  exponent 
)
noexcept

base to the power exponent (not negative) by repeated squaring, each product wrapped modulo 2ⁿ as NumPy's integer power (arithmetic).

Definition at line 3184 of file tensor_ops.hpp.

◆ inverse_factorials()

const std::array< double_double, 30 > & numerical::detail::inverse_factorials ( )
inline

The terms of dd_sin_cos' series: 1/k! for k = 0 to 29, each from the last by dd_div.

Definition at line 126 of file fft.hpp.

◆ inverse_iteration()

bool numerical::detail::inverse_iteration ( const buffer< double > &  diagonal,
const buffer< double > &  off,
std::size_t  n,
const buffer< double > &  values,
buffer< double > &  z,
std::size_t  most_solves = 5 
)
inlineprivate

The eigenvectors (rows of z, k × n) of the symmetric tridiagonal of diagonal diagonal and off-diagonal off (n from 2) for its eigenvalues values (k, ascending), as LAPACK's dstein: inverse iteration from a random vector, each solve of T − λ I scaled so that the iterate's largest entry reaches √(0.1 / n) once the eigenvector dominates, then two more; an eigenvalue within 10 ε |λ| of the one before moved up to that distance; the vectors of eigenvalues within 10⁻³ ‖T‖₁ of one another (a cluster) orthogonalised against the cluster's earlier ones at each step. The clusters on the threads.

Parameters
[in]diagonalT's diagonal.
[in]offT's off-diagonal (its first n − 1 entries read).
[in]nThe order.
[in]valuesThe eigenvalues, ascending.
[out]zThe eigenvectors as rows.
[in]most_solvesThe solves allowed a vector, dstein's five by default.
Returns
Whether every vector converged within most_solves and every cluster had at most inverse_iteration_cluster_most eigenvalues: if not, z holds nothing to use.

Definition at line 2698 of file svd.hpp.

◆ inverse_on_kept()

template<typename T >
matrix< T > numerical::detail::inverse_on_kept ( const vector< double > &  sigma,
const matrix< T > &  right,
const matrix< T > &  left,
rank_cutoff  cutoff 
)

Σ_d right_d (1 / σ_d) left_dᴴ over the columns d of right (cols × k) and left (rows × k) whose sigma is above cutoff (of the largest magnitude in sigma): a pseudo-inverse from a singular value decomposition (right = V, left = U) or from an eigendecomposition (V both, σ the eigenvalues, signed). Through the matrix product kernel: summed term by term it ran five times as long as NumPy's at order 512.

Definition at line 3101 of file svd.hpp.

◆ inverse_squaring()

template<typename T >
std::tuple< buffer< T >, std::size_t, std::size_t > numerical::detail::inverse_squaring ( const buffer< T > &  t0,
std::size_t  n,
const std::array< double, 8 > &  theta = log_pade_bounds,
const char *  name = "logm" 
)

Al-Mohy and Higham's inverse scaling and squaring (2012, algorithm 4.1) on the upper triangular t0, as SciPy's _inverse_squaring_helper: s square roots of T until T − I is small enough for the [m/m] Padé approximant whose bounds are theta (log(I + X)'s, or (I − X)ᵗ's; m from 1 to 7), the norms ‖(T − I)ᵖ‖₁^(1/p) exact; R = T − I with its diagonal and superdiagonal recomputed without cancellation where the principal branch exists.

Returns
R, s and m.

Definition at line 2886 of file matrix_functions.hpp.

◆ irfft_of()

template<std::floating_point T>
tensor< T > numerical::detail::irfft_of ( const fft_source< std::complex< T > > &  a,
const std::optional< std::size_t > &  n,
std::ptrdiff_t  axis,
fft_norm  norm 
)

irfft's computation, of complex elements a caller lends.

Definition at line 2927 of file fft.hpp.

◆ is_finite_entry() [1/2]

template<typename U >
bool numerical::detail::is_finite_entry ( const std::complex< U > &  x)

Whether x is finite: both of its parts.

Parameters
[in]xThe entry.

Definition at line 618 of file factorizations.hpp.

◆ is_finite_entry() [2/2]

template<typename T >
bool numerical::detail::is_finite_entry ( const T &  x)

Whether x is finite.

Parameters
[in]xThe entry.

Definition at line 611 of file factorizations.hpp.

◆ is_label()

bool numerical::detail::is_label ( char  ch)
inline

Is ch an ASCII letter (a valid index label)?

Definition at line 1212 of file tensor.hpp.

◆ is_nan() [1/2]

template<typename T >
bool numerical::detail::is_nan ( const T &  )

A NaN? Never for a scalar without one.

Definition at line 1564 of file tensor_ops.hpp.

◆ is_nan() [2/2]

bool numerical::detail::is_nan ( double  x)
inline

A NaN?

Definition at line 1570 of file tensor_ops.hpp.

◆ isolate()

template<typename T >
balancing numerical::detail::isolate ( buffer< T > &  a,
std::size_t  n 
)

Isolates eigenvalues of the row-major n × n a in place by permutations alone, as LAPACK's ‘dgebal('P’)`: balance without its scaling, a similarity by an orthogonal matrix.

Definition at line 215 of file eigen.hpp.

◆ jacobi_rotate()

void numerical::detail::jacobi_rotate ( buffer< double > &  a,
buffer< double > &  v,
std::size_t  n,
std::size_t  p,
std::size_t  q 
)
inline

One Jacobi rotation of the n × n a zeroing a[p][q], accumulated into v.

Precondition
a[p][q] is not 0 (cyclic_jacobi rotates a pair past DBL_MIN alone).

Definition at line 6266 of file factorizations.hpp.

◆ join_bidiagonal_halves()

void numerical::detail::join_bidiagonal_halves ( std::vector< bidiagonal_node > &  nodes,
std::size_t  index,
arrow_workspace &  work 
)
inline

Joins the halves of nodes' block index in work, then lets them go.

Definition at line 1429 of file svd.hpp.

◆ join_bidiagonal_node()

void numerical::detail::join_bidiagonal_node ( bidiagonal_node &  node,
const bidiagonal_node &  one,
const bidiagonal_node &  two,
arrow_workspace &  work 
)
inline

Joins the two decomposed halves of node, one and two, by merge_arrow, node started (start_bidiagonal_node).

Definition at line 1350 of file svd.hpp.

◆ join_divide_node()

void numerical::detail::join_divide_node ( const divide_node &  node,
merge_workspace &  work 
)
inline

Joins the two solved halves of node by merge_rank_one, in work.

Definition at line 6085 of file factorizations.hpp.

◆ kernel_asinh()

template<std::floating_point R>
std::complex< R > numerical::detail::kernel_asinh ( const std::complex< R > &  z,
bool  adjust 
)

asinh(z) of a finite nonzero z, as glibc's __kernel_casinh, its imaginary part taken from π/2 where adjust (acos and acosh through it): the first quadrant's |re| and |im|, by the region they lie in, log(2z) past 1/ε, log(z + √(1 + z²)) by log1p and atan2 of parts that do not cancel nearer 1, else that sum by complex_sqrt and complex_log; the signs of z's parts given back last. Each product whose every use is a sum fused where the C library's build fuses (libm_add_product, libm_sum_of_products): GCC's contraction of glibc on arm64.

Definition at line 818 of file complex_math.hpp.

◆ laid_along()

template<typename T >
std::optional< laid_axis< T > > numerical::detail::laid_along ( const tensor< T > &  t,
std::size_t  axis 
)

t laid out and the axis matching axis: nothing for any tensor but a transpose, or an axis past the rank (the operation's own to refuse).

Definition at line 1964 of file tensor_ops.hpp.

◆ larfg_in_place()

template<typename T >
std::pair< T, T > numerical::detail::larfg_in_place ( buffer< T > &  w)
private

The reflector LAPACK's dlarfg (zlarfg) forms of w, in place: H = I − τ w wᴴ with w₀ = 1 and Hᴴ x = β e₁, β real, of the sign opposite to Re x₀. Exact inputs stay exact where LAPACK's do (e₂ reflected onto e₁: w = (1, 1), τ = 1, where a unit v holds 1/√2, rounded). For an x with something below x₀, or x₀ complex (the identity otherwise, which the caller keeps).

Returns
β (as T, its imaginary part 0) and τ.

Definition at line 3214 of file factorizations.hpp.

◆ largest_magnitude()

template<typename Entry >
double numerical::detail::largest_magnitude ( std::size_t  rows,
std::size_t  cols,
const Entry &  entry 
)

The power_of_two_scale of the largest magnitude of entry over rows rows of cols: by bands of rows on the threads, each band's largest then the largest of those (a maximum, the same whatever the order).

Definition at line 839 of file svd.hpp.

◆ largest_magnitude_vectors()

template<typename T , typename Magnitude >
std::size_t numerical::detail::largest_magnitude_vectors ( const T *  ,
std::size_t  ,
Magnitude &  ,
std::size_t &   
)

Only x[0], the one best and at come in with, for a type no native vector holds.

Definition at line 1000 of file simd.hpp.

◆ largest_part() [1/2]

double numerical::detail::largest_part ( const std::complex< double > &  x)
inline

max(|re x|, |im x|): within √2 of |x|, without its square root.

Definition at line 92 of file eigen.hpp.

◆ largest_part() [2/2]

double numerical::detail::largest_part ( double  x)
inline

|x| of a real x.

Definition at line 86 of file eigen.hpp.

◆ largest_prime_factor()

std::size_t numerical::detail::largest_prime_factor ( std::size_t  n)
inlinenoexcept

The largest prime factor of n, at least 2.

Definition at line 1012 of file fft.hpp.

◆ left_kernel_for()

template<typename T >
left_kernel numerical::detail::left_kernel_for ( std::size_t  ,
std::size_t  ,
std::size_t   
)
noexcept

The tile gemm runs a product of m × n by depth k on, for a left operand packed once for many products: plain for a scalar other than double (written as an overload per scalar, as conj_of).

Definition at line 4871 of file gemm.hpp.

◆ left_kernel_for< double >()

template<>
left_kernel numerical::detail::left_kernel_for< double > ( std::size_t  m,
std::size_t  n,
std::size_t  k 
)
inlinenoexcept

left_kernel_for a double product: past gemm_rows, as gemm_elsewhere chooses, of a depth of one panel; plain otherwise.

Definition at line 4881 of file gemm.hpp.

◆ left_rows()

template<typename T >
packed_left< T > numerical::detail::left_rows ( const packed_left< T > &  left,
std::size_t  r0,
std::size_t  rows 
)
noexcept

The rows [r0, r0 + rows) of left, r0 a multiple of left.granule: their slivers in its packing, or their rows of A in place.

Definition at line 4857 of file gemm.hpp.

◆ less_product()

template<typename T >
T numerical::detail::less_product ( T  held,
const T &  factor,
const T &  x 
)
inline

held − factor · x for the scalar blocks beside GCC's vectors, rounded as those vectors round it: under GCC the product a statement of its own, since GCC fuses no multiply-add of its vectors but contracts a scalar expression where its target has the instruction (an x86-64-v3 baseline, the default of Ubuntu 26.04's GCC on GitHub's runners, or arm64), and a row would then be rounded two ways, split by the count each call gets (an inverse of order 129 by column slices on two threads: 2700 of its entries changed). Elsewhere the expression as written, as the compiler rounds it (Clang vectorises these blocks itself, with no vectors of ours beside them).

Definition at line 1180 of file simd.hpp.

◆ libm_add_product()

template<std::floating_point R>
R numerical::detail::libm_add_product ( R  s,
R  a,
R  b 
)

s + a·b as the C library's build computes s += a·b: fused where it fuses, as libm_sum_of_products.

Definition at line 145 of file complex_math.hpp.

◆ libm_fuses()

bool numerical::detail::libm_fuses ( )
inlinenoexcept

Whether the C library's complex functions fuse their sums of products, a·b + c·d as fma(a, b, c·d), off arm64 (where FMA is in every build's baseline, and they do): under glibc as its build does, read once from its ctan of a number whose cos² + sinh² the two round apart. glibc 2.43 fuses there on an AMD EPYC 7763 (GitHub's Ubuntu 26.04 runners) and not on an Intel Core i5-8500, the same library; NumPy's complex functions are glibc's. Elsewhere no.

Definition at line 100 of file complex_math.hpp.

◆ libm_sum_of_products()

template<std::floating_point R>
R numerical::detail::libm_sum_of_products ( R  a,
R  b,
R  c,
R  d 
)

a·b + c·d as the C library's build computes it: fma(a, b, c·d) where it fuses (libm_fuses), each product rounded elsewhere.

Definition at line 126 of file complex_math.hpp.

◆ library_power()

template<split_complex C>
C numerical::detail::library_power ( const C &  a,
const C &  b 
)

a ** b by std::pow: libstdc++'s calls the C library's cpow, which NumPy calls (the same bits under glibc); MSVC's is its own.

Definition at line 218 of file tensor_ops.hpp.

◆ lies_close()

template<typename T >
bool numerical::detail::lies_close ( const tensor< T > &  t)
noexcept

Whether an element-wise operation reads the view t's elements where they lie: stepping within a cache line along its last axis (a slice).

Precondition
t is a view (not tensor::contiguous). A view stepping farther (a transpose's last axis, its elements a row apart) is copied out row-major first, by blocks: read in place, each element would load its own line.

Definition at line 2019 of file tensor_ops.hpp.

◆ lies_row_major()

bool numerical::detail::lies_row_major ( const std::vector< std::size_t > &  shape,
const std::vector< std::ptrdiff_t > &  strides 
)
inline

Whether elements strides apart along the axes of shape lie row-major from the first (an axis of fewer than two elements stepping anything).

Definition at line 723 of file tensor.hpp.

◆ line_aligned()

template<typename T >
T * numerical::detail::line_aligned ( T *  at)
noexcept

The first element of at's run that starts a cache line (or at's nearest element towards one, for an element whose size does not divide the allocator's alignment).

Definition at line 2371 of file gemm.hpp.

◆ lines_along()

tensor< Out > numerical::detail::lines_along ( const fft_source< In > &  a,
std::size_t  index,
std::size_t  in_points,
fft_output< Out > &&  result,
std::size_t  work,
Line  line 
)

A transform of each line of a along axis index into the lines of result: each line gathered (its first in_points elements, padded with zeros), passed with its output line, both together, and the thread's working space of work elements of C to line, then scattered; a line together and not padded passed where it lies, its output line too. The lines shared among the threads. result holds elements: its callers return an empty one before they plan.

Definition at line 2776 of file fft.hpp.

◆ log1p_into()

void numerical::detail::log1p_into ( const double *  in,
double *  out,
std::size_t  count 
)
inline

log1p of the count doubles at in into out, each elementary_log1p's: table_log1p in vectors (with the FMA instructions on an x86-64 processor, run_fused), std::log1p throughout where exp_by_table says no.

Definition at line 890 of file tensor_ops.hpp.

◆ log1p_of() [1/3]

tensor< double > numerical::detail::log1p_of ( const double *  data,
const std::vector< std::size_t > &  shape 
)
inline

log1p of every one of the row-major doubles at data, of shape shape: log1p_into by chunks of elementwise_chunk among the threads.

Definition at line 927 of file tensor_ops.hpp.

◆ log1p_of() [2/3]

tensor< float > numerical::detail::log1p_of ( const float *  data,
const std::vector< std::size_t > &  shape 
)
inline

log1p of every one of the row-major floats at data, of shape shape: log1p_into of them as doubles (through_doubles).

Definition at line 948 of file tensor_ops.hpp.

◆ log1p_of() [3/3]

template<typename T >
tensor< T > numerical::detail::log1p_of ( const T *  data,
const std::vector< std::size_t > &  shape 
)

log1p of every one of the row-major scalars at data, of shape shape: std::log1p of each.

Definition at line 919 of file tensor_ops.hpp.

◆ log1p_viewed() [1/3]

tensor< double > numerical::detail::log1p_viewed ( const tensor< double > &  a)
inline

log1p of every double of a, a view read where it lies (by_runs).

Definition at line 2580 of file tensor_ops.hpp.

◆ log1p_viewed() [2/3]

tensor< float > numerical::detail::log1p_viewed ( const tensor< float > &  a)
inline

log1p of every float of a through doubles, a view read where it lies (by_runs).

Definition at line 2586 of file tensor_ops.hpp.

◆ log1p_viewed() [3/3]

template<typename T >
tensor< T > numerical::detail::log1p_viewed ( const tensor< T > &  a)

log1p of every element of a: of a contiguous copy (log1p_of).

Definition at line 2574 of file tensor_ops.hpp.

◆ log_parts()

void numerical::detail::log_parts ( double  u,
double  u_low,
double &  sum,
double &  small 
)
inlinenoexcept

ln(u + u_low) as sum + small unrounded, for u at least 2^−54 and finite and |u_low| at most half an ulp of it: u = 2^e m, m in [0.703125, 1.40625), c = k/128 the nearest point, r = m RN(1/c) − 1 + u_low RN(1/c)/2^e as a normalized pair; e ln 2 + (−ln RN(1/c)) + r − r²/2 summed exactly (an exact sum of multiples of 2^−42, two-sums, a fused square's error) into sum, and the rest, r³/3 − … − r¹⁰/10, the low parts and r's low part by the derivative 1 − r + r², into small. Near 0, u_low is as large as r, under which small would round to an ulp of the result. Every product beside a sum fused explicitly, as table_expm1's.

Definition at line 895 of file elementary.hpp.

◆ log_superdiagonal() [1/2]

std::complex< double > numerical::detail::log_superdiagonal ( const std::complex< double > &  l1,
const std::complex< double > &  l2,
const std::complex< double > &  t12 
)
inline

The superdiagonal entry of log T for the 2 × 2 upper triangular [[l1, t12], [0, l2]], as SciPy's _logm_superdiag_entry.

Definition at line 2744 of file matrix_functions.hpp.

◆ log_superdiagonal() [2/2]

double numerical::detail::log_superdiagonal ( double  l1,
double  l2,
double  t12 
)
inline

log_superdiagonal of real l1 and l2 above 0 (the unwinding number 0).

Definition at line 2795 of file matrix_functions.hpp.

◆ lower_is_finite()

template<typename T >
bool numerical::detail::lower_is_finite ( const matrix< T > &  a)

Whether the lower triangle of the square a, and its diagonal's real part, are finite: entry by entry, kept out of line: the rare road of require_finite_lower where the whole matrix is not finite.

Definition at line 699 of file factorizations.hpp.

◆ lower_triangle_product() [1/2]

bool numerical::detail::lower_triangle_product ( std::size_t  m,
std::size_t  depth,
const double *  a,
const double *  b,
double *  c,
std::size_t  ldc 
)
inline

lower_triangle_product for double.

Definition at line 6439 of file factorizations.hpp.

◆ lower_triangle_product() [2/2]

template<typename T >
bool numerical::detail::lower_triangle_product ( std::size_t  ,
std::size_t  ,
const T *  ,
const T *  ,
T *  ,
std::size_t   
)

C += A·B on the lower triangle of the m × m c (row stride ldc), a m × depth and b depth × m row-major, depth two blocks of pencil_block (past gemm_min_depth), where the matrix product kernel packs a product of that size: on the triangle's tiles alone (gemm_triangle).

Returns
whether it ran; a scalar other than double never does here.

Definition at line 6428 of file factorizations.hpp.

◆ lstsq_by_qr()

template<typename T >
std::optional< matrix< T > > numerical::detail::lstsq_by_qr ( const matrix< T > &  a,
const matrix< T > &  b,
rank_cutoff  cutoff,
buffer< double > &  values,
std::pair< buffer< T >, int > *const  r = nullptr 
)
private

The least-squares solution of A X ≈ B by a QR factorisation, of A, or of Aᴴ for a wide A (the minimum-norm solution), where it is the one the singular value decomposition gives: with A divided by a power of two near its largest magnitude, the bound 1 / ‖R⁻¹‖_F under its smallest singular value must stay qr_rank_margin times above cutoff of the bound ‖R‖_F over its largest, so that every singular value is far above the cutoff and the rank full (a zero on R's diagonal makes the first bound zero, or NaN, which fails). Else nothing. A fraction of the decomposition's work, the rest in matrix products.

Parameters
[in]aThe m × n matrix A.
[in]bThe right-hand sides, one per column.
[in]cutoffWhere a singular value counts as zero.
[out]valuesR's singular values (of A over 2^exponent), where they decided the rank because the bound was too loose; else left empty.
[out]rWhere given, the square R factorised, of A divided by 2^exponent, and that exponent: its singular values are A's over 2^exponent.
Returns
The solution, or nothing where the decomposition must decide the rank.

Definition at line 2052 of file svd.hpp.

◆ lstsq_from_svd()

template<typename T >
matrix< T > numerical::detail::lstsq_from_svd ( const svd_decomposition< T > &  s,
std::size_t  rank,
const matrix< T > &  b 
)
private

The minimum-norm least-squares solution of A X ≈ B from the singular value decomposition s of A (m × n), its first rank triplets: X = V Σ⁺ Uᴴ B.

Definition at line 2178 of file svd.hpp.

◆ lu_block_for()

constexpr std::size_t numerical::detail::lu_block_for ( std::size_t  n)
constexprnoexcept

The panel width of the LU of order n: a power of two, so a row's panel is a mask away, not a division.

Definition at line 174 of file factorizations.hpp.

◆ lu_reciprocal_condition()

template<typename T >
double numerical::detail::lu_reciprocal_condition ( const T *  lu,
std::size_t  n,
double  one_norm 
)

reciprocal_condition of A = P⁻¹ L U from the n × n row-major lu (L below the diagonal, its unit diagonal not stored, U on and above it) and one_norm, ‖A‖₁, as LAPACK's gecon: A⁻¹'s products U⁻¹ L⁻¹ and L⁻ᴴ U⁻ᴴ (the permutation leaves 1-norms alone). 0 where U has a zero on its diagonal.

Definition at line 2251 of file factorizations.hpp.

◆ make_cosine_line()

template<std::floating_point R>
cosine_line< R > numerical::detail::make_cosine_line ( int  type,
std::size_t  n 
)

The cosine_line of type on n points, its plan and roots formed: each root e^{−iπa/d} (d of n, 2n or 4n) as unit_roots of 8n rounds e^{−2πi(4n·a/d)/8n}, in double-double arithmetic on the threads, so the same on every platform.

Definition at line 3353 of file fft.hpp.

◆ make_fft_plan()

template<std::floating_point R>
fft_plan< R > numerical::detail::make_fft_plan ( std::size_t  n)

The fft_plan of length n, its roots computed on the threads (each the same on any).

Definition at line 1030 of file fft.hpp.

◆ make_small_radix_constants()

template<std::floating_point R>
small_radix_constants< R > numerical::detail::make_small_radix_constants ( )

small_radix_constants from the roots of order 3 and 5 (unit_root, whose imaginary parts are −sin).

Definition at line 862 of file fft.hpp.

◆ masked_select()

template<std::floating_point T>
T numerical::detail::masked_select ( bool  condition,
T  if_true,
T  if_false 
)
noexcept

if_true where condition holds, else if_false, by a mask of their bits: no branch, so a loop of it vectorises (GCC 13 makes a ?: of reals a branch).

Definition at line 2863 of file tensor_ops.hpp.

◆ matrix_times_vector()

template<typename T >
void numerical::detail::matrix_times_vector ( const T *  a,
std::size_t  lda,
std::size_t  rows,
std::size_t  cols,
const T *  x,
T *  y 
)

y = A x for the row-major rows × cols a, its rows lda apart: each element the dot product of its row with x, as chunked_dot sums it (a row of a chunk or less is one dot_product), the rows in parallel by groups (row_dots, which share each load of x). The same on any number of threads. A double matrix of amx_dot_least elements to fewer than amx_dot_most, on the AMX path, runs on the AMX (amx_dot_rows_f64): each element then its eight lanes of fused multiply-adds, added in order.

Definition at line 5153 of file gemm.hpp.

◆ merge_arrow()

void numerical::detail::merge_arrow ( buffer< double > &  d,
buffer< double >  z,
double  alpha,
double  beta,
buffer< double > &  u,
buffer< double > &  v,
std::size_t  n,
std::size_t  m,
std::size_t  split,
arrow_workspace &  work 
)
inline

The singular value decomposition of the arrow M (first row z, diagonal d, d₀ = 0), carried into the bases u (n × n) and v (m × m, its first n columns the arrow's; a last, null column when m = n + 1), as LAPACK's dlasd1–dlasd3: on return d holds the singular values ascending and u, v the singular vectors, v's null column last.

Deflation first (dlasd2): a negligible zⱼ leaves dⱼ and its vectors as they are, a dⱼ near zero is rotated into the zero pole, and two nearly equal dⱼ are rotated together so that one of them deflates. The remaining roots are found by singular_secular_root, in parallel; z is recomputed from them (Löwner's formula), so that the vectors are orthogonal to working precision, and they reach u and v by a matrix product each.

Parameters
[in,out]dThe diagonal; on return, the singular values.
[in]zThe first row.
[in]alpha,betaThe entries the arrow was built from: they set the deflation tolerance.
[in,out]uThe left basis; on return, the left singular vectors.
[in,out]vThe right basis; on return, the right singular vectors.
[in]nThe arrow's order.
[in]mThe right basis's order (n, or n + 1 with a null column).
[in]splitThe row of u the arrow's first row is (its upper half above, its lower below): the bases are block diagonal by it, and each half's rows take only the columns that reach them (dlasd3 multiplies the halves apart the same way).
[in,out]workThe buffers it works in, the caller's thread's (thread_arrow_workspace).

Definition at line 989 of file svd.hpp.

◆ merge_rank_one()

void numerical::detail::merge_rank_one ( buffer< double > &  d,
buffer< double >  z,
double  rho,
double *  q,
std::size_t  ldq,
std::size_t  n,
std::size_t  split,
merge_workspace &  work 
)
inline

The eigenvalues of diag(d) + ρ z zᵀ, ascending, into d, and its eigenvectors times q (the row-major n × n block of rows ldq apart whose columns span the halves) into q, as LAPACK's dlaed1.

Deflation first (dlaed2): a negligible zⱼ leaves dⱼ and its column as they are, and two nearly equal poles are rotated so that one of them does. The remaining k roots are found by secular_root, in parallel; z is then recomputed from them (Löwner's formula, Gu and Eisenstat), so that the eigenvectors (dⱼ − λᵢ)⁻¹ ẑⱼ are orthogonal to working precision, and they reach q by a matrix product per half: q is block diagonal, its first split columns zero below row split and the others above it, and only a deflating rotation mixes a column of each (dlaed3 multiplies the halves apart the same way).

Definition at line 5793 of file factorizations.hpp.

◆ merged_pieces()

std::size_t numerical::detail::merged_pieces ( std::size_t  threads)
inlinenoexcept

The pieces parallel_sort_run sorts a line in for threads: the least power of four that many.

Definition at line 3828 of file tensor_ops.hpp.

◆ mixed_precision_solve()

std::optional< matrix< double > > numerical::detail::mixed_precision_solve ( const matrix< double > &  a,
const matrix< double > &  b,
double *  reciprocal_condition = nullptr 
)
inlineprivate

X of A X = B (A square, B of its rows) as LAPACK's dsgesv: A factorised in single precision, its solution corrected in double precision by the residual R = B − A X (each correction a single-precision solve of A D = R) until every column meets ‖R‖∞ ≤ ‖X‖∞ ‖A‖∞ ε √n (ε double's unit roundoff). Each operand goes to single precision scaled by a power of two (scaled_to_float): none overflows, where dsgesv falls back. A correction must divide the largest ‖R‖∞ / ‖X‖∞ by 8 at least, which bounds their count (about a dozen from single precision to the bound) where dsgesv allows 30: an A too ill-conditioned for single precision converges slowly or not at all, and the double factorisation is then the faster. The single factorisation costs half the double one's; each correction a product and a solve of n² k.

Parameters
[in]aA, square.
[in]bB, of A's rows.
[out]reciprocal_conditionWhere given, set with X: A's reciprocal condition number estimated from the single-precision factors (lu_decomposition::reciprocal_condition). The corrections converged, so ‖A⁻¹‖ ‖E‖ is under 1 for the factors' error E, of order single precision's unit roundoff times ‖A‖: the estimate is the double factors' to a small factor, and far above double's unit roundoff.
Returns
X, or none where an entry of A or B is not finite, A is singular in single precision, or a correction falls short: the caller solves in double (and meets what the double solve meets).

Definition at line 1724 of file factorizations.hpp.

◆ modulus_1() [1/2]

double numerical::detail::modulus_1 ( const std::complex< double > &  x)
inline

|re x| + |im x|, the cheap modulus LAPACK's complex routines judge with.

Definition at line 62 of file eigen.hpp.

◆ modulus_1() [2/2]

double numerical::detail::modulus_1 ( double  x)
inline

|x| of a real x: what modulus_1 of a complex one stands for.

Definition at line 56 of file eigen.hpp.

◆ modulus_one()

double numerical::detail::modulus_one ( const std::complex< double > &  z)
inline

|Re z| + |Im z|, LAPACK's ABS1.

Definition at line 35 of file generalized.hpp.

◆ modulus_power_norm()

double numerical::detail::modulus_power_norm ( const buffer< double > &  moduli,
std::size_t  n,
std::size_t  p,
double  scale 
)
inline

‖ |A|ᵖ ‖₁ of the n × n row-major moduli |A| (n from 1) times scale, exactly: the row 1ᵀ |scale A|ᵖ by p products of a vector and |A|, a matrix of moduli's norm its largest entry.

Definition at line 90 of file matrix_functions.hpp.

◆ move_leading()

template<typename T >
std::size_t numerical::detail::move_leading ( buffer< T > &  t,
buffer< T > &  qt,
std::size_t  n,
const std::vector< bool > &  chosen 
)
private

Moves the eigenvalues of the Schur form t whose entries of chosen are set to its leading block, the rows of qt (Zᵀ) turning with it, in their order, as LAPACK's trsen (dtrexc or ztrexc): each chosen block in turn to the first place past those already moved; a real pair moves where either of its two is chosen.

Returns
the eigenvalues moved.
Exceptions
convergence_errorIf an exchange was refused (the blocks too close to separate), as trsen's info.

Definition at line 3462 of file eigen.hpp.

◆ move_real_block_up()

void numerical::detail::move_real_block_up ( real_qz_form &  f,
std::size_t  ifst,
std::size_t  ilst 
)
inline

Moves the diagonal block at ifst (from 1) of the real generalized Schur form f up to ilst, as LAPACK's dtgexc does it upwards: exchanged with each block above in turn (exchange_real_blocks), a 2 × 2 block that splits on the way moved as its two 1 × 1 ones; ifst and ilst taken to the first row of their blocks.

Exceptions
convergence_errorIf an exchange is refused (exchange_or_refuse); f then partly moved.

Definition at line 2052 of file generalized.hpp.

◆ multiply_real_left() [1/3]

void numerical::detail::multiply_real_left ( const buffer< double > &  left,
buffer< double > &  zt,
std::size_t  n 
)
inline

zt ← left · zt, n × n, row-major, for a real left factor.

Definition at line 6208 of file factorizations.hpp.

◆ multiply_real_left() [2/3]

void numerical::detail::multiply_real_left ( const buffer< double > &  left,
buffer< std::complex< double > > &  zt,
std::size_t  n 
)
inline

zt ← left · zt for a complex zt: the real and imaginary parts by a real product each.

Definition at line 6218 of file factorizations.hpp.

◆ multiply_real_left() [3/3]

template<typename T >
void numerical::detail::multiply_real_left ( const buffer< double > &  left,
buffer< T > &  zt,
std::size_t  n 
)

zt ← left · zt, n × n, row-major, for any other element type: the real factor converted.

Definition at line 6194 of file factorizations.hpp.

◆ multishift_from() [1/2]

constexpr lapack_index numerical::detail::multishift_from ( const std::complex< double > &  )
constexpr

The same for a complex one (order 76 1.8 against 2.1 ms, 128 7.5 against 6.5; with the vectors, 100, 6.4 against 5.6).

Definition at line 1291 of file eigen.hpp.

◆ multishift_from() [2/2]

constexpr lapack_index numerical::detail::multishift_from ( double  )
constexpr

The order from which a real Schur form is taken by multishift_schur, below which real_schur is faster (eight arm64 cores: order 128 1.85 against 2.60 ms; 224, 8.9 against 8.6; with the vectors, 192, 8.1 against 7.1).

Definition at line 1284 of file eigen.hpp.

◆ multishift_schur()

template<typename T >
void numerical::detail::multishift_schur ( const schur_target< T > &  target,
std::complex< double > *  w,
bool  recursive,
lapack_index  steps 
)

The Schur form of target's active block by the multishift QR algorithm with aggressive early deflation, as LAPACK's dlaqr0 and zlaqr0 (dlaqr4 without recursive): each iteration deflates what it can from a window at the bottom of the active block (early_deflation) and, unless that deflated enough, sweeps the block with the window's other eigenvalues as shifts (chase_bulges). The eigenvalues are left in w (one per row; a real matrix's complex pairs adjacent, the positive imaginary part first).

Parameters
[in]targetThe Hessenberg matrix, its active block, and the Schur vectors to update.
[out]wThe eigenvalues, one per row of the matrix.
[in]recursiveWhether a wide deflation window is itself taken by these iterations (dlaqr0), or by the single-shift ones (dlaqr4).
[in]stepsThe iterations allowed, LAPACK's 30 · max(10, order) where zero.
Exceptions
convergence_errorIf the iterations did not converge within steps.

Definition at line 2544 of file eigen.hpp.

◆ narrow_block()

template<std::size_t R, std::size_t N, term_rounding Mode, typename T >
void numerical::detail::narrow_block ( std::size_t  k,
const T *  a,
std::size_t  lda,
const T *  b,
std::size_t  ldb,
T *  c,
std::size_t  ldc,
c_start  start 
)
noexcept

C += A·B for R rows and N columns of C (C = A·B as start says), each sum in ascending order of the depth, as the plain loop's: the block held in registers, of a size the compiler knows; each term rounded as Mode says. Out of line: its unrolled sizes, inlined, spent GCC's growth budget, and the LU factorisation, which runs none of them, lost 3 to 8 % at every order (one arm64 thread, GCC 14).

Definition at line 2119 of file gemm.hpp.

◆ narrow_rows()

template<std::size_t R, term_rounding Mode, typename T >
void numerical::detail::narrow_rows ( std::size_t  n,
std::size_t  k,
const T *  a,
std::size_t  lda,
const T *  b,
std::size_t  ldb,
T *  c,
std::size_t  ldc,
c_start  start 
)
noexcept

C += A·B for R whole rows of C of n columns (C = A·B as start says): blocks of narrow_columns, the last narrower.

Definition at line 2171 of file gemm.hpp.

◆ nearly_diagonal_power() [1/2]

buffer< std::complex< double > > numerical::detail::nearly_diagonal_power ( const buffer< std::complex< double > > &  t,
std::size_t  n,
double  p 
)
inlineprivate

T^t of a triangle t of order n whose strict upper part is a similarity's rounding (Zᴴ A Z, Z a Hermitian A's eigenvectors): its diagonal's principal powers, above them each entry times the divided difference of x^t between its diagonal entries (fractional_power_superdiagonal), Daleckii and Krein's first order, exact to the square of the rounding.

Definition at line 3177 of file matrix_functions.hpp.

◆ nearly_diagonal_power() [2/2]

buffer< double > numerical::detail::nearly_diagonal_power ( const matrix< double > &  similar,
double  p 
)
inlineprivate

nearly_diagonal_power of the upper triangle of a real similar whose diagonal is positive: real.

Definition at line 3192 of file matrix_functions.hpp.

◆ newton_refined()

matrix< std::complex< double > > numerical::detail::newton_refined ( const riccati_terms &  terms,
matrix< std::complex< double > > &&  x 
)
inlineprivate

One step of Newton's method on the continuous equation from the Schur method's x (Kleinman, 1968), in its correction form: A_Kᴴ Δ E + Eᴴ Δ A_K = −Res(X) for the closed loop A_K (continuous_residual), a Lyapunov equation solved for Y = Eᴴ Δ E as Fᴴ Y + Y F = −Res(X), F = E⁻¹ A_K; X + Δ kept only where its residual's 1-norm is smaller. The generalized Schur form of a badly scaled equation (a graded A) loses digits that a step from a solution so near the stabilising one gains back.

Definition at line 3033 of file generalized.hpp.

◆ no_registers()

template<std::floating_point R>
std::size_t numerical::detail::no_registers ( std::size_t  ,
const std::complex< R > *  ,
const std::complex< R > *  ,
std::complex< R > *   
)
noexcept

No recombination by registers: the scalar loop's from k = 1.

Definition at line 2526 of file fft.hpp.

◆ nonnegative_diagonal()

bool numerical::detail::nonnegative_diagonal ( const matrix< double > &  t)
inline

Whether no diagonal entry of t is below 0.

Definition at line 3464 of file matrix_functions.hpp.

◆ norm_at_scale()

template<typename T >
carried_t< T > numerical::detail::norm_at_scale ( const T *  x,
std::size_t  count 
)

The Euclidean norm of the count elements at x, summed at the scale of their largest magnitude: a plain sum of squares overflows near 1e155 and underflows to 0 near 1e-155.

Definition at line 494 of file factorizations.hpp.

◆ norm_one()

template<typename T >
double numerical::detail::norm_one ( const matrix< T > &  a)
private

The 1-norm of a: its largest column sum of moduli.

Definition at line 2736 of file generalized.hpp.

◆ normalize_vector()

void numerical::detail::normalize_vector ( std::complex< double > *  v,
std::size_t  n,
bool  signed_real 
)
inline

Scales v to unit Euclidean norm and turns it so that its first component of largest modulus is real: positive for a complex matrix (zgeev), of its real part's sign for a real one's pair (dgeev).

Definition at line 2868 of file eigen.hpp.

◆ numbers_end() [1/2]

std::int64_t * numerical::detail::numbers_end ( std::int64_t *  ,
std::int64_t *  last,
const std::int64_t *  ,
std::size_t   
)
inline

An integer has no NaN: every index is a number's.

Definition at line 3734 of file tensor_ops.hpp.

◆ numbers_end() [2/2]

template<std::floating_point T>
std::int64_t * numerical::detail::numbers_end ( std::int64_t *  first,
std::int64_t *  last,
const T *  values,
std::size_t  stride 
)

Where a real's NaNs begin among the indices [first, last), moved behind the rest in their order.

Definition at line 3725 of file tensor_ops.hpp.

◆ numerical_rank()

std::size_t numerical::detail::numerical_rank ( const vector< double > &  sigma,
std::size_t  rows,
std::size_t  cols,
rank_cutoff  cutoff = {} 
)
inlineprivate

How many of the singular values sigma of an rows × cols matrix exceed cutoff: its numerical rank (the first that many, where sigma is descending).

Definition at line 1963 of file svd.hpp.

◆ numpy_absolute() [1/2]

template<split_complex C>
C::value_type numerical::detail::numpy_absolute ( const C &  x,
bool  fused 
)

NumPy's complex magnitude (its absolute's vector loop): the parts' magnitudes, an infinite one making both infinite and then a NaN both NaN, then the larger times sqrt(ratio² + 1) for the ratio of the smaller to it (0 where the larger is 0 or the smaller infinite), ratio² + 1 one fused multiply-add where fused.

Definition at line 1292 of file tensor_ops.hpp.

◆ numpy_absolute() [2/2]

template<detail::split_complex C>
tensor< typename C::value_type > numerical::detail::numpy_absolute ( const tensor< C > &  a,
const numpy_loops &  loops = numpy_wheel_loops() 
)

|x| of every element of a complex tensor as NumPy's absolute computes it (detail::numpy_absolute), fused as loops says of its vector loops.

Definition at line 2287 of file tensor_ops.hpp.

◆ numpy_cumprod()

template<detail::split_complex C>
tensor< C > numerical::detail::numpy_cumprod ( const tensor< C > &  t,
std::size_t  axis,
const numpy_loops &  loops = numpy_wheel_loops() 
)

The running products of a complex t along axis, in order, each as NumPy's loops compute it (detail::numpy_multiply): its vector loop's for a line of two (one product), its reduction loop's past it; fused as loops says. The binding's cumprod.

Exceptions
dimension_errorIf axis is not an axis.

Definition at line 3418 of file tensor_ops.hpp.

◆ numpy_divide() [1/2]

template<split_complex C>
C numerical::detail::numpy_divide ( const C &  a,
const C &  b,
bool  fused 
)

a / b of complex numbers as NumPy's array loop computes it: Smith's division by the larger part of b (an exact zero divided part by part, to an infinity or a NaN), each x + y·z of it one fused multiply-add where fused (NumPy's build contracts them on arm64, not on x86-64).

Definition at line 138 of file tensor_ops.hpp.

◆ numpy_divide() [2/2]

template<detail::split_complex C>
tensor< C > numerical::detail::numpy_divide ( const tensor< C > &  a,
const tensor< C > &  b,
const numpy_loops &  loops = numpy_wheel_loops() 
)

a / b of complex tensors, broadcast, each quotient as NumPy's array loop computes it (detail::numpy_divide, fused as loops says): the binding's /.

Definition at line 2238 of file tensor_ops.hpp.

◆ numpy_fuses_products()

bool numerical::detail::numpy_fuses_products ( )
inlinenoexcept

Whether NumPy's complex product fuses its multiply-adds on this processor: on arm64, and on x86-64 where its AVX2 loop runs (FMA with it); elsewhere its loop rounds each product.

Definition at line 1327 of file tensor_ops.hpp.

◆ numpy_fuses_quotients()

bool numerical::detail::numpy_fuses_quotients ( )
inlinenoexcept

Whether NumPy's complex quotient fuses its multiply-adds, as its PyPI wheels are built: on macOS arm64, whose clang contracts them (FMA is arm64's baseline); not on Linux, whose GCC does not, nor on x86-64, whose baseline has no FMA.

Definition at line 1362 of file tensor_ops.hpp.

◆ numpy_fuses_reductions()

bool numerical::detail::numpy_fuses_reductions ( )
inlinenoexcept

Whether NumPy's complex product fuses its multiply-adds in its reduction loop (the product's loop whose output is its own first operand: numpy.prod along an axis nothing follows, numpy.cumprod past two elements), as its PyPI wheels are built: a scalar loop, contracted as numpy_fuses_scalar_products says (GCC's on Linux rounds each product, though its vector loop fuses).

Definition at line 1354 of file tensor_ops.hpp.

◆ numpy_fuses_scalar_products()

bool numerical::detail::numpy_fuses_scalar_products ( )
inlinenoexcept

Whether NumPy's complex product fuses its multiply-adds in its scalar loop (where numpy_loop_is_scalar holds), as its PyPI wheels are built: that loop sits beside the vector one, and clang (macOS) and MSVC (Windows) contract it where the vector loop has FMA; GCC (Linux, ISO C) does not.

Definition at line 1341 of file tensor_ops.hpp.

◆ numpy_loop_is_scalar() [1/2]

bool numerical::detail::numpy_loop_is_scalar ( const std::vector< std::size_t > &  a_shape,
const std::vector< std::size_t > &  b_shape 
)
inlinenoexcept

Whether NumPy multiplies an operand of shape a_shape by one of b_shape in its scalar loop rather than its vector one: when the result is one element and an operand of rank above 0 is broadcast to a higher rank, NumPy's iterator hands its loop an output step of 0, which the vector loop refuses.

Definition at line 1377 of file tensor_ops.hpp.

◆ numpy_loop_is_scalar() [2/2]

template<typename T >
bool numerical::detail::numpy_loop_is_scalar ( const tensor< T > &  a,
const tensor< T > &  b 
)
noexcept

numpy_loop_is_scalar of the shapes of a and b.

Definition at line 1391 of file tensor_ops.hpp.

◆ numpy_muladd()

template<std::floating_point R>
R numerical::detail::numpy_muladd ( R  x,
R  y,
R  z,
bool  fused 
)

x · y + z: one fused multiply-add where fused, else the product rounded first (statements apart, so no compiler contracts them).

Definition at line 78 of file tensor_ops.hpp.

◆ numpy_multiply() [1/2]

template<split_complex C>
C numerical::detail::numpy_multiply ( const C &  a,
const C &  b,
bool  fused 
)

a · b of complex numbers as NumPy's array loop computes it: ai·bi and ai·br rounded, then ar·br − ai·bi and ar·bi + ai·br, each by a fused multiply-add where fused (NumPy's vectors on arm64 and on x86-64 with FMA), else each product rounded first. No recovery of an infinity from a NaN, as std::complex's (C Annex G) makes: 1j · (inf + inf·j) is NaN, (1e308 + 1e308j) · (1e308 − 1e308j) inf + inf·j when fused.

Definition at line 98 of file tensor_ops.hpp.

◆ numpy_multiply() [2/2]

template<detail::split_complex C>
tensor< C > numerical::detail::numpy_multiply ( const tensor< C > &  a,
const tensor< C > &  b,
const numpy_loops &  loops = numpy_wheel_loops() 
)

a · b of complex tensors, broadcast, each product as NumPy's array loop computes it (detail::numpy_multiply, fused as loops says): the binding's *, where std::complex's operator (this library's *) recovers infinities NumPy's makes NaN.

Definition at line 2227 of file tensor_ops.hpp.

◆ numpy_multiply_late()

template<split_complex C>
C numerical::detail::numpy_multiply_late ( const C &  a,
const C &  b 
)

a · b of complex numbers as NumPy's reduction loop computes it where MSVC contracts it (its Windows wheels from 2.5): ai·bi and ar·bi rounded, then ar·br − ai·bi and ai·br + ar·bi each one fused multiply-add (the imaginary part's other product fused than in numpy_multiply).

Definition at line 112 of file tensor_ops.hpp.

◆ numpy_power() [1/2]

template<split_complex C>
C numerical::detail::numpy_power ( const C &  a,
const C &  b,
bool  fused 
)

NumPy's complex power a ** b (npymath's npy_cpow): 1 for a zero exponent, 0 or NaN for a zero base; an integer exponent below 100 in magnitude by products (numpy_multiply, from 1 past the cubes) and, when negative, the quotient of 1 by them (numpy_divide), each fused where fused; any other by the C library's cpow, as NumPy (library_power).

Definition at line 233 of file tensor_ops.hpp.

◆ numpy_power() [2/2]

template<detail::split_complex C>
tensor< C > numerical::detail::numpy_power ( const tensor< C > &  a,
const tensor< C > &  b,
const numpy_loops &  loops = numpy_wheel_loops() 
)

a ** b of complex tensors, broadcast, as NumPy's power computes each (detail::numpy_power), fused as loops says of its scalar loops.

Definition at line 2317 of file tensor_ops.hpp.

◆ numpy_prod() [1/2]

template<detail::split_complex C>
C numerical::detail::numpy_prod ( const tensor< C > &  t,
const numpy_loops &  loops = numpy_wheel_loops() 
)

The product of every element of a complex t, in order from 1 (NumPy's identity), as NumPy's reduction loop computes it (detail::numpy_multiply, fused as loops says).

Definition at line 2274 of file tensor_ops.hpp.

◆ numpy_prod() [2/2]

template<detail::split_complex C>
tensor< C > numerical::detail::numpy_prod ( const tensor< C > &  t,
std::size_t  axis,
const numpy_loops &  loops = numpy_wheel_loops() 
)

The products of a complex t along axis, each line in order from 1 (NumPy's identity, which turns an infinite element alone into a NaN part), each product as NumPy's loops compute it (detail::numpy_multiply): its reduction loop's where nothing follows the axis, its vector loop's (between whole slices) where something does; fused as loops says. The binding's prod, where std::complex's operator (prod) recovers infinities NumPy's makes NaN.

Exceptions
dimension_errorIf axis is not an axis.

Definition at line 2254 of file tensor_ops.hpp.

◆ numpy_reciprocal() [1/2]

template<split_complex C>
C numerical::detail::numpy_reciprocal ( const C &  x,
bool  fused 
)

NumPy's complex reciprocal 1 / x (its reciprocal, the loop x ** -1 takes): the ratio of the smaller part to the larger, d the larger plus the smaller times it (one fused multiply-add where fused), then 1 / d and −ratio / d — not numpy_divide's quotient of 1.

Definition at line 164 of file tensor_ops.hpp.

◆ numpy_reciprocal() [2/2]

template<detail::split_complex C>
tensor< C > numerical::detail::numpy_reciprocal ( const tensor< C > &  a,
const numpy_loops &  loops = numpy_wheel_loops() 
)

x ** -1 of a complex tensor as NumPy's reciprocal computes it (the loop of an array to the Python integer -1, detail::numpy_reciprocal), fused as loops says of its scalar loops.

Definition at line 2307 of file tensor_ops.hpp.

◆ numpy_reduction_multiply()

template<split_complex C>
C numerical::detail::numpy_reduction_multiply ( const C &  a,
const C &  b,
const numpy_loops &  loops 
)

a · b as NumPy's reduction loop computes it, fused as loops says.

Definition at line 2202 of file tensor_ops.hpp.

◆ numpy_square()

template<detail::split_complex C>
tensor< C > numerical::detail::numpy_square ( const tensor< C > &  a,
const numpy_loops &  loops = numpy_wheel_loops() 
)

x ** 2 of a complex tensor as NumPy's square computes it (the loop of an array to the Python integer 2): its product's vector loop, fused as loops says of it.

Definition at line 2297 of file tensor_ops.hpp.

◆ numpy_wheel_loops()

numpy_loops numerical::detail::numpy_wheel_loops ( )
inlinenoexcept

NumPy's loops as its current PyPI wheels build them on this processor (detail::numpy_fuses_products, detail::numpy_fuses_scalar_products, detail::numpy_fuses_quotients).

Definition at line 2212 of file tensor_ops.hpp.

◆ octant_sin_cos()

std::pair< double_double, double_double > numerical::detail::octant_sin_cos ( std::size_t  share,
std::size_t  n 
)
inline

The cosine and sine of (π/4)·share/n by dd_sin_cos.

Definition at line 213 of file fft.hpp.

◆ on_release()

bool numerical::detail::on_release ( void(*)()  step)
inlinenoexcept

Registers step among the calling thread's thread_releases, while a place is left (a workspace past them is not freed by release_thread_memory).

Returns
Whether it was, for a thread_local's initialiser beside the workspace it empties.

Definition at line 65 of file buffer.hpp.

◆ one_norm()

template<typename T >
double numerical::detail::one_norm ( const buffer< T > &  a,
std::size_t  n 
)

The 1-norm of the n × n row-major a (n from 1): its largest column sum of moduli.

Definition at line 82 of file matrix_functions.hpp.

◆ one_norm_estimate()

template<typename T >
double numerical::detail::one_norm_estimate ( std::size_t  n,
const std::function< void(T *, bool)> &  apply 
)

An estimate of ‖B‖₁ for the n × n operator apply gives, as LAPACK's dlacn2 and zlacn2 (Hager's method, Higham's refinements): apply(x, false) overwrites the n entries at x with B x, apply(x, true) with Bᴴ x. Products on sign vectors (unit-modulus ones for complex T) climb to a column of large norm, at most five each way; one more on (−1)ⁱ (1 + i / (n − 1)) guards against the operators that fool the climb. A lower bound, rarely short of ‖B‖₁ by more than a factor of 3 (Higham, Accuracy and Stability of Numerical Algorithms, 2002, §15.3); infinity where a product overflows.

Parameters
[in]nThe order of B (from 1).
[in]applyThe products: apply(T* x, bool adjoint), behind one type, so one estimate per T whatever the operator (each product costs n operations at the least, the call through it nothing).
Returns
The estimate.

Definition at line 2150 of file factorizations.hpp.

◆ one_sided_jacobi()

template<typename T >
void numerical::detail::one_sided_jacobi ( buffer< T > &  w,
buffer< T > &  v,
std::size_t  n,
std::size_t  max_sweeps 
)

One-sided (Hestenes) Jacobi on the columns of the n × n row-major w: plane rotations, accumulated into v, until every pair of columns is orthogonal to rounding. The column norms are then the singular values of the input, each to high relative accuracy.

Parameters
[in,out]wThe matrix whose columns are orthogonalised.
[in,out]vThe rotations, accumulated (start from the identity).
[in]nThe order.
[in]max_sweepsThe bound of sweeps.
Exceptions
convergence_errorIf a sweep still rotated at the bound.

Definition at line 64 of file svd.hpp.

◆ operand_of() [1/2]

real_operand numerical::detail::operand_of ( const double *  v,
std::size_t  stride,
std::size_t  count 
)
inline

The count entries stride apart at v as a real_operand.

Definition at line 407 of file eigen.hpp.

◆ operand_of() [2/2]

complex_operand numerical::detail::operand_of ( const std::complex< double > *  v,
std::size_t  stride,
std::size_t  count 
)
inline

The same as a complex_operand.

Definition at line 419 of file eigen.hpp.

◆ operator==()

template<typename T >
bool numerical::detail::operator== ( const std::vector< T, default_init_allocator< T > > &  buffer,
const std::vector< T > &  other 
)

Element-wise equality of a numerical::buffer and a std::vector, found by argument-dependent lookup through the allocator: code that compared a tensor's elements with a std::vector keeps doing so.

Definition at line 430 of file buffer.hpp.

◆ operator_times()

template<typename T >
T numerical::detail::operator_times ( const T &  a,
const T &  b 
)
inline

a · b, as std::operator* computes it, in line. libc++'s complex product (the four products and their sums as statements of their own, so unfused) is copied here, its rescue of a product whose parts are both NaN left to it: where it was a call, out of line in a translation unit as large as the Python extension, it took 45 % of a complex LU of order 48. Elsewhere (libstdc++, whose product the compiler expands) and for a real T, the operator itself.

Definition at line 1122 of file simd.hpp.

◆ order_key()

template<std::floating_point T>
order_key_t< T > numerical::detail::order_key ( T  x)
noexcept

x's bits as an unsigned integer ordered as x is by <: the sign bit set for a positive real, every bit flipped for a negative one. A NaN and the zeros' signs are the caller's: −0 orders below +0.

An integer's bits ordered as it is: its sign bit flipped.

Definition at line 3513 of file tensor_ops.hpp.

◆ order_run()

template<radix_orderable T, typename Less >
void numerical::detail::order_run ( std::int64_t *  first,
std::int64_t *  last,
const T *  values,
std::size_t  stride,
const Less &  less,
std::size_t  parts = 1 
)

Sorts the indices [first, last) stably by their elements (values[k · stride]), as sorts_before orders them: comparisons for a run shorter than radix_sort_least, else the NaNs set apart in their order and the rest by argsort_key, a byte at a time (radix_passes, in parts among the threads: a stable counting sort of the keys and their indices together; a byte that every key shares skipped): equal elements by index either way.

Sorts the indices [first, last) stably by their elements, by comparisons (any other type), on the caller's thread.

Definition at line 3751 of file tensor_ops.hpp.

◆ order_shared()

template<radix_orderable T, typename Less >
void numerical::detail::order_shared ( std::int64_t *  order,
std::size_t  count,
const T *  values,
std::size_t  stride,
const Less &  less,
std::size_t  threads 
)

Sorts the indices of one long line among threads: each pass of its radix sort shared out.

Sorts the indices of one long line of any other type among threads: in pieces, then merged.

Definition at line 3897 of file tensor_ops.hpp.

◆ over_pivot() [1/2]

template<typename R >
std::complex< R > numerical::detail::over_pivot ( const std::complex< R > &  x,
const complex_divisor< R > &  divisor 
)

over_pivot of a complex x: times the pivot's reciprocal, the product in line; over a subnormal pivot itself.

Definition at line 99 of file factorizations.hpp.

◆ over_pivot() [2/2]

template<typename T >
T numerical::detail::over_pivot ( const T &  x,
const T &  divisor 
)

x over the pivot whose pivot_divisor is divisor: the same quotient wherever the LU takes it.

Definition at line 90 of file factorizations.hpp.

◆ over_real_diagonal() [1/2]

template<typename R >
std::complex< R > numerical::detail::over_real_diagonal ( const std::complex< R > &  x,
const std::complex< R > &  diagonal 
)

over_real_diagonal of a complex x: each part over the diagonal's real part, one rounding each, where a complex division took the C library's scaled one (several roundings, and other bits under another library).

Definition at line 119 of file factorizations.hpp.

◆ over_real_diagonal() [2/2]

template<typename T >
T numerical::detail::over_real_diagonal ( const T &  x,
const T &  diagonal 
)

x over a Cholesky factor's diagonal entry diagonal, real and positive whatever the scalar.

Definition at line 107 of file factorizations.hpp.

◆ pack_left()

template<typename T >
packed_left< T > numerical::detail::pack_left ( std::size_t  m,
std::size_t  ,
std::size_t  k,
const T *  a,
std::size_t  lda,
buffer< T > &   
)

The m × k a packed once for products with it of about n columns each (gemm_with), in storage (grown as needed, kept by the caller from one call to the next): on the tile gemm would choose for m × n × k. A scalar other than double (here), a depth of more than one panel, or a product gemm_rows takes, leaves A to be read in place.

Definition at line 4938 of file gemm.hpp.

◆ pack_left< double >()

template<>
packed_left< double > numerical::detail::pack_left< double > ( std::size_t  m,
std::size_t  n,
std::size_t  k,
const double *  a,
std::size_t  lda,
buffer< double > &  storage 
)
inline

pack_left for double, on the tile left_kernel_for chooses.

Definition at line 4938 of file gemm.hpp.

◆ pack_left_for()

template<typename Kernel , typename T >
const T * numerical::detail::pack_left_for ( std::size_t  m,
std::size_t  k,
const T *  a,
std::size_t  lda,
buffer< T > &  storage 
)

A packed for Kernel into storage, in parallel by its blocks of rows, as gemm_packed packs it.

Definition at line 4914 of file gemm.hpp.

◆ packs_a() [1/2]

template<typename Kernel , typename T >
requires reads_a_in_place<Kernel, T>
constexpr bool numerical::detail::packs_a ( const T *  a_packed,
std::size_t  n 
)
constexprnoexcept

packs_a for a kernel that reads A in place: nor where B is one sliver of columns (n), which reads each element of A once.

Definition at line 2527 of file gemm.hpp.

◆ packs_a() [2/2]

template<typename Kernel , typename T >
requires (!reads_a_in_place<Kernel, T>)
constexpr bool numerical::detail::packs_a ( const T *  a_packed,
std::size_t   
)
constexprnoexcept

Whether gemm_packed packs A for Kernel: unless given packed (a_packed).

Definition at line 2516 of file gemm.hpp.

◆ pade_approximant()

template<typename T >
buffer< T > numerical::detail::pade_approximant ( const buffer< T > &  a,
const std::array< const buffer< T > *, 4 > &  powers,
std::size_t  n,
std::size_t  m 
)

The [m/m] Padé approximant (V − U)⁻¹ (V + U) of the exponential of a, its even powers powers (A², A⁴, A⁶, and A⁸ for m = 9), as Higham's (2005): U the odd terms, V the even, A¹³'s terms by A⁶ (A⁶ (b₁₃ A⁶ + b₁₁ A⁴ + b₉ A²) + …).

Definition at line 197 of file matrix_functions.hpp.

◆ pade_coefficients()

const double * numerical::detail::pade_coefficients ( std::size_t  m)
inline

The coefficients b₀ … b_m of the [m/m] Padé approximant of the exponential, as Higham's (2005).

Definition at line 145 of file matrix_functions.hpp.

◆ pair_for_sweep() [1/2]

void numerical::detail::pair_for_sweep ( std::complex< double > *  ,
lapack_index  ,
lapack_index  ,
const std::complex< double > &   
)
inline

A complex matrix's shifts need no pairing.

Definition at line 2505 of file eigen.hpp.

◆ pair_for_sweep() [2/2]

void numerical::detail::pair_for_sweep ( std::complex< double > *  w,
lapack_index  ks,
lapack_index  kbot,
double   
)
inline

A real matrix's shifts brought into pairs, each a conjugate pair or two reals, as dlaqr0 does before its sweep (conjugate shifts already adjacent).

Definition at line 2489 of file eigen.hpp.

◆ pair_keeps()

std::string numerical::detail::pair_keeps ( const std::vector< std::string > &  labels,
std::size_t  i,
std::size_t  j,
const std::string &  output 
)
inline

The labels of labels[i] and labels[j] that the output or another operand carries, in their order of appearance, once each: what their contraction keeps.

Definition at line 2074 of file tensor.hpp.

◆ pair_shifts() [1/2]

void numerical::detail::pair_shifts ( std::complex< double > *  ,
lapack_index  ,
const std::complex< double > &   
)
inline

A complex matrix's shifts pair as they come.

Definition at line 1843 of file eigen.hpp.

◆ pair_shifts() [2/2]

void numerical::detail::pair_shifts ( std::complex< double > *  s,
lapack_index  count,
double   
)
inline

Arranges a real matrix's shifts so that each pair is a conjugate pair or two reals, as dlaqr5 begins (conjugate shifts already adjacent).

Definition at line 1828 of file eigen.hpp.

◆ pair_values()

std::pair< real_pencil_value, real_pencil_value > numerical::detail::pair_values ( real_qz_form &  f,
std::size_t  ilast,
const pencil_pair &  pair,
double  b11,
double  b22 
)
inline

The conjugate pair of the standardized 2 × 2 block ending at ilast of f (T diagonal, b11 and b22), as dhgeqz computes α and β from pair (dlag2's): β the norms of the diagonal of T after the complex rotations that would triangularise the block, α = (wr ± i wi) β / scale.

Definition at line 838 of file generalized.hpp.

◆ pairs_into()

template<typename T , typename Op , typename R = std::invoke_result_t<const Op&, const T&, const T&>>
tensor< R > numerical::detail::pairs_into ( const T *  a,
const T *  b,
const std::vector< std::size_t > &  shape,
const Op &  op 
)

op(a, b) element by element of the row-major a and b, both of shape shape, in op's result type R, by chunks of elementwise_chunk among the threads.

Definition at line 743 of file tensor.hpp.

◆ pairwise_combine()

template<typename T >
T numerical::detail::pairwise_combine ( std::size_t  count,
const T *&  next 
)

The subtrees' sums from next on (pairwise_leaves' order) added as pairwise_sum's tree over count values adds its halves.

Definition at line 1086 of file tensor.hpp.

◆ pairwise_leaves()

template<typename T >
void numerical::detail::pairwise_leaves ( std::size_t  count,
std::vector< std::size_t > &  out 
)

The lengths of pairwise_sum's subtrees of at most pairwise_leaf values of type T over count, in order.

Definition at line 1071 of file tensor.hpp.

◆ pairwise_quad()

template<typename T >
T numerical::detail::pairwise_quad ( const T *  p,
std::size_t  width 
)

Four partial sums at p, width apart, added pairwise: (p₀ + p₁) + (p₂ + p₃).

Definition at line 973 of file tensor.hpp.

◆ pairwise_sum()

template<typename T >
T numerical::detail::pairwise_sum ( const T *  first,
std::size_t  count,
std::size_t  stride 
)

Sum of count values from first at stride stride, by NumPy's pairwise summation (pairwise_tree).

Definition at line 1030 of file tensor.hpp.

◆ pairwise_sum_rows()

template<typename T >
void numerical::detail::pairwise_sum_rows ( const T *  first,
std::size_t  count,
std::size_t  stride,
std::size_t  width,
T *  out 
)

pairwise_sum of width neighbouring columns at once, into out: column q's sum that of pairwise_sum(first + q, count, stride), to the bit, each step adding a whole row of the columns (contiguous, stride apart) where a column was walked down stride at a time.

Definition at line 1127 of file tensor.hpp.

◆ pairwise_sums()

template<typename T >
void numerical::detail::pairwise_sums ( const T *  data,
std::size_t  outer,
std::size_t  count,
std::size_t  inner,
T *  out 
)

The pairwise sums of outer blocks of count rows of inner elements (row-major), each column of a block summed down its rows, into out (outer × inner): one column at a time where inner is 1, else by rows (pairwise_sum_rows), the same bits. The blocks, and their runs of pairwise_columns columns, shared among the threads: each sum computed alike by whichever thread.

Definition at line 1192 of file tensor.hpp.

◆ parallel_copy()

template<typename T >
void numerical::detail::parallel_copy ( const T *  in,
std::size_t  count,
T *  out 
)

Copies the count elements at in to out, split across the threads the work pays for, as parallel_zero does: a fresh copy's pages are faulted in by several threads, where a sequential copy pays every fault on one (a 2 MiB copy into fresh memory took 0.85 ms on one x86-64 core in a container, a third of the Cholesky factorisation of order 512).

Definition at line 930 of file parallel.hpp.

◆ parallel_copy_of()

template<typename T >
matrix< T > numerical::detail::parallel_copy_of ( const matrix< T > &  a)

A copy of a whose pages the threads fault in (parallel_copy), a decomposition's own storage: the matrix's copy faulted every one in on the calling thread (order 1000 on one arm64 core 0.64 ms, an eighth of the Cholesky factorisation; on eight cores 0.28 ms).

Definition at line 646 of file factorizations.hpp.

◆ parallel_extreme()

template<typename T , typename Order >
T numerical::detail::parallel_extreme ( const T *  first,
std::size_t  count,
const Order &  larger 
)

contiguous_extreme of count elements on the threads: runs of pairwise_leaf each its own, then their extremes'. Exact whatever the runs (a maximum rounds nothing), so the same answer on any number of threads. Four runs or fewer: on the caller, as parallel_pairwise_sum.

Definition at line 1793 of file tensor_ops.hpp.

◆ parallel_extreme_index()

template<typename T , typename Order >
std::int64_t numerical::detail::parallel_extreme_index ( const T *  first,
std::size_t  count,
const Order &  larger 
)

extreme_index of the count contiguous elements at first, by chunks on the threads: each chunk's own, then the first of the chunks' (a NaN's chunk wins at once, the earliest of equal ones otherwise).

Definition at line 4364 of file tensor_ops.hpp.

◆ parallel_fill()

template<typename T >
void numerical::detail::parallel_fill ( T *  out,
std::size_t  count,
const T &  value 
)

Sets the count elements at out to value, split across the threads as parallel_zero is, a fresh result's pages faulted in by several threads.

Definition at line 908 of file parallel.hpp.

◆ parallel_for()

template<typename Body >
void numerical::detail::parallel_for ( std::size_t  count,
std::size_t  work_per_index,
const Body &  body 
)

Runs body(i) for every i in [0, count).

The range is partitioned across as many threads as its work pays for (threads_for), and runs sequentially when that is fewer than two. body must be safe to invoke concurrently for distinct i (it is when each call touches a disjoint part of the output).

An exception body throws in a worker is kept and rethrown here once every worker has joined (the first chunk's, in index order): one escaping a worker thread would call std::terminate. A worker the system cannot start is a std::system_error, thrown once the others have joined.

Parameters
[in]countThe number of iterations.
[in]work_per_indexThe work of one iteration, in multiply-adds.
[in]bodyThe per-index work (void(std::size_t)).

Definition at line 790 of file parallel.hpp.

◆ parallel_for_beside()

template<typename Body , typename Side >
void numerical::detail::parallel_for_beside ( std::size_t  count,
std::size_t  work_per_index,
const Body &  body,
const Side &  side 
)

Runs side on the caller while the pool's workers run body(i) for every i in [0, count), then lets the caller take the chunks left: one task beside a parallel loop, each touching what the other does not (a factorisation's next panel beside its trailing update). side's own parallel loops find the pool busy and run in its thread. Where the work pays for fewer than two threads, or the pool is busy, side runs, then the loop, on the caller. An exception of side comes back after the loop's chunks have joined, before theirs.

Parameters
[in]countThe number of iterations.
[in]work_per_indexThe work of one iteration, in multiply-adds.
[in]bodyThe per-index work (void(std::size_t)).
[in]sideThe caller's task (void()).

Definition at line 827 of file parallel.hpp.

◆ parallel_map() [1/2]

template<typename T , typename F >
auto numerical::detail::parallel_map ( const T *  data,
const std::vector< std::size_t > &  shape,
const F &  f 
) -> tensor<std::decay_t<decltype(f(std::declval<const T&>()))>>

f(x) for every one of the row-major elements at data, of shape shape (another array's storage, read where it lies).

Definition at line 2033 of file tensor_ops.hpp.

◆ parallel_map() [2/2]

template<typename T , typename F >
auto numerical::detail::parallel_map ( const tensor< T > &  t,
const F &  f 
) -> tensor<std::decay_t<decltype(f(std::declval<const T&>()))>>

f(x) for every element of t, by chunks of elementwise_chunk among the threads (as broadcast_apply's same-shape pairs): for a stateless f, the library's element-wise functions.

Definition at line 2055 of file tensor_ops.hpp.

◆ parallel_pairwise_sum()

template<typename T >
T numerical::detail::parallel_pairwise_sum ( const T *  first,
std::size_t  count 
)

pairwise_sum of count contiguous values on the threads, to the bit: the tree's subtrees of at most pairwise_leaf values summed in parallel, each as pairwise_sum sums it, then added in the tree's order. The subtrees follow from the count alone. Four subtrees or fewer: on the caller (2¹⁶ doubles, ten AVX-512 cores: 11.7 µs alone, 14.5 shared; 2²⁰ 247 → 44 µs, 2²⁴ 12.5 → 2.2 ms).

Definition at line 1105 of file tensor.hpp.

◆ parallel_sort_run()

void numerical::detail::parallel_sort_run ( T *  data,
std::size_t  count,
std::size_t  pieces,
const SortRun &  sort_piece,
const Less &  less 
)

Sorts the count elements at data stably in pieces, a power of four, among the threads: each piece sorted apart by sort_piece, then the pieces merged in pairs by less (the order sort_piece sorts by), round after round, between data and a scratch buffer (std::merge takes the first piece's element of two equal ones, so the merge keeps their order). The rounds are even in number, so the last writes data.

Definition at line 3847 of file tensor_ops.hpp.

◆ parallel_threads()

std::size_t numerical::detail::parallel_threads ( std::size_t  count,
std::size_t  work_per_index 
)
inline

The threads parallel_for shares count indices of work_per_index out to (threads_for): the hardware's, at most the cap numerical::set_max_threads set. Fewer than two runs them on the caller.

Definition at line 766 of file parallel.hpp.

◆ parallel_zero()

template<typename T >
void numerical::detail::parallel_zero ( T *  out,
std::size_t  count 
)

Sets the count elements at out to zero, split across the threads the work pays for: a fresh result's pages are first touched, and faulted in, by several threads, where a sequential fill pays every fault on one (a product on eight arm64 threads: 16 % faster at order 512, 7 % at 1024). The zero is a constant in each chunk's fill, which the compiler turns into the C library's zeroing (on arm64 it clears whole cache lines without reading them; a value passed in, stored element by element, was 4 % slower at order 1024 on one thread).

Definition at line 891 of file parallel.hpp.

◆ parlett_estimate()

double numerical::detail::parlett_estimate ( const matrix< std::complex< double > > &  t,
double  smallest 
)
inline

SciPy's estimate for Parlett's recurrence on t: min(1, max(ε, ε ‖strict upper T‖₁ / δ)), δ smallest (ε where 0).

Definition at line 1734 of file matrix_functions.hpp.

◆ parlett_function()

std::pair< funm_result, double > numerical::detail::parlett_function ( const matrix< std::complex< double > > &  t,
const matrix< std::complex< double > > &  z,
const std::vector< std::complex< double > > &  values 
)
inline

funm_from_schur with the recurrence's own estimate beside (recurrence_error, without SciPy's floor, which reads 1 wherever T(0, 0) is 0): what funm weighs its attempts by.

Definition at line 1795 of file matrix_functions.hpp.

◆ part_begin()

std::size_t numerical::detail::part_begin ( std::size_t  count,
std::size_t  parts,
std::size_t  p 
)
inlinenoexcept

Where part p of parts sharing count elements begins: the first count % parts one longer.

Definition at line 3559 of file tensor_ops.hpp.

◆ part_size() [1/2]

template<std::floating_point U>
U numerical::detail::part_size ( const std::complex< U > &  x)

The larger magnitude of a complex x's parts: no sum of them, which overflows near the largest double.

Definition at line 414 of file factorizations.hpp.

◆ part_size() [2/2]

template<std::floating_point R>
R numerical::detail::part_size ( R  x)

The magnitude of a real x, x's exponent read from it.

Definition at line 407 of file factorizations.hpp.

◆ partition_positions()

std::vector< std::size_t > numerical::detail::partition_positions ( std::vector< std::size_t >  kths,
std::size_t  extent 
)
inline

kths ascending and each once, every one checked against extent.

Exceptions
index_errorIf a position is not below extent.

Definition at line 4032 of file tensor_ops.hpp.

◆ parts_finite()

template<split_complex C>
bool numerical::detail::parts_finite ( const C *  x,
std::size_t  count 
)
noexcept

Whether every part of count complex numbers from x is finite.

Definition at line 1577 of file tensor.hpp.

◆ pencil_eigenvalues()

pencil_pair numerical::detail::pencil_eigenvalues ( double  a11,
double  a12,
double  a21,
double  a22,
double  b11,
double  b12,
double  b22,
double  safmin 
)
inline

The eigenvalues of the pencil A − w B, A (a11 a12 / a21 a22) and the upper triangular B (b11 b12 / 0 b22), as LAPACK's dlag2, with scalings that keep scale · w from overflowing or underflowing; a diagonal entry of B below safmin's square root times its largest perturbed up to it.

Definition at line 685 of file generalized.hpp.

◆ pencil_operand_in_range()

template<typename T >
std::pair< matrix< T >, double > numerical::detail::pencil_operand_in_range ( const matrix< T > &  m)

m brought into the range [√safmin / ε, ε / √safmin] of its largest magnitude where it lies outside, as LAPACK's xgges scales A and B before the QZ iterations (their norms' squares, the shifts and the rotations stay in range): the matrix, and the factor that takes the scaled one back (1 where none).

Definition at line 2132 of file generalized.hpp.

◆ pencil_standard_form()

template<typename T >
buffer< T > numerical::detail::pencil_standard_form ( buffer< T >  y,
const T *  lower,
std::size_t  n,
bool  blocks 
)

C = L⁻¹ A L⁻ᴴ whole, y holding A's lower triangle (n × n, row-major, the diagonal real) and lower L: by blocks (reduce_pencil, C's lower triangle, then its upper from it) where blocks, else by two whole triangular solves, L⁻¹ A and L⁻¹ (L⁻¹ A)ᴴ.

Definition at line 6562 of file factorizations.hpp.

◆ pencil_wsize()

double numerical::detail::pencil_wsize ( const pencil_bounds &  b,
double  w 
)
inlinenoexcept

dlag2's WSIZE: the divisor that keeps an eigenvalue of magnitude w in range.

Definition at line 673 of file generalized.hpp.

◆ permuted() [1/2]

template<typename T >
buffer< T > numerical::detail::permuted ( const buffer< T > &  data,
const std::string &  from,
const std::string &  to,
const std::map< char, std::size_t > &  sizes 
)

permuted of a buffer's elements.

Definition at line 1831 of file tensor.hpp.

◆ permuted() [2/2]

template<typename T >
buffer< T > numerical::detail::permuted ( const T *  data,
std::size_t  count,
const std::string &  from,
const std::string &  to,
const std::map< char, std::size_t > &  sizes,
std::size_t  ld = 0 
)

The elements of data, whose axes carry the labels from (extents in sizes), laid out row-major along the labels to, a permutation of from that differs from it (so at least one label): a transpose of the last two labels by tiles, each slice of the leading ones; any other row by row of to's last label, a strided sweep. The threads share the work. data holds count elements, the product of the extents; a matrix (two labels) may have its rows ld elements apart (a NumPy array of every other row), 0 for contiguous ones.

Definition at line 1764 of file tensor.hpp.

◆ phase_of()

template<typename T >
T numerical::detail::phase_of ( const T &  x)

The unit-modulus factor of x: x / |x| (its sign, for a real scalar), and 1 at 0. A subnormal x is scaled up first, exactly: its modulus would round to the coarse subnormal grid, and x / |x| miss unit modulus by as much.

Definition at line 464 of file factorizations.hpp.

◆ physical_cores()

std::size_t numerical::detail::physical_cores ( )
inlinenoexcept

The physical cores this process may run on, or 0 where the system does not say. Linux: core_sets; macOS: hw.physicalcpu.

Definition at line 185 of file parallel.hpp.

◆ pivot_divisor() [1/2]

template<typename R >
complex_divisor< R > numerical::detail::pivot_divisor ( const std::complex< R > &  pivot)

pivot_divisor of a complex pivot: its reciprocal, as zgetf2 takes one at least the smallest normal number in modulus (sfmin), and one under it divided by, whose reciprocal would overflow. (A zero pivot never comes here: the factorisation leaves its column, and the solves refuse a singular factor.)

Definition at line 82 of file factorizations.hpp.

◆ pivot_divisor() [2/2]

template<typename T >
T numerical::detail::pivot_divisor ( const T &  pivot)

The divisor over_pivot takes for the pivot pivot. A complex pivot's reciprocal, once, as LAPACK's zgetf2 takes it (a complex division calls the C library's robust one, a quarter of a complex LU of order 48 when it ran for every element); a real pivot itself, each element divided.

Definition at line 63 of file factorizations.hpp.

◆ pivot_magnitude() [1/2]

template<typename U >
U numerical::detail::pivot_magnitude ( const std::complex< U > &  x)

The magnitude partial pivoting compares of a complex x: |Re x| + |Im x|, LAPACK's (izamax, cabs1), so the same pivots as its zgetrf, with no square root.

Definition at line 342 of file factorizations.hpp.

◆ pivot_magnitude() [2/2]

template<typename T >
magnitude_t< T > numerical::detail::pivot_magnitude ( const T &  x)

The magnitude partial pivoting compares: |x| of a real x.

Definition at line 334 of file factorizations.hpp.

◆ pivot_size() [1/2]

double numerical::detail::pivot_size ( const std::complex< double > &  z)
inline

LAPACK's measure of a complex entry, CABS1: |Re z| + |Im z|.

Definition at line 32 of file ldl.hpp.

◆ pivot_size() [2/2]

double numerical::detail::pivot_size ( double  x)
inline

LAPACK's measure of a real entry, |x|.

Definition at line 26 of file ldl.hpp.

◆ place_by_octant()

std::pair< double_double, double_double > numerical::detail::place_by_octant ( std::size_t  octant,
double_double  c,
double_double  s 
)
inlinenoexcept

e^{−2πi k/n} from the cosine c and sine s of its angle within the octant octant (reduce_to_octant): as double-double parts, real then imaginary.

Definition at line 193 of file fft.hpp.

◆ plain_loop_pays()

bool numerical::detail::plain_loop_pays ( std::size_t  m,
std::size_t  n,
std::size_t  k 
)
inlinenoexcept

Whether a double product of m rows, n columns and depth k, past gemm_rows's threshold, still runs faster in the plain loop than packed on the path in use: packing B's slivers for so few rows of A, or so short a depth, did not pay. One core, the plain loop's time over the packed product's: the NEON tile at a depth of 16 or less whatever the shape (0.5 to 0.6 up to 2000 × 2000), at 64 or less for 64 rows or fewer (0.4 to 0.9; the same under GCC 14 and on eight threads); the AVX2 tile for 16 rows or fewer to a depth of 64, or 32 rows to 32 (0.2 to 0.8, six cores too), as wide as plain_loop_least_width. The AMX's packing pays from 16 rows; a product below that (amx_outruns_plain_loop) runs the NEON tile's rule, where it went packed on that tile (one M1 Pro core, 8 to 15 rows to a depth of 8 to 64: 0.34 to 0.74 of the time).

Definition at line 4372 of file gemm.hpp.

◆ plain_loop_suits()

template<typename T >
bool numerical::detail::plain_loop_suits ( std::size_t  ,
std::size_t  ,
std::size_t   
)
noexcept

plain_loop_pays for a product of T: double's and float's, those it was measured for.

Definition at line 4397 of file gemm.hpp.

◆ plain_loop_suits< double >()

template<>
bool numerical::detail::plain_loop_suits< double > ( std::size_t  m,
std::size_t  n,
std::size_t  k 
)
inlinenoexcept

plain_loop_pays for double.

Definition at line 4406 of file gemm.hpp.

◆ plain_loop_suits< float >()

template<>
bool numerical::detail::plain_loop_suits< float > ( std::size_t  m,
std::size_t  n,
std::size_t  k 
)
inlinenoexcept

plain_loop_pays for float: double's rule, where the plain loop outran the packed tiles as for double (random products of its region: 140 on one M1 Pro core, 0.51 of the time, 0.52 on eight threads; 89 on eight AVX2 cores, 0.75 and 0.68 under GCC, 0.88 and 0.80 under Clang, one thread then eight). The same bits.

Definition at line 4420 of file gemm.hpp.

◆ plain_or_at_scale()

template<typename T >
double numerical::detail::plain_or_at_scale ( double  plain,
const T *  x,
std::size_t  count 
)

√plain, the plain sum of the squares of the count elements at x, where it neither overflowed nor underflowed (between plain_squares_low and plain_squares_high); else norm_at_scale.

Definition at line 523 of file factorizations.hpp.

◆ plain_schur() [1/2]

void numerical::detail::plain_schur ( const schur_target< double > &  t,
std::complex< double > *  w 
)
inline

The Schur form of t's active block by real_schur, its eigenvalues to w (one per row of t).

Definition at line 2128 of file eigen.hpp.

◆ plain_schur() [2/2]

void numerical::detail::plain_schur ( const schur_target< std::complex< double > > &  t,
std::complex< double > *  w 
)
inline

The same by complex_schur.

Definition at line 2140 of file eigen.hpp.

◆ plan_bidiagonal()

std::size_t numerical::detail::plan_bidiagonal ( std::vector< bidiagonal_node > &  nodes,
const double *  d,
const double *  e,
std::size_t  n,
std::size_t  extra,
std::size_t  depth 
)
inline

Appends the block (d, e, n, extra) and everything below it to nodes, parents before children, each scaled to work near unit size by a power of two (exactly): the secular equation works on squares, which overflow past 1e154.

Returns
Its index in nodes.

Definition at line 1243 of file svd.hpp.

◆ plan_block_exchange()

std::optional< block_exchange > numerical::detail::plan_block_exchange ( small_square  s,
small_square  t,
std::size_t  n1,
std::size_t  n2,
double  thresha 
)
inline

The exchange of blocks of orders n1 and n2 (one of them 2), as LAPACK's dtgex2 makes it: orthogonal bases of the deflating subspaces the generalized Sylvester equations give (generalized_sylvester), T made triangular again by an RQ or a QR factorisation (the one that leaves the smaller block below S's diagonal), each 2 × 2 block standardized (standardize_pencil). Nothing where the equations have no solution or that block passes thresha (the weak test).

Definition at line 1814 of file generalized.hpp.

◆ plan_divide()

void numerical::detail::plan_divide ( std::vector< divide_node > &  nodes,
double *  d,
const double *  e,
std::size_t  n,
double *  q,
std::size_t  ldq,
std::size_t  depth 
)
inline

Appends the block (d, e, n) and, before it is solved, everything below it to nodes, parents before children: each scaled to work near unit size by a power of two (exactly), as the deflation tolerance is relative to the weights, which have unit norm, and so to the tridiagonal only at that size; each half has its own. A block past divide_leaf is split into two halves plus a rank-one coupling, as LAPACK's dlaed0: T = diag(T₁ − ρ e_a e_aᵀ, T₂ − ρ e_b e_bᵀ) + ρ u uᵀ, u = e_a + sign(β) e_b, ρ = |β|, β the coupling.

Definition at line 6015 of file factorizations.hpp.

◆ plan_single_exchange()

std::optional< block_exchange > numerical::detail::plan_single_exchange ( small_square  s,
small_square  t,
double  thresha,
double  threshb 
)
inline

Two 1 × 1 blocks' exchange, as LAPACK's dtgex2 makes it: columns turned onto the second eigenvalue's right eigenvector, then rows back to triangular, the rotation from S's new first column where |S₂₂ T₁₁| ≥ |S₁₁ T₂₂|, else T's. Nothing where the new subdiagonal entries pass thresha and threshb (the weak test).

Definition at line 1776 of file generalized.hpp.

◆ plan_taylor()

taylor_plan numerical::detail::plan_taylor ( double  norm,
std::size_t  columns,
const std::function< double(std::size_t)> &  power_norm 
)
inline

Al-Mohy and Higham's code fragment 3.1 (2011), as SciPy's _fragment_3_1 (m_max 55, ℓ 2): the m and s of least cost m s for t A of 1-norm norm and columns columns of B. Where condition 3.13 holds (‖tA‖₁ small for the columns), s = ⌈‖tA‖₁ / θ_m⌉; else s = ⌈α_p / θ_m⌉ over the p from 2 to 8 and the m from p (p − 1) − 1, α_p = max(d_p, d_{p+1}), d_p power_norm(p) (an estimate of ‖(tA)ᵖ‖₁^{1/p}), at least one step.

Definition at line 541 of file matrix_functions.hpp.

◆ plane_rotation_to_zero()

plane_rotation_real numerical::detail::plane_rotation_to_zero ( double  f,
double  g 
)
inlinenoexcept

The rotation that takes (f, g) to (r, 0), as LAPACK's dlartg: r of f's sign, c 0 or more; none for g 0, a quarter turn for f 0. Its length by rotation_length.

Definition at line 5140 of file factorizations.hpp.

◆ pointwise_square_root()

template<typename T >
buffer< T > numerical::detail::pointwise_square_root ( const buffer< T > &  t,
std::size_t  n 
)

The principal square root of the upper triangular n × n row-major t (no real eigenvalue below 0 where T is real), entry by entry: √t_ii on the diagonal, the rest by square_root_columns.

Definition at line 1310 of file matrix_functions.hpp.

◆ pool_threads()

constexpr std::size_t numerical::detail::pool_threads ( std::size_t  logical,
std::size_t  physical,
std::size_t  quota = 0 
)
constexprnoexcept

The pool's threads from the logical hardware threads, the physical cores and the processors a CPU quota allows (each 0: unknown, or none): the cores where known, no more than the quota, at least 1.

Definition at line 201 of file parallel.hpp.

◆ positive_back_substitution() [1/2]

void numerical::detail::positive_back_substitution ( const buffer< double > &  d,
const buffer< double > &  e,
buffer< double > &  b,
std::size_t  n,
std::size_t  k 
)
inline

dptts2's last two passes, after L: D and Lᵀ in one, b ← b / d − b_next e from the end.

Definition at line 327 of file banded.hpp.

◆ positive_back_substitution() [2/2]

void numerical::detail::positive_back_substitution ( const buffer< double > &  d,
const buffer< std::complex< double > > &  e,
buffer< std::complex< double > > &  b,
std::size_t  n,
std::size_t  k 
)
inline

zptts2's (lower): D, then Lᴴ, b ← b − b_next conj(e) from the end.

Definition at line 344 of file banded.hpp.

◆ positive_pivot_step() [1/2]

double numerical::detail::positive_pivot_step ( double  d,
double &  e,
double  next 
)
inline

pttrf's step for a real tridiagonal: e ← e / d,

Returns
next − e·e_old, the next pivot.

Definition at line 304 of file banded.hpp.

◆ positive_pivot_step() [2/2]

double numerical::detail::positive_pivot_step ( double  d,
std::complex< double > &  e,
double  next 
)
inline

zpttrf's step for a complex one: e ← e / d by parts,

Returns
next − f·Re e − g·Im e.

Definition at line 314 of file banded.hpp.

◆ positive_tridiagonal_solve()

template<typename T >
void numerical::detail::positive_tridiagonal_solve ( buffer< double > &  d,
buffer< T > &  e,
buffer< T > &  b,
std::size_t  n,
std::size_t  k 
)

The k right-hand sides of b (n × k, row-major) solved in place against the Hermitian positive definite tridiagonal matrix of real diagonal d and subdiagonal e (n and n − 1 entries, overwritten), as LAPACK's ptsv, which scipy.linalg.solveh_banded calls for one off-diagonal: A = L D Lᴴ (pttrf, e becoming L's subdiagonal), then L, D and Lᴴ in turn (ptts2).

Exceptions
not_positive_definite_errorIf a pivot of D is not positive.

Definition at line 370 of file banded.hpp.

◆ power_of_two_scale()

double numerical::detail::power_of_two_scale ( double  largest)
inlinenoexcept

The power of two a decomposition divides a matrix by whose largest magnitude is largest: its leading bit's, the largest entry then in [1, 2); 1 where largest is 0. Dividing and multiplying back by it are exact, where a division by the largest itself rounded every entry.

Definition at line 240 of file factorizations.hpp.

◆ power_pade()

buffer< std::complex< double > > numerical::detail::power_pade ( const buffer< std::complex< double > > &  x,
std::size_t  n,
double  t,
std::size_t  m 
)
inlineprivate

The [m/m] Padé approximant of (I − X)ᵗ for the upper triangular x, as SciPy's _fractional_power_pade: its continued fraction from the bottom, Y = c₂ₘ X, then Y = (I + Y)⁻¹ cⱼ X for j = 2m − 1, …, 1, and I + Y.

Definition at line 3094 of file matrix_functions.hpp.

◆ power_pade_constant()

double numerical::detail::power_pade_constant ( std::size_t  i,
double  t 
)
inlineprivate

The coefficient cᵢ (i ≥ 1) of the continued fraction of (1 − x)ᵗ, as SciPy's _fractional_power_pade_constant (Higham and Lin, 2011, equation 4.1).

Definition at line 3075 of file matrix_functions.hpp.

◆ prefetch_next_row() [1/2]

template<typename Kernel , typename T >
void numerical::detail::prefetch_next_row ( const T *  ,
std::size_t   
)
noexcept

Nothing, for a tile that prefetches no C (prefetches_next_row_of_c).

Definition at line 2477 of file gemm.hpp.

◆ prefetch_next_row() [2/2]

void numerical::detail::prefetch_next_row ( const T *  c,
std::size_t  ldc 
)
noexcept

The C tile at c prefetched by Kernel's prefetch_rows.

Definition at line 2486 of file gemm.hpp.

◆ principal_branch() [1/2]

bool numerical::detail::principal_branch ( const buffer< double > &  ,
std::size_t   
)
inline

log has its principal branch at every diagonal entry of a real triangular form logm keeps real: none below 0, and its zeros taken as 10⁻²⁰ (force_nonsingular).

Definition at line 2859 of file matrix_functions.hpp.

◆ principal_branch() [2/2]

bool numerical::detail::principal_branch ( const buffer< std::complex< double > > &  t,
std::size_t  n 
)
inline

Whether log has its principal branch at every diagonal entry of t (none real and at most 0).

Definition at line 2866 of file matrix_functions.hpp.

◆ principal_power()

std::complex< double > numerical::detail::principal_power ( const std::complex< double > &  z,
double  p 
)
inlineprivate

z to the real power p, principal: exp(p log z) (0 for z = 0 and p > 0).

Definition at line 3067 of file matrix_functions.hpp.

◆ qd_flip()

void numerical::detail::qd_flip ( qd_index  i0,
qd_index  n0,
const qd_array &  z 
)
inlinenoexcept

Turns the block i0…n0 of the array end for end, both halves (LAPACK's flip in dlasq2 and dlasq3).

Definition at line 517 of file dqds.hpp.

◆ qd_pair()

void numerical::detail::qd_pair ( double &  a,
double  b,
double &  c 
)
inlinenoexcept

The eigenvalues of the 2 × 2 [[a, √b], [√b, c]]-like block of a qd array, a = q₁, b = e₁, c = q₂: the larger into a, the smaller into c, each to high relative accuracy. No change where b is negligible against c, nor where half the sum (a − c) + b underflows to zero.

Definition at line 71 of file dqds.hpp.

◆ quadratic_step()

double numerical::detail::quadratic_step ( double  a,
double  b,
double  c 
)
inline

The smaller root η of C η² − A η + B = 0, in the form that does not cancel: a step of solve_secular's fit (shared by its instantiations, whose data reach one form or the other).

Parameters
[in]aA.
[in]bB.
[in]cC.
Returns
η.

Definition at line 5588 of file factorizations.hpp.

◆ quota_limit()

std::size_t numerical::detail::quota_limit ( )
inlinenoexcept

The processors the CPU quotas over this process allow (cgroup_processors on Linux), 0 for none.

Definition at line 289 of file parallel.hpp.

◆ quota_processors()

constexpr std::size_t numerical::detail::quota_processors ( long long  quota,
long long  period 
)
constexprnoexcept

The processors a CPU quota of quota microseconds a period allows, rounded up (cgroup v1's cpu.cfs_quota_us, -1 for none, and cpu.cfs_period_us): 0 for none.

Definition at line 212 of file parallel.hpp.

◆ qz_iteration()

std::vector< std::pair< std::complex< double >, std::complex< double > > > numerical::detail::qz_iteration ( qz_form &  f,
std::size_t  steps 
)
inline

The Hessenberg-triangular f to generalized Schur form, as LAPACK's zhgeqz (Schur form, Q and Z updated): single-shift QZ sweeps on the active block, the shift the eigenvalue of the trailing 2 × 2 of S T⁻¹ nearer its corner (an exceptional one every tenth iteration), a negligible subdiagonal entry of S splitting the block, a negligible diagonal entry of T chased down (or up, after two small subdiagonal entries) to split off an infinite eigenvalue.

Returns
the eigenvalues as (α, β) pairs, in order. steps bounds the iterations (zhgeqz's 30 n).
Exceptions
convergence_errorIf the iterations did not converge within steps (zhgeqz's info).

Definition at line 189 of file generalized.hpp.

◆ radix_eight_pass_body()

template<bool Fused, typename C >
void numerical::detail::radix_eight_pass_body ( std::size_t  current,
std::size_t  stride,
const C *  in,
C *  out,
const fft_root_t< C > *  twiddles,
fft_real_t< C >  half_root_two 
)
inlinenoexcept

One Stockham pass of radix 8, as radix_four_pass is of 4: a_j = x_j + x_{j+4}, b_j = x_j − x_{j+4} (j below 4), b_j then times e^{−2πi j/8} (√½·(re + im, im − re), (im, −re), √½·(im − re, −(re + im)): each sum then its product, no fused operation, so the same bits on every path); the even outputs a's four_point, the odd ones b's; output k then times w^{kp} as Fused, none for p = 0. half_root_two is √½.

Definition at line 1492 of file fft.hpp.

◆ radix_five_pass_body()

template<bool Fused, typename C >
void numerical::detail::radix_five_pass_body ( std::size_t  current,
std::size_t  stride,
const C *  in,
C *  out,
const fft_root_t< C > *  twiddles,
const small_radix_constants< fft_real_t< C > > &  k 
)
inlinenoexcept

One Stockham pass of radix 5: t1 = x1 + x4, t2 = x2 + x3, t3 = x1 − x4, t4 = x2 − x3; a1 = x0 + c1·t1 + c2·t2, a2 = x0 + c2·t1 + c1·t2, b1 = s1·t3 + s2·t4, b2 = s2·t3 − s1·t4 (each sum in that order, its products fused as Fused); X0 = x0 + t1 + t2, X1 = a1 − i·b1, X4 = a1 + i·b1, X2 = a2 − i·b2, X3 = a2 + i·b2, then times w^{kp}. Inlined into each caller.

Definition at line 1445 of file fft.hpp.

◆ radix_four_pass()

template<bool Fused, std::floating_point R>
void numerical::detail::radix_four_pass ( std::size_t  current,
std::size_t  stride,
const std::complex< R > *  in,
std::complex< R > *  out,
const std::complex< R > *  twiddles 
)
noexcept

radix_four_pass_body of complex numbers, its products as Fused.

Definition at line 1200 of file fft.hpp.

◆ radix_four_pass_body()

template<bool Fused, typename C >
void numerical::detail::radix_four_pass_body ( std::size_t  current,
std::size_t  stride,
const C *  in,
C *  out,
const fft_root_t< C > *  twiddles 
)
inlinenoexcept

One Stockham pass of radix 4 over in into out: current points at stride stride, so current · stride is the length. Element q + stride · (p + j · current/4) of the input is the j-th input of butterfly (p, q), whose outputs go to q + stride · (4p + j), the last three times w^{jp} (twiddled), none for p = 0.

Definition at line 1164 of file fft.hpp.

◆ radix_passes()

template<typename U >
bool numerical::detail::radix_passes ( U *  keys,
U *  key_spare,
std::int64_t *  indices,
std::int64_t *  index_spare,
std::size_t  count,
std::vector< byte_counts< U > > &  counts 
)

Sorts the count keys at keys stably by their bytes from the least significant, the indices at indices with them unless null, in counts.size() parts among the threads. Each pass places a part's keys of one byte value after the earlier parts' and moves each part's in its order, so equal keys keep their order whatever the parts: the bits of one counting sort. counts holds each part's counts of every byte of the keys as they lie, counted again for a part once a pass has moved them (one part's are the whole's, whatever their order); a byte that every key shares is skipped.

Returns
Whether the sorted keys and indices lie in key_spare and index_spare.

Definition at line 3576 of file tensor_ops.hpp.

◆ radix_sixteen_pass()

template<bool Fused, std::floating_point R>
void numerical::detail::radix_sixteen_pass ( std::size_t  current,
std::size_t  stride,
const std::complex< R > *  in,
std::complex< R > *  out,
const std::complex< R > *  twiddles,
const std::array< std::complex< R >, 16 > &  sixteenths 
)
noexcept

radix_sixteen_pass_body of complex numbers, its products as Fused.

Definition at line 1278 of file fft.hpp.

◆ radix_sixteen_pass_body()

template<bool Fused, typename C >
void numerical::detail::radix_sixteen_pass_body ( std::size_t  current,
std::size_t  stride,
const C *  in,
C *  out,
const fft_root_t< C > *  twiddles,
const std::array< fft_root_t< C >, 16 > &  sixteenths 
)
inlinenoexcept

One Stockham pass of radix 16 over in into out, as radix_four_pass is of radix 4: each butterfly a 16-point transform as two of 4 points, x_{j1 + 4·j2} first transformed over j2, each result (j1, k2) then times e^{−2πi j1·k2/16} (sixteenths; −i exactly where j1·k2 is 4), then over j1 into output 4·k1 + k2; output k of butterfly p times w^{kp}, none for p = 0. Fused as twiddled.

Definition at line 1236 of file fft.hpp.

◆ radix_sort()

template<typename T >
void numerical::detail::radix_sort ( T *  first,
T *  last,
std::size_t  parts = 1 
)

Sorts [first, last) by order_key, a byte at a time from the least significant (by radix_passes, a stable counting sort each, so equal keys keep their order; a byte that every key shares skipped), in parts among the threads.

Definition at line 3648 of file tensor_ops.hpp.

◆ radix_three_pass_body()

template<bool Fused, typename C >
void numerical::detail::radix_three_pass_body ( std::size_t  current,
std::size_t  stride,
const C *  in,
C *  out,
const fft_root_t< C > *  twiddles,
const small_radix_constants< fft_real_t< C > > &  k 
)
inlinenoexcept

One Stockham pass of radix 3, as any_radix_pass_body is of any: t = x1 + x2, v = x0 − t/2 (exact halving, one rounding), u = (√3/2)·(x1 − x2); X0 = x0 + t, X1 = v − i·u, X2 = v + i·u, then times w^{kp} as Fused. Inlined into each caller.

Definition at line 1406 of file fft.hpp.

◆ radix_two_pass()

template<typename C >
void numerical::detail::radix_two_pass ( std::size_t  stride,
const C *  in,
C *  out 
)
inlinenoexcept

The last pass of an odd power of two: butterflies of radix 2 at stride stride, no factor.

Definition at line 1647 of file fft.hpp.

◆ range_within()

constexpr bool numerical::detail::range_within ( std::size_t  first,
std::size_t  count,
std::ptrdiff_t  step,
std::size_t  extent 
)
constexprnoexcept

Whether count indices from first, step apart, all lie in [0, extent): none is out for a count of 0. In unsigned arithmetic, the last index's distance from first never formed where it would pass the axis, so that no step and no count overflows ((count - 1) · step in std::ptrdiff_t did, a step of PTRDIFF_MIN and a count of 3 among them, and could then pass the test).

Definition at line 115 of file vector.hpp.

◆ rank_tolerance()

double numerical::detail::rank_tolerance ( double  largest,
std::size_t  rows,
std::size_t  cols 
)
inlineprivate

The magnitude under which a diagonal entry of R or a singular value counts as zero, as NumPy draws it: the largest times the machine epsilon times the larger dimension.

Definition at line 3703 of file factorizations.hpp.

◆ read_in_place()

template<typename T >
packed_left< T > numerical::detail::read_in_place ( std::size_t  m,
std::size_t  k,
const T *  a,
std::size_t  lda 
)
noexcept

The m × k a read in place by gemm_with, unpacked: each product packs it as gemm does.

Definition at line 4844 of file gemm.hpp.

◆ real_forward()

template<std::floating_point R>
void numerical::detail::real_forward ( const real_fft_plan< R > &  plan,
const R *  x,
std::complex< R > *  X,
std::complex< R > *  work,
bool  fused 
)

real_forward_body on the path in use.

Definition at line 2728 of file fft.hpp.

◆ real_forward_body()

template<bool Fused, std::floating_point R, typename Registers >
void numerical::detail::real_forward_body ( const real_fft_plan< R > &  plan,
const R *  x,
std::complex< R > *  X,
std::complex< R > *  work,
Registers  registers 
)
inline

The transform of n real points x (together) into X (n/2 + 1 complex numbers), work holding n/2 + 1 complex numbers beside the plan's work. An even n = 2h: the h complex numbers x_{2j} + i·x_{2j+1} transformed, then X_k = E_k + w^k·O_k, E_k = (Z_k + conj Z_{h−k})/2, O_k = −i·(Z_k − conj Z_{h−k})/2 (exact halvings and products by ±i; w^k·O_k as Fused), registers (h, Z, roots, X) computing the first of them and returning the next k. An odd n: the complex transform of the points, its first n/2 + 1 outputs. Inlined into each caller.

Definition at line 2594 of file fft.hpp.

◆ real_forward_whole()

template<std::floating_point R>
void numerical::detail::real_forward_whole ( std::size_t  n,
const R *  x,
std::complex< R > *  X,
bool  fused 
)

The first n/2 + 1 frequencies of the n real points x into X by the complex transform of n points, the products fused where fused: an even n's line holding an infinity or a NaN, which the recombination of its pairs would turn into NaN (∞ − ∞) where the complex transform carries it.

Definition at line 2540 of file fft.hpp.

◆ real_hessenberg_triangular()

real_qz_form numerical::detail::real_hessenberg_triangular ( const matrix< double > &  a,
const matrix< double > &  b 
)
inline

The real a and b to Hessenberg-triangular form, as LAPACK's dgges begins: B = Q R by the QR factorisation, A ← Qᵀ A, then dgghrd: each column of A from the first cleared below its subdiagonal from the bottom by rotations of rows, the fill each leaves below B's diagonal taken out by a rotation of columns; Q and Z accumulating them.

Definition at line 1098 of file generalized.hpp.

◆ real_input_transform()

template<std::floating_point R>
tensor< std::complex< R > > numerical::detail::real_input_transform ( const fft_source< R > &  a,
std::size_t  index,
std::size_t  points,
bool  inverse,
R  scale 
)

The transform of real a along the axis index into all n frequencies, or the inverse transform where inverse: each line's real transform (real_forward, its n/2 + 1 non-negative frequencies), the rest by Hermitian symmetry, X_{n−k} = conj X_k, exactly; a length whose complex transform takes four steps, which half of it does not, by that complex transform, on the threads. The inverse of a real line the conjugate of its transform. Then times scale.

Definition at line 2838 of file fft.hpp.

◆ real_inverse()

template<std::floating_point R>
void numerical::detail::real_inverse ( const real_fft_plan< R > &  plan,
const std::complex< R > *  X,
R *  x,
std::complex< R > *  work,
R  scale,
bool  fused 
)

real_inverse_body on the path in use.

Definition at line 2747 of file fft.hpp.

◆ real_inverse_body()

template<bool Fused, std::floating_point R, typename Registers >
void numerical::detail::real_inverse_body ( const real_fft_plan< R > &  plan,
const std::complex< R > *  X,
R *  x,
std::complex< R > *  work,
R  scale,
Registers  registers 
)
inline

The inverse of real_forward_body, unscaled save its own halvings: n real points x from the n/2 + 1 complex numbers X (the imaginary parts of X_0 and, for an even n, X_{n/2} read as 0, as NumPy's). An even n = 2h: Z_k = E_k + i·O_k, E_k = (X_k + conj X_{h−k})/2, O_k = conj(w^k)·(X_k − conj X_{h−k})/2, then Z's inverse transform (conj(fft(conj ·))), its parts x_{2j} and x_{2j+1}, times scale, registers (h, X, roots, conj Z) computing the first conj Z_k and returning the next k; an odd n: the Hermitian spectrum of n points, its inverse transform's real parts times scale. Inlined into each caller.

Definition at line 2647 of file fft.hpp.

◆ real_inverse_whole()

template<std::floating_point R>
void numerical::detail::real_inverse_whole ( std::size_t  n,
const std::complex< R > *  X,
R *  x,
R  factor,
bool  fused 
)

The n real points x from the n/2 + 1 frequencies X by the complex transform of n points (conj(fft(conj ·)) of the Hermitian spectrum, the imaginary parts of X_0 and X_{n/2} read as 0), times factor: an even n's spectrum holding an infinity or a NaN (real_forward_whole).

Definition at line 2561 of file fft.hpp.

◆ real_part() [1/2]

template<typename U >
carried_t< U > numerical::detail::real_part ( const std::complex< U > &  x)

The real part of a complex scalar.

Definition at line 369 of file factorizations.hpp.

◆ real_part() [2/2]

template<typename T >
carried_t< T > numerical::detail::real_part ( const T &  x)

The real part of a real (or general) scalar, in carried_t.

Definition at line 362 of file factorizations.hpp.

◆ real_qz_iteration()

std::vector< real_pencil_value > numerical::detail::real_qz_iteration ( real_qz_form &  f,
std::size_t  steps 
)
inline

The Hessenberg-triangular f to generalized real Schur form, as LAPACK's dhgeqz (Schur form, Q and Z updated): double-shift QZ sweeps (Moler and Stewart) on the active block, the shifts the eigenvalues of its trailing 2 × 2 (dlag2; a single shift where they are real, an exceptional one every tenth iteration), a negligible subdiagonal entry of S splitting it, a negligible diagonal entry of T chased to split off an infinite eigenvalue; a 2 × 2 block of a conjugate pair standardized with T diagonal and positive (dlasv2), every T(j, j) non-negative. steps bounds the iterations (dhgeqz's 30 n).

Returns
The eigenvalues, in the diagonal's order.
Exceptions
convergence_errorIf the iterations did not converge within steps (dhgeqz's info).

Definition at line 1150 of file generalized.hpp.

◆ real_schur()

void numerical::detail::real_schur ( const schur_target< double > &  t,
buffer< double > &  re,
buffer< double > &  im,
std::size_t  steps 
)
inline

The real Schur form of the upper Hessenberg t's active block, as LAPACK's dlahqr: Francis's implicit double-shift QR, with the deflation criterion of Ahues and Tisseur and exceptional shifts after 10 and 20 iterations without deflation. The eigenvalues are left in re and im, a complex pair's in two consecutive places, the positive imaginary part first.

Parameters
[in]tThe Hessenberg matrix, its active block, and the Schur vectors to update.
[in,out]reThe eigenvalues' real parts (the isolated ones already in place).
[out]imTheir imaginary parts.
[in]stepsThe iterations allowed between two deflations.
Exceptions
convergence_errorIf an eigenvalue has not converged within steps.

Definition at line 920 of file eigen.hpp.

◆ real_square_root()

std::optional< matrix< double > > numerical::detail::real_square_root ( const matrix< double > &  form,
const matrix< double > *  z 
)
inline

The real principal square root of the real a from its real Schur form (Higham, 1987), or none where a has a real eigenvalue below 0 (its principal root complex): a 1 × 1 block's root √t, a 2 × 2 block [θ b; c θ] of eigenvalues θ ± iμ the root α I + (T − θ I) / 2α for α + iβ = √(θ + iμ).

Definition at line 1266 of file matrix_functions.hpp.

◆ real_subdiagonal() [1/2]

double numerical::detail::real_subdiagonal ( double  t,
double  ,
double &   
)
inline

A real subdiagonal entry of the tridiagonal: kept with its sign (the rotations take any sign).

Parameters
[in]tThe entry.
Returns
t.

Definition at line 4769 of file factorizations.hpp.

◆ real_subdiagonal() [2/2]

double numerical::detail::real_subdiagonal ( std::complex< double >  t,
std::complex< double >  delta,
std::complex< double > &  next 
)
inline

A complex subdiagonal entry made real by the next row's unit phase: next = delta · conj(phase(t)).

Parameters
[in]tThe entry.
[in]deltaThe phase of the row above.
[out]nextThe phase of the row below.
Returns
|t|.

Definition at line 4783 of file factorizations.hpp.

◆ real_transform_of()

template<std::floating_point T>
tensor< std::complex< T > > numerical::detail::real_transform_of ( const fft_source< T > &  a,
const std::optional< std::size_t > &  n,
std::ptrdiff_t  axis,
fft_norm  norm,
bool  inverse 
)

fft or ifft (inverse) of the real elements a: fft of a real tensor's computation, of elements a caller lends.

Definition at line 2880 of file fft.hpp.

◆ real_values()

std::vector< real_pencil_value > numerical::detail::real_values ( real_qz_form &  f)
inline

The eigenvalues of the real generalized Schur form f after a reordering, as LAPACK's dtgsen gives them: a 2 × 2 block's pair by dlag2 (β its scale), a 1 × 1 block's from its diagonal, T(k, k) made non-negative (row k of S and T and column k of Q negated).

Definition at line 2099 of file generalized.hpp.

◆ real_vectors()

real_eigenvectors numerical::detail::real_vectors ( const buffer< double > &  t,
const buffer< double > &  re,
const buffer< double > &  im,
std::size_t  n 
)
inline

The eigenvectors of the real quasi-triangular t (row-major n × n, its 2 × 2 blocks standard) as the rows of Xᵀ, as LAPACK's dtrevc: real for a real eigenvalue, solved in real arithmetic; for a pair re ± i im (im [k] > 0 the first) the first's vector, the second's its conjugate.

Definition at line 2979 of file eigen.hpp.

◆ reciprocal_condition()

template<typename T >
double numerical::detail::reciprocal_condition ( std::size_t  n,
double  one_norm,
const std::function< void(T *, bool)> &  apply 
)

The reciprocal condition number 1 / (‖A‖₁ ‖A⁻¹‖₁) of A from one_norm, ‖A‖₁, and an estimate of ‖A⁻¹‖₁ by one_norm_estimate over apply, as LAPACK's *con routines end: 1 for an empty matrix, 0 for a zero one, and 0 where the estimate overflows.

Definition at line 2232 of file factorizations.hpp.

◆ recommended_shifts()

lapack_index numerical::detail::recommended_shifts ( lapack_index  nh)
inline

The shifts LAPACK's iparmq recommends for an active block of order nh.

Definition at line 2421 of file eigen.hpp.

◆ recommended_window()

lapack_index numerical::detail::recommended_window ( lapack_index  nh,
lapack_index  ns 
)
inline

The deflation window LAPACK's iparmq recommends for an active block of order nh and ns shifts.

Definition at line 2446 of file eigen.hpp.

◆ recurrence_error()

double numerical::detail::recurrence_error ( const matrix< std::complex< double > > &  t,
const std::vector< std::size_t > &  starts,
const buffer< std::complex< double > > &  fx,
buffer< std::complex< double > >  error 
)
inline

The error of f(T), fx, from the block recurrence (solve_upper_blocks) over the runs starts of t's diagonal, relative to its largest entry: the recurrence run again on the error alone, from its diagonal blocks' errors (error, n × n, its diagonal blocks set: circle_error for a cluster, f's rounding for one eigenvalue), each entry above them with its rounding as a source, ε (Σ |F_ik| |T_kj| + |T_ik| |F_kj|) over the products that form it (the moduli as |re| + |im|), in a direction of its own (rounding_direction). Where clusters are far from normal, the separation of their blocks falls far below their eigenvalues' gaps and the recurrence grows the diagonal blocks' errors and its own roundings by its inverse: eight clusters of 32 at order 256 (spread 0.01, T's strict upper triangle 0.3 N(0, 1)), integrals right to 10⁻¹¹ each and f(T) wrong by 10⁻⁴ above them (by 10⁻⁶ from the roundings alone, a recurrence in 30 digits from the same blocks shows), where SciPy's estimate, ε ‖strict upper T‖₁ over the smallest gap, said 7·10⁻¹³; this one says 2·10⁻⁴.

Definition at line 1698 of file matrix_functions.hpp.

◆ reduce_along()

template<typename T , typename Reduce >
tensor< T > numerical::detail::reduce_along ( const tensor< T > &  t,
std::size_t  axis,
const Reduce &  reduce 
)

Reduces each line of t along axis by reduce, which takes the line's first element, its length and its stride.

Definition at line 1492 of file tensor_ops.hpp.

◆ reduce_pencil()

template<typename T >
void numerical::detail::reduce_pencil ( T *  c,
const T *  l,
std::size_t  n 
)

The lower triangle of the Hermitian n × n c (row-major) taken to L⁻¹ C L⁻ᴴ for the lower-triangular l (n × n, row-major), as LAPACK's sygst (type 1, lower) by blocks of pencil_block columns: a block's diagonal by reduce_pencil_columns; the rows below it solved against its L₁₁ᴴ from the right, corrected by half of L₂₁ C₁₁ twice around the trailing rank-2k update C₂₂ −= C₂₁ L₂₁ᴴ + L₂₁ C₂₁ᴴ (its lower triangle); once every block is through, each one's rows solved against its L₂₂, the blocks on the threads together. Some n³ operations where two whole triangular solves took 2 n³. c's other triangle is neither read nor written.

Definition at line 6464 of file factorizations.hpp.

◆ reduce_pencil_columns()

template<typename T >
void numerical::detail::reduce_pencil_columns ( T *  c,
std::size_t  ldc,
const T *  l,
std::size_t  ldl,
std::size_t  n 
)

The lower triangle of the n × n c (row stride ldc) taken to L⁻¹ C L⁻ᴴ for the lower-triangular l (row stride ldl), one column after another, as LAPACK's sygs2 (type 1): each diagonal element divided by L's squared, the column below it scaled, corrected by half of it twice around the rank-2 update of the rest, then solved against the rest of L. c's other triangle is neither read nor written.

Definition at line 6389 of file factorizations.hpp.

◆ reduce_to_octant()

octant_share numerical::detail::reduce_to_octant ( std::size_t  k,
std::size_t  n 
)
inlinenoexcept

The octant_share of e^{−2πi k/n}.

Definition at line 182 of file fft.hpp.

◆ reduced_axes()

std::vector< std::size_t > numerical::detail::reduced_axes ( const std::vector< std::size_t > &  order,
std::size_t  axis 
)
inline

The axes that put a result along axis of a tensor laid out in order (view_axes), that axis taken out, back in the view's order.

Definition at line 1940 of file tensor_ops.hpp.

◆ reduced_sine()

template<bool Cosine>
double numerical::detail::reduced_sine ( double  x)
inlinenoexcept

sin(x), or cos(x) where Cosine, for |x| ≤ sine_reduction_bound (another x gives a value of no use): k the integer nearest x·2/π, by the sum with 1.5·2⁵² whose low bits are k's (its quadrant); r = x − k·π/2 by three multiply-adds of half_pi_parts, |r| ≤ π/4; sin r and cos r by fdlibm's polynomials (within 2⁻⁵⁸ there), the quadrant's taken and signed by masks. A float's value rounded to float is within half its ulp and 2⁻²⁸: the floats below 2²⁰ come no nearer a multiple of π/2 than 2⁻²⁷·⁸ (252.898…, near 161·π/2), where the reduction errs by some 2⁻⁸⁰. ±0 kept. Every multiply-add std::fma, so the bits are the same in vectors.

Definition at line 523 of file elementary.hpp.

◆ reflect_columns()

template<typename T >
void numerical::detail::reflect_columns ( T *  panel,
std::size_t  height,
const buffer< T > &  v,
std::size_t  first,
std::size_t  from,
std::size_t  to,
const T &  s = T {2} 
)
private

Applies I − s v vᴴ from the left (s 2 for a unit v) to the columns from…to-1 of the column-major panel (height rows per column), on its rows first…first+|v|-1: each column a dot product with v, then v scaled out of it, both along the column at unit stride.

Definition at line 3234 of file factorizations.hpp.

◆ reflect_columns_3()

void numerical::detail::reflect_columns_3 ( const std::array< double, 3 > &  u,
double  tau,
double *  t,
std::size_t  ld,
std::size_t  count 
)
inline

The same from the right, to the count × 3 columns at t.

Definition at line 1401 of file eigen.hpp.

◆ reflect_from_left()

template<typename T >
void numerical::detail::reflect_from_left ( T *  x,
std::size_t  stride,
const T &  tau,
const T &  v2,
const T &  v3 
)

The adjoint I − τ̄ v vᴴ of a reflection of three entries from the left on one column (x three entries stride apart).

Definition at line 1903 of file eigen.hpp.

◆ reflect_in_columns()

template<typename T >
std::vector< reflector_block< T > > numerical::detail::reflect_in_columns ( buffer< T > &  columns,
std::size_t  rows,
std::size_t  cols,
bool  reflectors = true 
)
private

The blocks of reflectors of a Householder QR of the column-major columns (rows per column, cols columns), factorised in place as LAPACK's dgeqrf leaves it: R on and above the diagonal, each reflector's w₁… below it. The columns right of a panel take its block as (I − V T Vᴴ)ᴴ, held transposed (apply_transposed_reflector_block): the products then have as many rows as there are columns to update, where the row-major form's first product has the block's 32 (one x86-64 core: 39 ms against 46 for the updates of order 1000). From qr_look_ahead_threads threads, the next panel is updated and factorised by one task while the others update the columns past it, a slice each. Where reflectors is false (R alone asked), a block no update reads is not gathered.

Definition at line 3323 of file factorizations.hpp.

◆ reflect_in_rows()

template<typename T >
std::vector< reflector_block< T > > numerical::detail::reflect_in_rows ( buffer< T > &  a,
std::size_t  rows,
std::size_t  cols,
buffer< T > *  r = nullptr,
bool  reflectors = true 
)
private

The blocks of reflectors of a Householder QR of the row-major a (rows × cols), factorised in place as reflect_in_columns leaves the column-major one (R on and above the diagonal, each reflector's w₁… below it): each panel in a column-major copy, the columns right of it updated by apply_reflector_block, whose projection reads A as the product's right operand. The AMX tile reads that in place, and NEON's packs it faster than the column-major form's left one (arm64, order 512: 6.3 ms against 7.2 under clang, 16.4 against 17.9 under GCC). Where r is given, it is R (min(rows, cols) × cols row-major, zeros below the diagonal).

Definition at line 3446 of file factorizations.hpp.

◆ reflect_rows()

template<typename T >
void numerical::detail::reflect_rows ( buffer< T > &  a,
std::size_t  cols,
const buffer< T > &  v,
std::size_t  first,
std::size_t  from,
const T &  s = T {2} 
)
private

Applies I − s v vᴴ from the left (s 2 for a unit v) to the rows first…first+|v|-1 of a row-major array, on its columns from…cols-1.

Row by row: the projection w = vᴴ A accumulates one contiguous row at a time, then each row takes −s vᵢ w. A column-by-column walk strides by cols on every access and runs several times slower. The columns are independent, so chunks of them run in parallel, each element's arithmetic in the same order.

Parameters
[in,out]aThe array, cols entries per row.
[in]colsIts row length.
[in]vThe reflector (unit where s is 2).
[in]firstThe first row it acts on.
[in]fromThe first column it acts on.
[in]sThe reflector's factor.

Definition at line 3021 of file factorizations.hpp.

◆ reflect_rows_3()

void numerical::detail::reflect_rows_3 ( const std::array< double, 3 > &  u,
double  tau,
double *  t,
std::size_t  ld,
std::size_t  count 
)
inline

H = I − τ u uᵀ of three entries applied from the left to the 3 × count rows at t (row stride ld).

Definition at line 1386 of file eigen.hpp.

◆ reflect_three_columns()

template<typename T >
void numerical::detail::reflect_three_columns ( T *  first,
std::size_t  rows,
std::size_t  ld,
const T &  tau,
const T &  v2,
const T &  v3 
)

A reflection I − τ v vᴴ, v = (1, v₂, v₃), from the right on rows rows (ld apart) of the three columns at first: x ← x − τ (x₁ + v₂ x₂ + v₃ x₃) (1, v̄₂, v̄₃) on each. The rows are short and apart: one call for them all.

Definition at line 862 of file eigen.hpp.

◆ reflect_three_rows()

template<typename T >
void numerical::detail::reflect_three_rows ( T *  r1,
T *  r2,
T *  r3,
std::size_t  from,
std::size_t  to,
const T &  tau,
const T &  v2,
const T &  v3,
bool  adjoint 
)

A reflection I − τ v vᴴ, v = (1, v₂, v₃), on the entries from … to − 1 of three rows: its adjoint from the left with adjoint (rows of H), else from the right (rows of Zᵀ, which are columns of Z). A reflection of two passes r3 = r2 and v3 = 0. One loop over the entries, which vectorises.

Definition at line 812 of file eigen.hpp.

◆ reflect_two_columns()

template<typename T >
void numerical::detail::reflect_two_columns ( T *  first,
std::size_t  rows,
std::size_t  ld,
const T &  tau,
const T &  v2 
)

The same for a reflection of two entries, v = (1, v₂), on two columns.

Definition at line 882 of file eigen.hpp.

◆ reflect_two_rows()

template<typename T >
void numerical::detail::reflect_two_rows ( T *  r1,
T *  r2,
std::size_t  from,
std::size_t  to,
const T &  tau,
const T &  v2,
bool  adjoint 
)

reflect_three_rows for a reflection of two entries, v = (1, v₂), on two rows: its own loop, which vectorises where the third row aliasing the second would not.

Definition at line 838 of file eigen.hpp.

◆ reflector_tau()

template<typename T >
T numerical::detail::reflector_tau ( T &  alpha,
T *  x,
std::size_t  count 
)

The Householder reflection H = I − τ v vᴴ, v = (1, v₁…), with Hᴴ (α, x) = (β, 0) and β real, as LAPACK's zlarfg (dlarfg for a real T): alpha becomes β and the count entries at x become v₁….

Returns
τ, zero where x is zero and α real (the identity).

Definition at line 350 of file eigen.hpp.

◆ refuse_boundary()

void numerical::detail::refuse_boundary ( bool  continuous)
inlineprivate

Refuses a Riccati equation whose pencil has eigenvalues on (or too close to) the boundary.

Exceptions
convergence_errorAlways.

Definition at line 2766 of file generalized.hpp.

◆ release_and_trim()

void numerical::detail::release_and_trim ( )
inline

release_thread_memory, then the allocator's free pages to the system, on the calling thread: glibc keeps an arena for each thread that allocates, whose free pages the thread trims best itself.

Definition at line 27 of file memory.hpp.

◆ release_large_arrow_workspace()

void numerical::detail::release_large_arrow_workspace ( arrow_workspace &  work)
inline

Releases the buffers of work longer than kept_workspace.

Definition at line 955 of file svd.hpp.

◆ release_large_workspace()

void numerical::detail::release_large_workspace ( merge_workspace &  work)
inline

Releases the buffers of work longer than kept_workspace.

Definition at line 5771 of file factorizations.hpp.

◆ release_memory_on()

void numerical::detail::release_memory_on ( thread_pool *  pool)
inline

numerical::release_memory with pool's workers, none where it is nullptr (a pool never started holds nothing, and starting it would cost threads, not free memory).

Definition at line 37 of file memory.hpp.

◆ release_thread_memory()

void numerical::detail::release_thread_memory ( )
inline

Empties every workspace the calling thread keeps between calls: its thread_releases, in turn.

Definition at line 77 of file buffer.hpp.

◆ remainder_power()

template<typename T >
std::optional< matrix< std::complex< double > > > numerical::detail::remainder_power ( const matrix< T > &  a,
double  t,
const std::optional< hermitian_similarity< T > > &  similarity 
)
private

A^t (−1 < t < 1) by the Schur form, as SciPy's _remainder_matrix_power: an upper triangular A directly, else Z T^t Zᴴ; none where the triangular form has a zero on its diagonal. An exactly Hermitian A not triangular by its eigenvectors (hermitian_remainder_power); a diagonal one's diagonal taken real, so that a negative eigenvalue's power is the principal value's.

Definition at line 3297 of file matrix_functions.hpp.

◆ reorder() [1/2]

bool numerical::detail::reorder ( one_based< double >  t,
one_based< double >  qt,
lapack_index  n,
lapack_index  ifst,
lapack_index &  ilst 
)
inline

Moves the diagonal block of the real quasi-triangular t at ifst to ilst by exchanges of adjacent blocks, the rows of qt (Qᵀ) turning with it, as LAPACK's dtrexc. ilst becomes where the block ended: short of its goal where an exchange was refused (then.

Returns
false).

Definition at line 1644 of file eigen.hpp.

◆ reorder() [2/2]

bool numerical::detail::reorder ( one_based< std::complex< double > >  t,
one_based< std::complex< double > >  qt,
lapack_index  n,
lapack_index  ifst,
lapack_index &  ilst 
)
inline

Moves the diagonal entry of the complex upper triangular t at ifst to ilst, as LAPACK's ztrexc.

Returns
true: exchanges of 1 × 1 blocks never fail.

Definition at line 1737 of file eigen.hpp.

◆ require_band()

template<typename T >
void numerical::detail::require_band ( const matrix< T > &  ab,
std::size_t  rows,
const matrix< T > &  b,
const std::string &  name 
)

Refuses a band that is not rows × n and right-hand sides not of n rows.

Definition at line 408 of file banded.hpp.

◆ require_finite() [1/3]

void numerical::detail::require_finite ( const matrix< double > &  a,
const char *  what 
)
inline

require_finite for double, its elements contiguous: all_finite.

Definition at line 678 of file factorizations.hpp.

◆ require_finite() [2/3]

void numerical::detail::require_finite ( const matrix< std::complex< double > > &  a,
const char *  what 
)
inline

The same for std::complex<double>, whose array holds its parts in order ([complex.numbers]).

Definition at line 687 of file factorizations.hpp.

◆ require_finite() [3/3]

template<typename T >
void numerical::detail::require_finite ( const matrix< T > &  a,
const char *  what 
)

The one non-finite policy, checked where every decomposition starts.

Parameters
[in]aThe input.
[in]whatThe decomposition, for the message.
Exceptions
non_finite_errorIf an entry is infinite or NaN.

Definition at line 628 of file factorizations.hpp.

◆ require_finite_lower() [1/3]

void numerical::detail::require_finite_lower ( const matrix< double > &  a,
const char *  what 
)
inline

require_finite_lower for double: all_finite over the whole matrix, the one vectorised pass require_finite takes, and the triangle entry by entry only where it finds a NaN or an infinity. Out of line, so that a decomposition's constructor stays small enough to be inlined: in it, the check kept GCC from inlining the Cholesky's (order 8, 0.204 → 0.220 µs), as an entry-by-entry check (6 % at order 64 on arm64) or a row-by-row one (21 % at order 8 under GCC) cost in time.

Definition at line 732 of file factorizations.hpp.

◆ require_finite_lower() [2/3]

void numerical::detail::require_finite_lower ( const matrix< std::complex< double > > &  a,
const char *  what 
)
inline

The same for std::complex<double>, whose array holds its parts in order ([complex.numbers]).

Definition at line 741 of file factorizations.hpp.

◆ require_finite_lower() [3/3]

template<typename T >
void numerical::detail::require_finite_lower ( const matrix< T > &  a,
const char *  what 
)

require_finite over what a Hermitian eigensolver reads: the lower triangle, and the diagonal's real part (as numpy.linalg.eigh reads them, whatever the rest holds).

Parameters
[in]aThe square input.
[in]whatThe decomposition, for the message.
Exceptions
non_finite_errorIf an entry read is infinite or NaN.

Definition at line 719 of file factorizations.hpp.

◆ require_finite_triangle()

template<typename T >
void numerical::detail::require_finite_triangle ( const T *  a,
std::size_t  n,
bool  lower,
const T *  x,
std::size_t  count 
)

Refuses an infinite or NaN entry of the triangle of a a triangular solve reads, or of its right-hand side x (count elements): one vectorised pass over the whole matrix, then the triangle entry by entry only where that pass finds one (the other triangle may hold anything); the right-hand side in one pass.

Exceptions
non_finite_errorIf an entry read is infinite or NaN.

Definition at line 2023 of file factorizations.hpp.

◆ require_hermitian()

void numerical::detail::require_hermitian ( const matrix< std::complex< double > > &  x,
const char *  name 
)
inline

Refuses x (named name in the message) unless Hermitian, as SciPy's _are_validate_args: ‖X − Xᴴ‖₁ within 100 spacings of ‖X‖₁.

Exceptions
argument_errorIf it is not.

Definition at line 3085 of file generalized.hpp.

◆ rescale_tiny()

template<typename T >
carried_t< T > numerical::detail::rescale_tiny ( buffer< T > &  x)

Scales x by rescue_factor of its largest magnitude, exactly, so that a Householder vector built from it is computed among normal numbers: a subnormal norm rounds to the coarse subnormal grid, and the reflector is then not orthogonal.

Returns
The factor (1 unless the entries are all below 2⁻⁵⁰⁰, 2⁻⁶⁰ in float).

Definition at line 478 of file factorizations.hpp.

◆ rescue_factor()

template<typename T >
carried_t< T > numerical::detail::rescue_factor ( carried_t< T >  largest)

The exact power of two that brings a magnitude below about the square root of the smallest normal number of T's range (2⁻⁵⁰⁰ in double, 2⁻⁶⁰ in float) up among normal numbers, as LAPACK's dlarfg rescales; 1 for any larger magnitude. It is a T itself: a zero takes the factor of the smallest normal number, which a double's range holds and a float's does not.

Definition at line 390 of file factorizations.hpp.

◆ residual_ratio()

double numerical::detail::residual_ratio ( const matrix< double > &  x,
const matrix< double > &  r 
)
inlineprivate

The largest of ‖r‖∞ / ‖x‖∞ over the columns of the residual r and the solution x (LAPACK's dsgesv compares each with ‖A‖∞ ε √n); NaN, which no bound meets, where an entry of r is not finite.

Definition at line 1686 of file factorizations.hpp.

◆ reversed_adjoint()

template<typename T >
buffer< T > numerical::detail::reversed_adjoint ( const buffer< T > &  t,
std::size_t  n 
)

S = J Tᴴ J of the upper (quasi-)triangular t (row-major n × n, J the reversal): upper (quasi-)triangular again, a 2 × 2 block of standard form one still, its eigenvalues t's conjugated in reverse order. A right eigenvector w of S for conj λ gives J w, a left one of t for λ (yᴴ t = λ yᴴ, LAPACK's trevc side L).

Definition at line 3036 of file eigen.hpp.

◆ reversed_rows()

template<typename T >
buffer< T > numerical::detail::reversed_rows ( const buffer< T > &  w,
std::size_t  n 
)

Row j of the rows w (n × n) its row n − 1 − j, its entries in reverse order: J applied to each of reversed_adjoint's vectors, in the order of t's eigenvalues.

Definition at line 3051 of file eigen.hpp.

◆ rfft_of()

template<std::floating_point T>
tensor< std::complex< T > > numerical::detail::rfft_of ( const fft_source< T > &  a,
const std::optional< std::size_t > &  n,
std::ptrdiff_t  axis,
fft_norm  norm 
)

rfft's computation, of real elements a caller lends.

Definition at line 2897 of file fft.hpp.

◆ riccati()

template<typename T >
matrix< T > numerical::detail::riccati ( const riccati_operands< T > &  terms,
bool  continuous,
bool  balanced 
)
private

The stabilising solution of the continuous (continuous) or discrete algebraic Riccati equation by SciPy's algorithm: the extended pencil, balanced unless balanced is false, deflated by the QR factorisation of its last n columns, its generalized Schur form sorted with the eigenvalues in the left half-plane (inside the unit circle) first, and X = U₁₀ U₀₀⁻¹ from the leading m columns of Z (with E, of the QR factorisation of [Ẽ U₀₀; U₁₀]), returned as (X + Xᴴ)/2 in the original scaling. Ẽ is E as the balanced pencil holds it, D₁ E D₁⁻¹: the basis U is of the balanced pencil, so X̃ Ẽ U₀₀ = U₁₀ for X̃ = D₁⁻¹ X D₁⁻¹, and the unscaled E there (as SciPy has it) answers a wrong X or none.

Exceptions
singular_matrix_errorIf U₀₀ is numerically singular (the condition of its LU factor U past 1/ε, as SciPy tests it): no finite solution. Where (A, B) cannot be stabilised, the computed U₀₀ is singular only to rounding, and its condition may stay below 1/ε: the answer is then a huge X, as SciPy's.
convergence_errorIf U₀₀ᴴ U₁₀ is not Hermitian: the pencil's eigenvalues too close to the imaginary axis (the unit circle).

Definition at line 2906 of file generalized.hpp.

◆ riccati_pencil()

template<typename T >
std::pair< buffer< T >, buffer< T > > numerical::detail::riccati_pencil ( const riccati_operands< T > &  terms,
bool  continuous 
)
private

The extended pencil (H, J) of order 2m + n of SciPy's solve_continuous_are (continuous) or solve_discrete_are, row-major.

Definition at line 2810 of file generalized.hpp.

◆ rolled()

template<typename T >
tensor< T > numerical::detail::rolled ( const tensor< T > &  a,
const std::vector< std::size_t > &  shifts 
)

a rolled along every axis, element j of axis d to (j + shifts[d]) mod n_d, in one copy: each row of the result along its last axis read from its source row, in two contiguous pieces; the rows shared among the threads.

Definition at line 3049 of file fft.hpp.

◆ root()

template<typename T >
T numerical::detail::root ( const T &  x)

The square root of a scalar, as std::sqrt gives it.

The square root of a complex number of float or double: complex_sqrt.

Definition at line 280 of file tensor_ops.hpp.

◆ rotate()

template<typename T >
void numerical::detail::rotate ( T *  x,
T *  y,
std::size_t  count,
std::size_t  stride,
double  cs,
double  sn 
)

(x, y) ← (cs x + sn y, cs y − sn x) over count pairs stride apart: a plane rotation.

Definition at line 791 of file eigen.hpp.

◆ rotate_band()

template<typename T >
void numerical::detail::rotate_band ( band_work< T > &  a,
std::size_t  p,
double  c,
const T &  s,
buffer< T > &  turned 
)

A ← G A Gᴴ of the band a, G the rotation [c s; −s̄ c] on rows and columns p and p + 1: the two rows within the bulge's reach, then their 2 × 2 block; turned (Mᴴ's rows, M the product of the rotations, where not empty) turned with them.

Definition at line 3747 of file eigen.hpp.

◆ rotate_columns()

void numerical::detail::rotate_columns ( buffer< double > &  m,
std::size_t  cols,
std::size_t  a,
std::size_t  b,
double  c,
double  s 
)
inline

Columns a and b of the row-major m (cols per row) rotated: a ← c a + s b, b ← c b − s a.

Definition at line 912 of file svd.hpp.

◆ rotate_pair()

template<typename T >
void numerical::detail::rotate_pair ( T *  x,
T *  y,
std::size_t  count,
std::size_t  stride,
double  c,
const T &  s 
)

(x, y) ← (c x + s y, c y − s̄ x) over count pairs stride apart.

Definition at line 1347 of file eigen.hpp.

◆ rotation_length()

double numerical::detail::rotation_length ( double  a,
double  b 
)
inlinenoexcept

√(a² + b²) for a plane rotation of the QR and QL iterations, as LAPACK's dlartg takes it: the plain sum of squares where the larger magnitude lies within 2^±500, so its square neither overflows nor underflows and the smaller one's underflow is below its rounding; std::hypot, which scales every time, outside. A graded matrix reaches both ends though scaled to its largest entry: a QL shift's ratio (d₁ − d₀) / 2e came to 5e199 on [[1e-200, 1e-200], [1e-200, 1]].

Definition at line 571 of file factorizations.hpp.

◆ rotation_runs()

std::vector< rotation_run > numerical::detail::rotation_runs ( const std::vector< row_operation > &  operations)
inline

The operations split into runs of chained rotations, every other operation a run of one.

Definition at line 5024 of file factorizations.hpp.

◆ rounded_alone()

template<std::floating_point R>
R numerical::detail::rounded_alone ( R  x)
inlinenoexcept

x, held in a register as its rounded value: GCC on arm64 with the complex-arithmetic unit (ARMv8.3's FCMLA: -mcpu=apple-m1, Neoverse V1) recognizes a·b − c·d, a·d + c·b as a complex product and fuses one product into the other's sum whatever -ffp-contract says; elsewhere x as it is.

Definition at line 291 of file fft.hpp.

◆ rounded_apart_add() [1/2]

template<typename R >
std::complex< R > numerical::detail::rounded_apart_add ( std::complex< R >  sum,
std::complex< R >  a,
std::complex< R >  b 
)
inlinenoexcept

rounded_apart_add of complex numbers: the product in line (operator_times, the operator's own rounding, unfused), where it was a call that took a sixth of a complex LU.

Definition at line 1732 of file gemm.hpp.

◆ rounded_apart_add() [2/2]

template<typename T >
T numerical::detail::rounded_apart_add ( T  sum,
T  a,
T  b 
)
inlinenoexcept

sum + a·b as the vector tile rounds it (vector_tile_kernel), so that the plain loop's elements are its own: under Clang's -ffp-contract=on one expression, fused in both; under GCC's, whose vector extensions fuse nothing while a scalar statement's multiply-add fuses, the product apart, a statement of its own.

Definition at line 1717 of file gemm.hpp.

◆ rounded_product()

template<std::floating_point R>
std::complex< R > numerical::detail::rounded_product ( const std::complex< R > &  p,
const std::complex< R > &  q 
)
inlinenoexcept

p · q of complex numbers, each product rounded apart from the sum it enters (statements of their own, rounded_alone).

Definition at line 302 of file fft.hpp.

◆ rounded_term() [1/2]

template<typename T >
T numerical::detail::rounded_term ( rounding_is< term_rounding::compiler >  ,
T  sum,
T  a,
T  b 
)
inlinenoexcept

sum + a·b as the compiler rounds it (rounded_apart_add).

Definition at line 1791 of file gemm.hpp.

◆ rounded_term() [2/2]

template<typename T >
T numerical::detail::rounded_term ( rounding_is< term_rounding::fused >  ,
T  sum,
T  a,
T  b 
)
inlinenoexcept

sum + a·b as one fused multiply-add.

Definition at line 1801 of file gemm.hpp.

◆ rounding_direction()

std::complex< double > numerical::detail::rounding_direction ( std::size_t  place)
inline

A direction of unit modulus for the rounding of entry place of an estimate's model, fixed and varying from one entry to the next.

Definition at line 1593 of file matrix_functions.hpp.

◆ row_dots() [1/5]

void numerical::detail::row_dots ( const double *  m,
std::size_t  ld,
std::size_t  rows,
const double *  x,
std::size_t  count,
double *  out 
)
inline

row_dots for double: on the x86-64 units each row by their dot_product (wider than four rows of the portable vectors), elsewhere four rows at a time.

Definition at line 825 of file gemm.hpp.

◆ row_dots() [2/5]

void numerical::detail::row_dots ( const float *  m,
std::size_t  ld,
std::size_t  rows,
const float *  x,
std::size_t  count,
float *  out 
)
inline

row_dots for float: on the x86-64 units each row by their float dot product (dot_product), elsewhere the portable vectors'.

Definition at line 848 of file gemm.hpp.

◆ row_dots() [3/5]

void numerical::detail::row_dots ( const std::complex< double > *  m,
std::size_t  ld,
std::size_t  rows,
const std::complex< double > *  x,
std::size_t  count,
std::complex< double > *  out 
)
inline

row_dots for std::complex<double>, through complex_row_dots.

Definition at line 802 of file gemm.hpp.

◆ row_dots() [4/5]

void numerical::detail::row_dots ( const std::complex< float > *  m,
std::size_t  ld,
std::size_t  rows,
const std::complex< float > *  x,
std::size_t  count,
std::complex< float > *  out 
)
inline

row_dots for std::complex<float>, through complex_row_dots.

Definition at line 813 of file gemm.hpp.

◆ row_dots() [5/5]

template<typename T >
void numerical::detail::row_dots ( const T *  m,
std::size_t  ld,
std::size_t  rows,
const T *  x,
std::size_t  count,
T *  out 
)

dot_product of each row of m with x, for a type no native vector holds.

Definition at line 1102 of file simd.hpp.

◆ row_major_strides()

std::vector< std::ptrdiff_t > numerical::detail::row_major_strides ( const std::vector< std::size_t > &  shape)
inline

The row-major strides of shape, in elements.

Definition at line 710 of file tensor.hpp.

◆ row_sum_norm()

template<typename T >
double numerical::detail::row_sum_norm ( const buffer< T > &  x,
std::size_t  n,
std::size_t  k 
)

The largest row sum of moduli of the n × k row-major x (‖X‖∞).

Definition at line 585 of file matrix_functions.hpp.

◆ rows_from_transpose()

template<typename T >
const T * numerical::detail::rows_from_transpose ( const T *  x,
std::size_t  ld,
std::size_t  rows,
std::size_t  cols,
buffer< T > &  scratch,
std::size_t &  ld_out 
)

A row-major rows × cols operand for the plain loop, copied into scratch from its transpose x (cols × rows, ld apart).

Definition at line 1195 of file gemm.hpp.

◆ rows_fuse()

bool numerical::detail::rows_fuse ( )
inlinenoexcept

Whether the tile a double or float product past the plain loop runs on fuses each term (gemm_rows rounds its own so): double on arm64's NEON and AMX paths, float on the NEON tile there. The x86-64 tiles' double takes term_rounding::x86_fused; the vector tile, and so the portable path, rounds as the compiler does, as the plain loop then does. On arm64 every cap but the portable one runs the NEON or the AMX path, so the cap alone answers: one load, where the path in use took an order-4 product on one arm64 core from 10.6 to 13.3 ns (the cap, 11.5).

Definition at line 168 of file gemm.hpp.

◆ rows_in_place()

template<typename T >
const T * numerical::detail::rows_in_place ( const T *  x,
std::size_t  ld,
std::size_t &  ld_out 
)
noexcept

A row-major rows × cols operand for the plain loop, x itself (ld apart).

Definition at line 1184 of file gemm.hpp.

◆ rows_of_tile()

template<std::size_t R, typename T , typename Full >
void numerical::detail::rows_of_tile ( std::size_t  rows,
T *  c,
std::size_t  ldc,
Full &  full 
)
noexcept

full's tile of its first R rows, or of rows of them where fewer (through_rows).

Definition at line 949 of file simd.hpp.

◆ rows_terms_fuse()

template<std::floating_point T>
bool numerical::detail::rows_terms_fuse ( )
noexcept

Whether gemm_rows rounds each term c + a·b as one fused multiply-add on the path in use, as its choice of term_rounding below makes it (the x86-64 AVX paths' double and float, arm64's fused paths' floating types): the first term of a product that sums nothing (0 + a·b) rounded as gemm_rows rounds it elsewhere.

Definition at line 2295 of file gemm.hpp.

◆ run_fused()

template<typename Body >
void numerical::detail::run_fused ( const Body &  body)

body(0), where std::fma is the instruction (with_fma_instruction) on an x86-64 processor with AVX2 and FMA: a body whose multiply-adds are all std::fma's, NUMERICAL_INLINE_LAMBDA, as what it calls must be.

Definition at line 698 of file tensor.hpp.

◆ runs_into()

template<typename T , typename Kernel >
tensor< T > numerical::detail::runs_into ( const tensor< T > &  t,
const Kernel &  kernel 
)

kernel (kernel(in, out, count), of contiguous elements) of every element of t: a contiguous tensor at once by chunks of elementwise_chunk among the threads; a view lying close (lies_close; a slice, its rows apart) read where it lies, each run of its last axis passed where it is contiguous and gathered by pieces of 512 on the stack where its elements are apart; any other copied first. Each element alone: the bits of the same elements laid row-major (a slice of 200 000 doubles' exp copied first took a third more time).

Definition at line 2386 of file tensor_ops.hpp.

◆ same_sign() [1/2]

template<std::floating_point R>
bool numerical::detail::same_sign ( R  x,
int  sign 
)
noexcept

Whether a real entry's sign is sign (sign_entry's), as dlacn2 finds a sign vector repeated.

Definition at line 2106 of file factorizations.hpp.

◆ same_sign() [2/2]

template<std::floating_point R>
bool numerical::detail::same_sign ( std::complex< R >  ,
int   
)
noexcept

A complex entry's sign never repeats (zlacn2 has no such test).

Definition at line 2114 of file factorizations.hpp.

◆ scalar_log() [1/2]

std::complex< double > numerical::detail::scalar_log ( const std::complex< double > &  z)
inline

complex_log of z.

Definition at line 2815 of file matrix_functions.hpp.

◆ scalar_log() [2/2]

double numerical::detail::scalar_log ( double  x)
inline

log x of a real x above 0.

Definition at line 2809 of file matrix_functions.hpp.

◆ scalar_sqrt() [1/2]

std::complex< double > numerical::detail::scalar_sqrt ( const std::complex< double > &  z)
inline

csqrt's √z, exact on the axes as NumPy's and SciPy's.

Definition at line 1302 of file matrix_functions.hpp.

◆ scalar_sqrt() [2/2]

double numerical::detail::scalar_sqrt ( double  x)
inline

√x of a real x at least 0.

Definition at line 1296 of file matrix_functions.hpp.

◆ scaled()

template<std::floating_point R>
std::complex< R > numerical::detail::scaled ( R  c,
const std::complex< R > &  v 
)
inlinenoexcept

c · v of a real c and a complex v: each part's product, rounded.

Definition at line 1290 of file fft.hpp.

◆ scaled_add() [1/2]

template<bool Fused, std::floating_point R>
requires Fused
std::complex< R > numerical::detail::scaled_add ( const std::complex< R > &  s,
R  c,
const std::complex< R > &  v 
)
inlinenoexcept

s + c · v of a complex s, a real c and a complex v on the vector paths: each part one fused multiply-add.

Definition at line 1301 of file fft.hpp.

◆ scaled_add() [2/2]

template<bool Fused, std::floating_point R>
requires (!Fused)
std::complex< R > numerical::detail::scaled_add ( const std::complex< R > &  s,
R  c,
const std::complex< R > &  v 
)
inlinenoexcept

scaled_add on the portable path: the product rounded first.

Definition at line 1312 of file fft.hpp.

◆ scaled_back()

template<typename T >
matrix< T > numerical::detail::scaled_back ( const buffer< T > &  m,
std::size_t  n,
double  factor 
)

The entries of m multiplied by factor, n × n row-major (the scaled form taken back).

Definition at line 2154 of file generalized.hpp.

◆ scaled_by_two() [1/2]

template<typename U >
std::complex< U > numerical::detail::scaled_by_two ( const std::complex< U > &  x,
int  exponent 
)
private

scaled_by_two for a complex number, part by part.

Definition at line 1995 of file svd.hpp.

◆ scaled_by_two() [2/2]

template<typename T >
T numerical::detail::scaled_by_two ( const T &  x,
int  exponent 
)
private

x times two to the exponent, exactly where the result is a normal number (no power of two is formed, so none overflows).

Definition at line 1987 of file svd.hpp.

◆ scaled_norm() [1/4]

template<typename T >
carried_t< T > numerical::detail::scaled_norm ( const buffer< T > &  x)

scaled_norm of a whole vector.

Definition at line 559 of file factorizations.hpp.

◆ scaled_norm() [2/4]

double numerical::detail::scaled_norm ( const double *  x,
std::size_t  count 
)
inline

The same for double: the plain sum of squares in one pass of dot_product where it holds.

Definition at line 542 of file factorizations.hpp.

◆ scaled_norm() [3/4]

double numerical::detail::scaled_norm ( const std::complex< double > *  x,
std::size_t  count 
)
inline

The same for std::complex<double>: dot_product over the parts, which an array of them holds in order ([complex.numbers]).

Definition at line 550 of file factorizations.hpp.

◆ scaled_norm() [4/4]

template<typename T >
carried_t< T > numerical::detail::scaled_norm ( const T *  x,
std::size_t  count 
)

The Euclidean norm of the count elements at x, by norm_at_scale for this type.

Definition at line 535 of file factorizations.hpp.

◆ scaled_product()

template<typename T , typename Factor >
T numerical::detail::scaled_product ( T  start,
std::size_t  count,
const Factor &  factor 
)

The product of count factors, factor(i) for each i, times start, as a determinant multiplies its pivots: each factor, and the running product after it, brought near 1 by a power of 2 (exactly) and the powers kept apart, then applied once at the end. The same bits as the plain product wherever that stays normal; where a partial product would overflow or underflow (pivots of 10¹⁵⁰ and 10⁻¹⁵⁰), the representable answer.

Definition at line 435 of file factorizations.hpp.

◆ scaled_to_float()

std::pair< matrix< float >, int > numerical::detail::scaled_to_float ( const matrix< double > &  m)
inlineprivate

The finite m times 2⁻ᵉ in single precision, 2ᵉ the least power of two above its largest magnitude: every entry below 1 in magnitude, so none overflows float, and the scaling exact.

Returns
The matrix and e.

Definition at line 1657 of file factorizations.hpp.

◆ scaled_tridiagonal()

double numerical::detail::scaled_tridiagonal ( const vector< double > &  d,
const vector< double > &  e,
buffer< double > &  diagonal,
buffer< double > &  off 
)
inlineprivate

The diagonal d and off-diagonal e of a symmetric tridiagonal matrix divided by the power of two of their largest magnitude (returned), e padded with a 0 to d's length, as the QL iteration takes them.

Exceptions
dimension_errorIf e has not one entry fewer than d (none for an empty d).
non_finite_errorIf an entry is infinite or NaN.

Definition at line 2369 of file svd.hpp.

◆ scaling_balance()

buffer< double > numerical::detail::scaling_balance ( buffer< double > &  m,
std::size_t  n 
)
inlineprivate

The scaling of LAPACK's ‘dgebal('S’)for the non-negativen × n\p m (row-major, scaled in place): rows and columns by powers of two until their 2-norms are within a factor of 2, a row or column of zeros left alone, the factors kept within the safe range. \return The scalesDofD⁻¹ M D`.

Definition at line 2673 of file generalized.hpp.

◆ scan_along()

template<typename T , typename Step >
tensor< T > numerical::detail::scan_along ( const tensor< T > &  t,
std::size_t  axis,
const Step &  step 
)

Each line of t along axis replaced by its running step (x₀, step(x₀, x₁), …), in order.

Definition at line 3378 of file tensor_ops.hpp.

◆ schur_block() [1/2]

std::size_t numerical::detail::schur_block ( const buffer< double > &  t,
std::size_t  n,
std::size_t  k 
)
inline

The order of the diagonal block of the upper quasi-triangular t (n × n, row-major) that starts at k: 2 where the entry below its diagonal is nonzero (a real Schur form's pair).

Definition at line 712 of file matrix_functions.hpp.

◆ schur_block() [2/2]

std::size_t numerical::detail::schur_block ( const buffer< std::complex< double > > &  ,
std::size_t  ,
std::size_t   
)
inline

The same for a complex Schur form, triangular: 1.

Definition at line 720 of file matrix_functions.hpp.

◆ schur_form() [1/2]

void numerical::detail::schur_form ( const schur_target< double > &  t,
buffer< double > &  re,
buffer< double > &  im,
std::size_t  steps 
)
inline

The real Schur form of t's active block: real_schur below multishift_from, else multishift_schur, as LAPACK's dhseqr chooses. steps bounds real_schur's iterations.

Definition at line 2654 of file eigen.hpp.

◆ schur_form() [2/2]

void numerical::detail::schur_form ( const schur_target< std::complex< double > > &  t,
buffer< std::complex< double > > &  values,
std::size_t  steps 
)
inline

The complex Schur form likewise: complex_schur or multishift_schur.

Definition at line 2672 of file eigen.hpp.

◆ schur_logm()

template<typename T >
logm_result numerical::detail::schur_logm ( buffer< T >  t,
std::size_t  n,
const matrix< T > *  z 
)

log A from the upper triangular n × n t of A = Z T Zᴴ (z, or none where T is A), its zero diagonal entries taken as 10⁻²⁰ (force_nonsingular).

Definition at line 3454 of file matrix_functions.hpp.

◆ schur_values() [1/2]

std::vector< std::complex< double > > numerical::detail::schur_values ( const double *  t,
std::size_t  n 
)
inlineprivate

The eigenvalues of the real quasi-triangular t (n × n, row-major), in its order: a 2 × 2 block of standard form [a b; c a] the pair a ± i √|b| √|c|, as LAPACK's dlanv2 gives them, the first positive.

Definition at line 3376 of file eigen.hpp.

◆ schur_values() [2/2]

std::vector< std::complex< double > > numerical::detail::schur_values ( const std::complex< double > *  t,
std::size_t  n 
)
inlineprivate

The eigenvalues of the complex upper triangular t: its diagonal.

Definition at line 3396 of file eigen.hpp.

◆ secular_root()

double numerical::detail::secular_root ( const buffer< double > &  d,
const buffer< double > &  z,
double  rho,
std::size_t  i,
std::size_t  max_iterations,
double *  delta 
)
inline

Root i of 1/ρ + Σⱼ zⱼ²/(dⱼ − λ) = 0 (d strictly ascending, ‖z‖ = 1), the i-th eigenvalue of diag(d) + ρ z zᵀ, by solve_secular.

Parameters
[in]dThe poles.
[in]zThe weights.
[in]rhoρ.
[in]iThe root.
[in]max_iterationsThe bound on the steps.
[out]deltadⱼ − λ for every j.
Returns
λ.

Definition at line 5697 of file factorizations.hpp.

◆ select_positions()

template<typename It , typename Less >
void numerical::detail::select_positions ( It  first,
std::size_t  count,
const std::vector< std::size_t > &  kths,
const Less &  less 
)

Rearranges [first, first + count) by less so that each of the ascending kths holds the element a sort puts there: std::nth_element over what lies past the previous one.

Definition at line 4048 of file tensor_ops.hpp.

◆ select_row() [1/2]

template<split_complex C>
void numerical::detail::select_row ( const C *  row,
std::size_t  count,
const std::int64_t *  at,
std::size_t  m,
C *  work,
C *  out 
)

A complex row is sorted whole: its classes under sorts_before are zeros and NaNs in either part.

Definition at line 4763 of file tensor_ops.hpp.

◆ select_row() [2/2]

template<typename T >
requires std::floating_point<T> || std::integral<T>
void numerical::detail::select_row ( const T *  row,
std::size_t  count,
const std::int64_t *  at,
std::size_t  m,
T *  work,
T *  out 
)

Into out, the elements of the count at row that a stable sort puts at the m positions at, work holding count elements of scratch: std::nth_element at each distinct position, smallest first, each over the part above the one before (the least or the largest of that part by a pass alone), then stable_choice; more positions than select_positions_sorted, the row sorted whole.

Definition at line 4715 of file tensor_ops.hpp.

◆ selected_leading() [1/2]

std::size_t numerical::detail::selected_leading ( const std::vector< std::complex< double > > &  ,
const eigenvalue_selection &  ,
std::size_t  moved,
std::type_identity< std::complex< double > >   
)
inlineprivate

The count of a complex Schur form's sort, as zgees takes trsen's: the eigenvalues it moved.

Definition at line 3418 of file eigen.hpp.

◆ selected_leading() [2/2]

std::size_t numerical::detail::selected_leading ( const std::vector< std::complex< double > > &  after,
const eigenvalue_selection &  select,
std::size_t  ,
std::type_identity< double >   
)
inlineprivate

The count of a real Schur form's sort, as dgees checks the order the exchanges left, from the eigenvalues as they now stand (after): a pair selected where either of its two is, counted twice.

Exceptions
convergence_errorIf a selected eigenvalue follows one not selected (dgees's info n + 2).

Definition at line 3431 of file eigen.hpp.

◆ separate_cosine()

template<std::floating_point R>
R numerical::detail::separate_cosine ( R  x)

std::cos of x in a call of its own, as separate_sine.

Definition at line 87 of file complex_math.hpp.

◆ separate_sine()

template<std::floating_point R>
R numerical::detail::separate_sine ( R  x)

std::sin of x in a call of its own, as NumPy's formulas take it: no build fuses it with a cos of the same argument into a sincos, which strays from them by a unit in the last place (glibc's sincosf before 2.29, macOS's __sincos).

Definition at line 80 of file complex_math.hpp.

◆ settled_zero()

template<std::floating_point T, typename Order >
T numerical::detail::settled_zero ( T  value,
const T *  first,
std::size_t  count,
std::size_t  stride,
const Order &  larger 
)

The extreme value of count elements stride apart from first, its zero's sign settled as IEEE 754-2019's maximum and minimum settle it, and NumPy's on arm64: no zero is larger than the other, so the first met stayed; max gives +0 where one is among them, min −0, whatever their order. A NaN, or any other value, as it is.

settled_zero of a type with one zero: value.

Definition at line 1447 of file tensor_ops.hpp.

◆ shape_text()

std::string numerical::detail::shape_text ( const std::vector< std::size_t > &  shape)
inline

A shape as NumPy prints it, (2, 3), for a message.

Definition at line 649 of file tensor.hpp.

◆ shared_cosine_line()

template<std::floating_point R>
std::shared_ptr< const cosine_line< R > > numerical::detail::shared_cosine_line ( int  type,
std::size_t  n 
)

The shared cosine_line of type on n points, built on first use and kept among the fft_plans_kept most recently used, as shared_fft_plan keeps its plans.

< The line's type.

< Its points.

< The line.

Definition at line 3401 of file fft.hpp.

◆ shared_fft_plan()

template<std::floating_point R>
std::shared_ptr< const fft_plan< R > > numerical::detail::shared_fft_plan ( std::size_t  n)

The shared fft_plan of length n, built on first use and kept among the fft_plans_kept most recently used.

< The plan's length.

< The plan.

Definition at line 1113 of file fft.hpp.

◆ shared_real_fft_plan()

template<std::floating_point R>
std::shared_ptr< const real_fft_plan< R > > numerical::detail::shared_real_fft_plan ( std::size_t  n)

The real_fft_plan of n, shared as shared_fft_plan shares the complex ones.

< The plan's length.

< The plan.

Definition at line 2478 of file fft.hpp.

◆ sign_entry() [1/2]

template<std::floating_point R>
R numerical::detail::sign_entry ( R  x,
int &  sign 
)
noexcept

A real entry's sign for one_norm_estimate's sign vectors, dlacn2's: 1 from zero up, else −1; sign set to it.

Definition at line 2086 of file factorizations.hpp.

◆ sign_entry() [2/2]

template<std::floating_point R>
std::complex< R > numerical::detail::sign_entry ( std::complex< R >  x,
int &  sign 
)
noexcept

A complex entry's unit-modulus sign, zlacn2's: x / |x|, 1 where |x| is not past the least normal number; sign 0, a complex climb having no sign vector to repeat.

Definition at line 2096 of file factorizations.hpp.

◆ similar_power()

buffer< std::complex< double > > numerical::detail::similar_power ( const matrix< std::complex< double > > &  similar,
double  t 
)
inlineprivate

T^t for T the upper triangle of similar (Zᴴ A Z, Z a Hermitian A's eigenvectors): the similarity's rounding above the diagonal kept as a Schur form keeps it (Λ alone erred two to five times more), its diagonal taken real (a negative eigenvalue's power the principal value's), T^t to first order in that rounding (nearly_diagonal_power, in place of the Padé steps: order 200 35 → 9 ms, the same errors).

Precondition
No real part on the diagonal is 0.

Definition at line 3231 of file matrix_functions.hpp.

◆ similarity_of()

template<typename T >
hermitian_similarity< T > numerical::detail::similarity_of ( const matrix< T > &  a)
private

hermitian_similarity of the Hermitian a.

Definition at line 3217 of file matrix_functions.hpp.

◆ sin_of() [1/3]

tensor< double > numerical::detail::sin_of ( const double *  data,
const std::vector< std::size_t > &  shape 
)
inline

sin of every one of the row-major doubles at data, of shape shape: double_sines_into by chunks of elementwise_chunk on the threads.

Definition at line 1136 of file tensor_ops.hpp.

◆ sin_of() [2/3]

tensor< float > numerical::detail::sin_of ( const float *  data,
const std::vector< std::size_t > &  shape 
)
inline

sin of every one of the row-major floats at data, of shape shape: float_sines_into of them as doubles (through_doubles).

Definition at line 1128 of file tensor_ops.hpp.

◆ sin_of() [3/3]

template<typename T >
tensor< T > numerical::detail::sin_of ( const T *  data,
const std::vector< std::size_t > &  shape 
)

sin of every one of the row-major scalars at data, of shape shape: std::sin of each.

Definition at line 1120 of file tensor_ops.hpp.

◆ sin_viewed() [1/3]

tensor< double > numerical::detail::sin_viewed ( const tensor< double > &  a)
inline

sin of every double of a, a view read where it lies (by_runs).

Definition at line 2599 of file tensor_ops.hpp.

◆ sin_viewed() [2/3]

tensor< float > numerical::detail::sin_viewed ( const tensor< float > &  a)
inline

sin of every float of a through doubles, a view read where it lies (by_runs).

Definition at line 2605 of file tensor_ops.hpp.

◆ sin_viewed() [3/3]

template<typename T >
tensor< T > numerical::detail::sin_viewed ( const tensor< T > &  a)

sin of every element of a: of a contiguous copy (sin_of).

Definition at line 2593 of file tensor_ops.hpp.

◆ sinch()

template<typename T >
T numerical::detail::sinch ( const T &  x)

sinh(x) / x, 1 at 0, by its series where x is small.

Definition at line 260 of file matrix_functions.hpp.

◆ sine_and_cosine() [1/2]

void numerical::detail::sine_and_cosine ( double  x,
double &  sine,
double &  cosine 
)
inline

sin and cos of x together, as glibc's complex functions take them: its sincos under glibc (whose single-precision one strayed from sinf and cosf by a unit in the last place before 2.29), the system's __sincos on macOS (a compiler may fuse a sin and a cos into it where it inlines them, and it strays from them by a unit: the same call in every context), std::sin and std::cos elsewhere.

Definition at line 47 of file complex_math.hpp.

◆ sine_and_cosine() [2/2]

void numerical::detail::sine_and_cosine ( float  x,
float &  sine,
float &  cosine 
)
inline

sine_and_cosine of a float.

Definition at line 62 of file complex_math.hpp.

◆ sine_terms_of()

template<bool Cosine>
sine_terms numerical::detail::sine_terms_of ( std::uint64_t  entry,
double  x 
)
noexcept

sine_terms of the table's row at entry (sin head, sin tail, cos head, cos tail) for x: the sine's where not Cosine, the cosine's otherwise; one specialization each, no choice at run time.

◆ sine_terms_of< false >()

template<>
sine_terms numerical::detail::sine_terms_of< false > ( std::uint64_t  entry,
double  x 
)
inlinenoexcept

The terms of the sine (sine_terms): a = S, b = C, x itself where tiny.

Definition at line 641 of file elementary.hpp.

◆ sine_terms_of< true >()

template<>
sine_terms numerical::detail::sine_terms_of< true > ( std::uint64_t  entry,
double   
)
inlinenoexcept

The terms of the cosine (sine_terms): a = C, b = −S, 1 where tiny.

Definition at line 649 of file elementary.hpp.

◆ singular_secular_root()

double numerical::detail::singular_secular_root ( const buffer< double > &  d,
const buffer< double > &  z,
std::size_t  i,
std::size_t  max_iterations,
double *  below,
double *  above 
)
inline

Singular value i of the arrow M whose first row is z and whose diagonal is d (d₀ = 0, d strictly ascending): root σ of 1 + Σⱼ zⱼ²/(dⱼ² − σ²) = 0, as LAPACK's dlasd4.

Solved by solve_secular in μ = σ² − d_o², the poles' offsets (dⱼ − d_o)(dⱼ + d_o) computed from their two accurate factors, so that both dⱼ − σ and dⱼ + σ come out with small relative errors.

Parameters
[in]dThe diagonal (the poles).
[in]zThe first row (the weights).
[in]iThe singular value.
[in]max_iterationsThe bound on the steps.
[out]belowdⱼ − σ for every j.
[out]abovedⱼ + σ for every j.
Returns
σ.

Definition at line 891 of file svd.hpp.

◆ singular_value_in_double() [1/2]

double numerical::detail::singular_value_in_double ( double  x)
inlinenoexcept

A singular value of a decomposition carried in double: x itself.

Definition at line 1559 of file svd.hpp.

◆ singular_value_in_double() [2/2]

double numerical::detail::singular_value_in_double ( long double  x)
inline

A singular value of a long double matrix, computed in double at the matrix's scale and scaled back, as the double its type holds. Where long double's range is wider than double's (x86-64's 80 bits), one past double's normal numbers cannot be held, and the decomposition refuses the matrix rather than answer 0 or ∞.

Exceptions
argument_errorIf x is nonzero and outside double's normal range.

Definition at line 1570 of file svd.hpp.

◆ sinh_into()

void numerical::detail::sinh_into ( const double *  in,
double *  out,
std::size_t  count 
)
inline

sinh of the count doubles at in into out, each elementary_sinh's: table_sinh in vectors (with the FMA instructions on an x86-64 processor, run_fused), std::sinh throughout where exp_by_table says no.

Definition at line 604 of file tensor_ops.hpp.

◆ sinh_of() [1/3]

tensor< double > numerical::detail::sinh_of ( const double *  data,
const std::vector< std::size_t > &  shape 
)
inline

sinh of every one of the row-major doubles at data, of shape shape: sinh_into by chunks of elementwise_chunk among the threads.

Definition at line 641 of file tensor_ops.hpp.

◆ sinh_of() [2/3]

tensor< float > numerical::detail::sinh_of ( const float *  data,
const std::vector< std::size_t > &  shape 
)
inline

sinh of every one of the row-major floats at data, of shape shape: sinh_into of them as doubles (through_doubles).

Definition at line 662 of file tensor_ops.hpp.

◆ sinh_of() [3/3]

template<typename T >
tensor< T > numerical::detail::sinh_of ( const T *  data,
const std::vector< std::size_t > &  shape 
)

sinh of every one of the row-major scalars at data, of shape shape: std::sinh of each.

Definition at line 633 of file tensor_ops.hpp.

◆ sinh_viewed() [1/3]

tensor< double > numerical::detail::sinh_viewed ( const tensor< double > &  a)
inline

sinh of every double of a, a view read where it lies (by_runs).

Definition at line 2529 of file tensor_ops.hpp.

◆ sinh_viewed() [2/3]

tensor< float > numerical::detail::sinh_viewed ( const tensor< float > &  a)
inline

sinh of every float of a through doubles, a view read where it lies (by_runs).

Definition at line 2535 of file tensor_ops.hpp.

◆ sinh_viewed() [3/3]

template<typename T >
tensor< T > numerical::detail::sinh_viewed ( const tensor< T > &  a)

sinh of every element of a: of a contiguous copy (sinh_of).

Definition at line 2523 of file tensor_ops.hpp.

◆ sits_out()

constexpr bool numerical::detail::sits_out ( std::size_t  worker,
std::size_t  caller 
)
constexprnoexcept

Whether the worker pinned to core worker sits out a run whose caller is on core caller: one thread per core, as two on one core's hyperthreads share its multiply-add units. A free worker never does.

Definition at line 83 of file parallel.hpp.

◆ small_norm()

double numerical::detail::small_norm ( const small_square &  x,
std::size_t  r0,
std::size_t  r1,
std::size_t  c0,
std::size_t  c1 
)
inline

The Frobenius norm of rows r0 … r1 − 1 and columns c0 … c1 − 1 of x.

Definition at line 1489 of file generalized.hpp.

◆ small_product()

small_square numerical::detail::small_product ( const small_square &  x,
bool  tx,
const small_square &  y,
bool  ty,
std::size_t  m 
)
inline

op(x) op(y) of order m, op the transpose where tx (ty).

Definition at line 1469 of file generalized.hpp.

◆ small_qr()

small_square numerical::detail::small_qr ( small_square  x,
std::size_t  m,
std::size_t  k 
)
inline

The orthogonal Q (order m) of the Householder QR factorisation of the first k columns of x (m × k, k ≤ m), as LAPACK's dgeqr2 then dorg2r: its first k columns span theirs, Qᵀ x upper trapezoidal.

Definition at line 1509 of file generalized.hpp.

◆ small_reflector()

double numerical::detail::small_reflector ( double(&)  v[3],
std::size_t  count 
)
inline

The Householder reflection I − τ v vᵀ, v = (1, x…), that takes (α, x…) to (β, 0…), as LAPACK's dlarfg for two or three entries: on return v[0] = β and v[1…] holds x's part of v.

Returns
τ, 0 where x is zero (the identity).

Definition at line 647 of file eigen.hpp.

◆ small_sincos()

template<std::floating_point R>
void numerical::detail::small_sincos ( R  x,
R &  sine,
R &  cosine 
)

sin and cos of x (sine_and_cosine), or x and 1 for one under the smallest normal value (an exact sine there).

Definition at line 163 of file complex_math.hpp.

◆ small_sylvester()

void numerical::detail::small_sylvester ( const one_based< double > &  tl,
const one_based< double > &  tr,
const one_based< double > &  b,
lapack_index  n1,
lapack_index  n2,
double(&)  x[4] 
)
inline

The solution X (n1 × n2, row-major in x) of TL X − X TR = B, n1 and n2 at most 2, by Gaussian elimination with complete pivoting on its Kronecker form, a pivot below ε times the largest coefficient taken as that, as LAPACK's dlasy2 (the blocks' entries near 1: no scaling against overflow).

Definition at line 1421 of file eigen.hpp.

◆ small_sylvester_block()

template<typename T >
void numerical::detail::small_sylvester_block ( const T *  r,
std::size_t  ldr,
const T *  s,
std::size_t  lds,
std::size_t  rows,
std::size_t  cols,
T *  c,
std::size_t  ldc,
double  smin 
)

The rows × cols y of R y + y S = c (each at most 2, row-major in c, overwritten) for the diagonal blocks r (row stride ldr) and s (row stride lds): the Kronecker system of order rows · cols by Gaussian elimination with complete pivoting, a pivot under smin taken as smin, as LAPACK's dlasy2 perturbs one (the eigenvalues of R and −S close).

Definition at line 734 of file matrix_functions.hpp.

◆ solve_are()

template<typename T >
matrix< T > numerical::detail::solve_are ( const matrix< T > &  a,
const matrix< T > &  b,
const matrix< T > &  q,
const matrix< T > &  r,
const matrix< T > *  e,
const matrix< T > *  s,
bool  balanced,
bool  continuous 
)

The checks of SciPy's _are_validate_args, then riccati on complex operands, the answer real for real ones.

Definition at line 3224 of file generalized.hpp.

◆ solve_bidiagonal_leaf()

void numerical::detail::solve_bidiagonal_leaf ( bidiagonal_node &  node)
inline

Decomposes the leaf node by implicit-shift QR.

Definition at line 1296 of file svd.hpp.

◆ solve_bidiagonal_subtree()

void numerical::detail::solve_bidiagonal_subtree ( std::vector< bidiagonal_node > &  nodes,
std::size_t  index,
arrow_workspace &  work 
)
inline

Decomposes the block index of nodes and everything below it, each block right after its halves.

Definition at line 1441 of file svd.hpp.

◆ solve_block()

template<typename S >
double numerical::detail::solve_block ( const S(&)  m[4],
S(&)  r[2],
double  floor 
)

Solves [a b; c d] x = r in place of r by Gaussian elimination with complete pivoting, a pivot below floor taken as floor, as LAPACK's dlaln2; r is scaled first, and.

Returns
the factor, wherever x would pass back_substitution_limit.

Definition at line 2689 of file eigen.hpp.

◆ solve_divide_leaf()

void numerical::detail::solve_divide_leaf ( const divide_node &  node)
inline

Solves the leaf node by implicit QL: its eigenvalues ascending into its diagonal, their eigenvectors as the columns of its block of q.

Definition at line 6060 of file factorizations.hpp.

◆ solve_divide_subtree()

void numerical::detail::solve_divide_subtree ( std::vector< divide_node > &  nodes,
std::size_t  index,
merge_workspace &  work 
)
inline

Solves the block index of nodes and everything below it, each block right after its halves.

Definition at line 6115 of file factorizations.hpp.

◆ solve_finite_triangular_in_place()

template<typename T >
void numerical::detail::solve_finite_triangular_in_place ( const T *  a,
std::size_t  n,
bool  lower,
triangular_op  op,
bool  unit_diagonal,
T *  x,
std::size_t  k 
)

solve_triangular_in_place of a triangle and a right-hand side known finite (require_finite_triangle, or a check that covered them): a zero on the diagonal still refused.

Definition at line 2042 of file factorizations.hpp.

◆ solve_secular()

template<typename Offset >
secular_solution numerical::detail::solve_secular ( const buffer< double > &  z,
double  rho,
std::size_t  i,
std::size_t  max_iterations,
double *  delta,
const Offset &  offset 
)

Root i of 1/ρ + Σⱼ zⱼ²/(Pⱼ − x) = 0, poles P strictly ascending, no zⱼ zero, ρ > 0, as LAPACK's dlaed4 and dlasd4 find it. The poles are given only as their offsets from one another, offset(o, j) = Pⱼ − P_o, which the caller computes accurately.

The root lies in (Pᵢ, Pᵢ₊₁), or for the last within 2ρ‖z‖² past its pole. It is sought as an offset τ from the nearer end P_o, so that every Pⱼ − x = offset(o, j) − τ is computed with a small relative error, which keeps the vectors built from them orthogonal. Each step fits the equation with its two nearest poles (Gragg's) and falls back to bisection when the fit leaves the bracket, which shrinks at every step.

Parameters
[in]zThe weights.
[in]rhoρ.
[in]iThe root.
[in]max_iterationsThe bound on the steps.
[out]deltaPⱼ − x for every j (size z.size()).
[in]offsetThe pole offsets.
Returns
The pole and the root's offset from it.
Exceptions
convergence_errorIf the root is not reached within max_iterations steps.

Definition at line 5615 of file factorizations.hpp.

◆ solve_shifted_tridiagonal()

void numerical::detail::solve_shifted_tridiagonal ( const shifted_tridiagonal &  f,
std::size_t  n,
double *  y,
double &  tol 
)
inlineprivate

Solves (T − λ I) x = y in place in y from factor_shifted_tridiagonal's factors, as LAPACK's dlagts (job −1): a pivot that would overflow the quotient, or is zero, moved away from zero by tol, then by twice as much, until it does not; tol, where 0, set first to ε times the factors' largest magnitude (ε itself if that is 0), for the next solves.

Definition at line 2608 of file svd.hpp.

◆ solve_triangular_in_place()

template<typename T >
void numerical::detail::solve_triangular_in_place ( const T *  a,
std::size_t  n,
bool  lower,
triangular_op  op,
bool  unit_diagonal,
T *  x,
std::size_t  k 
)

numerical::solve_triangular on the n × n row-major a, in place in x (n × k, row-major): the triangle checked, then read where it lies, transposed in place; the adjoint by the transpose on the conjugates (Aᴴ X = B is Aᵀ conj X = conj B).

Definition at line 2071 of file factorizations.hpp.

◆ solve_upper_blocks()

void numerical::detail::solve_upper_blocks ( const matrix< std::complex< double > > &  t,
const std::vector< std::size_t > &  starts,
buffer< std::complex< double > > &  x,
const std::complex< double > *  sources 
)
inline

Definition at line 1604 of file matrix_functions.hpp.

◆ solve_upward()

template<typename T , typename X >
void numerical::detail::solve_upward ( const buffer< T > &  t,
std::size_t  n,
std::size_t  k,
std::size_t  top,
typename X::scalar  lambda,
double  floor,
X &  x 
)

Solves (T − λ I) x = 0 upward from the component k, rows k − 1 … 0 of the row-major quasi-triangular t (its 2 × 2 blocks where a subdiagonal entry is nonzero), x [k … top] given: as LAPACK's dtrevc and ztrevc, a pivot below floor taken as floor, the components scaled down where they would grow past back_substitution_limit. The sums of rows against x are X 's, vectorised.

Definition at line 2895 of file eigen.hpp.

◆ sort_run() [1/4]

template<split_complex C>
void numerical::detail::sort_run ( C *  first,
C *  last 
)

Sorts [first, last) stably as sorts_before orders complex numbers.

Definition at line 3821 of file tensor_ops.hpp.

◆ sort_run() [2/4]

void numerical::detail::sort_run ( std::int64_t *  first,
std::int64_t *  last,
std::size_t  parts = 1 
)
inline

Sorts [first, last) as int64 sort: a long run by radix_sort in parts among the threads (equal integers are the same bits, so any order of them is the stable one).

Definition at line 3808 of file tensor_ops.hpp.

◆ sort_run() [3/4]

template<std::integral T>
void numerical::detail::sort_run ( T *  first,
T *  last 
)

Sorts [first, last) stably as integers sort.

Definition at line 3800 of file tensor_ops.hpp.

◆ sort_run() [4/4]

template<std::floating_point T>
void numerical::detail::sort_run ( T *  first,
T *  last,
std::size_t  parts = 1 
)

Sorts [first, last) stably as sorts_before orders reals: the NaNs moved behind the rest in their order first, the rest by <, which tests for no NaN. A long run by radix_sort, its zeros then rewritten with their signs in their order (the bits order −0 below +0, where < holds them equal); a short one by std::stable_sort with a lambda of its own, not the default std::less, which libc++ 22 sorts by the bits from 1024 doubles on, −0 first. The radix sort in parts among the threads.

Definition at line 3683 of file tensor_ops.hpp.

◆ sort_shared()

template<radix_orderable T>
void numerical::detail::sort_shared ( T *  run,
std::size_t  count,
std::size_t  threads 
)

Sorts one long line of count elements at run among threads: each pass of its radix sort shared out (radix_passes), the one-thread sort's bits with no merge.

Sorts one long line of any other type among threads: in pieces, a power of four, then merged (parallel_sort_run).

Definition at line 3876 of file tensor_ops.hpp.

◆ sorts_before() [1/2]

template<split_complex C>
bool numerical::detail::sorts_before ( const C &  a,
const C &  b 
)
noexcept

Whether a sorts before b as NumPy sorts complex numbers (npysort's CLT): those with no NaN by the real parts then the imaginary parts, then those with a NaN imaginary part alone by the real parts, then a NaN real part alone by the imaginary parts, then both NaN (R + Rj < R + nanj < nan + Rj < nan + nanj).

Definition at line 3456 of file tensor_ops.hpp.

◆ sorts_before() [2/2]

template<std::floating_point T>
bool numerical::detail::sorts_before ( T  a,
T  b 
)
noexcept

Whether a sorts before b as NumPy sorts reals: ascending, NaNs last.

Whether a sorts before b as integers sort: ascending.

Definition at line 3436 of file tensor_ops.hpp.

◆ source_of()

template<typename T >
fft_source< T > numerical::detail::source_of ( const tensor< T > &  a)

a's elements, where they lie.

Definition at line 2247 of file fft.hpp.

◆ spectral()

template<typename R , typename T >
matrix< R > numerical::detail::spectral ( const matrix< T > &  v,
const std::vector< R > &  f,
bool  hermitian 
)

V diag(f) Vᴴ in the arithmetic of R (double only for real v and f), the eigenvectors v orthonormal: one product. Made Hermitian to the bit where hermitian (every f real).

Definition at line 351 of file matrix_functions.hpp.

◆ spin_until()

template<typename Ready >
void numerical::detail::spin_until ( const Ready &  ready)

Polls ready, yielding the core between polls, until it holds or spin_time has passed.

Definition at line 321 of file parallel.hpp.

◆ split_by_parts()

bool numerical::detail::split_by_parts ( std::size_t  m)
inlinenoexcept

Whether gemm_split runs a product of m rows of A by four real products of each block's parts (split_parts_block), else by one of its expanded operands (split_expanded_block): the same bits, the parts' products faster where they read B's parts in place (gemm_packed). On the AMX path up to split_parts_rows; elsewhere (the NEON path) up to one row of the NEON tile, its b_in_place_rows: M1 Pro, all threads, by 400 × 500, 1 to 4 rows 2 to 18 % faster, 6 to 32 rows 5 to 52 % slower.

Definition at line 4163 of file gemm.hpp.

◆ split_expanded_block()

template<split_complex Z, typename B = b_rows, typename A = a_rows>
void numerical::detail::split_expanded_block ( std::size_t  m,
std::size_t  nb,
std::size_t  kb,
std::size_t  pc,
std::size_t  jc,
const Z *  a,
std::size_t  lda,
const Z *  b,
std::size_t  ldb,
typename Z::value_type *  ar,
typename Z::value_type *  br,
typename Z::value_type *  cr 
)

One block of gemm_split (split_parts_block's) by one real product of its expanded operands, [Re C | Im C] += [Re A | −Im A] · [[Re B, Im B], [Im B, −Re B]]: the same chains, B copied four times its size, the product as wide and as deep as the block's two.

Definition at line 4259 of file gemm.hpp.

◆ split_inputs()

std::vector< std::string > numerical::detail::split_inputs ( const std::string &  lhs)
inline

Splits an einsum left-hand side into per-operand subscripts.

Definition at line 1218 of file tensor.hpp.

◆ split_lu()

template<typename T >
std::pair< matrix< T >, matrix< T > > numerical::detail::split_lu ( const T *  lu,
std::size_t  rows,
std::size_t  cols,
std::size_t  k 
)

The rows of lu (rows × cols row-major, L below its diagonal, U on and above) as L, its first k columns (unit diagonal), and U, its first k rows.

Definition at line 2400 of file factorizations.hpp.

◆ split_parts_block()

template<split_complex Z, typename B = b_rows, typename A = a_rows>
void numerical::detail::split_parts_block ( std::size_t  m,
std::size_t  nb,
std::size_t  kb,
std::size_t  pc,
std::size_t  jc,
const Z *  a,
std::size_t  lda,
const Z *  b,
std::size_t  ldb,
typename Z::value_type *  ar,
typename Z::value_type *  br,
typename Z::value_type *  cr 
)

One block of gemm_split ([Re C | Im C] at cr, m × nb each, += A·B, A m × kb and B kb × nb) by four real products of its parts: Re C += Re A·Re B, then −Im A·Im B; Im C += Re A·Im B, then Im A·Re B, each continuing its elements' chains ((−x)·(−y) is x·y exactly: the bits of −Im A·−Re B). B's parts are a copy of its size, which a product of a few rows of A reads in place. The block is A's columns and B's rows from pc, B's columns from jc, A and B laid out as A and B say.

Definition at line 4214 of file gemm.hpp.

◆ split_product()

template<std::floating_point R>
std::complex< R > numerical::detail::split_product ( const std::complex< R > &  p,
const std::complex< R > &  q 
)
inlinenoexcept

0 + p·q of complex numbers as arm64's real kernel rounds it (complex_term::split), a product of [Re p | −Im p] by [[Re q, Im q], [Im q, −Re q]]: each part's first product from zero, the second fused into it.

Definition at line 1885 of file tensor.hpp.

◆ split_row_columns()

template<split_complex C>
void numerical::detail::split_row_columns ( const C *  a,
const C *  b,
std::size_t  row_step,
std::size_t  column_step,
std::size_t  inner,
std::size_t  first,
std::size_t  last,
C *  y,
bool  kept = false 
)

Columns first to last of the complex vector a (inner) times B (B[k][j] at b[(k · row_step) + (j · column_step)]) into y, each element summed as gemm_split sums it (complex_term::split): by blocks of split_depth of the depth, Re a · Re B then −Im a · Im B into its real part, Re a · Im B then Im a · Re B into its imaginary part, each one chain of fused multiply-adds from +0 through the blocks, or from y's own where kept (y += a · B, as gemm_split starts from C).

Definition at line 3895 of file gemm.hpp.

◆ split_storage()

template<typename R >
split_operands< R > & numerical::detail::split_storage ( )

The calling thread's split_operands. A product uses them whole on one thread, and the real product it runs on them nests no complex one.

Definition at line 4197 of file gemm.hpp.

◆ square()

template<typename T >
T numerical::detail::square ( const T &  x)

x · x, by the type's own product (a complex one's C Annex G, as the library's *).

Definition at line 1557 of file tensor_ops.hpp.

◆ square_in_place()

template<typename T >
void numerical::detail::square_in_place ( buffer< T > &  x,
buffer< T > &  spare,
std::size_t  n 
)

x squared count times, the products alternating between its storage and spare's (resized as needed): no fresh n × n mapped and faulted in per squaring.

Definition at line 136 of file matrix_functions.hpp.

◆ square_product()

template<typename T >
buffer< T > numerical::detail::square_product ( const buffer< T > &  a,
const buffer< T > &  b,
std::size_t  n 
)

The n × n product A B of row-major operands (square_product_into, into storage of its own).

Definition at line 124 of file matrix_functions.hpp.

◆ square_product_into()

template<typename T >
void numerical::detail::square_product_into ( const buffer< T > &  a,
const buffer< T > &  b,
std::size_t  n,
buffer< T > &  c 
)

The n × n product A B of row-major operands into c, whatever it held (its elements not read, as many made as needed): C = A·B, each part of it zeroed by the thread that computes it (c_start::zero), where a zeroed buffer was filled on the calling thread first. c is neither operand.

Definition at line 113 of file matrix_functions.hpp.

◆ square_root_columns()

template<typename T >
void numerical::detail::square_root_columns ( const buffer< T > &  t,
buffer< T > &  r,
std::size_t  n,
const std::vector< std::size_t > &  starts 
)

The off-diagonal blocks of the square root r (n × n, row-major) of the upper (quasi-)triangular t, its diagonal blocks already in r, column of blocks by column, each block row from the last: R_ii X + X R_jj = T_ij − Σ R_ik R_kj (Higham, 1987), between two scalars x = c / (r_ii + r_jj), a zero over a zero taken as 0 and a nonzero over a zero as infinite (no root there, as SciPy answers), else the small Sylvester system of the blocks (small_sylvester_block; principal roots of eigenvalues off the negative axis never sum to 0 there).

Definition at line 1218 of file matrix_functions.hpp.

◆ square_singular_values()

template<typename T >
buffer< double > numerical::detail::square_singular_values ( buffer< T >  b,
std::size_t  n,
qd_index  sweeps = dqds_sweeps 
)
private

Definition at line 2166 of file svd.hpp.

◆ stabilising_solution()

template<typename T >
matrix< T > numerical::detail::stabilising_solution ( const matrix< T > &  top,
const matrix< T > &  bottom,
bool  continuous 
)
private

X = U₂₁ U₁₁⁻¹ from the m leading Schur vectors' halves top (U₁₁) and bottom (U₂₁) of a Riccati equation's Hamiltonian (continuous) or symplectic matrix or pencil, as SciPy's checks have it.

Exceptions
singular_matrix_errorIf U₁₁ is numerically singular (the condition of its LU factor U past 1/ε, as SciPy's): no finite stabilising solution.
convergence_errorIf U₁₁ᴴ U₂₁ is not Hermitian: the eigenvalues too close to the imaginary axis (the unit circle).

Definition at line 2781 of file generalized.hpp.

◆ stable_choice()

template<std::floating_point T>
T numerical::detail::stable_choice ( const T *  row,
std::size_t  count,
std::size_t  k,
T  chosen 
)

Of the count elements at row, the one a stable sort (sort_run) puts at k, from the one chosen that a selection put there, equal to it under sorts_before. A zero (either sign) or a NaN (any payload) is not alone in its class, and the class's member of the same rank in row's order is the stable sort's.

An integer's class under < is itself alone.

Definition at line 4670 of file tensor_ops.hpp.

◆ stacked_matmul()

template<typename T >
tensor< T > numerical::detail::stacked_matmul ( const tensor< T > &  a,
const tensor< T > &  b 
)

numerical::matmul of a type whose sums and products are defined past its range: a float, a complex, an unsigned integer.

Definition at line 2788 of file tensor.hpp.

◆ standardize()

std::pair< std::complex< double >, std::complex< double > > numerical::detail::standardize ( qz_form &  f,
std::size_t  j 
)
inline

T(j, j) made real and non-negative, as zhgeqz standardises it: row j of T and S above the diagonal and column j of S, T and Z taking the phase.

Returns
(α, β), the eigenvalue α/β.

Definition at line 155 of file generalized.hpp.

◆ standardize_at()

void numerical::detail::standardize_at ( one_based< double >  t,
one_based< double >  qt,
lapack_index  n,
lapack_index  j 
)
inline

The standard form of the 2 × 2 block at j of the quasi-triangular t, its rotation carried to the rest of T and to the rows of qt, as dlaexc ends.

Definition at line 1494 of file eigen.hpp.

◆ standardize_block()

standard_block numerical::detail::standardize_block ( double &  a,
double &  b,
double &  c,
double &  d 
)
inline

The Schur factorisation of the real 2 × 2 block [a b; c d], as LAPACK's dlanv2: [a b; c d] ← [cs −sn; sn cs]ᵀ [a b; c d] [cs −sn; sn cs], upper triangular where its eigenvalues are real, else with equal diagonal entries and b c < 0.

Definition at line 710 of file eigen.hpp.

◆ standardize_pencil()

pencil_rotations numerical::detail::standardize_pencil ( std::array< std::array< double, 2 >, 2 > &  a,
std::array< std::array< double, 2 >, 2 > &  b 
)
inline

The 2 × 2 pencil (a, b), b upper triangular, standardized, as LAPACK's dlagv2: two real eigenvalues made triangular, a complex pair left with b diagonal and positive; both changed in place.

Definition at line 1565 of file generalized.hpp.

◆ start_bidiagonal_node()

void numerical::detail::start_bidiagonal_node ( bidiagonal_node &  node)
inline

Its singular values and vectors into node's, from scratch: u zero, v the identity.

Definition at line 1284 of file svd.hpp.

◆ start_vector()

void numerical::detail::start_vector ( std::size_t  index,
double *  x,
std::size_t  n 
)
inlineprivatenoexcept

A start vector of n entries in [−1, 1) for tridiagonal_eigenvectors' eigenvalue index, its own draw (splitmix64 from the index), so that each vector is the same on any number of threads.

Definition at line 2663 of file svd.hpp.

◆ starting_points()

std::vector< std::size_t > numerical::detail::starting_points ( const cluster_circles &  circles)
inline

The points to start each circle of circles about a cluster of two or more with: contour_points, funm doubling those its integral's estimate asks more of.

Definition at line 2059 of file matrix_functions.hpp.

◆ stockham_passes()

template<std::floating_point R, typename C , typename Four , typename Sixteen , typename Other >
void numerical::detail::stockham_passes ( const fft_plan< R > &  plan,
C *  data,
C *  work,
Four  four,
Sixteen  sixteen,
Other  other,
bool  from_work = false 
)
inlinenoexcept

The Stockham passes of the plan over data, work as long, the result in data: radix 4 by four (radix_four_pass's arguments), 16 by sixteen (radix_sixteen_pass's), 2 by radix_two_pass (always the last pass), any other by other (any_radix_pass's: 3 and 5 by their own butterflies). The input in data, or from_work in work (fft_starts_in_work: an odd number of passes then ends in data, where from data the last pass wrote work, copied back).

Definition at line 1671 of file fft.hpp.

◆ strided_column_group()

template<std::size_t Width, bool Fused = false, typename T >
void numerical::detail::strided_column_group ( const T *  a,
const T *  b,
std::size_t  row_step,
std::size_t  column_step,
std::size_t  inner,
std::size_t  j,
T *  y 
)

Columns j to j + Width of vector_times_columns into y, their sums held through the rows: each term one fused multiply-add where Fused (as add_product adds each row on the arm64 fused paths), else add_one_product.

Definition at line 1296 of file tensor.hpp.

◆ sturm_count()

std::size_t numerical::detail::sturm_count ( const double *  d,
const double *  e2,
std::size_t  n,
double  x,
double  pivmin 
)
inlineprivate

The number of eigenvalues below x of the symmetric tridiagonal of diagonal d and squared off-diagonal e2 (n and n − 1 entries), by Sturm's sequence: the negative pivots of T − x I, a pivot smaller than pivmin taken as −pivmin, as LAPACK's dlaebz.

Definition at line 2429 of file svd.hpp.

◆ subtract_combined_vectors()

template<typename T >
std::size_t numerical::detail::subtract_combined_vectors ( T *  ,
const T *  ,
const T *  ,
std::size_t  ,
std::size_t  ,
std::size_t   
)

No leading blocks in vectors, for a type no native vector holds or a compiler that vectorises the scalar block itself (Clang).

Definition at line 1011 of file simd.hpp.

◆ subtract_corrections() [1/2]

template<typename T >
requires std::is_floating_point_v<T>
void numerical::detail::subtract_corrections ( T *  y,
const T *  f,
const T *  x,
std::size_t  ldx,
std::size_t  terms,
std::size_t  count 
)

subtract_products of a real T, where the reduction's corrections are those of its type.

Definition at line 4585 of file factorizations.hpp.

◆ subtract_corrections() [2/2]

template<split_complex T>
void numerical::detail::subtract_corrections ( T *  y,
const T *  f,
const T *  x,
std::size_t  ldx,
std::size_t  terms,
std::size_t  count 
)

subtract_products of complex numbers, each term y[i] −= f · x[i] spelled out as the complex operator's statements (four part products rounded apart, ac − bd and ad + bc, then their differences with y): the same bits for finite parts whose products stay finite, where the operator's rescue of a product whose parts are both NaN never acts, and loops the compiler vectorises (the reduction's rows, scaled by their largest magnitude, are such). Complex double on the x86-64 units keeps its fused kernel.

Definition at line 4603 of file factorizations.hpp.

◆ subtract_panel_columns() [1/2]

void numerical::detail::subtract_panel_columns ( double *  column,
const double *  columns,
std::size_t  height,
const double *  l,
std::size_t  terms 
)
inline

subtract_panel_columns for double: subtract_products, which holds the column in registers through the terms, the same multiply-adds in the same order, so the same values (a complex one keeps the term-by-term calls, whose x · f the other would round as f · x).

Definition at line 770 of file factorizations.hpp.

◆ subtract_panel_columns() [2/2]

template<typename T >
void numerical::detail::subtract_panel_columns ( T *  column,
const T *  columns,
std::size_t  height,
const T *  l,
std::size_t  terms 
)

column −= Σₜ x_t · conj(l[t]) over the terms columns columns (height elements each, one after another), term by term in ascending order: an elimination step of a Cholesky panel's rows below (an overload per scalar, as conj_of). Each term subtract_product, the column read and written again for every one.

Definition at line 756 of file factorizations.hpp.

◆ subtract_product() [1/6]

void numerical::detail::subtract_product ( double *  y,
const double &  f,
const double *  x,
std::size_t  count 
)
inline

y[i] −= f · x[i] for double on the widest x86-64 units the matrix product's kernel may use (gemm_path_current), each element one fused multiply-add there; elsewhere subtract_product's template. The decompositions' row updates take it: a wheel built for the baseline x86-64 holds two double per vector and fuses none.

Definition at line 485 of file gemm.hpp.

◆ subtract_product() [2/6]

void numerical::detail::subtract_product ( double *  y,
const double *  x,
const double &  f,
std::size_t  count 
)
inline

y[i] −= x[i] · f for double, as the other (a real product is the same either way round).

Definition at line 512 of file gemm.hpp.

◆ subtract_product() [3/6]

void numerical::detail::subtract_product ( std::complex< double > *  y,
const std::complex< double > &  f,
const std::complex< double > *  x,
std::size_t  count 
)
inline

y[i] −= f · x[i] for std::complex<double>: one term of subtract_products on AVX-512 or AVX2 where the matrix product's kernel runs there, else the template.

Definition at line 614 of file gemm.hpp.

◆ subtract_product() [4/6]

void numerical::detail::subtract_product ( std::complex< double > *  y,
const std::complex< double > *  x,
const std::complex< double > &  f,
std::size_t  count 
)
inline

y[i] −= x[i] · f for std::complex<double>, as the other (the template keeps the expression's order).

Definition at line 631 of file gemm.hpp.

◆ subtract_product() [5/6]

template<typename T >
void numerical::detail::subtract_product ( T *  y,
const T &  f,
const T *  x,
std::size_t  count 
)

y[i] -= f · x[i] for count elements of y and x, which do not overlap. Each element is computed exactly as that scalar statement computes it (its product rounded before its difference, or fused with it where the compiler contracts the statement); the arguments follow the expression, since a compiler may contract f · x and x · f of complex numbers differently. Under GCC, over native vectors where T has them: at -O2 GCC leaves the scalar loop scalar, unable to rule out an overlap without a check it will not add. Clang vectorises the scalar loop itself, with that check, and better than these vectors (an LU solve of order 128 on arm64 took 19 % longer through them). For double, gemm.hpp's overload chooses the x86-64 units at run time, a fused multiply-add per element there.

Definition at line 1159 of file simd.hpp.

◆ subtract_product() [6/6]

template<typename T >
void numerical::detail::subtract_product ( T *  y,
const T *  x,
const T &  f,
std::size_t  count 
)

y[i] -= x[i] · f, as the other subtract_product (a real product is the same either way round).

Definition at line 1299 of file simd.hpp.

◆ subtract_products() [1/4]

void numerical::detail::subtract_products ( double *  y,
const double *  f,
const double *  x,
std::size_t  ldx,
std::size_t  terms,
std::size_t  count 
)
inline

y[i] -= Σₜ f[t] · x[(t · ldx) + i] for double, the units chosen as subtract_product chooses them, the elements held in registers through the terms there.

Definition at line 522 of file gemm.hpp.

◆ subtract_products() [2/4]

void numerical::detail::subtract_products ( std::complex< double > *  y,
const std::complex< double > *  f,
const std::complex< double > *  x,
std::size_t  ldx,
std::size_t  terms,
std::size_t  count 
)
inline

subtract_products for std::complex<double>: on AVX-512 or AVX2 with FMA where the matrix product's kernel runs there, each part of each term one fused multiply-add (Re f · x, then Im f times the swapped parts), the elements in registers through the terms; elsewhere the template, whole complex products as the compiler rounds them.

Definition at line 586 of file gemm.hpp.

◆ subtract_products() [3/4]

template<typename T >
void numerical::detail::subtract_products ( T *  y,
const T *  f,
const T *  x,
std::size_t  ldx,
std::size_t  terms,
std::size_t  count 
)

y[i] -= Σₜ f[t] · x[(t · ldx) + i] for count elements of y, terms rows of x apart by ldx: subtract_product for each term in ascending order, so each element is computed as that sequence of calls computes it. For double, gemm.hpp's overload holds the elements in registers through the terms.

Definition at line 1198 of file simd.hpp.

◆ subtract_products() [4/4]

template<typename T >
requires std::is_floating_point_v<T>
void numerical::detail::subtract_products ( T *  y,
const T *  f,
const T *  x,
std::size_t  ldx,
std::size_t  terms,
std::size_t  count 
)

subtract_products for a real T: eight elements of y held in a local array through the terms, which the compiler keeps in registers, where each term was a call through the whole row. Each element takes its terms in the same order as a sequence of subtract_product gives them (an LU solve of order 64 and four columns on arm64: 18 → 8.5 µs), and the same value under Clang, and under GCC on x86-64, which fuses neither; under GCC on arm64 the vectors of subtract_combined_vectors and subtract_scaled_vectors round apart and the scalar statements fused, and their splits differ.

Definition at line 1220 of file simd.hpp.

◆ subtract_rows() [1/2]

void numerical::detail::subtract_rows ( double *  y,
std::size_t  ldy,
const double *  f,
std::size_t  ldf,
const double *  x,
std::size_t  ldx,
std::size_t  terms,
std::size_t  count 
)
inline

subtract_rows for double, the units chosen as subtract_products chooses them: each element the value subtract_products gives it.

Definition at line 551 of file gemm.hpp.

◆ subtract_rows() [2/2]

template<typename T >
requires std::is_floating_point_v<T>
void numerical::detail::subtract_rows ( T *  y,
std::size_t  ldy,
const T *  f,
std::size_t  ldf,
const T *  x,
std::size_t  ldx,
std::size_t  terms,
std::size_t  count 
)

subtract_products for product_rows rows of y at once, ldy apart, each with its own factors (ldf apart in f) on the same rows of x: y_r[i] -= Σₜ f_r[t] · x[(t · ldx) + i]. Each element takes the statement of subtract_products, a term at a time in ascending order, so the same value; each element of x read serves every row, and the rows' chains of multiply-adds advance together, where a row alone waits on its last one through each term.

Definition at line 1262 of file simd.hpp.

◆ subtract_rows_vectors()

template<typename T >
std::size_t numerical::detail::subtract_rows_vectors ( T *  ,
std::size_t  ,
const T *  ,
std::size_t  ,
const T *  ,
std::size_t  ,
std::size_t  ,
std::size_t   
)

No leading columns in vectors, for a type no native vector holds or a compiler that vectorises the scalar block itself (Clang).

Definition at line 1024 of file simd.hpp.

◆ subtract_scaled_vectors()

template<typename T >
std::size_t numerical::detail::subtract_scaled_vectors ( T *  ,
const T *  ,
const T &  ,
std::size_t   
)

No leading elements in vectors, for a type no native vector holds.

Definition at line 989 of file simd.hpp.

◆ subtract_term() [1/2]

template<typename R >
void numerical::detail::subtract_term ( std::complex< R > &  sum,
const std::complex< R > &  a,
const std::complex< R > &  b 
)

subtract_term of complex numbers: the product in line (operator_times, the operator's rounding).

Definition at line 136 of file factorizations.hpp.

◆ subtract_term() [2/2]

template<typename T >
void numerical::detail::subtract_term ( T &  sum,
const T &  a,
const T &  b 
)

sum −= a · b in one statement, which the compiler may fuse (as it fuses the matrix solve's).

Definition at line 127 of file factorizations.hpp.

◆ sum_over_run()

template<typename T >
std::optional< tensor< T > > numerical::detail::sum_over_run ( const std::string &  in,
const std::string &  output,
const std::vector< std::size_t > &  shape,
const T *  data 
)

einsum of one row-major operand data (shape, its axes labelled in, each label once) into the labels output, where those it sums are one run of its axes and output the others in order (ij->i, ij->, ij->j, ijk->ik): each output element the sum of evenly spaced elements, by pairwise halving as sum() adds them (pairwise_sums), where einsum's general loop added them one after another. None for any other spec (nothing summed, the summed labels apart, the output reordered).

Definition at line 1720 of file tensor.hpp.

◆ svd_of_triangle()

triangle_svd numerical::detail::svd_of_triangle ( double  f,
double  g,
double  h 
)
inline

The SVD of the upper triangular [f g; 0 h], as LAPACK's dlasv2: [csl snl; −snl csl] [f g; 0 h] [csr −snr; snr csr] = diag(ssmax, ssmin), |ssmax| ≥ |ssmin|, each accurate to a few ulps.

Definition at line 557 of file generalized.hpp.

◆ symmetric_eigen_2x2()

symmetric_two_by_two numerical::detail::symmetric_eigen_2x2 ( double  a,
double  b,
double  c 
)
inlinenoexcept

The eigensystem of [a b; b c] as LAPACK's dlaev2 computes it: the larger eigenvalue in magnitude from the sum and the root of the difference, the smaller from the determinant (no cancellation), and the rotation (c, s) whose first row is the larger's eigenvector.

Definition at line 5165 of file factorizations.hpp.

◆ symmetric_rank_one()

template<typename T >
void numerical::detail::symmetric_rank_one ( buffer< T > &  a,
std::size_t  n,
std::size_t  k,
buffer< T > &  w 
)

The trailing matrix's update by a pivot of order 1 at k, as sytf2 (syr, then L's column scaled): A ← A − x xᵀ / d on and below the diagonal, x / d into column k; the arithmetic of T. Row by row (a is row-major), each entry's one operation as syr's column by column: the factors −x_j / d first, in w (n entries).

Definition at line 88 of file ldl.hpp.

◆ symmetric_rank_two()

template<typename T >
void numerical::detail::symmetric_rank_two ( buffer< T > &  a,
std::size_t  n,
std::size_t  k,
buffer< T > &  w 
)

The trailing matrix's update by a pivot of order 2 at rows k and k + 1, as sytf2: D21 scaled out, W = [x y] D⁻¹ from D11 / D21, D22 / D21 and t = 1 / (D11 D22 / D21² − 1), A ← A − [x y] Wᵀ, W into the two columns; the arithmetic of T. Row by row, W first, in w (2n entries), each row's x and y read before its two entries of W replace them.

Definition at line 137 of file ldl.hpp.

◆ symmetric_row()

template<typename T >
T numerical::detail::symmetric_row ( const T *  row,
const T *  v,
const T &  f,
T *  out,
std::size_t  count 
)

Σ row[i] · v[i] and out[i] += row[i] · f, in order, for a type no native vector holds.

Definition at line 1048 of file simd.hpp.

◆ symmetric_row_speedup()

template<typename T >
std::size_t numerical::detail::symmetric_row_speedup ( )

The weight of a symmetric row's multiply-adds in hermitian_product: on the x86-64 units, where its rows are paced by one core's loads, vector_speedup, as more threads pay than their multiply-adds alone would pay for (an i9-9900K's reduction of order 512 took 17 % less, an i9-10900X's 7 %); 1 elsewhere (arm64's, weighed so, took as long or longer).

Definition at line 4487 of file factorizations.hpp.

◆ symmetric_rows()

template<typename T >
std::array< T, symmetric_row_group > numerical::detail::symmetric_rows ( const T *  rows,
std::size_t  n,
const T *  v,
T *  out,
std::size_t  i 
)

symmetric_rows one column at a time, for a type no native vector holds.

Definition at line 1065 of file simd.hpp.

◆ symmetric_rows_tail()

template<typename T >
std::array< T, symmetric_row_group > numerical::detail::symmetric_rows_tail ( const T *  rows,
std::size_t  n,
const T *  v,
T *  out,
std::size_t  i,
std::size_t  done,
std::array< T, symmetric_row_group >  sums 
)

The columns done…i+r of symmetric_rows, one by one: for each row r in order, out[k] += row_r[k] · v[i+r] for done ≤ k < i + r and row_r[k] · v[k] added to sums[r], its diagonal term last.

Definition at line 165 of file simd.hpp.

◆ table_atan()

double numerical::detail::table_atan ( double  x)
inlinenoexcept

atan(x): y = |x|, or 1/|x| past 1 (its rounding error carried), c = k/64 the nearest point of atan_table, t = (y − c)/(1 + y c) as a pair, |t| ≤ 2^−7, atan(t) − t by its Taylor series to t^11; atan(c) + t summed exactly, π/2 less that past 1, the small rest added once. Within 0.501 ulp of the exact value; x itself where |x| < 2^−27 (±0 kept) and for NaN, ±π/2 past 2^54. Every product beside a sum fused explicitly, as table_expm1's.

Definition at line 709 of file elementary.hpp.

◆ table_cosh()

double numerical::detail::table_cosh ( double  x)
inlinenoexcept

cosh(x) = (d + u)/2 (hyperbolic_parts): within 0.502 ulp of the exact value up to the overflow; NaN for NaN.

Definition at line 381 of file elementary.hpp.

◆ table_exp()

double numerical::detail::table_exp ( double  x)
inlinenoexcept

exp(x) for an x whose result is normal or infinite (exp_normal_bound or above): x = (128 e + j) ln 2/128 + r, |r| ≤ ln 2/256, and exp(x) = 2^e · 2^(j/128) · exp(r), exp(r) − 1 by its Taylor polynomial of degree 6. Every multiply-add std::fma, written out, so that no compiler fuses otherwise; inlined into the loop that calls it (exp_into). Below the bound, its answer is not exp's; for NaN, NaN.

Definition at line 195 of file elementary.hpp.

◆ table_exp2()

double numerical::detail::table_exp2 ( double  x)
inlinenoexcept

2^x for an x whose result is normal or infinite (−1022 or above): x = (128 e + j)/128 + f, |f| ≤ 1/256, each step exact (the scaling by 128 and f's difference), and 2^x = 2^e · 2^(j/128) · exp(f ln 2), as in table_exp. Within 0.506 ulp of the exact value, and exact at every integer (f = 0, j = 0). Below the bound, its answer is not 2^x's; for NaN, NaN.

Definition at line 404 of file elementary.hpp.

◆ table_expm1()

double numerical::detail::table_expm1 ( double  x)
inlinenoexcept

expm1(x) by table_exp's reduction, x = (128 e + j) ln 2/128 + r: in units of 2^e, 2^(j/128) − 2^−e

  • 2^(j/128) r_head, each sum and product of them exact (two-sums and a fused product's error), plus the small rest, 2^(j/128) (r − r_head + exp(r) − 1 − r) and the table's tail times exp(r), then scaled by 2^e: the cancellation near 0 and ±ln 2 leaves no rounding but the last. Within 0.503 ulp of the exact value; below −40, −1 (exp(x) under half an ulp of it); x itself where |x| < 2^−54 (±0 kept) and for NaN.

Definition at line 284 of file elementary.hpp.

◆ table_log1p()

double numerical::detail::table_log1p ( double  x)
inlinenoexcept

ln(1 + x) by log_parts of 1 + x as a pair (its two-sum): near 0, where c = 1, r is x exactly. Within 0.501 ulp of the exact value; x itself where |x| < 2^−54 (±0 kept) and for NaN, −∞ at −1, +∞ at +∞, NaN below −1.

Definition at line 945 of file elementary.hpp.

◆ table_sine()

template<bool Cosine>
double numerical::detail::table_sine ( double  x)
inlinenoexcept

sin(x), or cos(x) where Cosine, of a double within half an ulp and a few thousandths, for |x| ≤ sine_reduction_bound (another x gives a value of no use): j the integer nearest x·32/π (its low six bits the table's entry), r = x − jπ/32 by sine_step_parts as a pair r_hi + r_lo (|r| ≤ π/64); sin r − r and cos r − 1 by their Taylor series to r¹¹ and r¹⁰; then sin x = S + (C·r + S·(cos r − 1) + C·(sin r − r)), cos x = C + (−S·r + …) for S, C the table's sin(jπ/32) and cos(jπ/32), the first sum exact (a two-sum and a two-product), the small rest added once. x itself where |x| < 2⁻²⁷ (±0 kept), 1 for the cosine. Every product beside a sum fused explicitly and every other operation its own, so the bits are the same in vectors and under every compiler with the FMA instructions. Within 0.502 ulp of the exact value on 80 000 points to 2²⁰, a third of them near multiples of π/32 (the C library's of macOS: 0.86).

Definition at line 667 of file elementary.hpp.

◆ table_sinh()

double numerical::detail::table_sinh ( double  x)
inlinenoexcept

sinh(x) = (d − u)/2 (hyperbolic_parts), the sign of x: within 0.503 ulp of the exact value up to the overflow (2^(e − 1) a normal double, where e^x alone would overflow from 709.78). x itself where |x| < 2^−27 (sinh(x) − x under half an ulp of it, ±0 kept) and for NaN.

Definition at line 362 of file elementary.hpp.

◆ table_tanh()

double numerical::detail::table_tanh ( double  x)
inlinenoexcept

tanh(x) = t / (t + 2), t = expm1(2|x|) as a pair (expm1_parts, its two parts summed exactly), the sum t + 2 a pair too, the quotient by the reciprocal of its head, corrected by its exact residual: within 0.502 ulp of the exact value. Past 22, ±1 (1 − tanh under half an ulp of it); x itself where |x| < 2^−27 (tanh(x) − x under half an ulp of it, ±0 kept) and for NaN.

Definition at line 305 of file elementary.hpp.

◆ taken_as() [1/2]

double numerical::detail::taken_as ( const std::complex< double > &  z,
std::type_identity< double >   
)
inline

A complex value as a real result takes it: its real part (the imaginary part rounding's alone).

Definition at line 393 of file banded.hpp.

◆ taken_as() [2/2]

std::complex< double > numerical::detail::taken_as ( const std::complex< double > &  z,
std::type_identity< std::complex< double > >   
)
inline

A complex value as a complex result takes it: whole.

Definition at line 400 of file banded.hpp.

◆ tall_lu_columns()

template<typename T >
void numerical::detail::tall_lu_columns ( T *  panel,
std::size_t  m,
std::size_t  n,
std::size_t  first,
std::size_t  last,
std::vector< std::size_t > &  rows 
)

Factorises the columns [first, last) of the m × n column-major panel (m ≥ n; column j at panel[j m]) by partial pivoting, in place, as LAPACK's dgetrf2: the left half factorised, the top of the right half solved against its unit-lower L, the bottom updated by one product, then the right half; a leaf of solve_leaf columns, or a panel too short for its products to pay, unblocked. Each pivot's row swap reaches every column and rows (row i of the factors is row rows[i] of the input); a zero pivot column is left as it is.

Definition at line 2429 of file factorizations.hpp.

◆ tall_singular_values()

template<typename T >
buffer< double > numerical::detail::tall_singular_values ( buffer< T >  b,
std::size_t  rows,
std::size_t  n,
qd_index  sweeps = dqds_sweeps 
)
private

The singular values of the square n × n row-major b, descending, without its vectors: those of its Golub–Kahan bidiagonal by dqds (dqds_singular_values), no basis formed; by the implicit-shift QR iteration (bidiagonal_qr) where dqds does not converge within sweeps steps per value.

Exceptions
convergence_errorIf neither converges within its bound (not observed).

The singular values of the rows × n row-major b (rows ≥ n), descending, as square_singular_values gives them for a square one; a tall one's Golub–Kahan bidiagonal reduced from b itself (bidiagonalize_unblocked), where a QR factorisation first costs about the bidiagonalisation's own time; for an order under bidiagonal_crossover and fewer than 1.25 times as many rows (LAPACK's dgesdd, 1.6: here a complex 100 × 159 took 1.27 times as long, a real one 1.06). One M1 Pro core, 39 × 40 80.5 → 57.3 µs (NumPy 51), complex 221 → 150 µs; 50 × 60 136 → 109 µs; 120 × 127 913 → 794 µs.

Definition at line 2144 of file svd.hpp.

◆ tanh_into()

void numerical::detail::tanh_into ( const double *  in,
double *  out,
std::size_t  count 
)
inline

tanh of the count doubles at in into out, each elementary_tanh's: table_tanh in vectors (with the FMA instructions on an x86-64 processor, run_fused), std::tanh throughout where exp_by_table says no.

Definition at line 1221 of file tensor_ops.hpp.

◆ tanh_of() [1/3]

tensor< double > numerical::detail::tanh_of ( const double *  data,
const std::vector< std::size_t > &  shape 
)
inline

tanh of every one of the row-major doubles at data, of shape shape: tanh_into by chunks of elementwise_chunk among the threads.

Definition at line 1258 of file tensor_ops.hpp.

◆ tanh_of() [2/3]

tensor< float > numerical::detail::tanh_of ( const float *  data,
const std::vector< std::size_t > &  shape 
)
inline

tanh of every one of the row-major floats at data, of shape shape: tanh_into of them as doubles (through_doubles).

Definition at line 1279 of file tensor_ops.hpp.

◆ tanh_of() [3/3]

template<typename T >
tensor< T > numerical::detail::tanh_of ( const T *  data,
const std::vector< std::size_t > &  shape 
)

tanh of every one of the row-major scalars at data, of shape shape: elementary_tanh of each.

Definition at line 1250 of file tensor_ops.hpp.

◆ tanh_viewed() [1/3]

tensor< double > numerical::detail::tanh_viewed ( const tensor< double > &  a)
inline

tanh of every double of a, a view read where it lies (by_runs).

Definition at line 2637 of file tensor_ops.hpp.

◆ tanh_viewed() [2/3]

tensor< float > numerical::detail::tanh_viewed ( const tensor< float > &  a)
inline

tanh of every float of a through doubles, a view read where it lies (by_runs).

Definition at line 2643 of file tensor_ops.hpp.

◆ tanh_viewed() [3/3]

template<typename T >
tensor< T > numerical::detail::tanh_viewed ( const tensor< T > &  a)

tanh of every element of a: of a contiguous copy (tanh_of).

Definition at line 2631 of file tensor_ops.hpp.

◆ task_rows_cap() [1/2]

template<typename Kernel >
requires requires(std::size_t kc) { Kernel::task_rows_for(kc); }
constexpr std::size_t numerical::detail::task_rows_cap ( std::size_t  kc)
constexprnoexcept

task_rows_cap for a tile whose task rows depend on the panel's depth (Kernel::task_rows_for).

Definition at line 2565 of file gemm.hpp.

◆ task_rows_cap() [2/2]

template<typename Kernel >
constexpr std::size_t numerical::detail::task_rows_cap ( std::size_t  )
constexprnoexcept

The most rows of a task of Kernel over a panel of depth kc: its block of rows.

Definition at line 2553 of file gemm.hpp.

◆ thread_arrow_workspace()

arrow_workspace & numerical::detail::thread_arrow_workspace ( )
inline

The calling thread's arrow_workspace, kept from one decomposition to the next. A join uses it whole on one thread and nests no other join; release_large_arrow_workspace lets a large buffer go after.

Definition at line 947 of file svd.hpp.

◆ thread_cap()

std::atomic< std::size_t > & numerical::detail::thread_cap ( )
inlinenoexcept

The most threads a parallel_for uses (0: as many as the pool has), set by numerical::set_max_threads.

Definition at line 720 of file parallel.hpp.

◆ thread_releases()

thread_release_steps & numerical::detail::thread_releases ( )
inlinenoexcept

The calling thread's thread_release_steps.

Definition at line 56 of file buffer.hpp.

◆ thread_workspace()

merge_workspace & numerical::detail::thread_workspace ( )
inline

The calling thread's merge_workspace, kept from one decomposition to the next: mapping its pages anew took 8–11 % of the eigenvectors' time (six AVX2 cores, an eigh of order 256 or 512). A join uses it whole on one thread, and nests no other join; release_large_workspace lets a large buffer go after.

Definition at line 5763 of file factorizations.hpp.

◆ threads_for()

std::size_t numerical::detail::threads_for ( std::size_t  count,
std::size_t  work_per_index,
std::size_t  hardware 
)
inline

How many threads count indices of work_per_index each pay for: one per work_per_thread, at most one per index and per hardware thread.

Parameters
[in]countThe number of indices.
[in]work_per_indexThe work of one index, in multiply-adds.
[in]hardwareThe hardware threads.

Definition at line 748 of file parallel.hpp.

◆ through_doubles()

template<typename Into >
tensor< float > numerical::detail::through_doubles ( const float *  data,
const std::vector< std::size_t > &  shape,
Into  into 
)

The function into (one of the *_into of doubles) of every one of the row-major floats at data, of shape shape: each piece of 512 converted to doubles on the stack, into of them, each rounded back once. The double within 0.51 of its ulp, the float is within half of its own and 2⁻²⁸ more, where the C library's float functions are within one or two; and the doubles' vectors are faster than its scalar calls (glibc's sinhf 4.4 times the doubles' sinh, atanf 3.5, log1pf 2.9).

Definition at line 300 of file tensor_ops.hpp.

◆ through_edge()

template<std::size_t Rows, std::size_t Cols, typename T , typename Full >
void numerical::detail::through_edge ( T *  c,
std::size_t  ldc,
std::size_t  rows,
std::size_t  cols,
Full  full 
)
noexcept

One tile of a matrix product kernel, rows × cols of C (at most Rows × Cols, leading dimension ldc), through full, which computes a whole tile in place given its first element and leading dimension: a whole tile in C itself, a partial one in a copy whose outside is zero. Only that copy is zeroed: zeroing it for every tile, whole ones included, took a twelfth of an x86-64 product's time.

Definition at line 921 of file simd.hpp.

◆ through_rows()

template<std::size_t Rows, std::size_t Cols, typename T , typename Full >
void numerical::detail::through_rows ( T *  c,
std::size_t  ldc,
std::size_t  rows,
std::size_t  cols,
Full  full 
)
noexcept

through_edge for a tile full computes for any count of its rows up to Rows, its template argument: a tile of its whole Cols columns and fewer rows computed in C itself, those rows alone (one row of complex products had six rows' multiply-adds computed through a whole tile); any other through a copy, as through_edge, its rows alone computed there too. Each element of C has the same operations either way.

Definition at line 974 of file simd.hpp.

◆ tile_gemm()

template<typename T >
tensor< T > numerical::detail::tile_gemm ( const T *  a,
std::size_t  lda,
const T *  b,
std::size_t  ldb,
std::size_t  rows,
std::size_t  inner,
std::size_t  cols,
std::vector< std::size_t >  out_shape 
)

Matrix product C(M×N) = A(M×K) · B(K×N), row-major.

The fast path for a plain two-operand single-contraction einsum (matmul, matrix–vector, dot), by detail::gemm. Every element of C is summed in ascending order of the contraction, the general engine's order, so no tolerance changes.

Definition at line 1246 of file tensor.hpp.

◆ tile_product()

template<std::floating_point R>
std::complex< R > numerical::detail::tile_product ( const std::complex< R > &  p,
const std::complex< R > &  q 
)
inlinenoexcept

0 + p·q of complex numbers as the x86-64 complex tiles round it (complex_term::tiles): each part's product from zero, their difference and their sum, each added to 0.

Definition at line 1873 of file tensor.hpp.

◆ times() [1/2]

template<typename A , typename B >
auto numerical::detail::times ( const A &  a,
const B &  b 
)

a · b.

Definition at line 70 of file eigen.hpp.

◆ times() [2/2]

std::complex< double > numerical::detail::times ( const std::complex< double > &  a,
const std::complex< double > &  b 
)
inline

a · b of two complex numbers by the schoolbook formula, without C Annex G's recovery of infinities, which a large translation unit leaves out of line (a call per product: complex eig of order 64 took 2.7 times as long in the Python binding as alone). The entries here are finite.

Definition at line 79 of file eigen.hpp.

◆ times_power_of_two() [1/2]

template<std::floating_point U>
std::complex< U > numerical::detail::times_power_of_two ( const std::complex< U > &  x,
int  e 
)

times_power_of_two of each part of a complex x.

Definition at line 421 of file factorizations.hpp.

◆ times_power_of_two() [2/2]

template<std::floating_point R>
R numerical::detail::times_power_of_two ( R  x,
int  e 
)

x times 2ᵉ, rounded once (a power of 2 scales a normal number exactly).

Definition at line 399 of file factorizations.hpp.

◆ to_matrix_basis() [1/2]

template<typename T >
void numerical::detail::to_matrix_basis ( const tridiagonal_reduction< T > &  reduction,
buffer< T > &  at,
std::size_t  n 
)

to_matrix_basis of all n rows.

Definition at line 4989 of file factorizations.hpp.

◆ to_matrix_basis() [2/2]

template<typename T >
void numerical::detail::to_matrix_basis ( const tridiagonal_reduction< T > &  reduction,
buffer< T > &  at,
std::size_t  n,
std::size_t  rows 
)

The rows rows of at (n columns, every row by default), vectors in the tridiagonal's basis, taken to the matrix's: at ← at Mᵀ = at Dᴴ Qᵀ, column j times conj(δⱼ), then the blocks of reflectors from the last back on every row, as LAPACK's dormtr: two thirds of the operations of forming M (tridiagonal_basis) and multiplying by it.

Definition at line 4968 of file factorizations.hpp.

◆ transform_axis()

template<split_complex T>
tensor< T > numerical::detail::transform_axis ( const fft_source< T > &  a,
const std::optional< std::size_t > &  n,
std::ptrdiff_t  axis,
fft_norm  norm,
bool  inverse 
)

The transform of complex a along axis, n points (the axis cut or padded with zeros to it, as NumPy's), forward or inverse (conj(fft(conj x)), exact), scaled for norm. Each line gathered, transformed and scattered on its own, the lines shared among the threads: a line's bits are its own. (A real input takes real_transform_of.)

Definition at line 2357 of file fft.hpp.

◆ transformed_along()

template<typename T , typename F >
auto numerical::detail::transformed_along ( const tensor< T > &  a,
std::ptrdiff_t  axis,
const F &  f 
) -> decltype(f(source_of(a), axis))

f (f(source, axis), a transform along axis, negative from the last) of a: a transpose's elements as they lie where as_laid_along reads them (each line transformed on its own, its bits the same); any other tensor's row-major.

Definition at line 2259 of file fft.hpp.

◆ transpose_into()

template<typename T >
void numerical::detail::transpose_into ( const T *  from,
std::size_t  rows,
std::size_t  cols,
T *  to 
)
private

The row-major rows × cols from into to as its transpose, cols × rows row-major, by tiles, bands of them on the threads (a pass bound by the memory, its bounds captured by value).

Definition at line 3389 of file factorizations.hpp.

◆ transpose_piece()

std::size_t numerical::detail::transpose_piece ( std::size_t  from_bytes,
std::size_t  to_bytes 
)
inlinenoexcept

The side of the square pieces a transposing copy walks a tile by, from the bytes between rows of its source and of its result: a tile's 32 rows a multiple of a power of two of bytes apart fall in a few of the L1 cache's sets and evict one another, where 8 rows fit them. Pieces of 8 where either stride is a multiple of 1 KiB on x86-64 (sets every 4 KiB: four AVX2 cores, 256 × 256 doubles 165 → 51 µs, 128 × 128 15.8 → 10.3, 2048 × 2048 15.4 → 9.2 ms; ten AVX-512 cores 256 × 256 142 → 51 µs), of 8 KiB on arm64 (M1 Pro, 1024 × 1024 3.8 → 0.95 ms, 2048 × 2048 19.7 → 7.2); elsewhere the whole tile, which pieces slowed (100 × 100 on the AVX2 cores 10.5 → 11.6 µs, most orders on the M1 Pro 10 to 40 %).

Definition at line 100 of file simd.hpp.

◆ transpose_pieces()

template<std::size_t Piece, typename Copy >
void numerical::detail::transpose_pieces ( std::size_t  i0,
std::size_t  i1,
std::size_t  j0,
std::size_t  j1,
const Copy &  copy 
)

copy(i, j) for every i in [i0, i1) and j in [j0, j1), by square pieces of side Piece, j the faster within a piece.

Definition at line 117 of file simd.hpp.

◆ transpose_walk()

template<typename Copy >
void numerical::detail::transpose_walk ( std::size_t  i0,
std::size_t  i1,
std::size_t  j0,
std::size_t  j1,
std::size_t  piece,
const Copy &  copy 
)

transpose_pieces of side piece (transpose_piece's 8 or the whole tile), each side its own instance: a side known only at run time slowed the whole tile's copy (100 × 100 doubles, four AVX2 cores, 10.4 → 11.2 µs).

Definition at line 142 of file simd.hpp.

◆ triangle_is_finite()

template<typename T >
bool numerical::detail::triangle_is_finite ( const T *  a,
std::size_t  n,
bool  lower 
)

Whether the triangle of the row-major n × n a a solve reads (the diagonal with it) is finite, entry by entry: the rare road of require_finite_triangle, where the whole matrix is not.

Definition at line 2003 of file factorizations.hpp.

◆ triangle_split()

std::size_t numerical::detail::triangle_split ( std::size_t  i,
std::size_t  rows,
std::size_t  cols,
std::ptrdiff_t  k,
bool  lower 
)
inline

The first column of row i past the part to and below its k diagonal (lower), or the first column of the part to and above it (else), within [0, cols]; k clamped first, so i + k cannot overflow.

Definition at line 5028 of file tensor_ops.hpp.

◆ triangular_halves()

template<typename T >
void numerical::detail::triangular_halves ( const T *  a,
std::size_t  n,
bool  lower,
bool  transposed,
bool  unit,
T *  x,
std::size_t  ldx,
std::size_t  k,
std::size_t  first,
std::size_t  last,
T *  minus 
)

A X = B in place in x (n × k, row-major), for the n × n row-major triangle of a, its lower one or its upper one (the other never read), its diagonal taken as ones where unit: by halves (triangular_halves), each half's terms out of the other by one matrix product as deep as it, as a recursive trsm; one column by a dot product of each row with the unknowns solved.

Rows first to last of the triangular solve of triangular_rows (M lower where lower, M the n × n row-major a or, transposed, its transpose read in place), the terms of the rows outside them already taken: up to triangular_block rows one by one, a row of terms at a time; past them by halves, the half solved first taken from the other by one matrix product as deep as the half, then the other half solved. x holds k columns, its rows ldx apart; minus a half's solution negated.

Definition at line 1862 of file factorizations.hpp.

◆ triangular_log()

template<typename T >
buffer< T > numerical::detail::triangular_log ( const buffer< T > &  t0,
std::size_t  n 
)

The logarithm of the upper triangular t0 (no zero on its diagonal), as SciPy's _logm_triu: R, s and m by inverse_squaring, U = 2ˢ Σ αₖ (I + βₖ R)⁻¹ R (the [m/m] Padé approximant as the partial fractions of m-point Gauss–Legendre quadrature on [0, 1], SciPy's nodes and weights), then the diagonal log λᵢ and the superdiagonal recomputed exactly where the principal branch exists.

Definition at line 2974 of file matrix_functions.hpp.

◆ triangular_pencil_vectors()

matrix< std::complex< double > > numerical::detail::triangular_pencil_vectors ( const matrix< std::complex< double > > &  s,
const matrix< std::complex< double > > &  t,
const std::vector< std::pair< std::complex< double >, std::complex< double > > > &  values,
const matrix< std::complex< double > > &  basis,
bool  left 
)
inlineprivate

The right (or, where left, the left) eigenvectors of the triangular pencil (s, t) whose diagonal is values, as LAPACK's ztgevc: for each (α, β), x with (β S − α T) x = 0 and x_k = 1 by back substitution (yᴴ (β S − α T) = 0 forward for a left one), then basis x made of unit 2-norm; a diagonal entry of β S − α T below ε (|β| ‖S‖ + |α| ‖T‖), or the smallest normal number, taken as that bound (SMIN), and α and β both zero (a singular pencil) given the unit vector of its place. The substitutions on the threads, each on its own; the basis applied to their nonzero parts by one product.

Returns
The vectors as columns.

Definition at line 2322 of file generalized.hpp.

◆ triangular_power()

buffer< std::complex< double > > numerical::detail::triangular_power ( const buffer< std::complex< double > > &  t0,
std::size_t  n,
double  t 
)
inlineprivate

T^t (−1 < t < 1) of the upper triangular t0 (no zero on its diagonal), as SciPy's _remainder_matrix_power_triu (Higham and Lin's Schur–Padé algorithm, 2013): a diagonal T entry by entry; else R = T^(1/2ˢ) − I and m by inverse_squaring with the bounds of (I − X)ᵗ, the [m/m] Padé approximant at −R, then s squarings. Where the principal branch exists, the diagonal λᵢ^(t/2ⁱ) and the superdiagonal (fractional_power_superdiagonal) are written exactly after each squaring, as Higham and Lin's algorithm 5.1 does (SciPy writes them before the first squaring only).

Definition at line 3129 of file matrix_functions.hpp.

◆ triangular_reciprocal_condition()

template<typename T >
double numerical::detail::triangular_reciprocal_condition ( const T *  a,
std::size_t  n,
bool  lower 
)

reciprocal_condition of the triangular A, the lower triangle of the n × n row-major a or its upper one, as LAPACK's trcon (1-norm): ‖A‖₁ the triangle's largest column sum of moduli, A⁻¹'s products by substitution. 0 where the diagonal holds a zero.

Definition at line 2290 of file factorizations.hpp.

◆ triangular_rows()

template<typename T >
void numerical::detail::triangular_rows ( const T *  a,
std::size_t  n,
bool  lower,
bool  transposed,
bool  unit,
T *  x,
std::size_t  k 
)

M X = B in place in x (n × k, row-major) for M the n × n row-major a or, transposed, its transpose, read in place; lower whether a (not M) is lower-triangular, the other triangle never read, its diagonal taken as ones where unit. By halves (triangular_halves), as a recursive trsm, the columns by blocks of triangular_slice on the threads; one column of M = a by a dot product of each row with the unknowns solved, of M = aᵀ by a sweep, each unknown solved taken from the others by its row of a.

Definition at line 1930 of file factorizations.hpp.

◆ triangular_square_root()

template<typename T >
buffer< T > numerical::detail::triangular_square_root ( const buffer< T > &  t,
std::size_t  n 
)

The principal square root of the upper triangular n × n row-major t (no real eigenvalue below 0 where T is real): pointwise_square_root up to square_root_block or where a diagonal entry is 0 (a zero over a zero taken as 0 there); else by blocks (Deadman, Higham and Ralha, 2013, as SciPy's _sqrtm_triu): each diagonal block's root entry by entry, then each block column from the diagonal up, R_ii X + X R_jj = T_ij − R_i,(i+1…j−1) R_(i+1…j−1),j, the product by gemm and the Sylvester equation by triangular_sylvester (R_ii + R_jj has no eigenvalue at 0: principal roots of nonzero eigenvalues).

Definition at line 1335 of file matrix_functions.hpp.

◆ triangular_sylvester()

template<typename T >
void numerical::detail::triangular_sylvester ( const buffer< T > &  r,
std::size_t  m,
const buffer< T > &  s,
std::size_t  n,
buffer< T > &  f 
)

Y of R Y + Y S = F in place in f (m × n, row-major) for the upper quasi-triangular (real) or triangular (complex) r (m × m) and s (n × n), as LAPACK's trsyl (no transposes, +): by blocks of columns of S from the first, each first relieved of the columns solved before it through the matrix product, then by blocks of rows of R from the last, each a system of at most four unknowns (small_sylvester_block), R and −S close in an eigenvalue perturbing its pivot as trsyl does.

Definition at line 813 of file matrix_functions.hpp.

◆ tridiagonal_basis()

template<typename T >
buffer< T > numerical::detail::tridiagonal_basis ( const tridiagonal_reduction< T > &  reduction,
std::size_t  n 
)

Mᵀ of a reduction, row-major (its rows are the columns of M): Qᵀ by blocks of reflectors from the last back, as LAPACK's dorgtr forms Q, each block on the rows it reaches of the identity, then row j times conj(δⱼ).

Definition at line 4940 of file factorizations.hpp.

◆ tridiagonal_divide_and_conquer()

template<typename T >
buffer< T > numerical::detail::tridiagonal_divide_and_conquer ( buffer< double > &  d,
buffer< double > &  e,
const tridiagonal_reduction< T > &  reduction,
std::size_t  n 
)

The eigenvalues of the tridiagonal (d, e) of reduction into d, and the matrix's eigenvectors as the rows of the result, by divide and conquer (LAPACK's dstedc, then dormtr): most of the work in matrix products. Up to divide_leaf, tridiagonal_ql on Mᵀ itself.

Exceptions
convergence_errorIf an iteration did not converge within its bound.

Definition at line 6242 of file factorizations.hpp.

◆ tridiagonal_ql()

template<typename T >
void numerical::detail::tridiagonal_ql ( buffer< double > &  d,
buffer< double > &  e,
buffer< T > &  zt,
std::size_t  n,
std::size_t  max_steps 
)

The eigenvalues of the symmetric tridiagonal (d, e) by implicit QL or QR as LAPACK's dsteqr, the rotations accumulated into the rows of zt. Each unreduced block, scaled into a safe range, is iterated from the end of its smaller diagonal entry (QL down a block graded downwards, QR up one graded upwards), an off-diagonal entry dropped against its neighbours' diagonal entries (e² ≤ ε² |dᵢ dᵢ₊₁|) rather than the matrix's norm, and no shift is subtracted from the whole diagonal: the small eigenvalues of a graded matrix keep their relative accuracy. EISPACK's tql2, here before, tested against the largest entry seen and moved every diagonal entry by the accumulated shift: on d = gⁱ, e near √(dᵢ dᵢ₊₁), the smallest eigenvalues lost every digit (g = 0.01, n = 10; NumPy's 3e-16).

Parameters
[in,out]dThe diagonal; on return, the eigenvalues (unsorted).
[in,out]eThe subdiagonal in e[0…n-2], e[n-1] = 0; destroyed.
[in,out]ztRow-major n × n whose rows are rotated together (Qᵀ from tridiagonalize); on return, row j is the eigenvector of d[j].
[in]nThe order.
[in]max_stepsThe bound of iterations in all.
Exceptions
convergence_errorIf a block has not split after max_steps iterations.

Definition at line 5398 of file factorizations.hpp.

◆ tridiagonal_root_free_ql()

void numerical::detail::tridiagonal_root_free_ql ( buffer< double > &  d,
buffer< double > &  e,
std::size_t  n,
std::size_t  max_steps 
)
inline

The eigenvalues of the symmetric tridiagonal (d, e), no vector, as LAPACK's dsterf: the Pal–Walker–Kahan variant of implicit QL or QR on the squares of the off-diagonal entries, no root in its inner loop, each block scaled into a safe range and iterated from its smaller end, an entry dropped against its neighbours (e² ≤ ε² |dᵢ dᵢ₊₁|). Faster than tridiagonal_ql's dsteqr without rotations (one core: order 500 4.4 → 3.4 ms, 2000 66 → 52) and more accurate on a graded matrix (the smallest of dᵢ = 0.01ⁱ, n = 10: 3e-16 against 4e-14).

Parameters
[in,out]dThe diagonal; on return, the eigenvalues (unsorted).
[in,out]eThe subdiagonal in e[0…n-2], e[n-1] = 0; destroyed.
[in]nThe order.
[in]max_stepsThe bound of iterations in all.
Exceptions
convergence_errorIf a block has not split after max_steps iterations.

Definition at line 5226 of file factorizations.hpp.

◆ tridiagonal_solve()

template<typename T >
void numerical::detail::tridiagonal_solve ( buffer< T > &  dl,
buffer< T > &  d,
buffer< T > &  du,
buffer< T > &  b,
std::size_t  n,
std::size_t  k 
)

The k right-hand sides of b (n × k, row-major) solved in place against the tridiagonal matrix of subdiagonal dl, diagonal d and superdiagonal du (n − 1, n, n − 1 entries, overwritten), as LAPACK's gtsv, which scipy.linalg.solve_banded calls for one sub- and one superdiagonal: Gaussian elimination with partial pivoting, row i + 1 exchanged with row i where its subdiagonal entry is the larger (dl then holding the second superdiagonal the exchange brings), then back substitution.

Exceptions
singular_matrix_errorIf a pivot is zero.

Definition at line 247 of file banded.hpp.

◆ tridiagonalize()

template<typename T >
tridiagonal_reduction< T > numerical::detail::tridiagonalize ( buffer< T > &  a,
std::size_t  n,
buffer< double > &  d,
buffer< double > &  e 
)

Reduces the Hermitian (real: symmetric) n × n row-major a, of which the lower triangle is read, to a real symmetric tridiagonal by Householder reflections, A = M T Mᴴ with M unitary.

Each reflection zeroes one column below the subdiagonal; the trailing block takes the rank-2 updates B − v wᴴ − w vᴴ of a panel of tridiagonal_block reflections at once, through the matrix product kernel, as LAPACK's dsytrd. A complex subdiagonal entry is then made real by a unit phase per row, as LAPACK's zhetrd leaves it for a real iteration; a real one keeps its sign.

Parameters
[in,out]aThe matrix; destroyed.
[in]nIts order (at least 1).
[out]dThe diagonal of T (size n).
[out]eThe subdiagonal of T in e[0…n-2], and e[n-1] = 0 (size n).
Returns
M's reflectors and phases: tridiagonal_basis forms M, to_matrix_basis applies it.

Definition at line 4814 of file factorizations.hpp.

◆ trig_transform() [1/2]

template<std::floating_point R>
tensor< R > numerical::detail::trig_transform ( const fft_source< R > &  source,
trig_family  family,
int  type,
const std::optional< std::size_t > &  n,
std::ptrdiff_t  axis,
fft_norm  norm,
const std::optional< bool > &  orthogonalize,
bool  inverse 
)

dct, idct, dst or idst (family, inverse) of source along axis, as scipy.fft's: the type computed (an inverse's of types 2 and 3 the other), on each line of n points (the axis cut or padded), the sines by the cosines: type 1 the real transform of the odd extension (2(n + 1) points), type 2 the cosines of the points of alternate signs, in reverse order; types 3 and 4 those of the points reversed, of alternate signs. Scaled as norm says, orthogonalize (by default where norm is ortho) the first or last points and values weighted by √2 as SciPy's. The elements of source are read where they lie.

Definition at line 3439 of file fft.hpp.

◆ trig_transform() [2/2]

template<std::floating_point R>
tensor< R > numerical::detail::trig_transform ( const tensor< R > &  a,
trig_family  family,
int  type,
const std::optional< std::size_t > &  n,
std::ptrdiff_t  axis,
fft_norm  norm,
const std::optional< bool > &  orthogonalize,
bool  inverse 
)

trig_transform of the tensor a.

Definition at line 3542 of file fft.hpp.

◆ twiddle_run()

template<std::floating_point R>
void numerical::detail::twiddle_run ( std::complex< R > *  t,
const std::complex< R > *  w,
std::size_t  count,
bool  fused 
)
noexcept

twiddle_run_body on the path in use.

Definition at line 1795 of file fft.hpp.

◆ twiddle_run_body()

template<bool Fused, std::floating_point R>
void numerical::detail::twiddle_run_body ( std::complex< R > *  t,
const std::complex< R > *  w,
std::size_t  count 
)
inlinenoexcept

t[i] = w[i] · t[i] for count elements, as Fused (twiddled). Inlined into each caller.

Definition at line 1772 of file fft.hpp.

◆ twiddled() [1/2]

template<bool Fused, std::floating_point R>
requires Fused
std::complex< R > numerical::detail::twiddled ( const std::complex< R > &  w,
const std::complex< R > &  v 
)
inlinenoexcept

A butterfly's factor times v on the vector paths: fused_product.

Definition at line 327 of file fft.hpp.

◆ twiddled() [2/2]

template<bool Fused, std::floating_point R>
requires (!Fused)
std::complex< R > numerical::detail::twiddled ( const std::complex< R > &  w,
const std::complex< R > &  v 
)
inlinenoexcept

A butterfly's factor times v on the portable path: rounded_product.

Definition at line 337 of file fft.hpp.

◆ twiddled_rows_fft()

template<std::floating_point R>
void numerical::detail::twiddled_rows_fft ( const fft_plan< R > &  plan,
std::complex< R > *  data,
std::size_t  rows,
std::size_t  apart,
const std::complex< R > *  twiddles,
bool  fused,
std::complex< R > *  out,
bool  inverse,
R  scale 
)

Each of the rows rows of data (apart from one another; conjugated first where inverse) times its run of twiddles (each as long as the rows), then transformed in place (plan's length), then written down a column of out, row r's element k to out[r + rows·k], conjugated where inverse and times scale: fft_four_step_band rows a task, still in the caches, each element of out written among that many neighbours.

Definition at line 2183 of file fft.hpp.

◆ two_real_shifts() [1/2]

bool numerical::detail::two_real_shifts ( const std::complex< double > *  ,
lapack_index  ks,
lapack_index  kbot,
const std::complex< double > &   
)
inline

The same for a complex matrix: two shifts of any kind.

Definition at line 2522 of file eigen.hpp.

◆ two_real_shifts() [2/2]

bool numerical::detail::two_real_shifts ( const std::complex< double > *  w,
lapack_index  ks,
lapack_index  kbot,
double   
)
inline

Whether a real matrix's sweep is down to two shifts, both real: then the one nearer the corner serves twice.

Definition at line 2513 of file eigen.hpp.

◆ two_sum() [1/2]

double_double numerical::detail::two_sum ( double  a,
double  b 
)
inlinenoexcept

a + b exactly, as a sum and its rounding error (Knuth's two-sum).

Definition at line 69 of file fft.hpp.

◆ two_sum() [2/2]

void numerical::detail::two_sum ( double  a,
double  b,
double &  sum,
double &  error 
)
inlinenoexcept

The sum of a and b as sum rounded and error exact (Knuth's two-sum: no order of magnitudes needed).

Definition at line 223 of file elementary.hpp.

◆ undeflated() [1/2]

lapack_index numerical::detail::undeflated ( one_based< double >  t,
one_based< double >  vt,
lapack_index  jw,
double  s,
double  smallest,
std::complex< double > *  w 
)
inline

The eigenvalues of the real Schur form t (jw × jw) of a deflation window that do not deflate, moved to its top, as dlaqr3: from the bottom, a block whose spike entries s v₁ⱼ are negligible deflates, any other is moved up; the undeflated blocks are then sorted by decreasing modulus (which helps graded matrices), and every eigenvalue of the window written to w.

Returns
Their count.

Definition at line 2154 of file eigen.hpp.

◆ undeflated() [2/2]

lapack_index numerical::detail::undeflated ( one_based< std::complex< double > >  t,
one_based< std::complex< double > >  vt,
lapack_index  jw,
const std::complex< double > &  s,
double  smallest,
std::complex< double > *  w 
)
inline

The same for a complex Schur form, as zlaqr3: 1 × 1 blocks only, sorted by selection.

Definition at line 2225 of file eigen.hpp.

◆ unit_circle_offset() [1/2]

double numerical::detail::unit_circle_offset ( double  x,
double  y 
)
inline

x² + y² − 1 for 0.5 ≤ x < 1, y ≤ x and x² + y² ≥ 0.5, almost exactly: each square split into its rounded value and its error (a fused multiply-add), the five terms summed by magnitude, each pair's error carried to the next.

Definition at line 531 of file complex_math.hpp.

◆ unit_circle_offset() [2/2]

float numerical::detail::unit_circle_offset ( float  x,
float  y 
)
inline

unit_circle_offset of floats, in double: (x − 1)(x + 1) + y².

Definition at line 567 of file complex_math.hpp.

◆ unit_circle_squares_reach_half()

template<std::floating_point R>
bool numerical::detail::unit_circle_squares_reach_half ( R  x,
R  y 
)

Whether x² + y² ≥ 0.5, as glibc's build computes the sum (libm_sum_of_products).

Definition at line 577 of file complex_math.hpp.

◆ unit_root()

std::complex< double > numerical::detail::unit_root ( std::size_t  k,
std::size_t  n 
)
inline

unit_root_parts rounded once to double, a zero's sign positive as the exact value's.

Definition at line 229 of file fft.hpp.

◆ unit_root_parts()

std::pair< double_double, double_double > numerical::detail::unit_root_parts ( std::size_t  k,
std::size_t  n 
)
inline

e^{−2πi k/n} as double-double parts: reduce_to_octant, octant_sin_cos, place_by_octant.

Definition at line 220 of file fft.hpp.

◆ unscale_bidiagonal_node()

void numerical::detail::unscale_bidiagonal_node ( bidiagonal_node &  node)
inline

The singular values of a solved node scaled back.

Definition at line 1419 of file svd.hpp.

◆ unscale_divide_node()

void numerical::detail::unscale_divide_node ( const divide_node &  node)
inline

The eigenvalues of a solved node, scaled back, where its parent reads them.

Definition at line 6105 of file factorizations.hpp.

◆ unsigned_zero()

double numerical::detail::unsigned_zero ( double  x)
inlinenoexcept

x with a zero's sign made positive: the twiddle factors' zeros are +0, as their exact values.

Definition at line 161 of file fft.hpp.

◆ unwinding()

double numerical::detail::unwinding ( const std::complex< double > &  z)
inline

ceil((Im z − π) / 2π), the unwinding number of z (SciPy's _unwindk).

Definition at line 2684 of file matrix_functions.hpp.

◆ upper_band()

template<typename T >
band_storage< T > numerical::detail::upper_band ( const matrix< T > &  ab,
bool  lower_form 
)

The upper band of kd superdiagonals of the band ab in either of scipy.linalg's forms (solveh_banded), AB(kd + 1 + i − j, j) = A(i, j) for i ≤ j, as band_cholesky takes it.

Definition at line 535 of file banded.hpp.

◆ upper_inverse()

template<typename T >
buffer< T > numerical::detail::upper_inverse ( const buffer< T > &  r,
std::size_t  n 
)
private

R⁻¹, upper triangular, of the n × n upper-triangular r (row-major): back substitution against the identity in parallel chunks of columns, each row's terms in registers (subtract_products). A chunk's rows past its last column stay zero; a zero on the diagonal leaves infinities or NaN.

Definition at line 2007 of file svd.hpp.

◆ upper_triangular()

template<typename T >
bool numerical::detail::upper_triangular ( const matrix< T > &  a)

Whether the square a is upper triangular: its Schur form as it stands, Z = I (as SciPy takes it, so an infinite entry of the root reaches no other).

Definition at line 1459 of file matrix_functions.hpp.

◆ vector_times_columns() [1/3]

buffer< double > numerical::detail::vector_times_columns ( const double *  a,
const double *  b,
std::size_t  row_step,
std::size_t  column_step,
std::size_t  inner,
std::size_t  cols 
)
inline

vector_times_columns for double: on the x86-64 fused paths, where add_product adds each row by fused multiply-adds, by avx2_strided_columns; on the arm64 NEON and AMX paths, where it does too, by fused terms; elsewhere the template's arithmetic.

Definition at line 1382 of file tensor.hpp.

◆ vector_times_columns() [2/3]

buffer< float > numerical::detail::vector_times_columns ( const float *  a,
const float *  b,
std::size_t  row_step,
std::size_t  column_step,
std::size_t  inner,
std::size_t  cols 
)
inline

vector_times_columns for float: each term as gemm_rows rounds it, fused on the x86-64 AVX paths (avx2_strided_columns) and on arm64's NEON and AMX paths, elsewhere the template's arithmetic.

Definition at line 1354 of file tensor.hpp.

◆ vector_times_columns() [3/3]

template<typename T , bool Fused = false>
buffer< T > numerical::detail::vector_times_columns ( const T *  a,
const T *  b,
std::size_t  row_step,
std::size_t  column_step,
std::size_t  inner,
std::size_t  cols 
)

canonical_product's vector a (inner) times a matrix b whose element B[k][j] is b[(k · row_step) + (j · column_step)]: each element of the product the sum canonical_product forms, a[k] · B[k][j] added in ascending order of k with add_product's arithmetic (add_one_product), so the same bits; strided_columns columns' sums held at once, where add_product stores and reloads them for every row of B.

Definition at line 1323 of file tensor.hpp.

◆ vector_times_transposed()

template<typename T >
buffer< T > numerical::detail::vector_times_transposed ( const T *  a,
const T *  bt,
std::size_t  ldbt,
std::size_t  inner,
std::size_t  cols 
)

vector_times_columns of a matrix given as its transpose bt (cols × inner, its rows ldbt apart, B's columns); a complex one past complex_term::compiler by complex_vector_times_columns.

Definition at line 1471 of file tensor.hpp.

◆ vector_work()

std::size_t numerical::detail::vector_work ( std::size_t  n)
inlinenoexcept

A triangular solve's work for parallel_for over the n eigenvectors: k²/2 products for vector k, n²/6 on average.

Definition at line 2931 of file eigen.hpp.

◆ view_axes()

std::vector< std::size_t > numerical::detail::view_axes ( const std::vector< std::size_t > &  order)
inline

The axes that put the elements of a tensor laid out in order (tensor::laid_out, its view's axes in the order they lie) back in the view's order: the permutation to transpose a result of the same axes by.

Definition at line 1929 of file tensor_ops.hpp.

◆ vigilant_deflation()

template<typename T >
void numerical::detail::vigilant_deflation ( one_based< T >  h,
lapack_index  k,
lapack_index  ktop,
lapack_index  kbot,
double  smallest 
)

H(k + 1, k) set to zero where the deflation criterion of Ahues and Tisseur holds, as the sweep of dlaqr5 tests after each step of a bulge (vigilant deflation).

Definition at line 1866 of file eigen.hpp.

◆ walk_along()

axis_walk numerical::detail::walk_along ( const std::vector< std::size_t > &  shape,
std::size_t  axis 
)
inline

axis_walk of shape along axis; outer and inner 0 where the result is empty.

Exceptions
dimension_errorIf axis is not an axis, or the result's elements overflow.

Definition at line 53 of file tensor_ops.hpp.

◆ walked_extreme()

template<typename T , typename Order >
T numerical::detail::walked_extreme ( const T *  first,
std::size_t  count,
std::size_t  stride,
const Order &  larger 
)

extreme of a line walked element by element: a NaN wins, as in NumPy (every comparison with it is false, so a NaN candidate is taken apart, and once taken nothing compares larger than it).

Definition at line 1813 of file tensor_ops.hpp.

◆ wants()

constexpr bool numerical::detail::wants ( std::size_t  core,
std::size_t  caller,
std::size_t  helpers 
)
constexprnoexcept

Whether the worker on core joins a run of helpers workers whose caller is on caller: by rank among the cores, the caller's aside, so a run of few threads takes the same cores from one run to the next.

Definition at line 387 of file parallel.hpp.

◆ whole_schur_form() [1/2]

void numerical::detail::whole_schur_form ( const schur_target< double > &  t,
std::size_t  steps 
)
inlineprivate

The real Schur form of the whole of t (its eigenvalues not kept), steps its iterations' bound.

Definition at line 3355 of file eigen.hpp.

◆ whole_schur_form() [2/2]

void numerical::detail::whole_schur_form ( const schur_target< std::complex< double > > &  t,
std::size_t  steps 
)
inlineprivate

The complex Schur form likewise.

Definition at line 3364 of file eigen.hpp.

◆ wins_any_order()

template<std::floating_point T>
bool numerical::detail::wins_any_order ( T  x)
noexcept

A real's NaN, which wins numpy.argmax and numpy.argmin.

An integer's: none.

Definition at line 4226 of file tensor_ops.hpp.

◆ with_decimal_rounding() [1/2]

template<std::floating_point R, typename F >
requires (!std::same_as<R, float>)
auto numerical::detail::with_decimal_rounding ( int  decimals,
const F &  f 
)

f(rounding) with the decimal_rounding of R to decimals places: in R, as NumPy's, but in double for a float whose 10ᵈ overflows it (a float under 2²⁴ · 10⁻ᵈ still has places to round there).

Definition at line 2952 of file tensor_ops.hpp.

◆ with_decimal_rounding() [2/2]

template<std::same_as< float > R, typename F >
auto numerical::detail::with_decimal_rounding ( int  decimals,
const F &  f 
)

with_decimal_rounding of a float: in double where its 10ᵈ overflows a float.

Definition at line 2961 of file tensor_ops.hpp.

◆ without_axis()

std::vector< std::size_t > numerical::detail::without_axis ( std::vector< std::size_t >  shape,
std::size_t  axis 
)
inline

The shape with axis axis removed.

Definition at line 1479 of file tensor_ops.hpp.

◆ worker_cores()

std::vector< std::size_t > numerical::detail::worker_cores ( std::size_t  cores,
std::size_t  threads 
)
inline

The core of each worker the pool starts beside its callers, from the cores known (0: none) and its threads (at least 1): one worker per core, pinned there, where the cores are known (each run then leaves out the one on the caller's core, sits_out); else threads − 1 workers free to move.

Definition at line 71 of file parallel.hpp.

◆ workspace_of()

double * numerical::detail::workspace_of ( workspace_buffer &  buffer,
std::size_t  size 
)
inline

buffer at least size elements long, its storage kept for the next join. A larger one is mapped fresh, and its pages are touched by every thread, chunk by chunk: on Linux a page costs about a microsecond the first time it is written, and the join that first wrote it waited on them one thread at a time (an eigh of order 1024 mapped some 30 MB of these buffers on one x86-64 core).

Definition at line 5739 of file factorizations.hpp.

◆ zero_block()

template<typename T >
void numerical::detail::zero_block ( T *  c,
std::size_t  ldc,
std::size_t  first,
std::size_t  last,
std::size_t  from,
std::size_t  to 
)
noexcept

Zeroes rows first to last (excluded) of C, columns from to to (excluded), its rows ldc apart.

Definition at line 2200 of file gemm.hpp.

◆ zero_c()

template<typename T >
void numerical::detail::zero_c ( T *  c,
std::size_t  ldc,
std::size_t  m,
std::size_t  n 
)

Zeroes C's m × n elements, its rows ldc apart: on the threads where they lie together and are many (parallel_zero), a fresh C's pages then faulted in by several.

Definition at line 2219 of file gemm.hpp.

◆ zeroes_c() [1/2]

template<typename Kernel , typename T >
requires (!writes_fresh<Kernel, T>)
constexpr bool numerical::detail::zeroes_c ( bool  empty)
constexprnoexcept

Whether gemm_packed zeroes a C that holds nothing (empty) for Kernel's tiles to add to.

Definition at line 2537 of file gemm.hpp.

◆ zeroes_c() [2/2]

template<typename Kernel , typename T >
requires writes_fresh<Kernel, T>
constexpr bool numerical::detail::zeroes_c ( bool  )
constexprnoexcept

zeroes_c for tiles that write C fresh: never.

Definition at line 2546 of file gemm.hpp.

Variable Documentation

◆ atan_half_pi_high

constexpr double numerical::detail::atan_half_pi_high {0x1.921fb54442d18p+0}
inlineconstexpr

π/2, the double nearest it (tools/atan_table.py derives each constant here).

Definition at line 431 of file elementary.hpp.

◆ atan_half_pi_low

constexpr double numerical::detail::atan_half_pi_low {0x1.1a62633145c07p-54}
inlineconstexpr

What atan_half_pi_high leaves out of π/2, to the nearest double.

Definition at line 434 of file elementary.hpp.

◆ atan_table

constexpr std::array<double, 130> numerical::detail::atan_table
constexpr

atan(k/64) for k = 0…64, at 2k the double nearest it and at 2k + 1 the double nearest what that one leaves out. Not inline, as exp_table.

Definition at line 438 of file elementary.hpp.

◆ back_substitution_limit

constexpr double numerical::detail::back_substitution_limit {0x1p400}
inlineconstexpr

The modulus an eigenvector's components are kept under during the back-substitution: the sums stay finite for any order a matrix can have, and a component that would grow past it scales the vector down.

Definition at line 53 of file eigen.hpp.

◆ balance_limit

constexpr double numerical::detail::balance_limit {0x1p256}
inlineconstexpr

The largest power of two a balancing scales a row or a column by, either way: the matrix enters at a largest magnitude near 1, so no entry overflows, and the scaling stops where LAPACK's would be one factor among dozens.

Definition at line 49 of file eigen.hpp.

◆ band_columns

constexpr std::size_t numerical::detail::band_columns {256}
inlineconstexpr

The columns of a band that one task of fold_along and scan_along walks row by row.

Definition at line 1507 of file tensor_ops.hpp.

◆ bidiagonal_basis_crossover

constexpr std::size_t numerical::detail::bidiagonal_basis_crossover {64}
inlineconstexpr

The order below which the bases of a bidiagonal reduction are formed a reflector at a time: the blocks' packing and triangular factor cost more than they save. Order 32: 11 % shorter on six x86-64 cores, 12 % on arm64; order 96: 7 % longer on both.

Definition at line 196 of file svd.hpp.

◆ bidiagonal_block

constexpr std::size_t numerical::detail::bidiagonal_block {32}
inlineconstexpr

The rows from which a step of the blocked bidiagonal reduction shares its coefficients and corrections with the threads: on x86-64 (six AVX2 cores, the reduction of order 512 −5 to −14 %, 1024 −12 %); never on arm64, where its serial part is smaller and the shared step gained nothing (M1 Pro, eight cores, order 1024 0.97 to 1.05 times as long).

The reflections from each side in one panel of the blocked bidiagonal reduction.

Definition at line 263 of file svd.hpp.

◆ bidiagonal_crossover

constexpr std::size_t numerical::detail::bidiagonal_crossover {128}
inlineconstexpr

The order below which a real bidiagonal reduction goes a reflection at a time, as LAPACK's dgebrd falls back to dgebd2: the panels' products are too small to pay for packing. Order 64: 33 % shorter on six x86-64 cores, 7 % on arm64; order 160: 30 % longer on arm64, where x86-64 still gains 8 %.

Definition at line 191 of file svd.hpp.

◆ bidiagonal_leaf

constexpr std::size_t numerical::detail::bidiagonal_leaf {32}
inlineconstexpr

The order up to which bidiagonal_divide decomposes by implicit-shift QR.

Definition at line 1213 of file svd.hpp.

◆ bidiagonal_product_weight

constexpr std::size_t numerical::detail::bidiagonal_product_weight {1}
inlineconstexpr

The weight of a multiply-add of the bidiagonal reduction's matrix-vector products, which read the trailing block once for each column: on x86-64 a thread shares them from a quarter of the work (six cores, singular values of order 256: 8.1 → 7.3 ms, 512: 35 → 30); on arm64 that slowed order 256 by 10 %.

Definition at line 271 of file svd.hpp.

◆ cancelled

constexpr double numerical::detail::cancelled {64.0}
inlineconstexpr

The fall of a column's norm, in units of √n ε, past which one_sided_jacobi takes what a rotation left of it for rounding alone.

Definition at line 51 of file svd.hpp.

◆ cholesky_block

constexpr std::size_t numerical::detail::cholesky_block {64}
inlineconstexpr

The columns of one panel of the blocked Cholesky factorisation (as lu_block).

Definition at line 262 of file factorizations.hpp.

◆ cholesky_diagonal_rows

constexpr std::size_t numerical::detail::cholesky_diagonal_rows {4}
inlineconstexpr

The rows of a Cholesky diagonal block factorised together, one chain of sums each (one x86-64 AVX2 core, order 300: the diagonal blocks 128 → 65 µs; eight rows did no better).

Definition at line 266 of file factorizations.hpp.

◆ cholesky_one_panel

constexpr std::size_t numerical::detail::cholesky_one_panel {48}
inlineconstexpr

The order below which the Cholesky factorisation is one panel: no trailing product, whose packing the few panels of a small order do not repay (order 17 on one thread: 0.8 µs whole, 1.3 µs in panels of 16). Panels of 16 win from order 40 on x86-64 and from 48 on arm64.

Definition at line 278 of file factorizations.hpp.

◆ cholesky_rows

constexpr std::size_t numerical::detail::cholesky_rows {64}
inlineconstexpr

The rows of one chunk of a Cholesky panel's rows below it, solved together along each column.

Definition at line 297 of file factorizations.hpp.

◆ cholesky_together_width

constexpr std::size_t numerical::detail::cholesky_together_width {16}
inlineconstexpr

The narrowest Cholesky diagonal block whose rows go together: in a block of 8, the groups' loops cost more than the chains save (order 8, one arm64 core: 0.21 → 0.23 µs).

Definition at line 270 of file factorizations.hpp.

◆ cholesky_update_columns

constexpr std::size_t numerical::detail::cholesky_update_columns {64}
inlineconstexpr

The fewest columns of the trailing block one Cholesky update's product takes: each product packs every row below its columns again, a panel's width apart the packing outweighed the products on the AMX (M1 Pro, order 128: 61 → 53 µs), and the triangle a wider block computes past the diagonal is wasted.

Definition at line 302 of file factorizations.hpp.

◆ cholesky_update_rows

constexpr std::size_t numerical::detail::cholesky_update_rows {256}
inlineconstexpr

The rows of one tile of a Cholesky update: the tiles of a panel's update, each one product on one thread, share the threads in one parallel loop, where a parallel product for each block of columns started and joined the threads some thirty times a panel.

Definition at line 307 of file factorizations.hpp.

◆ cluster_share

constexpr double numerical::detail::cluster_share {0.1}
inlineconstexpr

The share of the largest modulus (or of 1) within which eigenvalues gather (cluster_bound): Davies and Higham's 0.1, relative here.

Definition at line 1907 of file matrix_functions.hpp.

◆ cluster_share_fine

constexpr double numerical::detail::cluster_share_fine {0.01}
inlineconstexpr

The finer share funm falls back to where clusters at cluster_share leave its estimate past funm_fallback_error, or where, chained into irregular shapes (a random matrix's spectrum), a cluster's circle about its centre reaches another's eigenvalues, and no count of points helps (order 192: error 1.4).

Definition at line 1913 of file matrix_functions.hpp.

◆ column_rows

constexpr std::size_t numerical::detail::column_rows {128}
inlineconstexpr

The rows of a column-major matrix columns_times_vector takes at once: their partial sums (32 KB on any path) stay in the L1 cache, and a matrix of a few hundred rows still makes a block for each core (six AVX2 cores, 500 × 500 by a vector: 33 µs in two blocks of 256 rows, 19.6 in four of 128).

Definition at line 5188 of file gemm.hpp.

◆ column_sums_below

constexpr std::size_t numerical::detail::column_sums_below {9}
inlineconstexpr

The columns under which canonical_product sums a vector times a matrix by vector_times_columns, the same bits as its rows added in order: below it faster for every type from a depth of a hundred (a few nanoseconds either way under it), from it slower for a real type. On x86-64 the rows' vectors of eight float (AVX2) make the rows' loop the faster sooner.

Definition at line 1623 of file tensor.hpp.

◆ complex_row_columns

constexpr std::size_t numerical::detail::complex_row_columns {32}
inlineconstexpr

The columns of a task of complex_vector_times_columns divide: the most its kernels hold in registers at once (sixteen NEON registers of complex float).

Definition at line 1421 of file tensor.hpp.

◆ contour_points

constexpr std::size_t numerical::detail::contour_points {64}
inlineconstexpr

The trapezoidal rule's points on each circle of Cauchy's integral.

Definition at line 1994 of file matrix_functions.hpp.

◆ contour_points_most

constexpr std::size_t numerical::detail::contour_points_most {4096}
inlineconstexpr

The most points on one cluster's circle: funm doubles them from contour_points while the integrals' estimate passes contour_tolerance and halves at least (the trapezoidal rule converges geometrically on a circle: a cluster of 128 eigenvalues of a random matrix took 512 to 25 ε).

Definition at line 1999 of file matrix_functions.hpp.

◆ contour_tolerance

constexpr double numerical::detail::contour_tolerance {32.0 * std::numeric_limits<double>::epsilon()}
inlineconstexpr

The integrals' estimate below which funm takes no more points.

Definition at line 2002 of file matrix_functions.hpp.

◆ corrected_product_least

constexpr std::size_t numerical::detail::corrected_product_least {512}
inlineconstexpr

The rows from which corrected_hermitian_product shares its coefficients and bands with the threads: on six x86-64 AVX2 cores from 256 (below, eigh of order 128 took 13 % longer), on an M1 Pro from 512 (from 256, eigh of order 512 on eight cores took 3 to 9 % longer).

Definition at line 4579 of file factorizations.hpp.

◆ correlation_block

constexpr std::size_t numerical::detail::correlation_block {16}
inlineconstexpr

The lags numerical::correlate sums together: a block's sums held in registers (four AVX2 vectors of doubles) through the terms of v.

Definition at line 4824 of file tensor_ops.hpp.

◆ divide_leaf

constexpr std::size_t numerical::detail::divide_leaf {25}
inlineconstexpr

The order up to which tridiagonal_divide_and_conquer diagonalises by implicit QL.

Definition at line 5567 of file factorizations.hpp.

◆ divided_exponentials_past

constexpr double numerical::detail::divided_exponentials_past {64.0}
inlineconstexpr

The half gap past which exact_triangular_bands divides the difference of the exponentials by the gap: one exponential then outweighs the other by e¹²⁸, and the difference loses nothing, where the product e^{half sum} sinch(half gap) overflows sinh while the exponential underflows (a gap past about 1420).

Definition at line 272 of file matrix_functions.hpp.

◆ dot_chunk

constexpr std::size_t numerical::detail::dot_chunk {std::size_t {1} << 14U}
inlineconstexpr

The elements of one partial sum of chunked_dot: fixed, so the sum does not depend on the threads.

Definition at line 5045 of file gemm.hpp.

◆ dqds_sweeps

constexpr qd_index numerical::detail::dqds_sweeps {100}
inlineconstexpr

The steps of the inner iteration per value of its block before dqds gives up (LAPACK's 100).

Definition at line 64 of file dqds.hpp.

◆ dqds_tolerance_squared

constexpr double numerical::detail::dqds_tolerance_squared
inlineconstexpr
Initial value:
{(std::numeric_limits<double>::epsilon() * 100.0) *
(std::numeric_limits<double>::epsilon() * 100.0)}

LAPACK's bound of the off-diagonal against the diagonal under which a value deflates: (100 ε)².

Definition at line 60 of file dqds.hpp.

◆ elementwise_chunk

constexpr std::size_t numerical::detail::elementwise_chunk {work_per_thread / 4}
inlineconstexpr

The elements one task of an element-wise operation takes, a quarter of a thread's work: a tensor of a million shares out (M1 Pro, 2000 × 2000 doubles added: 3.1 → 1.4 ms), one of a few hundred thousand stays within its time (a sixteenth: 300 × 300 plus a row 13 % slower, each task setting up its odometer).

Definition at line 734 of file parallel.hpp.

◆ exceptional_after

constexpr lapack_index numerical::detail::exceptional_after {6}
inlineconstexpr

The iterations without deflation after which the shifts are exceptional (kexsh).

Definition at line 1300 of file eigen.hpp.

◆ exhaustive_path_operands

constexpr std::size_t numerical::detail::exhaustive_path_operands {4}
inlineconstexpr

The operands at most whose contraction order contraction_path searches exhaustively; more take the cheapest pair at each step.

Definition at line 2062 of file tensor.hpp.

◆ exp2_ln2

constexpr double numerical::detail::exp2_ln2 {0x1.62e42fefa39efp-1}
inlineconstexpr

ln 2 to the nearest double.

Definition at line 396 of file elementary.hpp.

◆ exp2_normal_bound

constexpr double numerical::detail::exp2_normal_bound {-1022.0}
inlineconstexpr

2^x below which exp2 is subnormal, where table_exp2's two scalings would round twice.

Definition at line 755 of file tensor_ops.hpp.

◆ exp_bound

template<std::floating_point R>
constexpr R numerical::detail::exp_bound {static_cast<R>(static_cast<int>(static_cast<R>(std::numeric_limits<R>::max_exponent - 1) * std::numbers::ln2_v<R>))}
inlineconstexpr

The real part past which exp of it overflows, rounded down: 709 for double, 88 for float.

Definition at line 37 of file complex_math.hpp.

◆ exp_inv_ln2_128

constexpr double numerical::detail::exp_inv_ln2_128 {0x1.71547652b82fep+7}
inlineconstexpr

128/ln 2 (tools/exp_table.py derives each constant here).

Definition at line 29 of file elementary.hpp.

◆ exp_ln2_128_high

constexpr double numerical::detail::exp_ln2_128_high {0x1.62e42fee00000p-8}
inlineconstexpr

ln 2/128 to 32 significant bits, toward zero: its product by an integer of 18 bits or fewer is exact, and so is that product's difference from an argument it reduces.

Definition at line 33 of file elementary.hpp.

◆ exp_ln2_128_low

constexpr double numerical::detail::exp_ln2_128_low {0x1.a39ef35793c76p-40}
inlineconstexpr

What exp_ln2_128_high leaves out of ln 2/128, to the nearest double.

Definition at line 36 of file elementary.hpp.

◆ exp_normal_bound

constexpr double numerical::detail::exp_normal_bound {-0x1.6232bdd7abcd2p+9}
inlineconstexpr

ln 2^-1022 toward zero: below it, exp is subnormal, where table_exp's two scalings would round twice.

Definition at line 39 of file elementary.hpp.

◆ exp_table

constexpr std::array<double, 256> numerical::detail::exp_table
constexpr

2^(j/128) for j = 0…127, at 2j the double nearest it and at 2j + 1 the double nearest what that one leaves out. Not inline: a copy in each translation unit, whose loads Clang's vectoriser bounds, where it kept a loop of table_exp scalar over an inline variable's ("cannot identify array bounds").

Definition at line 44 of file elementary.hpp.

◆ fft_column_block

constexpr std::size_t numerical::detail::fft_column_block {8}
inlineconstexpr

The lines a task of transform_axis takes together along an axis whose lines lie apart: each row read and written as that many neighbours, where one line at a time took one element of each row.

Definition at line 2124 of file fft.hpp.

◆ fft_four_step_band

template<std::floating_point R>
constexpr std::size_t numerical::detail::fft_four_step_band {256 / sizeof(std::complex<R>)}
inlineconstexpr

The rows of the four steps' working space a task transforms together, then writes down the columns of the output: 256 bytes of neighbours (four cache lines) for each of a row's elements.

Definition at line 901 of file fft.hpp.

◆ fft_four_step_least

constexpr std::size_t numerical::detail::fft_four_step_least {std::size_t {1} << 22}
inlineconstexpr

The fewest bytes of complex numbers a transform takes in four steps (four_step_plan), whose columns and rows the threads share: fewer stay in the caches through the passes of one transform, faster on a thread alone (ten AVX-512 cores: complex64 2¹⁸ points 1.5 ms in passes, 2.0 in four steps). From 4 MiB (complex128 2¹⁸ points: 5.0 ms in passes, 3.8 in four steps on all threads, 5.4 on one; M1 Pro, one thread, 3.6 against 2.1 ms). By the length and the type alone, so that a transform's bits are its own on any threads.

Definition at line 892 of file fft.hpp.

◆ fft_four_step_pad

constexpr std::size_t numerical::detail::fft_four_step_pad {8}
inlineconstexpr

The complex numbers past each row of the four steps' working space: rows a power of two of bytes apart fall in a few sets of the first-level cache, a column's elements evicting one another.

Definition at line 896 of file fft.hpp.

◆ fft_largest_radix

constexpr std::size_t numerical::detail::fft_largest_radix {61}
inlineconstexpr

The largest prime a Stockham pass takes as its radix, by its own r-point sum: a length with a larger prime factor is transformed by Bluestein's convolution.

Definition at line 804 of file fft.hpp.

◆ fft_lines_width

template<std::floating_point R>
constexpr std::size_t numerical::detail::fft_lines_width {1}
inlineconstexpr

One line at a time where no register holds lines.

Definition at line 2092 of file fft.hpp.

◆ fft_plans_kept

constexpr std::size_t numerical::detail::fft_plans_kept {32}
inlineconstexpr

The plans kept, the least recently used dropped past it: a plan in use lives on in its holders.

Definition at line 1108 of file fft.hpp.

◆ fft_root_cost

constexpr std::size_t numerical::detail::fft_root_cost {100}
inlineconstexpr

A root of unity's cost in parallel_for's units, about a nanosecond each: some 95 ns in double-double (unit_roots::parts).

Definition at line 1026 of file fft.hpp.

◆ fft_scratch_kept

constexpr std::size_t numerical::detail::fft_scratch_kept {std::size_t {1} << 16}
inlineconstexpr

The most elements fft_scratch keeps on a thread between transforms: 1 MiB of complex double.

Definition at line 1997 of file fft.hpp.

◆ fft_split_least

constexpr std::size_t numerical::detail::fft_split_least {8192}
inlineconstexpr

The fewest points of a length with a prime factor p past fft_largest_radix, not prime, that takes Bluestein's convolution for p alone, in two steps (four_step_plan: p points one way, n/p the other), n/p at least 16: fewer, or fewer than 16 transforms of p points, run the convolution over the whole length faster (ten AVX-512 cores: 1009·16 points 521 µs whole, 372 in two steps; 67·64 98 against 109; 67·1024 3.7 against 2.6 ms).

Definition at line 1009 of file fft.hpp.

◆ finite_chunk

constexpr std::size_t numerical::detail::finite_chunk {std::size_t {1} << 15U}
inlineconstexpr

The elements all_finite_shared gives a thread at a time.

Definition at line 654 of file factorizations.hpp.

◆ first_touch_chunk

constexpr std::size_t numerical::detail::first_touch_chunk {std::size_t {1} << 15}
inlineconstexpr

The elements a task of workspace_of touches first: 64 pages of 4 KiB, so a buffer of a few megabytes spreads over every core.

Definition at line 5731 of file factorizations.hpp.

◆ funm_batch_elements

constexpr std::size_t numerical::detail::funm_batch_elements {std::size_t {1} << 20}
inlineconstexpr

The elements of one batch of add_circle_terms's points (their p × p inverses, and as many of their shifted blocks): 2²⁰, 16 MiB each, and at least one point.

Definition at line 2120 of file matrix_functions.hpp.

◆ funm_fallback_error

constexpr double numerical::detail::funm_fallback_error {1e3 * std::numeric_limits<double>::epsilon()}
inlineconstexpr

The estimate past which funm tries the finer clusters (SciPy reports a result past it inaccurate).

Definition at line 1916 of file matrix_functions.hpp.

◆ gemm_kc

constexpr std::size_t numerical::detail::gemm_kc {256}
inlineconstexpr

The depth of one panel: a B sliver (gemm_kc × gemm_nr) stays in the L1 cache.

Definition at line 878 of file gemm.hpp.

◆ gemm_kernel_speedup

constexpr std::size_t numerical::detail::gemm_kernel_speedup {16}
inlineconstexpr

The multiply-adds the register tile does in the time of one in work_per_thread's dependent chain (about 20 G/s on one arm64 or x86-64 AVX2 core): a task weighs its work by it, so a product of 0.1 ms does not start eight threads.

Definition at line 893 of file gemm.hpp.

◆ gemm_mc

constexpr std::size_t numerical::detail::gemm_mc {64}
inlineconstexpr

The rows of one task: an A block (gemm_mc × gemm_kc) stays in the L2 cache.

Definition at line 881 of file gemm.hpp.

◆ gemm_min_depth

constexpr std::size_t numerical::detail::gemm_min_depth {8}
inlineconstexpr

Below this depth the plain loop runs too: each panel reads and writes its tiles of C once, which a shallow product does not amortise (1000×1000×1 ran six times slower packed, ×4 still lost 10 %).

Definition at line 915 of file gemm.hpp.

◆ gemm_mr

constexpr std::size_t numerical::detail::gemm_mr {4}
inlineconstexpr

The rows of C one register tile covers.

Definition at line 870 of file gemm.hpp.

◆ gemm_nc

template<typename T >
constexpr std::size_t numerical::detail::gemm_nc {gemm_nr<T> * 128}
inlineconstexpr

The columns of B packed at once, a whole number of slivers.

Definition at line 897 of file gemm.hpp.

◆ gemm_nr

template<typename T >
constexpr std::size_t numerical::detail::gemm_nr {sizeof(T) >= 16 ? std::size_t {6} : std::size_t {96} / sizeof(T)}
inlineconstexpr

The columns of C one register tile covers: 96 bytes of elements (12 double, 24 float), which with gemm_mr rows fills the vector registers of both arm64 and x86-64 with AVX.

Definition at line 875 of file gemm.hpp.

◆ gemm_packed_threshold

constexpr std::size_t numerical::detail::gemm_packed_threshold {std::size_t {1} << 13U}
inlineconstexpr

Below this many multiply-adds the plain loop (gemm_rows, by blocks of rows held in registers) runs instead of the packed kernel, both summing in the same order. Against the packed kernel on 400 random shapes from 512 multiply-adds, the plain loop took 0.25 to 0.41 times as long below 2^13 on arm64 (M1 Pro, geometric means by octave, the slowest shape 0.91); above, it lost up to 2.5 times on shapes the AMX suits. On x86-64 the plain loop has the baseline's vectors and the packed one the AVX2 or AVX-512 units: there it took 0.43 to 0.55 times as long below 2^11, 0.86 above with shapes up to 2 times slower (six AVX2 cores, GCC 13).

Definition at line 909 of file gemm.hpp.

◆ gemm_rows_columns

constexpr std::size_t numerical::detail::gemm_rows_columns {24}
inlineconstexpr

The most columns of C for which gemm runs any product by gemm_rows, whose blocks keep C's rows in registers over the whole depth: the packed kernel's tile spans 16 or more columns, so a narrower C left most of it idle (x86-64, 169 × 579 by 579 × 2: 82 µs, OpenBLAS 13). Both round each term as the path's tile does, so the same bits. Against the packed kernel on 200 random products of 1 to 24 columns: x86-64 0.42 to 0.64 times as long by width (ten AVX-512 cores, six AVX2 cores, GCC); arm64 under GCC 0.44 to 0.54; under Clang 0.60 up to 8 columns, while from 9 its packed tile kept pace and some products took twice as long by rows (M1 Pro).

Definition at line 1681 of file gemm.hpp.

◆ gemm_rows_share

constexpr std::size_t numerical::detail::gemm_rows_share {std::size_t {1} << 19U}
inlineconstexpr

The multiply-adds by rows a thread takes at most past gemm_rows_work, the blocks of rows shared out: the packed product's pass of packing and its tasks scale less well there (all threads, order 128: six AVX2 cores 31 → 24 µs, ten AVX-512 cores 32 → 21, eight arm64 cores under GCC 72 → 30; of 150 drawn products, those it moves 0.64, 0.58 and 0.38 times as long). On one thread the packed product stays, 2.3 times as fast at order 128 on AVX-512.

Definition at line 1749 of file gemm.hpp.

◆ gemm_rows_weight

constexpr std::size_t numerical::detail::gemm_rows_weight {2}
inlineconstexpr

The divisor of a block of gemm_rows' rows' multiply-adds as its threads are weighed: on x86-64 its work itself, so a product of order 64 takes the threads (six AVX2 cores: 11.5 → 7.4 µs; 1000 × 64 by 64 × 5 28.3 → 10.8); on arm64 half of it, where weighing the whole slowed narrow products a fifth (M1 Pro, 300 × 300 by 300 × 8 under Clang, 128 × 128 by 128 × 8 5.9 → 7.3 µs) and a quarter kept tall ones on one thread (eight threads: 4000 × 16 by 16 × 8 28 → 13 µs, 4000 × 64 by 64 × 2 58 → 26, 200 × 200 by 200 × 8 14 → 10; GCC 14 alike).

Definition at line 1708 of file gemm.hpp.

◆ gemm_rows_work

constexpr std::size_t numerical::detail::gemm_rows_work {std::size_t {1} << 20U}
inlineconstexpr

The most multiply-adds of a product gemm runs by rows (gemm_rows) whatever its width: up to there its rows' blocks, on the threads (gemm_rows_weight), finished before the packed kernel had packed and shared its work. Six AVX2 cores: order 64 14.5 → 7.2 µs, 80 20.3 → 8.9, 96 24.5 → 12.7, 40 × 60 by 60 × 100 14.2 → 7.5; arm64 under GCC order 80 32 → 15, 96 49 → 18. Under Clang on arm64 the packed tiles (the AMX's) kept small products ahead, order 64 3.3 µs against 12 by rows: none there (by_rows). Fewer rows than one of gemm_rows' blocks (narrow_group) stay packed, B then read in place (arm64 under GCC, 3 × 137 by 137 × 138: 1.48 times as long by rows). The same bits.

Definition at line 1695 of file gemm.hpp.

◆ gemm_slivers_per_task

constexpr std::size_t numerical::detail::gemm_slivers_per_task {4}
inlineconstexpr

The B slivers of one task.

Definition at line 884 of file gemm.hpp.

◆ half_exp_bound

template<std::floating_point R>
constexpr R numerical::detail::half_exp_bound {static_cast<R>(static_cast<int>(static_cast<R>(std::numeric_limits<R>::max_exponent - 1) * std::numbers::ln2_v<R> / R {2}))}
inlineconstexpr

Half of it, past which tanh's parts are ±1 and a quotient by exp(2x): 354 for double, 44 for float.

Definition at line 41 of file complex_math.hpp.

◆ half_pi_parts

constexpr std::array<double, 3> numerical::detail::half_pi_parts {0x1.921fb54442d18p+0, 0x1.1a62633145c07p-54, -0x1.f1976b7ed8fbcp-110}
inlineconstexpr

π/2 in three parts: its double, then the double of the rest, then the double of what is left (their sum within 6·10⁻⁵⁰ of π/2).

Definition at line 508 of file elementary.hpp.

◆ hessenberg_block

constexpr std::size_t numerical::detail::hessenberg_block {32}
inlineconstexpr

The columns of a panel of hessenberg_blocked.

Definition at line 325 of file eigen.hpp.

◆ hessenberg_crossover

constexpr std::size_t numerical::detail::hessenberg_crossover {128}
inlineconstexpr

The active order from which hessenberg reduces by panels.

Definition at line 328 of file eigen.hpp.

◆ inverse_band

constexpr std::size_t numerical::detail::inverse_band {16}
inlineconstexpr

The rows an LU inverse writes its identity into, and permutes the columns of, on one thread at a time.

Definition at line 230 of file factorizations.hpp.

◆ inverse_iteration_cluster_most

constexpr std::size_t numerical::detail::inverse_iteration_cluster_most {64}
inlineconstexprprivate

The most eigenvalues tridiagonal_eigenvectors reorthogonalises together: a larger cluster's vectors come from the whole spectrum's divide and conquer, n k² reorthogonalisation passing its n³'s share.

Definition at line 2680 of file svd.hpp.

◆ is_complex_number

template<typename T >
constexpr bool numerical::detail::is_complex_number< std::complex< R > > {false}
inlineconstexpr

Whether T is a std::complex of a floating-point type.

is_complex_number of std::complex<R>.

Definition at line 239 of file buffer.hpp.

◆ jacobi_large

constexpr double numerical::detail::jacobi_large {1e150}
inlineconstexpr

The |ζ| past which one-sided Jacobi takes the rotation's tangent as 1 / (2|ζ|): ζ² stays finite under it.

Definition at line 47 of file svd.hpp.

◆ jacobi_sweeps

constexpr std::size_t numerical::detail::jacobi_sweeps {60}
inlineconstexpr

The sweeps a Jacobi method may take: it converges quadratically, in under 15 on every matrix measured; the bound turns a non-convergence into an error instead of a hang.

Definition at line 4323 of file factorizations.hpp.

◆ jacobi_tiny

constexpr double numerical::detail::jacobi_tiny {std::numeric_limits<double>::min() / std::numeric_limits<double>::epsilon()}
inlineconstexpr

The sum of squares under which one-sided Jacobi judges a pair of columns over their norms: below it a square has lost digits to underflow (DBL_MIN / ε).

Definition at line 44 of file svd.hpp.

◆ kept_pages_largest

constexpr std::size_t numerical::detail::kept_pages_largest {std::size_t {1} << 25U}
inlineconstexpr

The bytes under which a buffer's block is a kept one, no huge pages asked: 32 MiB, four of them at most 128 MiB kept.

Definition at line 222 of file buffer.hpp.

◆ kept_pages_least

constexpr std::size_t numerical::detail::kept_pages_least {std::size_t {1} << 20U}
inlineconstexpr

The fewest bytes a buffer takes a kept block of 4 KiB pages for (kept_page_blocks): below, the allocator's own free lists reuse what is freed.

Definition at line 235 of file buffer.hpp.

◆ kept_workspace

constexpr std::size_t numerical::detail::kept_workspace {std::size_t {1} << 18}
inlineconstexpr

The most elements a buffer of a thread's merge_workspace keeps from one decomposition to the next: 2 MiB, every buffer an eigh of order 512 needs, so a thread keeps at most 8 MiB.

Definition at line 5756 of file factorizations.hpp.

◆ ldl_block

constexpr std::size_t numerical::detail::ldl_block {64}
inlineconstexpr

The columns of one panel of the blocked factorisation (bunch_kaufman_blocked_lower), LAPACK's nb for sytrf.

Definition at line 323 of file ldl.hpp.

◆ ldl_blocked_least

constexpr std::size_t numerical::detail::ldl_blocked_least {384}
inlineconstexpr

The order from which bunch_kaufman_lower runs by panels (bunch_kaufman_blocked_lower): below it the panels' columns, each brought up to date one dot product at a time, cost about what the unblocked steps' updates do (arm64, real: order 256 1.08× their time under Clang and 1.25× under GCC, 384 0.74× and 0.78×; complex 0.55× and 0.76× at 256).

Definition at line 329 of file ldl.hpp.

◆ log_inverse

constexpr std::array<double, 91> numerical::detail::log_inverse
constexpr

For k = 90…180, at k − 90 the double nearest 128/k. Not inline, as exp_table.

Definition at line 756 of file elementary.hpp.

◆ log_ln2_high

constexpr double numerical::detail::log_ln2_high {0x1.62e42fefa3800p-1}
inlineconstexpr

ln 2 to 42 significant bits, toward zero: its product by an exponent of 11 bits or fewer is exact (tools/log_table.py derives each constant here).

Definition at line 750 of file elementary.hpp.

◆ log_ln2_low

constexpr double numerical::detail::log_ln2_low {0x1.ef35793c76730p-45}
inlineconstexpr

What log_ln2_high leaves out of ln 2, to the nearest double.

Definition at line 753 of file elementary.hpp.

◆ log_pade_bounds

constexpr std::array<double, 8> numerical::detail::log_pade_bounds {0.0, 1.59e-5, 2.31e-3, 1.94e-2, 6.21e-2, 1.28e-1, 2.06e-1, 2.88e-1}
inlineconstexpr

The bounds θ_m of the [m/m] Padé approximant of log(I + X) (Al-Mohy and Higham's table 2.1, 2012), m from 1 to 7.

Definition at line 2676 of file matrix_functions.hpp.

◆ log_table

constexpr std::array<double, 182> numerical::detail::log_table
constexpr

For k = 90…180, at 2(k − 90) −ln of log_inverse's entry to the nearest multiple of 2^−42 (its sum with an exponent times log_ln2_high exact) and at 2(k − 90) + 1 the double nearest what that leaves out. Not inline, as exp_table.

Definition at line 793 of file elementary.hpp.

◆ lowner_block

constexpr std::size_t numerical::detail::lowner_block {32}
inlineconstexpr

The poles a task of merge_rank_one's Löwner products takes, and the side of its tiles of u.

Definition at line 5710 of file factorizations.hpp.

◆ lu_block

constexpr std::size_t numerical::detail::lu_block {64}
inlineconstexpr

The columns of one panel of the blocked LU: the trailing update, a product of this depth, is deep enough for the packed kernel while the unblocked panel stays a small part of the work.

Definition at line 145 of file factorizations.hpp.

◆ lu_fused_columns

constexpr std::size_t numerical::detail::lu_fused_columns {512}
inlineconstexpr

The trailing columns at most past which each block of the update beside the panel solves its own rows of U₁₂: a wider update keeps every thread busy past the panel, and its rows solved on all of them first ran faster than on the workers alone (order 1000 on four AVX2 cores, 2 %).

Definition at line 192 of file factorizations.hpp.

◆ lu_lookahead_columns

constexpr std::size_t numerical::detail::lu_lookahead_columns {1024}
inlineconstexpr

The trailing columns at most past which the LU factorises its next panel beside the update of the rest (a look-ahead): a wider update runs well as one product, and cut in blocks beside the panel it ran longer (order 2048 on six x86-64 cores, 9 %); a narrower one leaves threads idle while the panel runs.

Definition at line 187 of file factorizations.hpp.

◆ lu_small_block

constexpr std::size_t numerical::detail::lu_small_block {32}
inlineconstexpr

The columns of one panel of the LU of an order up to lu_small_order.

Definition at line 161 of file factorizations.hpp.

◆ lu_small_order

constexpr std::size_t numerical::detail::lu_small_order {256}
inlineconstexpr

The order up to which the LU takes panels of lu_small_block columns: there the panels, a column at a time on one thread, are most of the time, and narrower ones halve their work (arm64, order 300 on every core: 14 % slower at 32 columns than at 64).

Definition at line 157 of file factorizations.hpp.

◆ lu_tall_ratio

constexpr std::size_t numerical::detail::lu_tall_ratio {2}
inlineconstexpr

The ratio of rows to columns up to which lu_factors factorises a tall matrix as the square of it beside zero columns (m³ / 3, by the blocked LU), past which by tall_lu_columns (m n² on narrow products): the two even near twice as tall as wide (arm64, 1200 rows).

Definition at line 2492 of file factorizations.hpp.

◆ lu_update_speedup

constexpr std::size_t numerical::detail::lu_update_speedup {gemm_kernel_speedup}
inlineconstexpr

The multiply-adds of an LU's trailing update one of the tile's dependent ones stands for, as threads are weighed (gemm_kernel_speedup): the AMX tile's, on arm64; the AVX tiles' quarter of it on x86-64, where the update of order 256 waited on one thread (six cores: 690 → 622 µs, 1024: 12.5 → 12.1 ms).

Definition at line 200 of file factorizations.hpp.

◆ lu_wide_block

constexpr std::size_t numerical::detail::lu_wide_block {128}
inlineconstexpr

The columns of one panel of the LU from lu_wide_order.

Definition at line 171 of file factorizations.hpp.

◆ lu_wide_order

constexpr std::size_t numerical::detail::lu_wide_order {3072}
inlineconstexpr

The order from which the LU takes panels of lu_wide_block columns: each trailing update rereads the whole trailing block, and from there the memory traffic of panels of 64, not their arithmetic, sets the time; 128 halves it. Order 4096: ten AVX-512 cores 322 → 273 ms, six AVX2 cores 416 → 330, eight arm64 cores 285 → 252; order 3072 10 to 17 % faster; 2560 the same; 2048 10 % slower, the wider panel's own work then the longer. The factors are the same to the bit at either width.

Definition at line 168 of file factorizations.hpp.

◆ matrix_vector_weight

constexpr std::size_t numerical::detail::matrix_vector_weight {1}
inlineconstexpr

The weight of a matrix element in matrix_times_vector's share of the threads, beside its multiply-add: each element read once, the product the memory's, so more threads read it. On x86-64, 4: six AVX2 cores, a double matrix of 1000 × 100 times a vector 19.1 → 6.6 µs, 256 × 256 10.6 → 5.2, a complex one of 671 × 36 16.4 → 6.1, 539 × 68 14.3 → 9.8, the smallest the same. On arm64, 1: the M1 Pro's complex products took longer on more threads (100 × 100 3.1 → 7.0 µs), its double ones the same.

Definition at line 5131 of file gemm.hpp.

◆ memory_element_weight

constexpr std::size_t numerical::detail::memory_element_weight {4}
inlineconstexpr

The weight of an element a pass bound by the memory writes or moves, against a multiply-add, as the BLAS routines' transposing copies (weighed as one, an LU inverse of order 300 permuted its columns on one thread).

Definition at line 760 of file parallel.hpp.

◆ multiply_add_weight

template<typename T >
constexpr std::size_t numerical::detail::multiply_add_weight {1}
inlineconstexpr

The real multiply-adds one multiply-add of T weighs: 1 for a real number.

Definition at line 5108 of file gemm.hpp.

◆ multishift_crossover

constexpr lapack_index numerical::detail::multishift_crossover {75}
inlineconstexpr

LAPACK's nmin: the active order below which dlaqr0's iterations skip a sweep after enough deflation.

Definition at line 1279 of file eigen.hpp.

◆ narrow_columns

constexpr std::size_t numerical::detail::narrow_columns {8}
inlineconstexpr

The columns of C one block of gemm_rows keeps in registers over the whole depth.

Definition at line 1669 of file gemm.hpp.

◆ narrow_group

constexpr std::size_t numerical::detail::narrow_group {4}
inlineconstexpr

The rows of C one block of gemm_rows takes together: each element a chain of multiply-adds that waits on its last, four rows give the units four times the chains (M1 Pro, order 8 to 24: 1.6 to 1.9 times as fast as one row; two rows on six x86-64 AVX2 cores, the fastest there, were within 7 %).

Definition at line 1742 of file gemm.hpp.

◆ narrow_least_depth

constexpr std::size_t numerical::detail::narrow_least_depth {2}
inlineconstexpr

The least depth at which rows of C wider than one block (narrow_columns) run by blocks in gemm_rows; one block or narrower takes any depth past 1. Against the plain loop of unknown width, on 500 random shapes below the packed kernel, the blocks took 0.11 to 0.63 times as long from a depth of 2 on arm64 (M1 Pro, geometric means by width); on x86-64, where the loop has the baseline's vectors, 0.35 to 0.90 from a depth of 8 but 1.08 to 1.31 below it past one block (six AVX2 cores, GCC 13).

Definition at line 1665 of file gemm.hpp.

◆ narrow_solve

constexpr std::size_t numerical::detail::narrow_solve {16}
inlineconstexpr

The most right-hand sides an LU solve takes row by row rather than by blocks through the product kernel.

Definition at line 204 of file factorizations.hpp.

◆ neon_compiled

constexpr bool numerical::detail::neon_compiled {false}
inlineconstexpr

Whether the NEON path's tile is compiled in: on every arm64 target GCC or Clang builds.

Definition at line 98 of file gemm.hpp.

◆ nibble

constexpr lapack_index numerical::detail::nibble {14}
inlineconstexpr

The share of the window, in percent, whose deflation makes the sweep unnecessary (nibble).

Definition at line 1303 of file eigen.hpp.

◆ no_core

constexpr std::size_t numerical::detail::no_core {std::numeric_limits<std::size_t>::max()}
inlineconstexpr

No core: a worker free to run anywhere, or a caller on a processor the pool does not know.

Definition at line 64 of file parallel.hpp.

◆ packed_line

constexpr std::size_t numerical::detail::packed_line {64}
inlineconstexpr

The cache line a packed operand starts.

Definition at line 2366 of file gemm.hpp.

◆ pairwise_columns

constexpr std::size_t numerical::detail::pairwise_columns {512}
inlineconstexpr

The columns pairwise_sum_rows sums at once: its partial sums of eight rows, 32 KiB of doubles, stay in the L1 cache.

Definition at line 1183 of file tensor.hpp.

◆ pairwise_lanes

constexpr std::size_t numerical::detail::pairwise_lanes {8}
inlineconstexpr

The partial sums pairwise_sum interleaves, and the multiple of which its first half is long.

Definition at line 969 of file tensor.hpp.

◆ pairwise_leaf

constexpr std::size_t numerical::detail::pairwise_leaf {std::size_t {1} << 15U}
inlineconstexpr

The longest subtree of pairwise_sum's tree parallel_pairwise_sum sums on one thread: 256 KiB of double.

Definition at line 1066 of file tensor.hpp.

◆ pencil_block

constexpr std::size_t numerical::detail::pencil_block {64}
inlineconstexpr

The columns reduce_pencil reduces a block at a time, as LAPACK's sygst by its block size.

Definition at line 6379 of file factorizations.hpp.

◆ pencil_reduction_least

template<typename T >
constexpr std::size_t numerical::detail::pencil_reduction_least {is_complex_number<T> ? 3072 : 2048}
inlineconstexpr

The order from which a pencil's standard form is reduced by blocks (reduce_pencil): half the operations of two whole triangular solves, but a block at a time, where the two solves share their columns among the threads whole. Below it the solves ran faster: ten AVX-512 cores up to order 1100 or so, six AVX2 cores up to 500, the M1 Pro (its products on AMX) up to some 2000 real and 2500 complex. From it: order 3072, six AVX2 cores 683 → 335 ms (complex 1749 → 884), ten AVX-512 cores 423 → 224 ms (complex 900 → 528), M1 Pro 204 → 161 ms (complex 724 → 684). One order whatever the threads: the bits of C do not depend on their number.

Definition at line 6554 of file factorizations.hpp.

◆ plain_loop_least_width

constexpr std::size_t numerical::detail::plain_loop_least_width {64}
inlineconstexpr

The fewest columns of a product plain_loop_pays sends to the plain loop on the x86-64 tiles: there an order-24 SVD, whose two products of 24 × 24 × 24 it took, ran 9 % slower (GCC 13, the same bits), though such a product alone ran faster in the plain loop. On the NEON tile the small square products gain most (GCC 14, one core: a product of order 24 to 64 −32 to −50 %, a QR −5 to −21 %).

Definition at line 4360 of file gemm.hpp.

◆ plain_squares_high

constexpr double numerical::detail::plain_squares_high {0x1p900}
inlineconstexpr

The largest plain sum of squares plain_or_at_scale keeps: no term of it overflowed.

Definition at line 518 of file factorizations.hpp.

◆ plain_squares_low

constexpr double numerical::detail::plain_squares_low {0x1p-900}
inlineconstexpr

The smallest plain sum of squares plain_or_at_scale keeps: a term that underflowed is below 2⁻¹⁰²², so a sum this large loses at most count · 2⁻¹²² of itself to them.

Definition at line 515 of file factorizations.hpp.

◆ power_pade_bounds

constexpr std::array<double, 8> numerical::detail::power_pade_bounds {0.0, 1.51e-5, 2.24e-3, 1.88e-2, 6.04e-2, 1.24e-1, 2.00e-1, 2.79e-1}
inlineconstexpr

The bounds θ_m of the [m/m] Padé approximant of (I − X)ᵗ (Higham and Lin's table 3.1, 2013, as SciPy's _remainder_matrix_power_triu), m from 1 to 7.

Definition at line 2680 of file matrix_functions.hpp.

◆ prefetches_next_row_of_c

template<typename Kernel >
constexpr bool numerical::detail::prefetches_next_row_of_c
inlineconstexpr
Initial value:
{requires(const double* c, std::size_t ldc) {
Kernel::prefetch_rows(c, ldc);
}}

Whether Kernel has the C tile of the next row of tiles prefetched while a tile computes (Kernel::prefetch_rows): the tiles within a row walk one page's lines, which the processor's own prefetch follows, while the next row lands on other pages.

Definition at line 2470 of file gemm.hpp.

◆ product_rows

constexpr std::size_t numerical::detail::product_rows {8}
inlineconstexpr

The rows of y subtract_rows takes at once.

Definition at line 731 of file simd.hpp.

◆ ql_steps_per_value

constexpr std::size_t numerical::detail::ql_steps_per_value {30}
inlineconstexpr

The implicit QL steps the tridiagonal eigensolver may take per eigenvalue, as LAPACK's dsteqr bounds them (30 n in all): it converges cubically, in two or three per eigenvalue.

Definition at line 4763 of file factorizations.hpp.

◆ qr_block

constexpr std::size_t numerical::detail::qr_block {32}
inlineconstexprprivate

The reflectors of one panel of the blocked QR.

Definition at line 3250 of file factorizations.hpp.

◆ qr_look_ahead_threads

constexpr std::size_t numerical::detail::qr_look_ahead_threads {3}
inlineconstexprprivate

The fewest threads for which reflect_in_columns looks ahead, a panel factorised while the threads update the columns past it: on two, one thread would update the rest alone (ten AVX-512 cores, order 1024: 29.4 ms as it is, 36.3 looking ahead), from three a panel no longer waits for the update (four threads 22.1 → 16.5 ms, eight 19.8 → 13.9).

Definition at line 3305 of file factorizations.hpp.

◆ qr_rank_margin

constexpr double numerical::detail::qr_rank_margin {100.0}
inlineconstexprprivate

How far above the rank cutoff a bound on the smallest singular value must stay for lstsq_by_qr to answer: nearer, the singular value decomposition decides the rank.

Definition at line 1982 of file svd.hpp.

◆ qr_transpose_tile

constexpr std::size_t numerical::detail::qr_transpose_tile {32}
inlineconstexprprivate

The side of the tiles in which the QR transposes (transpose_into): a tile's rows and columns both stay in the cache, where a whole column read across rows misses on every element.

Definition at line 3384 of file factorizations.hpp.

◆ qr_update_slice

constexpr std::size_t numerical::detail::qr_update_slice {32}
inlineconstexprprivate

The fewest columns a task of reflect_in_columns' look-ahead updates: each task repacks the block, which narrow slices pay for too often (fixed slices of 64 columns on one thread, order 1024: 41 → 61 ms; with a slice per thread, a least of 32 or of 64 alike).

Definition at line 3310 of file factorizations.hpp.

◆ quarter_pi

constexpr double_double numerical::detail::quarter_pi {0x1.921fb54442d18p-1, 0x1.1a62633145c07p-55}
inlineconstexpr

π/4 as a double-double number.

Definition at line 123 of file fft.hpp.

◆ radix_sort_least

constexpr std::size_t numerical::detail::radix_sort_least {256}
inlineconstexpr

Runs this long or longer are sorted by their bits (radix_sort); shorter ones by comparisons.

Definition at line 3500 of file tensor_ops.hpp.

◆ reads_a_in_place

template<typename Kernel , typename T >
constexpr bool numerical::detail::reads_a_in_place
inlineconstexpr
Initial value:
{requires(const T * x, T * y, std::size_t z) {
Kernel::tile_a_in_place(z, x, z, z, x, z, y, z, z, z);
}}

Whether Kernel reads A where it lies (Kernel::tile_a_in_place).

Definition at line 2503 of file gemm.hpp.

◆ reads_partial_b_in_place

template<typename Kernel >
constexpr bool numerical::detail::reads_partial_b_in_place {requires { requires Kernel::partial_b_in_place; }}
inlineconstexpr

Whether Kernel reads B's last partial sliver in place (Kernel::partial_b_in_place): a tile that stores only its columns inside C, its loads past them landing in accumulators it never stores (the AMX's).

Definition at line 2510 of file gemm.hpp.

◆ real_rows

template<typename T >
constexpr bool numerical::detail::real_rows {std::is_same_v<T, double> || std::is_same_v<T, float>}
inlineconstexpr

Whether a solve of T goes by rows of real terms (lu_decomposition's solve_narrow): double and float.

Definition at line 785 of file factorizations.hpp.

◆ rescue_below

template<typename T >
constexpr carried_t<T> numerical::detail::rescue_below {std::is_same_v<value_range_t<T>, float> ? 0x1p-60 : 0x1p-500}
inlineconstexpr

The magnitude under which rescue_factor scales a T up: about the square root of the smallest normal number of its range, or below it (2⁻⁵⁰⁰ for a long double, whose range is the platform's).

Definition at line 383 of file factorizations.hpp.

◆ row_group

constexpr std::size_t numerical::detail::row_group {64}
inlineconstexpr

The rows a thread takes of a matrix times a vector (matrix_times_vector).

Definition at line 5119 of file gemm.hpp.

◆ schur_steps_per_order

constexpr std::size_t numerical::detail::schur_steps_per_order {30}
inlineconstexpr

The iterations one eigenvalue (or pair) may take per unit of the active order, as dlahqr's 30.

Definition at line 44 of file eigen.hpp.

◆ schur_ulp

constexpr double numerical::detail::schur_ulp {std::numeric_limits<double>::epsilon()}
inlineconstexpr

The relative spacing LAPACK's ‘dlamch('P’)` names: the roundoff a QR sweep judges a subdiagonal by.

Definition at line 41 of file eigen.hpp.

◆ secular_iterations

constexpr std::size_t numerical::detail::secular_iterations {100}
inlineconstexpr

The iterations one root of the secular equation may take: it converges in a few; the bound turns a non-convergence into an error instead of a hang.

Definition at line 5571 of file factorizations.hpp.

◆ select_positions_sorted

constexpr std::size_t numerical::detail::select_positions_sorted {8}
inlineconstexpr

A line with more positions than these is sorted whole rather than selected position by position.

Definition at line 4661 of file tensor_ops.hpp.

◆ sign_bit

template<typename U >
constexpr U numerical::detail::sign_bit {U {1} << ((sizeof(U) * 8) - 1)}
inlineconstexpr

The highest bit of U, a real's or an integer's sign.

Definition at line 3508 of file tensor_ops.hpp.

◆ sine_inverse_step

constexpr double numerical::detail::sine_inverse_step {0x1.45f306dc9c883p+3}
inlineconstexpr

32/π, the double nearest it (tools/sine_table.py derives each constant here).

Definition at line 543 of file elementary.hpp.

◆ sine_reduction_bound

constexpr double numerical::detail::sine_reduction_bound {0x1p20}
inlineconstexpr

The magnitude to which reduced_sine reduces its argument: k below 2²⁰, k·π/2's parts exact enough.

Definition at line 511 of file elementary.hpp.

◆ sine_step_parts

constexpr std::array<double, 3> numerical::detail::sine_step_parts {0x1.921fb54442d18p-4, 0x1.1a62633145c07p-58, -0x1.f1976b7ed8fbcp-114}
inlineconstexpr

π/32 in three parts: its double, then the double of the rest, then the double of what is left.

Definition at line 546 of file elementary.hpp.

◆ sine_table

constexpr std::array<double, 256> numerical::detail::sine_table
constexpr

For j = 0…63, at 4j sin(jπ/32) and at 4j + 1 what that double leaves out, at 4j + 2 cos(jπ/32) and at 4j + 3 what that one leaves out, each the double nearest. Not inline, as exp_table.

Definition at line 550 of file elementary.hpp.

◆ slice_columns

constexpr std::size_t numerical::detail::slice_columns {64}
inlineconstexpr

The fewest columns an LU solve gives a thread of its own: a narrower slice starves the product kernel (arm64, order 256 and 64 right-hand sides in eight slices: 112 → 194 µs; 48 columns, order 300 and 300 right-hand sides: 8 % slower than 64).

Definition at line 215 of file factorizations.hpp.

◆ solve_leaf

constexpr std::size_t numerical::detail::solve_leaf {16}
inlineconstexpr

The columns of one leaf of the recursive LU panel.

Definition at line 227 of file factorizations.hpp.

◆ sort_rows

constexpr std::size_t numerical::detail::sort_rows {64}
inlineconstexpr

The rows a task of merge_arrow reorders in place, one row's buffer between them.

Definition at line 928 of file svd.hpp.

◆ spin_time

constexpr std::chrono::microseconds numerical::detail::spin_time {500}
inlineconstexpr

How long a thread polls before it sleeps: kernels run in quick succession (a matrix product packs, then computes, panel after panel), and a thread woken from sleep starts tens of microseconds late, its wake-up paid by the caller of the next run, about 10 µs a sleeping worker on six x86-64 cores. A factorisation's steps last about 100 µs at order 256, so at 100 µs the workers one step left out were asleep for the next (an LU of order 256: six x86-64 cores 560 → 500 µs at 500; eight arm64 cores 358 → 324).

Definition at line 317 of file parallel.hpp.

◆ split_columns

constexpr std::size_t numerical::detail::split_columns {512}
inlineconstexpr

The columns of B and C, and the depth, of one real product of gemm_split: its real operands stay a few megabytes whatever the shapes.

Definition at line 3877 of file gemm.hpp.

◆ split_depth

constexpr std::size_t numerical::detail::split_depth {256}
inlineconstexpr

The depth of one real product of gemm_split (split_columns).

Definition at line 3880 of file gemm.hpp.

◆ split_kept

constexpr std::size_t numerical::detail::split_kept {std::size_t {1} << 18}
inlineconstexpr

The most elements an operand of gemm_split keeps from one product to the next on a thread: 2 MiB of double, every operand of a product of order 512; a larger one is let go after its product.

Definition at line 4148 of file gemm.hpp.

◆ split_narrow_area

constexpr std::size_t numerical::detail::split_narrow_area {1024}
inlineconstexpr

The largest n·k of a complex product gemm_split shares out by blocks of rows: the real product of its expanded parts then cannot share its own threads. Past it, a product of depth 64 by 64 columns, which that product shares, ran slower so on eight GCC threads (300 rows: 86 → 124 µs).

Definition at line 4179 of file gemm.hpp.

◆ split_parts_rows

constexpr std::size_t numerical::detail::split_parts_rows {64}
inlineconstexpr

The most rows of A for which gemm_split runs four real products of a block's parts on the AMX path (split_by_parts). M1 Pro, all threads, A of 16 to 64 rows by B of 256 to 1024: complex float 1.1 to 2.4 times faster, complex double 0.97 to 1.6; 96 to 128 rows by 512, complex double 1.2 to 1.4 times slower, where one wide product suited the AMX.

Definition at line 4154 of file gemm.hpp.

◆ split_row_width

constexpr std::size_t numerical::detail::split_row_width {32}
inlineconstexpr

The columns split_row_columns holds at once: a block of B's rows read twice stays in the caches.

Definition at line 3884 of file gemm.hpp.

◆ square_root_block

constexpr std::size_t numerical::detail::square_root_block {64}
inlineconstexpr

The order of the diagonal blocks of triangular_square_root (SciPy's _sqrtm_triu's).

Definition at line 1324 of file matrix_functions.hpp.

◆ strided_columns

constexpr std::size_t numerical::detail::strided_columns {8}
inlineconstexpr

The columns of B one task of vector_times_columns sums together: their sums' chains interleaved, each waiting on its own last addition.

Definition at line 1269 of file tensor.hpp.

◆ sturm_bisection_work

constexpr std::size_t numerical::detail::sturm_bisection_work {256}
inlineconstexprprivate

One eigenvalue's bisection in parallel_for's units per order of the matrix: some 60 halvings, each a Sturm count whose step divides (some four multiply-adds).

Definition at line 2424 of file svd.hpp.

◆ svd_band

constexpr std::size_t numerical::detail::svd_band {32}
inlineconstexpr

The rows of an SVD's input one thread scales and copies at a time.

Definition at line 233 of file factorizations.hpp.

◆ svd_qr_steps_per_value

constexpr std::size_t numerical::detail::svd_qr_steps_per_value {30}
inlineconstexpr

The QR steps the bidiagonal singular value iteration may take per value: it converges in two or three; the bound turns a non-convergence into an error instead of a hang.

Definition at line 186 of file svd.hpp.

◆ symmetric_row_group

constexpr std::size_t numerical::detail::symmetric_row_group {4}
inlineconstexpr

The rows of a lower triangle symmetric_rows takes at once.

Definition at line 158 of file simd.hpp.

◆ tasks_per_thread

constexpr std::size_t numerical::detail::tasks_per_thread {4}
inlineconstexpr

The tasks a product's compute pass hands each thread at least, while its rows allow: a slower core then holds back no one.

Definition at line 888 of file gemm.hpp.

◆ taylor_bounds

constexpr std::array<taylor_bound, 35> numerical::detail::taylor_bounds
inlineconstexpr

The table of taylor_bound, m ascending.

Definition at line 491 of file matrix_functions.hpp.

◆ term_work

template<typename T >
constexpr std::size_t numerical::detail::term_work {is_complex_number<T> ? 4 : 1}
inlineconstexpr

The real multiply-adds of a term a · b of T, as parallel_for counts work: four for a complex one.

Definition at line 1265 of file tensor.hpp.

◆ thread_release_places

constexpr std::size_t numerical::detail::thread_release_places {256}
inlineconstexpr

The most workspaces a thread registers (on_release): one per kind, scalar type and slot, a few dozen where every type is used (the C++ tests' thread: 34), so that registering allocates nothing (their accessors throw nothing).

Definition at line 45 of file buffer.hpp.

◆ transpose_tile

constexpr std::size_t numerical::detail::transpose_tile {32}
inlineconstexpr

The side of the tiles a transposing copy goes by (an einsum permutation of two axes, the eigenvectors' columns): a tile read and a tile written fit the L1 cache, where element by element each write, a row apart, missed it (order 1024 on arm64: 4.7 → 1.5 ms; the eigenvectors of order 1024, a tenth of an eigh on six x86-64 cores).

Definition at line 89 of file simd.hpp.

◆ triangular_block

constexpr std::size_t numerical::detail::triangular_block {32}
inlineconstexpr

The rows triangular_halves solves one by one; past them, by halves of a multiple of them.

Definition at line 1843 of file factorizations.hpp.

◆ triangular_slice

constexpr std::size_t numerical::detail::triangular_slice {64}
inlineconstexpr

The columns of the right-hand sides a thread takes in triangular_halves' rows one by one.

Definition at line 1846 of file factorizations.hpp.

◆ tridiagonal_band

constexpr std::size_t numerical::detail::tridiagonal_band {512}
inlineconstexpr

The rows of each band of the tridiagonal reduction's trailing update: a band reaches its diagonal, so a wider one computes more of the upper triangle the reduction never reads, a narrower one runs the product kernel on smaller products.

Definition at line 4759 of file factorizations.hpp.

◆ tridiagonal_block

constexpr std::size_t numerical::detail::tridiagonal_block {32}
inlineconstexpr

The reflections of one panel of the blocked tridiagonal reduction.

Definition at line 4754 of file factorizations.hpp.

◆ upper_block_product_least

constexpr std::size_t numerical::detail::upper_block_product_least {64}
inlineconstexpr

The blocks' size (rows × columns) from which solve_upper_blocks forms a block's terms by gemm: below it, a call costs more than its sums (a random matrix's clusters, most of one eigenvalue).

Definition at line 1602 of file matrix_functions.hpp.

◆ vector_chunk

constexpr std::size_t numerical::detail::vector_chunk {128}
inlineconstexpr

The columns a thread takes of a vector times a matrix: each of its rows added to them in order.

Definition at line 1261 of file tensor.hpp.

◆ vector_speedup

constexpr std::size_t numerical::detail::vector_speedup {4}
inlineconstexpr

The multiply-adds a vectorised inner loop does in the time of one in work_per_thread's dependent chain: a product weighs its work by it (at order 64, a matrix product on eight threads took 95 µs, on one 35 µs).

Definition at line 739 of file parallel.hpp.

◆ window_growth

constexpr lapack_index numerical::detail::window_growth {5}
inlineconstexpr

The iterations without deflation after which the window grows (kexnw).

Definition at line 1297 of file eigen.hpp.

◆ work_per_thread

constexpr std::size_t numerical::detail::work_per_thread {std::size_t {1} << 16}
inlineconstexpr

The work one thread must have to pay for starting it: about 60 µs of multiply-adds, six times the cost of a start (measured on arm64: 16 indices of 4096 ran 80 µs sequentially, 92 µs on eight threads; of 16384, 324 µs and 128 µs).

Definition at line 729 of file parallel.hpp.

◆ writes_fresh

template<typename Kernel , typename T >
constexpr bool numerical::detail::writes_fresh
inlineconstexpr
Initial value:
{requires(const T * x, T * y, std::size_t z) {
Kernel::tile(z, x, x, z, y, z, z, z, true);
}}

Whether Kernel's tiles write C as it holds nothing, their sums from zero (Kernel::tile's fresh): C then not zeroed first.

Definition at line 2496 of file gemm.hpp.

◆ zero_chunk

constexpr std::size_t numerical::detail::zero_chunk {std::size_t {1} << 12}
inlineconstexpr

The elements one index of parallel_zero writes: eight 4 KiB pages of double.

Definition at line 853 of file parallel.hpp.

◆ zero_work

constexpr std::size_t numerical::detail::zero_work {1}
inlineconstexpr

The work of zeroing one element, in multiply-adds: a second thread from 1 MiB of double. Below, the allocator mostly hands back memory already touched, and zeroing 512 KiB took 4.5 µs on one arm64 thread, 9 µs split across eight.

Definition at line 880 of file parallel.hpp.

◆ zeroed_rows

constexpr std::size_t numerical::detail::zeroed_rows {32}
inlineconstexpr

The rows of a Cholesky factor whose upper part one thread zeroes at a time.

Definition at line 246 of file factorizations.hpp.