This manuscript contains preprint of a chapter under consideration for inclusion in the forthcoming third edition of {\em Cover and Thomas's Elements of Information Theory}, posted with permission from Wiley. The table of contents EIT-3 ToC of the new edition can be found at: https://docs.google.com/document/d/1L-m4oQEJw1PJhoxBeMwrrBD8S_HmvzMEkPbYvS24980/edit?usp=sharing . For feedback, please contact abbas@ee.stanford.edu Learning and information theory intersect in both model training and the characterization of fundamental performance limits. This manuscript provides a concise and accessible treatment of the first intersection, requiring only basic background in information theory and statistics at the senior undergraduate or first-year graduate level. End-of-chapter exercises make the material well suited for classroom use as well as self-study. The chapter focuses on the role of divergence measures in model training, with examples ranging from linear and logistic regression to autoregressive models, variational autoencoders, diffusion models, generative adversarial networks, and score-based models. It introduces the evidence lower bound (ELBO), f-divergences, and the Fisher divergence. In particular, the treatment of the generative diffusion model provides a more systematic and explicit derivation than is typical in the literature.

We introduce the degree filtration on the discrete cubical chain complex of a graph, defined in terms of the maximal injective dimension of the facets of singular $n$-cubes, and study the degree spectral sequence which arises from this filtration. This spectral sequence interpolates between the discrete cubical homology of a graph $H_n(G)$ and the injective homology $H_n^{inj}(G)$, a variant of the discrete cubical homology based on injective singular cubes. Building on the work of Babson et al. we introduce the combinatorial condition of quasimonophobicity on graphs, and show quasimonophobicity implies both the vanishing of the degree spectral sequence in certain bidegrees, and implies $H_n^{inj}(G)$ is isomorphic to the homology of the CW complex obtained by ``filling in'' subcubes of the graph. These results are applied to compute $H_2(G_n^{sph})$ for the Greene sphere graphs $G^{sph}_n$.

We consider a general class of ``inexactly smooth'' convex functions, providing a universal model capturing as special cases $L$-smooth, $M$-Lipschitz, and Hölder smooth functions, and any combination thereof. Such functions possess a calculus closely following that of smooth functions. Our main results provide inexactly smooth functions with interpolation theorems that are necessary and sufficient up to modest universal constants. These enable analysis of first-order methods for any inexactly smooth convex problem class via solving convex Performance Estimation Problems (PEPs). Further, these enable the extension of Drori and Taylor's constructive approach to algorithm design. From this, we derive an exactly minimax optimal method for $(β,0)$-Hölder smooth problems, methods with the best-known convergence guarantees up to constants for any $(β,p)$-Hölder smooth convex minimization, and a new universal fast backtracking method for any inexactly smooth convex problem.

Many learning tasks in science and engineering are characterized by sparse datasets, which limits the effectiveness of purely data-driven approaches. At the same time, these problems are often accompanied by rich domain knowledge derived from physical laws, operational requirements, and expert heuristics. Such knowledge is frequently expressed as rules involving logical propositions and linear inequalities. Existing neuro-symbolic methods typically enforce these rules approximately through soft penalties, assume input-independent rules when designing specialized architectures, or rely on non-differentiable post-processing at inference time to achieve hard constraint satisfaction. While recent advances in differentiable optimization layers enable end-to-end feasibility enforcement within neural networks, extending these approaches to logical or mixed-integer rules remains challenging due to inherent nonconvexity. In this work, we propose a unified end-to-end framework for enforcing hard, input-dependent mixed integer linear constraints within neural networks. Our approach represents rules as disjunctive constraints and applies hierarchical convex relaxations to obtain convex hull formulations. These relaxations yield tractable linear constraints that can be embedded as differentiable optimization layers while enabling exact rule satisfaction. We demonstrate the effectiveness of the proposed framework on real-world datasets, achieving perfect rule satisfaction and strong predictive performance.

Uncrewed aerial vehicles (UAVs) are increasingly used for exploration-driven monitoring in hazardous environments such as disaster zones, contaminated sites, wildfire areas, and damaged infrastructure, where limited flight endurance must be allocated between visiting reported locations and gathering new information. In these settings, prior information regarding hazards is often incomplete, spatially imprecise, and subject to change during execution. For example, initial reports may identify a region where a hazard is likely to exist, but the actual hazard may be displaced, partially observed, or entirely unreported. We present an integrated exploration-aware UAV route optimization and path planning framework for hazard monitoring under uncertain and evolving prior information. The environment is represented as a spatial risk map, where each location has an associated belief of hazardous conditions. Reported hazards are modeled as uncertain regions of interest (ROIs) rather than confirmed target locations, requiring the UAV to inspect reported areas while also using its limited flight endurance to explore informative regions. The proposed method solves a vehicle routing problem over reported ROIs, augments the route with auxiliary pseudo-nodes to improve spatial coverage, allocates the remaining flight distance budget across route segments, and optimizes dynamically feasible B-spline trajectories for local exploration. During execution, UAV measurements update a grid-based belief map, and the remaining trajectory is replanned when new information and the remaining budget justify adaptation. Across 48 scenario configurations, online replanning improves average KL reduction by 15.9% over the offline optimized planner and 48.6% over straight-line traversal.

In this paper, we study a class of double Lambert series and establish several identities and transformation relations for them. These formulae provide useful tools for reducing certain double Lambert series to single Lambert series. As applications, we derive identities related to recent conjectures of Andrews, Dixit, Schultz, and Yee, and of Amdeberhan, Andrews, and Ballantine. We also propose a new proof of a result of Amdeberhan, Andrews, and Ballantine.

We study Rota--Baxter operators on vertex algebras using the integrated $λ$-bracket formalism. A Rota--Baxter operator produces a deformed vertex algebra structure, and the difference between the deformed and original brackets yields a two-cocycle in vertex algebra cohomology. This generalizes the classical relation between Rota--Baxter operators and Hochschild two-cocycles. We also characterize when this two-cocycle is trivial, showing that non-scalar operators give rise to non-trivial cohomology classes.

We establish the existence and uniqueness of discrete Einstein metrics on trees under Lin-Lu-Yau Ricci curvature using Perron-Frobenius theory. We establish a sharp upper bound for the largest eigenvalue of the associated Ricci matrix in terms of the maximum degree. Turning to structural properties, notably, the existence of a positive-curvature Einstein metric implies the tree must be a caterpillar. Furthermore, these metrics exhibit radial monotonicity, with edge weights decreasing strictly away from the maximal edge.

Let $G$ be a simple connected graph on $n$ vertices, and let $λ_1(G),λ_2(G),\ldots,λ_n(G)$ be the eigenvalues of its adjacency matrix $A(G)$. For $p>0$, define the $p$-energy of $G$ by $\mathcal E_p(G)=\sum_{i=1}^n |λ_i(G)|^p$. We prove that, for every real number $p\ge 2$ and every simple connected graph $G$ on $n$ vertices, $$ \mathcal E_p(G)\ge \mathcal E_p(P_n), $$ where $P_n$ denotes the path on $n$ vertices. Moreover, for each fixed $p>2$, equality holds if and only if $G\cong P_n$. Together with the previously known star-minimality results, this completes the solution of two questions of Nikiforov. The proof combines two different comparison principles. For $2<p<4$, we use a bipartite reduction, a Mellin representation of fractional powers, and a determinant comparison involving matching generating polynomials and tree shifts. For $p\ge4$, we prove a second-order stop-loss comparison for the squared singular values of bipartite graphs. This comparison is established by rank-one spectral-shift estimates, deletion-minimal counterexamples, and a finite certified analysis of the terminal sparse-sun configurations.

Answering a question of Goode, we show that $k$-triviality collapses to (1-)triviality among simple theories. In particular, every stable theory with quantifier elimination in a relational language of bounded arity is trivial. We use our collapse result, along with other facts about $k$-triviality and $k$-total triviality, to generate examples of (strongly) $k$-distal theories. The collapse result immediately implies that no stable theory can be strictly $k$-distal for some $k\geq 3$, partially answering a question of Walker. Moreover, all known examples of non-distal (strongly) $k$-distal theories are $k$-ary, rendering (strong) $k$-distality moot as a $(k+1)$-ary dividing line; we give four classes of examples that are not $k$-ary. We also show that just as distality is not preserved under taking reducts, neither is (strong) $k$-distality.

This paper shows that the concept of complex frequency, originally introduced to characterize the dynamics of signals with complex values, constitutes a generalization of eigenvalues when applied to the states of linear time-invariant (LTI) systems. Starting from the definition of geometric frequency, which provides a geometrical interpretation of frequency in electric circuits that admits a natural decomposition into symmetric and antisymmetric components associated with amplitude variation and rotational motion, respectively, we show that complex frequency arises as its restriction to the two-dimensional Euclidean plane. For LTI systems, it is shown that the complex frequencies computed from the system's states subject to a non-isometric transformation, coincide with the original system's eigenvalues. This equivalence is demonstrated for diagonalizable systems of any order. The paper provides a unified geometric interpretation of eigenvalues, bridging classical linear system theory with differential geometry of curves. The paper also highlights that this equivalence does not generally hold for nonlinear systems. On the other hand, the geometric frequency of the system can always be defined, providing a geometrical interpretation of the system flow. A variety of examples based on linear and nonlinear circuits illustrate the proposed framework.

The expected signature uniquely determines the law of a random rough path under a moment-growth condition, yet finite-sample bounds for estimating its truncations from a single long dependent trajectory remain unavailable. We study a strictly stationary stochastic process equipped with a geometric rough-path lift, observed in non-overlapping blocks of equally-spaced samples, and prove a non-asymptotic mean-squared error (MSE) bound for the block-averaging estimator of its truncated expected signature. Under moment and stationarity assumptions together with a direct covariance-decay condition on block signatures -- strictly weaker than $α$-mixing and applicable to long-range-dependent processes -- the error separates into a discretization term and a fluctuation term, with rates determined respectively by path regularity and dependence strength. A levelwise rough-factorial variance analysis keeps finite-truncation constants explicit and yields an optimal allocation rule under a fixed observation budget. We verify the assumptions for independent-coordinate fractional Ornstein--Uhlenbeck processes in three regimes: short-range (Hurst $1/4<H<1/2$), semimartingale ($H=1/2$), and long-range ($H>1/2$); in all three, the block-signature covariance is summable, so the fluctuation term decays at the same rate as in the independent-block case, even under long memory at $H>1/2$. Monte Carlo experiments show empirical slopes steeper than the guaranteed upper-bound rates.

We study attention mechanisms through the lens of a canonical unsupervised problem: principal component analysis (PCA). We show that, when trained on Gaussian data, both softmax and linear attention layers learn parameters that align with the principal eigenvectors of the covariance matrix, thereby establishing a direct and explicit connection with PCA. Our analysis covers both finite and infinite prompt regimes. In the infinite-prompt limit, we prove convergence to globally optimal solutions aligned with the leading spectral direction, while in the finiteprompt setting we show that the same behavior emerges up to sampling effects. We further extend the analysis to an in-context setting with spiked Wishart covariances, where attention successfully recovers the underlying signal direction. These results demonstrate that attention inherently performs PCA-like computations under unsupervised objectives, providing a theoretical foundation for its representation-learning capabilities.

We undertake a systematic study of the mapping properties of forms based on distance graphs in $\mathbb{Z}^{d}$ to see how the structure of a graph, $G$, affects the $\ell^{p}$ improving estimates of the form, $Λ_{G}$, based on $G$. This extends previous work on $\ell^{p}$ improving properties for the spherical averaging operator, which corresponds to a distance graph of a single distance. We obtain $\ell^{p}$ improving estimates for the collection of forms based on all graphs with 2, 3, and 4 vertices, as well as chains and simplexes of any size in $\mathbb{Z}^{d}$. Surprisingly, certain mapping properties only seem to depend on the number of vertices in the graph, not its structure, and forms based on subgraphs of a graph, $G$, do not necessarily inherit all mapping properties from $G$.

We study a class of bilevel interdiction problems in which the follower's optimization problem is computationally intractable. Motivated by network defense applications, we introduce the Randomized Max-Vertex-Coverage Interdiction (RMVCI) problem under matroid constraints. In this zero-sum Stackelberg game, the leader commits to a randomized interdiction strategy over feasible vertex subsets, while the follower, after observing the induced protection probabilities, chooses a matroid-constrained attack to maximize the expected coverage of network edges. The main challenge stems from the fact that the follower's problem is a matroid-constrained maximum vertex coverage problem and is therefore NP-hard. To address this difficulty, we first develop a general approximation framework for bilevel optimization problems with hard follower responses. The framework is based on replacing the follower's value function by a surrogate objective that approximates the follower's optimal payoff while preserving tractability of the leader's optimization problem. For the RMVCI problem, we formulate the follower's problem as an integer linear program, establish a tight integrality gap of $4/3$ for its linear relaxation, and derive a polynomial-time $4/3$-approximation algorithm via pipage rounding. We then show that a carefully designed surrogate objective admits a marginal-probability reformulation that transforms the randomized interdiction problem into a tractable optimization problem over the leader's matroid polytope. This yields a polynomial-time $2$-approximation algorithm for RMVCI under general matroid constraints. Beyond the specific application studied here, our results provide a new perspective on approximation methods for {general} bilevel optimization problems.

We provide counterexamples to the unimodal minimal filling architecture conjecture for polynomial neural networks (PNNs) with power activation functions. Fixing the input and output widths, the conjecture states that any minimal filling architecture has unimodal widths for the hidden layers. We found counterexamples via a frontier search, recursive dimension bounds on neurovarieties, and symbolic computation. Notably, several subarchitectures of our main example exhibit large defect, in contrast with the predominantly small-defect behavior observed in prior literature.

We show that the realization theorem of Fernández de Bobadilla, which identifies the Milnor fiber of a weighted-homogeneous polynomial with the complement of a germ of analytic set, can be combined with the systematic Massey product constructions of Grbić-Linton for moment-angle complexes $\mathcal{Z}_K = \mathcal{Z}_K(D^2, S^1)$ to produce weighted-homogeneous polynomials whose Milnor fibers are arbitrarily highly connected and non-formal. The original application of this strategy, due to Fernández de Bobadilla, used the Denham-Suciu classification of lowest-degree triple Massey products and yielded only 2-connected non-formal Milnor fibers. The Grbić-Linton framework, which constructs non-trivial $n$-fold Massey products in $H^*(\mathcal{Z}_K;\mathbb{Z})$ for arbitrary $n$ and in arbitrary cohomological degrees, removes this connectivity restriction entirely.

Let $\mathbb{F}$ be an algebraically closed field of characteristic $0$. Given a square matrix $A \in \mathbb{F}^{n \times n}$ and a polynomial $f \in \mathbb{F}[w]$, we determine the Jordan canonical form of the formal Fréchet derivative of $f(A)$, in terms of that of $A$ and of $f$. When $\mathbb{F}\subseteq \mathbb{C}$, via Hermite interpolation, our result provides a solution to [N.J. Higham, \emph{Functions of Matrices: Theory and Computation}, Research Problem 3.11]. A generalization consists of finding the Jordan canonical form of linear combinations of Kronecker products of powers of two square matrices, i.e., $\sum_{i,j} a_{ij} (X^i \otimes Y^j)$. For this generalization, we provide some new partial results, including a partial solution under certain assumptions and general bounds on the number and the sizes of Jordan blocks.

A variant of the flatness problem from integer programming is studied, in which one considers convex bodies in $\mathbb{R}^d$ with at most $k$ interior lattice points. The maximum lattice width of such a body is denoted by Flt(d,k) and it is related to the classical flatness constant as well as a conjectural dual version of Minkowski's convex body theorem due to Makai. Moreover, it is shown that Flt(2, 1) = 3, i.e., any planar convex body with at most one interior point has lattice width at most three. This leads to an isominwidth inequality for the lattice point enumerator of planar convex bodies.

We develop a geometric procedure for finding the Apéry set of any numerical semigroup with embedding dimension four. Previous methods of comparable strength worked only for embedding dimension three or under very specific conditions. We illustrate our method by finding the Frobenius numbers, genera, Betti elements, minimal presentations, and catenary degrees of numerical semigroups generated by four consecutive squares and by four consecutive triangular numbers.

The Lie algebra $\bar{S}_2$ of polynomial vector fields on $\mathbb{C}^2$ with constant divergence is an important Cartan type Lie algebra. In this paper, we study Whittaker $\bar{S}_2$-modules that are locally finite over $\text{span}\{\frac{\partial}{\partial t_1}, \frac{\partial}{\partial t_2}\}$. We first show that each block $Ω^{\widetilde{S}_2}_{\mathbf{a}}$ of the category of $(A_2, \bar{S}_2)$-Whittaker modules with finite-dimensional Whittaker vector spaces is equivalent to the category of finite-dimensional modules over the parabolic subalgebra $\bar{S}_2^{\geq 0}$. Then we classify all simple Whittaker $\bar{S}_2$-modules in every block $Ω^{\bar{S}_2}_{\mathbf{a}}$ . Finally, we establish an equivalence between $Ω^{\bar{S}_2}_{\mathbf{1}}$ and the category $H_{\mathbf{1}}$-fmod of finite-dimensional modules over an associative algebra $H_{\mathbf{1}}$, whose generators are also determined.

We introduce large $p$ asymptotic geometric quantities associated with $p$-capacity, the first $p$-eigenvalue, and the Maz'ya constant on complete noncompact Riemannian manifolds. We prove the hierarchy $$ \mathcal{V}(M)\geq \mathcal C(Ω)\geq Λ(M)=\mathcal M(M)\geq0, $$ and show that, under a centered-ball isoperimetric condition or a rotational symmetry condition, these quantities coincide with the volume entropy or the dimension. In the Hadamard nonpositively curved case it also agrees with the topological entropy of the geodesic flow. As an application, combining with the entropy rigidity theorem, we obtain a characterization of hyperbolic manifolds. We also prove a second-order refinement. For a Hadamard manifold with compact quotient, under certain condition, the first-order large $p$ capacitary limit detects volume entropy, whereas the logarithmic second-order correction detects the rank.

We revisit size controllability results in Pötscher and Preinerstorfer (2025) concerning heteroskedasticity robust test statistics in regression models. For the special, but important, case of testing a single restriction (e.g., a zero restriction on a single coefficient), we povide a necessary and sufficient condition for size controllability, whereas the condition in Pötscher and Preinerstorfer (2025) is, in general, only sufficient (even in the case of testing a single restriction).

We extend the algebraic diversity (AD) framework from classical signal processing to quantum measurement theory. The Quantum Algebraic Diversity (QAD) Theorem establishes that a group-structured positive operator-valued measure (POVM) applied to a single copy of a quantum state produces a full-rank, group-averaged density matrix estimator whose eigenbasis and eigenvalue ordering track those of the true density matrix, with a bias toward the symmetrized state, analogous to the classical recovery of covariance eigenstructure from a single observation. We establish a Classical-Quantum Duality Map connecting classical covariance estimation to quantum state tomography, and an Optimality Inheritance Theorem showing that classical group optimality transfers to quantum settings via the Born map within the group-averaged family. SIC-POVMs are identified as AD with the Heisenberg-Weyl group and mutually unbiased bases as AD with the Clifford group, revealing the hierarchy $\mathrm{HW}(d) \subseteq \mathcal{C}(d) \subseteq S_d$ that mirrors the classical $\mathbb{Z}_M \subseteq G_{\min} \subseteq S_M$. The double-commutator eigenvalue theorem gives polynomial-time adaptive POVM selection. A worked qubit example shows the group-averaged estimator from a single computational-basis measurement, averaged over a matched $\mathbb{Z}_2$ group, reaching fidelity 0.99 where standard single-basis tomography gives a rank-1 estimate of fidelity 0.80. Monte Carlo simulations for $d = 2$ to $13$ confirm fidelity above 0.90 from a single outcome while standard fidelity degrades as $\sim 1/d$. The growing ratio reflects collapse of the rank-1 standard estimator, not fewer copies per parameter: the biased single-copy estimator reduces the number of distinct measurement settings, not the per-parameter sampling cost, and a genuine copy reduction holds only under exact symmetry.

Let $β$ be a non-unit real algebraic integer greater than one and $\{a_{n}\}_{n \geq 0}$ be a sequence satisfying a linear recurrence relation $a_{n+3}=aa_{n+2}+ba_{n+1}+ca_{n}$. Under certain conditions, we prove that the number of $a_{n}$ which are palindromic concatenations of two repdigits in base $β$ is finite.

This paper is the second and final part of a two-part study. We construct positive-energy relativistic spatial localization observables in Minkowski spacetime within standard quantum field theory, using the stress--energy--momentum tensor smeared with suitable test functions. For each fixed timelike direction, the construction gives positive operator-valued measures (POVMs) on spacelike hypersurfaces, well defined on every $n$-particle sector and satisfying a relativistic causality condition excluding superluminal propagation of detection probabilities. The observables are built from local or quasi-local field-theoretic quantities, thus providing a rigorous version of earlier heuristic proposals. In the one-particle sector, the construction reduces to the observable previously introduced by the author, and its first moment gives the Newton--Wigner position operator under appropriate normalization and centering assumptions. Because the Reeh--Schlieder theorem prevents the normally ordered stress--energy--momentum tensor from being positive on the full Fock space, we use quantum energy inequalities to obtain lower bounds controlling deviations from positivity. This leads to regularized operator families, bounded from below, which approximate the localization effects. Finally, we define conditional localization observables for finite laboratories through modified local energy operators. By Haag duality, the corresponding conditional POVMs belong to local von Neumann algebras and commute for causally separated regions, in accordance with the Araki--Haag--Kastler framework. The results show how commutativity of localization observables is recovered for conditional measurements in finite spacetime regions.

The article establishes a rigorous shift operator calculus for stationary time series modeling, addressing a certain gap in the literature. It provides proofs of existence and isometry for the transfer function operators $f(B)$ and $f(T)$ where $B$ is the bilateral shift operator and $T$ is the unilateral shift operator for different families of functions $f$. The article establishes convergence of the power series of $f(B)$ and $f(T)$ under the operator norm for the Wiener algebra $\mathbb{W}_+$, and convergence under strong operator topology for $f$ in $H^{\infty}$, based on the use of Abel sums. Based on this calculus, it unifies the notion of stationary process invertibility with the operator invertibility of the transfer function $f(T)$.

For a non-zero parameter $q$, we define Hermite trace polynomials, which are multivariate polynomials indexed by permutations. We prove several combinatorial properties for them, such as expansions and product formulas. The linear functional determined by these trace polynomials is a state for $q = \frac{1}{N}$ for $N$ a non-zero integer. For such $q$, Hermite trace polynomials of different degrees are orthogonal. The product formulas extend to the closure with respect to the state. The state can be identified with the expectation induced by the $N \times N$ Hermitian Brownian motion. Hermite trace polynomials are martingales for this Brownian motion, while the elements in the closure can be interpreted as stochastic integrals with respect to it. Using the grading on the algebra, we prove several chaos decompositions for such integrals, as well as analyze corresponding creation and annihilation operators. In the univariate, pure trace polynomial case, trace Hermite polynomials can be identified with the Hermite polynomials of matrix argument.

We study a novel type of mechanical structures, composed of spatial four-bar linkages, that are bistable, that is, they allow for two distinct configurations. These structures have an interpretation as quad nets in the Study quadric which we use to prove existence of assemblies with an unbounded number of links and joints. We propose a purely geometric construction of such objects, starting from infinitesimally flexible quad nets in Euclidean space and applying Whiteley de-averaging. This point of view situates the problem within the broader framework of discrete differential geometry and enables the construction of bistable structures from well-known classes of quad nets, such as discrete minimal surfaces. In contrast to many other construction methods for bistable structures, our approach does not rely on numerical optimization and it allows for simple control of relevant geometric parameters such as axis positions and snap angles.

We give an explicit description of the BPS Lie algebra of any quiver with zero potential, by relating the perverse filtration on the cohomological Hall algebra with certain limit conditions on polynomials. Our results also give a partial description of the perverse filtration for arbitrary potential, which we conjecture is complete in the case of tripled quivers with canonical cubic potential.