Operator SVD: from spectral theorem to polar form to singular values

The singular value decomposition is the operator analogue of the polar form $z = re^{i\theta}$ for complex scalars: a magnitude $|L|$ that does the stretching and a partial isometry $U$ that does the rotation. The magnitude is built from the spectral theorem applied to $L^* L$, the polar form $L = U|L|$ collects the pieces, and the SVD reads off the spectral basis of $|L|$. The construction is uniform; what varies is the spectral theorem available at each level.

Three companion posts build the spectral theorems. Spectral Theorem I covers finite dimensions. Spectral Theorem II covers compact operators on a Hilbert space. Spectral Theorem III covers the general bounded case. This post applies each in turn to read off the SVD at three levels of generality, and asks where the SVD breaks (it does, for most bounded operators). The Koopman operator is the place where the question matters in practice.

The polar reduction

For any bounded operator $L: H_1 \to H_2$ between Hilbert spaces, three facts hold simultaneously and are the only ingredients the SVD construction needs.

1. $L^* L$ is bounded, self-adjoint, positive semi-definite. Self-adjointness is direct: $(L^* L)^* = L^* L^{**} = L^* L$. Positivity is best seen as a Gram structure rather than a spectral fact:

   $\langle L^* L\, h, h\rangle = \langle L h, L h\rangle = \|L h\|^2 \ge 0.$

   The quadratic form is a norm squared.

2. The positive square root $|L| := (L^* L)^{1/2}$ exists. The spectral theorem for self-adjoint operators (Part I in finite dimensions, Part III in general) applied to $L^* L$ produces a functional calculus. Since $f(t) = \sqrt{t}$ is continuous on the non-negative spectrum of $L^* L$, the calculus delivers $|L|$ as a bounded, self-adjoint, positive operator with $|L|^2 = L^* L$.

3. The polar decomposition $L = U |L|$ holds with $U$ a partial isometry. Define $U$ on $\mathrm{ran}(|L|)$ by $U(|L| h) := L h$. This is well-defined and isometric on $\overline{\mathrm{ran}(|L|)}$ because

   $\| |L| h \|^2 = \langle |L|^2 h, h\rangle = \langle L^* L\, h, h\rangle = \|L h\|^2,$

   so $|L| h = 0 \Rightarrow L h = 0$ and the assignment respects the norm. Extend $U$ by zero on $\overline{\mathrm{ran}(|L|)}^\perp = \ker(L^*)$. The result is a partial isometry with $L = U |L|$. When $L$ is invertible, $U$ is unitary.

The polar form has clear geometric content: every bounded $L$ factors as a stretch $|L|$ followed by a rotation $U$. The SVD is the spectral diagonalization of the stretch, written so that source and target bases are made explicit.

Finite dimensions: the classical SVD

$L: V \to W$ between finite-dimensional inner-product spaces. Part I’s spectral theorem applied to $L^* L$ produces an orthonormal eigenbasis $\{\mathbf{v}_i\}$ of $V$ with non-negative eigenvalues $\mu_i \ge 0$:

$L^* L = \sum_i \mu_i\, \mathbf{v}_i \mathbf{v}_i^*, \qquad |L| = \sum_i \sqrt{\mu_i}\, \mathbf{v}_i \mathbf{v}_i^*.$

Apply the polar form and define singular values $\sigma_i := \sqrt{\mu_i}$, right singular vectors $\mathbf{v}_i$, left singular vectors $\mathbf{w}_i := U \mathbf{v}_i$. Reorganizing the basis gives the standard factorization

$L = U |L| = U V \Sigma V^* = W \Sigma V^*,$

where $V$ has the $\mathbf{v}_i$ as columns, $W$ has the $\mathbf{w}_i$ as columns, and $\Sigma = \mathrm{diag}(\sigma_1, \sigma_2, \ldots)$ is non-negative diagonal. Geometrically: every linear map, in suitable orthonormal coordinates on source and target, is multiplication by a non-negative diagonal matrix, i.e., a set of independent stretches.

How the operator SVD relates to the matrix SVD

The factorization just derived is the matrix SVD. In the standard textbook notation $A = U\Sigma V^*$, the matrix $A \in \mathbb{C}^{m\times n}$ is the coordinate array of $L:\mathbb{C}^n \to \mathbb{C}^m$ in the standard bases, the columns of $U$ are the left singular vectors $\mathbf{w}_i$ (the $W$ above), the columns of $V$ the right singular vectors $\mathbf{v}_i$, and $\Sigma$ holds the singular values. The operator view adds no new theory; it names two pieces of structure the matrix array silently fixes.

The SVD spends an inner product that the eigenproblem does not. Both the matrix form and the intrinsic form $L = \sum_i \sigma_i \langle \cdot, \mathbf{v}_i\rangle\, \mathbf{w}_i$ are coordinate-free: once the inner products are fixed, the $\sigma_i, \mathbf{v}_i, \mathbf{w}_i$ are determined by $L$ alone, so the dependence is on the inner products, not on a basis. The sharp statement is in the invariance group. The eigenvalues of a square matrix are invariant under every similarity $A \mapsto SAS^{-1}$ and need no inner product ($\det(A - \lambda I)$ has none); the singular values are invariant only under unitary $A \mapsto Q_2 A Q_1^*$, and “unitary” is exactly what preserves the chosen inner products. Change those inner products and the singular values move with them: the $\sigma_i^2$ are the eigenvalues of $L^*L$, whose matrix is $G_1^{-1} A^\dagger G_2 A$ for the source and target Gram matrices $G_1, G_2$ (Part I), while the eigenproblem of $A$ never references $A^*$ at all. The SVD asks for more structure and respects a smaller symmetry in return.

Source and target are different spaces. The matrix SVD lumps $U$ ($m\times m$) and $V$ ($n\times n$) together as “the rotations,” but they are orthonormal bases of different Hilbert spaces: $V$ of the source, $U$ of the target. This is why the SVD handles rectangular $A$ and maps between unrelated spaces, while the eigendecomposition needs a square endomorphism with source and target identified. The SVD asks only for two inner-product spaces and a map between them; it never needs them to be the same.

Why the SVD is the right low-rank object

The SVD matters in practice less because it exists than because it ranks the action of $L$ by importance, and truncating that ranking is provably the best low-rank approximation. Three views of the same fact.

Importance-ordered rank-1 pieces. Write $L = \sum_i \sigma_i\, \mathbf{w}_i \mathbf{v}_i^*$ with $\sigma_1 \ge \sigma_2 \ge \cdots \ge 0$. Each term is a rank-1 operator sending $\mathbf{v}_i$ to $\sigma_i \mathbf{w}_i$ and annihilating everything orthogonal to $\mathbf{v}_i$; the sum stacks them in decreasing strength. The rank-$k$ truncation $L_k = \sum_{i \le k} \sigma_i \mathbf{w}_i \mathbf{v}_i^*$ keeps the $k$ strongest pieces. The decomposition arrives already sorted, so a low-rank approximation is just a prefix of it.

The singular values are the semi-axes. Extending the “set of stretches” picture: $L$ carries the source unit sphere to a hyperellipse in the target, and the $\sigma_i$ are the lengths of its semi-axes along the directions $\mathbf{w}_i$. Truncation flattens the thinnest axes to zero. When the $\sigma_i$ decay fast the hyperellipse is nearly degenerate, a few long axes holding almost all of its extent, and $L_k$ is a faithful low-rank picture of the full map.

Truncation is optimal, not just convenient. The rank-$k$ truncation is the best rank-$k$ approximation of $L$ in both the Frobenius norm $\|\cdot\|_F$ and the operator norm $\|\cdot\|_{\mathrm{op}}$ (Eckart and Young, 1936), and the two minimizers coincide. The Frobenius statement is an energy account: $\|L\|_F^2 = \sum_i \sigma_i^2$, the truncation retains the largest $\sum_{i \le k} \sigma_i^2$ of that energy and leaves error exactly $\sum_{i > k} \sigma_i^2$ (the operator-theoretic core of PCA). In the operator norm the error is exactly $\sigma_{k+1}$, the first dropped singular value.

Proof: why no rank-k operator beats the SVD truncation

Write the SVD as $L = \sum_n \sigma_n \langle \cdot, \psi_n\rangle\, \phi_n$ with $\sigma_1 \geq \sigma_2 \geq \cdots$ (a finite sum in the finite-dim case, a discrete sequence with $\sigma_n \to 0$ in the compact case treated in the next section). The truncation $L_k$ achieves $\|L - L_k\|_{\mathrm{op}} = \sigma_{k+1}$ directly from the SVD. For the lower bound, let $B$ be any rank-$k$ operator. Then $\ker B$ has codimension $\leq k$, so it intersects the $(k+1)$-dimensional subspace $V_{k+1} := \mathrm{span}\{\psi_1, \ldots, \psi_{k+1}\}$ nontrivially. Pick a unit vector $v$ in the intersection; $Bv = 0$, so

$\|(L - B)v\|^2 = \|L v\|^2 = \sum_{n=1}^{k+1} \sigma_n^2\, |\langle v, \psi_n\rangle|^2 \;\geq\; \sigma_{k+1}^2 \sum_{n=1}^{k+1} |\langle v, \psi_n\rangle|^2 \;=\; \sigma_{k+1}^2$

(the last equality is Parseval, using $v \in V_{k+1}$). Hence $\|L - B\|_{\mathrm{op}} \geq \sigma_{k+1} = \|L - L_k\|_{\mathrm{op}}$. The Frobenius case is the same argument applied coordinatewise: $\|L - B\|_F^2 \geq \sum_{n > k} \sigma_n^2$ via orthogonality of the singular vectors, with equality at the truncation. The structural ingredient (a finite-codimension intersection with a singular-vector subspace) is the same in both regimes.

Compact regime: discrete SVD with decaying singular values

$L: H_1 \to H_2$ compact between Hilbert spaces. Part II gives $L^* L$ compact (Schauder plus the ideal property), self-adjoint, and positive, with a discrete eigendecomposition

$L^* L = \sum_n \mu_n P_n, \qquad \mu_n > 0, \quad \mu_n \to 0,$

each $P_n$ a finite-rank orthogonal projection onto the $\mu_n$-eigenspace. The functional calculus is trivial because the spectrum is discrete: $|L| = \sum_n \sqrt{\mu_n}\, P_n$. The polar form $L = U |L|$ goes through as above, and the SVD is

$L = \sum_n \sigma_n\, \langle \cdot, \psi_n\rangle\, \phi_n, \qquad \sigma_n := \sqrt{\mu_n},$

with $\{\psi_n\}$ orthonormal eigenvectors of $L^* L$ and $\phi_n := U \psi_n$.

The singular values are nonzero and decay to zero. A compact operator is the norm limit of its finite-rank truncations $L_k := \sum_{n \le k} \sigma_n \langle \cdot, \psi_n\rangle\, \phi_n$, and the truncation is the best rank-$k$ approximation of $L$ in operator norm. (The Frobenius best-approximation statement requires more care here: it is the right form only when $L$ is Hilbert–Schmidt, $\sum_n \sigma_n^2 < \infty$, in which case the Frobenius norm $\|L\|_{\mathrm{HS}} = \bigl( \sum_n \sigma_n^2 \bigr)^{1/2}$ makes sense and Eckart–Young extends verbatim.)

General bounded operators: SVD typically does not exist

For $L: H_1 \to H_2$ merely bounded, Part III gives $L^* L$ a spectral resolution

$L^* L = \int_0^{\|L\|^2} \lambda \, dE(\lambda)$

against a projection-valued measure $E$. The polar form $L = U |L|$ still holds and $|L| = \int \sqrt{\lambda}\, dE(\lambda)$ via the same functional calculus. But the SVD as a discrete sum requires $E$ to be purely atomic, and that is exactly the compactness condition: $L^* L$ has atomic spectral measure iff $L^* L$ is compact, which by the ideal property holds iff $L$ is compact. So §4 is the only regime where SVD-as-discrete-sum exists.

The textbook counter-example is the multiplication operator on $L^2[0, 1]$,

$(M_t f)(x) = x f(x),$

which is bounded ($\|M_t\| = 1$), self-adjoint, and positive. Its spectral measure is $E(S) =$ multiplication by the indicator $\mathbf{1}_S$, purely continuous on $[0, 1]$. So $M_t = \int_0^1 \lambda\, dE(\lambda)$ as an integral, with no atoms, no eigenvalues, no eigenvectors, no singular values, and no singular vectors. Its decomposition is irreducibly continuous. Many naturally occurring operators look like this, and for them the SVD as a discrete sum simply does not exist.

This is a substantive negative result, not a deficiency of construction. The SVD is the right object exactly when the underlying $L^* L$ has the right spectral shape, and “the right spectral shape” is the compactness condition that Part II established.

When this matters: the Koopman SVD

The pattern appears in dynamical systems through the Koopman operator $\mathcal{K}$ of a Markov process on $L^2(\pi)$. Whether $\mathcal{K}$ admits an SVD is decided by the dynamics:

• Deterministic measure-preserving dynamics. $\mathcal{K}$ is an isometry; every singular value is $1$; the operator is not compact and the spectrum lives continuously on the unit circle. No SVD exists as a discrete sum. Modern data-driven Koopman SVD methods do not apply here, for a fundamental reason.
• Stochastic dynamics with bounded conditional density. Stochastic noise regularizes the transition kernel, often pushing $\mathcal{K}$ into the Hilbert–Schmidt class, in particular compact. Then the compact-regime SVD of §4 applies: $\sigma_n \to 0$, the truncation is the best rank-$k$ approximation in operator norm (and in Frobenius once $\sum \sigma_n^2 < \infty$ is in hand), and the standard data-driven recoveries are well-posed.

The Koopman post takes this dichotomy up in detail. The key fact is that the existence of the SVD is itself a non-trivial property of the system, decided one rung up by whether $L^* L$ is compact, decided two rungs up by the structure that produces compactness.

What it amounts to

SVD = spectral theorem on $L^* L$ + polar form. The three-rung spectral-theorem ladder transfers verbatim, with the SVD at each level reading off as:

• Finite dimension. A finite sum $L = \sum_{i=1}^r \sigma_i \mathbf{w}_i \mathbf{v}_i^*$, with Eckart–Young in both Frobenius and operator norms.
• Compact. A discrete series $L = \sum_n \sigma_n \langle \cdot, \psi_n\rangle \phi_n$ with $\sigma_n \to 0$, with Eckart–Young in operator norm and (under Hilbert–Schmidt) in Frobenius norm.
• General bounded. No SVD as a discrete sum. The spectral measure of $L^* L$ may be continuous, and the operator’s decomposition is irreducibly an integral.

The spectral theorem is a property of self-adjoint operators, but the operators interesting in practice (Koopman of a non-reversible chain, NN forward maps, integral operators with non-symmetric kernels) are typically not self-adjoint. The polar reduction $L \to L^* L$ is what bridges the gap: it lets the spectral theorem do work on operators that are not self-adjoint themselves, and the SVD is the working form of that bridge.

References

1. Eckart, C., and Young, G. (1936). The approximation of one matrix by another of lower rank. Psychometrika 1(3), 211–218.
2. Lax, P. D. (2002). Functional Analysis. Wiley-Interscience.
3. Conway, J. B. (2007). A Course in Functional Analysis (2nd ed.). Springer.