Algorithmic incompressibility, chaos, and the data-scaling ceiling in scientific machine learning

In 2009, Halevy, Norvig, and Pereira (Halevy et al., 2009) published The Unreasonable Effectiveness of Data, deliberately echoing Wigner’s Unreasonable Effectiveness of Mathematics. The argument: web-scale corpora of human-generated text combined with simple non-parametric models beat feature-engineered alternatives on language tasks. Seventeen years on, the same recipe (scaled up to neural rather than non-parametric models) defines modern NLP. In language-model pretraining it now runs to its logical end: past a large enough compute budget, even data filtering stops paying off, and the full unfiltered corpus, junk included, overtakes every hand-curated subset (Mohri et al., 2026).

Transposed to scientific machine learning (SciML), the record is heterogeneous. AlphaFold (Jumper et al., 2021) predicts well-folded protein structures to near-experimental accuracy, and modern ML interatomic potentials match quantum-chemistry benchmarks on energies and forces. AI weather models, by contrast, post strong RMSE scores but suppress small-scale spectra, fail dynamical-consistency tests, and degrade out of distribution. The heterogeneity is not noise; it is structure, and one axis organizes all of it: how much of the data a compact physical law can compress. A scientific law is operationally a compression of observations. The data-scaling recipe of Halevy et al. (2009) works to the extent that what the task asks the model to predict is pattern a short law can regenerate, rather than detail no law can. Pure data scaling succeeds where the law compresses the data, the regime of i.i.d. samples of a finite-dimensional target, and stalls where it cannot, the per-trajectory detail of a chaotic multi-scale system. A ceiling in what follows is exactly that incompressible remainder: a level of fidelity that more data and bigger generic architectures do not push past, because the missing information is structurally absent from the data.

Of the four task types we will distinguish, chaotic dynamics carries by far the hardest ceiling. The rest asks why, and what breaks it.

A class-by-class view of SciML

Four classes, distinguished by what the data look like, how the model is evaluated, and how much of the data a physical law can compress, rather than by scientific domain:

Class	Target object	Data structure	Law compresses	Ceiling	Examples
A Chaotic dynamics	Evolution map $\phi^t: \mathcal{S} \to \mathcal{S}$	Chaotic trajectories	Statistical layer only, not per-trajectory	Structural (McAllister–Chibbaro)	Weather forecasting, turbulent flow rollouts, long-time MD
B Equilibrium / regression	Function $f: \mathcal{X} \to \mathbb{R}^d$	i.i.d. samples of a finite-dim target	Fully; a compact function regenerates samples	None (classical statistics)	DFT and VMC ground states, ML interatomic potentials, materials property regression
C Generative	Distribution $p$ on high-dim space	i.i.d. samples	The bulk distribution, not the OOD tails	Coverage / OOD	Molecule and material generation, AlphaFold-style conditional structure
D Symbolic / formal	Discrete object (proof, program)	Tokens + bivalent verifier	N/A; verifier replaces it	None (verifier closes loop)	AI for mathematics, code generation

Tasks, not models. The same neural network potential is an equilibrium regressor when scored against quantum-chemistry samples, and a chaotic-dynamics rollout when used for long-time molecular dynamics. Same weights, different evaluation, different ceiling.

Equilibrium and chaotic dynamics share a target. Both seek a deterministic function; what differs is whether the data are i.i.d. samples (equilibrium) or chaotic trajectories (dynamics).

Generative is a different target. Generative tasks learn a distribution rather than a function. AlphaFold straddles equilibrium and generative: it treats a multimodal conditional distribution $p(\mathrm{structure} \mid \mathrm{sequence})$ as a function, which works for sharp-mode globular proteins but fails on intrinsically disordered proteins, novel folds, and single-point mutations (Buel and Walters, 2022).

Symbolic tasks are a separate exception. They are the only class with a hard verifier, which is what lets verifier-in-the-loop training (reinforcement learning with verifiable rewards, RLVR, for AlphaProof and DeepSeek-Prover; LLM-guided evolutionary search for FunSearch) substitute for data scaling. We come back to them at the end, since their escape mechanism does not transfer to the physical sciences.

How each class fails

The failures differ in character, not just magnitude. Equilibrium tasks fail through approximation error within physically valid outputs: AlphaFold’s predicted folds may be wrong but they have realistic bond lengths and packing. Chaotic-dynamics tasks fail categorically: AI weather models break conservation laws, fail dynamical balances, and produce unphysical small-scale spectra over rollout. Generative tasks fail distributionally: modern architectures avoid invalid outputs by construction, leaving residual failures that are functional, such as a synthesizable molecule with no activity, or distributional, such as a missed conformational mode.

What the marginal byte carries

McAllister (2003) decomposes an empirical data set as

$F(x) = \mathrm{pattern}(x) + R(x),$

where the pattern is whatever a scientific law would recover, and $R(x)$ collects perturbation that no law of the system is meant to account for (instrument calibration, environmental noise, observer-dependent disturbance). The claim is that $R(x)$ is algorithmically incompressible in the Kolmogorov sense: a fixed Kolmogorov-random string, not an i.i.d. zero-mean noise process. An i.i.d. process has a population mean to average against, and sample averaging concentrates the estimate at $\sigma/\sqrt{n}$; an algorithmically random string has no such structure, passes statistical tests for randomness, and offers nothing for the statistician to estimate.

For equilibrium and generative tasks the standard i.i.d. picture applies. Each PDB structure is an independent draw from a distribution over fold space, each DFT calculation is an independent sample of an energy surface, sample averaging works as expected, and the AlphaFold-class wins follow. For chaotic dynamics the data are closer to the algorithmically incompressible regime: more cadence on the same trajectory adds incompressible content, not pattern, and standard sample averaging does not recover what is structurally not there.

Chaos propagates incompressibility

Chibbaro and Vulpiani (2017) sharpen the point on the dynamical-systems side. Almost all initial conditions are algorithmically incompressible (Martin-Löf, 1966), and deterministic dynamics propagate that incompressibility forward in time. Mechanistically: a finite-precision initial condition carries finite information; chaotic dynamics generate new fine-scale information at the system’s Kolmogorov–Sinai entropy rate. After the Lyapunov time $t_\lambda \sim 1/\lambda_{\max}$, initial-condition uncertainty has grown by $\mathcal{O}(1)$ and the trajectory has decorrelated from any predictor. Per-trajectory error past $t_\lambda$ is bounded below regardless of training data, because the future fine-scale information has not yet emerged at the time of prediction.

The trajectory-statistics asymmetry. Per-trajectory prediction is subject to the Lyapunov bound; statistical learning of the attractor (climatology, ergodic averages, scaling laws, energy spectra) is data-scalable and is precisely what the physical law compresses. The Navier–Stokes equation makes this concrete: “we cannot say that the NSE are able to compress the turbulent behaviors” at the trajectory level (Chibbaro and Vulpiani, 2017, p. 12), but the NSE do compress the statistical layer of fully developed turbulence.

The cost of working purely from time series is steep. Takens delay embedding requires $T_R \sim (L/\epsilon)^D$ samples in attractor dimension $D$; for modest $D=8$ and resolution $100$ this is $10^{16}$ samples. Chibbaro and Vulpiani close with a sentence that reads as a direct rebuttal of the data-scaling recipe (Halevy et al., 2009) in this setting:

Perhaps we live in a “big data” era, but actually not big enough to model complex phenomena, without the help of some theory. (Chibbaro and Vulpiani, 2017, p. 15)

The ceiling is exactly this asymmetry: the part of the data the model is asked to predict (per-trajectory continuation past $t_\lambda$) is the part the dynamics structurally do not let the law compress.

What the picture predicts for AI weather

AI weather model failures are now in print:

Reference	Models tested	Finding	Consistent with McAllister–Chibbaro?
Bonavita (2024)	GraphCast, Pangu-Weather, FourCastNet	Small-scale tail of kinetic-energy spectrum is under-resolved; scaling does not close the gap	Yes: this is the exact spectral region where chaotic trajectory data are least compressible
Hakim and Masanam (2024)	Pangu-Weather (dynamical perturbation experiments)	Passes climatological-pattern tests; fails geostrophic-balance recovery and specific linear-response amplitudes	Yes: statistical layer captured, structural layer missed
Subramanian et al. (2023)	Neural operators across PDE families	SciML scaling laws unlike NLP’s: worse sample efficiency, brittle transfer, OOD (not capacity) is the bottleneck	Adjacent: a transfer / OOD ceiling, not the trajectory-incompressibility argument per se. The reverse of language-model pretraining, where the limit is capacity and scaling it closes the gap (Mohri et al., 2026)

The three results sit at different operating points (forecast spectra, dynamical-consistency tests, cross-PDE scaling) and converge on the same shape: more data and bigger architectures push these models up to the McAllister–Chibbaro ceiling, not through it.

What "pure data" actually means for AI weather models

The AI weather models above are trained on ERA5 reanalysis, which is itself produced by primitive equations and 4D-Var data assimilation. So “pure data scaling” here is more precisely “pure-data emulation of physics-informed reanalysis.” The ceiling argument still applies because the learned model rolls out off the reanalysis distribution and accumulates errors the assimilation cycle would otherwise correct, but the framing should be read with this caveat.

Two paths around the ceiling

The chaotic-dynamics ceiling has two known workarounds, and each restores compressibility the raw trajectory data lack: supply the missing law, or switch the target to the layer the law already compresses. Both depart from the equivariance-centric consensus of recent AI for science (Zhang et al., 2025): that architectural bias suffices for equilibrium and generative tasks but not for chaotic dynamics.

Put the law in the model. Generic architectural bias is insufficient for chaotic dynamics even with unlimited data; the missing structure is the physical law (conservation, locality, scale separation), and it has to be written in, not learned. This inverts the bitter lesson of language-model pretraining, where hand-designed inductive bias such as data filtering loses to scale (Mohri et al., 2026); for chaotic dynamics, scale loses to bias. de Bézenac, Pajot, and Gallinari (2018) is the clean ablation: two sea-surface-temperature forecasters with matched data, capacity, and objective, and the one that wires an advection PDE into its forward pass wins decisively. At operational scale, NeuralGCM (Kochkov et al., 2024) couples a differentiable primitive-equation solver with learned sub-grid parameterizations; it matches pure ML on short-range deterministic skill and dominates on long-term stability and physical consistency. This is the standard hybrid view (Reichstein et al., 2019; Karniadakis et al., 2021).

Stop predicting the trajectory; predict its distribution. Refuse the per-trajectory framing and learn $p(\text{future} \mid \text{present})$, then sample ensembles. GenCast (Price et al., 2025) does this with a conditional diffusion model and beats the ECMWF operational ensemble; the FGN follow-up (Alet et al., 2025) recovers joint spatial structure from per-location training alone. In the taxonomy this is a chaotic-dynamics $\to$ generative reframing of the output object: the data are still one chaotic trajectory, but the model emits a distribution rather than a point. That swaps the structural McAllister–Chibbaro ceiling for a coverage ceiling, softer in principle but, for a planet with exactly one trajectory, equally hard on rare joint extremes. The question is changed, not answered.

Why the conditional kernel is a well-defined learning target

$p(s_{t+\Delta t} \mid s_t)$ is a real mathematical object even for a deterministic chaotic flow. It evolves under the Liouville equation (Fokker–Planck with stochastic forcing), the Chapman–Kolmogorov relation gives it semigroup structure, and in the ergodic limit it converges to the Sinai–Ruelle–Bowen measure on the attractor, so $\phi^t(s_0)$ loses meaning past $t_\lambda$ while $p(\cdot \mid s_t)$ stays smooth. Estimation is statistically valid: one trajectory sliced into $\{(s_t, s_{t+\Delta t})\}$ pairs after enough mixing time gives a consistent estimator of the kernel. The approach dates to Epstein (1969) and Leith (1974), who introduced ensemble forecasting precisely because chaos forces distribution prediction, a line Palmer’s stochastic parameterizations (Palmer, 2001) extend.

In operator terms, learning this kernel is finite-dimensional approximation of the Perron–Frobenius operator $\mathcal{P}^{\Delta t}$, the predual of the Koopman operator (Mezić, 2005; Williams et al., 2015). For dissipative chaotic flows the Koopman operator acquires discrete spectrum (Pollicott–Ruelle resonances) and effective compactness on suitable function spaces, which is what makes finite-data spectral approximation (DMD, EDMD, ResDMD) converge; the SVD rather than the eigendecomposition is the natural object once the operator is non-normal. See Spectral Theorem II, Operator SVD, and Koopman operator: adjoint, normality, SVD.

The math exception

Math is the case where verifier-in-the-loop training fully replaces data scaling. Two variants carry the headline results.

RLVR against a Lean proof checker. AlphaProof reaches olympiad-level theorem proving (Google DeepMind, 2024); DeepSeek-Prover handles autoformalization at scale (Xin et al., 2024). The reward signal is bivalent: Lean either compiles the proof or it does not, and that exact gradient through reinforcement learning is what drives planning and search.

LLM-guided evolutionary search. FunSearch (Romera-Paredes et al., 2024) uses the LLM as a program proposer and a fitness function as evaluator rather than RL reward, and improved bounds on the cap-set problem and online bin packing. The mechanism differs from RLVR but the structural property is the same: a verifier closes the loop.

Why the physical sciences have no Lean. PDE residuals are continuous, approximate, and gameable; lattice QCD is too expensive to play the same role for general physical theories. Until SciML grows a hard verifier for chaotic dynamics, the only known workarounds are the hybrid and the probabilistic reframing above.

Closing

A learning recipe has two pieces: the inductive bias it brings and the bytes of data it consumes. How far a compact law compresses the data differs by task type, and the four classes line up as follows.

Class	How far the law compresses the data	Recipe that works
A Chaotic dynamics	Statistical layer only; per-trajectory detail past Lyapunov time is incompressible	Physics-informed hybrid (NeuralGCM) breaks the ceiling; probabilistic reframing (GenCast, FGN) sidesteps it
B Equilibrium	Fully; sample averaging recovers the target as standard statistics predicts	Pure data + equivariant architecture (AlphaFold, ML interatomic potentials)
C Generative	The bulk distribution; OOD tails are not in the data	Pure data + generative model, with coverage caveats
D Symbolic	Replaced by a bivalent verifier	Pretraining + RLVR with verifier (AlphaProof, FunSearch)

The data-scaling recipe of Halevy et al. (2009) describes equilibrium tasks cleanly and the math exception via verifier, but undersells coverage limits in generative settings and undersells the structural ceiling in chaotic dynamics most of all. This is why the headline AI for science failures cluster in chaotic dynamics and at the equilibrium-generative boundary (AlphaFold on disordered proteins), and why the proposed remedies for chaotic dynamics are physics-informed or task-reframing rather than larger.

References

1. Martin-Löf, P. (1966). The definition of random sequences. Information and Control 9(6), 602–619.
2. Epstein, E. S. (1969). Stochastic dynamic prediction. Tellus 21(6), 739–759.
3. Leith, C. E. (1974). Theoretical skill of Monte Carlo forecasts. Monthly Weather Review 102(6), 409–418.
4. Palmer, T. N. (2001). A nonlinear dynamical perspective on model error: a proposal for non-local stochastic-dynamic parametrization in weather and climate prediction models. Quarterly Journal of the Royal Meteorological Society 127(572), 279–304.
5. McAllister, J. W. (2003). Algorithmic randomness in empirical data. Studies in History and Philosophy of Science Part A 34(3), 633–646.
6. Mezić, I. (2005). Spectral properties of dynamical systems, model reduction and decompositions. Nonlinear Dynamics 41, 309–325.
7. Halevy, A., Norvig, P., and Pereira, F. (2009). The unreasonable effectiveness of data. IEEE Intelligent Systems 24(2), 8–12.
8. Williams, M. O., Kevrekidis, I. G., and Rowley, C. W. (2015). A data-driven approximation of the Koopman operator: extending dynamic mode decomposition. Journal of Nonlinear Science 25(6), 1307–1346.
9. Chibbaro, S., and Vulpiani, A. (2017). Compressibility, laws of nature, initial conditions and complexity. arXiv:1709.07062.
10. de Bézenac, E., Pajot, A., and Gallinari, P. (2018). Deep learning for physical processes: incorporating prior scientific knowledge. International Conference on Learning Representations.
11. Reichstein, M., Camps-Valls, G., Stevens, B., Jung, M., Denzler, J., Carvalhais, N., and Prabhat (2019). Deep learning and process understanding for data-driven Earth system science. Nature 566, 195–204.
12. Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., Bridgland, A., Meyer, C., Kohl, S. A. A., Ballard, A. J., Cowie, A., Romera-Paredes, B., Nikolov, S., Jain, R., Adler, J., Back, T., Petersen, S., Reiman, D., Clancy, E., Zielinski, M., Steinegger, M., Pacholska, M., Berghammer, T., Bodenstein, S., Silver, D., Vinyals, O., Senior, A. W., Kavukcuoglu, K., Kohli, P., and Hassabis, D. (2021). Highly accurate protein structure prediction with AlphaFold. Nature 596, 583–589.
13. Karniadakis, G. E., Kevrekidis, I. G., Lu, L., Perdikaris, P., Wang, S., and Yang, L. (2021). Physics-informed machine learning. Nature Reviews Physics 3, 422–440.
14. Buel, G. R., and Walters, K. J. (2022). Can AlphaFold2 predict the impact of missense mutations on structure. Nature Structural & Molecular Biology 29, 1–2.
15. Subramanian, S., Harrington, P., Keutzer, K., Bhimji, W., Morozov, D., Mahoney, M. W., and Gholami, A. (2023). Towards foundation models for scientific machine learning: characterizing scaling and transfer behavior. Advances in Neural Information Processing Systems 36.
16. Bonavita, M. (2024). On some limitations of current machine-learning weather-prediction models. Geophysical Research Letters 51, e2023GL107377.
17. Hakim, G. J., and Masanam, S. (2024). Dynamical tests of a deep-learning weather prediction model. Artificial Intelligence for the Earth Systems.
18. Kochkov, D., Yuval, J., Langmore, I., Norgaard, P., Smith, J., Mooers, G., Klöwer, M., Lottes, J., Rasp, S., Düben, P., Hatfield, S., Battaglia, P., Sanchez-Gonzalez, A., Willson, M., Brenner, M. P., and Hoyer, S. (2024). Neural general circulation models for weather and climate. Nature 632, 1060–1066.
19. Romera-Paredes, B., Barekatain, M., Novikov, A., Balog, M., Kumar, M. P., Dupont, E., Ruiz, F. J. R., Ellenberg, J. S., Wang, P., Fawzi, O., Kohli, P., and Fawzi, A. (2024). Mathematical discoveries from program search with large language models. Nature 625, 468–475.
20. Xin, H., Guo, D., Shao, Z., Ren, Z., Zhu, Q., Liu, B., Ruan, C., Li, W., and Liang, X. (2024). DeepSeek-Prover: advancing theorem proving in LLMs through large-scale synthetic data. arXiv:2405.14333.
21. Google DeepMind (2024). AI achieves silver-medal standard solving International Mathematical Olympiad problems. Google DeepMind blog post.
22. Alet, F., Price, I., El-Kadi, A., Masters, D., Markou, S., Andersson, T. R., Stott, J., Lam, R., Willson, M., Sanchez-Gonzalez, A., and Battaglia, P. (2025). Skillful joint probabilistic weather forecasting from marginals. arXiv:2506.10772.
23. Price, I., Sanchez-Gonzalez, A., Alet, F., Andersson, T. R., El-Kadi, A., Masters, D., Ewalds, T., Stott, J., Mohamed, S., Battaglia, P., Lam, R., and Willson, M. (2025). Probabilistic weather forecasting with machine learning. Nature 637, 84–90.
24. Zhang, X., Wang, L., Helwig, J., Luo, Y., Fu, C., Xie, Y., Liu, M., Lin, Y., Xu, Z., Yan, K., Adams, K., Weiler, M., Li, X., Fu, T., Wang, Y., Strasser, A., Yu, H., Xie, Y., Fu, X., Xu, S., Liu, Y., Du, Y., and Ji, S. (2025). Artificial intelligence for science in quantum, atomistic, and continuum systems. arXiv:2307.08423.
25. Mohri, C., Duchi, J., and Hashimoto, T. (2026). A bitter lesson for data filtering. arXiv:2605.19407.