Gaussian Processes
A distribution over functions, a kernel that carries the assumptions, and uncertainty that comes out of the model rather than bolted onto it.
A Gaussian process specifies what a plausible function looks like rather than fixing a functional form and estimating coefficients. The covariance function — the kernel — carries every assumption about smoothness, periodicity and growth, and predictions arrive as a mean with a variance.
Why they matter here
Kernels are closed under addition and multiplication. That closure turns the set of possible models into a grammar, which is what makes automated structure search possible and keeps the discovered model interpretable: each term corresponds to a phrase in ordinary language.
Practical constraints
Exact inference costs O(n³) time and O(n²) memory, so beyond a few thousand points you need sparse approximations or structured kernels. High-dimensional inputs degrade the notion of distance the kernel depends on; for wide tabular data, trees usually win.
In this section