Bayesian Model Selection Explained

Why a more flexible model can be less probable, and how that lets a search terminate.

Elizabeth Sramek Published 1 September 2026 1 MIN READ

Key takeaways

– The marginal likelihood is the probability of the data under a model, averaged over its parameters.
– Flexible models spread that probability thinly, so they lose to simpler models that fit adequately.
– No penalty term is added; the razor is a consequence of averaging.
– Information criteria approximate this when the exact quantity is intractable.

The quantity

Rather than the best fit a model can achieve, Bayesian model selection asks how probable the observed data is under the model as a whole, integrating over parameters weighted by the prior. A model with many parameters can fit many datasets; it must therefore assign less probability to each.

p(D \\mid M) = \\int p(D \\mid \\theta, M)\\, p(\\theta \\mid M)\\, d\\theta

Marginal likelihood — the evidence for model M.

Why this matters for automation

A search that scored candidates by best fit alone would always prefer the most elaborate structure, and would never stop. Scoring by evidence gives the search a natural termination point and produces the parsimonious structures that make an explanation readable.

The practical caveats

The integral is rarely tractable, so BIC, WAIC or cross-validation stand in for it. The evidence is also sensitive to the prior in ways the maximum likelihood is not — a vague prior on a parameter that matters can dominate the comparison. Automated systems should report the priors they used.

Comparing models is not validating one

Selection tells you which candidate is best supported. It says nothing about whether any candidate is adequate. That is the job of model criticism.

Written by

Elizabeth Sramek

Elizabeth Sramek is an independent advisor on search visibility and demand architecture for B2B companies operating in high-competition markets. Based in Prague and working globally, she specializes in designing search presence for AI-mediated discovery and building category visibility that survives algorithmic shifts.

LinkedIn elizabethsramek.com All articles