Bayesian Statistics
Inference, model comparison and the razor that penalises complexity without a penalty term.
Bayesian inference treats parameters as uncertain quantities and updates beliefs about them with data. For automated analysis its most useful property is not philosophical but practical: the marginal likelihood provides a principled way to compare models of different complexity.
Occam’s razor, built in
A more flexible model spreads its predictive mass over more possible datasets, so it assigns less probability to the one actually observed. Complexity is penalised automatically, without an added term. This is what allows a model search to stop rather than always preferring the most elaborate structure.
Where priors bite
In small samples the prior does real work, which is a feature when it encodes something true and a liability when it encodes a default nobody examined. Automated systems should report the priors they used.