Skip to content
Automated Data Analysis

How Automated Statistical Analysis Works

From data profiling through model search and scoring to generated explanation.

Key takeaways
  • – The pipeline is: profile, propose, fit, score, criticise, translate.
  • – Search is usually greedy: expand the best current structure, keep what improves the score.
  • – Scoring must penalise complexity or the search never terminates usefully.
  • – The criticism step is the one most systems omit, and the one that catches bad answers.

1. Profile

Types, ranges, missingness, cardinality, obvious duplicates, and the shape of the index. This step is cheap and catches a surprising share of real problems before any modelling starts.

2. Propose

Candidate structures are drawn from a grammar. In the Gaussian process formulation the base kernels are smooth (SE), periodic (PER), linear (LIN), changepoint (CP) and noise (WN), combined by addition and multiplication.

3. Fit and score

Each candidate is fitted and scored by marginal likelihood or an information criterion. Because a more flexible model spreads its probability mass more thinly, complexity is penalised automatically — see Bayesian Occam’s razor.

TEXTsearch.txt

SE score -1402
SE + PER score -1330
SE x PER score -1298
SE x PER + SE score -1284 <- selected SE x PER + SE + CP score -1286 [/code]

4. Criticise

Simulate from the fitted model and compare against the observations. Where they diverge systematically, the model is describing the data badly regardless of its score relative to the alternatives it happened to be compared with.

5. Translate

Each component maps to a phrase. The product of a periodic and a smooth term becomes "a cycle whose amplitude varies smoothly"; a changepoint becomes "behaviour changes around 2008". Because the sentence is generated from the structure, it can be verified against it.

Where it breaks

A grammar that cannot express the phenomenon will still return a winner. Search is greedy and can miss better structures behind a worse intermediate step. And nothing in the pipeline knows whether the question was worth asking.

Written by
Elizabeth Sramek

Leave a response