Explainable AI for Data Analysis

Explainability is usually framed as a regulatory obligation. It is more useful framed as a debugging tool: an explanation you can check is an explanation that can be found wrong.

Interpretable by construction vs explained after the fact

A model built from named components explains itself. A post-hoc attribution method explains a black box approximately, and the approximation can be wrong in ways that are hard to detect. Both have uses; conflating them does not.

Model criticism

The step almost every automated system skips: not which model won, but whether the winner is any good in absolute terms. Simulating from the fitted model and comparing against the observations will show where the description breaks down.

Uncertainty is not confidence

A confident tone is not a calibrated interval. Systems that generate prose about data should be judged on whether the prose is derived from a fitted structure or generated alongside it.