Your Pipeline Learned to Call Yesterday's Breakage Normal

Databricks made data quality monitoring generally available in June 2024 with a feature that needs no manual configuration: it watches every table's freshness and completeness by learning what each table normally does.

Per the product documentation, the system 'analyzes the history of commits to a table and builds a per-table model to predict the time of the next commit'; if a commit is unusually late, the table is marked stale. Completeness works the same way — it predicts a range of expected row counts from history and flags volumes below the lower bound. Databricks 'automatically scans each table at the same frequency it's updated,' with no per-table setup. The baseline for 'good' is the table's own past behavior.

This is enormously useful and quietly load-bearing. Statistical baselining catches the breaks no one wrote a rule for, which is most of them. But 'anomalous' and 'wrong' are different words. A table that has shipped duplicate rows every morning for a month has taught the model that duplicates are Tuesday. The monitor will defend the steady state, including the parts of the steady state that are already broken, because nobody supplied an external definition of correct for it to compare against.

It reveals the field's central substitution: we adopted observability faster than we adopted definitions. Monte Carlo's 2023 survey of 200 data professionals found respondents averaging 67 incidents a month, detection taking four-plus hours, resolution averaging fifteen, and — the quiet part — 74% saying business stakeholders spot the issue first, up from 47% the year before. A learned baseline tells you a number moved. It cannot tell you the number was wrong, because no one ever told it what right was.

Watch for the gap between a freshness alert and a freshness SLA. The first says the data changed later than usual; the second says someone promised consumers a deadline and named who answers when it slips. Anomaly detection is a fine smoke alarm and a poor fire code. Use it to catch the unruly, but keep the human-authored thresholds — the contract, the owner, the definition of 'good' — for the things you actually promised. A model that learns normal will eventually ratify whatever you tolerate.