DLT Expectations as Quality Gates, Fail, Drop, or Quarantine

Pull-quote: “Every expectation encodes a business judgment about how bad a bad record is. The only question is whether the business made that judgment, or a default did.”

Why this matters

Declarative pipelines on Databricks come with a quality mechanism built into the table definition: expectations. An expectation is a named predicate attached to a DLT table, evaluated on every record that flows in, with a declared response when the predicate fails. The mechanics take an afternoon to learn. The design question they expose is the durable one: for each rule, should a violation warn, drop the record, or stop the pipeline, and who in the organization is entitled to make that call? Most teams answer by accident. The default gets copied from the last pipeline, and a business judgment ships as a code habit.

The three responses, and the fourth

Quarantine deserves its asterisk: it is not a fourth keyword but a pattern built from the primitives, typically by splitting flow with inverted rules so failing records land in a companion table instead of vanishing. It exists because drop has a hidden cost. A dropped record is an unanswerable question during an incident review. A quarantined record is a preserved piece of evidence with a timestamp and a reason attached.

Severity is a business scale

The right response is a function of what the violation means downstream, which is exactly the knowledge engineering does not have alone.

   Violation severity, read from the consumer's side
   ─────────────────────────────────────────────────
   cosmetic          suspicious           structural            poisonous
   (trailing         (out-of-range        (null key,            (wrong grain,
   whitespace,       amount, future       broken reference)     duplicate primary
   casing)           date)                                      keys, bad join)
      │                  │                     │                     │
      ▼                  ▼                     ▼                     ▼
    warn             quarantine              drop                  fail
   and track         and review           and count            and page someone

A finance consumer may declare that any out-of-range amount must halt the pipeline, because a monthly close built on one wrong number is worse than a late close. An operations dashboard may prefer dropped records and an alert, because staleness costs more than a small gap. Same predicate, opposite verdicts, and only the data’s owner and its consumers can rank them. That is the meaning of the “who decides” clause: expectation policy is data governance expressed in pipeline code, so the severity mapping belongs to the data owner, in writing, with engineering implementing rather than improvising it.

Operating the gates

Three practices keep expectations honest after the first deploy. Publish the numbers: DLT records pass, drop, and violation counts in pipeline metrics, and those figures belong on the same dashboards as freshness, because a table that is on time but quietly dropping four percent of records is not healthy. Review the quarantine: a holding table nobody reads is a slower delete; every quarantined record needs an owner and a disposition path, fixed upstream, reprocessed, or formally rejected. Revisit the mapping: severity judgments drift as consumers change, and an expectation set that has not been reviewed since launch is a snapshot of last year’s risk tolerance enforcing itself on this year’s data.

Closing

Expectations make data quality declarative, visible, and cheap to enforce, which removes the last excuse for leaving quality policy implicit. Write the predicates with engineering, but put the fail, drop, or quarantine decision where it belongs: with the people who own the data and live with its failures. The pipeline enforces the policy. It should not be the one that wrote it.