The pipeline that fails silently
- Why it happens
- A nightly load has been partially failing for weeks; downstream reports looked plausible so nobody noticed.
- How we work it
- Add row-count and freshness assertions at each stage, make loads atomic, fail the run rather than the record, alert on staleness rather than on job status.
- What you end up with
- Pipelines that stop instead of lying, with freshness SLAs published per dataset and an owner named for each.