We often spend hours days and even weeks analyzing whether a

Question

We often spend hours days and even weeks analyzing whether a metric is correct exploding head Sandeep Uttamchandani Engineering Unravel shared an interesting approach <https modern cdo medium com taming data quality with circuit breakers dbe550d3ca78|circuit breakers> lt quality gates gt for production data pipelines gt TL DR gt bulb When data quality issues occur the circuit opens preventing low quality data from propagating to downstream processes gt dart Result This data will be missing in the reports for time periods of low quality gt point right you re automating data availability to be directly proportional to data quality gt fire No more fire fighting amp verifying amp fixing metrics reports on a case by case basis This probably reads familiar to you as in the lakefs world we refer to it as CI CD for data pipelines I wonder what you think about this approach any insights