Adi Polak

01/12/2023, 9:03 AM
We often spend hours, days, and even weeks analyzing whether a metric is correct! 🤯 Sandeep Uttamchandani (Engineering @ Unravel) shared an interesting approach: circuit-breakers<quality gates> for production data pipelines!
💡 When data quality issues occur, the circuit opens, preventing low-quality data from propagating to downstream processes
🎯 Result? This data will be missing in the reports for time periods of low quality
👉 you’re automating data availability to be directly proportional to data quality
🔥 No more fire-fighting & verifying-&-fixing metrics/reports on a case-by-case basis
This probably reads familiar to you as in the :lakefs: world we refer to it as CI/CD for data pipelines .. I wonder what you think about this approach.. any insights?
💡 4