Adi Polak01/12/2023, 9:03 AM
💡 When data quality issues occur, the circuit opens, preventing low-quality data from propagating to downstream processes
🎯 Result? This data will be missing in the reports for time periods of low quality
👉 you’re automating data availability to be directly proportional to data quality
🔥 No more fire-fighting & verifying-&-fixing metrics/reports on a case-by-case basisThis probably reads familiar to you as in the :lakefs: world we refer to it as CI/CD for data pipelines .. I wonder what you think about this approach.. any insights?