Hey all! I came across this article mentioning the use of lakeFS in the CI/CD of data pipelines. I’m wondering if anyone knows and can recommend of tools (OSS or not) that helps in continuous deployment of AI inference workflows in prod. The scenario: complex pipeline in prod serving users in real-time, involving several models and business logic that is constantly changing. Challenge: testing how a change will impact this DAG before landing a change (model, data, code) Any pointers will be greatly appreciated, and sorry if this is a noob question or out of context 🙂

Hi @Ori Kabeli! Thanks for posting your question! Can you please tell more about how your DAG is built and mention if you are using dbt?

Also, are you looking for a tool that supports continues deployment of code and data at once?

We’re using an internal framework for building our DAGs (equivalent to Airflow). dbt is not used. To be clear, this pipeline is essentially a backend logic that responds to a user request and is executed in real-time (not a batch pipeline). Unlike a standard backend endpoint, this one involves execution of several ML models and additional logic to produce a response. The aim is to test how a change to this pipeline effects past results. This can be something we build ourselves, but I’m wondering if there are tools out there that help with these integration/simulation tests.

PS I can also jump on a call to further discuss it if it helps! 🙂

It sounds like you are looking for a way to build a development environment in which you can test changes done to your pipeline before you apply them to production. lakeFS (for data) together with git (for code) can help you do exactly that! I’m sharing here a few blog posts that can give you ideas for how to build a development environment with lakeFS, and test your changes before you apply them to production. In any case, i’m happy to have a call to hear more about your use case and help with figuring out the right tooling and practices 🙂

Thanks Tal! let me look at these and get back to you