Typically using lakeFS, you would have a production branch (main) and any type of transformation / ingestion will be done on a separate branch, and then, merged into main.

For example, let’s say you want to run an ETL against your data: 1. You create a branch from prod (main) - a zero clone copy operation 2. You make the changes on your isolated branch. 3. Once you are ready, you merge the changes back into main, promoting the data to production

That’s a good ingest example.

Or for transformation, what I mentioned above.

(only mentioned it because you mentioned “upload new data”) You are correct that it is the same concept

What about if you’re running an automated workflow that does ETL? Is it basically the same process?

Do you use an airflow / dagster / kubeflow / other orchestrator?

Which is just another workflow engine - but for kubernetes

I’ll watch it and then try it out with my project. Thank you!

Oh sorry one more quick thing - I was planning to use mlflow for experiment tracking but a recent blog post in lakeFS mentions it can also track experiments. Any good resource for me to dig into how lakeFS does it and what the pros/cons are compared to something like mlflow?

Another resource available is

We are actually working on another ML webinar that might be even more relevant. Stay tuned 🙂