Hi, I’m just getting started with lakeFS and a little confused about the workflow. In git, you usually commit -> push. For lakeFS, there’s no push, so if I want to add objects to my dataset in s3, how do I do that?

Welcome to the lake @Seungchan Lee!

Typically using lakeFS, you would have a production branch (main) and any type of transformation / ingestion will be done on a separate branch, and then, merged into main.

For example, let’s say you want to run an ETL against your data: 1. You create a branch from prod (main) - a zero clone copy operation 2. You make the changes on your isolated branch. 3. Once you are ready, you merge the changes back into main, promoting the data to production

OK great thanks - so when does commit happen?

When you execute it.

You branch out -> upload new data -> commit -> merge back to main if required?

For example, yes.

That’s a good ingest example.

Or for transformation, what I mentioned above.

Wait - I don’t see the difference? Is transformation workflow different? Isn’t it still branch out -> transform -> commit -> merge back?

(only mentioned it because you mentioned “upload new data”) You are correct that it is the same concept

Ah ok

What about if you’re running an automated workflow that does ETL? Is it basically the same process?

Yup

Do you use an airflow / dagster / kubeflow / other orchestrator?

Yes Flyte

Which is just another workflow engine - but for kubernetes

You will have the orchestrator run the ETL in a branch.

RIght

We have a good example with airflow:

OK this is helpful

I’ll watch it and then try it out with my project. Thank you!

Sure thing. Hope you enjoy lakeFS!

It looks very promising - thank you!

If you have additional questions, feel free to post those.

Definitely - thanks!

Oh sorry one more quick thing - I was planning to use mlflow for experiment tracking but a recent blog post in lakeFS mentions it can also track experiments. Any good resource for me to dig into how lakeFS does it and what the pros/cons are compared to something like mlflow?

That’s a great point. lakeFS will version the data on top of (as opposed to instead of).

Another resource available is

We are actually working on another ML webinar that might be even more relevant. Stay tuned 🙂

OK thank you!

Enjoy!