Hi, I am just wondering if there are any plans in the near future to make

lakectl local

more like

git

? I would just like to say, I have so far really enjoyed using

lakefs

, and it seems to be the answer to my pain point for a long time. I mainly use it as I would use git -- to version data through the

lakectl local

interface. However, compared to git, it's missing quite a lot of features, which makes it, while useable, somewhat finicky when I want to do things in certain ways. Here I list a few examples: 1. I would like to checkout only a directory or file rather than the entire repository for a commit with

lakectl checkout

. 2. I would like to examine changes to certain files between different commits, similar to

git difftool

. Are there plans to include features such as the above in the near future? Or am I not quite using

lakefs

in the intended way?

Hey @Timothy Mak, welcome to the lake lakefs Did you have a chance to explore our docs? 1. There’s a

lakectl local

example there that copies a prefix instead of a whole commit. 2.

lakectl diff

accepts 2 lakeFS refs (like

<lakefs://repo/branch>

) and a prefix (like

images/dogs/

) and presents the changes between them.

Hi Itai, thanks for the reply. I'm still fairly new to lakefs, so am not too familiar with the docs. I had a look at what you suggested just now. Some comments: 1. I can't find the example where you use

lakectl local

to copy (checkout?) a prefix instead of a whole commit. 2. I see that

lakectl diff

does compare two refs, but they have to be different branches. This is somewhat different from

git

where I often compare different commits just to see changes were made in them. What's the rationale that

diff

allows comparison between branches but not commits? How would you advise structuring our workflow so as to be able to better keep track of the differences between the commits? Perhaps by having many branches?

lakectl local accepts commits in the form of their complete lakeFS refs:

lakefs://<REPO>/<COMMIT_ID>

I see. That's really useful to know. I hope some days there will be a guide to these things. Thanks again for all your help.

Hi @Timothy Mak, Could you explain what you'd like to do with diff? It accepts two diffs, and they certainly don't have to belong to different branches. In fact, lakeFS -- like Git -- has no concept of a commit belonging to a branch. Could you give an example of a diff command on Git? Then we could try to give a lakectl example, or understand where we fail.

Thanks Ariel. I think i understand from Itai’s answer how to do what i want now. But i wouldn’t have figured this out by looking at the docs or the user guides. so it would help if there were more guides on this topic. i guess one of the points of confusion was why is there “lakectl” and “lakectl local” where some commands such as “commit” and “status” fall under the latter and others such as “diff” fall under the former. I come from a background where i just train machine learning models but i don’t have understanding of the internals of databases.

Point well taken, and I completely understand. Given that lakectl local is a bit newer than some parts, I hope we can and will improve our documentation. I know that's small consolation for someone who needs to figure things out today, though...

Thanks for the script @Ariel Shaqed (Scolnicov)

Sure thing. Hope it helps, or at least serves as the base for a quick hack that does 90% of something useful.