Hi, it looks like the api allows for only one file upload at a time - what’s the best practice for uploading an image folder (i.e. with subfolders, multiple images, csv with image index, etc) via lakeFS?

Hi @Seungchan Lee, I can think of two alternatives you can use- 1. In case you want to import data from S3, you can use lakeFS import capability. https://docs.lakefs.io/howto/import.html 2. You can use lakeFS S3 gateway which supports multipart uploads (I'm assuming you are using S3 according to your last message). https://docs.lakefs.io/reference/s3.html Please let me know if it was helpful or if I can help somehow else 🙏🏽 .

OK thanks! Just a quick follow up - if lakeFS S3 gateway supports multipart uploads, why isn’t there client api for bulk upload?

Hey @Seungchan Lee - nice to meet you! I want to better understand what you're trying to do.. is it: 1. Upload a single file in smaller chunks, in parallel (i.e. a multipart upload) 2. Upload multiple items concurrently?

I’m trying to upload multiple items concurrently - imagine creating an image dataset in s3 and having to add/update the dataset over time. We may use sdk or cli.

Got it! Thanks. Follow up question: the list of files on S3 - is that a directory or common prefix that you'd like to have synced, or an arbitrary list of different files scattered throughout the bucket?

Each dataset we create will have a unique prefix and we plan to match it to a lakeFS repo

got it! thanks again. I think this is a good use case for the zero-copy import functionality in lakeFS - it allows you to have lakeFS "point" to a set of objects in another location without actually having to copy the files into the lakeFS repository. If you import path

foo/

and later import that same path again on the same lakeFS branch, lakeFS would actually show you in its diff the changes that occured since last import (what was removed from that directory, what was added, etc. One thing to keep in mind - if you delete anything from

foo/

, naturally you won't be able to access it even from historical commits on the lakeFS side

Yeah this is what I’m a bit concerned about - if I change the underlying bucket (for example, delete some files), then lakeFS won’t be able to rollback

Copying in parallel is the alternative - you can use rclone / aws s3 sync / distcp or any other tool that allows parallel copying

But if I’m copying, I’m duplicating the dataset

True - that's the tradeoff. import doesn't require copy but can't guarantee immutability. Copying can guarantee immutability but requires storage..

Strange lakeFS hasn’t run into this issue before - I consider dataset creation and versioning a pretty basic use case

Once the data is managed in lakeFS, you enjoy deduplication across branches (i.e. creating another branch doesn't copy the data again). You then define garbage collection rules to determine when files can get cleaned up

I understand, but without bulk upload, how does one create a large dataset via lakeFS in the first place?

a single lakeFS instance would let you upload many hundreds (and even thousands) of objects in parallel

So I’m still a bit confused - your recommendation is then to use the lakeFS s3 gateway?

either to use the s3 gateway with the tools suggested above, or do so programatically with e.g. the lakeFS python SDK - set up multiple threads or routines that in parallel perform uploads.

OK so with the sdk, basically use:

client.objects.upload_object

But handle the concurrency myself

indeed

Or use the s3 gateway

I'm wondering, what would an ideal solution look like for you? what is lakeFS missing in your eyes to make that easier?

I think s3 gateway api might work but it’s more that the quickstart tutorials do not provide any usable examples for this. If I were to create an image dataset using lakeFS, what’s the best way to do it?

Thanks for the great feedback! we've been discussing this use case a lot lately and we do have some ideas on how to make it better. I'd love it if you could spare a few minutes some time this week for a quick session? would love to run some ideas by you 🙂

Oh sorry - one more question regarding the gateway

All you have to do is configure your s3 endpoint to point to your lakeFS server. See this example.

OK thank you

For each s3 client it might be configured a bit differently, but most of them allow overriding the endpoint url

Hmm this one is python sdk though - how would I use multipart upload with this?

whenever works for you

Got it thanks

cool, I'm at oz.katz@treeverse.io

Do you prefer another time?

will your 9am work?

Sure - let’s do 9am

cool, thanks!

No problem!

I did! looking forward to talking 🙂

Great- talk to you then!