Hello, I'm new to lakeFS, and trying it out for my work. The infrastructure team has set up an instance (I believe the production instance) of lakeFS but they couldn't get s3 to work for now so they said I should use local storage.

Hi, the local storage should be used when running lakeFS locally on your device. If you’re looking to use lakeFS on a remote machine you should configure an S3 bucket it’ll make more sense. Why can’t they get s3 to work?

Thanks! Is that just what is recommended or is there absolutely no way to use local storage when lakeFS is running on Kubernetes then? This is just for a trial/test to see if lakeFS is useful for us so it doesn't need to be perfect, just a minimal working example for now.

Hi Anna, LakeFS can work on local storage. Import doesn’t work at this point in time on local storage (you can use the import functionality working with cloud storage). I would be happy to jump on a call with the infrastructure team and help configure S3 if that would be helpful. Would that be helpful?

Potentially, yes, I've asked them if they're willing to configure s3 again and will update you if they have any issues, but they've finished work for the day (we're UK based). Any pointers on how to ingest data from local storage for now? If import doesn't work, does ingest still work?

Upload still works.

hello! thanks for the help. I got stuck on this because I created a repository in the bucket local://example-bucket/ and then deleted it while I was figuring out what was wrong before you messaged, and now I tried to create it again but I got the following error: failed to create repository: found lakeFS objects in the storage namespace(local://example-bucket): lakeFS repositories can't share storage namespace any idea what this is?

Hi Anna, you should create the repository with a different namespace, storage namespace is underlying path to your new repository. For example, “Repository A” with S3 might be:

<s3://annas-bucket/a>

another “Repository B” with S3 could be

<s3://annas-bucket/b>

So in your case the error you get saying that that specific path

<local://example-bucket>

is already reserved for a previous repository you created. Try giving it something else, even

<local://example-bucket/test>

or something

thanks! so once I have created a repository with a given name, that name will be forever "reserved", even after I've deleted the repo?

Specifically regarding storage namespace: we never delete it, it might be a very complex and nuanced operation. Generally regarding deleting data we don’t delete the data rather mark it with a tombstone, eventually deleted data will get picked up in different scenarios by our GC mechanism you can read more about here.

ah that's very useful to know, thanks so much!

YW 🙏