Using existing bucket on lakefs: I would like to use the currently existing bucket location of lakefs, but when I tried to use it shows following error. There are some files on the directory on s3 and I want to use that exact same location on S3. I want to use NodeJS for programmatically access and how can I do that?

failed to create repository: found lakeFS objects in the storage namespace(s3://i-am-bigdata) key(_lakefs/dummy): storage namespace already in use

@Nethsara Do you mean you want to use lakeFS with existing objects in S3? A lakeFS repository can be created only in a path that doesn't contain any objects. You can then use the import functionality to import existing objects from S3 to the repository

thanks @Niro, When the service is starting I need to create a repo, now I am creating, but sometimes it is not working as expected, are there any workaround to handle this?

Can you please provide the error message

image.png

This happens on the create repository command?

there is no repo is created even though

What is your underlying storage service?

S3

This can happen if your AWS credentials are not configured properly

No, If I create repo from UI it works

OK I see. Are you using a lakectl.yaml file?

in docker

That is the lakeFS configuration. the Python SDK package takes the client credentials from either a lakectl.yaml file or environment variables (see here). To test your lakefs package is configured correctly you can try to create a repository in the UI (for example "test") and then try to query for its metadata using the python package

print(lakefs.repository("test").metadata)

Can you tell me if this works for you?

image.png

Not sure I understand the question. You can access existing repositories using the lakefs python package.

Lets say there is lakefs on location s3://test/1, and after restarting the service I want to use that same repo on that

Restart the lakeFS server?

yeah, this is on docker so it restarts sometimes. I want to continue from where I left off

Which database is configured for the lakeFS server?

local

local on a docker?

yeah

As long as you configure the volume on which the local database path is pointing to to be persistent, that should not be a problem. Keep in mind though this is not recommended as a production setup

you mean to use a DB? so it will resume where I left off

Yes, also with a DB, the volume must be persistent

got it

GL

Thanks