Hi i am new to lakefs and i can write little bit of go ,can we extend lakefs to support other s3 storage options as well ,example: https://github.com/joyent/manta . Can some one please help me here with any pointers/documentations so that i can try implementing the same

Hello @sumesh kanayi lakeFS supports several types of a block storage, which is the underlying object store, such as:

S3

,

GCS

,

Azure Blob Storage

and few non-prod like

local

and

mem

. All block stores implements the same adapter, so it should be the best place to start. Read our contribution guide for general guidelines. I am also interested to know what is the potential use-case for Manta<->lakeFS integration and how we could help Manta users even more.

thanks @Itai Admi . I work in a project where data security is top notch priority and dont want to send anything to internet. We are already using manta inside our private cloud but want to see the possibility of integrating all the great features of lakefs with manta as backend .

BTW I did find this project, that let your clients use s3 operations with manta but it is archived. If you’re familiar with any other project that let your apps communicate with manta the same way they would with S3, you could use lakeFS with the S3 adapter without implementing a designated one for manta.

Hi i am stuck at this point where i am trying to commit an object .I can see files are getting stored in manta but in UI it fails with error.Can any one tell me what is going on here

commit: close writer ns=<manta://sumesh-demo-2> id=: batch write closer wait: sstable store (2c9f1ab23b771c1702cfa9ef3ff72dfe023cfd1a876dadf254732a7bb08a0f03): closing file 2c9f1ab23b771c1702cfa9ef3ff72dfe023cfd1a876dadf254732a7bb08a0f03: close /Users/sdata/lakefs/cache/range/sumesh-demo-2/workspace/d4b16c91-a56c-4588-85ad-579efe6d01b9: file already closed

It seems like getting failed here https://github.com/treeverse/lakeFS/blob/6a01ee35453fee68b784500a8a7062ac10f75dcc/pkg/graveler/sstable/writer.go#L149

Hi @sumesh kanayi, Can you please provide some more details? what files do you see in manta? do you see files under the path

_lakefs

? Does it happen every time you try to commit? What lakeFS version are you using?

Hi @Guy Hardonag i can see files getting created under _lakefs

mls ~~/stor/sumesh-demo-2/_lakefs

2c9f1ab23b771c1702cfa9ef3ff72dfe023cfd1a876dadf254732a7bb08a0f03

one thing noticed is that i cant see any

actions/log

directory under _lakefs .I am running the latest version from master.Yes this happens every-time when i commit

Thats OK, Seems like the issue is that lakeFS manages to write the file and when trying to close the file locally it fails so it doesn’t save the commit.

log is kept here @Guy Hardonag https://gist.github.com/sumeshkanayi/0019537044dbd51ec682e21a31834475 . This was the first commit .

Hi @sumesh kanayi, which version of lakeFS are you using?

Hi @Yoni Augarten i am here https://github.com/treeverse/lakeFS/commit/9d35f14eba038648902a59dbb091b27590525e87 .cloned from master

Thanks. And you're running lakeFS on your local machine? A cloud provider? Something else?

yes its on my mac .

This information is very helpful

i will check and let you know this @Yoni Augarten

The closing is done by default in Golang's http. Notice what we did on

block/s3/adapter.go:192

- we wrap the reader with a NopCloser.

you are right @Yoni Augarten its failing at

tier_gs.go:162

.Got it..

block/s3/adapter.go:192

certainly is super helpful.I will let you know how it goes

🙏🏻 here if you need anything else

Hi guyz, i have a question around

func (l *Adapter) Remove(_ context.Context, obj block.ObjectPointer)

as you can see under

<https://github.com/treeverse/lakeFS/blob/master/pkg/block/local/adapter.go#L143>

couldn't find out when it really comes under play. Do we need to implement it to remove objects/buckets ? When i tried using "local" block i dont see it really deleting from file system but its updated under UI as deleted .So what exactly this method do?

Hey @sumesh kanayi , You didn’t find it is because it currently isn’t used :) Delete object in lakeFS is a metadata command, meaning it won’t be visible in staging and won’t have a reference in the next commit. The reason objects can’t be deleted is that they may be referenced from previous commits. The function Remove implements a

hard delete

and it will be used in the future (probably as part of this issue).

ohh 🙂 this was a surprise for sure..thanks a lot @Guy Hardonag ..