Is there a way to turn off/change CORS settings wh...
# help
s
Is there a way to turn off/change CORS settings when running LakeFS in a docker container? I was trying to hit the LakeFS API (localhost:8000 from localhost:3000, for example)
i
Hi @Steve Willard LakeFS is agnostic to this, whatever you configure the listen address be enough for it ( See the lakeFS config) Other than that it might be something else in your browser or another server running.
s
what about in the case of generating a presigned s3 url for lakefs, and uploading it from a browser? If I used nginx, for example, which allowed CORS…. can I generate a presigned url for nginx, but rewrite it for the underlying lakefs server?
i
Im not quite sure I fully understand your use case, can you explain what are you trying to achieve?
s
I have an internal tool that is a web UI. That UI has a way to select potentially large files, and upload them into a lakeFS repository. Ideally I’d like to directly upload to LakeFS from the browser, and avoid large data hitting another server. From the browser, I’d be ok with either using the AWS sdk to upload, or uploading to a presigned URL to LakeFS, if that’s possible. I can’t currently go from the web UI -> LakeFS because they are hosted on different domains.
i
That Web UI -> lakeFS problem with occur with any server redirect not just lakeFS. You should host them under the same domain if possible. Usually testing those things locally is done with some proxy like Nginx or via dev browser configuration that are not secure for real world
👍 1
s
Assuming I had an nginx proxy to get around the cors issue… does lakefs support something like pre-signed URL uploads? Basically I’m wondering if I can avoid potential problems with uploading large files to lakefs, and maybe just tell it it exists after uploading it
i
Yes, it’s supported. I think I better understand your use case now so I compiled some options for you to test, including the CORS issue and uploading to lakeFS via pre-sign url 🙂
First, besides using NGINX, for local development if you want to work a UI let’s say on localhost:3000 (some React App) and the lakeFS server on localhost:8000. You can use NGINX but this is exactly what happens in lakeFS when develop our own UI, we use a reverse proxy so here’s a reference: Run npm run dev and the UI will start with a reverse-proxy on localhost:3000 which will speak with lakeFS on localhost:8000 - check the reverse proxy config. Using that same logic you can achieve with your own ui for local development. In production they would sit on the same domain of course. Second, regarding pre-sign URL: both put and get is available (for S3, GCS, Azure - not for local storage or anything else). Check this doc reference: To use this functionality in the lakeFS UI: you would have to allow CORS on your bucket policy and in lakeFS config enable in the UI. To use this functionality in lakectl: check lakectl fs upload and lakectl fs cat example. To write your own API either use a generated SDK or based on our HTTP OAPI spec then for pre-sign URL best would be to look at what
lakectl
does, for example to upload an object with a pre-sign feature look at clientUploadPreSignHelper API calls to lakeFS: First it calls getPhysicalAddress to get a presigned URL to upload (without the need for AWS credentials, just lakeFS) and then LinkPhysicalAddressWithResponse.
s
Using the links you provided to look at
lakectl
was super helpful. I ended up generating presigned URLs and linking them with another lakefs API call. That was super easy, and it worked great — thank you!!!!
jumping lakefs 1
i
Glad that worked out! 🤝