I m new to lakeFS We have now setup lakeFS on our own server

Question

I m new to lakeFS We have now setup lakeFS on our own server with MinIO providing the s3 interface So far this works as far as I can tell Now I want to read a simple iris data file from lakeFS that was uploaded manually to the main branch In my understanding this should be possible using the lakeFS API following this example <https pydocs lakefs io docs ObjectsApi html get object> I assum that the username password is equivalent to the Access Key pair I also assume that the host is the same port as used for the UI specified in lakectl yaml Unfortunately with this setup I receive the following error message Exception when calling ObjectsApi gt get object 401 Reason Unauthorized HTTP response headers HTTPHeaderDict Content Type application json X Content Type Options nosniff X Request Id 02494329 b95e 4652 98d8 5733efd05f70 Date Fri 27 Jan 2023 13 41 49 GMT Content Length 43 HTTP response body message error authenticating request Any help is appreciated

Or Tzabary · Answer

Hi < Jonas> welcome to lakeFS In order to use the API you should use access secret key pair generated for your user Is this the case The host and port should match the UI yes

Jonas · Answer

Hi and thanks for the fast reply Yes both is the case Unfortunately I still get the same error

Or Tzabary · Answer

Are you able to login to the UI using these credentials and download the object

Jonas · Answer

Well I was until someone has removed my credentials thanks it works now after resetting them slightly smiling face

Or Tzabary · Answer

upside down face happy to hear Happy lakeFSing

Or Tzabary · Answer

Is there anything else I can help with

Jonas · Answer

Not for now thank you very much

Jonas · Answer

< Or Tzabary> There is actually a question I have what is the intended recommended way to read images from lakeFS I have a small sample in my repo with the following structure repo | train | cat | png | dog | png | test | cat | png | dog | png One way I found was to us the lakeFS Python API to get a list of all images and to iterate over it using PIL on the io Buffer The issue though is that as far as I can tell I get all files in the branch so I cannot select the train or test set separately without filtering the paths on the results object Is that how it is supposed to work Would you recommend to keep that structure or do you suggest to use separate branches to store train and test data

Oz Katz · Answer

< Jonas> hey I think a directory structure is actually the correct way of doing this when calling the object listing api you can pass a prefix e g train dog and the listing api will only return objects under that prefix so it scales better and is less complex than filtering on the client

Oz Katz · Answer

does that make sense

Jonas · Answer

< Oz Katz> Prefect I was looking for something like that ~The doc just didn t show any possible prefix ~ I should learn to scroll down I was just looking for the signature shown at the very top

Jonas · Answer

Thank you slightly smiling face

Oz Katz · Answer

blush

Oz Katz · Answer

the docs could be improved however there s no clear example of how to use the prefix parameter so thanks for catching that slightly smiling face

Jonas · Answer

I was also not able to find an example of the get object for tabular data in your sample git repo If there is none I m happy to provide my code snippets to be included if that helps

Or Tzabary · Answer

pray skin tone 2 we appreciate contributions

Or Tzabary · Answer

thank you