https://lakefs.io/ logo
Powered by
Title
t

Temilola Onaneye

03/08/2023, 7:05 PM
Hi, Please is it possible to read data like a csv or load a json file directly from a lakefs repo?? if yes, is there any documentation around it
g

Guy Hardonag

03/08/2023, 7:13 PM
Hi @Temilola Onaneye, lakeFS is data agnostic. Reading and Writing CSV or JSON is done the same as any other file.
I’d also add that lakeFS is S3 compatible so you can Read and Write CSV and JSON files the same way you would using your object store.
If you’d like to share some information about your use case, and how you’ve done it without lakeFS I can help you with running it on lakeFS
t

Temilola Onaneye

03/08/2023, 7:16 PM
Oh, I didn't complete my question, I meant with Python. Is it possible to do this directly with python??
Can I do this directly with Python like I would with any typical filesystem
a

Amit Kesarwani

03/08/2023, 7:24 PM
Python integration doc: https://docs.lakefs.io/integrations/python.html An example using Python and Spark to read CSV file: https://github.com/treeverse/lakeFS-samples/blob/main/03-multiple-samples/Spark%20Demo.ipynb
@Temilola Onaneye If above example is helpful then you can test this example/sample: https://github.com/treeverse/lakeFS-samples/tree/main/03-multiple-samples
This is also helpful: https://github.com/treeverse/lakeFS/tree/master/clients/python It lists all Python APIs including get_object to read file from lakeFS repo.
t

Temilola Onaneye

03/08/2023, 7:44 PM
So apparently, what I want to achieve is using pandas to read a csv directly from lakefs as a pandas dataframe. The documentation speaks to spark
g

Guy Hardonag

03/08/2023, 8:12 PM
Pandas can read/write remote files from S3 You could check out this URL for more information Due to the fact lakeFS is S3 compatible you will be able to read from lakeFS the same way you would read from S3 There are two slight changes you should consider: 1. you will need to set the S3 endpoint to be your lakeFS installations 2. you should take into consideration the 
repo
and 
branch
when accessing files Assuming your lakeFS is running on 
localhost:8000
and you want to access a file that exists on Repo: 
your-repo
Branch: 
your-branch
file-path: 
patth/to/file.csv
It should look something like this: 
df = pd.read_csv(
    "<s3://your-repo/your-branch/path/to/file.csv>",
    storage_options={
        "key": AWS_ACCESS_KEY_ID,
        "secret": AWS_SECRET_ACCESS_KEY,
        "token": AWS_SESSION_TOKEN,
        "client_kwargs": {"endpoint_url": "localhost:8000"}
t

Temilola Onaneye

03/08/2023, 8:56 PM
Thanks @Guy Hardonag
But Unfortunately, my object storage is azure blob
I was thinking of a workaround for the workflow. Since I am working with small json files in terms of size, I am considering downloading the files at runtime to the working dir and reading from there.
Don't know if direct download of the files is possible with python from lakefs repo. I only found file upload in the documentation shared
b

Barak Amar

03/08/2023, 8:59 PM
The above code uses lakefs S3 compatible API - your underlaying storage used by lakeFS can still be Azure
import pandas as pd

df = pd.read_csv(
    f"<s3://repo/main/data.csv>",
    storage_options={
        "key": "AKIAIOSFDNN7EXAMPLEQ",
        "secret": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
		"client_kwargs": {
			"endpoint_url": "<http://localhost:8000>",
		}
    },
)

print(df.to_string())
Same as @Guy Hardonag mentioned - the call here goes to lakefs, the example is running on a local instance
t

Temilola Onaneye

03/08/2023, 9:17 PM
Okay thanks, I will try this out
Thanks it works
:lakefs: 1
Hi @Barak Amar I wanted to find out if it is possible to download a data file from lakefs repo directly with python??
b

Barak Amar

03/09/2023, 11:07 AM
Hi, @Temilola Onaneye the above code does that. You can use python's 
boto
package or 
s3fs
(like the code above) - both uses lakeFS's S3 compatible API. lakeFS also provide API + python SDK 
lakefs_client
package - https://pydocs.lakefs.io/.
t

Temilola Onaneye

03/09/2023, 11:09 AM
Okay thanks, would check it out
b

Barak Amar

03/09/2023, 11:10 AM
Both talk directly with lakeFS which read the data from the underlying storage.
Let me know if you need help in using any of the above.
t

Temilola Onaneye

03/09/2023, 11:11 AM
I don't want to read it, I just want to download directly to a local filesystem
b

Barak Amar

03/09/2023, 11:11 AM
read == download for me 🙂
https://docs.lakefs.io/integrations/python.html
^ think here you will find what you are looking for
t

Temilola Onaneye

03/09/2023, 11:12 AM
Okay thanks
b

Barak Amar

03/09/2023, 11:17 AM
import boto3

session = boto3.session.Session()

s3_client = session.client(
    service_name='s3',
    aws_access_key_id='AKIAIOSFDNN7EXAMPLEQ',
    aws_secret_access_key='wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY',
    endpoint_url='<http://localhost:8000>',
)

s3_client.download_file('repo', 'main/userdata1.parquet', 'userdata1.parquet')
🙏 1
Here is example on how to download the file using boto
you can find s3fs example similar to the one we wrote yesterday
If you need one that uses lakefs api - let me know
🙏 1
t

Temilola Onaneye

03/09/2023, 11:21 AM
Yeah thanks Barak, it would be great to have the one that uses the lakefs api.
Yeah I need one that uses the lakefs api
b

Barak Amar

03/09/2023, 11:34 AM
import lakefs_client
from lakefs_client.client import LakeFSClient

configuration = lakefs_client.Configuration()
configuration.username = 'AKIAIOSFDNN7EXAMPLEQ'
configuration.password = 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY'
configuration.host = '<http://localhost:8000>'

client = LakeFSClient(configuration)
with (
        client.objects.get_object(repository='ugc', ref='main', path='userdata1.parquet') as f,
        open('userdata1.parquet', "wb") as o
    ):
    o.write(f.read())
🙌 1
t

Temilola Onaneye

03/09/2023, 11:35 AM
Okay thanks
:lakefs: 1
4 Views
#helpJoin Slack
Powered by