#help
Title
# help
t
Temilola Onaneye
03/08/2023, 7:05 PMHi, Please is it possible to read data like a csv or load a json file directly from a lakefs repo??
if yes, is there any documentation around it
g
Guy Hardonag
03/08/2023, 7:13 PMHi @Temilola Onaneye, lakeFS is data agnostic.
Reading and Writing CSV or JSON is done the same as any other file.
I’d also add that lakeFS is S3 compatible so you can Read and Write CSV and JSON files the same way you would using your object store.
If you’d like to share some information about your use case, and how you’ve done it without lakeFS I can help you with running it on lakeFS
t
Temilola Onaneye
03/08/2023, 7:16 PMOh, I didn't complete my question, I meant with Python.
Is it possible to do this directly with python??
Can I do this directly with Python like I would with any typical filesystem
a
Amit Kesarwani
03/08/2023, 7:24 PMPython integration doc: https://docs.lakefs.io/integrations/python.html
An example using Python and Spark to read CSV file: https://github.com/treeverse/lakeFS-samples/blob/main/03-multiple-samples/Spark%20Demo.ipynb
@Temilola Onaneye If above example is helpful then you can test this example/sample: https://github.com/treeverse/lakeFS-samples/tree/main/03-multiple-samples
This is also helpful: https://github.com/treeverse/lakeFS/tree/master/clients/python
It lists all Python APIs including get_object to read file from lakeFS repo.
t
Temilola Onaneye
03/08/2023, 7:44 PMSo apparently, what I want to achieve is using pandas to read a csv directly from lakefs as a pandas dataframe.
The documentation speaks to spark
g
Guy Hardonag
03/08/2023, 8:12 PMPandas can read/write remote files from S3
You could check out this URL for more information
Due to the fact lakeFS is S3 compatible you will be able to read from lakeFS the same way you would read from S3
There are two slight changes you should consider:
1. you will need to set the S3 endpoint to be your lakeFS installations
2. you should take into consideration the
repobranchlocalhost:8000your-repoyour-branchpatth/to/file.csv Copy code
df = pd.read_csv(
"<s3://your-repo/your-branch/path/to/file.csv>",
storage_options={
"key": AWS_ACCESS_KEY_ID,
"secret": AWS_SECRET_ACCESS_KEY,
"token": AWS_SESSION_TOKEN,
"client_kwargs": {"endpoint_url": "localhost:8000"}t
Temilola Onaneye
03/08/2023, 8:56 PMThanks @Guy Hardonag
But Unfortunately, my object storage is azure blob
I was thinking of a workaround for the workflow. Since I am working with small json files in terms of size, I am considering downloading the files at runtime to the working dir and reading from there.
Don't know if direct download of the files is possible with python from lakefs repo.
I only found file upload in the documentation shared
b
Barak Amar
03/08/2023, 8:59 PM
The above code uses lakefs S3 compatible API - your underlaying storage used by lakeFS can still be Azure
Copy code
import pandas as pd
df = pd.read_csv(
f"<s3://repo/main/data.csv>",
storage_options={
"key": "AKIAIOSFDNN7EXAMPLEQ",
"secret": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
"client_kwargs": {
"endpoint_url": "<http://localhost:8000>",
}
},
)
print(df.to_string())t
Temilola Onaneye
03/08/2023, 9:17 PMOkay thanks, I will try this out
Thanks it works
lakefs 1
Hi @Barak Amar
I wanted to find out if it is possible to download a data file from lakefs repo directly with python??
b
Barak Amar
03/09/2023, 11:07 AM
Hi, @Temilola Onaneye the above code does that.
You can use python's
botos3fslakefs_clientt
Temilola Onaneye
03/09/2023, 11:09 AMOkay thanks, would check it out
b
Barak Amar
03/09/2023, 11:10 AM
Both talk directly with lakeFS which read the data from the underlying storage.
Let me know if you need help in using any of the above.
t
Temilola Onaneye
03/09/2023, 11:11 AMI don't want to read it, I just want to download directly to a local filesystem
b
Barak Amar
03/09/2023, 11:11 AM
read == download for me 🙂
^ think here you will find what you are looking for
t
Temilola Onaneye
03/09/2023, 11:12 AMOkay thanks
b
Barak Amar
03/09/2023, 11:17 AM
Copy code
import boto3
session = boto3.session.Session()
s3_client = session.client(
service_name='s3',
aws_access_key_id='AKIAIOSFDNN7EXAMPLEQ',
aws_secret_access_key='wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY',
endpoint_url='<http://localhost:8000>',
)
s3_client.download_file('repo', 'main/userdata1.parquet', 'userdata1.parquet')🙏 1
Here is example on how to download the file using boto
you can find s3fs example similar to the one we wrote yesterday
If you need one that uses lakefs api - let me know
🙏 1
t
Temilola Onaneye
03/09/2023, 11:21 AMYeah thanks Barak, it would be great to have the one that uses the lakefs api.
Yeah I need one that uses the lakefs api
b
Barak Amar
03/09/2023, 11:34 AM
Copy code
import lakefs_client
from lakefs_client.client import LakeFSClient
configuration = lakefs_client.Configuration()
configuration.username = 'AKIAIOSFDNN7EXAMPLEQ'
configuration.password = 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY'
configuration.host = '<http://localhost:8000>'
client = LakeFSClient(configuration)
with (
client.objects.get_object(repository='ugc', ref='main', path='userdata1.parquet') as f,
open('userdata1.parquet', "wb") as o
):
o.write(f.read())🙌 1
t
Temilola Onaneye
03/09/2023, 11:35 AMOkay thanks
lakefs 1
48 Views