Hi again guys Really like lakefs I have a problem regarding

Question

Hi again guys Really like lakefs I have a problem regarding reading and writing delta tables using spark It works but spark is huge and its complex and i will try everything to avoid it So my question to you guys is if there are any other way to read and write delta tables to lakefs avoiding spark

Itai Admi · Answer

Hi < Eirik Knævelsrud> welcome to the lake wave jumping lakefs Delta tables can be written to lakeFS regardless of the compute engine Spark in this case How do you normally read write delta tables without Spark Have you tried writing them to lakeFS in the same manner Could you expand some more on your use case

Robin Moffatt · Answer

there are a bunch of integrations listed here that you could use <https delta io integrations>

Eirik Knævelsrud · Answer

Ok thank you for fast response Ill do it the spark way now using this code sc jsc hadoopConfiguration set fs s3a path style access true sc jsc hadoopConfiguration set fs s3a access key access key sc jsc hadoopConfiguration set fs s3a secret key secret key sc jsc hadoopConfiguration set fs s3a endpoint sc jsc hadoopConfiguration set fs s3 impl org apache hadoop fs s3a S3AFileSystem df = sc read parquet parquet path header=True df write format delta mode overwrite save delta save path So im using python at this point and will continue with that Im looking for the most light way approach of doing this logic Thanks for the link of integrations the most lightway i can see is this Any experience Or other approaces

Lynn Rozen · Answer

As Itai mentioned it would be great if you can elaborate a bit so we can fully understand and help with your use case slightly smiling face How do you currently using delta tables regardless of lakeFS And do you mean you are using pyspark What do you mean looking for the most light way approach of doing this logic

Eirik Knævelsrud · Answer

To start Im totally new to this so dont shoot me if its a stupid question Yes im using pyspark I use pyspark to read the parquet and to write that df as a deltatable Im having azure data lake as foundation and lakefs on top of this So my question is How can i do this without spark or any other havier systems My desired approach would be something like 1 pip isntall python package 2 Set up connection to lakefs without spark Use boto3 3 pythonPackage readParquet 4 pythonpackage writeAsDelta

Oz Katz · Answer

This could be useful using Delta rs with lakeFS S3 Gateway ```from deltalake import DeltaTable Replace with your lakeFS configuration and credentials lakefs = AWS ACCESS KEY ID AKIAIOSFODNN7EXAMPLE AWS SECRET ACCESS KEY wJalrXUtnFEMI K7MDENG bPxRfiCYEXAMPLEKEY AWS ENDPOINT URL AWS REGION us east 1 dt = DeltaTable storage options=lakefs df = dt to pandas ```

Eirik Knævelsrud · Answer

yes < Oz Katz> im looking for something like this For now im just have lakefs installed on a local minikube environment with port forward on lakefs service In other words hosting lakefs locally How is the auth info then

Oz Katz · Answer

Probably the same values you used for Spark access key secret and endpoint the region is likely `us east 1` unless you explicitly changed the s3 gateway configuration in your lakeFS server

Robin Moffatt · Answer

I m trying to build a notebook for this to go in but can t get it to work ```storage options = AWS ACCESS KEY ID lakefsAccessKey AWS SECRET ACCESS KEY lakefsSecretKey AWS ENDPOINT lakefsEndPoint AWS STORAGE ALLOW HTTP true ``` ``` AWS ACCESS KEY ID AKIAIOSFOLKFSSAMPLES AWS SECRET ACCESS KEY wJalrXUtnFEMI K7MDENG bPxRfiCYEXAMPLEKEY AWS ENDPOINT AWS STORAGE ALLOW HTTP true ``` ```deltalake write deltalake table or uri= data = dataframe storage options=storage options ``` Fails with ```Generic S3 error response error lt xml version= 1 0 encoding= UTF 8 gt lt Error gt lt Code gt MissingFields lt Code gt lt Message gt Missing fields in request lt Message gt lt Resource gt lt Resource gt lt Region gt us east 1 lt Region gt lt RequestId gt fc5ae35f aaa6 4891 b0e4 715b574c894a lt RequestId gt lt HostId gt 79C9987FC956D83E lt HostId gt lt Error gt after 0 retries HTTP status client error 400 Bad Request for url ``` in the lakeFS log is ```level=error msg= log header does not match v2 structure func= pkg gateway sig V2SigAuthenticator Parse file= build pkg gateway sig v2 go 94 header=Authorization level=warning msg= failed to parse signature func=pkg gateway AuthenticationHandler func1 file= build pkg gateway middleware go 41 error=MissingFields``` any ideas

Oz Katz · Answer

I wonder why sigv2 is even getting used here Which version of Python Delta rs lakeFS are you using Also you probably need to set `AWS REGION` for this to work

Oz Katz · Answer

Where is `AWS STORAGE ALLOW HTTP` coming from I m not familiar with that config value

Robin Moffatt · Answer

```Python version 3 10 10 | packaged by conda forge | main Mar 24 2023 19 56 21 GCC 11 3 0 deltalake version 0 10 1 lakeFS version 0 106 0```

Robin Moffatt · Answer

`AWS STORAGE ALLOW HTTP` is buried in the S3 crate documentation I stumbled across it in <https github com delta io delta rs discussions 796 discussioncomment 4608911|a delta rs GH issue>

Robin Moffatt · Answer

okay this worked ```storage options = AWS ACCESS KEY ID lakefsAccessKey AWS SECRET ACCESS KEY lakefsSecretKey AWS ENDPOINT lakefsEndPoint AWS REGION us east 1 AWS STORAGE ALLOW HTTP true AWS S3 ALLOW UNSAFE RENAME true ```

Robin Moffatt · Answer

< Eirik Knævelsrud> you can see working code example here Let me know if this is different from what you re trying to do

Eirik Knævelsrud · Answer

Ok thank you very much Ill check it tomorrow morning and will give gove you an update Atleast its this way i wanted

Eirik Knævelsrud · Answer

got this error Traceback most recent call last File pathToPythonFile test2 py line 61 in lt module gt deltalake write deltalake table or uri= <http s3a bucketr branch deltaTable> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File C Users project venv Lib site packages deltalake writer py line 148 in write deltalake table table uri = try get table and table uri table or uri storage options ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File C Users project venv Lib site packages deltalake writer py line 393 in try get table and table uri table = try get deltatable table or uri storage options ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File C Users project venv Lib site packages deltalake writer py line 406 in try get deltatable return DeltaTable table uri storage options=storage options ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File C Users project venv Lib site packages deltalake table py line 238 in init self table = RawDeltaTable ^^^^^^^^^^^^^^ internal DeltaError Delta protocol violation Failed to serialize operation expected value at line 1 column 1

Robin Moffatt · Answer

at a glance `table or uri= bucketr branch deltaTable ` should probably be `table or uri= `

Eirik Knævelsrud · Answer

dont know why slack fortmatted it with http Thats not the case in my code

Robin Moffatt · Answer

can you share the code you re running

Eirik Knævelsrud · Answer

for sure import fsspec import pandas as pd import deltalake import lakefs client from lakefs client models import from lakefs client client import LakeFSClient import boto3 from lakefs client api import objects api from deltalake writer import write deltalake import s3fs import pyarrow as pa lakefsEndPoint = MINIKUBE SERVICE lakefs service n namespace Forwarding port etc lakefsAccessKey = key lakefsSecretKey = secret repo name = internal import os os environ AWS ACCESS KEY ID = lakefsAccessKey os environ AWS SECRET ACCESS KEY = lakefsSecretKey os environ AWS ENDPOINT = lakefsEndPoint os environ AWS REGION = us east 1 Not sure about this one but just did what you said lakeFS credentials and endpoint configuration = lakefs client Configuration configuration username = lakefsAccessKey configuration password = lakefsSecretKey configuration host = lakefsEndPoint lakefs = LakeFSClient configuration storage options = AWS ACCESS KEY ID lakefsAccessKey AWS SECRET ACCESS KEY lakefsSecretKey AWS ENDPOINT lakefsEndPoint AWS REGION us east 1 Not sure about this one but just did what you said AWS STORAGE ALLOW HTTP true AWS S3 ALLOW UNSAFE RENAME true fs = s3fs S3FileSystem endpoint url= secret=lakefsSecretKey key=lakefsAccessKey with fs open rb as f This works as expected df = pd read parquet f df = df astype str print df head Initialize the Delta table deltalake write deltalake table or uri= data=df mode= overwrite storage options=storage options

Robin Moffatt · Answer

hmmm I don t know sorry I d suggest trying the Delta Lake community e g and also their Slack I d also maybe start from the notebook that I shared and see if you can modify it one step at a time towards your code and see where it starts to break

Eirik Knævelsrud · Answer

yes i will try thank you alot for the help I think this is more of a compabillity issue in delta lake more than lakefs