Eirik Knævelsrud
08/23/2023, 10:58 AMItai Admi
08/23/2023, 11:05 AMRobin Moffatt
08/23/2023, 11:06 AMEirik Knævelsrud
08/23/2023, 12:07 PMLynn Rozen
08/23/2023, 12:29 PMEirik Knævelsrud
08/23/2023, 12:34 PMOz Katz
from deltalake import DeltaTable
# Replace with your lakeFS configuration and credentials
lakefs = {
"AWS_ACCESS_KEY_ID": "AKIAIOSFODNN7EXAMPLE",
"AWS_SECRET_ACCESS_KEY": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
"AWS_ENDPOINT_URL": "<https://my.lakefs.installation.com>",
"AWS_REGION": "us-east-1"
}
dt = DeltaTable("<s3://my-repo/branch/path/to/delta_table/>", storage_options=lakefs)
df = dt.to_pandas()
Eirik Knævelsrud
08/23/2023, 1:30 PMOz Katz
us-east-1
unless you explicitly changed the s3 gateway configuration in your lakeFS serverRobin Moffatt
08/23/2023, 1:38 PMstorage_options = {"AWS_ACCESS_KEY_ID": lakefsAccessKey,
"AWS_SECRET_ACCESS_KEY":lakefsSecretKey,
"AWS_ENDPOINT": lakefsEndPoint,
"AWS_STORAGE_ALLOW_HTTP": "true"
}
{'AWS_ACCESS_KEY_ID': 'AKIAIOSFOLKFSSAMPLES',
'AWS_SECRET_ACCESS_KEY': 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY',
'AWS_ENDPOINT': '<http://lakefs:8000>',
'AWS_STORAGE_ALLOW_HTTP': 'true'}
deltalake.write_deltalake(table_or_uri='<s3a://example/main/userdata/>',
data = dataframe,
storage_options=storage_options)
Fails with
Generic S3 error: response error "<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>MissingFields</Code><Message>Missing fields in request.</Message><Resource></Resource><Region>us-east-1</Region><RequestId>fc5ae35f-aaa6-4891-b0e4-715b574c894a</RequestId><HostId>79C9987FC956D83E</HostId></Error>", after 0 retries: HTTP status client error (400 Bad Request) for url (<http://lakefs:8000/example/main/userdata/_delta_log/_last_checkpoint>)
in the lakeFS log is
level=error msg="log header does not match v2 structure" func="pkg/gateway/sig.(*V2SigAuthenticator).Parse" file="build/pkg/gateway/sig/v2.go:94" header=Authorization
level=warning msg="failed to parse signature" func=pkg/gateway.AuthenticationHandler.func1 file="build/pkg/gateway/middleware.go:41" error=MissingFields
any ideas?Oz Katz
AWS_REGION
for this to work.AWS_STORAGE_ALLOW_HTTP
coming from? I'm not familiar with that config value..Robin Moffatt
08/23/2023, 2:13 PMPython version 3.10.10 | packaged by conda-forge | (main, Mar 24 2023, 19:56:21) [GCC 11.3.0]
deltalake version 0.10.1
lakeFS version 0.106.0
AWS_STORAGE_ALLOW_HTTP
is buried in the S3 crate documentation, I stumbled across it in a delta-rs GH issuestorage_options = {"AWS_ACCESS_KEY_ID": lakefsAccessKey,
"AWS_SECRET_ACCESS_KEY":lakefsSecretKey,
"AWS_ENDPOINT": lakefsEndPoint,
"AWS_REGION": "us-east-1",
"AWS_STORAGE_ALLOW_HTTP": "true",
"AWS_S3_ALLOW_UNSAFE_RENAME": "true"
}
Eirik Knævelsrud
08/23/2023, 3:20 PMRobin Moffatt
08/24/2023, 8:09 AMtable_or_uri='<http://s3a>://bucketr/branch/deltaTable',
should probably be table_or_uri='<s3a://bucketr/branch/deltaTable>',
Eirik Knævelsrud
08/24/2023, 8:13 AMRobin Moffatt
08/24/2023, 8:14 AMEirik Knævelsrud
08/24/2023, 8:19 AMRobin Moffatt
08/24/2023, 8:26 AMEirik Knævelsrud
08/24/2023, 8:28 AM