https://lakefs.io/ logo
Join Slack
Powered by
Hello everyone, can anyone explain to me why my la...
# help
d
Hello everyone, can anyone explain to me why my lakefs files are not reflected in MinIO? I'm using Trino with Delta tables, the entire versioning process works, only the files that are not reflected in MinIO. The branch 
<lakefs://data-platform-silver/main/customers/>
is not reflected in 
<s3://data-platform-silver/main/customers>
n
Hi @Davi Gomes, When lakeFS manages your data, the physical address of the data is different from the logical one. Therefore you will not find it in the underlying storage where you'd expect. To understand more you can read this great article written by our colleague
d
Thanks @Niro
levitating lakefs 1
I understand how it works, I'm now looking at export, but I'm not able to export to MinIO via Docker. I've already tried setting AWS_ENDPOINT, but it doesn't work. Do you have any ideas?
There is this example in the documentation, but I didn't find a list of variables that I can send in the command.
Copy code
docker run \
    -e LAKEFS_ACCESS_KEY_ID=XXX -e LAKEFS_SECRET_ACCESS_KEY=YYY \
   -e LAKEFS_ENDPOINT=https://<LAKEFS_ENDPOINT>/ \
   -e AWS_ACCESS_KEY_ID=XXX -e AWS_SECRET_ACCESS_KEY=YYY \
   treeverse/lakefs-rclone-export:latest \
      example-repo \
      <s3://destination-bucket/prefix/> \
      --branch="example-branch"
n
Are you looking to export data from lakeFS? What exactly does not work?
d
Yes, I need test to export data from lakeFS. I'm running this command:
Copy code
docker run \
    -e LAKEFS_ACCESS_KEY_ID=celerodata -e LAKEFS_SECRET_ACCESS_KEY=celero123 \
    -e LAKEFS_ENDPOINT=<http://192.168.1.60:8000/> \
    -e AWS_ACCESS_KEY_ID=lzYdM55vUqPc4PHbinLp -e AWS_SECRET_ACCESS_KEY=wXuCcfkeaBrxuKpsAEAHW2kEetjWwsfBktMSxace -e AWS_ENDPOINT="<http://192.168.1.60:9000>" \
    treeverse/lakefs-rclone-export:latest \
    data-platform-gold \
    <s3://data-platform-metastore/> \
    --branch="main"
And receive this message:
Copy code
024/10/18 20:34:22 ERROR : S3 bucket data-platform-metastore: error reading destination root directory: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
	status code: 403, request id: 701HV30M75GCS0AC, host id: 1rhMDFWyjBptK8qrbno5CxlSvbwVnU+aWgVJFRyYzpjYauxp0zhN8cxDNX1+1ZWLg8aPrIYNAPg=
But, the access_key and secret_access_key are corrected.
n
So the error reported is that the AWS_ACCESS_KEY_ID you provided is invalid. This could mean 1 of 2 things: 1. You provided the wrong AWS_ACCESS_KEY_ID (this should be credentials of the underlying storage and not of lakeFS) 2. Your environment configuration is not setup correctly and is trying to access a different endpoint (AWS for example instead of your minio server)
d
In this case is option 2, I need configure the endpoint with environment variable.
AWS ENDPOINT does not work. In this case, is there another variable?
n
Can you try with 
AWS_ENDPOINT_URL
 instead?
d
Unfortunately no, I've already tried 
AWS_ENDPOINT
, 
AWS_ENDPOINT_URL
, 
S3_ENDPOINT
and 
S3_ENDPOINT_URL
n
Would you mind submitting a bug on this issue?
d
Okay
Thanks
n
Thank you
h
would it be simplier to just install rclone and do a 
rclone sync
??
lakefs 1
15 Views
Open in Slack
PreviousNext
Powered by