I'm referring to <https://docs.lakefs.io/integrati...
# helpr
Robin Moffatt
03/02/2023, 5:35 PMI'm referring to https://docs.lakefs.io/integrations/duckdb.html to try and work with some parquet files that I have in lakeFS. I'm trying to set the endpoint but the config's not taking (and if I try to query them it goes to S3)
Copy code
drones.duckdb> SET s3_endpoint='<http://rmoff-test.us-east-2.lakefscloud.io|rmoff-test.us-east-2.lakefscloud.io>';
0 rows in set
Time: 0.000s
drones.duckdb> SELECT * FROM duckdb_settings() where name like 's3_e%';
+-------------+------------------+------------------------------------------+------------+
| name | value | description | input_type |
+-------------+------------------+------------------------------------------+------------+
| s3_endpoint | <http://s3.amazonaws.com|s3.amazonaws.com> | S3 Endpoint (default '<http://s3.amazonaws.com|s3.amazonaws.com>') | VARCHAR |
+-------------+------------------+------------------------------------------+------------+
1 row in set
Time: 0.006sRobin Moffatt
03/02/2023, 5:54 PMhmmm the above was with
duckcliduckdb Copy code
D SET s3_endpoint='<http://rmoff-test.us-east-2.lakefscloud.io|rmoff-test.us-east-2.lakefscloud.io>';
D SELECT * FROM duckdb_settings() where name like 's3_e%';
┌─────────────┬──────────────────────┬────────────────────────────┬────────────┐
│ name │ value │ description │ input_type │
│ varchar │ varchar │ varchar │ varchar │
├─────────────┼──────────────────────┼────────────────────────────┼────────────┤
│ s3_endpoint │ rmoff-test.us-east… │ S3 Endpoint (default 's3… │ VARCHAR │
└─────────────┴──────────────────────┴────────────────────────────┴────────────┘b
Barak Amar
03/03/2023, 6:59 AM
The docs really talks about running the above from lakefs. but the same should work similar to query data files from duckdb with S3
Barak Amar
03/03/2023, 6:59 AM
will look into it
Barak Amar
03/03/2023, 7:13 AM
Running lakeFS locally access using duckdb:
Copy code
INSTALL httpfs;
LOAD httpfs;
SET s3_region='us-east-1';
SET s3_access_key_id='AKIAIOSFDNN7EXAMPLEQ';
SET s3_secret_access_key='wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY';
SET s3_endpoint='localhost:8000';
SET s3_use_ssl=false;
SET s3_url_style='path';
SELECT * FROM read_parquet('<s3://repo/main/userdata1.parquet>');Barak Amar
03/03/2023, 7:13 AM
I guess in your case you just need keep ssl true and it should work too.
5 Views