#help
Title
# help
r
Robin Moffatt
03/02/2023, 5:35 PMI'm referring to https://docs.lakefs.io/integrations/duckdb.html to try and work with some parquet files that I have in lakeFS. I'm trying to set the endpoint but the config's not taking (and if I try to query them it goes to S3)
Copy code
drones.duckdb> SET s3_endpoint='<http://rmoff-test.us-east-2.lakefscloud.io|rmoff-test.us-east-2.lakefscloud.io>';
0 rows in set
Time: 0.000s
drones.duckdb> SELECT * FROM duckdb_settings() where name like 's3_e%';
+-------------+------------------+------------------------------------------+------------+
| name | value | description | input_type |
+-------------+------------------+------------------------------------------+------------+
| s3_endpoint | <http://s3.amazonaws.com|s3.amazonaws.com> | S3 Endpoint (default '<http://s3.amazonaws.com|s3.amazonaws.com>') | VARCHAR |
+-------------+------------------+------------------------------------------+------------+
1 row in set
Time: 0.006shmmm the above was with
duckcliduckdb Copy code
D SET s3_endpoint='<http://rmoff-test.us-east-2.lakefscloud.io|rmoff-test.us-east-2.lakefscloud.io>';
D SELECT * FROM duckdb_settings() where name like 's3_e%';
┌─────────────┬──────────────────────┬────────────────────────────┬────────────┐
│ name │ value │ description │ input_type │
│ varchar │ varchar │ varchar │ varchar │
├─────────────┼──────────────────────┼────────────────────────────┼────────────┤
│ s3_endpoint │ rmoff-test.us-east… │ S3 Endpoint (default 's3… │ VARCHAR │
└─────────────┴──────────────────────┴────────────────────────────┴────────────┘b
Barak Amar
03/03/2023, 6:59 AM
The docs really talks about running the above from lakefs. but the same should work similar to query data files from duckdb with S3
will look into it
Running lakeFS locally access using duckdb:
Copy code
INSTALL httpfs;
LOAD httpfs;
SET s3_region='us-east-1';
SET s3_access_key_id='AKIAIOSFDNN7EXAMPLEQ';
SET s3_secret_access_key='wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY';
SET s3_endpoint='localhost:8000';
SET s3_use_ssl=false;
SET s3_url_style='path';
SELECT * FROM read_parquet('<s3://repo/main/userdata1.parquet>');I guess in your case you just need keep ssl true and it should work too.
4 Views