Channels
announcements
blockers_for_windward
career-opportunities
celebrations
cuddle-corner
data-discussion
data-events
dev
events
general
help
iceberg-integration
lakefs-for-beginners
lakefs-hubspot-cloud-registration-email-automation
lakefs-releases
lakefs-suggestions
lakefs-twitter
linen-dev
memes-and-banter
say-hello
stackoverflow
test
Title
r
Robin Moffatt
03/02/2023, 5:35 PMI'm referring to https://docs.lakefs.io/integrations/duckdb.html to try and work with some parquet files that I have in lakeFS. I'm trying to set the endpoint but the config's not taking (and if I try to query them it goes to S3)
drones.duckdb> SET s3_endpoint='<http://rmoff-test.us-east-2.lakefscloud.io|rmoff-test.us-east-2.lakefscloud.io>';
0 rows in set
Time: 0.000s
drones.duckdb> SELECT * FROM duckdb_settings() where name like 's3_e%';
+-------------+------------------+------------------------------------------+------------+
| name | value | description | input_type |
+-------------+------------------+------------------------------------------+------------+
| s3_endpoint | <http://s3.amazonaws.com|s3.amazonaws.com> | S3 Endpoint (default '<http://s3.amazonaws.com|s3.amazonaws.com>') | VARCHAR |
+-------------+------------------+------------------------------------------+------------+
1 row in set
Time: 0.006shmmm the above was with
duckcliduckdbD SET s3_endpoint='<http://rmoff-test.us-east-2.lakefscloud.io|rmoff-test.us-east-2.lakefscloud.io>';
D SELECT * FROM duckdb_settings() where name like 's3_e%';
┌─────────────┬──────────────────────┬────────────────────────────┬────────────┐
│ name │ value │ description │ input_type │
│ varchar │ varchar │ varchar │ varchar │
├─────────────┼──────────────────────┼────────────────────────────┼────────────┤
│ s3_endpoint │ rmoff-test.us-east… │ S3 Endpoint (default 's3… │ VARCHAR │
└─────────────┴──────────────────────┴────────────────────────────┴────────────┘b
Barak Amar
03/03/2023, 6:59 AMThe docs really talks about running the above from lakefs. but the same should work similar to query data files from duckdb with S3
will look into it
Running lakeFS locally access using duckdb:
INSTALL httpfs;
LOAD httpfs;
SET s3_region='us-east-1';
SET s3_access_key_id='AKIAIOSFDNN7EXAMPLEQ';
SET s3_secret_access_key='wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY';
SET s3_endpoint='localhost:8000';
SET s3_use_ssl=false;
SET s3_url_style='path';
SELECT * FROM read_parquet('<s3://repo/main/userdata1.parquet>');I guess in your case you just need keep ssl true and it should work too.