https://lakefs.io/ logo
Title
v

Vino

10/11/2022, 2:14 AM
Hello! I'm configuring Airbyte's S3 destination to use lakeFS as Airbyte sync following these steps. I'm using lakeFS playground and the storage namespace for my repo is
<s3://treeverse-demo-lakefs-storage-production/user_saved-katydid/my-repo>
, so what bucket name should I use to configure the Airbyte destination? Using either
treeverse-demo-lakefs-storage-production
or
my-repo
returns
could not connect to the bucket
error. What am I missing?
n

Niro

10/11/2022, 2:29 AM
Hi @Vino! I'll try to reproduce your issue, but before that can you please check if treeverse-demo-lakefs-storage-production/user_saved-katydid works?
v

Vino

10/11/2022, 2:33 AM
Sure. I get
Could not connect to the S3 bucket with the provided configuration. Bucket name should not contain '/'
error this time.
n

Niro

10/11/2022, 2:36 AM
Ok, thanks. It was just a hunch - I'll bring up an environment and get back to you in the following hour.
Can you please let me know which Airbyte version you are using?
v

Vino

10/11/2022, 2:41 AM
Sure. While you are reproducing it locally, here are the configs that I used: Airbyte Cloud Env:
Destination Type: S3
Destination Name: lakefs-airbyte-repo
S3 Key ID: <lakeFS Playground Access Key ID>
S3 Access Key: <lakeFS Playground Secret Key>
S3 Bucket Name: treeverse-demo-lakefs-storage-production
S3 Bucket Path: main/raw
S3 Bucket Region: us-east-1
Output Format: csv
Normalization: No Flattening
Compression: No compression
Endpoint: <https://saved-katydid.lakefs-demo.io/>
S3 Path Format: <left it empty>
S3 Filename Pattern: <left it empty>
👍🏽 1
@Niro I'm not running on Airbyte OSS. Spun up airbyte cloud free trial instead.
n

Niro

10/11/2022, 5:37 AM
@Vino the failure you've seen is due to code changes in Airbyte. I was able to reproduce it with the latest version and also successfully managed to configure the connector using an older version of Airbyte. I'm continuing the investigation and will provide additional updates soon
:jumping-lakefs: 1
💪🏻 1
💪 1
v

Vino

10/11/2022, 5:42 AM
Ofc. Thank you so much!