Hello! I'm configuring Airbyte's S3 destination to use lakeFS as Airbyte sync following these steps. I'm using lakeFS playground and the storage namespace for my repo is

<s3://treeverse-demo-lakefs-storage-production/user_saved-katydid/my-repo>

, so what bucket name should I use to configure the Airbyte destination? Using either

treeverse-demo-lakefs-storage-production

or

my-repo

returns

could not connect to the bucket

error. What am I missing?

Hi @Vino! I'll try to reproduce your issue, but before that can you please check if treeverse-demo-lakefs-storage-production/user_saved-katydid works?

Sure. I get

Could not connect to the S3 bucket with the provided configuration. Bucket name should not contain '/'

error this time.

Ok, thanks. It was just a hunch - I'll bring up an environment and get back to you in the following hour.

Sure. While you are reproducing it locally, here are the configs that I used: Airbyte Cloud Env:

Destination Type: S3
Destination Name: lakefs-airbyte-repo
S3 Key ID: <lakeFS Playground Access Key ID>
S3 Access Key: <lakeFS Playground Secret Key>
S3 Bucket Name: treeverse-demo-lakefs-storage-production
S3 Bucket Path: main/raw
S3 Bucket Region: us-east-1
Output Format: csv
Normalization: No Flattening
Compression: No compression
Endpoint: <https://saved-katydid.lakefs-demo.io/>
S3 Path Format: <left it empty>
S3 Filename Pattern: <left it empty>

When I need a quick lakeFS installation for a week, I usually try demo.lakefs.io.

@Niro I'm not running on Airbyte OSS. Spun up airbyte cloud free trial instead.

@Vino the failure you've seen is due to code changes in Airbyte. I was able to reproduce it with the latest version and also successfully managed to configure the connector using an older version of Airbyte. I'm continuing the investigation and will provide additional updates soon

Ofc. Thank you so much!