Hi there 🙂 I'm working with Kafka and wanted to try your LakeFS. S3LinkConnector works great with LakeFS, I'm successfuly sending data from Kafka to LakeFS repo, but I have a problem with S3SourceConnector. I'm using the same credentials, same store.url (and it's my LakeFS demo url), same topic etc. but it crashes with AmazonS3Exception. In logs it looks like this: ERROR Unable to read objects using the configured S3 credentials, bucket, and path. (io.confluent.connect.s3.source.S3SourceConnectorValidation:105) AmazonS3Exception: 400 BadRequest (Service: Amazon S3; Status Code: 400; Error Code: BadRequest: Request ID... Does anyone know what I did wrong? Is there any additional configuration/properties I should add to the source connector to make it work?

Hi @Olagfigh, without sharing any secrets and you share how the S3SourceConnector is configured? Make sure you specify the branch you are reading from - include trailing '/'

Hi @Barak Amar, thanks for quick response. Both connectors have the same configurations so I guess the urls etc. are correct. S3LinkConnector works fine. For S3SourceConnector I have provided these properties: connector.class=...S3SourceConnector format.class=...JsonFormat flush.size=1 s3.bucket.name=LakeFS-repo-name aws.access.key.id=my-id-from-LakeFS aws.secret.access.key=my-secret-key-from-LakeFS storage.class=...S3Storage topics.dir=LakeFS-branch-name topics=Kafka-topic-name name=connectorName confluent.topic.bootstrap.servers=server-with-Kafka store.url=url-to-LakeFS key.converter=...StringConverter value.converter=...StringConverter plugin.path=path-to-S3SourceConnector-plugin offset.storage.file.filename=offset

@Olagfigh the issue is not with the S3SourceConnector. The problem is with the lakeFS's S3 gateway. While listing the connector uses the custom delimiter ".json" and currently it is blocked as unsupported by our code. I'll open an issue (bug) to track the issue and give estimation on how much time it will take to fix it tomorrow.

@Barak Amar Thanks for investigation and help 🙂

Hi again @Barak Amar. I have changed format.class to AvroFormat and again got the same error as before. Is there any workaround to this issue?

Hi @Olagfigh, we are working on the issue, a fix will be released next week. In the meantime we will check if there is any workaround and update today

Hey @Olagfigh, we couldn’t think of any workaround, sorry for the late response. We will provide a fix next week.

Hi @Guy Hardonag, thanks for info. Let me know please when fix will be released. Have a great weekend 🙂

Will do Have a great weekend 🙂

@Olagfigh lakeFS release v0.67.0 holds the above fix. (s3 listing with custom delimiter)

@Barak Amar Thanks. Is that version released already?

yes, yesterday

That's great, thanks once more 🙂 Cheers

Hi @Olagfigh, I might be reading the text incorrectly - JPEG text is murder on my eyes, please copy and paste text so that it becomes accessible. But I think you are accessing demo.lakefs.io, which is probably not the correct endpoint. I don't think it speaks S3.

Sorry guys, my bad. Now I got again this: ```[2022-06-20 122908,279] DEBUG http-outgoing-0: set socket timeout to 0 (org.apache.http.impl.conn.DefaultManagedHttpClientConnection:88) [2022-06-20 122908,279] DEBUG Connection released: [id: 0][route: {s}->https://close-panda.lakefs-demo.io:443][total available: 1; route allocated: 1 of 50; total allocated: 1 of 50] (org.apache.http.impl.conn.PoolingHttpClientConnectionManager:351) [2022-06-20 122908,291] TRACE Class 'com.sun.xml.internal.stream.XMLInputFactoryImpl' not found. Delegating to parent (org.apache.kafka.connect.runtime.isolation.PluginClassLoader:99) [2022-06-20 122908,302] DEBUG Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: 400 BadRequest (Service: Amazon S3; Status Code: 400; Error Code: BadRequest; Request ID: 880f3805-6e2b-4c8a-b467-d9668686d552; S3 Extended Request ID: 2F82337482963829), S3 Extended Request ID: 2F82337482963829 (com.amazonaws.request:1783) [2022-06-20 122908,304] ERROR Unable to read objects using the configured S3 credentials, bucket, and path. (io.confluent.connect.s3.source.S3SourceConnectorValidation:105) com.amazonaws.services.s3.model.AmazonS3Exception: 400 BadRequest (Service: Amazon S3; Status Code: 400; Error Code: BadRequest; Request ID: 880f3805-6e2b-4c8a-b467-d9668686d552; S3 Extended Request ID: 2F82337482963829), S3 Extended Request ID: 2F82337482963829 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1799) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1383) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1359) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:796) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5052) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4998) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4992) at com.amazonaws.services.s3.AmazonS3Client.listObjectsV2(AmazonS3Client.java:938) at io.confluent.connect.s3.source.S3Storage.listUpToMaxObjects(S3Storage.java:126) at io.confluent.connect.s3.source.S3SourceConnectorValidation.validateS3ReadPermissions(S3SourceConnectorValidation.java:98) at io.confluent.connect.s3.source.S3SourceConnectorValidation.performValidation(S3SourceConnectorValidation.java:55) at io.confluent.connect.utils.validators.all.ConfigValidation.validate(ConfigValidation.java:185) at io.confluent.connect.cloud.storage.source.CompositeSourceConnector.validate(CompositeSourceConnector.java:101) at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:471) at org.apache.kafka.connect.runtime.AbstractHerder.lambda$validateConnectorConfig$2(AbstractHerder.java:367) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) [2022-06-20 122908,306] DEBUG Completed validating configuration for …

demo was created today but still with version 0.66.0 but fix was prepared for v0.67.0

Hey @Olagfigh, you are correct, the demo doesn’t contain the fix yet. We are upgrading it today

thanks :)

Updated playground to use lakeFS v0.67.0. Use new env to use it.