Hey all, first time poster. S3 gateway question:

I'm trying to get Spark (via AWS Glue) to talk to my LakeFS running on an EC2. I can use my LakeFS UI on port 8000 and that is working fine. To connect from Spark, I'm providing the EC2 IP address as described in the Spark settings documentation. However my glue job fails because it cant connect to port 80 on my EC2 (connection refused). I presume this means that the S3 gateway is expected to be on Port 80. If I telnet to my EC2 on port 80, I also can't connect. So I suspect the S3 gateway component isn't running as part of my LakeFS environment. Or perhaps I misunderstand how to use it. Any advice would be helpful :)

The docs here https://docs.lakefs.io/understand/architecture.html#s3-gateway say "To achieve this, the gateway exposes an HTTP listener on a dedicated host and port." What I am missing are the details of where this listener is / how to start it.

Hi Bjorn, welcome aboard! LakeFS uses a single port for both the UI and the S3 gateway components. If you can access the UI on port 8000, you can direct Spark to the same port and it should work. Please let us know if that helped 🙂

Thanks, I'll give that a try 🙂

That worked, thank you. I think the docs should be updated a bit as this wasn't clear to me. I'll log an issue on Github with the details 🙂

I'm glad that helped! Any input regarding the docs would be awesome. Thanks!