Hi I am trying to use airflow example DAG with my docker lakefs setup. when I create and test the connection on airflow GUI I get connection time out. Need some help

Hi @Vaibhav Kumar, If you're using Docker, you probably need to expose lakeFS on localhost:8000 if you want to access it from another container. Also, make sure to remember to set credentials in the extra properties. It will look something like this:

$ airflow connections add conn_lakefs --conn-type=HTTP --conn-host=http://<LAKEFS_ENDPOINT> --conn-extra='{"access_key_id":"<LAKEFS_ACCESS_KEY_ID>","secret_access_key":"<LAKEFS_SECRET_ACCESS_KEY>"}'

Yes , extra property is set

@Vaibhav Kumar let me know if is helpful for you working against this sample repo that includes airflow.

Could you share the command line you use to run lakeFS in Docker?

I used docker compose to run it @Ariel Shaqed (Scolnicov)

If you're in Docker for both containers... You may be able use the lakeFS container name as its address. Docker likes to introduce a fake DNS on what us usually is default internal network. See our "everything bagel docker compose" - that sets up a large number of applications that communicate with lakeFS, all in containers. https://lakefs.io/the-everything-bagel-ii-versioned-data-lake-tables-with-lakefs-and-trino/

I tried with the name of the lakefs docker container but the results are same , Please also find the screenshots of the running containers.

Unfortunately I find it quite hard to read accurately from JPEGs, and they probably don't contain all the required information. Could you please send the docker-compose you use, or the docker command line?

this is the docker compose for lakefs I am using

Thanks! That looks correct, so I'll try to reproduce tomorrow. How are you running Airflow?

I used airflow from here https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html

@Vaibhav Kumar You can use endpoint http://host.docker.internal:8000 to connect to lakeFS in another container

Hi @Vaibhav Kumar, I'm sorry, I'm unable to run the Airflow docker-compose. I think it might rely on an older docker-compose version, or on some strange default security configuration pre-existing on the machine (it explicitly tries to run as

root

in l. 239

user: "0:0"

, so I will not run it on my machine). Instead I investigated. Fundamentally your problem is that Docker isolates different docker-compose files by default. @Amit Kesarwani explains one way to hit a container on another network, or you could make them run in the same network. I would probably merge the two Docker compose files (so all the

services

will live under the same

services:

key at the top level. If you want to keep them separate, you can add a network to your Airflow containers -- lakeFS will be on

lakefs_default

. See this StackOverflow answer (as well as the other answers on that question) for how to do it.

@Ariel Shaqed (Scolnicov) , I tried with what @Amit Kesarwani suggested. It worked. Thanks for helping me out as always. Appreciated .