Hi I am trying to use airflow example DAG with my ...
# dev
Hi I am trying to use airflow example DAG with my docker lakefs setup. when I create and test the connection on airflow GUI I get connection time out. Need some help
Hi @Vaibhav Kumar, If you're using Docker, you probably need to expose lakeFS on localhost:8000 if you want to access it from another container. Also, make sure to remember to set credentials in the extra properties. It will look something like this:
Copy code
$ airflow connections add conn_lakefs --conn-type=HTTP --conn-host=http://<LAKEFS_ENDPOINT> --conn-extra='{"access_key_id":"<LAKEFS_ACCESS_KEY_ID>","secret_access_key":"<LAKEFS_SECRET_ACCESS_KEY>"}'
Yes , extra property is set
Can you please share some command for exposing those ports
@Vaibhav Kumar let me know if

this webinar

is helpful for you working against this sample repo that includes airflow.
lakefs 1
jiggling lakefs 1
Could you share the command line you use to run lakeFS in Docker?
I used docker compose to run it @Ariel Shaqed (Scolnicov)
I will have to create some kind of bridge to let these containers communicate
If you're in Docker for both containers... You may be able use the lakeFS container name as its address. Docker likes to introduce a fake DNS on what us usually is default internal network. See our "everything bagel docker compose" - that sets up a large number of applications that communicate with lakeFS, all in containers. https://lakefs.io/the-everything-bagel-ii-versioned-data-lake-tables-with-lakefs-and-trino/
I tried with the name of the lakefs docker container but the results are same , Please also find the screenshots of the running containers.
Unfortunately I find it quite hard to read accurately from JPEGs, and they probably don't contain all the required information. Could you please send the docker-compose you use, or the docker command line?
this is the docker compose for lakefs I am using
Thanks! That looks correct, so I'll try to reproduce tomorrow. How are you running Airflow?
@Vaibhav Kumar You can use endpoint http://host.docker.internal:8000 to connect to lakeFS in another container
Hi @Vaibhav Kumar, I'm sorry, I'm unable to run the Airflow docker-compose. I think it might rely on an older docker-compose version, or on some strange default security configuration pre-existing on the machine (it explicitly tries to run as
in l. 239
user: "0:0"
, so I will not run it on my machine). Instead I investigated. Fundamentally your problem is that Docker isolates different docker-compose files by default. @Amit Kesarwani explains one way to hit a container on another network, or you could make them run in the same network. I would probably merge the two Docker compose files (so all the
will live under the same
key at the top level. If you want to keep them separate, you can add a network to your Airflow containers -- lakeFS will be on
. See this StackOverflow answer (as well as the other answers on that question) for how to do it.
@Ariel Shaqed (Scolnicov) , I tried with what @Amit Kesarwani suggested. It worked. Thanks for helping me out as always. Appreciated .
👍 1
👍🏼 1