https://lakefs.io/ logo
Title
v

Vaibhav Kumar

10/19/2022, 12:56 PM
Hi I am trying to use airflow example DAG with my docker lakefs setup. when I create and test the connection on airflow GUI I get connection time out. Need some help
a

Ariel Shaqed (Scolnicov)

10/19/2022, 1:10 PM
Hi @Vaibhav Kumar, If you're using Docker, you probably need to expose lakeFS on localhost:8000 if you want to access it from another container. Also, make sure to remember to set credentials in the extra properties. It will look something like this:
$ airflow connections add conn_lakefs --conn-type=HTTP --conn-host=http://<LAKEFS_ENDPOINT> --conn-extra='{"access_key_id":"<LAKEFS_ACCESS_KEY_ID>","secret_access_key":"<LAKEFS_SECRET_ACCESS_KEY>"}'
v

Vaibhav Kumar

10/19/2022, 1:12 PM
Yes , extra property is set
Can you please share some command for exposing those ports
i

Iddo Avneri

10/19/2022, 1:14 PM
@Vaibhav Kumar let me know if

this webinar

is helpful for you working against this sample repo that includes airflow.
:lakefs: 1
:jiggling-lakefs: 1
a

Ariel Shaqed (Scolnicov)

10/19/2022, 1:28 PM
Could you share the command line you use to run lakeFS in Docker?
v

Vaibhav Kumar

10/19/2022, 5:11 PM
I used docker compose to run it @Ariel Shaqed (Scolnicov)
I will have to create some kind of bridge to let these containers communicate
a

Ariel Shaqed (Scolnicov)

10/19/2022, 5:22 PM
If you're in Docker for both containers... You may be able use the lakeFS container name as its address. Docker likes to introduce a fake DNS on what us usually is default internal network. See our "everything bagel docker compose" - that sets up a large number of applications that communicate with lakeFS, all in containers. https://lakefs.io/the-everything-bagel-ii-versioned-data-lake-tables-with-lakefs-and-trino/
v

Vaibhav Kumar

10/19/2022, 6:07 PM
I tried with the name of the lakefs docker container but the results are same , Please also find the screenshots of the running containers.
a

Ariel Shaqed (Scolnicov)

10/19/2022, 6:16 PM
Unfortunately I find it quite hard to read accurately from JPEGs, and they probably don't contain all the required information. Could you please send the docker-compose you use, or the docker command line?
v

Vaibhav Kumar

10/19/2022, 6:27 PM
this is the docker compose for lakefs I am using
a

Ariel Shaqed (Scolnicov)

10/19/2022, 6:30 PM
Thanks! That looks correct, so I'll try to reproduce tomorrow. How are you running Airflow?
v

Vaibhav Kumar

10/19/2022, 6:57 PM
a

Amit Kesarwani

10/19/2022, 9:49 PM
@Vaibhav Kumar You can use endpoint http://host.docker.internal:8000 to connect to lakeFS in another container
a

Ariel Shaqed (Scolnicov)

10/20/2022, 7:35 AM
Hi @Vaibhav Kumar, I'm sorry, I'm unable to run the Airflow docker-compose. I think it might rely on an older docker-compose version, or on some strange default security configuration pre-existing on the machine (it explicitly tries to run as
root
in l. 239
user: "0:0"
, so I will not run it on my machine). Instead I investigated. Fundamentally your problem is that Docker isolates different docker-compose files by default. @Amit Kesarwani explains one way to hit a container on another network, or you could make them run in the same network. I would probably merge the two Docker compose files (so all the
services
will live under the same
services:
key at the top level. If you want to keep them separate, you can add a network to your Airflow containers -- lakeFS will be on
lakefs_default
. See this StackOverflow answer (as well as the other answers on that question) for how to do it.
v

Vaibhav Kumar

10/20/2022, 5:57 PM
@Ariel Shaqed (Scolnicov) , I tried with what @Amit Kesarwani suggested. It worked. Thanks for helping me out as always. Appreciated .
👍 1
👍🏼 1