Hey I understand you are trying to install

Question

Hey < Vaibhav Kumar> I understand you are trying to install airflow using Docker Compose and running into an issue when adding the lakeFS airflow provider as a requirement Can you share the final Compose file you are trying this with

Vaibhav Kumar · Answer

Here is the curl as well `curl LfO <https airflow apache org docs apache airflow 2 4 1 docker compose yaml> `

Barak Amar · Answer

Hi < Vaibhav Kumar> I think the issue is that you set the additional lakefs package by editing the docker compose at the place it is set to the init container This is a fix to solve the issue of installing packages as root

Barak Amar · Answer

You need to pass the value as environment variable and it will set it at the right places except this one

Barak Amar · Answer

A quick way to do it is creating a ` env` file in the same directory with the values you like ``` PIP ADDITIONAL REQUIREMENTS=airflow provider lakefs```

Barak Amar · Answer

I ll also recommend ```echo e AIRFLOW UID=$ id u gt gt env``` based on the docs

Barak Amar · Answer

let me know if it solves your docker compose up issue

Vaibhav Kumar · Answer

Ok so my env will have the pip and airflow uid right

Vaibhav Kumar · Answer

And the original docker compose should be kept intact

Barak Amar · Answer

yes the env vars docker compose will use

Barak Amar · Answer

don t forget to run docker compose down before

Barak Amar · Answer

just verify that my what I wrote is true you modified the docker compose file right didn t set env with the above value

Barak Amar · Answer

The above docker compose doesn t include image build spec just using a pre built images so I assumed the above

Barak Amar · Answer

Unless you are trying to build a new image that includes lakefs package so please paste the command line you used to build it

Vaibhav Kumar · Answer

I have done the changes as per your suggestions it worked But now while running the lakefs dag <https github com treeverse airflow provider lakeFS blob main lakefs provider example dags lakefs dag py|example> I get below error

Yoni Augarten · Answer

< Vaibhav Kumar> this looks a lot like an issue that has already been resolved see Can you let me know which version of the lakeFS provider you are using

Yoni Augarten · Answer

< Vaibhav Kumar> I think I know what s going on here It seems like we forgot to fix this issue for the example file that you are using

Yoni Augarten · Answer

I ve opened a <https github com treeverse airflow provider lakeFS pull 39|PR >to try and handle that The team will test it during the week You are welcome to try it yourself and let me know if it worked

Vaibhav Kumar · Answer

Thanks meanwhile I will try to start my work by freezing it to 0 42 version hope that works

Yoni Augarten · Answer

Where are you getting the example DAG from

Yoni Augarten · Answer

Also please share errors in textual format rather than a photo

Vaibhav Kumar · Answer

`<https github com treeverse airflow provider lakeFS blob main lakefs provider example dags lakefs dag py>`

Vaibhav Kumar · Answer

Is that fine

Yoni Augarten · Answer

So you re using the script from the main branch with version 0 42 from the provider

Vaibhav Kumar · Answer

yes

Vaibhav Kumar · Answer

I know the example is not from 0 42 0 where can I find it

Yoni Augarten · Answer

I think it s better if you used the most up to date provider and modified the script according to the above PR

Amit Kesarwani · Answer

< Vaibhav Kumar> I have a Docker container with Airflow and Airflow demo notebook everything packaged in one container Also I have a webinar tomorrow on Airflow + lakeFS You can join

Vaibhav Kumar · Answer

Thanks for sharing this < Amit Kesarwani>

Amit Kesarwani · Answer

< Vaibhav Kumar> You can check my Dockerfile for any installation issues

Vaibhav Kumar · Answer

Hello I am not aware of the below syntax though I can understand the file is being copied in the below syntax here is the for any reference Can someone help `command AWS ACCESS KEY ID=$ env KEY AWS SECRET ACCESS KEY=$ env SECRET aws s3 cp endpoint url=` ` `

Vaibhav Kumar · Answer

< Ariel Shaqed Scolnicov > < Yoni Augarten> Can you guys please help on the above question

Niro · Answer

< Vaibhav Kumar> just to understand you correctly You are trying to understand what this step does or you have a different question

Vaibhav Kumar · Answer

First what this step does is my question

Niro · Answer

This step waits until lakeFS DAG completes successfully by polling on the SUCCESS file before continuing to the next steps

Vaibhav Kumar · Answer

Polling But I see a copy command Where is the DAG being called in this command

Niro · Answer

This is just a means to an end This command uses a github action called retry which tries the following copy command until it succeeds The SUCCESS file will be available only at the end of the lakeFS airflow when this file is merged to `main` branch Once this happens the copy command succeeds and we continue to the next step You can follow the code to understand the flow better