I'm trying to get https://github.com/treeverse/lakeFS-samples/tree/main/06-airflow-dag-example to work. However, the

commit

is failing with

AirflowException("access_key_id must be specified in the lakeFS connection details")

I can see the connection defined as an env var in the Docker Compose, but it wasn't listed in the Airflow UI. I went ahead and created it and verified it from the CLI

airflow@f3ee9d2b2072:/opt/airflow$ airflow connections get lakefs
/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py:386: FutureWarning: The auth_backends setting in [api] has had airflow.api.auth.backend.session added in the running config, which is needed by the UI. Please update your config before Apache Airflow 3.0.
  FutureWarning,
     |         |           |             |                   |        |       |          |      |              | is_extra_encrypte |                   |
id   | conn_id | conn_type | description | host              | schema | login | password | port | is_encrypted | d                 | extra_dejson      | get_uri
=====+=========+===========+=============+===================+========+=======+==========+======+==============+===================+===================+===================
None | lakefs  | HTTP      | None        | <http://lakefs:800> | None   | None  | None     | None | None         | None              | {'access_key_id': | <http://http>%3A%2F%
     |         |           |             | 0/api/v1          |        |       |          |      |              |                   | 'AKIAIOSFODNN7EXA | 2Flakefs%3A8000%2F
     |         |           |             |                   |        |       |          |      |              |                   | MPLE',            | api%2Fv1/?access_k
     |         |           |             |                   |        |       |          |      |              |                   | 'secret_access_ke | ey_id=AKIAIOSFODNN
     |         |           |             |                   |        |       |          |      |              |                   | y':               | 7EXAMPLE&secret_ac
     |         |           |             |                   |        |       |          |      |              |                   | 'wJalrXUtnFEMI/K7 | cess_key=wJalrXUtn
     |         |           |             |                   |        |       |          |      |              |                   | MDENG/bPxRfiCYEXA | FEMI%2FK7MDENG%2Fb
     |         |           |             |                   |        |       |          |      |              |                   | MPLEKEY'}         | PxRfiCYEXAMPLEKEY

But the

commit

still fails with the same error above. I'm new to Airflow so any pointers on how to get this working much appreciated 🙂

I hope you don’t find this answer silly 🙂 I find that sometimes working with these example on my local computer with limited resources I simply need to restart the kernel and rerun. Can you try that? (sorry for the “did you try to restart it” answer 🙂 )

Try host without “api/v1”: http://lakefs:8000

thanks both. I found the cause and filed a fix for it here: https://github.com/treeverse/lakeFS-samples/pull/71