Hi all. Looks like a great product you are building ❤️. Out of curiosity: • I’m interested in running tekton pipelines/tasks in CI pre-merge https://tekton.dev/ has anyone tried this approach? • Anyone here have any experience using https://docs.feast.dev/ together with lakeFS? • Successful, dex Integration?

Hi @Leonard Aukea, welcome! happy to have you on-board.

@Tal Sofer so dex is a mighty popular OpenID Connect identity service https://github.com/dexidp/dex . We use it in our clusters to allow authentication of users via our AD. Dex can then provide signed JWT tokens. https://dexidp.io/docs/using-dex/. So what would be nice is if LakeFS can consume these tokens so that we may have a single point of sign on.

@Leonard Aukea to answer your questions in more details:

I’m interested in running tekton pipelines/tasks in CI pre-merge https://tekton.dev/ has anyone tried this approach?

I would like to confirm that I understand your use-case. Do you want to (1) run a Tekton pipeline during lakefs’s pre-merge, or you want (2) the lakefs merge operation to be the final stage of your pipeline? Option 1 can be done with lakefs webhooks, example usage, and option 2 using our cli/openAPI/operator (such as lakefs airflow operator).

Anyone here have any experience using https://docs.feast.dev/ together with lakeFS?

We’ve looked into feast, and saw that their registry is object storage based. This is the component that we may be able to integrate with, but we need to check if feast communicates with the object store via s3 api which lakefs supports. Is this what you had in mind? What are you hoping to achieve from integrating lakefs with feast? As for dex, unfortunately we don’t support that atm, lakefs does not have any external authentication. You are welcome to open a feature request in github, describing how you envision it will be integrated with the current lakefs app.

@Leonard Aukea looking forward to your answers on the above so we can further check the compatibility with the tools you require.

Dex integration is maybe on the top thing we care about and is a bit of a blocker for full adoption. • wrt. to feast integration. It is rather unclear at this point. But I would like to be able to manage feature versioning with LakeFS somehow. • For tekton, I guess we would just set up an eventlistener that would consume http requests from lakeFS (webhook) and trigger tekton tasks and pipelines. • Also, more interesting to us is Kubeflow pipelines integration (argo workflows, tekton) in comparison with airflow, which we have deprecated from our stack.

There's a great python sdk. Check out https://docs.lakefs.io/integrations/python.html

Thanks @Leonard Aukea! • I’m going to open a github issue to track your request to support dex. • As for Feast, let me investigate further and update you with my findings early next week. I’m new to Feast and would like to get a deeper understanding of it to find the right way to integrate with lakefs. • The same holds for kubeflow pipelines, I will investigate the potenial integration with lakefs and will be in touch with you early next week.