Is there an ability or interest in implementing a way to add custom headers and post commit/merge actions? We were exploring how to automate a kubeflow pipeline on commits to master and would be helpful to pass auth in the headers so that the jobs are triggered only from LakeFS

Hi @Thomas Vander Wal! • We haven’t heard users asking for post commit/merge hooks, what is your use case? we will be happy to explore it! • Regarding custom headers, if I understand correctly you want to use it to integrate Kubeflow and lakeFS? can you please elaborate on what you mean by “jobs are triggered from lakeFS” in this context? Following our chat in https://lakefs.slack.com/archives/C017S6YFFSP/p1624573779028400?thread_ts=1624545254.024800&cid=C017S6YFFSP, let me know if you still need custom headers while using

ContianerOp

.

@Tal Sofer for the first part of that yea, after a successful merge it would be nice to kick off some downstream processing like a KF pipeline in that example. As far as the headers that would allow us to pass along a kibernetes secret or some type of auth that we could verify a pipeline can be ran. Otherwise anyone could just post to our web hook endpoint and simulate a successful merge

hey @Thomas Vander Wal 🙂 you could pass a token as part of the

query_params

dictionary as described here: https://docs.lakefs.io/setup/hooks.html#action-file-webhook-properties of course, if you’re passing secrets, i’d advise to make sure the passed url is using

https

Thanks @Oz Katz ! That’s definitely an option, we could use that for our prototyping but I’m not sure that would pass our security audits for prod work. Mainly because even though the network level is encrypted the URL could be stored in server logs and referrer header leaks

Hmm that’s a good point, and justifies another GitHub issue to add custom header support 🙂 I believe we can address both requests in upcoming versions of lakeFS, probably in 2-4 weeks. Would you be able to open the relevant issues?

That’s awesome! Yea I can open them up later today.

Great! thanks. we’ll make sure to keep them updated as we progress