Is there an ability or interest in implementing a ...
# help
t
Is there an ability or interest in implementing a way to add custom headers and post commit/merge actions? We were exploring how to automate a kubeflow pipeline on commits to master and would be helpful to pass auth in the headers so that the jobs are triggered only from LakeFS
t
Hi @Thomas Vander Wal! • We haven’t heard users asking for post commit/merge hooks, what is your use case? we will be happy to explore it! • Regarding custom headers, if I understand correctly you want to use it to integrate Kubeflow and lakeFS? can you please elaborate on what you mean by “jobs are triggered from lakeFS” in this context? Following our chat in https://lakefs.slack.com/archives/C017S6YFFSP/p1624573779028400?thread_ts=1624545254.024800&cid=C017S6YFFSP, let me know if you still need custom headers while using
ContianerOp
.
t
@Tal Sofer for the first part of that yea, after a successful merge it would be nice to kick off some downstream processing like a KF pipeline in that example. As far as the headers that would allow us to pass along a kibernetes secret or some type of auth that we could verify a pipeline can be ran. Otherwise anyone could just post to our web hook endpoint and simulate a successful merge
I’m envisioning we could pass along a service account token in the request somehow
o
hey @Thomas Vander Wal 🙂 you could pass a token as part of the
query_params
dictionary as described here: https://docs.lakefs.io/setup/hooks.html#action-file-webhook-properties of course, if you’re passing secrets, i’d advise to make sure the passed url is using
https
regarding post-commit and post-merge hooks: expanding on what @Tal Sofer wrote, sounds like there’s a good use case here. Would you be able to open a GitHub issue for it?
t
Thanks @Oz Katz ! That’s definitely an option, we could use that for our prototyping but I’m not sure that would pass our security audits for prod work. Mainly because even though the network level is encrypted the URL could be stored in server logs and referrer header leaks
o
Hmm that’s a good point, and justifies another GitHub issue to add custom header support 🙂 I believe we can address both requests in upcoming versions of lakeFS, probably in 2-4 weeks. Would you be able to open the relevant issues?
👍🏾 1
t
That’s awesome! Yea I can open them up later today.
o
Great! thanks. we’ll make sure to keep them updated as we progress