:bulb: lakeFS Pro Tip # 348,921: Branch Protection...
# devo
Oz Katz
07/20/2023, 12:04 PM
💡 lakeFS Pro Tip # 348,921: Branch Protection! 🔒
You can configure certain lakeFS branches (or branch patterns) to be protected: this means they can't be directly written to, you have to merge another branch into them.
🤔 Why is that useful? Because of lakeFS Hooks, of course!
🪝 Using pre-merge hooks you can run your own logic as a condition for the merge to happen. Think GitOps, for data!
This is really powerful because it allows:
⭐️ Performing schema validation prior to applying modifications to tables
⭐️ Running data quality checks - only high quality data on main!
⭐️ Ensuring consistency and referential integrity - since branches work across tables and datasets, you can make sure foreign key relations are consistent!
⭐️ Enforcing data contracts: ensure all values meet a specific, codified set of constraints
⭐️ Compliance - Branch protection makes it much easier to ensure sensitive data doesn't leak by running PII checks before data is exposed to consumers
And many more! See first comment for more interesting use cases and patterns enabled by branch protection
💡 1
lakefs 2
Oz Katz
07/20/2023, 12:04 PM
Read more about these patterns and how to implement them: https://docs.lakefs.io/use_cases/cicd_for_data.html
h
HT
07/20/2023, 9:17 PMis there any plan for supporting python for hooks ?
i
Iddo Avneri
07/20/2023, 9:54 PMHi @HT - You can use python with Web hooks today (you’ll need the proper infra of course).
What is your use case?
h
HT
07/21/2023, 1:42 AMI would like do some QC at pre-merge hook but we have no LUA dev in our team 😛
HT
07/21/2023, 1:55 AMthe QC need to check that the data that is going to be merge is correct: like if a file is present, an uuid is uniq, etc .... but may be it's wrong way to use hook ...
i
Iddo Avneri
07/21/2023, 1:56 AMTake a look at the Lua samples
Iddo Avneri
07/21/2023, 1:56 AMMight be very helpful
Iddo Avneri
07/21/2023, 1:58 AMIt’s a good way to use hooks :)
h
HT
07/21/2023, 1:59 AMThanks. I think I saw that before. I will give it another go
i
Iddo Avneri
07/21/2023, 1:59 AMGreat. Let me know if this was helpful.
h
HT
07/21/2023, 2:00 AMideally if we can do python, then that would make our life easier. Thus the initial question 😛
HT
07/21/2023, 2:00 AMbut I guess you do python behind the webhook
i
Iddo Avneri
07/21/2023, 2:00 AMGot it. Will bring it with our product
Iddo Avneri
07/21/2023, 2:01 AMRight - python can work with the webhooks
h
HT
07/21/2023, 2:01 AMbut then the server that running the QC need to access to the data that is beeing merged ... which I don't really understand how yet ...
HT
07/21/2023, 2:03 AMI am actually curious about: why LUA ??? the only other place that I came across lua is Wireshark ...
i
Iddo Avneri
07/21/2023, 2:04 AMOnce again, I’ll leave that to our product to answer :)
3 Views