Is there a way for Lakefs to emit events every time an object is uploaded? I see hooks, but it's at a less frequent granularity. I'm looking for something like S3 Event Notifications.

Hi Sid, Unfortunately we don't currently have such a feature. Would you like to open an issue for something like this, maybe with a sketch of a use case? Originally (AFAIR, from listening to the "generate thumbnail" example at some talks) the S3 feature was intended to postprocess S3 when used for serverless uploads. Out of curiosity (for now at least...), how would you expect notifications to be delivered? There are so many mechanisms, I think aws offer 3 different ways to deliver.

Yes, we're using S3 notifications already. We're also seeing if we can handle data reformating within lakefs - so if someone uploads a CSV we can automatically convert it to parquet and register a dataset in hive or dremio. The notification could just be sent to an HTTP endpoint, in our case we can set up a URL-triggered Lambda function that can be kicked off.

Wow, that would make sense. Would you like me to open the issue, or do you want to do it yourself? (Lambdas are cool, but it might be hard to implement reliable delivery on top of them...)

I can open up a ticket, thanks

Thanks!