Not sure if this has any benefit (or implications)...
# dev
o
Not sure if this has any benefit (or implications) for the lakeFS metadata client, but as of Spark 3.4 which was recently released, there's an official Protobuf Data Source
a
Uggh: not beneficial, IIRC Graveler RocksDB tables do not have protobuf values. Instead they use some varint encoding, and hold a buffer of bytes in the value. That ends up as a protobuf, but I don't think it is exposed as an actual Spark column before we finish decoding. But a great opportunity to see whether our shading rules are good enough to separate "their" protobuf from "our" protobuf. We shall see once we start supporting 3.4.
🙏 1
😄 1