Hi fellow devs,
This is about making the lakeFS me...
# deva
Ariel Shaqed (Scolnicov)
07/27/2021, 12:45 PMHi fellow devs,
This is about making the lakeFS metadata client work on DataBricks (https://github.com/treeverse/lakeFS/issues/1847), and will likely make less sense if you're unfamiliar with the bug.
My work on loading a second copy of org.rocksdb.rocksdbjni has stalled. While there is a path forward, it is both more expensive than anticipated and so far seems very unlikely to give us the desired advantage of an easy upgrade path for the rocksdbjni package.
So instead I've decided with @Oz Katz to change course on this and implement a (very) minimal RocksDB reader on the JVM in Scala, with no JNI needed. I'm looking for someone to join me in writing this; please let me know!
👍🏻 1
✋ 1
👍 1
Ariel Shaqed (Scolnicov)
07/27/2021, 2:47 PMGiven that we're going to start without even a binary search (client doesn't need
seekGEsbinaryAriel Shaqed (Scolnicov)
07/27/2021, 2:47 PM(Also it's a really simple library, you just tell it to parse one thing and then sequence many things to parse them in order...)
Ariel Shaqed (Scolnicov)
07/28/2021, 7:55 AMWhat we're planning to try now (as well as ramblings why we tried something else first): https://github.com/treeverse/lakeFS/issues/1847#issuecomment-871944539, look for "Fine, I'll write my own SSTable reader!".
@Itai Admi
Ariel Shaqed (Scolnicov)
07/28/2021, 1:14 PM"Documentation" for the structure of index blocks is at https://github.com/facebook/rocksdb/blob/74b7c0d24997e12482105c09b47c7223e7b75b96/table/block_based/block_builder.cc#L10-L32.
They have prefix-compressed keys, so a bit like Gills if you were here at Treeverse at the time. But unlike Gills it cannot be memory-mapped, it has to be decoded. SO not quite as nice.
4 Views