• Ariel Shaqed (Scolnicov)

    Ariel Shaqed (Scolnicov)

    5 days ago
    Hi devs, Something I've been thinking about. We see Spark running a very large number of "statObject" calls and "listObject" - and I would like to speed those up. (This is not an immediate suggestion, or even a suggestion, I'm just trying to wrap my head around things and understand the feasible solution space!) Now almost all of these calls seem very racy: delete a file, see if its "directory" is empty; create a file, see if we want to give it a directory marker - stuff like that. The point about these calls is that the calling code does not expect a consistent answer! Supposewe could identify on the Spark side these calls (because some calls really need consistency - it's just that others don't...). Then we could ask lakeFS for an inconsistent ("eventually" consistent) answer! Would it be possible? How much faster would it be, say on DynamoDB and on PostgreSQL?
    Ariel Shaqed (Scolnicov)
    Itai Admi
    4 replies
    Copy to Clipboard
  • t

    taylor schneider

    2 weeks ago
    can anyone tell me what is in the _lakefs directory with a repo? I am guessing it's rocksdb related?
    Oz Katz
    26 replies
    Copy to Clipboard
  • e

    Elad Lachmi

    3 hours ago
    Hi, I've created the first PR for the TypeScript conversion This is step 1 - tooling The idea is that breaking down this effort into multiple PRs will make it much easier to review and provide feedback vs. one big changesetTooling PR is awaiting feedback and review
  • v

    Vaibhav Kumar

    1 week ago
    Hi Team I am working on this issue https://github.com/treeverse/lakeFS/issues/2344 As far as I know I have to change the the README under the python package https://github.com/treeverse/lakeFS/tree/master/clients/python . Not sure why the changes doesn't reflect in pypy of lakefs already. Or if it is not how it was supposed to be done please guide me where to add the changes?
    Guy Hardonag
    62 replies
    Copy to Clipboard