• e

    Elad Lachmi

    6 days ago
    Hi again, I'm working on a new File Viewer for the lakeFS webui This is the issue (related PR mentioned), in case anyone would like to chime in on the issue and/or PRhttps://github.com/treeverse/lakeFS/issues/4225
  • Ariel Shaqed (Scolnicov)

    Ariel Shaqed (Scolnicov)

    6 days ago
    To recap: I'm seeing reasonable speeds for writing 4000 partitions from Spark, but as soon as I switch SaveMode over to overwrite things are slow again. Can we count DeleteObjects operations and their paths, to see how overwrite deletes the old data?
  • Barak Amar

    Barak Amar

    6 days ago
    Merged by mistake a failed integration test into master branch - working on it.
    Barak Amar
    1 replies
    Copy to Clipboard
  • Ariel Shaqed (Scolnicov)

    Ariel Shaqed (Scolnicov)

    6 days ago
    @Barak Amar let's continue our lakeFSFS recursive deletes discussion here rather than on DMs.
    Ariel Shaqed (Scolnicov)
    Barak Amar
    14 replies
    Copy to Clipboard
  • Barak Amar

    Barak Amar

    5 days ago
    Hi, planning to release lakeFS - let me know if there is something to wait for or any issues you see.
  • Ariel Shaqed (Scolnicov)

    Ariel Shaqed (Scolnicov)

    5 days ago
    Starting to release lakeFSFS -- let me know if I should wait.
  • Ariel Shaqed (Scolnicov)

    Ariel Shaqed (Scolnicov)

    5 days ago
    Hi devs, Something I've been thinking about. We see Spark running a very large number of "statObject" calls and "listObject" - and I would like to speed those up. (This is not an immediate suggestion, or even a suggestion, I'm just trying to wrap my head around things and understand the feasible solution space!) Now almost all of these calls seem very racy: delete a file, see if its "directory" is empty; create a file, see if we want to give it a directory marker - stuff like that. The point about these calls is that the calling code does not expect a consistent answer! Supposewe could identify on the Spark side these calls (because some calls really need consistency - it's just that others don't...). Then we could ask lakeFS for an inconsistent ("eventually" consistent) answer! Would it be possible? How much faster would it be, say on DynamoDB and on PostgreSQL?
    Ariel Shaqed (Scolnicov)
    Itai Admi
    +1
    4 replies
    Copy to Clipboard
  • t

    taylor schneider

    2 weeks ago
    can anyone tell me what is in the _lakefs directory with a repo? I am guessing it's rocksdb related?
    t
    Oz Katz
    +3
    26 replies
    Copy to Clipboard
  • v

    Vaibhav Kumar

    1 week ago
    Hi Team I am working on this issue https://github.com/treeverse/lakeFS/issues/2344 As far as I know I have to change the the README under the python package https://github.com/treeverse/lakeFS/tree/master/clients/python . Not sure why the changes doesn't reflect in pypy of lakefs already. Or if it is not how it was supposed to be done please guide me where to add the changes?
    v
    Guy Hardonag
    +1
    58 replies
    Copy to Clipboard
  • e

    Elad Lachmi

    1 hour ago
    Hi, I've created the first PR for the TypeScript conversion This is step 1 - tooling The idea is that breaking down this effort into multiple PRs will make it much easier to review and provide feedback vs. one big changesetTooling PR is awaiting feedback and review