• Yoni Augarten

    Yoni Augarten

    9 months ago
    Hey, I've been experimenting with the lakeFS metadata client - and managed to use Databricks to create a diff between two commits! I want to publish the notebook as an example to the project. However all examples in the project are Scala files (see
    examples/
    directory) and my notebook is exported in HTML. Where in the project can/should I put this HTML?
    Yoni Augarten
    Adi Polak
    5 replies
    Copy to Clipboard
  • Ori Adijes

    Ori Adijes

    9 months ago
    Hello all, Is there an option for bulk delete of object paths through the lakeFS API? At the moment we use the delete object api one by one (async) in order to delete many objects. I think performance wise, it will be much better to do this parallelism server side instead of issuing many http requests in the client side (We can have more than 1000 files requests for deletion)
    Ori Adijes
    Barak Amar
    +2
    24 replies
    Copy to Clipboard
  • Ariel Shaqed (Scolnicov)

    Ariel Shaqed (Scolnicov)

    9 months ago
    I want us to do a quick patch fix for part of https://github.com/treeverse/lakeFS/issues/2773 tomorrow (we have 2 users stuck on this). The quick fix is to remove username format validation entirely or almost entirely. I believe our code is robust against injection attacks; we can and will verify that usernames are never involved in constructing sql queries, but only ever passed as parameters. Does anyone know of a good reason to validate username formats on usage? (We might validate a format during internal user registration, but that would be for business reasons...) Thanks!
    Ariel Shaqed (Scolnicov)
    Oz Katz
    +1
    3 replies
    Copy to Clipboard
  • Ariel Shaqed (Scolnicov)

    Ariel Shaqed (Scolnicov)

    9 months ago
    No immediate action from us. But Spark users will probably be vulnerable, regardless of whether or not they use lakeFS clients (depending on how much 3rd parties can control what they log). Operators should set the relevant property to avoid loading executable code from user-supplied urls. But this is a nasty bug, and one that was completely avoidable. https://www.theregister.com/2021/12/10/log4j_remote_code_execution_vuln_patch_issued/ I reckon once an actual official patch is available, we can bump versions in our Java clients.
    Ariel Shaqed (Scolnicov)
    2 replies
    Copy to Clipboard
  • Ariel Shaqed (Scolnicov)

    Ariel Shaqed (Scolnicov)

    9 months ago
    I've just deleted branch
    to-be-deleted-main
    from the lakeFS repository, which used to be called "main". Nobody's ever used it, and it contained some filename case issues. The branch tip was
    8ff9846ab6e9aacdea56606ff9dde135c42e4f83
    . I am purposefully not generating a tag there. To be clear: the lakeFS trunk is named "master". WHY? This branch was an aborted attempt to rename
    master
    (the lakeFS repo was opened back when this horribly incorrect name was the default). This particular renaming was broken because of some uppercase/lowercase renamings. And those cause issues on a case-insensitive filesystem -- which MacOS helpfully provides by default. So it used to be that after
    git checkout main
    on a macOS box you'd need to seek expert help to recover your local repo on that box... WHY does trunk remain
    master
    ?
    It is an incorrect name with horrible connotations (and not one used historically). However renaming trunk requires a simultaneous (not concurrent!) change: all developers must rename
    master
    ->
    main
    on the same commit, all actions must change to accept the new name, and other unknown dependencies will break. (I would appreciate pointers to articles from any open-source projects about good practices for doing and testing this!)
    Ariel Shaqed (Scolnicov)
    1 replies
    Copy to Clipboard
  • Ariel Shaqed (Scolnicov)

    Ariel Shaqed (Scolnicov)

    9 months ago
    I'm getting
    java.lang.NoSuchMethodException: com.databricks.spark.metrics.FileSystemWithMetrics.getAmazonS3Client()
    when writing using HadoopFS (
    io.lakefs:hadoop-lakefs-assembly:0.1.4
    ). Has anyone seen this?
    Ariel Shaqed (Scolnicov)
    Barak Amar
    5 replies
    Copy to Clipboard
  • Oz Katz

    Oz Katz

    9 months ago
    could be useful for the lakeFS monorepo - supports running steps only if certain paths changed:https://github.com/dorny/paths-filter
    Oz Katz
    Barak Amar
    3 replies
    Copy to Clipboard
  • Tal Sofer

    Tal Sofer

    9 months ago
    Is there a way to view Exclidraw charts that are part of github PRs?
    Tal Sofer
    Barak Amar
    3 replies
    Copy to Clipboard
  • Itai Admi

    Itai Admi

    8 months ago
    Is anyone familiar with speedb ? They claim to have developed a 10X rockdb compatible replacement.
    Itai Admi
    Oz Katz
    4 replies
    Copy to Clipboard
  • Ariel Shaqed (Scolnicov)

    Ariel Shaqed (Scolnicov)

    8 months ago
    https://github.com/github/roadmap/issues/372 We should try some of these mermaid diagrams. 🧜🏼 (Says the guy whose name Disney stole for The Little Mermaid...)
    Ariel Shaqed (Scolnicov)
    1 replies
    Copy to Clipboard