• Oz Katz

    Oz Katz

    1 month ago
    This might be useful..? https://acorn.io/
  • Lynn Rozen

    Lynn Rozen

    4 weeks ago
    Hi, I would like to release a new version of lakeFS, any objections?
    Lynn Rozen
    Itai Admi
    5 replies
    Copy to Clipboard
  • Oz Katz

    Oz Katz

    4 weeks ago
    Thought i’d share a few references that might be highly relevant to lakeFS’ scalability: 1. https://devblogs.microsoft.com/devops/supercharging-the-git-commit-graph-iii-generations/#using-generation-number-in-merge-base-questions 2. Git's https://git-scm.com/docs/commit-graph-format 3. some more https://devblogs.microsoft.com/devops/supercharging-the-git-commit-graph-ii-file-format/ 4. logCommits for a path: https://devblogs.microsoft.com/devops/super-charging-the-git-commit-graph-iv-bloom-filters/ Obviously these are for longer term improvements that include additional indexing or data model tweaks, so putting here as reference
  • Barak Amar

    Barak Amar

    4 weeks ago
  • Ariel Shaqed (Scolnicov)

    Ariel Shaqed (Scolnicov)

    4 weeks ago
    Interesting but tough reading: Conflict resolution for structured merge via version space algebra. Adds some algebraic representations of merging. (AFAICT this is not the operational transform approach behind Google Docs, rather something like a description of Git's 3-way merge (which is also our model.)
    Ariel Shaqed (Scolnicov)
    1 replies
    Copy to Clipboard
  • s

    Slackbot

    4 weeks ago
    This message was deleted.
    s
    einat.orr
    +2
    4 replies
    Copy to Clipboard
  • n

    Niro

    3 weeks ago
    I'm about to release lakeFS with KV version, any objections?
  • n

    Niro

    3 weeks ago
    Hi all, A lot of code changes were merged over the weekend towards release of lakeFS with KV. I highly recommend you rebase your work over latest master and apply the changes. Please let us know if you encounter any issues.
  • Ariel Shaqed (Scolnicov)

    Ariel Shaqed (Scolnicov)

    3 weeks ago
    I have a small fix https://github.com/treeverse/lakeFS/pull/4097. It is related to https://github.com/treeverse/lakeFS/issues/4094, but it does not fix that issue. Think of it as a partial resolution. How should I mark it to get around the
    No linked issues found. Please add the corresponding issues in the pull request description.
    message we get on GitHub? Just open another issue? (@Tal Sofer, can you advise?)
    Ariel Shaqed (Scolnicov)
    Tal Sofer
    4 replies
    Copy to Clipboard
  • Ariel Shaqed (Scolnicov)

    Ariel Shaqed (Scolnicov)

    3 weeks ago
    Does anyone know of an ASCII representation for timestamps, similar to ISO8601, but that sorts in reverse order? Long form: @Yoni Augarten and I are working on improving usability of run_id in GC. Currently in order to discover which run_id to provide to a run, you pretty much need a KV store to keep the last few run_ids by time. That's an ops problem. As a first step we want to change the format of run_ids to be sort -- so instead of
    <UUID>
    we will use something like
    <TS_ISO_8601>-<RANDOM>
    . Now the files
    .../retention/gc/commits/run_id.../commits.csv
    will list in time-sorted order. So no need for KV, we can simply list these objects in S3. Unfortunately, it is still hard to go backwards in such a list. And that's actually the preferred mode: "find the last run_id from at least 2 weeks ago". So... I'd really like to name them
    <REVERSE_8601_ISH>-<TS_ISO_8601>-<RANDOM
    , where RANDOM_8601_ISH is an ISO 8601-like printable representation of timestamp that sorts in reverse order, and we include TS_ISO_8601 merely for readability. Now we just need an encoding 🙂 Where I've gotten so far: use a 9s-complement ASCII representation. An even less-readable representation would use e.g.
    10000000000 - EPOCH
    (right now would be
    8337539453
    (== 10000000000-1662460547), and 5 seconds later it would be
    8337539448
    . A more-readable representation might use
    <10000-YYYY>-<100-mm>-<100-DD>N<100-HH>:<100-MM>:<100-SS>
    , and let you decode in your head with a little effort. Is there any more standard way to do this?
    Ariel Shaqed (Scolnicov)
    Barak Amar
    6 replies
    Copy to Clipboard