• Ariel Shaqed (Scolnicov)

    Ariel Shaqed (Scolnicov)

    1 year ago
    Working on upgrade script for https://github.com/treeverse/lakeFS/pull/2430 - *Add "attach storage namespace [to repository]" IAM action type. There's a split between possibilities here, complexity of SQL balanced against code complexity for a script that should run only once. I would like the script to "add[...] a new rule allowing action
    fs:AttachRepositoryNamespace
    on
    *
    for any policy that allows
    fs:CreateRepository
    ". Three possibilities that I can see: • Golang all the way! Pros: probably more readable, easier to get right. uses a good programming language. Cons: (very) poor ops experience: not part of the migration library that we use (or, indeed, any reasonable migration library). • PostgreSQL
    jsonb
    SQL whackiness. Write 100% safe pure-SQL code that updates the table. Pros: Perfect ops experience. Cons: The function itself will be tricky to write - it needs to handle IAM-style wildcards (
    *
    ,
    ?
    ) entirely correctly. • PostgreSQL
    jsonb
    SQL whackiness but less whacky. Write SQL code that works on every reasonable IAM policy, but fails if the policy uses special (unused!) characters in action names. Fail migration if such a policy is found (unlikely; such a policy includes an action that is unused!). Specifically, fail for actions that include SQL-style wildcards (
    _
    .
    %
    ); there are no such actions in lakeFS (or anywhere in IAM, really). Unless I hear strenuous objections -- hopefully accompanied by suggestions how to do it better -- I am going with the third option. (Also posting to the PR, of course.)
    Ariel Shaqed (Scolnicov)
    Itai Admi
    2 replies
    Copy to Clipboard
  • datadavd

    datadavd

    1 year ago
    Hey friends, I was wondering if someone can help point me in the right direction as to where this would necessarily be implemented. https://github.com/treeverse/lakeFS/issues/2415#issuecomment-923728336
    datadavd
    Itai Admi
    2 replies
    Copy to Clipboard
  • Ariel Shaqed (Scolnicov)

    Ariel Shaqed (Scolnicov)

    1 year ago
    TIL about GopherJS. (And I'm pleased, amused, and terrified... concurrently...).
    Ariel Shaqed (Scolnicov)
    Oz Katz
    2 replies
    Copy to Clipboard
  • Ariel Shaqed (Scolnicov)

    Ariel Shaqed (Scolnicov)

    1 year ago
    About https://github.com/treeverse/lakeFS/issues/2491: while I disagree with the decision that we made, I am committed to it. So I am trying to use this issue in order to get some explicit arguments for our current design onto the record, which is probably https://github.com/treeverse/lakeFS/pull/2369 or the repo settings PR. My ideal resolution for this issue is to close it as "invalid". AFAIR @Itai Admi and I supported the kind of change that appears in #2491 in the F2F and written design reviews of protected branches. The general feeling in the room was against re-using existing mechanisms and instead creating a new mechanism, IIUC for simplicity of operation by users. However it is very likely I do not understand correctly. Most of all I want to maintain commitment to the agreed design. I agree that this is hard given the current state as documented, and I would be grateful if the people who best understand why we need a new mechanism would be willing to take the time to spell it out. But I do believe that we should continue the discussion if necessarily rather than re-open it, separately and with no context.
    Ariel Shaqed (Scolnicov)
    Itai Admi
    3 replies
    Copy to Clipboard
  • Itai Admi

    Itai Admi

    11 months ago
    Waiting for @Barak Amar to complete his PR and releasing a version, let me know if there are any other changes you want me to wait for.
    Itai Admi
    1 replies
    Copy to Clipboard
  • datadavd

    datadavd

    11 months ago
    Any devs familiar with React? Trying to determine how to place paragraph level text under a React Bootstrap Modal.Header’s Modal.Title. My react/js skills are very week so I’m struggling with this. All my tests place the text to the right of the Modal.Header’s Modal.Title Also, I’m trying to similarly center paragraph level text right under component (closer to the component than its border below it; hopefully that makes sense). Right now I’ve got it looking good but the paragraph text is more “attached/closer” to the border than the component above (which is what I want). For reference, I’m trying to match the mocks @Yoni Augarten (I believe) provided me here for Create a Repository and Admin pages. I’m kinda struggling getting it perfect tbh 😿 https://github.com/treeverse/lakeFS/issues/2316#issuecomment-939459473
    datadavd
    Yoni Augarten
    2 replies
    Copy to Clipboard
  • Ariel Shaqed (Scolnicov)

    Ariel Shaqed (Scolnicov)

    11 months ago
    Trying to test multipart upload to our gateway using Spark as part of Nessie. I'm having a lot of trouble making Spark run multipart uploads to our S3 gateway. It's tough because we need big files and our containers are not so big. I want to be lazy, and run a program that exercises the S3a hadoopfs directly rather than going through Spark. Now I can just easily write a 20MiB file with 5MiB parts, read it back, and be done with it.Pros: Short, nonbrittle (small changes don't make the test silently stop testing what it should) and easy to write.Cons: Doesn't actually test Spark performing multipart uploads to the lakeFS S3 gateway, but merely the S3a hadoopfs. There might be differences between some Spark writer, due to arbitrary changes that it might make to S3a config or similar. I believe it still makes sense: the cons that we overcome are relatively minor; most would occur regardless unless we managed to hit the exact combination that triggers the hypothetical failure. (E.g. something like https://github.com/treeverse/lakeFS/issues/2429). WDYT?
    Ariel Shaqed (Scolnicov)
    Itai Admi
    3 replies
    Copy to Clipboard
  • Itai Admi

    Itai Admi

    11 months ago
    I think we all agreed that the current way we handle the changelog is a mess (more details on the current process below). My suggestion is simple, revert it to the way it was before. The releaser looks at the completed PRs and gathers a list of meaningful changes. Yes, it means that the responsibility returns to the releaser. It also means that our commit messages need to describe the change better, which I think we’ve improved at. In case of ambiguity, we can always ask for more details and/or edit the release notes later. Current Process Author and reviewer are responsible for adding a line when required to the changelog. What normally happens is that they forget, only to open an additional PR to add that line when/if they remember later. When releasing the changelog becomes the release notes as is without further actions needed from the releaser.
    Itai Admi
    Guy Hardonag
    +4
    24 replies
    Copy to Clipboard
  • Oz Katz

    Oz Katz

    11 months ago
    FYI - Apache Spark (i.e. the OSS, not the Databricks distribution) added a RocksDB StateStore in Spark 3.2.0 (recently released). This means new Spark versions now add a dependency on RocksDB - https://issues.apache.org/jira/browse/SPARK-34198 The version seems to be pinned to 6.20.3 - as we're currently bundling our our sstable parser, not sure if/how this affects us. Perhaps @Tal Sofer or @Ariel Shaqed (Scolnicov) can shed some light?
    Oz Katz
    Ariel Shaqed (Scolnicov)
    5 replies
    Copy to Clipboard
  • Itai Admi

    Itai Admi

    11 months ago
    Planning on releasing a minor version soon, any PRs worth waiting for?
    Itai Admi
    Paul Singman
    +1
    6 replies
    Copy to Clipboard