• Ariel Shaqed (Scolnicov)

    Ariel Shaqed (Scolnicov)

    8 months ago
    TIL Kubernetes lets me use JSON Patch to edit objects. For instance, I can add a backend service to an ingress named "my-ingress" by saying
    kubectl patch ingress myingress --type json --patch-file=/tmp/patch.json
    where
    /tmp/patch.json
    is:
    [
      {
        "op": "add",
        "path": "/spec/rules/-",
        "value": {
          "host": "<http://another.example.com|another.example.com>",
          "http": {
    	"paths": [
              {
              "backend": {
                "service": {
                  "name": "another-svc",
                  "port": {
                    "number": 5678
                  }
                }
              },
              "path": "/another",
    	  "pathType": "Prefix"
            }]
          }
        }
      }
    ]
    A (more) readable introduction to JSON Patch is http://jsonpatch.com/; there's also https://datatracker.ietf.org/doc/html/rfc6902 of course.
    Ariel Shaqed (Scolnicov)
    Oz Katz
    2 replies
    Copy to Clipboard
  • Guy Hardonag

    Guy Hardonag

    8 months ago
    Hey, we got a feature request from one of our users requesting to add the option to set the
    time
    of the commit. git does support it and I believe we should to, Checking hear to see if anyone has any objections
    Guy Hardonag
    Itai Admi
    +1
    4 replies
    Copy to Clipboard
  • Ariel Shaqed (Scolnicov)

    Ariel Shaqed (Scolnicov)

    8 months ago
    @Guy Hardonag after we talked yesterday, I came up with this neat trick for transforming any iterator into a prefetching iterator! The basic trick is for the wrapping iterator to start a goroutine that reads values from the base iterator and writes them to a channel of size 10K (for instance). Now to advance and read a value, the wrapping iterator simply reads from the channel. If goroutine gets too far ahead, it fills up the output channel... and blocks. So it cannot move too far, but if it goes to read file or network it still lets the wrapping iterator continue for a bit. How to handle
    NextRange
    (and even
    SeekGE
    )? Basically, have another channel from the wrapping iterator to the base iterator. Every time the wrapping iterator needs to change location, it drops the old channel (@Barak Amar do we need to drain a channel, or can it just be garbage-collected while full?), creates a new channel in its stead, and sends a command
    <"NextRange", ptrToNewChannel>
    to the base iterator. The base iterator now
    select
    s on being able to write on its output or read from its input; if it gets a command on the input channel it can implement it. Not perfect, but a cheap way to prefetch.
    Ariel Shaqed (Scolnicov)
    Barak Amar
    6 replies
    Copy to Clipboard
  • Ariel Shaqed (Scolnicov)

    Ariel Shaqed (Scolnicov)

    8 months ago
    Reversim Summit 2021 conference videos are all out -- tech content in Hebrew (sorry all non-speakers...). https://www.youtube.com/playlist?list=PLqXy0aX6TzQryGoAdbyPevKocQxMJzg8_. Includes some content by familiar faces from lakeFS users, as well as my very own

    https://youtu.be/ZeE-JxMZDLk

    . 🙂
    Ariel Shaqed (Scolnicov)
    2 replies
    Copy to Clipboard
  • Tal Sofer

    Tal Sofer

    7 months ago
    I need to use lfs golang sdk from my code, what go package I should get? I only see this github.com/dollarkillerx/lakefs-sdk and it is not published by the lakefs team (this is very cool!). Is there a reason why we don’t publish this? cc @Barak Amar
    Tal Sofer
    Barak Amar
    2 replies
    Copy to Clipboard
  • mishraprafful

    mishraprafful

    7 months ago
    Hey, I just created a PR for a good first issue, but seems like I need approvals from maintainers before the CI checks run. Could you please approve the same. Thanks Ref: https://github.com/treeverse/lakeFS/pull/2900
    mishraprafful
    Barak Amar
    +1
    5 replies
    Copy to Clipboard
  • mishraprafful

    mishraprafful

    7 months ago
    Hey, I’m just a bit curious, I get this message every-time I create a PR.
    First-time contributors need a maintainer to approve running workflows. Learn more.
    but I am not a first time contributor, is it that every PR needs to be approved by a maintainer for running the workflows. Ref: https://github.com/treeverse/lakeFS/pull/2902
    mishraprafful
    Barak Amar
    3 replies
    Copy to Clipboard
  • Adi Polak

    Adi Polak

    7 months ago
    hi devs, installing statik with
    go get <http://github.com/rakyll/statik|github.com/rakyll/statik>
    returned the following message:
    go get: installing executables with 'go get' in module mode is deprecated.
    	To adjust and download dependencies of the current module, use 'go get -d'.
    	To install using requirements of the current module, use 'go install'.
    	To install ignoring the current module, use 'go install' with a version,
    	like 'go install <http://example.com/cmd@latest|example.com/cmd@latest>'.
    	For more information, see <https://golang.org/doc/go-get-install-deprecation>
    	or run 'go help get' or 'go help install'.
    which one is should I use to install statik?
    go get -d
    or
    go install
    ?
    Adi Polak
    Barak Amar
    6 replies
    Copy to Clipboard
  • Itai David

    Itai David

    7 months ago
    Hi all. I'm attaching a technical discussion that we were (mistakenly) having in an internal thread. It concerns an issue regarding the way we handle Windows paths. TL;DR - we want to change backslashes (
    \
    ) in Windows' style paths to Unix's style forward slashes (
    /
    ) to match S3 behavior. Out main concern is the effect it might have on users currently using Windows' style paths and the effect it might have on their data. The original message is attached hereafter. I'm attaching all responses as comments. All further comments are very appreciated. Thanks Hi. I would love to here some opinions and some your of concerns regarding an issue I'm working on - BUG : Upload directory path not parsed correctly #2880 TL;DR - we are not treating Windows' path delimiter - the backslash - as such Upon uploading a file, from Windows, the original path, e.g. 
    C:\some\path\to\file
     is kept and used as a key. Later, when the file is returned to the client(s), as a response for ListObjects, we do not parse the key as a path correctly. While 
    lakectl
     shows the key as a full path either ways, the web UI fails to create the nice (and expected) tree display, as described in the bug On @Tal Sofer's advice, I looked at S3 behaviour, and found out that AWS actually converts Windows` style paths to Unix style. Namely, all backslashes are turned into forward slashes. That's it for the problem description, now for possible solutions:Option 1 - change lakefs behaviour to match S3 - this means we actively change backslashes in path to forward slashes. I can think of 2 problems here: • Users who use Windows clients and has their data in place (let's say they do not use the web UI and are not bothered by the path parsing difference) will be affected, as new files will be 'named' differently • Not sure this is a real issue, but Unix allows forward slashes as part of a valid path name. If we change these to backslashes it will affect the file path in lakefs.Option 2 - Store the file path/name/key the same way we do today, but treat both backslash and forward slash as path delimiters. This will solve the web UI problem and will not affect the way we keep data, and so, existing user data will not be affected . However, this is handling the symptom rather than the problem, and I'm not sure of where else this problem gonna pop. Moreover, it does not align with our S3-like approach, so not sure this is even an option Any advice, opinion or other considerations I fell to mention? All comments will be greatly appreciated 🙏
    Itai David
    7 replies
    Copy to Clipboard
  • Barak Amar

    Barak Amar

    7 months ago
    build with the new docker uses buildx as far as I know
    Barak Amar
    Or Tzabary
    4 replies
    Copy to Clipboard