• Ariel Shaqed (Scolnicov)

8 months ago
TIL Kubernetes lets me use JSON Patch to edit objects. For instance, I can add a backend service to an ingress named "my-ingress" by saying
kubectl patch ingress myingress --type json --patch-file=/tmp/patch.json
where
/tmp/patch.json
is:
[
{
"path": "/spec/rules/-",
"value": {
"host": "<http://another.example.com|another.example.com>",
"http": {
"paths": [
{
"backend": {
"service": {
"name": "another-svc",
"port": {
"number": 5678
}
}
},
"path": "/another",
"pathType": "Prefix"
}]
}
}
}
]
A (more) readable introduction to JSON Patch is http://jsonpatch.com/; there's also https://datatracker.ietf.org/doc/html/rfc6902 of course.
2 replies
• Guy Hardonag

8 months ago
Hey, we got a feature request from one of our users requesting to add the option to set the
time
of the commit. git does support it and I believe we should to, Checking hear to see if anyone has any objections
+1
4 replies
• Ariel Shaqed (Scolnicov)

8 months ago
@Guy Hardonag after we talked yesterday, I came up with this neat trick for transforming any iterator into a prefetching iterator! The basic trick is for the wrapping iterator to start a goroutine that reads values from the base iterator and writes them to a channel of size 10K (for instance). Now to advance and read a value, the wrapping iterator simply reads from the channel. If goroutine gets too far ahead, it fills up the output channel... and blocks. So it cannot move too far, but if it goes to read file or network it still lets the wrapping iterator continue for a bit. How to handle
NextRange
(and even
SeekGE
)? Basically, have another channel from the wrapping iterator to the base iterator. Every time the wrapping iterator needs to change location, it drops the old channel (@Barak Amar do we need to drain a channel, or can it just be garbage-collected while full?), creates a new channel in its stead, and sends a command
<"NextRange", ptrToNewChannel>
to the base iterator. The base iterator now
select
s on being able to write on its output or read from its input; if it gets a command on the input channel it can implement it. Not perfect, but a cheap way to prefetch.
6 replies
• Ariel Shaqed (Scolnicov)

8 months ago
Reversim Summit 2021 conference videos are all out -- tech content in Hebrew (sorry all non-speakers...). https://www.youtube.com/playlist?list=PLqXy0aX6TzQryGoAdbyPevKocQxMJzg8_. Includes some content by familiar faces from lakeFS users, as well as my very own

https://youtu.be/ZeE-JxMZDLk

. 🙂
2 replies
• Tal Sofer

7 months ago
I need to use lfs golang sdk from my code, what go package I should get? I only see this github.com/dollarkillerx/lakefs-sdk and it is not published by the lakefs team (this is very cool!). Is there a reason why we don’t publish this? cc @Barak Amar
2 replies
• mishraprafful

7 months ago
Hey, I just created a PR for a good first issue, but seems like I need approvals from maintainers before the CI checks run. Could you please approve the same. Thanks Ref: https://github.com/treeverse/lakeFS/pull/2900
+1
5 replies
• mishraprafful

7 months ago
Hey, I’m just a bit curious, I get this message every-time I create a PR.
First-time contributors need a maintainer to approve running workflows. Learn more.
but I am not a first time contributor, is it that every PR needs to be approved by a maintainer for running the workflows. Ref: https://github.com/treeverse/lakeFS/pull/2902
3 replies

7 months ago
hi devs, installing statik with
go get <http://github.com/rakyll/statik|github.com/rakyll/statik>
returned the following message:
go get: installing executables with 'go get' in module mode is deprecated.
To install using requirements of the current module, use 'go install'.
To install ignoring the current module, use 'go install' with a version,
like 'go install <http://example.com/cmd@latest|example.com/cmd@latest>'.
or run 'go help get' or 'go help install'.
which one is should I use to install statik?
go get -d
or
go install
?
6 replies
• Itai David

7 months ago
Hi all. I'm attaching a technical discussion that we were (mistakenly) having in an internal thread. It concerns an issue regarding the way we handle Windows paths. TL;DR - we want to change backslashes (
\
) in Windows' style paths to Unix's style forward slashes (
/
) to match S3 behavior. Out main concern is the effect it might have on users currently using Windows' style paths and the effect it might have on their data. The original message is attached hereafter. I'm attaching all responses as comments. All further comments are very appreciated. Thanks Hi. I would love to here some opinions and some your of concerns regarding an issue I'm working on - BUG : Upload directory path not parsed correctly #2880 TL;DR - we are not treating Windows' path delimiter - the backslash - as such Upon uploading a file, from Windows, the original path, e.g.
C:\some\path\to\file
is kept and used as a key. Later, when the file is returned to the client(s), as a response for ListObjects, we do not parse the key as a path correctly. While
lakectl
shows the key as a full path either ways, the web UI fails to create the nice (and expected) tree display, as described in the bug On @Tal Sofer's advice, I looked at S3 behaviour, and found out that AWS actually converts Windows` style paths to Unix style. Namely, all backslashes are turned into forward slashes. That's it for the problem description, now for possible solutions:Option 1 - change lakefs behaviour to match S3 - this means we actively change backslashes in path to forward slashes. I can think of 2 problems here: • Users who use Windows clients and has their data in place (let's say they do not use the web UI and are not bothered by the path parsing difference) will be affected, as new files will be 'named' differently • Not sure this is a real issue, but Unix allows forward slashes as part of a valid path name. If we change these to backslashes it will affect the file path in lakefs.Option 2 - Store the file path/name/key the same way we do today, but treat both backslash and forward slash as path delimiters. This will solve the web UI problem and will not affect the way we keep data, and so, existing user data will not be affected . However, this is handling the symptom rather than the problem, and I'm not sure of where else this problem gonna pop. Moreover, it does not align with our S3-like approach, so not sure this is even an option Any advice, opinion or other considerations I fell to mention? All comments will be greatly appreciated 🙏
7 replies
• Barak Amar

7 months ago
build with the new docker uses buildx as far as I know
4 replies