• Tal Sofer

    Tal Sofer

    11 months ago
    I’m using the getObject endpoint to get the contents of two objects (of size <= 100MB) that I would like to compare to calculate their diff. The getObject operation returns the object contents as an
    application/octet-stream
    , and I’m looking into using a react library that can calculate the diff for me, this library gets file contents as strings. • Should I first read the contents and save it in-memory and then compare it? • What is the right way to read from a stream in javascript? @Barak Amar @Ariel Shaqed (Scolnicov) do you have useful tips to share?
    Tal Sofer
    Itai Admi
    +1
    4 replies
    Copy to Clipboard
  • Tal Sofer

    Tal Sofer

    11 months ago
    Another React question - I’m trying to add a dependency to our project and i’m getting the following error:
    webui % npm i react-diff-viewer
    
    npm ERR! code ERESOLVE
    npm ERR! ERESOLVE unable to resolve dependency tree
    npm ERR! 
    npm ERR! While resolving: lakefs-ui@0.1.0
    npm ERR! Found: react@17.0.2
    npm ERR! node_modules/react
    npm ERR!   react@"^17.0.0" from the root project
    npm ERR! 
    npm ERR! Could not resolve dependency:
    npm ERR! peer react@"^15.3.0 || ^16.0.0" from react-diff-viewer@3.1.1
    npm ERR! node_modules/react-diff-viewer
    npm ERR!   react-diff-viewer@"*" from the root project
    npm ERR! 
    npm ERR! Fix the upstream dependency conflict, or retry
    npm ERR! this command with --force, or --legacy-peer-deps
    npm ERR! to accept an incorrect (and potentially broken) dependency resolution.
    npm ERR!
    Is this safe to use the
    --legacy-peer-deps
    flag to resolve this? reading about this make me suspect that it can lead to potential conflicts in the future. has anyone ran into a similar error?
    Tal Sofer
    Barak Amar
    8 replies
    Copy to Clipboard
  • Barak Amar

    Barak Amar

    11 months ago
    Going to create a branch
    release/0.53.0
    based on the tag
    v0.53.0
    in order to include the fixes to release the python client and re-tag it as the version. If you see any issue please let me know
    Barak Amar
    Ariel Shaqed (Scolnicov)
    2 replies
    Copy to Clipboard
  • Barak Amar

    Barak Amar

    11 months ago
    Let me know if there is anything pending you would like to include in the next release - for now it is bug fixes and clients.
    Barak Amar
    1 replies
    Copy to Clipboard
  • Tal Sofer

    Tal Sofer

    10 months ago
    Working on the content diff feature and looking for feedback on visualisation - Specifically speaking, after adding the content diff feature, a directory and object entries differ only in their icon and directories ends with the “/” separator. both are expandable and the results of expanding them are different. @Oz Katz do you have thoughts about this?
    Tal Sofer
    Itai Admi
    +2
    21 replies
    Copy to Clipboard
  • Tal Sofer

    Tal Sofer

    10 months ago
    Hi! I will appreciate an advice on how to implement the following: This is my code that wraps a call to our objects API -
    async get(repoId, ref, path, additionalHeaders) {
            const query = qs({path});
            const response = await apiRequest(`/repositories/${repoId}/refs/${ref}/objects?${query}`, {
                method: 'GET',
                headers: new Headers(additionalHeaders)
            });
            if (response.status !== 200 && response.status !== 206) {
                throw new Error(await extractError(response));
            }
            return response.text()
        }
    The promise it returns reads the object text from a stream and then returning. This get operation receives a Range header as a parameter and I need to track the response “Content-Range” to determine whether the full object content is returned or only part of it. The problem i’m facing is how to return this information while the get function must return a promise because i’m invoking it with useAPI? Can I somehow wrap the
    response.text(), response.headers.get("Content-Range")
    with a promise? @Barak Amar @Guy Hardonag @Ariel Shaqed (Scolnicov) maybe you have an advice?
    Tal Sofer
    Barak Amar
    +1
    8 replies
    Copy to Clipboard
  • Itai Admi

    Itai Admi

    10 months ago
    Working on issue#2629, trying to figure out what are the FileSystem calls being made to lakeFSFS and what are the calls it does to the underlying s3a FS. I’ve configured the following in
    /usr/lib/hadoop/etc/hadoop/log4j.properties
    file:
    log4j.appender.A1=org.apache.log4j.ConsoleAppender
    log4j.appender.A1.layout=org.apache.log4j.PatternLayout
    log4j.appender.A1.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n
    log4j.logger.io.lakefs=TRACE, A1
    But I still can’t see the TRACE logs in my stdout. @Tal Sofer @Ariel Shaqed (Scolnicov) @Barak Amar how should I configure the logs?
    Itai Admi
    1 replies
    Copy to Clipboard
  • Ariel Shaqed (Scolnicov)

    Ariel Shaqed (Scolnicov)

    10 months ago
    Does anyone know how to add fields from context to those emitted by an AWS client? This is for https://github.com/treeverse/lakefs/issues/2682. Context (sorry): Looking to "thread" our request_id field throughout all lakeFS logs. One issue that has me confused is how to get AWS to log with request IDs. Something like https://github.com/aws/aws-sdk-go/pull/3485 would be great, but that PR went silent. I do no want to dupe the client only to change its configuration, that may be overpowering for the issue.
    Ariel Shaqed (Scolnicov)
    Barak Amar
    8 replies
    Copy to Clipboard
  • Ariel Shaqed (Scolnicov)

    Ariel Shaqed (Scolnicov)

    10 months ago
    Looking hard at upgrading to aws-sdk-go-v2. Why does the S3 block adapter have
    streamToS3
    that hanldes its own HTTP request? What did v1 not offer in the interface that made us have to use our own client code?
    Ariel Shaqed (Scolnicov)
    Barak Amar
    +1
    12 replies
    Copy to Clipboard
  • Ariel Shaqed (Scolnicov)

    Ariel Shaqed (Scolnicov)

    10 months ago
    Looking at
    StreamingReader
    in our block/s3 package (click to see): Size is
    int
    . Is that correct?
    Ariel Shaqed (Scolnicov)
    1 replies
    Copy to Clipboard