https://lakefs.io/ logo
Join Slack
Powered by
# help
  • r
    Hi @all, I am trying build a golang wrapper over the APIs provider in the swagger docs, but when I am trying basic auth with the access_key_id and secret_key, it always says authentication fails. bdw, I just go introduced to lakefs and it is amazing. Please help me out
    o
    b
    • 3
    • 7
  • γ
    Hi everyone 👋 I am trying to build the project (I have contributed some years ago) but I am unable to get a successful build when running 
    make build
    . The command reaches the following point: 
    /usr/local/go/bin/go build -o lakefs -ldflags "-X <http://github.com/treeverse/lakefs/pkg/version.Version=dev-c5680959f.with.local.changes|github.com/treeverse/lakefs/pkg/version.Version=dev-c5680959f.with.local.changes>" -v ./cmd/lakefs
    and then crashes with this error: 
    pkg/api/lakefs.gen.go:23001:30: undefined: openapi3.Swagger
    pkg/api/lakefs.gen.go:23011:27: undefined: openapi3.NewSwaggerLoader
    make: *** [build-binaries] Error 1
    . Checking the piece of code apparently there is an error with that specific package which doesn’t exist
    b
    • 2
    • 3
  • i
    does lakectl config support multiple endpoints at once? It seems I can only configure one single endpoint in the config
    i
    • 2
    • 1
  • t
    Hi, I am just wondering if there are any plans in the near future to make 
    lakectl local
    more like 
    git
    ? I would just like to say, I have so far really enjoyed using 
    lakefs
    , and it seems to be the answer to my pain point for a long time. I mainly use it as I would use git -- to version data through the 
    lakectl local
    interface. However, compared to git, it's missing quite a lot of features, which makes it, while useable, somewhat finicky when I want to do things in certain ways. Here I list a few examples: 1. I would like to checkout only a directory or file rather than the entire repository for a commit with 
    lakectl checkout
    . 2. I would like to examine changes to certain files between different commits, similar to 
    git difftool
    . Are there plans to include features such as the above in the near future? Or am I not quite using 
    lakefs
    in the intended way?
    i
    a
    • 3
    • 13
  • t
    Another question. With reference to https://lakefs.slack.com/archives/C016726JLJW/p1760420043187809?thread_ts=1760416389.902419&amp;cid=C016726JLJW above, there isn't a 
    lakectl mv
    command that can handle renaming. I would just like to know, if I simply rename a folder, since 
    lakefs
    will remove and re-upload everything, does it mean that I will also double the storage usage, since 
    lakefs
    tracks changes to my directory?
    i
    • 2
    • 4
  • a
    Hi all, Thanks for hosting this community. I have a question about LakeFS API in Java (Scala), I am using the following simple function to generate presignedURL:
    Copy code
    statObject(repoName, commitHash, filePath).presign(true).execute().getPhysicalAddress
    But it does not support filename (content disposition header), I went through full library io.lakefs but there is no option to add headers to presigned URL, the only option is to use S3 API and use LakeFS directly which is not standard. Is there any way that LakeFS add filename to presigned URL coming from MinIO?
    a
    • 2
    • 6
  • r
    There is a Dec 2nd, LakeFS fireside chat that I want to register for, but the "Register" button does nothing. Tested on latest Google Chrome, Firefox and Safari. What am I missing?
    y
    k
    • 3
    • 7
  • t
    Thanks for the help earlier. Just now, I was experimenting with the 
    lakectl fs upload
    command. I was hoping it can allow me to commit parts of the changes to a repo instead of all changes all at once, as I would do via 
    lakectl local commit
    . 
    lakectl fs upload
    did allow me to upload changes to a single file -- I suppose it will allow me a single directory also. I also manged to commit the changes with 
    lakectl commit
    . However, now if I run 
    lakectl local status
    , I get a "conflict" tag with the changed file, which I am unable to resolve. It looks like this.
    Copy code
    ╔════════╦══════════╦═══════════╗
║ SOURCE ║ CHANGE   ║ PATH      ║
╠════════╬══════════╬═══════════╣
║ local  ║ conflict ║ file2.txt ║
║ local  ║ added    ║ file3.txt ║
╚════════╩══════════╩═══════════╝
    In my experiment, I modified 
    file2.txt
    and added 
    file3.txt
    . But I only ran 
    lakectl fs upload <URI> -s file2.txt
    , followed by 
    lakectl commit -m
    . How do I now resolve the conflict? I suppose I need to somehow sync the commit ID in the server with the 
    .lakefs_ref.yaml
    file?
    👀 1
  • h
    Switching 
    main
    branch hi all, we are using OSS version of lakefs. We have the situation where we mistakenly merge a wrong branch to main. We started with this: A good branch, then branch out and had a bunch of bad commit on a "bad-branch" And then merged bad-branch to main ! With git, we would just delete main, and create a new main out of the good branch. But I cannot delete main branch, nor changing the default branch of the repo to something else ... One solution is just delete data in main and then re-copy all files from the good branch. But is there a more efficient way ?
    i
    a
    • 3
    • 52
  • c
    Hi all! I recently observed a weird behavior regarding upload object process. I uploaded a bunch of objects using lakectl, none of them thrown a error so I supposed everything was okay. This morning I retrieve some of that data and 2% of the data was corrupted (they were binary objects instead of csv). However the size of the file is the correct one. Any idea of what happened?? 😞 version: 1.29.0
    😱 1
    a
    • 2
    • 9
  • d
    Hi All, We are currently using LakeFS version 1.5. We're planning to update it, and I noticed that the latest version available is 1.73, which is quite a jump. Are there any breaking changes between version 1.5 and 1.73? If upgrading directly to 1.73 is not advisable, which version would you recommend we update to? Any thoughts or suggestions are welcome.
    o
    • 2
    • 2
  • i
    Too many attempts, try again later
    seeing this sometimes when merging, in the logs I see 
    service_name=graveler_graveler try=8
    are these related? I assume so, but I can't see how I can increase the retries in the docs
    lakefs 1
  • i
    time="2025-12-08T11:18:24Z" level=debug msg="Retried too many times" func="pkg/api.(*Controller).handleAPIErrorCallback" file="build/pkg/api/controller.go:3061" error="update branch: too many tries (last predicate failed)" host=[redacted] method=POST operation_id=MergeIntoBranch path=/api/v1/repositories/bronze/refs/delta-tx-3caf13d7-4ab6-4878-9c15-a9874497ba4c/merge/main service=api_gateway user=[redacted]
    lakefs 1
  • i
    Is there a way to stop lakefs choking on concurrent branch merges?
    a
    • 2
    • 16
  • d
    Hi all, I'm experiencing an unusual behavior with garbage collection, and I need some assistance. Below is the current configuration:
    Copy code
    {
  "branches": [
    {
      "branch_id": "main",
      "retention_days": 7
    }
  ],
  "default_retention_days": 1
}
    However, I've noticed that some branches are much older than 1 day, even though the retention period should be respected. Can you help me understand what might be causing this behavior? Is there a known issue in this area? Thank you for your help! server version: 1.5
    b
    n
    • 3
    • 13
  • d
    Hello, Where can I find the latest Garbage Collector JAR? One more question: there’s a file at 
    s3://${BUCKET_NAME}/_lakefs/retention/gc/rules/config.json
    , and its content is 
    main
    . Is that expected behavior?
    n
    • 2
    • 1
  • u
    Hi team, I think https://github.com/treeverse/lakeFS/pull/10016 this feature is very useful to monitor and troubleshoot performance issues. Are there more detailed instructions for use?
    a
    • 2
    • 1
  • m
    Hey all! I'm new to lakeFS. Are there limitations in using lakeFS without docker? We have our data lake stored in the cloud and my teammates share a VM to work directly with datasets. Currently, we never needed docker. Is it ok not use it?
    o
    • 2
    • 2
  • y
    Hello, I am experimenting hook according to this script: https://docs.lakefs.io/latest/integrations/glue_metastore/#the-exporter-script. We deployed lakefs in a EKS with lakefs docker image. I can trigger the script now but I failed to login aws. I tried several ways: 1. use key/secret/session_token, but aws.s3_client() only accept 3 2. use empty aws.s3_client() 3. aws.s3_client({}) 4. aws.s3_client(region) All failed. I also tried to call aws.sts_clieng() but it returned nil My question is, my lakefs running in a EKS with a proper IAM role in AWS. How can I get s3_client in this environment? Thanks
    o
    • 2
    • 5
  • d
    Hey there! I have been using lakefs OSS with postgres in RDS. Is there any guides on how I might migrate the KV store from RDS to Dynamo? Is it as simple as reading out the current KV store in postgres and populating a new Dynamo table with it? Thanks
    o
    • 2
    • 3
  • h
    Hi! I am interested in coming to https://lakefs.io/ai-ready-data-summit/ but seems unable to register at all on the website (the form fails to clear upon clicking submit and I didn't receive any confirmation email). Was wondering if there's another way to sign up?
    lakefs 1
  • g
    Hello @Hong Ha Vu, sorry to hear that you are having issues. We are using a standard HubSpot form that works for most users. In rare instances it could be blocked by browser or company firewall settings. Alternatively, you can sign up via our LinkedIn events page or DM me your email address and we’ll add you manually. See you at the Summit!
    h
    • 2
    • 1
  • a
    Hello 🙂 I'm new to LakeFS. I'm interested by the enterprise version, would it be possible to discuss about this ?
    e
    v
    • 3
    • 5
  • g
    Hello! I am wondering if anyone here has experience integrating Cloudflare R2 with LakeFS? I have seen some discussions online from several years ago claiming that some R2 quirks made it impossible (since R2 is apparently "almost" fully S3 compatible). Would be great to know if someone has tried this, or if someone knows whether this integration works / is supported? Thanks!
    a
    • 2
    • 3
  • u
    Hi, We are currently using LakeFS with PostgreSQL. We have allocated a large PVC to PostgreSQL, but we’ve noticed that its memory usage continues to grow, eventually leading to the error: “could not write to log file:
    Copy code
    No space left on device"
2026-04-19 23:37:15,504 WARNING: Retry got exception: connection problems
2026-04-19 23:37:15,504 INFO: primary start has timed out, but continuing to wait because failover is not possible
    Upon investigating the resource usage of the PostgreSQL pod, we found that the majority of the usage is in the base memory. Could you please recommend a configuration solution? Is there a way to prevent PostgreSQL from consuming excessive memory and to trigger periodic memory release?
    Copy code
    4.0K    /home/postgres/pgdata/pgroot/data/pg_commit_ts
4.0K    /home/postgres/pgdata/pgroot/data/pg_dynshmem
4.0K    /home/postgres/pgdata/pgroot/data/pg_notify
4.0K    /home/postgres/pgdata/pgroot/data/pg_replslot
4.0K    /home/postgres/pgdata/pgroot/data/pg_serial
4.0K    /home/postgres/pgdata/pgroot/data/pg_snapshots
4.0K    /home/postgres/pgdata/pgroot/data/pg_stat
4.0K    /home/postgres/pgdata/pgroot/data/pg_tblspc
4.0K    /home/postgres/pgdata/pgroot/data/pg_twophase
8.0K    /home/postgres/pgdata/pgroot/data/pg_stat_tmp
16K     /home/postgres/pgdata/pgroot/data/pg_logical
28K     /home/postgres/pgdata/pgroot/data/pg_multixact
44K     /home/postgres/pgdata/pgroot/data/pg_subtrans
612K    /home/postgres/pgdata/pgroot/data/global
49M     /home/postgres/pgdata/pgroot/data/pg_xact
993M    /home/postgres/pgdata/pgroot/data/pg_wal
469G    /home/postgres/pgdata/pgroot/data/base
470G    /home/postgres/pgdata/pgroot/data
    a
    • 2
    • 5
  • t
    Hi, I'm looking into using the mount functionality but the docs page doesn't seem to load: https://docs.lakefs.io/reference/mount/
    i
    • 2
    • 1
  • j
    Hello, are the garbage collection spark binaries all backwards compatible? I’m still running LakeFS server version 0.99.0 and I'd like to run committed and uncommitted GC individually. Should I used the spark client version that shipped with 0.99.0 or should I use a more recent version?
    n
    i
    a
    • 4
    • 16
  • j
    Hi, I'm not sure if this is the right place to ask this - I've been trying to contact sales to ask about pricing but I haven't heard back yet. Can anyone help?
    y
    f
    • 3
    • 3
  • m
    Hello, did anyone used lakeFS with AWS SageMaker Pipelines? I saw in the docs, that described SageMaker use case is more about overriding the endpoint URL with lakeFS one, and manually downloading/uploading the data from/to lakeFS instead of S3 via the session object, but I'm more interested with integrating lakeFS directly with a Pipelines environment, for example using lakeFS URI instead of S3 URI as an input to particular pipeline steps, like processing jobs Does anyone know if this is possible with the lakeFS, or have any experience with that?
    💡 1
    i
    • 2
    • 1
  • c
    hey, we have hit minio's minimum free space threshold with a major upload. Looks like deletion from this position w/ lakefs is not easy. Is there a recommended way to delete some items?
    n
    • 2
    • 6