when using a self host of lakefs with s3 connection i tried using lakectl commands on a repo while a team member tried reading the files from a branch they failed with reading the files until i disconnected from the program do you have any advice on why it happens

Hey @mayans20122012 - welcome to the lake. Can you share more details on the error? Client error logs, server error logs, specific operations that failed, …

there is no error message i use a remote machine that contains a mount to our lakefs from s3, i connect to the machine using vscode ssh. when we mount our lakefs from s3 into vscode the folder where we mount to dosent load and any attempt to get into it like with cd commands would cause the vs code to freeze, but other bucket in our s3 are still accessible thanks for your help

Hi @mayans20122012, I think we might be missing some details. Could you explain what you are trying to do? • Where is your lakeFS installed? Does "from S3" mean you're using the S3 blockstore, or something else? • Have you gone over our lakeFS quickstart? (This is highly recommended!) • What does "mount to our lakefs" mean? This could be Everest, but could also mean something else. • What does "doesn't load" mean? How are you connecting lakeFS into VSCode, and is S3 involved in any way other than as a backend for lakeFS? • How are you accessing "other buckets" in S3? Thanks! I'm sure we can help once we better understand what you are doing, what you expect, and what you actually get.

i have them mounted through runai as data sources both lakefs and the other s3 buckets im accessing both types of buckets through the vscode file system using the runai mounting in the lakefs buckets the s3 is the source of the data and the backend for the lakefs system

I am not sure what "run AI" here means; is this the nVidia run:ai facility? Let's try to break down the issue into small steps: rather than trying to debug run:ai + lakeFS + S3 + VSCode, ideally we would work on each part separately. • Does your lakeFS installation work? • Can you run run:ai without VSCode? What happens when you do? • Can you connect lakeFS to work with Boto (Python S3 library)? That is probably the most-used integration, once you get that going you'll know a lot about configuration. • How did you configure lakeFS to work with run:ai? If using the S3 gateway you will need to configure all of these: ◦ access key (from lakeFS) ◦ secret key (from lakeFS) ◦ URL of lakeFS endpoint

for point 3 i worked with lakectl and it does connect to the repo i connect my lakefs to run:ai exactly as you said using the s3 gateway using the keys and url my lakefs url works fine even in the effected repo i will try to check point 2

Okay, obviously you will not be able to read an object whose data you deleted 🙂. Regarding other objects: do you see any messages on your lakeFS log? I would imagine that run:ai fails to access your lakeFS.

all the deleted object have been removed from the branches and im not trying to access them i tried connecting to the pod using mobaxterm and still face this issue the thing is run:ai manages to access some of the lakefs repoes without issue but specific repos fail to load when used

How did you identify which objects to delete from S3?

I've looked through the logs, and there are no "Warning" logs or anything else that appears abnormal. I don't believe the problem is related to those deletes because I also created a brand-new repository and used

lakectl

to properly delete a batch of pointers, yet the problem still reappeared. Since I can see the changes correctly in the LakeFS web interface, I suspect the problem has less to do with LakeFS itself and more to do with how I am connecting it to my work environment. I am mounting LakeFS through Run:AI using an S3-compatible container, and I am also using VS Code as my IDE. Furthermore, once a repository becomes "ruined," the issue persists across pods. In some cases, it even prevents VS Code from connecting to the pod, despite the pod still showing a "Running" status. I have also checked the logs of the affected pods, and they appear to be normal.

Hi @mayans20122012, this doesn't sound like repo corruption. Quick things to check: 1. Logs. lakeFS logs per-request lines at DEBUG, so a hanging request shows nothing at INFO. Set logging.audit_log_level: INFO and check the took value on ListObjects/HeadObject from the run:ai pod. 2. Branch count. At a repo root the S3 gateway lists branches, capped at 1000. With the older ListObjects v1 API, a repo with >1000 branches can put the client in a paging loop that never ends: exactly a spinner that never resolves, and it follows the repo across pods. How many branches do the broken repos have versus working ones? Use ListObjectsV2 if you can. 3. Uncommitted deletes. lakectl status <repo>/<branch>. Uncommitted deletes sit in the staging area and every listing page scans past them. Try lakectl commit and see if the mount recovers. 4. Mount a branch, not the repo root. VS Code recursively walks the workspace root for file watching and search. At the repo root that fans out to every branch × every object. Mount a single branch/prefix and add it to files.watcherExclude / search.exclude. Branch count, whether the mount uses v1 or v2 listing, and one took log line would narrow it down.