If a branch has multiple folders and each folder has multiple files, how can we load all files at once? Is there any docs for this?

Hi @Bhawana Prasain, Option 1: Immutable Data - Import For data you consider "fixed" and won't change outside LakeFS, import is a good choice. Here's the simplified workflow: • Create a branch: Isolate your work from the main branch. • Import data: Add your files to the isolated branch. • Merge changes: Integrate your imported data into the main branch. Option 2: Working Directly with LakeFS If you actively work within LakeFS, consider this approach: • Branch out: Create a dedicated branch for your modifications. • Edit data: Add, delete, or modify files directly in the branch. • Commit and merge: Integrate your changes back to the main branch. Note: Each merge instantly pushes all your changes to the main branch. For deeper understanding - Data Lifecycle Management details: https://docs.lakefs.io/understand/data_lifecycle_management/

I am using LakeFSLoader form langchain. I see that we can set repo ,ref(branch) and path(file name). I instead of filename , can I set folder name and load contents of all files withing the folder?

I believe the lakeFS loader in langchain supports prefixes, so setting the path to

a/

should load any document under the

a/

prefix/directory.. For example, if I had

a/doc1.pdf

and

a/doc2.pdf

, both should be returned.

Thanks Oz, it is specified that path can be object or a prefix (https://python.langchain.com/docs/integrations/document_loaders/lakefs)

indeed!

These are my document structure and my reference for langchain. Am I doing anything wrong?

from the screenshots, no.

@Barak Amar Second part? That's not clear to me. What do mean to say? Can you please make it more clear?

everything looks okay, I'm waiting for the loader to run as you expect.

Thanks for the clarification!

I'm happy to hear that!