https://lakefs.io/ logo
#help
Title
# help
b

Bhawana Prasain

02/03/2024, 2:11 PM
If a branch has multiple folders and each folder has multiple files, how can we load all files at once? Is there any docs for this?
b

Barak Amar

02/03/2024, 2:33 PM
Hi @Bhawana Prasain, Option 1: Immutable Data - Import For data you consider "fixed" and won't change outside LakeFS, import is a good choice. Here's the simplified workflow: • Create a branch: Isolate your work from the main branch. • Import data: Add your files to the isolated branch. • Merge changes: Integrate your imported data into the main branch. Option 2: Working Directly with LakeFS If you actively work within LakeFS, consider this approach: • Branch out: Create a dedicated branch for your modifications. • Edit data: Add, delete, or modify files directly in the branch. • Commit and merge: Integrate your changes back to the main branch. Note: Each merge instantly pushes all your changes to the main branch. For deeper understanding - Data Lifecycle Management details: https://docs.lakefs.io/understand/data_lifecycle_management/
b

Bhawana Prasain

02/03/2024, 2:41 PM
I am using LakeFSLoader form langchain. I see that we can set repo ,ref(branch) and path(file name). I instead of filename , can I set folder name and load contents of all files withing the folder?
o

Oz Katz

02/03/2024, 2:57 PM
I believe the lakeFS loader in langchain supports prefixes, so setting the path to
a/
should load any document under the
a/
prefix/directory.. For example, if I had
a/doc1.pdf
and
a/doc2.pdf
, both should be returned.
b

Barak Amar

02/03/2024, 2:58 PM
Thanks Oz, it is specified that path can be object or a prefix (https://python.langchain.com/docs/integrations/document_loaders/lakefs)
o

Oz Katz

02/03/2024, 2:59 PM
indeed!
b

Bhawana Prasain

02/04/2024, 7:11 AM
These are my document structure and my reference for langchain. Am I doing anything wrong?
b

Barak Amar

02/04/2024, 7:19 AM
from the screenshots, no.
feels like a tricky question... so I'm waiting for the second part :)
b

Bhawana Prasain

02/04/2024, 7:28 AM
@Barak Amar Second part? That's not clear to me. What do mean to say? Can you please make it more clear?
b

Barak Amar

02/04/2024, 7:30 AM
everything looks okay, I'm waiting for the loader to run as you expect.
b

Bhawana Prasain

02/04/2024, 7:32 AM
Thanks for the clarification!
@Barak Amar @Oz Katz Thank you for your response! It works fine.
jumping lakefs 1
🎉 1
b

Barak Amar

02/04/2024, 10:07 AM
I'm happy to hear that!
gratitude thank you 1