Hello! Just had a great Lakefs demo, I'm very new to data versioning and am interested in integrating to my workflow. However, I am super attached to HuggingFace datasets, it is how my entire lab passes data to each other. It seems possible to version the HuggingFace cache files with Lakefs, which would allow for added versioning details and rollback. Has anyone thought about this or gotten that to work?

lakeFS integrates with LangChain and LangChain integrates with HuggingFace. So, you can try this route. Here is a sample of LangChain & OpenAI integration: https://github.com/treeverse/lakeFS-samples/tree/main/01_standalone_examples/llm-openai-langchain-integration https://github.com/treeverse/lakeFS-samples/blob/main/01_standalone_examples/llm-openai-langchain-integration/LLM%20OpenAI%20LangChain%20Demo.ipynb

Hey @Logan Hallee! good news is that Huggingface datasets should be able to work natively with lakeFS! I've opened a PR to document exactly how to read/write datasets from lakeFS, however this requires a small change to a python library (

lakefs-spec

) which is currently open. Once merged and released you'll be able to simply use

lakefs://

file URIs when saving/loading datasets 🤗🤗🤗 I'll update on this thread once both are merged, I don't believe it will take long..

Awesome @Oz Katz. This is a great feature! I'll try it out once merged

Aaaand it's live 🙂 Documentation is available on the lakeFS docs