we did this maybe can be useful https://github.com/RstorLabs/rs-benchmark

HI Itai, Im looking for the recommended instance size on AWS. Can it run on RISK or it needs X86?

Welcome @Joydeep Banik Roy!

If you guys want me to specifically look into some issues or point me to some doc

I'll be giving a talk this Tuesday on Data Versioning - use cases, tool comparison and an overview of lakeFS - feel free to join here: https://www.meetup.com/Big-things-are-happening-here/events/272943689/

Welcome, Devarsh! You'll probably want to read https://hacktoberfest.digitalocean.com/details/. But tl;dr: • Optional: Find an issue https://github.com/treeverse/lakeFS/issues?q=is%3Aissue+is%3Aopen+label%3Ahacktoberfesthttps://docs.lakefs.io/contributing that isn't assigned to anyone • Optional: Ping us to assign you to that issue • Please read https://docs.lakefs.io/contributing (it's short and simple!) • Send us a PR during October We're always happy to accept PRs for stuff that isn't on our list! Please do ask any questions here or on #dev, or just DM me. Looking forward to seeing you on GitHub!

Hey @ABC 🙂 - in short: DVC is more focused on ML pipeline management including versioning of sample data, model and metrics. lakeFS focuses on Data Lake development life cycle - making changes to very large scales of data safely by allowing isolation, revert and CI/CD capabilities on the data itself. For a more thorough overview, you can check out this talk I gave recently comparing the different data versioning tools: (including DVC)

hi, everyone

Hey!

Are you currently using an object storage?

i have a working pg, minio, lakefs within eks in case anyone needs help in future, please ping me

@Barak Amar really appreciate your two points above as you mentioned, they are not in docs 🙂 those were the key, again, thanks

is lakefs k8s operator in development ?

what are you planning for testing

hello, will there be additional swag for Hacktoberfest contributors? :P

Hi, I'm a newbie on lakefs and congrats to the team contributing to it. It might be me but I couldn't find the supported operations on S3. I've setup a docker swarm env using minio and lakefs(for testing purposes) When I try to revert a branch to a certain commit id, i get the following error: lakectl branch revert lakefs://example-repo@master --commit ~KJ8Wd1Rs96Z Error executing command: feature not supported is that expected?

we support Python using both boto - https://docs.lakefs.io/using/boto.html and our full api - https://docs.lakefs.io/using/python.html

where iceberg is limited to tabular data, this sort of design would be good for other types as well

I think time travel is sufficient, and probably easier to reason about than a DAG for tabular data. If I can point a workload to a point in time, I have reproducibility in terms of interacting with the same data

Hello there, How does the integration of lakefs with Airflow work? Thanks

Main two things to note are they only support Iceberg and potentially Delta, and their architecture is less cost effective (dynamo).

Hi, tl;dr: Please fill out https://forms.gle/cLAzDMpTB9vpndR6A if you can answer some questions about a Spark client for reading lakeFS metadata. We're designing a thick client for lakeFS metadata. It will allow writing Spark jobs to perform operations on metadata (inventory of files in a commit) on lakeFS. I'm interested in what features would be most useful to you. If you could fill out https://forms.gle/cLAzDMpTB9vpndR6A (should take 5 minutes) it would be great! I shall summarize results here. If you like, fill out your email on the form and I'll be happy to email you the results. Thanks!

Hi Barrett, Yes, a client is on our road map to be released on a few weeks. Please share your use case, it is important for our learning of the needs.

Following up on that previous question. I have another, I think I know the answer based on the reading of the docs, based it's worth an ask. What is assumption of ownership of the underlying object on the part of LakeFS?

Hi Nishant, yes, you can. Here's how: https://lakefs.io/git-like-operations-over-minio-with-lakefs/

You can use lakeFS with Hadoop, Spark and Hive. See the section "use lakeFS with" in our documentation for how. However, lakeFS doesn't support HDFS as a storage layer. For storage you need to use an object storage such as minIO.

Hello guys, I found out this project on yesterday and it's amazing what you're doing here, I am really impressed. I need to start navigating the tool but I wanna make sure if it integrates with AWS glue or not, if so, could u please send me some reference of the way of doing that? I googled a bit but seems there's no much resources.

Hi @Jai Jagani, welcome! This post explains how to use lakeFS over MinIO: https://lakefs.io/git-like-operations-over-minio-with-lakefs/ The example there starts lakeFS using docker-compose, with postgres also running as a docker container.

Can you please let me know how to install lakectl ?