• einat.orr

    einat.orr

    1 year ago
    Main two things to note are they only support Iceberg and potentially Delta, and their architecture is less cost effective (dynamo).
    einat.orr
    Kyle Bader
    +2
    8 replies
    Copy to Clipboard
  • Ariel Shaqed (Scolnicov)

    Ariel Shaqed (Scolnicov)

    1 year ago
    Hi, tl;dr: Please fill out https://forms.gle/cLAzDMpTB9vpndR6A if you can answer some questions about a Spark client for reading lakeFS metadata. We're designing a thick client for lakeFS metadata. It will allow writing Spark jobs to perform operations on metadata (inventory of files in a commit) on lakeFS. I'm interested in what features would be most useful to you. If you could fill out https://forms.gle/cLAzDMpTB9vpndR6A (should take 5 minutes) it would be great! I shall summarize results here. If you like, fill out your email on the form and I'll be happy to email you the results. Thanks!
    Ariel Shaqed (Scolnicov)
    1 replies
    Copy to Clipboard
  • einat.orr

    einat.orr

    1 year ago
    Hi Barrett, Yes, a client is on our road map to be released on a few weeks. Please share your use case, it is important for our learning of the needs.
    einat.orr
    Barrett Strausser
    +1
    7 replies
    Copy to Clipboard
  • Barrett Strausser

    Barrett Strausser

    1 year ago
    Following up on that previous question. I have another, I think I know the answer based on the reading of the docs, based it's worth an ask. What is assumption of ownership of the underlying object on the part of LakeFS?
    Barrett Strausser
    Oz Katz
    8 replies
    Copy to Clipboard
  • einat.orr

    einat.orr

    1 year ago
    einat.orr
    1 replies
    Copy to Clipboard
  • einat.orr

    einat.orr

    1 year ago
    You can use lakeFS with Hadoop, Spark and Hive. See the section "use lakeFS with" in our documentation for how. However, lakeFS doesn't support HDFS as a storage layer. For storage you need to use an object storage such as minIO.
    einat.orr
    Nishant Shrivastav
    10 replies
    Copy to Clipboard
  • Adly Mousa

    Adly Mousa

    1 year ago
    Hello guys, I found out this project on yesterday and it's amazing what you're doing here, I am really impressed. I need to start navigating the tool but I wanna make sure if it integrates with AWS glue or not, if so, could u please send me some reference of the way of doing that? I googled a bit but seems there's no much resources.
    Adly Mousa
    Yael Rivkind
    4 replies
    Copy to Clipboard
  • Yoni Augarten

    Yoni Augarten

    1 year ago
    Hi @Jai Jagani, welcome! This post explains how to use lakeFS over MinIO: https://lakefs.io/git-like-operations-over-minio-with-lakefs/ The example there starts lakeFS using docker-compose, with postgres also running as a docker container.
    Yoni Augarten
    Jai Jagani
    18 replies
    Copy to Clipboard
  • Jai Jagani

    Jai Jagani

    1 year ago
    Can you please let me know how to install lakectl ?
    Jai Jagani
    Oz Katz
    2 replies
    Copy to Clipboard
  • Dinakar Chennubotla

    Dinakar Chennubotla

    1 year ago
    i am working on POC, i.e. need to submit data from kafka to lakefs and then finally to Minio. can anyone help with docs or knowledge base. please do the needful
    Dinakar Chennubotla
    Ariel Shaqed (Scolnicov)
    8 replies
    Copy to Clipboard