• Lucas

    Lucas

    1 year ago
    Hello everyone! I have a question about lakeFS. In DVC, I can commit a data file with a tag and on my code I can choose which from "commit" or "tag" it's gonna pull the data from. Can I do that with lakeFS or the versioning is only system wise? I've used both Pachyderm and DVC and was wondering if lakeFS could be used to replace them for data versioning. Also, how would you describe the difference between lakeFS and Pachyderm/DVC?
    Lucas
    Ariel Shaqed (Scolnicov)
    +1
    11 replies
    Copy to Clipboard
  • Dinakar Chennubotla

    Dinakar Chennubotla

    1 year ago
    Hi All, can anyone help me here. i am trying to connect from lakefs to minio. to get the benefits of lakefs over minio i am doing so. 1. minio is up and running. 2. from doc "git-like operations over minio with lakefs", I am trying to run the below but curl https://compose.lakefs.io | docker-compose --env-file $LAKEFS_CONFIG_FILE -f - up

    error as below:

    lakefs is not coming up with the config provided
    Dinakar Chennubotla
    Yoni Augarten
    3 replies
    Copy to Clipboard
  • Karen

    Karen

    1 year ago
    Slack structure Hello everyone! I've been thinking recently about the current channel structure in this Slack space. Are there any channels that you would like to be introduced here or would you like to change/rename any of the current channels? Many of our members are surely participating at many other slack spaces. Are there any best practices that we here can adopt? We strongly encourage you to express and discuss such ideas in this thread. 🙂 🙏
    Karen
    Yael Rivkind
    4 replies
    Copy to Clipboard
  • Itai Admi

    Itai Admi

    1 year ago
    Did anyone ever experiment with AWS SageMaker with lakeFS? Planning on doing it myself today and wondering if anyone succeeded doing it before.
    Itai Admi
    1 replies
    Copy to Clipboard
  • Welly Tambunan

    Welly Tambunan

    1 year ago
    Hi all, I found about nessie from dremio is quite simmiliar with lakefs. can anyone give a difference and pro/cons? Currently trying to decide which one for our lake house. We also trying to decide between hudi, iceberg or delta-lake Is that just the matter of spesific format supported? like iceberg only for nessie? cc @Karen @Paul Singman
    Welly Tambunan
    Karen
    +2
    10 replies
    Copy to Clipboard
  • Welly Tambunan

    Welly Tambunan

    1 year ago
    hi just found another table sharing engine from delta .. just want to know your thought on this. from my quick read. it’s look like couple to delta lake format.. it’s open standard basically. but not sure where this one will headed…. https://github.com/delta-io/delta-sharing
    Welly Tambunan
    Paul Singman
    +1
    5 replies
    Copy to Clipboard
  • k

    Kesav Kolla

    1 year ago
    I'm new to this concept of Git for data. Can someone give me a pointer on how does it perform with large datasets? I'm dealing with datasets with > 500 million records. How does the git merge happens with this kind of large datasets?
    k
    Paul Singman
    3 replies
    Copy to Clipboard
  • Ariel Shaqed (Scolnicov)

    Ariel Shaqed (Scolnicov)

    1 year ago
    Moving to #general, which may be better for a discussion like this 🙂
    Ariel Shaqed (Scolnicov)
    Xubo Fei
    5 replies
    Copy to Clipboard
  • sumesh kanayi

    sumesh kanayi

    1 year ago
    Hi is copy-object across buckets supported by lakefs .I am trying to copy objects across buckets by running this using local storage
    aws --endpoint-url <http://s3.local.lakefs.io:8000> s3api copy-object --bucket demo-2 --key main/mytest7.py --copy-source demo-1/main/mytest.py
    and returns below error
    An error occurred (InvalidArgument) when calling the CopyObject operation: Copy Source must mention the source bucket and key: sourcebucket/sourcekey.
    I am following https://docs.aws.amazon.com/cli/latest/reference/s3api/copy-object.html#examples
    sumesh kanayi
    Oz Katz
    4 replies
    Copy to Clipboard
  • Leonard Aukea

    Leonard Aukea

    1 year ago
    Hi all! I’m curious, is anybody using lakefs together with https://www.amundsen.io ? it would be kind of cool to make use of Amundsen to “discover” the objects in the lakefs repositories. • I guess on
    AWS
    side of things make
    create-symlink
    might be helpful. But there is probably more valuable metadata that could potentially be made visibile in Amundsen or some similar tool. eg current commit hash, passes pre-commit DQ tests tags etc.
    Leonard Aukea
    Paul Singman
    3 replies
    Copy to Clipboard