• v

    Verun Rahimtoola

    6 months ago
    reset branch question
    v
    Itai David
    5 replies
    Copy to Clipboard
  • Bjorn Olsen

    Bjorn Olsen

    6 months ago
    Hey all, first time poster. S3 gateway question:
    Bjorn Olsen
    Jonathan Rosenberg
    6 replies
    Copy to Clipboard
  • Ori Kabeli

    Ori Kabeli

    6 months ago
    Hey all! I came across this article mentioning the use of lakeFS in the CI/CD of data pipelines. I’m wondering if anyone knows and can recommend of tools (OSS or not) that helps in continuous deployment of AI inference workflows in prod. The scenario: complex pipeline in prod serving users in real-time, involving several models and business logic that is constantly changing. Challenge: testing how a change will impact this DAG before landing a change (model, data, code) Any pointers will be greatly appreciated, and sorry if this is a noob question or out of context 🙂
    Ori Kabeli
    Tal Sofer
    6 replies
    Copy to Clipboard
  • Bjorn Olsen

    Bjorn Olsen

    6 months ago
    Hey all. A question on talking to S3 using the Hadoop-LakeFS-assembly:
    Bjorn Olsen
    Ariel Shaqed (Scolnicov)
    +1
    11 replies
    Copy to Clipboard
  • Jelle De Jong

    Jelle De Jong

    6 months ago
    Hi, I just joined this channel and have a question on how to best use lakefs for a specific use case. I am looking for some kind of 'best practice' workflow for reprocessing data. So the situation would be that you have some pipeline logic (code) that you changed and want to apply to your historical production data. I guess that you would have a git branch with your modified pipeline code, and could have a lakefs branch to test the changes on in isolation. But if you want to release those changes to production, how would you go about it? Also taking into account that during the development and testing of your new pipeline code, new data might have been ingested and processed by the current pipeline logic.
    Jelle De Jong
    Barak Amar
    +2
    8 replies
    Copy to Clipboard
  • Adi Polak

    Adi Polak

    6 months ago
    Been following the great tutorial on how to use lakeFS and airflow together, to branch out of the main data branch, run spark logic, and later commit automatically. thanks! one question, using the
    CommitOperator
    can I add
    application
    there too? I want to run validation checks on the data. do I need a spark engine to qualify the data?
    Adi Polak
    Itai Admi
    2 replies
    Copy to Clipboard
  • Adi Polak

    Adi Polak

    6 months ago
    Following this integration tutorial on running lakeFS with Databricks, I think I am missing something there. is there a library I need to install on the databricks cluster to make them work? Which dependencies should be there?
    Adi Polak
    Guy Hardonag
    +4
    23 replies
    Copy to Clipboard
  • m

    Matias Stanislavsky

    6 months ago
    Hello everybody...! I have a question, and I'm sorry in advance if it's a stupid one. But I work for a company that has a lot of PII on our datalake, if we create a branch, we'd be able to see that info and it would be a problem. Did anybody has this issue and solved it somehow? Thanks in advance
    m
    Ariel Shaqed (Scolnicov)
    +3
    8 replies
    Copy to Clipboard
  • Darwin Schweitzer

    Darwin Schweitzer

    6 months ago
    I have been getting a ERROR 403: Forbidden when trying to run the wget script sudo wget http://treeverse-clients-us-east.s3-website-us-east-1.amazonaws.com/lakectl/0.60.0/darwin_amd64/lakectl -O /usr/local/bin/lakectl I am trying to install the lakectl per the guidance at https://demo.lakefs.io/# What’s Next to use the playground environment. Is the s3 not accessible for some reason?
    Darwin Schweitzer
    Barak Amar
    4 replies
    Copy to Clipboard
  • d

    donald

    5 months ago
    I first started lakefs by running "docker-compose up", then do some initial setup, etc, then run "docker-compose down" to stop lakefs, then run "docker-compose up" to start lakefs again, but all previous setup are lost, I have to re-setup again. Does anyone know how I can solve this?
    d
    Barak Amar
    +1
    11 replies
    Copy to Clipboard