Hi all, I'm pretty new to lakefs and I've been trying to get it going locally with Minio before we try it out on the prod database. I have the following

docker-compose.yml

file:

version: '3.8'
services:

  lakefs:
    image: treeverse/lakefs:latest
    container_name: lakefs
    ports:
      - 8000:8000
    environment:
      - LAKEFS_DATABASE_TYPE=local
      - LAKEFS_BLOCKSTORE_TYPE=s3
      - LAKEFS_BLOCKSTORE_S3_FORCE_PATH_STYLE=true
      - LAKEFS_BLOCKSTORE_S3_ENDPOINT=<http://minio:9000>
      - LAKEFS_BLOCKSTORE_S3_CREDENTIALS_ACCESS_KEY_ID=${MINIO_USER}
      - LAKEFS_BLOCKSTORE_S3_CREDENTIALS_SECRET_ACCESS_KEY=${MINIO_PASSWORD}
      - LAKEFS_AUTH_ENCRYPT_SECRET_KEY=some random secret string
      - LAKECTL_SERVER_ENDPOINT_URL=<http://localhost:8000>
    depends_on:
      - minio
    networks:
      - dbnetwork


  minio:
    image: minio/minio
    container_name: minio
    ports:
      - "9000:9000"
      - "9001:9001"
    volumes:
      - ./data/s3data:/mnt/data
    environment:
      MINIO_ROOT_USER: ${MINIO_USER}
      MINIO_ROOT_PASSWORD: ${MINIO_PASSWORD}
      MINIO_VOLUMES: "/mnt/data"
    command: ['server', '/mnt/data', '--console-address', ":9001"]
    networks:
      - dbnetwork

  createbuckets:
    image: minio/mc
    depends_on:
      - minio
    volumes:
     - ./data/:/data
    entrypoint: >
      /bin/sh -c "
      /usr/bin/mc alias set srv <https://minio:9000> \${MINIO_USER} \${MINIO_PASSWORD};
      /usr/bin/mc mb srv/app;
      /usr/bin/mc anonymous set public srv/app;
      exit 0;
      "
    networks:
      - dbnetwork

networks:
  dbnetwork:
    name: dbnetwork
    driver: bridge

Now, this works fine with running and setting up lakefs, and from the

lakectl

tool I can create a branch, ingest my data from Minio and then merge it into main. However, when I try and clone the data to my local directory with

lakectl local clone <lakefs://my-repo/main/data> test-lakefs-data

I get the following error:

download _lakefs/dummy failed: Get "<http://minio:9000/example-data/_lakefs/dummy>: dial tcp: lookup minio: no such host

It might be something obvious I am missing, as I said I'm quite new to this. The compose file already existed so may contain things that isn't relevant to this issue. Thanks in advance

The issue is probably that your client is cooking objects by default using presigned URLs. Your host machine probably has no network access to container minio under that name. If I am correct then this is a client-side issue. You have multiple options; I would probably pick whichever one of these sounds most familiar to you: • Tell your local DNS on your machine that minio is localhost, and expose port 9000 on your container to your house. • Run your lakectl client inside a Docker container on the same network. • Use lakectl local on your host, but turn off "presigned URLs"

--pre-sign=false

The last one is probably easiest but the least performant. But if you're not planning a massive local deployment it's probably the first thing to try. Please let us know how you get along!

That last one worked perfectly, thank you! I'll give the others more of a look as I get more involved :)

Great, glad you're not blocked by this. Only to confirm: while your setup is great for kicking the tyres, I can not recommend it for any use in production.

It won't be, still testing things about!

Also forgot to say the important thing... Welcome to the lake!

I'll carry on this thread as it is sort of related, but should my Minio storage update with lakefs, or does it need some 'push' command? I modify a file locally, commit it to my repo and I can see the change, however my Minio database doesn't change. Perhaps my containers are not linked properly?

Hi James, We're on holiday here, back on Thursday, so expect slower responses please. As soon as you uploaded an object, it went into your banking store - so it's on minio. When you committed it you probably got at least 2 more metadata objects. The thing is, the names will not be what you expect. If you're curious, some details here, lakeFS: where's my data . This is not required to use lakeFS, of course.

thanks for that link (and hope you're having a good holiday!). Thats helped my understanding a lot