Title
#lakefs-for-beginners
d

dylan butler

10/27/2022, 4:27 AM
hello, for a dynamodb deployment do you need to specify a specific hash key? I do not see anything regarding that in the documentation
Idan Novogroder

Idan Novogroder

10/27/2022, 5:27 AM
Hi Dylan, Welcome to the lake! I'm not sure I fully understand your question. Do you mean for the key schema your table will use? And do you mean to ask if you need to specify it in lakeFS configuration or in your DynamoDB table configuration when you create it?
d

dylan butler

10/27/2022, 3:00 PM
thanks for replying, and yes when building a dynamodb to use for lakeFS here:https://docs.lakefs.io/deploy/aws.html the hash or partition key is a required attribute
Idan Novogroder

Idan Novogroder

10/27/2022, 3:34 PM
Hi Dylen, I'm not sure I understand where do you see any of those keys. Maybe you mean to the encryption key? It's needed for the auth (you can see it's under the auth section) For the DynamoDB you only need to mention the table name and region
5:04 PM
Please let me know if it was helpful or you still missing something!
d

dylan butler

10/27/2022, 5:07 PM
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/dynamodb_table required_arguments: hash_key, Attribute to use as the hash (partition) key. Must also be defined as an
attribute
. See below. but im wondering if you can just use anything there
Idan Novogroder

Idan Novogroder

10/27/2022, 6:00 PM
Ho, I see the confusion here. You don't have to create a table for lakeFS, It's already created as part of the lakeFS setup. Maybe we can make sure this point is clearer on our docs.
d

dylan butler

10/27/2022, 6:06 PM
Thank you for clearing my confusion, so if I want to run this on ecs, I would run the treeverse/lakefs:latest imagine and that builds the dynamodb table?
Idan Novogroder

Idan Novogroder

10/27/2022, 6:07 PM
Yep 🙂
d

dylan butler

10/27/2022, 6:09 PM
I appreciate your help! I will go through and try that out
Idan Novogroder

Idan Novogroder

10/27/2022, 6:10 PM
Sure thing, feel free to reach out
d

Dylan Butler

10/29/2022, 2:32 AM
so I tried pulling down the docker image here https://hub.docker.com/r/treeverse/lakefs and got it to serve the page with the postgres backend I tried downloading that file from the compose link and swapping the vars to match what I think the dynamodb settings want and its not working. Are there any other examples beyond this: https://docs.lakefs.io/deploy/aws.html? I tried searching in the git repo and couldnt find any
Idan Novogroder

Idan Novogroder

10/29/2022, 2:43 AM
What's the error message you get?
d

Dylan Butler

10/29/2022, 2:46 AM
lakefs | time="2022-10-29T02:44:28Z" level=fatal msg="Failed to open KV store" func=cmd/lakefs/cmd.glob..func6 file="cmd/run.go:117" error="missing dynamodb settings: driver configuration" lakefs exited with code 1
Idan Novogroder

Idan Novogroder

10/29/2022, 2:50 AM
I'm not sure you need the dynamodb section in your compose file. Can you try to remove it and see if it helps?
d

Dylan Butler

10/29/2022, 2:51 AM
sure
2:54 AM
removed the dynamo section and the depends on, it just builds the lakefs container and fails
[+] Running 9/9
⠿ lakefs Pulled                                                                                                                      7.5s
⠿ 59bf1c3509f3 Pull complete                                                                                                              1.1s
⠿ 931e5fa88a0b Pull complete                                                                                                              1.2s
⠿ 1eccfaca96c8 Pull complete                                                                                                              1.2s
⠿ c2713e584ff0 Pull complete                                                                                                              1.3s
⠿ e2e76a08fb81 Pull complete                                                                                                              1.5s
⠿ 59f9dc3b9d21 Pull complete                                                                                                              5.3s
⠿ 853a87f17da4 Pull complete                                                                                                              5.4s
⠿ 4f4fb700ef54 Pull complete                                                                                                              5.4s
[+] Running 2/2
⠿ Network lakefs_default Created                                                                                                             0.0s
⠿ Container lakefs    Created                                                                                                             0.1s
Attaching to lakefs
lakefs | Operation timed out
lakefs exited with code 1
Idan Novogroder

Idan Novogroder

10/29/2022, 2:57 AM
I think you should also remove it from the entrypoint params
d

Dylan Butler

10/29/2022, 2:59 AM
I removed just the "dynamodb:8000", and it says lakefs | Error: you need to provide a host and port to test. lakefs | Usage: lakefs | host😛ort [-t timeout] [-- command args] lakefs | -q | --quiet Do not output any status messages lakefs | -t TIMEOUT | --timeout=timeout Timeout in seconds, zero for no timeout lakefs | -- COMMAND ARGS Execute command with args after the test finishes lakefs exited with code 2
Idan Novogroder

Idan Novogroder

10/29/2022, 3:56 AM
I'll try to look more into this, but I believe something like this should do the job:
version: '3'
services:
  lakefs:
    image: "${REPO:-treeverse}/lakefs:${TAG:-latest}"
    command: "${COMMAND:-run}"
    ports:
      - "8000:8000"
    depends_on:
      - "dynamodb"
    environment:
      - LAKEFS_AUTH_ENCRYPT_SECRET_KEY=some random secret string
      - LAKEFS_BLOCKSTORE_TYPE=${LAKEFS_BLOCKSTORE_TYPE:-local}
      - LAKEFS_BLOCKSTORE_LOCAL_PATH=/home/lakefs
      - LAKEFS_LOGGING_LEVEL=DEBUG
      - LAKEFS_DATABASE_DYNAMODB_ENDPOINT=<http://dynamodb:8000>
      - LAKEFS_DATABASE_TYPE=dynamodb
      - LAKEFS_DATABASE_KV_ENABLED=true
      - LAKEFS_DATABASE_DYNAMODB_AWS_REGION=us-east-1
      - LAKEFS_DATABASE_DYNAMODB_AWS_ACCESS_KEY_ID=AKIAIO5FODNN7EXAMPLE
      - LAKEFS_DATABASE_DYNAMODB_AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K3MDENG/bPxRfiCYEXAMPLEKEY
    entrypoint:
      [
        "/app/wait-for",
        "dynamodb:8000",
        "--",
        "/app/lakefs",
        "run"
      ]
  dynamodb:
    image: amazon/dynamodb-local
    container_name: my-dynamodb
    hostname: dynamodb
    restart: always
    volumes:
      - ./my-dynamodb-data:/home/dynamodblocal/data
    ports:
      - 6432:8000
    command: "-jar DynamoDBLocal.jar -sharedDb -dbPath /home/dynamodblocal/data/"
d

Dylan Butler

10/29/2022, 3:58 AM
that worked, you're so helpful thank you so much. I will look through this to figure out what I need to do
Idan Novogroder

Idan Novogroder

10/29/2022, 3:59 AM
Sure 🙂