Channels
announcements
blockers_for_windward
career-opportunities
celebrations
cuddle-corner
data-discussion
data-events
dev
events
general
help
iceberg-integration
lakefs-for-beginners
lakefs-hubspot-cloud-registration-email-automation
lakefs-releases
lakefs-suggestions
lakefs-twitter
linen-dev
memes-and-banter
new-channel
new-channel
say-hello
stackoverflow
test
Title
d
dylan butler
10/27/2022, 4:27 AMhello, for a dynamodb deployment do you need to specify a specific hash key? I do not see anything regarding that in the documentation
i
Idan Novogroder
10/27/2022, 5:27 AMHi Dylan,
Welcome to the lake!
I'm not sure I fully understand your question.
Do you mean for the key schema your table will use?
And do you mean to ask if you need to specify it in lakeFS configuration or in your DynamoDB table configuration when you create it?
d
dylan butler
10/27/2022, 3:00 PMthanks for replying, and yes when building a dynamodb to use for lakeFS here:https://docs.lakefs.io/deploy/aws.html the hash or partition key is a required attribute
i
Idan Novogroder
10/27/2022, 3:34 PMHi Dylen,
I'm not sure I understand where do you see any of those keys.
Maybe you mean to the encryption key? It's needed for the auth (you can see it's under the auth section)
For the DynamoDB you only need to mention the table name and region
Please let me know if it was helpful or you still missing something!
d
dylan butler
10/27/2022, 5:07 PMhttps://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/dynamodb_table required_arguments: hash_key, Attribute to use as the hash (partition) key. Must also be defined as an
attributei
Idan Novogroder
10/27/2022, 6:00 PMHo,
I see the confusion here.
You don't have to create a table for lakeFS, It's already created as part of the lakeFS setup.
Maybe we can make sure this point is clearer on our docs.
d
dylan butler
10/27/2022, 6:06 PMThank you for clearing my confusion, so if I want to run this on ecs, I would run the treeverse/lakefs:latest imagine and that builds the dynamodb table?
i
Idan Novogroder
10/27/2022, 6:07 PMYep 🙂
d
dylan butler
10/27/2022, 6:09 PMI appreciate your help! I will go through and try that out
i
Idan Novogroder
10/27/2022, 6:10 PMSure thing, feel free to reach out
d
Dylan Butler
10/29/2022, 2:32 AMso I tried pulling down the docker image here https://hub.docker.com/r/treeverse/lakefs and got it to serve the page with the postgres backend
I tried downloading that file from the compose link and swapping the vars to match what I think the dynamodb settings want and its not working. Are there any other examples beyond this: https://docs.lakefs.io/deploy/aws.html? I tried searching in the git repo and couldnt find any
i
Idan Novogroder
10/29/2022, 2:43 AMWhat's the error message you get?
d
Dylan Butler
10/29/2022, 2:46 AMlakefs | time="2022-10-29T02:44:28Z" level=fatal msg="Failed to open KV store" func=cmd/lakefs/cmd.glob..func6 file="cmd/run.go:117" error="missing dynamodb settings: driver configuration"
lakefs exited with code 1
i
Idan Novogroder
10/29/2022, 2:50 AMI'm not sure you need the dynamodb section in your compose file.
Can you try to remove it and see if it helps?
d
Dylan Butler
10/29/2022, 2:51 AMsure
removed the dynamo section and the depends on, it just builds the lakefs container and fails
[+] Running 9/9â ¿ lakefs Pulled 7.5sâ ¿ 59bf1c3509f3 Pull complete 1.1sâ ¿ 931e5fa88a0b Pull complete 1.2sâ ¿ 1eccfaca96c8 Pull complete 1.2sâ ¿ c2713e584ff0 Pull complete 1.3sâ ¿ e2e76a08fb81 Pull complete 1.5sâ ¿ 59f9dc3b9d21 Pull complete 5.3sâ ¿ 853a87f17da4 Pull complete 5.4sâ ¿ 4f4fb700ef54 Pull complete 5.4s[+] Running 2/2â ¿ Network lakefs_default Created 0.0sâ ¿ Container lakefs Created 0.1sAttaching to lakefslakefs | Operation timed outlakefs exited with code 1i
Idan Novogroder
10/29/2022, 2:57 AMI think you should also remove it from the entrypoint params
d
Dylan Butler
10/29/2022, 2:59 AMI removed just the "dynamodb:8000", and it says lakefs | Error: you need to provide a host and port to test.
lakefs | Usage:
lakefs | host:port [-t timeout] [-- command args]
lakefs | -q | --quiet Do not output any status messages
lakefs | -t TIMEOUT | --timeout=timeout Timeout in seconds, zero for no timeout
lakefs | -- COMMAND ARGS Execute command with args after the test finishes
lakefs exited with code 2
i
Idan Novogroder
10/29/2022, 3:56 AMI'll try to look more into this, but I believe something like this should do the job:
version: '3'
services:
lakefs:
image: "${REPO:-treeverse}/lakefs:${TAG:-latest}"
command: "${COMMAND:-run}"
ports:
- "8000:8000"
depends_on:
- "dynamodb"
environment:
- LAKEFS_AUTH_ENCRYPT_SECRET_KEY=some random secret string
- LAKEFS_BLOCKSTORE_TYPE=${LAKEFS_BLOCKSTORE_TYPE:-local}
- LAKEFS_BLOCKSTORE_LOCAL_PATH=/home/lakefs
- LAKEFS_LOGGING_LEVEL=DEBUG
- LAKEFS_DATABASE_DYNAMODB_ENDPOINT=<http://dynamodb:8000>
- LAKEFS_DATABASE_TYPE=dynamodb
- LAKEFS_DATABASE_KV_ENABLED=true
- LAKEFS_DATABASE_DYNAMODB_AWS_REGION=us-east-1
- LAKEFS_DATABASE_DYNAMODB_AWS_ACCESS_KEY_ID=AKIAIO5FODNN7EXAMPLE
- LAKEFS_DATABASE_DYNAMODB_AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K3MDENG/bPxRfiCYEXAMPLEKEY
entrypoint:
[
"/app/wait-for",
"dynamodb:8000",
"--",
"/app/lakefs",
"run"
]
dynamodb:
image: amazon/dynamodb-local
container_name: my-dynamodb
hostname: dynamodb
restart: always
volumes:
- ./my-dynamodb-data:/home/dynamodblocal/data
ports:
- 6432:8000
command: "-jar DynamoDBLocal.jar -sharedDb -dbPath /home/dynamodblocal/data/"d
Dylan Butler
10/29/2022, 3:58 AMthat worked, you're so helpful thank you so much. I will look through this to figure out what I need to do
i
Idan Novogroder
10/29/2022, 3:59 AMSure 🙂