Clinton Monk
07/19/2022, 9:24 PMVino
07/25/2022, 7:57 AMfailed to create storage repository: failed to access storage
error. What am I missing?Sid Senthilnathan
07/25/2022, 5:44 PMVino
07/26/2022, 10:20 AMKevin Vasko
07/27/2022, 1:30 AMAriel Shaqed (Scolnicov)
07/27/2022, 1:36 AMAriel Shaqed (Scolnicov)
07/27/2022, 1:49 AMKevin Vasko
07/27/2022, 1:53 AMAriel Shaqed (Scolnicov)
07/27/2022, 2:00 AMNuno Carvalho dos Santos
07/28/2022, 8:22 AMNuno Carvalho dos Santos
07/28/2022, 8:51 AMNuno Carvalho dos Santos
07/28/2022, 8:54 AMKevin Vasko
08/02/2022, 4:03 PMVino
08/03/2022, 1:56 AMimport lakefs_client
I run into this error ModuleNotFoundError: No module named 'lakefs_client'
. What am I missing?carlos osuna
08/11/2022, 5:10 PMaws s3 --endpoint-url <https://bucket.s3.my.path.net> ls
It returns the same as base path:
- aws s3 --endpoint-url <https://s3.my.path.net> ls
in that it returns the list of available buckets instead of the contents of the bucket. Is there something wrong with my configuration here? Is the domain_name
field supposed to be something else? Traffic is being sent to the service, but I can't seem to identify why the bucket is not getting resolved. Below is my values.yaml
for Helm installation:
lakefsConfig: |
blockstore:
type: s3
gateways:
s3:
domain_name: <http://s3.my.path.net|s3.my.path.net>
ingress:
enabled: true
ingressClassName: "nginx"
tls:
- hosts:
- <http://my.path.net|my.path.net>
- <http://s3.my.path.net|s3.my.path.net>
- "*.<http://s3.my.path.net|s3.my.path.net>"
secretName: <secret tls>
hosts:
- host: <http://my.path.net|my.path.net>
paths: ["/"]
- host: <http://s3.my.path.net|s3.my.path.net>
paths: ["/"]
- host: "*.<http://s3.my.path.net|s3.my.path.net>"
paths: ["/"]
extraEnvVarsSecret: <secret env>
Thank you!Mike Logaciuk
08/18/2022, 6:36 AMversion: '3.9'
# Settings and configurations that are common for all containers
x-minio-common:
_&_minio-common
image: 'minio/minio:latest'
command: server --console-address ":9001" <http://lakefs-minio>{1...4}/data{1...2}
expose:
- "9000"
- "9001"
environment:
MINIO_ROOT_USER: ${AWS_ACCESS_KEY_ID}
MINIO_ROOT_PASSWORD: ${AWS_SECRET_ACCESS_KEY}
networks:
- lakefs-network
healthcheck:
test:
[
"CMD",
"curl",
"-f",
"<http://localhost:9000/minio/health/live>"
]
interval: 30s
timeout: 20s
retries: 3
# starts 4 docker containers running minio server instances.
# using nginx reverse proxy, load balancing, you can access
# it through port 9000.
services:
lakefs-minio1:
<<: _*_minio-common
hostname: lakefs-minio1
container_name: lakefs-minio1
networks:
- lakefs-network
volumes:
- data1-1:/data1
- data1-2:/data2
lakefs-minio2:
<<: _*_minio-common
hostname: lakefs-minio2
container_name: lakefs-minio2
networks:
- lakefs-network
volumes:
- data2-1:/data1
- data2-2:/data2
lakefs-minio3:
<<: _*_minio-common
hostname: lakefs-minio3
container_name: lakefs-minio3
networks:
- lakefs-network
volumes:
- data3-1:/data1
- data3-2:/data2
lakefs-minio4:
<<: _*_minio-common
hostname: lakefs-minio4
container_name: lakefs-minio4
networks:
- lakefs-network
volumes:
- data4-1:/data1
- data4-2:/data2
lakefs-minio-setup:
image: minio/mc
container_name: lakefs-minio-setup
environment:
- MC_HOST_lakefs=http://${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}@lakefs-minio1:9000
depends_on:
- lakefs-minio1
- lakefs-minio2
- lakefs-minio3
- lakefs-minio4
command: [ "mb", "lakefs/example" ]
networks:
- lakefs-network
lakefs-nginx:
image: nginx:1.19.2-alpine
hostname: lakefs-nginx
container_name: lakefs-nginx
networks:
- lakefs-network
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf:ro
ports:
- "9000:9000"
- "9001:9001"
depends_on:
- lakefs-minio1
- lakefs-minio2
- lakefs-minio3
- lakefs-minio4
lakefs-postgres:
image: "postgres:11"
hostname: lakefs-postgres
container_name: lakefs-postgres
environment:
POSTGRES_USER: ${POSTGRES_USER}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
POSTGRES_DB: ${POSTGRES_DB}
networks:
- lakefs-network
lakefs-setup:
image: treeverse/lakefs:latest
container_name: lakefs-setup
depends_on:
- lakefs-POSTGRES_DB
- lakefs-minio-setup
environment:
- LAKEFS_AUTH_ENCRYPT_SECRET_KEY=${LAKEFS_AUTH_ENCRYPT_SECRET_KEY}
- LAKEFS_DATABASE_CONNECTION_STRING=postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@lakefs-postgres/postgres?sslmode=disable
- LAKECTL_CREDENTIALS_ACCESS_KEY_ID=${LAKECTL_CREDENTIALS_ACCESS_KEY_ID}
- LAKECTL_CREDENTIALS_SECRET_ACCESS_KEY=${LAKECTL_CREDENTIALS_SECRET_ACCESS_KEY}
- LAKECTL_SERVER_ENDPOINT_URL=<http://lakefs:8000>
- LAKEFS_BLOCKSTORE_TYPE=s3
entrypoint: ["/app/wait-for", "postgres:5432", "--", "sh", "-c",
"lakefs setup --user-name docker --access-key-id ${LAKECTL_CREDENTIALS_ACCESS_KEY_ID} --secret-access-key ${LAKECTL_CREDENTIALS_SECRET_ACCESS_KEY} && lakectl repo create <lakefs://example>
<s3://example>"
]
networks:
- lakefs-network
lakefs:
image: "treeverse/lakefs:latest"
container_name: lakefs
ports:
- "8001:8000"
depends_on:
- "lakefs-postgres"
environment:
- LAKEFS_AUTH_ENCRYPT_SECRET_KEY=${LAKEFS_AUTH_ENCRYPT_SECRET_KEY}
- LAKEFS_DATABASE_CONNECTION_STRING=postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@lakefs-postgres/postgres?sslmode=disable
- LAKEFS_BLOCKSTORE_TYPE=s3
- LAKEFS_BLOCKSTORE_LOCAL_PATH=${LAKEFS_BLOCKSTORE_LOCAL_PATH:-/home/lakefs}
- LAKEFS_GATEWAYS_S3_DOMAIN_NAME=${LAKEFS_GATEWAYS_S3_DOMAIN_NAME:-<http://s3.local.lakefs.io:8000|s3.local.lakefs.io:8000>}
- LAKEFS_BLOCKSTORE_S3_CREDENTIALS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
- LAKEFS_BLOCKSTORE_S3_CREDENTIALS_ACCESS_SECRET_KEY=${AWS_SECRET_ACCESS_KEY}
- LAKEFS_LOGGING_LEVEL=DEBUG
- LAKEFS_STATS_ENABLED
- LAKEFS_BLOCKSTORE_S3_ENDPOINT=<http://localhost:9000>
- LAKEFS_BLOCKSTORE_S3_FORCE_PATH_STYLE=true
- LAKEFS_COMMITTED_LOCAL_CACHE_DIR=${LAKEFS_COMMITTED_LOCAL_CACHE_DIR:-/home/lakefs/.local_tier}
entrypoint:
[
"/app/wait-for",
"postgres:5432",
"--",
"/app/lakefs",
"run"
]
volumes:
data1-1:
name: lakefs-minio1-storage-1
data1-2:
name: lakefs-minio1-storage-2
data2-1:
name: lakefs-minio2-storage-1
data2-2:
name: lakefs-minio2-storage-2
data3-1:
name: lakefs-minio3-storage-1
data3-2:
name: lakefs-minio3-storage-2
data4-1:
name: lakefs-minio4-storage-1
data4-2:
name: lakefs-minio4-storage-2
networks:
lakefs-network:
driver: bridge
name: lakefs-network
Damon C
08/18/2022, 7:16 PMgenerate-db-auth-token
API call is made to the RDS service and then the returned value (valid for 15 minutes) is used as a password. Peeking at ConnectDBPool
(and the corresponding pgx library) it seems like I might need to tweak that method to get the token…curious if others have run into this before or have thoughts.
Context: I’m trying to deploy a full stack using AWS CDK and didn’t want to create an environment variable for my ECS task definition with a hard-coded password as it would show up in the CloudFormation template. 😅Clay Buxton
08/22/2022, 3:04 AMFarman Pirzada
08/23/2022, 10:53 PM张飞跃
08/29/2022, 8:32 AMAnandkarthick Krishnakumar
09/06/2022, 4:44 AMsymlink
thanksVino
09/08/2022, 1:58 PMHTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /api/v1/repositories/example/branches (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc728391760>: Failed to establish a new connection: [Errno 111] Connection refused'))
From the jupyter notebook that comes with everything bagel, I'm calling list_branches(), which results in this error. What am I missing? I'm able to access the lakeFS UI though.Hugh Nolan
09/14/2022, 4:02 PMApiTypeError: Invalid type for variable 'received_data'. Required value type is ObjectErrorList and passed type was str at ['received_data']
using the client.objects.delete_objects
call. Despite the error, I can see my files are deleted.
Diving in, it looks like the HTTP response received is an empty string, with a 204 header. This appears to be correct according to the API - https://docs.lakefs.io/reference/api.html#/objects/deleteObjects
I seem to be getting a valid response here so I'm hoping I'm not doing something wrong?
but the autogenerated API only seems to know about the ObjectErrorList which comes with a 200 header:
self.delete_objects_endpoint = _Endpoint(
settings={
'response_type': (ObjectErrorList, ),
'auth': [
'basic_auth',
'cookie_auth',
'jwt_token',
'oidc_auth'
],
'endpoint_path': '/repositories/{repository}/branches/{branch}/objects/delete',
'operation_id': 'delete_objects',
'http_method': 'POST',
'servers': None,
},
Is this a bug (and where should I report if so), or am I doing something wrong?Robin Moffatt
09/14/2022, 8:39 PMCREATE TABLE s3.tiny_v2.orders (
orderkey bigint,
custkey bigint,
orderstatus varchar(1),
totalprice double,
orderdate date,
orderpriority varchar(15),
clerk varchar(15),
shippriority integer,
comment varchar(79)
) WITH (
external_location = '<s3a://example/main/tiny_v2/orders>',
format = 'ORC'
);
My questions so far:
1. If the Trino schema has a location location = '<s3a://example/v2/tiny>'
then why is the external location for the table still under main
(external_location = '<s3a://example/main/tiny_v2/orders>'
) ?
2. What should I see as a result of creating this if it does work? In my head I’m expecting, since it’s a branch off main, to see 15k records, is that right?
trino:> select count(*) from s3.tiny.orders;
_col0
-------
15000
(1 row)
trino:> select count(*) from s3.tiny_v2.orders;
_col0
-------
0
(1 row)
TIA 🙂Guy Hardonag
09/15/2022, 1:26 PMvenkadesan elangovan
09/27/2022, 9:13 AMsetu suyagya
09/29/2022, 12:06 PMSG
09/29/2022, 12:16 PMsetu suyagya
09/29/2022, 2:55 PMBarak Amar