Hi, I need to get Lakefs implement git-like operations on our minio storage However, I'm getting an error after I setup everything (as described in https://lakefs.io/git-like-operations-over-minio-with-lakefs/). The error on the CLI is "request failed: [400 Bad Request] error creating repository: could not access storage namespace" when I try to add a new repository in Lakefs. The same error is also thrown on the GUI.

Hi @Tamilvanan A

Hi Oz, Thank you for the prompt support. It's clear how to point LakeFS to internally hosted MinIO running in a docker container. After testing the initial setup, I need LakeFS to point to another public IP where MinIO is deployed from a docker image. How do I configure LakeFS to point to MinIO hosted on another Public IP?

by changing the

endpoint

setting to the proper url (i.e. http://myminioserver.example.com) please check there’s no firewall or security rule preventi you from communicating with the minio server though

Hi @Oz Katz But, I already tried adding the end point details in this configuration file line by running the below script before running the docker image:

echo "LAKEFS_BLOCKSTORE_S3_ENDPOINT=<minio_endpoint>" >> $LAKEFS_CONFIG_FILE

But, I still end up with the same problem when I run

curl <https://compose.lakefs.io> | docker-compose --env-file $LAKEFS_CONFIG_FILE -f - up -d

Tried this link: https://lakefs.io/git-like-operations-over-minio-with-lakefs/ Is there any issue with the documentation in this link? Are there any documentation on getting LakeFS up and running for a Remote instance of MinIO accessible via Public IP? Do I have to setup networking for docker using the examples in your previous link?

the example there should work. if you could share the log and the configuration file you used, I might be able to better understand the issue

Just a moment, I'm posting the setup details and logs as well

Thanks @Tamilvanan A! What OS and Docker version are you using?

Running on Cent OS 8 Docker version 20.10.6, build 370c289 docker-compose version 1.29.1, build c34c88b2

Thanks.. From the looks of it, it seems like you're running into this: https://stackoverflow.com/a/61424570

yes

this host works automatically in MacOS and Windows, but for some reason, not on Linux

Hi @Oz Katz Have you found out any solution? Should I try the below option when trying docker-compose up?

--add-host=host.docker.internal:host-gateway

i’m not sure this will work with docker-compose. Im looking into adding an option to run minio as part of the docker compose stack: this way you’ll simply do

docker-compose up

and it’ll automatically setup lakefs and minio including the communication between the two.

But, for production deployment, MinIO instance is running in a separate VM. I need LakeFS to connect to the other VM over Public IP / Private VPN IP

Yes, but the problem here occurs when lakeFS is running on the same host as lakeFS. Docker networking is a strange beast.. The current docker-compose should work just fine for remote minIO deployments

But, the current setup doesn't work for both MinIO on localhost and External VM. I'm unable to deploy anything.

Ah, did not realize that. for the remote endpoint, can you connect to that host:port from the machine running lakeFS? using curl or mc?

Hi @Oz Katz Can you help me out with the docker deployment of LakeFS?? I really need help with deploying LakeFS from the docker image. Just to demonstrate working of LakeFS, I'm working on setting up a deployment as mentioned in your production deployment for AWS. But, my management demanding that I deploy the solution using the docker at the earliest.

Hi @Tamilvanan A can help you with using lakeFS using docker. can we schedule a Zoom call or any other way for me to better understand your issues?

Sure. I'm ready any time. Are we going to have a call now or are we going to schedule it for later?

2 min

ok. I'm ready

Working with Presto with lakeFS or S3 requires the same. Connecting a metastore as it specified in the Trino documentation - https://trino.io/docs/current/connector/hive.html.

I already got this link. But, for working on it, do I need to install a standalone version of Hive-metastore?

You will need Hive-metastore. How are you using Presto to query MinIO today?

I'm using MinIO, LakeFS and Trino on docker containers

before lakeFS - did you have any Presto setup / connected to your data?

Trino is working out of a docker container. But, it is not connected to LakeFS or MinIO

No

Shall I install Hive metastore as Docker container as well?

It is possible - but this part depends on your requirements - not related to lakeFS.

I prefer to have Hive as a docker container if possible. That is my end customer requirement

You can checkout this blog post - https://lakefs.io/the-quick-guide-for-running-presto-locally-on-s3/

I have seen this link too. So, you are suggesting to use LakeFS details for endpoint?

all I'm saying that if you setup your Presto/Trino to work with any S3 source - pointing the metastore and presto to lakeFS endpoint/credentials should work.

ok

I can get on zoom in an hour. but you can ask, usually it is preferred as others can help too

Sure. I went through the link which says "The Quick Guide for Running Presto Locally on S3" According to the instructions given there, I have cloned the repository and got everything working properly. I'm having trouble with the following query shown below:

CREATE EXTERNAL TABLE amazon_reviews_parquet(
marketplace string,
customer_id string,
review_id string,
product_id string,
product_parent string,
product_title string,
star_rating int,
helpful_votes int,
total_votes int,
vine string,
verified_purchase string,
review_headline string,
review_body string,
review_date int,
year int)
PARTITIONED BY (product_category string)
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION
'<s3a://amazon-reviews-pds/parquet/>';

What location should I use? The error is related to the location. I just want to point to my lakefs instance

This is great - notice that for each configuration file in the post there are sections in comment. example:

etc/catalog/s3.properties

#hive.s3.endpoint=http:[WANTED_ENDPOINT]

The yaml file which I used with docker-compose up for LakeFS is as follows: version: '3' services: lakefs: image: "treeverse/lakefs:${VERSION:-latest}" ports: - "8000:8000" depends_on: - "postgres" environment: - LAKEFS_AUTH_ENCRYPT_SECRET_KEY=${LAKEFS_AUTH_ENCRYPT_SECRET_KEY:-some random secret string} - LAKEFS_DATABASE_CONNECTION_STRING=${LAKEFS_DATABASE_CONNECTION_STRING:-postgres://lakefs:lakefs@postgres/postgres?sslmode=disable} - LAKEFS_BLOCKSTORE_TYPE=${LAKEFS_BLOCKSTORE_TYPE:-local} - LAKEFS_BLOCKSTORE_LOCAL_PATH=${LAKEFS_BLOCKSTORE_LOCAL_PATH:-/home/lakefs} - LAKEFS_GATEWAYS_S3_DOMAIN_NAME=${LAKEFS_GATEWAYS_S3_DOMAIN_NAME:-dinakar-etl.eastus.cloudapp.azure.com:8000} - LAKEFS_BLOCKSTORE_S3_CREDENTIALS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID:-} - LAKEFS_BLOCKSTORE_S3_CREDENTIALS_ACCESS_SECRET_KEY=${AWS_SECRET_ACCESS_KEY:-} - LAKEFS_LOGGING_LEVEL=${LAKEFS_LOGGING_LEVEL:-INFO} - LAKEFS_STATS_ENABLED - LAKEFS_BLOCKSTORE_S3_ENDPOINT - LAKEFS_BLOCKSTORE_S3_FORCE_PATH_STYLE - LAKEFS_CATALOGER_TYPE - LAKEFS_COMMITTED_LOCAL_CACHE_DIR=${LAKEFS_COMMITTED_LOCAL_CACHE_DIR:-/home/lakefs/.local_tier} entrypoint: ["/app/wait-for", "postgres:5432", "--", "/app/lakefs", "run"] postgres: image: "postgres:${PG_VERSION:-11}" command: "-c log_min_messages=FATAL" environment: POSTGRES_USER: lakefs POSTGRES_PASSWORD: lakefs logging: driver: none

Hey @Tamilvanan A What are you experiencing now? When trying to create a table in presto, do you get any error? Do you see any logs in lakeFS?

Last time I noticed that it caused a deadlock with the CREATE TABLE query. It hanged and was not responding

What path did you use in the create table query?

I tried location as s3a://dinakar-etl.eastus.cloudapp.azure.com:8000/dinakar-test-bucket/test I'm sure I did a minor mistake with the END POINT details. For a clean setup, can you help me out over a zoom call? It shouldn't take more than 10 or 15 minutes. Posting the logs and config file contents exposes credentials as well. I also configured the DNS name for LakeFS as @Barak Amar mentioned in our last meeting.

1 min

ok

sent the link on private

Hi @Barak Amar Here's the log you wanted to check out.