Title
#lakefs-for-beginners
w

Walter Johnson

09/14/2022, 11:28 PM
Does anybody know how to skip or mimic the /app/wait-for function in the entrypoint of the docker file? I have a situation where the container keeps exiting because it receives an error at this point.
Guy Hardonag

Guy Hardonag

09/14/2022, 11:43 PM
Hi @Walter Johnson, can you please provide more information about your situation. the
wait-for
function waits until
postgres
is up. it runs this script. you could skip it by editing the docker-compose file and removing
"/app/wait-for", "postgres:5432", "--",
But you will probably experience some other error.
w

Walter Johnson

09/14/2022, 11:44 PM
I am using Terraform and my postgres instance is setup before and I am certain it is there before I need lakeFS.
11:47 PM
I have attmepted to remove that and just use the /app/start function but it doesn't like that at all. So am I wondering if there is a way to mimic that output in place of the /app/wait-for postgres:5432 -- command.
12:04 AM
It is blowing up right here ... {wait-for} /bin/sh /app/wait-for localhost:5432 -- /app/lakefs run It starts for a second and then that function times out. I have confirmed that pg is running and able to be connected to on port 5432 with user lakefs and password lakefs, Any ideas?
Guy Hardonag

Guy Hardonag

09/15/2022, 12:05 AM
How did you confirm that?
w

Walter Johnson

09/15/2022, 12:06 AM
I have another container running that I am able to connect to my pg instance with.
12:06 AM
from that containers cli I was able to log in.
Guy Hardonag

Guy Hardonag

09/15/2022, 12:07 AM
can you please send here the connection string you are using to connect?
Guy Hardonag

Guy Hardonag

09/15/2022, 12:07 AM
what is Opostgres ?
w

Walter Johnson

09/15/2022, 12:08 AM
Please dont let that be it....😄
Guy Hardonag

Guy Hardonag

09/15/2022, 12:08 AM
😅
12:09 AM
It is, the wait-for just waits for port 5432 on domain
postgres
12:09 AM
because lakefs will later try to connect to it
12:09 AM
the wait for fails because there is no open port on that domain
w

Walter Johnson

09/15/2022, 12:09 AM
My container still shut down.
Guy Hardonag

Guy Hardonag

09/15/2022, 12:12 AM
If you’d like to use 0postgres, you should edit the run command and the properies as well
entrypoint: ["/app/wait-for", "postgres:5432", "--", "/app/lakefs", "run"] -> entrypoint: ["/app/wait-for", "0postgres:5432", "--", "/app/lakefs", "run"]

- LAKEFS_DATABASE_CONNECTION_STRING=<postgres://lakefs:lakefs@postgres/postgres?sslmode=disable> -> - LAKEFS_DATABASE_CONNECTION_STRING=<postgres://lakefs:lakefs@0postgres/call-center?sslmode=disable>
w

Walter Johnson

09/15/2022, 12:13 AM
nc -z postgres 5432 returns nc: getaddrinfo: Try again
12:14 AM
The container starts and I can go in there and run the same function that wait for is running and that is the result.
12:15 AM
nc -z localhost 5432 just returns to the next line with no output
Guy Hardonag

Guy Hardonag

09/15/2022, 12:16 AM
Sorry but I am not following, which container ?
w

Walter Johnson

09/15/2022, 12:16 AM
The lakeFS container.
12:17 AM
The operation times out.
Guy Hardonag

Guy Hardonag

09/15/2022, 12:18 AM
Can you please share your docker-compose file ?
12:19 AM
From the lakefs container you could run wait-for directly
12:19 AM
app/wait-for postgres:5432
w

Walter Johnson

09/15/2022, 12:22 AM
I am using Terraform not docker compose, it is very similar though.. It just pulls the latest image from treeverse/lakefs and run docker run on that. Here is the relevant code.
resource "docker_container" "lakeFS" {
  image = docker_image.lakeFS.name
  name  = "LakeFS"
  env = [
    "LAKEFS_AUTH_ENCRYPT_SECRET_KEY=secret",
    "LAKEFS_DATABASE_TYPE:postgres",
    "LAKEFS_DATABASE_POSTGRES_CONNECTION_STRING=<postgres://lakefs:lakefs@postgres:5432/call-center?sslmode=disable>",
    "LAKEFS_BLOCKSTORE_TYPE=local",
    "LAKEFS_BLOCKSTORE_LOCAL_PATH=/home/lakefs",
    "LAKEFS_GATEWAYS_S3_DOMAIN_NAME=<http://s3.local.lakefs.io:8000|s3.local.lakefs.io:8000>",
    "LAKEFS_LOGGING_LEVEL=INFO",
  "LAKEFS_COMMITTED_LOCAL_CACHE_DIR=/home/lakefs/.local_tier"]
  ports {
    internal = 8000
    external = 8000
  }
  hostname = "localhost"

  logs     = true
  must_run = true
  attach = false

  entrypoint = ["/app/wait-for", "postgres:5432", "--", "/app/lakefs", "run"]
 

}
12:24 AM
wait for times out.
Guy Hardonag

Guy Hardonag

09/15/2022, 12:24 AM
try
/app/wait-for 0postgres:5432
w

Walter Johnson

09/15/2022, 12:24 AM
I switched that to postgres. It was a type.
12:24 AM
typo
Guy Hardonag

Guy Hardonag

09/15/2022, 12:25 AM
Oh ok
12:26 AM
If it times out it means that there is a connectivity problem
12:26 AM
I assume there is a problem with the domain name
w

Walter Johnson

09/15/2022, 12:29 AM
I have to assume lakefs cant make any outside connections because I have verified that I can remotely connect to my pg instance.
12:29 AM
How could I test that?
Guy Hardonag

Guy Hardonag

09/15/2022, 12:35 AM
That’s a good question, lakeFS is quite limited,
w

Walter Johnson

09/15/2022, 12:36 AM
That is why I am wondering if there is amy way to mimic a positve response from /app/wait-for in the entrypoint
Guy Hardonag

Guy Hardonag

09/15/2022, 12:38 AM
you could run
nc  postgres 5432
12:38 AM
If it returns it didn’t connect
w

Walter Johnson

09/15/2022, 12:39 AM
So could I just return a 1 or something at that point, I tried miserably to attempt something like that but I am not the best bash coder.
Guy Hardonag

Guy Hardonag

09/15/2022, 12:40 AM
I would start by running a different image ( instead of lakeFS ) and trying to connect with that image
12:41 AM
Or is that the case that is already working for you?
12:42 AM
you could us nc with v flag for more information
12:43 AM
~ $ nc -zv postgres 5431
nc: connect to postgres port 5431 (tcp) failed: Connection refused

~ $ nc -zv postgres 5432
Connection to postgres 5432 port [tcp/postgresql] succeeded!
~ $
w

Walter Johnson

09/15/2022, 12:49 AM
nc: getaddrinfo: Try again
Guy Hardonag

Guy Hardonag

09/15/2022, 12:50 AM
Seems like its failing to get the address
12:55 AM
I think you should try running the container with a different image that has a postgres client and stronger debugging capabilities
w

Walter Johnson

09/15/2022, 12:55 AM
~ $ nc -zv postgres 5432 nc: getaddrinfo: Try again ~ $ nc -zv localhost 5432 nc: connect to localhost port 5432 (tcp) failed: Connection refused nc: connect to localhost port 5432 (tcp) failed: Connection refused nc: connect to localhost port 5432 (tcp) failed: Address not available ~ $ nc -zv 0.0.0.0 5432 nc: connect to 0.0.0.0 port 5432 (tcp) failed: Connection refused
12:56 AM
My problem is right there in front of me.
12:58 AM
Is there no way to skip /app-wait-for?
Guy Hardonag

Guy Hardonag

09/15/2022, 12:59 AM
You could skip it
12:59 AM
but than lakeFS will fail to connect
12:59 AM
wait-for is just there to protect you from running before postgres is available from the lakeFS machine.
1:05 AM
IIUC you already skipped it and got a different error
w

Walter Johnson

09/15/2022, 1:05 AM
What if I already have Postgres running and I am certain of it?
Guy Hardonag

Guy Hardonag

09/15/2022, 1:06 AM
I understand, I am also certain of that. But lakeFS doesn’t manage to connect to it for some reason
w

Walter Johnson

09/15/2022, 1:06 AM
When the instance starts up is it impossible to skip or mimic the outcome wait-for?
1:07 AM
It is because that nc function fails. What does lakefs actually use to connect to the pg instance?
1:08 AM
psql?
Guy Hardonag

Guy Hardonag

09/15/2022, 1:09 AM
OK, I might of got you wrong at the start, thought you already tried without wait-for and it failed. try changing
entrypoint = ["/app/wait-for", "postgres:5432", "--", "/app/lakefs", "run"]
to
entrypoint = ["/app/lakefs", "run"]
w

Walter Johnson

09/15/2022, 1:10 AM
I did but will try it again just for my own sanity.
Guy Hardonag

Guy Hardonag

09/15/2022, 1:10 AM
What happened when you did it?
1:10 AM
Because that is the way to skip
wait-for
1:11 AM
By the way, what machine are you running your docker on?
w

Walter Johnson

09/15/2022, 1:11 AM
The container exits immediately and I cant even ssh into for a few seconds.
1:11 AM
I am running on windows in a WSL2 Ubuntu
1:12 AM
But all my containers are running on windows
Guy Hardonag

Guy Hardonag

09/15/2022, 1:13 AM
So lakeFS probably fails when trying to connect to the database ( nothing to do with
wait-for
now )
w

Walter Johnson

09/15/2022, 1:14 AM
So this is basically some port issue. I can start lakeFS fine with docker compose.
Guy Hardonag

Guy Hardonag

09/15/2022, 1:15 AM
Port or domain resolving
w

Walter Johnson

09/15/2022, 1:16 AM
can app run be more verbose?
Guy Hardonag

Guy Hardonag

09/15/2022, 1:19 AM
Are you seeing any errors?
1:19 AM
or logs
1:21 AM
Don’t think the verbosity is the problem.
w

Walter Johnson

09/15/2022, 1:22 AM
I am not seeing errors. I start the instance that download and look at the nc function in there and I need to get my hostname to resolve to postgres. That is what will make lakeFS happy and I want to make lakeFS happy. 😁 Time to go further down the rabbit.
Guy Hardonag

Guy Hardonag

09/15/2022, 1:23 AM
If you can provide here the configurations you are using to run both lakeFS and postgres I will try to run it on my end and see if I can reproduce it.
1:26 AM
Otherwise we can set a meeting tomorrow and try to debug it on your end.
w

Walter Johnson

09/15/2022, 1:27 AM
All of your help was super enlightening. I think I will track down the issue shortly.
1:27 AM
I will definitely reconvene with you and let you know my progress. Are you familiar with Terraform?
Guy Hardonag

Guy Hardonag

09/15/2022, 1:30 AM
OK, I am familiar a bit with terraform. Waitingto hear about your progress. Good luck
w

Walter Johnson

09/15/2022, 2:57 AM
I am off to bed a sad little man. I did not solve my issue but I did get one more piece of valuable knowledge. I was able to get into the lakeFS server and run nc -zv {IP-OF-MY-PG-INSTANCE} 5432 and it successfully connected. So I put that in the wait-for function and the container exits with errors but that makes me believe there is a problem with the connection string but I can't seem t locate it yet. Good night.
"LAKEFS_DATABASE_POSTGRES_CONNECTION_STRING=<postgresql://lakefs:lakefs@172.17.0.3:5432/postgres?sslmode=disable>"
Amit Kesarwani

Amit Kesarwani

09/15/2022, 3:47 PM
@Guy Hardonag @Walter Johnson Should variable name be
LAKEFS_DATABASE_CONNECTION_STRING
instead of
LAKEFS_DATABASE_POSTGRES_CONNECTION_STRING
(no POSTGRES in name) unless using version 0.80?
Itai Admi

Itai Admi

09/15/2022, 4:00 PM
That’s right - for new installations with version >=v0.80.0 , the env var containing
POSTGRES
should be used
w

Walter Johnson

09/15/2022, 7:29 PM
I have gotten into the container for a few seconds by making /app/wait-for hang. While in there I ran /app/lakefs run and I got this response ->
Failed to open KV store                       error="unknown driver: "
Guy Hardonag

Guy Hardonag

09/15/2022, 7:35 PM
Hi @Walter Johnson, Nice progress. Try adding
LAKEFS_DATABASE_TYPE=postgres
above
LAKEFS_DATABASE_POSTGRES_CONNECTION_STRING
w

Walter Johnson

09/15/2022, 7:49 PM
One step closer to nirvana
FATAL  [2022-09-15T19:49:12Z]cmd/run.go:123 cmd/lakefs/cmd.glob..func8 Failed to open KV store                       error="connect failed: failed to connect to `host=/tmp user=lakefs database=`: dial error (dial unix /tmp/.s.PGSQL.5432: connect: no such file or directory)"
7:52 PM
SUCCESS!
7:52 PM
It was a combination of typos and configuration errors.
7:53 PM
You get 3 or four of those put together and its a nightmare.
7:54 PM
But through out it all I gained a little better understanding of lakeFS
Guy Hardonag

Guy Hardonag

09/15/2022, 7:55 PM
Happy to hear 😄 😄 Hope it goes smooth from now on. We are here if you need any help!
w

Walter Johnson

09/15/2022, 8:46 PM
This inspired me to create my first npm package https://www.npmjs.com/package/lakefs
Guy Hardonag

Guy Hardonag

09/15/2022, 8:57 PM
Cool 😒unglasses_lakefs: Will check it out Thanks!