Thread
#lakefs-for-beginners
    w

    Walter Johnson

    1 week ago
    Does anybody know how to skip or mimic the /app/wait-for function in the entrypoint of the docker file? I have a situation where the container keeps exiting because it receives an error at this point.
    Guy Hardonag

    Guy Hardonag

    1 week ago
    Hi @Walter Johnson, can you please provide more information about your situation. the
    wait-for
    function waits until
    postgres
    is up. it runs this script. you could skip it by editing the docker-compose file and removing
    "/app/wait-for", "postgres:5432", "--",
    But you will probably experience some other error.
    w

    Walter Johnson

    1 week ago
    I am using Terraform and my postgres instance is setup before and I am certain it is there before I need lakeFS.
    I have attmepted to remove that and just use the /app/start function but it doesn't like that at all. So am I wondering if there is a way to mimic that output in place of the /app/wait-for postgres:5432 -- command.
    It is blowing up right here ... {wait-for} /bin/sh /app/wait-for localhost:5432 -- /app/lakefs run It starts for a second and then that function times out. I have confirmed that pg is running and able to be connected to on port 5432 with user lakefs and password lakefs, Any ideas?
    Guy Hardonag

    Guy Hardonag

    1 week ago
    How did you confirm that?
    w

    Walter Johnson

    1 week ago
    I have another container running that I am able to connect to my pg instance with.
    from that containers cli I was able to log in.
    Guy Hardonag

    Guy Hardonag

    1 week ago
    can you please send here the connection string you are using to connect?
    Guy Hardonag

    Guy Hardonag

    1 week ago
    what is Opostgres ?
    w

    Walter Johnson

    1 week ago
    Please dont let that be it....😄
    Guy Hardonag

    Guy Hardonag

    1 week ago
    😅
    It is, the wait-for just waits for port 5432 on domain
    postgres
    because lakefs will later try to connect to it
    the wait for fails because there is no open port on that domain
    w

    Walter Johnson

    1 week ago
    My container still shut down.
    Guy Hardonag

    Guy Hardonag

    1 week ago
    If you’d like to use 0postgres, you should edit the run command and the properies as well
    entrypoint: ["/app/wait-for", "postgres:5432", "--", "/app/lakefs", "run"] -> entrypoint: ["/app/wait-for", "0postgres:5432", "--", "/app/lakefs", "run"]
    
    - LAKEFS_DATABASE_CONNECTION_STRING=<postgres://lakefs:lakefs@postgres/postgres?sslmode=disable> -> - LAKEFS_DATABASE_CONNECTION_STRING=<postgres://lakefs:lakefs@0postgres/call-center?sslmode=disable>
    w

    Walter Johnson

    1 week ago
    nc -z postgres 5432 returns nc: getaddrinfo: Try again
    The container starts and I can go in there and run the same function that wait for is running and that is the result.
    nc -z localhost 5432 just returns to the next line with no output
    Guy Hardonag

    Guy Hardonag

    1 week ago
    Sorry but I am not following, which container ?
    w

    Walter Johnson

    1 week ago
    The lakeFS container.
    The operation times out.
    Guy Hardonag

    Guy Hardonag

    1 week ago
    Can you please share your docker-compose file ?
    From the lakefs container you could run wait-for directly
    app/wait-for postgres:5432
    w

    Walter Johnson

    1 week ago
    I am using Terraform not docker compose, it is very similar though.. It just pulls the latest image from treeverse/lakefs and run docker run on that. Here is the relevant code.
    resource "docker_container" "lakeFS" {
      image = docker_image.lakeFS.name
      name  = "LakeFS"
      env = [
        "LAKEFS_AUTH_ENCRYPT_SECRET_KEY=secret",
        "LAKEFS_DATABASE_TYPE:postgres",
        "LAKEFS_DATABASE_POSTGRES_CONNECTION_STRING=<postgres://lakefs:lakefs@postgres:5432/call-center?sslmode=disable>",
        "LAKEFS_BLOCKSTORE_TYPE=local",
        "LAKEFS_BLOCKSTORE_LOCAL_PATH=/home/lakefs",
        "LAKEFS_GATEWAYS_S3_DOMAIN_NAME=<http://s3.local.lakefs.io:8000|s3.local.lakefs.io:8000>",
        "LAKEFS_LOGGING_LEVEL=INFO",
      "LAKEFS_COMMITTED_LOCAL_CACHE_DIR=/home/lakefs/.local_tier"]
      ports {
        internal = 8000
        external = 8000
      }
      hostname = "localhost"
    
      logs     = true
      must_run = true
      attach = false
    
      entrypoint = ["/app/wait-for", "postgres:5432", "--", "/app/lakefs", "run"]
     
    
    }
    wait for times out.
    Guy Hardonag

    Guy Hardonag

    1 week ago
    try
    /app/wait-for 0postgres:5432
    w

    Walter Johnson

    1 week ago
    I switched that to postgres. It was a type.
    typo
    Guy Hardonag

    Guy Hardonag

    1 week ago
    Oh ok
    If it times out it means that there is a connectivity problem
    I assume there is a problem with the domain name
    w

    Walter Johnson

    1 week ago
    I have to assume lakefs cant make any outside connections because I have verified that I can remotely connect to my pg instance.
    How could I test that?
    Guy Hardonag

    Guy Hardonag

    1 week ago
    That’s a good question, lakeFS is quite limited,
    w

    Walter Johnson

    1 week ago
    That is why I am wondering if there is amy way to mimic a positve response from /app/wait-for in the entrypoint
    Guy Hardonag

    Guy Hardonag

    1 week ago
    you could run
    nc  postgres 5432
    If it returns it didn’t connect
    w

    Walter Johnson

    1 week ago
    So could I just return a 1 or something at that point, I tried miserably to attempt something like that but I am not the best bash coder.
    Guy Hardonag

    Guy Hardonag

    1 week ago
    I would start by running a different image ( instead of lakeFS ) and trying to connect with that image
    Or is that the case that is already working for you?
    you could us nc with v flag for more information
    ~ $ nc -zv postgres 5431
    nc: connect to postgres port 5431 (tcp) failed: Connection refused
    
    ~ $ nc -zv postgres 5432
    Connection to postgres 5432 port [tcp/postgresql] succeeded!
    ~ $
    w

    Walter Johnson

    1 week ago
    nc: getaddrinfo: Try again
    Guy Hardonag

    Guy Hardonag

    1 week ago
    Seems like its failing to get the address
    I think you should try running the container with a different image that has a postgres client and stronger debugging capabilities
    w

    Walter Johnson

    1 week ago
    ~ $ nc -zv postgres 5432 nc: getaddrinfo: Try again ~ $ nc -zv localhost 5432 nc: connect to localhost port 5432 (tcp) failed: Connection refused nc: connect to localhost port 5432 (tcp) failed: Connection refused nc: connect to localhost port 5432 (tcp) failed: Address not available ~ $ nc -zv 0.0.0.0 5432 nc: connect to 0.0.0.0 port 5432 (tcp) failed: Connection refused
    My problem is right there in front of me.
    Is there no way to skip /app-wait-for?
    Guy Hardonag

    Guy Hardonag

    1 week ago
    You could skip it
    but than lakeFS will fail to connect
    wait-for is just there to protect you from running before postgres is available from the lakeFS machine.
    IIUC you already skipped it and got a different error
    w

    Walter Johnson

    1 week ago
    What if I already have Postgres running and I am certain of it?
    Guy Hardonag

    Guy Hardonag

    1 week ago
    I understand, I am also certain of that. But lakeFS doesn’t manage to connect to it for some reason
    w

    Walter Johnson

    1 week ago
    When the instance starts up is it impossible to skip or mimic the outcome wait-for?
    It is because that nc function fails. What does lakefs actually use to connect to the pg instance?
    psql?
    Guy Hardonag

    Guy Hardonag

    1 week ago
    OK, I might of got you wrong at the start, thought you already tried without wait-for and it failed. try changing
    entrypoint = ["/app/wait-for", "postgres:5432", "--", "/app/lakefs", "run"]
    to
    entrypoint = ["/app/lakefs", "run"]
    w

    Walter Johnson

    1 week ago
    I did but will try it again just for my own sanity.
    Guy Hardonag

    Guy Hardonag

    1 week ago
    What happened when you did it?
    Because that is the way to skip
    wait-for
    By the way, what machine are you running your docker on?
    w

    Walter Johnson

    1 week ago
    The container exits immediately and I cant even ssh into for a few seconds.
    I am running on windows in a WSL2 Ubuntu
    But all my containers are running on windows
    Guy Hardonag

    Guy Hardonag

    1 week ago
    So lakeFS probably fails when trying to connect to the database ( nothing to do with
    wait-for
    now )
    w

    Walter Johnson

    1 week ago
    So this is basically some port issue. I can start lakeFS fine with docker compose.
    Guy Hardonag

    Guy Hardonag

    1 week ago
    Port or domain resolving
    w

    Walter Johnson

    1 week ago
    can app run be more verbose?
    Guy Hardonag

    Guy Hardonag

    1 week ago
    Are you seeing any errors?
    or logs
    Don’t think the verbosity is the problem.
    w

    Walter Johnson

    1 week ago
    I am not seeing errors. I start the instance that download and look at the nc function in there and I need to get my hostname to resolve to postgres. That is what will make lakeFS happy and I want to make lakeFS happy. 😁 Time to go further down the rabbit.
    Guy Hardonag

    Guy Hardonag

    1 week ago
    If you can provide here the configurations you are using to run both lakeFS and postgres I will try to run it on my end and see if I can reproduce it.
    Otherwise we can set a meeting tomorrow and try to debug it on your end.
    w

    Walter Johnson

    1 week ago
    All of your help was super enlightening. I think I will track down the issue shortly.
    I will definitely reconvene with you and let you know my progress. Are you familiar with Terraform?
    Guy Hardonag

    Guy Hardonag

    1 week ago
    OK, I am familiar a bit with terraform. Waitingto hear about your progress. Good luck
    w

    Walter Johnson

    1 week ago
    I am off to bed a sad little man. I did not solve my issue but I did get one more piece of valuable knowledge. I was able to get into the lakeFS server and run nc -zv {IP-OF-MY-PG-INSTANCE} 5432 and it successfully connected. So I put that in the wait-for function and the container exits with errors but that makes me believe there is a problem with the connection string but I can't seem t locate it yet. Good night.
    "LAKEFS_DATABASE_POSTGRES_CONNECTION_STRING=<postgresql://lakefs:lakefs@172.17.0.3:5432/postgres?sslmode=disable>"
    Amit Kesarwani

    Amit Kesarwani

    1 week ago
    @Guy Hardonag @Walter Johnson Should variable name be
    LAKEFS_DATABASE_CONNECTION_STRING
    instead of
    LAKEFS_DATABASE_POSTGRES_CONNECTION_STRING
    (no POSTGRES in name) unless using version 0.80?
    Itai Admi

    Itai Admi

    1 week ago
    That’s right - for new installations with version >=v0.80.0 , the env var containing
    POSTGRES
    should be used
    w

    Walter Johnson

    1 week ago
    I have gotten into the container for a few seconds by making /app/wait-for hang. While in there I ran /app/lakefs run and I got this response ->
    Failed to open KV store                       error="unknown driver: "
    Guy Hardonag

    Guy Hardonag

    1 week ago
    Hi @Walter Johnson, Nice progress. Try adding
    LAKEFS_DATABASE_TYPE=postgres
    above
    LAKEFS_DATABASE_POSTGRES_CONNECTION_STRING
    w

    Walter Johnson

    1 week ago
    One step closer to nirvana
    FATAL  [2022-09-15T19:49:12Z]cmd/run.go:123 cmd/lakefs/cmd.glob..func8 Failed to open KV store                       error="connect failed: failed to connect to `host=/tmp user=lakefs database=`: dial error (dial unix /tmp/.s.PGSQL.5432: connect: no such file or directory)"
    SUCCESS!
    It was a combination of typos and configuration errors.
    You get 3 or four of those put together and its a nightmare.
    But through out it all I gained a little better understanding of lakeFS
    Guy Hardonag

    Guy Hardonag

    1 week ago
    Happy to hear 😄 😄 Hope it goes smooth from now on. We are here if you need any help!
    w

    Walter Johnson

    1 week ago
    This inspired me to create my first npm package https://www.npmjs.com/package/lakefs
    Guy Hardonag

    Guy Hardonag

    1 week ago
    Cool 😒unglasses_lakefs: Will check it out Thanks!