I am trying for multi container Bagel. but all containers are not running, Hive, Jupiter notebook etc

Hey @Ashwath, let me try and help. Can you share the output logs from the

docker-compose

command?

docker-compose logs

@Itai Admi I executed above in cmd and got below output. let me know is this you expected. Hive says no entrypoint.sh found.

Thanks, looking..

I am sorry, first time i generated log not sure what actually looking for 🙂 . Here is the services highlighted are not running. Thank you

Thanks! The

hive

container fails to run. It looks like the

hive

container was built but is missing the

entrypoint.sh

file. Can you try running

docker-compose build

and share the logs please?

here is the log minio uses an image, skipping minio-setup uses an image, skipping postgres uses an image, skipping lakefs uses an image, skipping lakefs-setup uses an image, skipping mariadb uses an image, skipping trino uses an image, skipping spark uses an image, skipping spark-worker uses an image, skipping spark-thrift uses an image, skipping Building hive-metastore [+] Building 5.1s (15/15) FINISHED => [internal] load build definition from Dockerfile 0.1s => => transferring dockerfile: 1.36kB 0.0s => [internal] load .dockerignore 0.1s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/library/openjdk:8u242-jre 4.7s => [ 1/10] FROM docker.io/library/openjdk:8u242-jre@sha256:f437249498cd370546035d6dc73e03a00ec637bcf47b740e60283 0.0s => [internal] load build context 0.1s => => transferring context: 243B 0.0s => CACHED [ 2/10] WORKDIR /opt 0.0s => CACHED [ 3/10] RUN mkdir /opt/hadoop /opt/apache-hive-bin 0.0s => CACHED [ 4/10] RUN curl -sSL https://www.apache.org/dist/hive/hive-3.1.2/apache-hive-3.1.2-bin.tar.gz | tar - 0.0s => CACHED [ 5/10] RUN curl -sSL https://archive.apache.org/dist/hadoop/common/hadoop-3.2.0/hadoop-3.2.0.tar.gz | 0.0s => CACHED [ 6/10] RUN curl -sSL https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-8.0.19.tar. 0.0s => CACHED [ 7/10] RUN ln -s /opt/hadoop/share/hadoop/tools/lib/aws-java-sdk-bundle-1.11.375.jar /opt/apache-hive 0.0s => CACHED [ 8/10] RUN ln -s /opt/hadoop/share/hadoop/tools/lib/hadoop-aws-3.2.0.jar /opt/apache-hive-bin/lib/ 0.0s => CACHED [ 9/10] RUN apt-get update -y && apt-get install -y netcat procps wait-for-it 0.0s => CACHED [10/10] COPY entrypoint.sh / 0.0s => exporting to image 0.1s => => exporting layers 0.0s => => writing image sha256:0179f2cd672e574060c17166455ca3c977cc4a9a9e0a1f3cb5a3969007a4cbe1 0.0s => => naming to docker.io/library/compose_hive-metastore 0.0s Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them Building hive-server [+] Building 1.4s (15/15) FINISHED => [internal] load build definition from Dockerfile 0.1s => => transferring dockerfile: 32B 0.0s => [internal] load .dockerignore 0.3s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/library/openjdk:8u242-jre 0.9s => [internal] load build context 0.0s => => transferring context: 35B 0.0s => [ 1/10] FROM docker.io/library/openjdk:8u242-jre@sha256:f437249498cd370546035d6dc73e03a00ec637bcf47b740e60283 0.0s => CACHED [ 2/10] WORKDIR /opt 0.0s => CACHED [ 3/10] RUN mkdir /opt/hadoop /opt/apache-hive-bin 0.0s => CACHED [ 4/10] RUN curl -sSL https://www.apache.org/dist/hive/hive-3.1.2/apache-hive-3.1.2-bin.tar.gz | tar - 0.0s => CACHED [ 5/10] RUN curl -sSL https://archive.apache.org/dist/hadoop/common/hadoop-3.2.0/hadoop-3.2.0.tar.gz | 0.0s => CACHED [ 6/10] RUN curl -sSL https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-8.0.19.tar. 0.0s => CACHED [ 7/10] RUN ln -s /opt/hadoop/share/hadoop/tools/lib/aws-java-sdk-bundle-1.11.375.jar /opt/apache-hive 0.0s => CACHED [ 8/10] RUN ln -s /opt/hadoop/share/hadoop/tools/lib/hadoop-aws-3.2.0.jar /opt/apache-hive-bin/lib/ 0.0s => CACHED [ 9/10] RUN apt-get update -y && apt-get install -y netcat procps wait-for-it 0.0s => CACHED [10/10] COPY entrypoint.sh / 0.0s => exporting to image 0.1s => => exporting layers 0.0s => => writing image sha256:0179f2cd672e574060c17166455ca3c977cc4a9a9e0a1f3cb5a3969007a4cbe1 0.0s => => naming to docker.io/library/compose_hive-server 0.0s Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them Building notebook [+] Building 0.6s (7/11) => [internal] load build definition from Dockerfile 0.1s => => transferring dockerfile: 491B 0.0s => [internal] load .dockerignore 0.1s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/bitnami/spark:3 0.0s => [1/7] FROM docker.io/bitnami/spark:3 0.0s => [internal] load build context 0.1s => => transferring context: 38.93kB 0.0s => CACHED [2/7] RUN useradd -ms /bin/bash notebook 0.0s => ERROR [3/7] RUN chown notebook:notebook-R /app/ 0.4s ------ > [3/7] RUN chown notebook:notebook-R /app/: #6 0.392 chown: invalid group: 'notebook:notebook-R' ------ executor failed running [/bin/sh -c chown notebook:notebook-R /app/]: exit code: 1 ERROR: Service 'notebook' failed to build : Build failed

I see some different commands from the ones on the HEAD of

master

branch. Did you change anything? Are you synced with the origin?

@Itai Admi everything same. when I run log command these were not appearing. if i go to particular service log from the container i was seeing these errors.

ok taking another look

@Itai Admi I replaced docker file from above and did build again. Here is the log and i do not see that error now. minio uses an image, skipping minio-setup uses an image, skipping postgres uses an image, skipping lakefs uses an image, skipping lakefs-setup uses an image, skipping mariadb uses an image, skipping trino uses an image, skipping spark uses an image, skipping spark-worker uses an image, skipping spark-thrift uses an image, skipping Building hive-metastore [+] Building 5.2s (15/15) FINISHED => [internal] load build definition from Dockerfile 0.1s => => transferring dockerfile: 32B 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/library/openjdk:8u242-jre 4.7s => [ 1/10] FROM docker.io/library/openjdk:8u242-jre@sha256:f437249498cd370546035d6dc73e03a00ec637bcf47b740e60283 0.0s => [internal] load build context 0.1s => => transferring context: 35B 0.0s => CACHED [ 2/10] WORKDIR /opt 0.0s => CACHED [ 3/10] RUN mkdir /opt/hadoop /opt/apache-hive-bin 0.0s => CACHED [ 4/10] RUN curl -sSL https://www.apache.org/dist/hive/hive-3.1.2/apache-hive-3.1.2-bin.tar.gz | tar - 0.0s => CACHED [ 5/10] RUN curl -sSL https://archive.apache.org/dist/hadoop/common/hadoop-3.2.0/hadoop-3.2.0.tar.gz | 0.0s => CACHED [ 6/10] RUN curl -sSL https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-8.0.19.tar. 0.0s => CACHED [ 7/10] RUN ln -s /opt/hadoop/share/hadoop/tools/lib/aws-java-sdk-bundle-1.11.375.jar /opt/apache-hive 0.0s => CACHED [ 8/10] RUN ln -s /opt/hadoop/share/hadoop/tools/lib/hadoop-aws-3.2.0.jar /opt/apache-hive-bin/lib/ 0.0s => CACHED [ 9/10] RUN apt-get update -y && apt-get install -y netcat procps wait-for-it 0.0s => CACHED [10/10] COPY entrypoint.sh / 0.0s => exporting to image 0.1s => => exporting layers 0.0s => => writing image sha256:0179f2cd672e574060c17166455ca3c977cc4a9a9e0a1f3cb5a3969007a4cbe1 0.0s => => naming to docker.io/library/compose_hive-metastore 0.0s Building hive-server [+] Building 3.0s (15/15) FINISHED => [internal] load build definition from Dockerfile 0.1s => => transferring dockerfile: 32B 0.0s => [internal] load .dockerignore 0.1s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/library/openjdk:8u242-jre 2.6s => [ 1/10] FROM docker.io/library/openjdk:8u242-jre@sha256:f437249498cd370546035d6dc73e03a00ec637bcf47b740e60283 0.0s => [internal] load build context 0.0s => => transferring context: 35B 0.0s => CACHED [ 2/10] WORKDIR /opt 0.0s => CACHED [ 3/10] RUN mkdir /opt/hadoop /opt/apache-hive-bin 0.0s => CACHED [ 4/10] RUN curl -sSL https://www.apache.org/dist/hive/hive-3.1.2/apache-hive-3.1.2-bin.tar.gz | tar - 0.0s => CACHED [ 5/10] RUN curl -sSL https://archive.apache.org/dist/hadoop/common/hadoop-3.2.0/hadoop-3.2.0.tar.gz | 0.0s => CACHED [ 6/10] RUN curl -sSL https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-8.0.19.tar. 0.0s => CACHED [ 7/10] RUN ln -s /opt/hadoop/share/hadoop/tools/lib/aws-java-sdk-bundle-1.11.375.jar /opt/apache-hive 0.0s => CACHED [ 8/10] RUN ln -s /opt/hadoop/share/hadoop/tools/lib/hadoop-aws-3.2.0.jar /opt/apache-hive-bin/lib/ 0.0s => CACHED [ 9/10] RUN apt-get update -y && apt-get install -y netcat procps wait-for-it 0.0s => CACHED [10/10] COPY entrypoint.sh / 0.0s => exporting to image 0.1s => => exporting layers 0.0s => => writing image sha256:0179f2cd672e574060c17166455ca3c977cc4a9a9e0a1f3cb5a3969007a4cbe1 0.0s => => naming to docker.io/library/compose_hive-server 0.0s Building notebook [+] Building 0.4s (11/11) FINISHED => [internal] load build definition from Dockerfile 0.1s => => transferring dockerfile: 452B 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/bitnami/spark:3 0.0s => [1/6] FROM docker.io/bitnami/spark:3 0.0s => [internal] load build context 0.1s => => transferring context: 41B 0.0s => CACHED [2/6] RUN useradd -ms /bin/bash notebook 0.0s => CACHED [3/6] WORKDIR /home/notebook 0.0s => CACHED [4/6] RUN pip install --user notebook findspark pyspark-kernel spylon-kernel 0.0s => CACHED [5/6] COPY notebook-config.py /home/notebook/.jupyter/jupyter_notebook_config.py 0.0s => CACHED [6/6] RUN mkdir /home/notebook/notebooks 0.0s => exporting to image 0.1s => => exporting layers 0.0s => => writing image sha256:e00d5e9a9b9cedfa0c278fdd38a4a9a5f210764016e2e4170d1ec00488c5e99c 0.0s => => naming to docker.io/library/compose_notebook 0.0s C:\Users\ATK\lakeFS\deployments\compose>docker-compose ps Name Command State Ports ------------------------------------------------------------------------------------------------------------------- compose_notebook_1 jupyter notebook Exit 1 compose_spark-thrift_1 /opt/bitnami/scripts/spark ... Exit 126 compose_spark-worker_1 /opt/bitnami/scripts/spark ... Up 0.0.0.0:56620->8081/tcp compose_spark-worker_2 /opt/bitnami/scripts/spark ... Up 0.0.0.0:56623->8081/tcp compose_spark-worker_3 /opt/bitnami/scripts/spark ... Up 0.0.0.0:56622->8081/tcp compose_spark_1 /opt/bitnami/scripts/spark ... Up 0.0.0.0:18080->8080/tcp hive /bin/sh -c "entrypoint.sh" Exit 127 hiveserver2 hive --service hiveserver2 ... Up 0.0.0.0:10001->10000/tcp lakefs /app/wait-for postgres:543 ... Up 0.0.0.0:8000->8000/tcp lakefs-setup /app/wait-for postgres:543 ... Exit 1 mariadb docker-entrypoint.sh mariadbd Up 3306/tcp minio minio server /data --conso ... Up 0.0.0.0:9000->9000/tcp, 0.0.0.0:9001->9001/tcp minio-setup mc mb lakefs/example Exit 1 postgres docker-entrypoint.sh postgres Up 5432/tcp trino /usr/lib/trino/bin/run-trino Up 0.0.0.0:48080->8080/tcp

What do you mean by “replaced docker file from above”?

@Itai Admi okie let me try

Hi @Ashwath Can you

cd notebook

and run:

@Itai Admi docker pull gave error "docker pull" requires exactly 1 argument. See 'docker pull --help'. Usage: docker pull [OPTIONS] NAME[:TAG|@DIGEST] Pull an image or a repository from a registry

Missed the image name - use:

docker pull <http://docker.io/bitnami/spark:3|docker.io/bitnami/spark:3>

@Itai Admi oh okie , i missed dot(.)