I m having some difficulty bringing up DBT dbt on the lakeFS lakeFS #dev

I'm having some difficulty bringing up DBT :dbt: o...

Ariel Shaqed (Scolnicov)

08/07/2022, 11:03 AM

I'm having some difficulty bringing up DBT dbt on the lakeFS lakefs Everything Bagel 🥯 . I've updated some things to use up-to-date DBT, avoid docker-compose-prefixes, and bump

ulimit -n

. Now my hive-metastore container fails to start; AFAICT the issue is

Copy code

hive                      | MetaException(message:Required table missing : "`DBS`" in Catalog "" Schema "". DataNucleus requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable "datanucleus.schema.autoCreateTables")

(full logs are longer). It looks like Hive needs some startup script (or to set that value). Has anyone run into this?

Jonathan Rosenberg

08/07/2022, 11:50 AM

At what stage does it happen? I’ve applied the changes in your PR and the metastore is up and running. The only thing I changed is I used numbers instead of strings in the ulimit part…

Ariel Shaqed (Scolnicov)

08/07/2022, 1:12 PM

hive-metastore container gets it while starting up. I hate to suggest it, but perhaps you should

docker-compose build

? (But that may require additional work to recover to your current working state, of course. So tag at least the

hive

image?)

Ariel Shaqed (Scolnicov)

08/07/2022, 1:58 PM

Hmmm.... looks like a bad startup sequence for the

hive-server

container. I waited and ran

Copy code

❯ docker-compose restart hive-server

and that succeeded, now I can run the dbt container. I'll see if I can get hive-serve to retry some more.

Ariel Shaqed (Scolnicov)

08/08/2022, 10:51 AM

Opened https://github.com/treeverse/lakeFS/pull/3853 to use updated and consistent versions of DBT, do a bit more waiting, etc. But... you will still need to wait after

docker-compose up -f

for 10-20s to give Hive a chance to load.

32 Views

Open in Slack

Previous Next