https://lakefs.io/ logo
Title
a

Ariel Shaqed (Scolnicov)

08/07/2022, 11:03 AM
I'm having some difficulty bringing up DBT :dbt: on the lakeFS :lakefs: Everything Bagel 🥯 . I've updated some things to use up-to-date DBT, avoid docker-compose-prefixes, and bump
ulimit -n
. Now my hive-metastore container fails to start; AFAICT the issue is
hive                      | MetaException(message:Required table missing : "`DBS`" in Catalog "" Schema "". DataNucleus requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable "datanucleus.schema.autoCreateTables")
(full logs are longer). It looks like Hive needs some startup script (or to set that value). Has anyone run into this?
j

Jonathan Rosenberg

08/07/2022, 11:50 AM
At what stage does it happen? I’ve applied the changes in your PR and the metastore is up and running. The only thing I changed is I used numbers instead of strings in the ulimit part…
a

Ariel Shaqed (Scolnicov)

08/07/2022, 1:12 PM
hive-metastore container gets it while starting up. I hate to suggest it, but perhaps you should
docker-compose build
? (But that may require additional work to recover to your current working state, of course. So tag at least the
hive
image?)
Hmmm.... looks like a bad startup sequence for the
hive-server
container. I waited and ran
❯ docker-compose restart hive-server
and that succeeded, now I can run the dbt container. I'll see if I can get hive-serve to retry some more.
Opened https://github.com/treeverse/lakeFS/pull/3853 to use updated and consistent versions of DBT, do a bit more waiting, etc. But... you will still need to wait after
docker-compose up -f
for 10-20s to give Hive a chance to load.