I'm having some difficulty bringing up DBT :dbt: o...
# dev
I'm having some difficulty bringing up DBT dbt on the lakeFS lakefs Everything Bagel 🥯 . I've updated some things to use up-to-date DBT, avoid docker-compose-prefixes, and bump
ulimit -n
. Now my hive-metastore container fails to start; AFAICT the issue is
Copy code
hive                      | MetaException(message:Required table missing : "`DBS`" in Catalog "" Schema "". DataNucleus requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable "datanucleus.schema.autoCreateTables")
(full logs are longer). It looks like Hive needs some startup script (or to set that value). Has anyone run into this?
At what stage does it happen? I’ve applied the changes in your PR and the metastore is up and running. The only thing I changed is I used numbers instead of strings in the ulimit part…
hive-metastore container gets it while starting up. I hate to suggest it, but perhaps you should
docker-compose build
? (But that may require additional work to recover to your current working state, of course. So tag at least the
Hmmm.... looks like a bad startup sequence for the
container. I waited and ran
Copy code
❯ docker-compose restart hive-server
and that succeeded, now I can run the dbt container. I'll see if I can get hive-serve to retry some more.
Opened https://github.com/treeverse/lakeFS/pull/3853 to use updated and consistent versions of DBT, do a bit more waiting, etc. But... you will still need to wait after
docker-compose up -f
for 10-20s to give Hive a chance to load.