Ariel Shaqed (Scolnicov)

    Ariel Shaqed (Scolnicov)

    1 month ago
    I'm having some difficulty bringing up DBT :dbt: on the lakeFS :lakefs: Everything Bagel 🥯 . I've updated some things to use up-to-date DBT, avoid docker-compose-prefixes, and bump
    ulimit -n
    . Now my hive-metastore container fails to start; AFAICT the issue is
    hive                      | MetaException(message:Required table missing : "`DBS`" in Catalog "" Schema "". DataNucleus requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable "datanucleus.schema.autoCreateTables")
    (full logs are longer). It looks like Hive needs some startup script (or to set that value). Has anyone run into this?
    Jonathan Rosenberg

    Jonathan Rosenberg

    1 month ago
    At what stage does it happen? I’ve applied the changes in your PR and the metastore is up and running. The only thing I changed is I used numbers instead of strings in the ulimit part…
    Ariel Shaqed (Scolnicov)

    Ariel Shaqed (Scolnicov)

    1 month ago
    hive-metastore container gets it while starting up. I hate to suggest it, but perhaps you should
    docker-compose build
    ? (But that may require additional work to recover to your current working state, of course. So tag at least the
    Hmmm.... looks like a bad startup sequence for the
    container. I waited and ran
    ❯ docker-compose restart hive-server
    and that succeeded, now I can run the dbt container. I'll see if I can get hive-serve to retry some more.
    Opened to use updated and consistent versions of DBT, do a bit more waiting, etc. But... you will still need to wait after
    docker-compose up -f
    for 10-20s to give Hive a chance to load.