https://lakefs.io/ logo
Join Slack
Powered by
Hey <@U02LK72MULX>, the first part of the URL is t...
# help
y
Hey @Chaim Turkel, the first part of the URL is the repository name, and the second one is the reference (branch/tag/commit). So in this case you are pointing to the branch 
production
. Does such branch exist?
c
yes
so if i want to use a new branch i need to change the location of the database?
y
For this purpose, use the 
lakectl dbt create-branch-schema
command, which will copy the schema to a new name, along with the tables managed by dbt. Everything in the created schema will point to your new branch. See the docs for more info.
c
thanks
i don't see that it is supported in the api: https://pydocs.lakefs.io/
y
Unfortunately it's currently only supported using the command-line tool. I'm happy to help you to configure it.
c
running $HOME/lakefs/lakectl dbt create-branch-schema --branch production --to-schema test_branch getting error: exit status 1 Error executing command. $HOME/lakefs/lakectl repo list works
tried with trace:
Copy code
root@53f535723ab7:/dbt# $HOME/lakefs/lakectl dbt --log-level=TRACE create-branch-schema --branch production --to-schema test_branch
DEBU[0000]/home/runner/work/lakeFS/lakeFS/cmd/lakectl/cmd/root.go:67 <http://github.com/treeverse/lakefs/cmd/lakectl/cmd.glob..func69()|github.com/treeverse/lakefs/cmd/lakectl/cmd.glob..func69()> loaded configuration from file                fields.file=/root/.lakectl.yaml file=/root/.lakectl.yaml
exit status 1
Error executing command.
y
This command should be run from somewhere you can run the dbt command. Can you share the output of 
dbt debug
?This will help us narrow down the cause
Newer versions of lakectl will show more informative errors
c
to run dbt debug it use the following: dbt debug --profiles-dir /dbt --target dev
is there a way to configure this on lakefs dbt?
y
Let me check
c
found a way around: export DBT_PROFILES_DIR=/dbt export DBT_TARGET=dev dbt debug
y
Great! I was just about to write the same 🙂 Thanks for updating
c
Copy code
root@30449b82af06:/dbt# $HOME/lakefs/lakectl dbt --log-level=DEBUG create-branch-schema --branch test_me
DEBU[0000]/home/runner/work/lakeFS/lakeFS/cmd/lakectl/cmd/root.go:67 <http://github.com/treeverse/lakefs/cmd/lakectl/cmd.glob..func69()|github.com/treeverse/lakefs/cmd/lakectl/cmd.glob..func69()> loaded configuration from file                fields.file=/root/.lakectl.yaml file=/root/.lakectl.yaml
dbt debug succeeded with schema dbt_chaim
address <thrift://10.1.1.98:9083>: too many colons in address
my config is: metastore: type: hive hive: uri: thrift://10.1.1.98:9083
g
Hey @Chaim Turkel, Try to remove the 
thrift://
from the uri in the 
lakectl.yaml
file
It’s a mistake in the documentation, Fixing it now
c
Copy code
Error executing command.
root@ec2c30d380b0:/dbt# $HOME/lakefs/lakectl dbt --log-level=DEBUG create-branch-schema --branch test_me
DEBU[0000]/home/runner/work/lakeFS/lakeFS/cmd/lakectl/cmd/root.go:67 <http://github.com/treeverse/lakefs/cmd/lakectl/cmd.glob..func69()|github.com/treeverse/lakefs/cmd/lakectl/cmd.glob..func69()> loaded configuration from file                fields.file=/root/.lakectl.yaml file=/root/.lakectl.yaml
dbt debug succeeded with schema dbt_chaim
MetaException({Message:Got exception: java.io.FileNotFoundException innerMkdirs on <lakefs://data-application/test_me/dbt>: com.amazonaws.services.s3.model.AmazonS3Exception: The specified bucket does not exist (Service: Amazon S3; Status Code: 404; Error Code: NoSuchBucket; Request ID: 775ea914-ee37-4fff-a53d-0485a99112be; S3 Extended Request ID: BFB30E4CCEBF4A71), S3 Extended Request ID: BFB30E4CCEBF4A71})
Error executing command.
g
Seems like you don't have a brach called ‘test_me’ on lakeFS is the correct?
c
correct i thought it would create the branch. i need to create it first?
g
It currently does not
c
there is also an issue with the macro generate_schema_name.sql, i am still trying to unserstand it
y
Can you share the error you are encountering?
g
lakectl uses dbt to create views, for that to work you would need run the 
lakectl dbt generate-schema-macro
command. If you don’t want to create the macro, you could start with the 
--skip-view
flag. That would copy all the tables to the new database. You could later create the views using dbt by running 
dbt run --select config.materialized:view
c
Copy code
root@7b944fccfdd4:/dbt# $HOME/lakefs/lakectl dbt create-branch-schema --branch production --to-schema new_branch
dbt debug succeeded with schema dbt_chaim
schema new_branch created
copied dbt_chaim.dim_loyalty_merchants -> new_branch.dim_loyalty_merchants
copied dbt_chaim.dim_opportunity -> new_branch.dim_opportunity
copied dbt_chaim.fact_loyalty_purchases -> new_branch.fact_loyalty_purchases
copied dbt_chaim.fact_points_redemptions -> new_branch.fact_points_redemptions
copied dbt_chaim.fact_referral_codes -> new_branch.fact_referral_codes
copied dbt_chaim.loyalty__stg_daily_points_redemptions -> new_branch.loyalty__stg_daily_points_redemptions
copied dbt_chaim.loyalty__stg_fact_loyalty_purchases -> new_branch.loyalty__stg_fact_loyalty_purchases
failed creating views with err: exit status 2
Error executing command.
y
Hey @Chaim Turkel, it looks like command failed when trying to create the views in your new schema. After pointing your 
profiles.yml
file to use your new schema, you can trigger the failed phase manually. To do so, run 
dbt run --select config.materialized:view
. We are working on improving the error messages, so it will be easier to debug in the future.
c
suck again: $HOME/lakefs/lakectl dbt --log-level=TRACE create-branch-schema --branch production --to-schema test2 --skip-views DEBU[0000]/home/runner/work/lakeFS/lakeFS/cmd/lakectl/cmd/root.go:67 github.com/treeverse/lakefs/cmd/lakectl/cmd.glob..func69() loaded configuration from file fields.file=/root/.lakectl.yaml file=/root/.lakectl.yaml dbt debug succeeded with schema dbt_chaim schema test2 created exit status 2 Error executing command.
y
Let me apologize again for the logs being non-informative.
@Chaim Turkel, can you share the output of 
lakectl --version
, to make sure you’re using the latest one?
g
The fix for more informative logs for 
lakectl dbt create-branch-schema
command is currently in master and wasn’t yet released
We will release a version tomorrow, in the meantime we could send you a binary if it’s fine with you
c
that is great, thanks
Open in Slack
PreviousNext
Powered by