<@U02BRC5J7U0> is there a tutorial on how to use S...
# helpe
Edmondo Porcu
05/02/2022, 5:54 PM@Iddo Avneri is there a tutorial on how to use Spark tables with LakeFS? I was wondering what's the right approach. Should one create a clone of a database but pointing it to a different branch?
i
Iddo Avneri
05/02/2022, 5:56 PMAre you on Databricks?
o
Or Tzabary
05/02/2022, 5:59 PMhey @Edmondo Porcu !
mind sharing some context on the use-case?
Or Tzabary
05/02/2022, 6:04 PMWe have our Spark integration guide which contains a high-level introduction on how to use lakeFS with Spark, you can also read Similarweb's case study.
Looking forward on hearing more on you use-case.
e
Edmondo Porcu
05/02/2022, 9:44 PMThis uses s3 API directly, while people often use the hive metastore and interacts with data using tables
Edmondo Porcu
05/02/2022, 9:44 PMIn practice if you want to restore a previous version of a certain database, you need to change the location at which the database is pointing
o
Or Tzabary
05/02/2022, 10:00 PMYou can use lakeFS to revert a branch to specific commits/tags and this way you won't need to change the location
Or Tzabary
05/02/2022, 10:01 PMYou can see it in the first example here: https://docs.lakefs.io/usecases/production.html
e
einat.orr
05/03/2022, 3:30 AM
@Edmondo Porcu You can use lakeFS with Hive Metastore. Instructions on how to do so are here: https://docs.lakefs.io/integrations/hive.html
🙏 1
o
Or Tzabary
05/05/2022, 2:26 PMhi @Edmondo Porcu
I wonder, did you manage to solve your issue?
e
Edmondo Porcu
05/05/2022, 10:14 PMnope 😞
Edmondo Porcu
05/05/2022, 10:15 PMI don't understand how you switch the data underneath a table transparently, but maybe is the wrong thing to do. However that's how people on Spark are used to deal with tables
o
Or Tzabary
05/05/2022, 10:23 PMI'm not sure I understand, are you referring to lakeFS in general? Mind clarifying?
e
Edmondo Porcu
05/05/2022, 10:25 PMSo, if you have a pipeline that works with "file API" Spark, you have a prefix that point to the current branch. You create a new branch, you change the prefix, great.
But what about when you work with tables API?
o
Or Tzabary
05/05/2022, 10:30 PMlakeFS does support Hive integration. Have you seen the link Einat shared?
e
Edmondo Porcu
05/08/2022, 7:06 PMyes, it sounds thought the workflow would not what people expect
o
Or Tzabary
05/08/2022, 7:11 PMcare to share how would you expect to see it?
we're open source and open to changes 🙂
e
Edmondo Porcu
06/23/2022, 6:17 PMYou would want to change the branch of a datasource and get all the hive tables updated, probably...
3 Views