Asher Song
03/29/2023, 3:48 PMYoni Augarten
03/29/2023, 3:55 PM--packages <...>
flag with:
// wrong jar was here
--jars <http://treeverse-clients-us-east.s3-website-us-east-1.amazonaws.com/hadoop/hadoop-lakefs-assembly-0.1.12.jar>
Asher Song
03/29/2023, 4:09 PMChandra Akkinepalli
03/29/2023, 4:11 PMYoni Augarten
03/29/2023, 4:13 PMChandra Akkinepalli
03/29/2023, 4:14 PMYoni Augarten
03/29/2023, 4:15 PM--packages
flag in my command. Could this be your case as well?Chandra Akkinepalli
03/29/2023, 4:36 PMspark-submit spark_changes.py --jars <http://treeverse-clients-us-east.s3-website-us-east-1.amazonaws.com/hadoop/hadoop-lakefs-assembly-0.1.12.jar>
and read from lakefs, i am able to follow the instructions on https://docs.lakefs.io/integrations/spark.html , created hdfs-site.xml to have appropriate settings.Asher Song
04/05/2023, 3:14 PMAriel Shaqed (Scolnicov)
04/05/2023, 3:33 PMAsher Song
04/05/2023, 3:35 PMAriel Shaqed (Scolnicov)
04/05/2023, 4:05 PMElad Lachmi
04/05/2023, 4:14 PMeinat.orr