Thanks for the help so far trying to recreate exam...
# helpr
Richard Gilmore
07/05/2021, 11:11 AMThanks for the help so far trying to recreate example in https://lakefs.io/guarantee-consistency-in-your-delta-lake-tables-with-lakefs/
Copy code
spark.sql("select * from example.batch_summary version as of 16").write.format("delta").partitionBy("aggregated_date").save("<s3a://lakefs-data/master/example/batch_summary>") Copy code
AWSSecurityTokenServiceException: The security token included in the request is invalid. (Service: AWSSecurityTokenService; Status Code: 403; Error Code: InvalidClientTokenId; Request ID: b06a9752-c07c-4f5d-8a18-0099d3fb31be) Copy code
spark.sparkContext.hadoopConfiguration.set(s"fs.s3a.bucket.${repo}.access.key", dbutils.secrets.get("development","LAKEFS_ACCESS_KEY"))
spark.sparkContext.hadoopConfiguration.set(s"fs.s3a.bucket.${repo}.secret.key", dbutils.secrets.get("development","LAKEFS_SECRET_ACCESS_KEY"))
spark.sparkContext.hadoopConfiguration.set(s"fs.s3a.bucket.${repo}.endpoint", "<http://s3.lakefs.dev.example.com>")
spark.sparkContext.hadoopConfiguration.set(s"fs.s3a.bucket.${repo}.connection.ssl.enabled", "false")t
Tal Sofer
07/05/2021, 11:21 AMHi @Richard Gilmore!
We are looking into your error and will be with you soon 🙂
🙏 1
Tal Sofer
07/05/2021, 11:40 AM@Richard Gilmore can you please try following the configurations section in https://docs.lakefs.io/integrations/databricks.html#configuration
and add these two config parameters in particular
Copy code
spark.hadoop.fs.s3a.bucket.<repo-name>.aws.credentials.provider shaded.databricks.org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
spark.hadoop.fs.s3a.bucket.<repo-name>.session.token lakefs👀 1
Tal Sofer
07/05/2021, 11:41 AMPlease let me know if the error went away after adding these
r
Richard Gilmore
07/05/2021, 12:00 PMthat worked in addition to adding => spark.databricks.delta.multiClusterWrites.enabled false thank you
👍 1
t
Tal Sofer
07/05/2021, 12:04 PMSure! Thanks for letting me know