Robin Moffatt04/25/2023, 5:03 PM
? I've tried that but getting
and no obvious error on the lakeFS side that I can see.
org.apache.spark.SparkException: Unable to create database default as failed to create its directory <s3://example/main>.
Oz Katz04/25/2023, 5:23 PM
Amit Kesarwani04/25/2023, 9:28 PM
Robin Moffatt04/26/2023, 9:53 AM
. Is there a useful reference on why the URI is sometimes one or the other?
from Spark SQL it fails:
However, if I create a dummy file (e.g.
org.apache.spark.SparkException: Unable to create database default as failed to create its directory <s3a://example/main> java.io.FileNotFoundException: PUT 0-byte object on main/: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: null; S3 Extended Request ID: null; Proxy: null), S3 Extended Request ID: null:404 Not Found
) then the
works fine. Is there a more elegant/proper way to do this than creating a dummy file first? Notebook: https://gist.github.com/rmoff/28211a28adf7c55607f7ed7e4c4efc8f
Oz Katz04/26/2023, 5:07 PM
or something of that sort. Otherwise, Spark will attempt to write an zero-length-named file at the root of that location, which is ambiguous with the branch URI.
Robin Moffatt04/26/2023, 5:07 PM
worked just fine - it was the
prefix that broke things
Ariel Shaqed (Scolnicov)04/27/2023, 3:44 AM