https://lakefs.io/ logo
Join Slack
Powered by
Hello all, we are trying to implement lakefs as a ...
# help
s
Hello all, we are trying to implement lakefs as a versioned data lake layer on top of S3. We've followed the docs and everything seems to be working fine, until we try integrating with Spark. Writing a small 10 byte dataframe into S3 takes 20+ seconds. On the lakefs console logs (set to debug) we see a ton of "Object set for deletion" and "failed deleting object" outputs. We are wondering if this is due to some configuration that we need to set on our end, or maybe something to do with the Hadoop output committers? Happy to provide more details if needed.
g
Hey @Sid Senthilnathan, welcome. Can you please provide part of the log and  more details about spark job ?
s
We're getting a whole lot of these: 
{"action":"delete_objects","file":"lakeFS/pkg/gateway/middleware.go:97","func":"pkg/gateway.EnrichWithOperation.func1.1","level":"debug","message_type":"action","msg":"performing S3 action","time":"2021-04-19T19:48:18Z"}
{"file":"lakeFS/pkg/gateway/operations/deleteobjects.go:75","func":"pkg/gateway/operations.(*DeleteObjects).Handle","host":"<http://origin.s3.lakefs.paigeai.net|origin.s3.lakefs.paigeai.net>","key":"branch-1/foo/bar/","level":"debug","method":"POST","msg":"object set for deletion","path":"/?delete","repository":"origin","request_id":"e31e6b81-a3b0-4db0-b4a7-3fe329558be9","service_name":"s3_gateway","time":"2021-04-19T19:48:18Z","user":"svc.emr"}
{"file":"lakeFS/pkg/gateway/operations/deleteobjects.go:75","func":"pkg/gateway/operations.(*DeleteObjects).Handle","host":"<http://origin.s3.lakefs.paigeai.net|origin.s3.lakefs.paigeai.net>","key":"branch-1/foo/","level":"debug","method":"POST","msg":"object set for deletion","path":"/?delete","repository":"origin","request_id":"e31e6b81-a3b0-4db0-b4a7-3fe329558be9","service_name":"s3_gateway","time":"2021-04-19T19:48:18Z","user":"svc.emr"}
{"error":"argument path: required value","file":"lakeFS/pkg/gateway/operations/deleteobjects.go:67","func":"pkg/gateway/operations.(*DeleteObjects).Handle","host":"<http://origin.s3.lakefs.paigeai.net|origin.s3.lakefs.paigeai.net>","key":"branch-1/","level":"error","method":"POST","msg":"failed deleting object","path":"/?delete","repository":"origin","request_id":"e31e6b81-a3b0-4db0-b4a7-3fe329558be9","service_name":"s3_gateway","time":"2021-04-19T19:48:18Z","user":"svc.emr"}
{"file":"lakeFS/pkg/httputil/logging.go:77","func":"pkg/httputil.DebugLoggingMiddleware.func1","host":"<http://origin.s3.lakefs.paigeai.net|origin.s3.lakefs.paigeai.net>","level":"debug","method":"POST","msg":"HTTP call ended","path":"/?delete","repository":"origin","request_id":"e31e6b81-a3b0-4db0-b4a7-3fe329558be9","sent_bytes":249,"service_name":"s3_gateway","status_code":200,"time":"2021-04-19T19:48:18Z","took":23154915,"user":"svc.emr"}
And we are just trying to write a real simple dataframe. We initialize the dataframe manually with 3 rows, and do a df.write.mode("overwrite").parquet("s3a://repo/branch/name")
g
@Sid Senthilnathan, this was fixed in this PR and will be part of the next lakeFS version which will be published in the following days.
👍 1
s
Could you briefly explain what the underlying issue is? We saw this closed issue but it did not contain too many details.
And until the fix is deployed, is there any guidance on pulling the code and building it ourselves?
g
when the spark job removes the files it tries to remove also the path of the branch, which is mapped to an empty path and fails
you could find guidance for setting up an environment here
Let me know if you need any help
Hey @Sid Senthilnathan, The version was just released. I would like to add that we have many spark users that don’t experience performance issues. If the performance issues continue with the new version, let me know and we could go over your environment and identify the cause for the performance issues.
s
Awesome! I will check it out and see if it has addressed the issue. Thanks for the quick response.
@Guy Hardonag looks like it works - thanks!
g
Happy to hear, let us know if you need anything. Thanks!
s
Hey @Guy Hardonag, we're running into a similar issue again. We are running a simple Spark SQL command ( 
INSERT INTO lakefs.stg_image
SELECT * FROM dbt_stg.stg_image
) which takes some data from our regular S3 bucket and writes to the lakefs path. This command works with a few hundred lines but runs indefinitely when its the whole table (about a million records).
Similar to the previous case, we see in the UI that a 
_temporary/0
directory is created, and it seems like it wants to delete some files that aren't there
{"file":"lakeFS/pkg/logging/logger.go:79","func":"pkg/logging.logrusEntryWrapper.Debug","host":"<http://origin.s3.lakefs.paigeai.net|origin.s3.lakefs.paigeai.net>","key":"lakefs_lakefs.db/stg_image/_temporary/0/_temporary/attempt_20210420193535_202823_m_000002_442722/part-00002-ed3e7766-1673-487a-b9a7-69e0de32d18d_00002.c000.snappy.orc","level":"debug","method":"DELETE","msg":"aborted multipart upload","path":"lakefs_lakefs.db/stg_image/_temporary/0/_temporary/attempt_20210420193535_202823_m_000002_442722/part-00002-ed3e7766-1673-487a-b9a7-69e0de32d18d_00002.c000.snappy.orc","qualified_key":"lakefs_lakefs.db/stg_image/_temporary/0/_temporary/attempt_20210420193535_202823_m_000002_442722/part-00002-ed3e7766-1673-487a-b9a7-69e0de32d18d_00002.c000.snappy.orc","qualified_ns":"paige-data-s3-preprod-use1-lakefs-datalake","ref":"master","repository":"origin","request_id":"fc00fb53-de16-4a93-a1d2-6c2f05f55171","service_name":"s3_gateway","time":"2021-04-20T19:44:04Z","upload_id":"Qgj7c3Uio1sJiODl7SnUyoajVgmAXQJaRUGIH5hHdKktsSmXVqvRWcYTVzr2AKJkSZparmreqTCjcXRtd6kHs4EteFpmDNxYcjUkb3O3R.wb32AJ7xYKgYk0JMH_fvrJ","user":"svc.emr"}
{"error":"NoSuchUpload: The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.\n\tstatus code: 404, request id: 7AA7FH7WMZKBQRRF, host id: NmJ2kSWHUguuNChVK66er1gDqrstk0hmXNqvRHEoULmhmpXcD9Z4PmtcnDaBjYpNDKohokjdwUA=","file":"lakeFS/pkg/logging/logger.go:95","func":"pkg/logging.logrusEntryWrapper.Error","host":"<http://origin.s3.lakefs.paigeai.net|origin.s3.lakefs.paigeai.net>","level":"error","method":"DELETE","msg":"could not abort multipart upload","path":"lakefs_lakefs.db/stg_image/_temporary/0/_temporary/attempt_20210420193535_202823_m_000002_442722/part-00002-ed3e7766-1673-487a-b9a7-69e0de32d18d_00002.c000.snappy.orc","ref":"master","repository":"origin","request_id":"fc00fb53-de16-4a93-a1d2-6c2f05f55171","service_name":"s3_gateway","time":"2021-04-20T19:44:04Z","upload_id":"Qgj7c3Uio1sJiODl7SnUyoajVgmAXQJaRUGIH5hHdKktsSmXVqvRWcYTVzr2AKJkSZparmreqTCjcXRtd6kHs4EteFpmDNxYcjUkb3O3R.wb32AJ7xYKgYk0JMH_fvrJ","user":"svc.emr"}
g
Hey @Sid Senthilnathan, seems like there is a problem with multipart upload. Let me try to reproduce it. Will update soon
Sorry, doesn’t seem like a problem with multipart. it looks like for some reason multipart upload was aborted and after that when trying to write a the upload it fails because it was aborted ( which is supposed to happen) Just Need to understand why the upload was aborted. can you please provide some logs prior to these?
s
The more we look at it the more it seems like this and the previous errors may be an upstream problem with Spark, and that Lakefs may not be handling the error gracefully. So those may be two different issues.
I am able to upload files up to 100MB into lakefs without an issue, and beyond that is when I start getting problems
Do you have any examples of a Spark application that has been writing larger amounts of data? We are thinking that there may be an issue with the default configurations, such as the output formatter, multipart upload thresholds, EMR settings, etc
g
We have examples, we didn’t get that error in any of them. Might be because the network is slow for some reason. can you try to configure a larger timeout? configure 
fs.s3a.connection.timeout
to be 
600000
( the default is 
200000
)
Let me know if this works, I will check for other configurations that might help out. can you please provide information about your environment? how are you running lakeFS?
s
Ok I will check. We suspect that it might have something to do with the output committer, so if there are any configs around that, that would help.
We are running lakefs on Linux EC2, a single VM behind a load balancer. Block storage is S3. Writing to Lakefs over S3 from EMR/Spark via the S3A adapter.
g
Thanks, I will update you about the configurations. What is the Instance type of your EC2?
Hey @Sid Senthilnathan, please try tuning the following lakeFS configurations
Copy code
blockstore.s3.streaming_chunk_size (int : 1048576) - Object chunk size to buffer before streaming to blockstore (use a lower value for less reliable networks). Minimum is 8192.
blockstore.s3.streaming_chunk_timeout (time duration : "60s") - Per object chunk timeout for blockstore streaming operations (use a larger value for less reliable networks).
start by setting 
blockstore.s3.streaming_chunk_size
to 
262144
s
We are using m5.xlarge. I will try increasing decreasing the streaming chunk size
g
Thanks, you should try to decrease. Let me know if it helps
s
So we found this article (https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.6.1/bk_cloud-data-access/content/s3a-fast-upload.html) and what caught my eye was the 
fs.s3a.multipart.size
hadoop configuration, which defaults to 100MB (the threshold that I was noticing our problems started). I bumped it up to 200MB and everything seems to be working now
g
Amazing! did you try configuring 
blockstore.s3.streaming_chunk_size
?
s
Yeah, it didn't seem to change anything. As a suggestion, can Lakefs logic be updated to handle these situations more gracefully? Similar to the PR that you linked earlier
i
Hey Sid, your suggestion seems reasonable to me. Created this issue to track it down. I’ll continue to update on the issue and link any relevant PRs.
s
Excellent!
5 Views
Open in Slack
PreviousNext
Powered by