https://lakefs.io/ logo
#help
Title
# help
l

Lior Resisi

01/28/2024, 3:08 PM
Hi there, is there any compatibility matrix of Java version, Spark 3.x, Hadoop, AWS libs and lakeFS?
l

Lynn Rozen

01/28/2024, 3:45 PM
Hi, Unfortunately I can't find something like that. Do you have specific version that you'd like to know about?
l

Lior Resisi

01/28/2024, 3:57 PM
Thanks for the prompt response! We want to start a migration project and start working with Java 17 and Spark 3.5, and want to make sure we will be able to use the latest lakeFS jars as part of it (for example, lakeFS output committer, garbage collector and so on). We’re wondering which
hadoop-aws
and
aws-java-sdk
we’ll be able to use (among other dependencies)
a

Ariel Shaqed (Scolnicov)

01/29/2024, 6:55 AM
Hi @Lior Resisi, welcome back! 1. We stepped away from lakeFS OutputCommitter, and are back to advising the use of default committers (FileOutputCommitter for lakeFSFS, latest magic for S3A). 2. lakeFSFS (the Hadoop filesystem) requires only Hadoop compatibility, and should work with Hadoop 3.x. It should work with your hadoop-aws. That said, I don't think we regularly test it with 3.5; could you open an issue, please? 3. Spark metadata client should "just work", but I would definitely open an issue there before 4. Are you still running on-prem Spark versions, or is it managed? Managed versions can have a lot of cruft that may cause issues.
l

Lior Resisi

01/29/2024, 8:46 AM
1. Thanks 2. Will do once we start the project, thanks! 3. Same 4. Still standalone (on-prem). I’m glad that there is finally an advantage for that 🙂