Hi there, is there any compatibility matrix of Java version, Spark 3.x, Hadoop, AWS libs and lakeFS?

Hi, Unfortunately I can't find something like that. Do you have specific version that you'd like to know about?

Thanks for the prompt response! We want to start a migration project and start working with Java 17 and Spark 3.5, and want to make sure we will be able to use the latest lakeFS jars as part of it (for example, lakeFS output committer, garbage collector and so on). We’re wondering which

hadoop-aws

and

aws-java-sdk

we’ll be able to use (among other dependencies)

Hi @Lior Resisi, welcome back! 1. We stepped away from lakeFS OutputCommitter, and are back to advising the use of default committers (FileOutputCommitter for lakeFSFS, latest magic for S3A). 2. lakeFSFS (the Hadoop filesystem) requires only Hadoop compatibility, and should work with Hadoop 3.x. It should work with your hadoop-aws. That said, I don't think we regularly test it with 3.5; could you open an issue, please? 3. Spark metadata client should "just work", but I would definitely open an issue there before 4. Are you still running on-prem Spark versions, or is it managed? Managed versions can have a lot of cruft that may cause issues.

1. Thanks 2. Will do once we start the project, thanks! 3. Same 4. Still standalone (on-prem). I’m glad that there is finally an advantage for that 🙂