Hi, I’m new to lakfs. I’m trying to run lakefs loc...
# helpg
gang ye
10/06/2023, 9:09 PMHi, I’m new to lakfs. I’m trying to run lakefs locally and use spark to write Iceberg table. But the write failed with
Copy code
exception org.apache.spark.sql.connector.catalog.CatalogNotFoundException: Catalog 'lakefs' plugin class not found: spark.sql.catalog.lakefs is not defined
at org.apache.spark.sql.errors.QueryExecutionErrors$.catalogPluginClassNotFoundError(QueryExecutionErrors.scala:1904)gang ye
10/06/2023, 9:10 PMMy spark conf
Copy code
Copy code
conf.set("spark.jars.packages", "org.apache.iceberg:iceberg-spark-runtime-3.3_2.12:1.3.0,io.lakefs:lakefs-iceberg:v0.1.2,org.apache.hadoop:hadoop-aws:3.3.3,io.lakefs:hadoop-lakefs-assembly:0.1.13")
conf.set("spark.sql.catalog.lakefs", "org.apache.iceberg.spark.SparkCatalog")
conf.set("spark.sql.catalog.lakefs.catalog-impl", "io.lakefs.iceberg.LakeFSCatalog")
conf.set("spark.sql.catalog.lakefs.warehouse", f"<lakefs://quickstart>")
conf.set("spark.sql.catalog.lakefs.uri", "<http://127.0.0.1:8000>")
conf.set("spark.sql.defaultCatalog", "lakefs")
conf.set("spark.hadoop.fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
conf.set("spark.hadoop.fs.s3a.endpoint", "<http://127.0.0.1:8000>")
conf.set("spark.hadoop.fs.s3a.access.key", "xxx")
conf.set("spark.hadoop.fs.s3a.secret.key", "xxx")
conf.set("spark.hadoop.fs.s3a.path.style.access", "true")
conf.set("spark.sql.catalog.lakefs.cache-enabled", "false")a
Amit Kesarwani
10/06/2023, 9:24 PMDon’t use
v Copy code
conf.set("spark.jars.packages", "org.apache.iceberg:iceberg-spark-runtime-3.3_2.12:1.3.0,io.lakefs:lakefs-iceberg:0.1.2,org.apache.hadoop:hadoop-aws:3.3.3,io.lakefs:hadoop-lakefs-assembly:0.1.13")g
gang ye
10/06/2023, 9:25 PMOK. Let me have a try
gang ye
10/06/2023, 11:45 PMstrange thing is, if I remove
v Copy code
:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Exception in thread "main" java.lang.RuntimeException: [unresolved dependency: io.lakefs#lakefs-iceberg;0.1.2: not found]
at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1537)gang ye
10/06/2023, 11:47 PMv0.1.2 Copy code
:: resolving dependencies :: org.apache.spark#spark-submit-parent-1579771d-c966-4f4a-a52e-389072d95ac0;1.0
confs: [default]
found org.apache.iceberg#iceberg-spark-runtime-3.3_2.12;1.3.0 in central
found io.lakefs#lakefs-iceberg;v0.1.2 in centralgang ye
10/06/2023, 11:49 PMIs there any specific jar repo we need to add
gang ye
10/06/2023, 11:51 PMhere https://mvnrepository.com/artifact/io.lakefs/lakefs-iceberg
0.1.2 is not in the version list
a
Amit Kesarwani
10/06/2023, 11:57 PMI used 0.1.1. Check sample notebooks for Iceberg demo: https://github.com/treeverse/lakeFS-samples/
g
gang ye
10/06/2023, 11:58 PMOK. Thanks! I’m trying out with 0.1.3
gang ye
10/07/2023, 12:14 AMit works! Thanks!
jumping lakefs 2
a
Amit Kesarwani
10/07/2023, 12:16 AM👍
3 Views