Hi, There are more exemple in pyspark? Similar thi...
# help
j
Hi, There are more exemple in pyspark? Similar this
Copy code
import io.treeverse.clients.LakeFSContext
    
val commitID = "a1b2c3d4"
val df = LakeFSContext.newDF(spark, "example-repo", commitID)
o
hi @Jennifer Cristina Evangelista Da Silva! this is an example how to read a path at a specific commit into pandas datafame through lakeFS. at the moment our repository does include one more example how to execute spark sql function on the same dataframe:
Copy code
df.createOrReplaceTempView("files")
 spark.sql("SELECT DATE(last_modified), COUNT(*) FROM files GROUP BY 1 ORDER BY 1")
 /* output example:
    +----------+--------+
    |        dt|count(1)|
    +----------+--------+
    |2021-03-05|       2|
    |2021-03-07|       2|
    +----------+--------+
  */
mind sharing what you're trying to do?