It’s me again 🙂 I’m trying to understand more about LakeFS by experimenting through the Jupyter notebook. It’s a long time since I used pySpark, so I’m not sure if the problem I’m getting is because I’m doing something plain stupid there and nothing to do with LakeFS 😉 Attached is my notebook. The tl;dr is that I write a parquet file, branch and read it back, modify it — and then when I try to write it back to the branch I get an error. • If I write it without

mode=overwite

I get

AnalysisException: path <s3a://example/remove_pii/demo/users> already exists

• If I write it with

mode=overwrite

I get

FileNotFoundException: No such file or directory: <s3a://example/remove_pii/demo/users/part-00000-83b53955-b055-4ca8-8adc-fc42010cfb80-c000.snappy.parquet>

Welcome back 😄 As far as I know the first error has nothing todo with lakeFS, writing without overwrite to an existing should fail. Looking into to the second error… will be back with an answer today

You might want to use this notebook - could be easier: https://github.com/treeverse/lakeFS-samples/tree/main/03-apache-spark-python-demo

oh cool @Iddo Avneri, that looks useful. thanks!

There is also a youtube video going over setting up one of these (the delta specifically, but SPARK will be the same):

@Robin Moffatt in your notebook I've configured the spark session this way:

from pyspark.context import SparkContext
from pyspark import SparkFiles
from pyspark.sql.session import SparkSession

spark = SparkSession \
    .builder \
    .config("spark.hadoop.fs.s3a.access.key", "AKIAIOSFODNN7EXAMPLE") \
    .config("spark.hadoop.fs.s3a.secret.key", "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY") \
    .config("spark.hadoop.fs.s3a.endpoint", "<http://lakefs:8000>") \
    .getOrCreate()
sc = spark.sparkContext

Thanks @Barak Amar. So does that override https://github.com/treeverse/lakeFS/blob/master/deployments/compose/etc/hive-site.xml#L51-L62 and do something different?

right, I need to check why the spark inside the notebook doesn't use these values.

It seems to use something because it writes the first dataframe just fine

the error is on the second branch

let me check

the last error you paste was from a request to read the file?

no, a write statement

df2.write.mode('overwrite').parquet('s3a://'+repo+'/'+branch+'/demo/users')

ok, I'll be next to my laptop a bit later will rerun the notebook and get back with better answers in an hour

thanks!

Hi Robin, the problem is that when we (notebook) remove the pii - we need to read the data and write the data. In this case you like to read the data from the main branch and write the data to the remove_pii branch. Currently the branch remove_pii holds the right data (at the point you branch), but we write to the same location we read from.

hey @Barak Amar, thanks for this. By using your approach (read from main, write to branch) I got it to work. However I’m still not clear on this, because in my mental model of branching (based on git) I would expect to also read from my branch;

main

could have changed since I created my

remove_pii

branch, so reading from it feels incorrect. Can you help clarify this?

Sure, the thing is not directly related to lakeFS. We try to transform and overwrite the same path. So the data is lost as part of the override process.

heh I was just reading https://community.databricks.com/s/question/0D53f00001HKHkcCAH/saving-to-parquet-with-savemodeoverwrite-throws-exception which I think is similar

Yes, about the branch information - I think it would be the right way to go. About spark - learning the pitfalls with you, so this is great 🙂

thanks, I appreciate it!

Saw this one - don't like to count on cache to prevent data access. Don't know when it will not have enough space and it will go to the actual storage. Or it is based on writing the data and move it at the end. So read/process/write without the cache will probably be faster for later sets.

got it. thanks.