Hey all. I’m following <this> setup to work with L...
# help
u
Hey all. I’m following this setup to work with LakeFs and Athena (hoping that work with Spectrum too). In my understanding the setup uses symlink. Let’s imagine that the underling data in S3 (via LakeFs), changes every hour (partitioning max till the hour). Do I have to run
Copy code
lakectl metastore create-symlink \
--repo example \
--branch main \
--path my_table \
--from-client-type hive \
--from-schema default \
--from-table my_table \
--to-schema default \ 
--to-table my_table
command each time that the data change? if so will it be fast enough each time that new partitions are added? Will it run incrementally (create sysmlink only for new partitions) or it will re-create from scratch? Also during such operation, will the table be still queryable? Or I should expect a not found error from my clients. Thanks.