Isn’t there file-level deduplication in lakeFS? I ...
# helpa
Angela Bovo
11/21/2023, 6:39 PMIsn’t there file-level deduplication in lakeFS? I thought there might be, but then I uploaded the same file twice and got two different physical addresses.
i
Iddo Avneri
11/21/2023, 8:18 PMHi @Angela Bovo, if you upload the same exact file twice, there will be no uncommitted changes (so of course, data duplication).
Iddo Avneri
11/21/2023, 8:23 PMPlease notice that when you upload a file into lakeFS the file is copied over to the repository’s bucket (storage namespace). Instead, you can import a file or directory to lakeFS and then the files will not be copied at all.
Iddo Avneri
11/21/2023, 8:36 PMGMT20231121-201924_Recording_1920x1080.mp4
a
Angela Bovo
11/21/2023, 9:58 PMThank you Iddo for your answer. Importing data is interesting and I will look into it. Meanwhile, with upload, here are the results of my commands, you will see that there are two different physical addresses. How do you explain this?
12 Views