HT
06/05/2023, 5:54 AMlakefs repo list
can only list a maximum of 1000
Is there a way to bump this number up ? Because to programatically list more 1000 repos is a bit a pain, greping for for more results run with --amount 1000 --after "a-repo-name"
for more results run with --amount 1000 --after "a-repo-name"
....Guy Hardonag
06/05/2023, 6:22 AMHT
06/05/2023, 6:24 AMGuy Hardonag
06/05/2023, 6:29 AMHT
06/05/2023, 6:32 AMGuy Hardonag
06/05/2023, 6:35 AMHT
06/05/2023, 6:35 AMGuy Hardonag
06/05/2023, 6:49 AMHT
06/05/2023, 7:08 AMrclone copy --files-from /dev/shm/downloadParquet.sh.tmp sandbox: /data/hieu/deleteme/demo/ --dry-run --transfers=64
with :
$ head /dev/shm/downloadParquet.sh.tmp
20180523-140551-spurderuel-clone000/main/handy/annotations.parquet
20180523-140551-spurderuel-clone001/main/handy/annotations.parquet
20180523-140551-spurderuel-clone002/main/handy/annotations.parquet
20180523-140551-spurderuel-clone003/main/handy/annotations.parquet
20180523-140551-spurderuel-clone004/main/handy/annotations.parquet
20180523-140551-spurderuel-clone005/main/handy/annotations.parquet
20180523-140551-spurderuel-clone006/main/handy/annotations.parquet
20180523-140551-spurderuel-clone007/main/handy/annotations.parquet
20180523-140551-spurderuel-clone008/main/handy/annotations.parquet
20180523-140551-spurderuel-clone009/main/handy/annotations.parquet
rclone do 1000 files then stop without error ...Guy Hardonag
06/05/2023, 7:17 AMHT
06/05/2023, 7:23 AMGuy Hardonag
06/05/2023, 7:34 AMHT
06/05/2023, 7:42 AMGuy Hardonag
06/05/2023, 7:43 AMHT
06/05/2023, 7:45 AMGuy Hardonag
06/05/2023, 7:49 AMDoes that mean that i will need a separate system that track what files do exist and which are ghost only?No, it’s deleted by a retention policy This should make it more clear https://docs.lakefs.io/howto/gc-internals.html
HT
06/05/2023, 8:01 AMGuy Hardonag
06/05/2023, 8:02 AMHT
06/05/2023, 8:04 AMGuy Hardonag
06/05/2023, 8:13 AMHT
06/05/2023, 8:16 AMrclone lsd sandbox: | wc -l
will only return 1000$ rclone --version
rclone v1.62.2
- os/version: opensuse-leap 15.4 (64 bit)
- os/kernel: 5.14.21-150400.24.60-default (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.2
- go/linking: static
- go/tags: none
--no-traverse
is the magic ...rclone --no-traverse lsd sandbox: | wc -l
still only give 1000Guy Hardonag
06/05/2023, 9:22 AMHT
06/05/2023, 9:24 AMGuy Hardonag
06/05/2023, 9:25 AMHT
06/05/2023, 9:40 AMGuy Hardonag
06/05/2023, 11:42 AMHT
06/05/2023, 10:12 PMIddo Avneri
06/05/2023, 10:25 PM