Oscar Wong
09/23/2024, 12:31 PMOscar Wong
09/23/2024, 12:32 PM# Download data
with tqdm(total=len(objects), desc="Downloading files", unit="file") as pbar:
for obj in objects:
print(f"Downloading {obj.path}...")
# Define the local file path
local_file_path = os.path.join(local_training_data_storage, obj.path)
# Create the directory if not exists
os.makedirs(os.path.dirname(local_file_path), exist_ok=True)
try:
with (
open(local_file_path, "wb") as local_file,
tag.object(obj.path).reader("rb") as r,
):
local_file.write(r.read())
except Exception as e:
print(f"Failed to download {obj.path}: {e}")
pbar.update(1)
Ariel Shaqed (Scolnicov)
09/23/2024, 12:49 PMlakectl local
, which provides an experience somewhat closer to "Everest".
Everest itself is a feature of lakeFS Enterprise.Oscar Wong
09/23/2024, 12:55 PMNiro
09/23/2024, 2:22 PM