Yaphet Kebede
02/02/2023, 8:53 PMVino
02/02/2023, 9:02 PMroot temp dir configured on the client object
? Would you mind sharing how you configured that?Yaphet Kebede
02/02/2023, 9:22 PMdef init_client(lakefs_conf_path):
logger.debug('initializing ')
configuration = lakefs_client.Configuration()
configuration.temp_folder_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'data')
with open(lakefs_conf_path) as stream:
config_raw = yaml.load(stream, Loader=yaml.FullLoader)
configuration.username = config_raw["credentials"]["access_key_id"]
configuration.password = config_raw["credentials"]["secret_access_key"]
configuration.host = config_raw["server"]["endpoint_url"]
the_lake = LakeFsWrapper(configuration=configuration)
return the_lake
here's my client init, i set
configuration.temp_folder_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'data')
as the correct path which is great, I can see the files being downloaded there when i make a call like
# path is lakefs location
self._client.objects.get_object(repository=repository, ref=branch, path=location)
but what i wanted to do was say i have
some-path/some.txt
on lake fs ,
and wanted to download it to a local
<temp_folder_path>/some-path/some.txt
the client.objects.get_objects
wouldn't let me as it would just dump some.txt
directly under <temp_folder_path>
, ignoring some-path
Vino
02/02/2023, 9:42 PMget_object
and upload_object
work with a single file at a time. You'd have to iterate over the files under a specific lakeFS path and create required subdirectories in your local storage to achieve what you are looking for.