What is a fast way to retrieve physicalAddress ? C...
# help
h
What is a fast way to retrieve physicalAddress ? Currently:
Copy code
client = lakefs_sdk.client.LakeFSClient(lakefs_conf)

res = []
for object_path in paths:
    response = client.objects_api.stat_object(repository=repo,
                                                ref=commit,
                                                path=object_path,
                                                presign=presign)

    res.append(response.physical_address)
Can client be used in multi processing ?
Copy code
import concurrent.futures
import os

def stat_object_path(args):
    client, repo, commit, presign, object_path = args
    response = client.objects_api.stat_object(
        repository=repo,
        ref=commit,
        path=object_path,
        presign=presign
    )
    return response.physical_address

def get_physical_addresses(client, repo, commit, presign, paths):
    with concurrent.futures.ThreadPoolExecutor(max_workers=alot) as executor:
        res = list(executor.map(
            stat_object_path,
            [(client, repo, commit, presign, p) for p in paths]
        ))
    return res

client = lakefs_sdk.client.LakeFSClient(lakefs_conf)
res = get_physical_addresses(client, repo, commit, presign, paths)
n
Stat is the way to go. You can definitely use multiprocessing. Also I recommend using the high level sdk stat for simpler code
👍 1
b
You can get the data as part of listing