Miguel RodrĂguez
01/04/2023, 10:19 PMMissing Credential Scope
error, which I guess comes from lakeFS not being able to authenticate to the storage account where the files actually are.
The ingest
command worked and could authenticate cause I set the Azure Storage Account key in the AZURE_STORAGE_ACCESS_KEY
environment variable locally, but how can I make LakeFS authenticate later to read the files when I need them?Iddo Avneri
01/04/2023, 10:23 PMIddo Avneri
01/04/2023, 10:24 PMIddo Avneri
01/04/2023, 10:24 PMMiguel RodrĂguez
01/04/2023, 10:24 PMIddo Avneri
01/04/2023, 10:25 PMMiguel RodrĂguez
01/04/2023, 10:29 PMAmit Kesarwani
01/04/2023, 10:39 PMlakectl import
command instead of lakectl ingest
? As per documentation:
The lakectl cli supports import and ingest commands to import objects from an external source.
• The import command acts the same as the UI import wizard. It imports (zero copy) and commits the changes on _<branch_name>_imported branch with an optional flag to also merge the changes to <branch_name>.
• The Ingest is listing the source bucket (and optional prefix) from the client, and creating pointers to the returned objects in lakeFS. The objects will be staged on the branch.Miguel RodrĂguez
01/04/2023, 10:41 PMlakectl import
because the storage account I want to import from is different from my LakeFS underlying storage accountIddo Avneri
01/04/2023, 10:44 PMMiguel RodrĂguez
01/04/2023, 10:52 PMIddo Avneri
01/04/2023, 10:57 PMAmit Kesarwani
01/04/2023, 11:06 PMAmit Kesarwani
01/04/2023, 11:15 PMAmit Kesarwani
01/04/2023, 11:34 PMblockstore:
type: azure
azure:
auth_method: access-key
storage_account: "abc"
storage_access_key: "xxxxxxxx"
Amit Kesarwani
01/05/2023, 12:31 AMlakectl ingest
command works with 2 different storage accounts. Please use following commands:
lakectl config
Access key ID: <lakeFS Access Key>
Secret access key: <lakeFS Secret Key>
Server endpoint URL: http://<lakeFS_server_ip_address>:8000/api/v1
export AZURE_STORAGE_ACCOUNT="from_storage_account"
export AZURE_STORAGE_ACCESS_KEY="access_key_for_storage_account"
lakectl ingest \
--from <https://from_storage_account.blob.core.windows.net/container/> \
--to <lakefs://my-repo/main/>
Niro
01/05/2023, 8:17 AMMiguel RodrĂguez
01/05/2023, 10:05 AMAmit Kesarwani
01/05/2023, 4:18 PMMiguel RodrĂguez
01/05/2023, 5:29 PMAmit Kesarwani
01/05/2023, 5:42 PM