Miguel Rodríguez
01/04/2023, 10:19 PMMissing Credential Scope
error, which I guess comes from lakeFS not being able to authenticate to the storage account where the files actually are.
The ingest
command worked and could authenticate cause I set the Azure Storage Account key in the AZURE_STORAGE_ACCESS_KEY
environment variable locally, but how can I make LakeFS authenticate later to read the files when I need them?Iddo Avneri
01/04/2023, 10:23 PMMiguel Rodríguez
01/04/2023, 10:24 PMIddo Avneri
01/04/2023, 10:25 PMMiguel Rodríguez
01/04/2023, 10:29 PMAmit Kesarwani
01/04/2023, 10:39 PMlakectl import
command instead of lakectl ingest
? As per documentation:
The lakectl cli supports import and ingest commands to import objects from an external source.
• The import command acts the same as the UI import wizard. It imports (zero copy) and commits the changes on _<branch_name>_imported branch with an optional flag to also merge the changes to <branch_name>.
• The Ingest is listing the source bucket (and optional prefix) from the client, and creating pointers to the returned objects in lakeFS. The objects will be staged on the branch.Miguel Rodríguez
01/04/2023, 10:41 PMlakectl import
because the storage account I want to import from is different from my LakeFS underlying storage accountIddo Avneri
01/04/2023, 10:44 PMMiguel Rodríguez
01/04/2023, 10:52 PMIddo Avneri
01/04/2023, 10:57 PMAmit Kesarwani
01/04/2023, 11:06 PMblockstore:
type: azure
azure:
auth_method: access-key
storage_account: "abc"
storage_access_key: "xxxxxxxx"
lakectl ingest
command works with 2 different storage accounts. Please use following commands:
lakectl config
Access key ID: <lakeFS Access Key>
Secret access key: <lakeFS Secret Key>
Server endpoint URL: http://<lakeFS_server_ip_address>:8000/api/v1
export AZURE_STORAGE_ACCOUNT="from_storage_account"
export AZURE_STORAGE_ACCESS_KEY="access_key_for_storage_account"
lakectl ingest \
--from <https://from_storage_account.blob.core.windows.net/container/> \
--to <lakefs://my-repo/main/>
Niro
01/05/2023, 8:17 AMMiguel Rodríguez
01/05/2023, 10:05 AMAmit Kesarwani
01/05/2023, 4:18 PMMiguel Rodríguez
01/05/2023, 5:29 PMAmit Kesarwani
01/05/2023, 5:42 PM