Miguel Rodríguez01/04/2023, 10:19 PM
error, which I guess comes from lakeFS not being able to authenticate to the storage account where the files actually are. The
Missing Credential Scope
command worked and could authenticate cause I set the Azure Storage Account key in the
environment variable locally, but how can I make LakeFS authenticate later to read the files when I need them?
Iddo Avneri01/04/2023, 10:23 PM
Miguel Rodríguez01/04/2023, 10:24 PM
Iddo Avneri01/04/2023, 10:25 PM
Miguel Rodríguez01/04/2023, 10:29 PM
Amit Kesarwani01/04/2023, 10:39 PM
command instead of
? As per documentation:
• The import command acts the same as the UI import wizard. It imports (zero copy) and commits the changes on _<branch_name>_imported branch with an optional flag to also merge the changes to <branch_name>. • The Ingest is listing the source bucket (and optional prefix) from the client, and creating pointers to the returned objects in lakeFS. The objects will be staged on the branch.
The lakectl cli supports import and ingest commands to import objects from an external source.
Miguel Rodríguez01/04/2023, 10:41 PM
because the storage account I want to import from is different from my LakeFS underlying storage account
Iddo Avneri01/04/2023, 10:44 PM
Miguel Rodríguez01/04/2023, 10:52 PM
Iddo Avneri01/04/2023, 10:57 PM
Amit Kesarwani01/04/2023, 11:06 PM
blockstore: type: azure azure: auth_method: access-key storage_account: "abc" storage_access_key: "xxxxxxxx"
command works with 2 different storage accounts. Please use following commands:
lakectl config Access key ID: <lakeFS Access Key> Secret access key: <lakeFS Secret Key> Server endpoint URL: http://<lakeFS_server_ip_address>:8000/api/v1 export AZURE_STORAGE_ACCOUNT="from_storage_account" export AZURE_STORAGE_ACCESS_KEY="access_key_for_storage_account" lakectl ingest \ --from <https://from_storage_account.blob.core.windows.net/container/> \ --to <lakefs://my-repo/main/>
Niro01/05/2023, 8:17 AM
Miguel Rodríguez01/05/2023, 10:05 AM
Amit Kesarwani01/05/2023, 4:18 PM
Miguel Rodríguez01/05/2023, 5:29 PM
Amit Kesarwani01/05/2023, 5:42 PM