user
06/13/2022, 9:21 AMBroken DAG: [/opt/airflow/dags/dagRun.py] Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/lakefs_provider/hooks/lakefs_hook.py", line 26, in <module>
class LakeFSHook(BaseHook):
File "/home/airflow/.local/lib/python3.7/site-packages/lakefs_provider/hooks/lakefs_hook.py", line 106, in LakeFSHook
def log_commits(self, repo: str, ref: str, size: int=100) -> Iterator[Any]:
TypeError: 'ABCMeta' object is not subscriptable
user
06/13/2022, 9:24 AMuser
06/13/2022, 9:25 AMuser
06/13/2022, 9:26 AMuser
06/13/2022, 9:27 AMairflow-provider-lakeFS==0.42.0
.user
06/13/2022, 9:30 AMuser
06/13/2022, 9:30 AMuser
06/13/2022, 9:30 AMuser
06/13/2022, 9:31 AMuser
06/13/2022, 11:12 AMuser
06/15/2022, 6:48 AMuser
06/15/2022, 8:35 AMuser
06/15/2022, 10:15 AMuser
06/22/2022, 7:27 AMlakefs_client.exceptions.UnauthorizedException: (401)
for extra details, my code looks like this sense_eventData = LakeFSFileSensor(
task_id="sense_eventData_files",
repo='test',
branch='main',
lakefs_conn_id='conn_1',
path='bronzelayer/APIs/events.parquet'
)user
06/22/2022, 7:46 AM~/.lakectl.yaml
that looks like this (just fill in the 3 fields).
credentials:
access_key_id: AKIA...
secret_access_key: shhh...
server:
endpoint_url: <https://lakefs.example.com/api/v1>
HTH!user
06/22/2022, 9:47 AMuser
06/22/2022, 10:56 AMuser
06/22/2022, 11:28 AM~/.lakectl.yaml
you didn't manage to use lakectl? What error do you get?user
06/22/2022, 11:37 AMERROR [2022-06-22T10:10:32Z]lakeFS/pkg/api/auth_middleware.go:157 pkg/api.userByAuth authenticate error="2 errors occurred:\n\t* built in authenticator: could not decrypt value\n\t* email authenticator: not found: no rows in result set\n\n" host="137.184.147.128:8000" method=GET path="/api/v1/repositories/test/refs/main/objects/stat?path=bronzelayer%2FWEB%2Ffootball-data.parquet" request_id=6f72101a-43c4-4964-8e23-92683f509dfc service=api_gateway service_name=rest_api user=AKIAJSEJPLE4NOPT4KEQ
user
06/22/2022, 11:38 AMuser
06/22/2022, 11:40 AMuser
06/22/2022, 11:42 AMuser
06/22/2022, 11:54 AMlakectl
command with those credentials.
So, did you had the change of using lakectl before? Did you downloaded it?
Where did you get the last error?user
06/22/2022, 12:20 PMuser
06/22/2022, 12:23 PMuser
06/22/2022, 12:30 PM.lakefs.yaml
or .lakectl.yaml
?user
06/22/2022, 12:46 PMuser
06/22/2022, 12:49 PMuser
06/22/2022, 12:51 PMAKIA
... In particular, AFAIK even on lakeFS Cloud email+password work only for logging into the GUI -- and you must define a service credentials access key for lakectl and other REST API access.
2. Credentials to access lakeFS go into .lakectl.yaml
. And the access key and secret key there are those that you got from lakeFS -- not your S3 credentials.
Sorry if you've covered both of these. They are just my 2 favourite mistakes to make with lakeFS 🙂 , I probably get into a credentials mixup at least once a fortnight.user
06/22/2022, 12:57 PM