I m trying to use Hadoop FS but hitting `Failed to get lakeF

Question

I m trying to use Hadoop FS but hitting `Failed to get lakeFS blockstore type` Details in notebook could someone point where I m going wrong please I m using Everything Bagel with MinIO for storage <https gist github com rmoff c3d6553aec11b569d8f1e1761b7182e5>

Niro · Answer

Do you have the right permissions for this request GetStorageConfig requires ReadConfigAction permissions

Robin Moffatt · Answer

permissions on the storage or lakeFS

Niro · Answer

lakefs permissions

Robin Moffatt · Answer

I m using a user that s in the `Admins` group

Niro · Answer

Is there anyway to get some more information on the error What status code was returned from lakefs

Robin Moffatt · Answer

I don t see any message in the lakeFS log

Niro · Answer

What s the lakeFS log level

Robin Moffatt · Answer

it was `INFO` Even with it set to `DEBUG` I don t see anything from lakeFS just the notebook ```jupyter notebook | 23 05 31 16 01 03 WARN FileSystem Failed to initialize fileystem <lakefs test main users> java io IOException Failed to get lakeFS blockstore type```

Robin Moffatt · Answer

ah if I turn up Spark logging I get this ```jupyter notebook | 23 05 31 16 05 39 DEBUG FileSystem Starting Acquiring creator semaphore for jupyter notebook | 23 05 31 16 05 39 DEBUG FileSystem Acquiring creator semaphore for duration 0 00 001s jupyter notebook | 23 05 31 16 05 39 DEBUG FileSystem Starting Creating FS jupyter notebook | 23 05 31 16 05 39 DEBUG FileSystem Looking for FS supporting lakefs jupyter notebook | 23 05 31 16 05 39 DEBUG FileSystem looking for configuration option fs lakefs impl jupyter notebook | 23 05 31 16 05 39 DEBUG FileSystem Filesystem lakefs defined in configuration option jupyter notebook | 23 05 31 16 05 39 DEBUG FileSystem FS for lakefs is class io lakefs LakeFSFileSystem jupyter notebook | 23 05 31 16 05 39 DEBUG FsUrlStreamHandlerFactory Creating handler for protocol http jupyter notebook | 23 05 31 16 05 39 DEBUG FsUrlStreamHandlerFactory Unknown protocol http delegating to default implementation jupyter notebook | 23 05 31 16 05 39 WARN FileSystem Failed to initialize fileystem java io IOException Failed to get lakeFS blockstore type jupyter notebook | 23 05 31 16 05 39 DEBUG FileSystem Failed to initialize fileystem jupyter notebook | java io IOException Failed to get lakeFS blockstore type```

Niro · Answer

```jupyter notebook | 23 05 31 16 05 39 DEBUG FsUrlStreamHandlerFactory Creating handler for protocol http jupyter notebook | 23 05 31 16 05 39 DEBUG FsUrlStreamHandlerFactory Unknown protocol http delegating to default implementation``` This looks suspicious

Niro · Answer

Can you check if you can curl to the endpoint directly

Robin Moffatt · Answer

the lakeFS or minio one

Niro · Answer

lakeFS

Robin Moffatt · Answer

I ll need to install curl on the container

Robin Moffatt · Answer

it s the same endpoint that I use from the lakeFS client on the same container though so the network should be good

Niro · Answer

I suspect the spark configuration is the problem just wanted to make sure the lakefs endpoint you provided is valid

Robin Moffatt · Answer

```root be6ae10d9902 curl <http lakefs 8000 api v1> message invalid API endpoint ```

Niro · Answer

```curl <http lakefs 8000 health>```

Robin Moffatt · Answer

``` curl <http lakefs 8000 health> alive c0d5c906 e316 4029 b093 6b6057d079fc```

Robin Moffatt · Answer

TIL slightly smiling face

Niro · Answer

So I suspect some thing with the spark configuration is not right Specifically this might be a direction to investigate ```jupyter notebook | 23 05 31 16 05 39 DEBUG FsUrlStreamHandlerFactory Creating handler for protocol http jupyter notebook | 23 05 31 16 05 39 DEBUG FsUrlStreamHandlerFactory Unknown protocol http delegating to default implementation``` Unfortunately I m not a spark expert Maybe < Yoni Augarten> or < Ariel Shaqed Scolnicov > can help

Ariel Shaqed (Scolnicov) · Answer

< Yoni Augarten> could this be a manifestation of what you fixed with

Robin Moffatt · Answer

I ve logged <https github com treeverse lakeFS issues 5997> to track and for visibility beyond the walls of Slack slightly smiling face