https://lakefs.io/ logo
#help
Title
# help
r

Robin Moffatt

05/31/2023, 12:10 PM
I'm trying to use Hadoop FS but hitting
Failed to get lakeFS blockstore type
. Details in notebook - could someone point where I'm going wrong please? I'm using Everything Bagel with MinIO for storage https://gist.github.com/rmoff/c3d6553aec11b569d8f1e1761b7182e5
n

Niro

05/31/2023, 1:28 PM
Do you have the right permissions for this request? GetStorageConfig requires ReadConfigAction permissions
r

Robin Moffatt

05/31/2023, 1:29 PM
permissions on the storage? or lakeFS?
n

Niro

05/31/2023, 1:29 PM
lakefs permissions
r

Robin Moffatt

05/31/2023, 1:30 PM
I'm using a user that's in the
Admins
group
n

Niro

05/31/2023, 1:40 PM
Is there anyway to get some more information on the error? What status code was returned from lakefs?
r

Robin Moffatt

05/31/2023, 2:13 PM
I don't see any message in the lakeFS log
n

Niro

05/31/2023, 3:31 PM
What's the lakeFS log level?
r

Robin Moffatt

05/31/2023, 4:01 PM
it was
INFO
. Even with it set to
DEBUG
I don't see anything from lakeFS, just the notebook
Copy code
jupyter-notebook  | 23/05/31 16:01:03 WARN FileSystem: Failed to initialize fileystem <lakefs://test/main/users>: java.io.IOException: Failed to get lakeFS blockstore type
ah, if I turn up Spark logging I get this:
Copy code
jupyter-notebook  | 23/05/31 16:05:39 DEBUG FileSystem: Starting: Acquiring creator semaphore for <lakefs://test/main/users>
jupyter-notebook  | 23/05/31 16:05:39 DEBUG FileSystem: Acquiring creator semaphore for <lakefs://test/main/users>: duration 0:00.001s
jupyter-notebook  | 23/05/31 16:05:39 DEBUG FileSystem: Starting: Creating FS <lakefs://test/main/users>
jupyter-notebook  | 23/05/31 16:05:39 DEBUG FileSystem: Looking for FS supporting lakefs
jupyter-notebook  | 23/05/31 16:05:39 DEBUG FileSystem: looking for configuration option fs.lakefs.impl
jupyter-notebook  | 23/05/31 16:05:39 DEBUG FileSystem: Filesystem lakefs defined in configuration option
jupyter-notebook  | 23/05/31 16:05:39 DEBUG FileSystem: FS for lakefs is class io.lakefs.LakeFSFileSystem
jupyter-notebook  | 23/05/31 16:05:39 DEBUG FsUrlStreamHandlerFactory: Creating handler for protocol http
jupyter-notebook  | 23/05/31 16:05:39 DEBUG FsUrlStreamHandlerFactory: Unknown protocol http, delegating to default implementation
jupyter-notebook  | 23/05/31 16:05:39 WARN FileSystem: Failed to initialize fileystem <lakefs://test/main/users>: java.io.IOException: Failed to get lakeFS blockstore type
jupyter-notebook  | 23/05/31 16:05:39 DEBUG FileSystem: Failed to initialize fileystem
jupyter-notebook  | java.io.IOException: Failed to get lakeFS blockstore type
n

Niro

05/31/2023, 4:07 PM
Copy code
jupyter-notebook  | 23/05/31 16:05:39 DEBUG FsUrlStreamHandlerFactory: Creating handler for protocol http
jupyter-notebook  | 23/05/31 16:05:39 DEBUG FsUrlStreamHandlerFactory: Unknown protocol http, delegating to default implementation
This looks suspicious
Can you check if you can curl to the endpoint directly?
r

Robin Moffatt

05/31/2023, 4:08 PM
the lakeFS or minio one?
n

Niro

05/31/2023, 4:08 PM
lakeFS
r

Robin Moffatt

05/31/2023, 4:10 PM
I'll need to install curl on the container
it's the same endpoint that I use from the lakeFS client on the same container though, so the network should be good.
n

Niro

05/31/2023, 4:11 PM
I suspect the spark configuration is the problem - just wanted to make sure the lakefs endpoint you provided is valid
👍 1
r

Robin Moffatt

05/31/2023, 4:12 PM
Copy code
root@be6ae10d9902:/# curl <http://lakefs:8000/api/v1>
{"message":"invalid API endpoint"}
n

Niro

05/31/2023, 4:13 PM
Copy code
curl <http://lakefs:8000/_health>
r

Robin Moffatt

05/31/2023, 4:13 PM
Copy code
# curl <http://lakefs:8000/_health>
alive! c0d5c906-e316-4029-b093-6b6057d079fc
👍🏽 1
(TIL 🙂 )
😄 1
n

Niro

05/31/2023, 4:15 PM
So I suspect some thing with the spark configuration is not right. Specifically this might be a direction to investigate:
Copy code
jupyter-notebook  | 23/05/31 16:05:39 DEBUG FsUrlStreamHandlerFactory: Creating handler for protocol http
jupyter-notebook  | 23/05/31 16:05:39 DEBUG FsUrlStreamHandlerFactory: Unknown protocol http, delegating to default implementation
Unfortunately I'm not a spark expert. Maybe @Yoni Augarten or @Ariel Shaqed (Scolnicov) can help?
a

Ariel Shaqed (Scolnicov)

06/01/2023, 7:52 AM
@Yoni Augarten could this be a manifestation of what you fixed with this PR?
r

Robin Moffatt

06/01/2023, 8:20 AM
I've logged https://github.com/treeverse/lakeFS/issues/5997 to track and for visibility beyond the walls of Slack 🙂
43 Views