Title
#lakefs-for-beginners
r

Ronnie Ning

11/23/2022, 6:28 PM
if we run lakefs local settings, it will create a folder called
lakefs
under the current user. Can we customize where
lakefs
can be?
Yoni Augarten

Yoni Augarten

11/23/2022, 6:32 PM
Hey @Ronnie Ning, yes! You need to create a config file for lakeFS, with the following settings:
blockstore:
  type: local
  local:
    path: "/path/to/data"
Then run lakeFS like so:
lakefs run --local-settings --config example-config.yaml
r

Ronnie Ning

11/23/2022, 6:35 PM
6:43 PM
"/path/to/data"
here it means the path forlakefs or
lakefs/data
?
Yoni Augarten

Yoni Augarten

11/23/2022, 6:45 PM
The data will be saved under this path. It will not add anything to it
r

Ronnie Ning

11/23/2022, 6:46 PM
you mean
lakefs/data/block
?
Yoni Augarten

Yoni Augarten

11/23/2022, 6:47 PM
I'm not sure what you're asking.
6:48 PM
lakeFS needs a storage space to operate on. It will use the given path as this space.
6:48 PM
Why are you concerned about the internal structure of this data?
r

Ronnie Ning

11/23/2022, 6:50 PM
since I mount s3 as local storage and run lakefs locally and try to keep data in s3.
Iddo Avneri

Iddo Avneri

11/23/2022, 6:53 PM
Do you want to run lakeFS locally and manage data on an S3 bucket?
r

Ronnie Ning

11/23/2022, 7:01 PM
Since I can not load data into s3 directly when I run
lakefs run --config s3.yaml
due to sse issue, I am trying to mount s3 as local storage and load data on it. I found lakefs created a folder called
lakefs
which has sub-folder
data
and
metadata
.
lakefs/data/block
is the place to keep files. Folder
lakefs
is under current user who run the lakefs. If I want to put
lakefs
on the mounted S3 local storage, how to do it.
Yoni Augarten

Yoni Augarten

11/23/2022, 7:10 PM
Suppose you mount to the local path /a/b/c, use /a/b/c as the value for the configuration.
r

Ronnie Ning

11/23/2022, 7:17 PM
figured it out.
"/path/to/data"
in config is the path for
lakefs/data/block
n

Niro

11/23/2022, 7:18 PM
@Ronnie Ning I feel the need to emphasize that using local settings (either for the ref-store or the block storage) is not a viable production setup
r

Ronnie Ning

11/23/2022, 7:20 PM
yeah, for exploration purpose. we still want to solve the aws sse issue so that we can load files directly on s3.
3:26 PM
For lakefs api, there are several places to put token / keys in the examples. Do I need to set them all or just one of them? how do I create them? I did not find any places in the gui for their creation.
Barak Amar

Barak Amar

11/28/2022, 3:45 PM
The credentials you create in the setup process. Or new ones you can create under http://localhost:8000/auth/credentials should be use by the API. You need to associate the credentials to your client and it will pass them on each request. If you have a specific code example / question I can be more specific.
r

Ronnie Ning

11/28/2022, 3:47 PM
But the credentials are used to create configuration and the api token / keys are on top of the config. Check the example https://pydocs.lakefs.io/docs/ObjectsApi.html
Barak Amar

Barak Amar

11/28/2022, 3:52 PM
And the config is pass to an ApiClient to get api instance that you can use
3:52 PM
any call from the api instance (api_client variable in the example) uses the configuration, and use the credentials.
3:53 PM
Check the first example - we created a client object that holds all APIs instances
3:53 PM
from lakefs_client.client import LakeFSClient
3:54 PM
if you create a single instance of this client - you can access all API sets. Example:
client.repositories.<method>
for all api calls associated with repositories
r

Ronnie Ning

11/28/2022, 3:55 PM
So the cookie_auth, jwt_token, and oidc_auth are just one of the auths, if I already set up configuration, I don't need to set one of them?
Barak Amar

Barak Amar

11/28/2022, 3:58 PM
The above uses 'http basic auth' for passing credentials - lakeFS also support JWT token as part of the headers and cookie based checks.
r

Ronnie Ning

11/28/2022, 4:00 PM
client.repositories.<method>
where I can find docs for it?
Barak Amar

Barak Amar

11/28/2022, 4:02 PM
Only the page about and the code - it is a small helper that instantiate all the types on initialize.
4:04 PM
This client is not required, in some cases it can simplify the code
r

Ronnie Ning

11/28/2022, 4:04 PM
The doc you mentioned above is not complete, just a part of the code since I can not find much help related repo actions I guess
Barak Amar

Barak Amar

11/28/2022, 4:05 PM
https://pydocs.lakefs.io/ holds the complete api documentation
4:05 PM
the client.repositories holds instance of RepositoriesApi
4:05 PM
4:06 PM
class LakeFSClient:
    def __init__(self, configuration=None, header_name=None, header_value=None, cookie=None, pool_threads=1):
        if configuration:
            configuration = LakeFSClient._ensure_endpoint(configuration)
        self._api = _WrappedApiClient(configuration=configuration, header_name=header_name,
                                          header_value=header_value, cookie=cookie, pool_threads=pool_threads)
        for key, value in inspect.getmembers(sys.modules['lakefs_client.apis'], inspect.isclass):
            name = key.lower()
            if not name.endswith(_API_CLASS_SUFFIX):
                continue
            api_instance = value(self._api)
            attr_name = name[:-len(_API_CLASS_SUFFIX)]
            setattr(self, attr_name, api_instance)
            setattr(self, attr_name + '_api', api_instance)
4:06 PM
this is the LakeFSClient - it is a collection of all lakefs_client.apis instances
4:07 PM
I've referenced it as I thought you wanted to pass the configuration once and use all the APIs
4:07 PM
The generated code you need to pass the configuration to each API set/collection.
4:08 PM
So this helper holds all the sets.
r

Ronnie Ning

11/28/2022, 4:12 PM
The example I put above shows I have to use cookie_auth, jwt_key, oidc_auth on top of configuration in order to use lakefs_client to do api works, which makes me confused.
Barak Amar

Barak Amar

11/28/2022, 4:14 PM
The docs are generated based on the swagger spec - the server support multiple methods. The non generated docs explains the simple way to work with the API.
4:15 PM
The server support for cookies is relevant for example to the web UI - it is easier to manage login/logout using cookies.
4:16 PM
Hope the above helps - let me know if there is a specific use case you are trying to execute using the API
r

Ronnie Ning

11/28/2022, 4:18 PM
Thanks