https://lakefs.io/ logo
Powered by
Title
r

Ronnie Ning

11/23/2022, 6:28 PM
if we run lakefs local settings, it will create a folder called 
lakefs
under the current user. Can we customize where 
lakefs
can be?
y

Yoni Augarten

11/23/2022, 6:32 PM
Hey @Ronnie Ning, yes! You need to create a config file for lakeFS, with the following settings: 
blockstore:
  type: local
  local:
    path: "/path/to/data"
Then run lakeFS like so: 
lakefs run --local-settings --config example-config.yaml
r

Ronnie Ning

11/23/2022, 6:35 PM
Thanks
👍🏻 1
"/path/to/data"
here it means the path for`lakefs` or 
lakefs/data
?
y

Yoni Augarten

11/23/2022, 6:45 PM
The data will be saved under this path. It will not add anything to it
r

Ronnie Ning

11/23/2022, 6:46 PM
you mean 
lakefs/data/block
?
y

Yoni Augarten

11/23/2022, 6:47 PM
I'm not sure what you're asking.
lakeFS needs a storage space to operate on. It will use the given path as this space.
Why are you concerned about the internal structure of this data?
r

Ronnie Ning

11/23/2022, 6:50 PM
since I mount s3 as local storage and run lakefs locally and try to keep data in s3.
i

Iddo Avneri

11/23/2022, 6:53 PM
Do you want to run lakeFS locally and manage data on an S3 bucket?
r

Ronnie Ning

11/23/2022, 7:01 PM
Since I can not load data into s3 directly when I run 
lakefs run --config s3.yaml
due to sse issue, I am trying to mount s3 as local storage and load data on it. I found lakefs created a folder called 
lakefs
which has sub-folder 
data
and 
metadata
. 
lakefs/data/block
is the place to keep files. Folder 
lakefs
is under current user who run the lakefs. If I want to put 
lakefs
on the mounted S3 local storage, how to do it.
y

Yoni Augarten

11/23/2022, 7:10 PM
Suppose you mount to the local path /a/b/c, use /a/b/c as the value for the configuration.
r

Ronnie Ning

11/23/2022, 7:17 PM
figured it out. 
"/path/to/data"
in config is the path for 
lakefs/data/block
n

Niro

11/23/2022, 7:18 PM
@Ronnie Ning I feel the need to emphasize that using local settings (either for the ref-store or the block storage) is not a viable production setup
r

Ronnie Ning

11/23/2022, 7:20 PM
yeah, for exploration purpose. we still want to solve the aws sse issue so that we can load files directly on s3.
👍🏽 1
For lakefs api, there are several places to put token / keys in the examples. Do I need to set them all or just one of them? how do I create them? I did not find any places in the gui for their creation.
b

Barak Amar

11/28/2022, 3:45 PM
The credentials you create in the setup process. Or new ones you can create under http://localhost:8000/auth/credentials should be use by the API. You need to associate the credentials to your client and it will pass them on each request. If you have a specific code example / question I can be more specific.
r

Ronnie Ning

11/28/2022, 3:47 PM
But the credentials are used to create configuration and the api token / keys are on top of the config. Check the example https://pydocs.lakefs.io/docs/ObjectsApi.html
b

Barak Amar

11/28/2022, 3:52 PM
And the config is pass to an ApiClient to get api instance that you can use
any call from the api instance (api_client variable in the example) uses the configuration, and use the credentials.
https://docs.lakefs.io/integrations/python.html
Check the first example - we created a client object that holds all APIs instances
from lakefs_client.client import LakeFSClient
if you create a single instance of this client - you can access all API sets. Example: 
client.repositories.<method>
for all api calls associated with repositories
r

Ronnie Ning

11/28/2022, 3:55 PM
So the cookie_auth, jwt_token, and oidc_auth are just one of the auths, if I already set up configuration, I don't need to set one of them?
b

Barak Amar

11/28/2022, 3:58 PM
The above uses 'http basic auth' for passing credentials - lakeFS also support JWT token as part of the headers and cookie based checks.
r

Ronnie Ning

11/28/2022, 4:00 PM
client.repositories.<method>
where I can find docs for it?
b

Barak Amar

11/28/2022, 4:02 PM
Only the page about and the code - it is a small helper that instantiate all the types on initialize.
https://github.com/treeverse/lakeFS/blob/10a4be3e8df8aee193177785e7c6f4b218336ec2/clients/python/lakefs_client/client.py
This client is not required, in some cases it can simplify the code
r

Ronnie Ning

11/28/2022, 4:04 PM
The doc you mentioned above is not complete, just a part of the code since I can not find much help related repo actions I guess
b

Barak Amar

11/28/2022, 4:05 PM
https://pydocs.lakefs.io/ holds the complete api documentation
the client.repositories holds instance of RepositoriesApi
the one from https://pydocs.lakefs.io/
class LakeFSClient:
    def __init__(self, configuration=None, header_name=None, header_value=None, cookie=None, pool_threads=1):
        if configuration:
            configuration = LakeFSClient._ensure_endpoint(configuration)
        self._api = _WrappedApiClient(configuration=configuration, header_name=header_name,
                                          header_value=header_value, cookie=cookie, pool_threads=pool_threads)
        for key, value in inspect.getmembers(sys.modules['lakefs_client.apis'], inspect.isclass):
            name = key.lower()
            if not name.endswith(_API_CLASS_SUFFIX):
                continue
            api_instance = value(self._api)
            attr_name = name[:-len(_API_CLASS_SUFFIX)]
            setattr(self, attr_name, api_instance)
            setattr(self, attr_name + '_api', api_instance)
this is the LakeFSClient - it is a collection of all lakefs_client.apis instances
I've referenced it as I thought you wanted to pass the configuration once and use all the APIs
The generated code you need to pass the configuration to each API set/collection.
So this helper holds all the sets.
r

Ronnie Ning

11/28/2022, 4:12 PM
The example I put above shows I have to use cookie_auth, jwt_key, oidc_auth on top of configuration in order to use lakefs_client to do api works, which makes me confused.
b

Barak Amar

11/28/2022, 4:14 PM
The docs are generated based on the swagger spec - the server support multiple methods. The non generated docs explains the simple way to work with the API.
The server support for cookies is relevant for example to the web UI - it is easier to manage login/logout using cookies.
Hope the above helps - let me know if there is a specific use case you are trying to execute using the API
👍 1
r

Ronnie Ning

11/28/2022, 4:18 PM
Thanks
40 Views
#lakefs-for-beginnersJoin Slack
Powered by