https://lakefs.io/ logo
Join Slack
Powered by
if we run lakefs local settings, it will create a ...
# help
u
if we run lakefs local settings, it will create a folder called 
lakefs
under the current user. Can we customize where 
lakefs
can be?
u
Hey @Ronnie Ning, yes! You need to create a config file for lakeFS, with the following settings:
Copy code
blockstore:
  type: local
  local:
    path: "/path/to/data"
Then run lakeFS like so:
Copy code
lakefs run --local-settings --config example-config.yaml
u
Thanks
u
Copy code
"/path/to/data"
here it means the path for`lakefs` or 
lakefs/data
?
u
The data will be saved under this path. It will not add anything to it
u
you mean 
lakefs/data/block
?
u
I'm not sure what you're asking.
u
lakeFS needs a storage space to operate on. It will use the given path as this space.
u
Why are you concerned about the internal structure of this data?
u
since I mount s3 as local storage and run lakefs locally and try to keep data in s3.
u
Do you want to run lakeFS locally and manage data on an S3 bucket?
u
Since I can not load data into s3 directly when I run 
lakefs run --config s3.yaml
due to sse issue, I am trying to mount s3 as local storage and load data on it. I found lakefs created a folder called 
lakefs
which has sub-folder 
data
and 
metadata
. 
lakefs/data/block
is the place to keep files. Folder 
lakefs
is under current user who run the lakefs. If I want to put 
lakefs
on the mounted S3 local storage, how to do it.
u
Suppose you mount to the local path /a/b/c, use /a/b/c as the value for the configuration.
u
figured it out. 
"/path/to/data"
in config is the path for 
lakefs/data/block
u
@Ronnie Ning I feel the need to emphasize that using local settings (either for the ref-store or the block storage) is not a viable production setup
u
yeah, for exploration purpose. we still want to solve the aws sse issue so that we can load files directly on s3.
u
For lakefs api, there are several places to put token / keys in the examples. Do I need to set them all or just one of them? how do I create them? I did not find any places in the gui for their creation.
u
The credentials you create in the setup process. Or new ones you can create under http://localhost:8000/auth/credentials should be use by the API. You need to associate the credentials to your client and it will pass them on each request. If you have a specific code example / question I can be more specific.
u
But the credentials are used to create configuration and the api token / keys are on top of the config. Check the example https://pydocs.lakefs.io/docs/ObjectsApi.html
u
And the config is pass to an ApiClient to get api instance that you can use
u
any call from the api instance (api_client variable in the example) uses the configuration, and use the credentials.
u
u
Check the first example - we created a client object that holds all APIs instances
u
Copy code
from lakefs_client.client import LakeFSClient
u
if you create a single instance of this client - you can access all API sets. Example:
Copy code
client.repositories.<method>
for all api calls associated with repositories
u
So the cookie_auth, jwt_token, and oidc_auth are just one of the auths, if I already set up configuration, I don't need to set one of them?
u
The above uses 'http basic auth' for passing credentials - lakeFS also support JWT token as part of the headers and cookie based checks.
u
Copy code
client.repositories.<method>
where I can find docs for it?
u
Only the page about and the code - it is a small helper that instantiate all the types on initialize.
u
u
This client is not required, in some cases it can simplify the code
u
The doc you mentioned above is not complete, just a part of the code since I can not find much help related repo actions I guess
u
https://pydocs.lakefs.io/ holds the complete api documentation
u
the client.repositories holds instance of RepositoriesApi
u
u
Copy code
class LakeFSClient:
    def __init__(self, configuration=None, header_name=None, header_value=None, cookie=None, pool_threads=1):
        if configuration:
            configuration = LakeFSClient._ensure_endpoint(configuration)
        self._api = _WrappedApiClient(configuration=configuration, header_name=header_name,
                                          header_value=header_value, cookie=cookie, pool_threads=pool_threads)
        for key, value in inspect.getmembers(sys.modules['lakefs_client.apis'], inspect.isclass):
            name = key.lower()
            if not name.endswith(_API_CLASS_SUFFIX):
                continue
            api_instance = value(self._api)
            attr_name = name[:-len(_API_CLASS_SUFFIX)]
            setattr(self, attr_name, api_instance)
            setattr(self, attr_name + '_api', api_instance)
u
this is the LakeFSClient - it is a collection of all lakefs_client.apis instances
u
I've referenced it as I thought you wanted to pass the configuration once and use all the APIs
u
The generated code you need to pass the configuration to each API set/collection.
u
So this helper holds all the sets.
u
The example I put above shows I have to use cookie_auth, jwt_key, oidc_auth on top of configuration in order to use lakefs_client to do api works, which makes me confused.
u
The docs are generated based on the swagger spec - the server support multiple methods. The non generated docs explains the simple way to work with the API.
u
The server support for cookies is relevant for example to the web UI - it is easier to manage login/logout using cookies.
u
Hope the above helps - let me know if there is a specific use case you are trying to execute using the API
u
Thanks
Open in Slack
PreviousNext
Powered by