Context: using the LakeFS API or Python SDK to fet...
# help
o
Context: using the LakeFS API or Python SDK to fetch list of repositories and other API's Problem: First, making the
v1/auth/login
API call or the
Client
from SDK. They are successful with 200. Login API call returns the
token
and
token_expiration
However, when subsequently trying to call
/api/v1/repositories
, I m getting 401
error authenticating request
Question: Do I need to attach the login token being returned in order to make subsequent calls? If so, how?
API implementation with Python Streamlit:
Copy code
def login_to_lakefs():
    """Authenticate with LakeFS and return the access token and response."""
    login_url = "<https://url/api/v1/auth/login>"
    credentials = {
        "access_key_id": "xxx",
        "secret_access_key": "yyy"
    }

    try:
        response = <http://requests.post|requests.post>(login_url, json=credentials)
        ...

def get_repositories(access_token):
    """Fetch repositories using the provided access token."""
    repos_url = "<https://lakefs.us-east-1.devfangmik.data-catalog.nile.amazon.dev/api/v1/repositories>"
    # How do I attach the login token as header here????
    headers = {
        "Authorization": f"Bearer {access_token}"
    }

    try:
        <http://st.info|st.info>("Fetching repositories...")
        response = requests.get(repos_url, headers=headers)
SDK implementation with Python Streamlit
Copy code
import streamlit as st
import lakefs
from lakefs.client import Client

def initialize_lakefs_client():
    """Initialize the lakeFS client with credentials."""
    try:
        client = Client(
            host="<https://url>",
            username="xxx",
            password="yyy"
        )
        return client
    except Exception as e:
        st.error(f"Failed to initialize lakeFS client: {str(e)}")
        return None
Copy code
def main():
    st.title("LakeFS Repository Explorer")

    # Initialize client
    with st.spinner("Connecting to LakeFS..."):
        client = initialize_lakefs_client()

        if client:
            st.success("Successfully connected to LakeFS!")
            try:
                # List repositories
                st.subheader("Available Repositories")
                <http://st.info|st.info>(f"repo call resp: {lakefs.repositories(client=client)}")
                repos = list(lakefs.repositories(client=client))
i
Hey @Ocean Chang, that login endpoint is best suited for UI flows. Follow the docs for best practices of the python sdk
m
@Itai Admi When I pass a valid username and password (access_key and secret_key) to the Python API Client for an endpoint, I end up getting a 401 credentials not found using the default built-in authenticator. However, this username and password is able to be used to log in to the web UI. Is there something I am missing with regards to how the authentication works for the API? The way this initial admin user is created is with the /setup_lakefs during deployment onto an ECS cluster. We are creating the keys for this initial user with randomly generated strings from AWS secret manager
Here is what I see in the server logs
Copy code
time="2024-11-08T06:09:27Z" level=info msg="Failed to authenticate user" func=pkg/auth.ChainAuthenticator.AuthenticateUser file="lakeFS/pkg/auth/authenticator.go:43" error="1 error occurred:\n\t* built in authenticator: credentials not found\n\n" host=lakefs.us-east-1.data-catalog.test.dev method=GET operation_id=GetRepository path=/api/v1/repositories/test username="mO%B?&_[=P96+/xl-R-b(0id{"

time="2024-11-08T06:09:27Z" level=error msg=authenticate func=pkg/api.userByAuth file="lakeFS/pkg/api/auth_middleware.go:409" error="1 error occurred:\n\t* built in authenticator: credentials not found\n\n" host=lakefs.us-east-1.data-catalog.test.dev method=GET operation_id=GetRepository path=/api/v1/repositories/test service=api_gateway user="mO%B?&_[=P96+/xl-R-b(0id{"
i
The user name value seems weird.. not sure I understand how you create the keys
m
We are creating the keys on our own with secret manager calling /setup_lakefs API endpoint
Here is an example:
Copy code
logging.info("Checking LakeFS Setup State")
            internal_api = lakefs_sdk.InternalApi(api)
            setup_state = internal_api.get_setup_state()
            logging.info(f"The current state of the LakeFS instance is {setup_state}")
            if setup_state.state == "not_initialized":
                logging.info(f"LakeFS not initialized, setting up initial LakeFS Admin Account")
                setup_params = lakefs_sdk.Setup.from_dict({
                    "username": "admin",
                    "key": {
                        "access_key_id": helper.download_secret_value(os.environ.get("LAKEFS_ADMIN_ACCESS_KEY_ID_SECRET_ARN")),
                        "secret_access_key": helper.download_secret_value(os.environ.get("LAKEFS_ADMIN_SECRET_KEY_ID_SECRET_ARN"))
                    }
                })
                setup_response = internal_api.setup(setup_params)
                logging.info(f"The setup was successful: {setup_response}")
i
We normally see setup requests with access key and secret that match AWS keys format. But we don’t enforce it in the API and it should work with any key pair. It’s possible that we have some encoding issue with non-standard formats. I’ll open a bug for that, meanwhile I suggest you try: 1. Doing a setup request without passing creds. lakeFS will generate the admin keys instead and you can save them in the secrets manager (one time operation for a lakeFS installation) 2. Passing AWS-like key and secret
btw what’s the output of the setup request?
m
Ok so if I pass nothing to the setup endpoint lakeFS will autogenerate in the default format?
Here is an example of the output of the setup request:
INFO:root:The
setup was successful:
access_key_id='^K1y:.Il&_bfd_,k;\\V12P3Rg3&S9Je+'
secret_access_key='#eN_ry5$v~L^R*8mGp)!p5lSc=yZ"\'M|'
creation_date=1730914328
i
Ok so if I pass nothing to the setup endpoint lakeFS will autogenerate in the default format?
Yes
Something feels off with the secret, it has a
'
near the end.
m
Ah we don't make any modification this is auto generated by AWS
o
@Itai Admi, https://docs.lakefs.io/security/authentication.html Do you know why the Authorization header here is not working and getting 401 error?
Copy code
Authorization: Basic <base64 encoded access_key_id:secret_access_key>
m
Thanks for your help Itai, looks like using autogenerated credentials works!
👌 1
o
@Itai Admi, I tried with the SDK login method documented here: https://docs.lakefs.io/integrations/python.html#using-the-lakefs-sdk
Copy code
import lakefs
from lakefs.client import Client

clt = Client(
    host="<http://localhost:8000>",
    username="AKIAIOSFODNN7EXAMPLE",
    password="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
)

lakefs.repositories():
still getting 401 error from the
lakefs.repositories():
i
Can you login with the same creds through the UI?
o
yes. can manually login with the same UI, and confirmed the base64 encoding is working as expected for the API implementation.
Copy code
credentials = f"{access_key_id}:{secret_access_key}"
    encoded_credentials = base64.b64encode(credentials.encode('utf-8')).decode('utf-8')
Also tried a different SDK implementation following this Code:
Copy code
import lakefs_client
from lakefs_client.client import LakeFSClient

lakefs_config = lakefs_client.Configuration()
lakefs_config.username = 'xxx'
lakefs_config.password = 'yyy'
lakefs_config.host = 'host url'

lakefs = LakeFSClient(lakefs_config)
lakefs_api_client = lakefs_client.ApiClient(lakefs_config)
repo=lakefs.repositories.list_repositories().results[0]
Copy code
UnauthorizedException: (401) Reason: Unauthorized HTTP response headers: HTTPHeaderDict({'Date': 'Fri, 08 Nov 2024 08:45:42 GMT', 'Content-Type': 'application/json', 'Content-Length': '43', 'Connection': 'keep-alive', 'X-Content-Type-Options': 'nosniff', 'X-Request-Id': 'xxx'}) HTTP response body: {"message":"error authenticating request"}
i
Can you share lakeFS server logs too?
i
@Mike Fang - I thought you configured fluffy and SSO - no? Why are you using the default authenticator?
m
@Iddo Avneri You might be referring to a different person with the same name, I have been using default auth in my own application