• Paul Singman

    Paul Singman

    4 months ago
    Hello everyone! Question about the Docker Everything Bagel –– would it be possible to use the UI Import feature when running it locally?
    Paul Singman
    Eden Ohana
    3 replies
    Copy to Clipboard
  • Rao

    Rao

    4 months ago
    Hi there, I want to install LakeFS on my PC. Is it possible to install LakeFS without internet I could download exe or Deb file from LakeFS website, then from USB I paste this exe or Deb file to the PC
    Rao
    Itai David
    2 replies
    Copy to Clipboard
  • a

    Angela Busato

    3 months ago
    Hi there! I'm developing a web application and I need to store some files on the server where the application will be. I think lakeFs is the system that I'm searching but I don't understand if necessary to use a cloud. Does it manage the storing of the data in the server?
    a
    Tal Sofer
    11 replies
    Copy to Clipboard
  • Olagfigh

    Olagfigh

    3 months ago
    Hi there 🙂 I'm working with Kafka and wanted to try your LakeFS. S3LinkConnector works great with LakeFS, I'm successfuly sending data from Kafka to LakeFS repo, but I have a problem with S3SourceConnector. I'm using the same credentials, same store.url (and it's my LakeFS demo url), same topic etc. but it crashes with AmazonS3Exception. In logs it looks like this: ERROR Unable to read objects using the configured S3 credentials, bucket, and path. (io.confluent.connect.s3.source.S3SourceConnectorValidation:105) AmazonS3Exception: 400 BadRequest (Service: Amazon S3; Status Code: 400; Error Code: BadRequest: Request ID... Does anyone know what I did wrong? Is there any additional configuration/properties I should add to the source connector to make it work?
    Olagfigh
    Barak Amar
    +3
    22 replies
    Copy to Clipboard
  • j

    Jude

    3 months ago
    Hi everyone! So I have this pipeline that extracts data from a database and sends to lakefs periodically. Everything has been working fine for over a month now until I checked my aiflow dashboard and now getting this error
    Broken DAG: [/opt/airflow/dags/dagRun.py] Traceback (most recent call last):
    File "/home/airflow/.local/lib/python3.7/site-packages/lakefs_provider/hooks/lakefs_hook.py", line 26, in <module>
    class LakeFSHook(BaseHook):
    File "/home/airflow/.local/lib/python3.7/site-packages/lakefs_provider/hooks/lakefs_hook.py", line 106, in LakeFSHook
    def log_commits(self, repo: str, ref: str, size: int=100) -> Iterator[Any]:
    TypeError: 'ABCMeta' object is not subscriptable
    j
    Ariel Shaqed (Scolnicov)
    +1
    29 replies
    Copy to Clipboard
  • Marvellous

    Marvellous

    3 months ago
    Hi everyone! So, I am having this (SQLSTATE 57P01) error with LakeFS which has been persistent
    ERROR [2022-06-13T18:42:46+01:00]lakeFS/pkg/api/auth_middleware.go:142 pkg/api.userByAuth authenticate error="2 errors occurred:\n\t* built in authenticator: FATAL: terminating connection due to administrator command (SQLSTATE 57P01)\n\t* email authenticator: FATAL: terminating connection due to administrator command (SQLSTATE 57P01)\n\n" service=api_gateway user=AKIAJYXNNRWG66FDRZDQ
    In the event this error surfaces on the terminal log, I am always logged out from the LakeFS UI which prevents communication. Upon attempting to login again, I receive an “invalid credential” error which remains persistent until after several attempts. I have LakeFS deployed with digital ocean object storage using MinIO configuration. I am currently running on lakefs version 0.66.0 and postgres 12 for logging. These are the screenshots I get from LakeFS log, Postgres log, and LakeFS UI.
    Marvellous
    Barak Amar
    +1
    5 replies
    Copy to Clipboard
  • j

    Jude

    3 months ago
    Hi everyone. I am having trouble accessing the lakefs UI I keep getting this authentication issue error each time I try to login and it is still the same credentials I used to access it in the past that is failing now.
    FO   [2022-06-15T19:14:04Z]lakeFS/pkg/auth/authenticator.go:54 pkg/auth.ChainAuthenticator.AuthenticateUser Failed to authenticate user                   error="2 errors occurred:\n\t* built in authenticator: could not decrypt value\n\t* email authenticator: not found: no rows in result set\n\n" host="137.184.147.128:8000" method=POST path=/api/v1/auth/login request_id=2a7fad06-5a0f-4a36-9369-d18f60bff2da service_name=rest_api username=AKIAJSEJPLE4NOPT4KEQ
    ERROR  [2022-06-15T19:14:04Z]lakeFS/pkg/api/auth_middleware.go:157 pkg/api.userByAuth authenticate                                  error="2 errors occurred:\n\t* built in authenticator: could not decrypt value\n\t* email authenticator: not found: no rows in result set\n\n" service=api_gateway user=AKIAJSEJPLE4NOPT4KEQ
    INFO   [2022-06-15T19:14:06Z]lakeFS/pkg/auth/authenticator.go:54 pkg/auth.ChainAuthenticator.AuthenticateUser Failed to authenticate user                   error="2 errors occurred:\n\t* built in authenticator: could not decrypt value\n\t* email authenticator: not found: no rows in result set\n\n" host="137.184.147.128:8000" method=POST path=/api/v1/auth/login request_id=d7c57613-ca24-43b3-a268-df0033bd0a10 service_name=rest_api username=AKIAJSEJPLE4NOPT4KEQ
    j
    Barak Amar
    16 replies
    Copy to Clipboard
  • j

    Jude

    3 months ago
    Hi Team, quick one please. I am unable to send data to lakefs using the lake-client. The upload process fails with this message in the airflow log
    File "/home/airflow/.local/lib/python3.7/site-packages/lakefs_client/api/branches_api.py", line 862, in list_branches
    return self.list_branches_endpoint.call_with_http_info(**kwargs)
    File "/home/airflow/.local/lib/python3.7/site-packages/lakefs_client/api_client.py", line 851, in call_with_http_info
    collection_formats=params['collection_format'])
    File "/home/airflow/.local/lib/python3.7/site-packages/lakefs_client/api_client.py", line 415, in call_api
    _check_type)
    File "/home/airflow/.local/lib/python3.7/site-packages/lakefs_client/api_client.py", line 189, in __call_api
    url = self.configuration.host + resource_path
    TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'
    My code looks something like this:                                                                                                                                                                                                                                                                                                        def upload_events_data_to_s3():
        file_path = "/opt/airflow/Dataset/v1/events.parquet"  
        print(client.branches.list_branches('test').results)
        with open(file_path, 'rb') as f:
            client.objects.upload_object(repository='test', branch='main', path='bronzelayer/ApiData/events.parquet', content=f)
            f.close()
    j
    Barak Amar
    5 replies
    Copy to Clipboard
  • Gideon Catz

    Gideon Catz

    3 months ago
    Hi Team, I’m trying to write a parquet to lakeFS for the first time. Working with scala, locally, using SBT. I included this in my dependencies:
    libraryDependencies += "io.lakefs" % "hadoop-lakefs" % "0.1.6"
    Yet when trying to write out a dataframe:
    outputDf.write.parquet(s"lakefs://${repo}/${branch}/example.parquet")
    I’m getting:
    Exception in thread "main" org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "lakefs"
    I’m executing the program in intellij, in a main method of an object. What am I missing here? Maybe I should somehow exclusively mention --packages somewhere in the run configuration, even though lakefs is in the dependencies?
    Gideon Catz
    Itai Admi
    +1
    66 replies
    Copy to Clipboard
  • j

    Jude

    2 months ago
    Hi team, please is there is a way to read data from LakeFS using lakefs_client just the same way it can be used to upload file to LakeFS storage. I am trying to run an ML pipeline on kubeflow
    j
    Itai Admi
    10 replies
    Copy to Clipboard