Hi!, I'm trying to setup Trino to access lakeFS, It's working perfectly for s3a:// style tables, but not working for lakefs:// filesystem (as expected, it says that the filesystem is not supported), I will give some tries but... someone can confirm if Trino can be made to work with lakefs filesystem? (and in addition to adding hadoop-lakefs-assembly.jar to the classpath, which options to set in the connector)

Hi, I am trying to setup the lakefs server on GKE with terraform, but running into the issue that its not able to find the

index.yaml

file For reference, this is the relevant HCL script (Have removed the GKE and Cloud SQL modules) I am sure I have specified either the chart or the repository arguments wrongly, but not sure what is the right one

provider "helm" {

  kubernetes {
    host                   = "${google_container_cluster.primary.endpoint}"
    token                  = "${data.google_client_config.current.access_token}"
    client_certificate     = "${base64decode(google_container_cluster.primary.master_auth.0.client_certificate)}"
    client_key             = "${base64decode(google_container_cluster.primary.master_auth.0.client_key)}"
    cluster_ca_certificate = "${base64decode(google_container_cluster.primary.master_auth.0.cluster_ca_certificate)}"
  }
}

resource "helm_release" "my-lakefs" {
  name  = "my-lakefs"
  chart = "charts/lakefs"
  repository = "<https://github.com/treeverse/lakeFS>"
}

I'm trying to get

rclone

to work with the sample repo one gets by clicking "try without installing". I've setup my rclone config file like:

[lakefs]
type = s3
provider = AWS
env_auth = false
access_key_id = <redacted>
secret_access_key = <redacted>
endpoint = <https://big-zebra.us-east-1.lakefscloud.io>
no_check_bucket = true

where

big-zebra.us-east-1...

is, I think, the correct endpoint for that sample repo (it's what shows in my address bar). But I get certificate errors:

$ rclone -vv ls lakefs:sample-repo/testing/  
2023/08/15 12:35:09 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "-vv" "ls" "lakefs:sample-repo/testing/"]
2023/08/15 12:35:09 DEBUG : Creating backend with remote "lakefs:sample-repo/testing/"
2023/08/15 12:35:09 DEBUG : Using config file from "/home/oliver/.config/rclone/rclone.conf"
2023/08/15 12:35:09 DEBUG : fs cache: renaming cache item "lakefs:sample-repo/testing/" to be canonical "lakefs:sample-repo/testing"
2023/08/15 12:35:49 DEBUG : 2 go routines active
2023/08/15 12:35:49 Failed to ls: RequestError: send request failed
caused by: Get "<https://sample-repo.big-zebra.us-east-1.lakefscloud.io/?delimiter=&encoding-type=url&max-keys=1000&prefix=testing%2F>": x509: certificate is valid for *.<http://us-east-1.lakefscloud.io|us-east-1.lakefscloud.io>, <http://us-east-1.lakefscloud.io|us-east-1.lakefscloud.io>, not <http://sample-repo.big-zebra.us-east-1.lakefscloud.io|sample-repo.big-zebra.us-east-1.lakefscloud.io>

I see it's a wildcard cert for

*.<http://us-east-1.lakefscloud.io|us-east-1.lakefscloud.io>

which seems like it should work, but it also looks like

rclone

is adding the repo name (

sample-repo

) to the hostname which seems incorrect. What am I doing wrong?

Trying out the new

local

command, I get an error when I try to commit - any ideas?

lakectl local commit . -m "Remove employee data and add ratings information" --pre-sign=false

Getting branch: main

diff 'local:///Users/rmoff/work/orion' <--> '<lakefs://quickstart/7eaeafd3fb90df419ca26eca69ba2b153eae13bd3dccf80ba47cbed1260cb0c1/data/OrionStar/>'...

diff '<lakefs://quickstart/38df1480d835fe52dc6db31b87ef35b78403d0c9c621c74a6d032c686551abc5/data/OrionStar/>' <--> '<lakefs://quickstart/main/data/OrionStar/>'...
delete remote path: EMPLOYEE_ORGANIZATI~ ... done! [%0 in 8ms]
delete remote path: EMPLOYEE_ADDRESSES.~ ... done! [%0 in 8ms]
delete remote path: EMPLOYEE_DONATIONS.~ ... done! [%0 in 8ms]
delete remote path: EMPLOYEE_PAYROLL.csv ... done! [%0 in 8ms]
upload RATINGS.csv                       ... done! [6.86KB in 9ms]
upload RATINGS.csv failed: upload to backing store: MissingRegion: could not find region configuration
Error executing command.

Hi ,all

Hi ,all I have a problem when use the python lakefs_client: I have deployed the lakefs service locally. when i use the example api: client.repositories_api.list_repositories(prefix="",after="", amount=100 ) .It reported an error: urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='xxx', port=xxx): Max retries exceeded with url: /api/v1/repositories (Caused by ProtocolError('Connection aborted.', ConnectionResetError(10054, '远程主机强迫关闭了一个现有的连接。', None, 10054, None))) However , other api :such as :client.repositories_api.delete_repository(repo_name),,client.branches_api.create_branch(repository=repo_name, branch_creation=branch_creation) are ok. list_repositories and list_branches api are have the error

Hi guys, I am going to deploy Lakefs in AWS. please help

Hi teams. I have a weird bug that I don't know if it coming from lakefs or fsspec/s3fs : The code:

import s3fs
import smartfs
conf = smartfs.load_rclone_profile("lakefs")



fs = s3fs.S3FileSystem(
                    anon=False,
                    endpoint_url=conf["endpoint"],
                    key=conf["access_key_id"],
                    secret=conf["secret_access_key"],
                )

## Comment/uncomment following to trigger the bug. 
fs.find("<s3://test-repo/main/>",maxdepth=None,withdirs=False)

print(fs.find("<s3://test-repo/main/>",maxdepth=1,withdirs=True))

The content looks like this:

$ rclone ls lakefs:test-repo/main/
        0 dir1/fileB
        0 fileA

When I run the code above, the second

find

should list 2 entries:

fileA

,

dir1

but somehow it is missing

dir1

But if I comment out the first

find

, then the second

find

will list the 2 entries correctely. I don't have an AWS account in order to check if it's a lakefs issue or s3fs issue. Can someone with a real AWS account test this ? Context: Lakefs self hosted: 0.104.0 pip freeze:

aiobotocore==2.5.2
boto3==1.21.21
botocore==1.29.161
s3fs==2023.6.0
s3transfer==0.5.2

Edit: I was decribing the expected result slightly wrong.

hi. how to use the latest lakectl . i want to clone the files to my local. while it doesnot get it .

lakectl local init <lakefs://datasetest/test1/> E:\WorkProjects\data

lakectl local list
+-----------+------------+---------------+
| DIRECTORY | REMOTE URI | SYNCED COMMIT |
+-----------+------------+---------------+
+-----------+------------+---------------+

D:\Develops\lakefs\lakeFS_0.107.0_Windows_x86_64>lakectl local clone <lakefs://datasetest/test1/> E:\lakedata
EOF
Error executing command.

Hi, is it possible to restrict

fs:ListRepositories

by something like

arn:lakefs:fs:::repository/{prefix}*

? We have a an internal structure with different teams who don't have to see all the repos by all other teams. The doc only states

*

as possible resource

Hi, I am trying to deploy LakeFS via an azure webapp (containers) using docker and we are able to run successfully with LAKEFS_DATABASE_TYPE=Local but when we try LAKEFS_DATABASE_TYPE=postgress and pass the connection string via LAKEFS_DATABASE_CONNECTION_STRING= host=Server.postgres.database.azure.com port=5432 dbname=postgres user=%USERNAME%@%Server password=################ sslmode=require It fails to start with the following errors in docker:

2023-08-22T13:54:49.349664594Z time="2023-08-22T13:54:49Z" level=info msg="lakeFS run" func=cmd/lakefs/cmd.glob..func8 file="cmd/run.go:94" version=0.107.0
2023-08-22T13:54:49.349684195Z time="2023-08-22T13:54:49Z" level=fatal msg="Failed to open KV store" func=cmd/lakefs/cmd.glob..func8 file="cmd/run.go:102" error="connect failed: failed to connect to `host=/tmp user=lakefs database=`: dial error (dial unix /tmp/.s.PGSQL.5432: connect: no such file or directory)"

Connection string works in psql are we missing variables?

Hi guys another funny here. We are looking to use an Azure Webapp (containers) to host lakeFS and Azure Postgress flexible db and i have this working now which is great however i am having a problem with the container itself. I can only run successfully if i use docker compose with the following settings:

version: "3.5"
name: lakefs
services:
  lakefs:
    image: treeverse/lakefs:latest
    ports:
      - "8000:8000"

When i try to use the container on its own it fails with the following errors

2023-08-23T08:20:56.611Z ERROR - Container jigsaw-lakefs_0_61e8c142 for site jigsaw-lakefs has exited, failing site start
2023-08-23T08:20:56.620Z ERROR - Container jigsaw-lakefs_0_61e8c142 didn't respond to HTTP pings on port: 8000, failing site start. See container logs for debugging.
2023-08-23T08:20:56.641Z INFO  - Stopping site jigsaw-lakefs because it failed during startup.

We have the WEBSITES_PORT set to 8000 and have tried specifying the ports in the start up command but just will not work. Unfortunately we can not use docker compose option as we are not able to bind the website address to a vnet to secure the Postgress DB. Any help would be appreciated.

Hi again guys! Really like lakefs. I have a problem regarding reading and writing delta tables using spark. It works but spark is huge and its complex and i will try everything to avoid it. So my question to you guys is if there are any other way to read and write delta tables to lakefs avoiding spark?

Hi guys! A question is very confusing to me, in /pkg/local/sync.go, we use maxParallelism to limit the number of goroutines, but it seems like we don't wait until the goroutine ends and release the ch at once，can you explain this? Thanks a lot! :) https://github.com/treeverse/lakeFS/blob/e02ece385d7e672957a7323d3f45ce69208767ef/pkg/local/sync.go#L94C8-L94C8

Hey everyone. I'm new to lakeFS, and I'm trying to use it a pyspark job running in AWS Glue 4.0. We're processing data in Iceberg tables, so I've been following this setup. I'm running into this error:

An error occurred while calling o128.getDynamicFrame. io/lakefs/iceberg/LakeFSCatalog has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0

I've used the same libraries listed in the setup link above (lakefs-iceberg:0.1.2, iceberg-spark-runtime-3.3_2.12:1.3.0), and I've also tried using the same spark runtime version that was listed as a test dependency for

io.lakefs:lakefs-iceberg:0.1.2

in Maven. Am I missing something obvious, or is there a version of the lakefs-iceberg lib that was compiled with Java runtime 52.0?

Do you guys have any example known good working configuration for LakeFS with Azure CosmosDB? We have postgres working but were interested in this other technology currently get the website up but when trying to sign up get the error:

I have a weird message error when trying to do a merge using

lakefs_client

python sdk:

File "/path/lakefs_helper.py", line 102, in merge
    return self.lakefs.refs.merge_into_branch(
  File "/path/venv/lib64/python3.10/site-packages/lakefs_client/api/refs_api.py", line 869, in merge_into_branch
    return self.merge_into_branch_endpoint.call_with_http_info(**kwargs)
  File "/path/venv/lib64/python3.10/site-packages/lakefs_client/api_client.py", line 835, in call_with_http_info
    return self.api_client.call_api(
  File "/path/venv/lib64/python3.10/site-packages/lakefs_client/api_client.py", line 409, in call_api
    return self.__call_api(resource_path, method,
  File "/path/venv/lib64/python3.10/site-packages/lakefs_client/api_client.py", line 203, in __call_api
    raise e
  File "/path/venv/lib64/python3.10/site-packages/lakefs_client/api_client.py", line 196, in __call_api
    response_data = self.request(
  File "/path/venv/lib64/python3.10/site-packages/lakefs_client/api_client.py", line 455, in request
    return <http://self.rest_client.POST|self.rest_client.POST>(url,
  File "/path/venv/lib64/python3.10/site-packages/lakefs_client/rest.py", line 267, in POST
    return self.request("POST", url,
  File "/path/venv/lib64/python3.10/site-packages/lakefs_client/rest.py", line 224, in request
    raise ServiceException(http_resp=r)
lakefs_client.exceptions.ServiceException: (503)
Reason: Service Unavailable
HTTP response headers: HTTPHeaderDict({'content-length': '118', 'content-type': 'text/plain', 'date': 'Thu, 24 Aug 2023 23:47:57 GMT'})
HTTP response body: upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection termination

My helper function that do the merge is:

def merge(self, source_branch, dest_branch, metadata={}, message=None):
        try:
            # Check if there is a need to merge:
            diffs = self.diff(left_ref=dest_branch, right_ref=source_branch)
            if len(diffs) == 0:
                print(f"No changes detected. Skipping merge {source_branch} to {dest_branch}")
                return
            if message:
                merge = models.Merge(message=message, metadata=metadata)
            else:
                merge = models.Merge(metadata=metadata)
            return self.lakefs.refs.merge_into_branch(
                repository=self.repo_name,
                source_ref=source_branch,
                destination_branch=dest_branch,
                merge=merge,
            )
        except Exception as e:
            # traceback.print_exc()
            print(f"ERROR: Failed to merge {source_branch} to {dest_branch}")
            raise e

With self.lakefs as :

configuration = lakefs_client.Configuration()
        configuration.username = conf["access_key_id"]
        configuration.password = conf["secret_access_key"]
        configuration.host = conf["endpoint"]
        self.lakefs = LakeFSClient(configuration)

But then if I do the merge of that 2 exact same branch via the UI, it succeed ! Our self deployed server running : 107.0 And the lakefs_client is 0.107.0

I’m setting up an on-premise lakefs instance and so far everything works, except importing files from S3, which returns this error :

$ lakectl import \                                                                     
  --from <s3://jaffle_shop_repo/raw_payments.csv> \
  --to <lakefs://jaffle-shop-repo/main/raw_payments.csv> 
Import failed: import error: error on ingest: NoSuchKey: 
        status code: 404, request id: tx00000000000000a72c046-0064ec5ed8-b6658b2f-fra1b, host id:

the Lakefs repo uses

<s3://jaffle_shop_repo>

as the storage namespace, and is located at the root of the S3 bucket. It doesn’t seem like a credentials problem, since the

lakectl fs upload

command works to add data in the repo, and write this data in the S3 storage repo’s folder. I suspect that it’s just a uri path problem, tried several variations but none seem to work Any idea on what could be wrong?

hey 👋 quick question: how can I load a yaml file in Lua hooks? I don’t see a way to do that in the lua library reference

Hello, I'm new to lakeFS and would like to ask, does anyone have an example of writing a Zarr store using xarray with lakeFS? Here is my code trying to do that but the last

to_zarr()

step is getting errors.

import dask.array as da
import xarray as xr
import numpy as np

configuration = lakefs_client.Configuration()
configuration.username = access_key_id
configuration.password = secret_access_key
configuration.host = endpoint_url
client = LakeFSClient(configuration)

repo = "zarr-test"
branch = "zarr-store"
client.branches.create_branch(
    repository=repo,
    branch_creation=models.BranchCreation(
    name=branch,
    source="main"))

# Create random array
state = da.random.RandomState(1234)
shape = (180, 360, 400)
chunk_shape = (36, 72, 200)
nlats, nlons, ntimes = shape
arr = state.random(shape, chunks=chunk_shape)
arr = da.random.random(shape, chunks=chunk_shape)
ds = xr.Dataset(
    data_vars={
        "precipitation": xr.DataArray(arr, dims=('lat', 'lon', 'time'))
    },
    coords={
        "lat": xr.DataArray(np.linspace(-90, 90, num=nlats, endpoint=False), dims='lat'),
        "lon": xr.DataArray(np.linspace(-180, 180, num=nlons, endpoint=False), dims='lon'),
        "time": xr.date_range(start="2000-06-01", freq="D", periods=ntimes)
    },
    attrs={
        "description": "GPM IMERG test dataset"
    }
)

# Write the first 200 time slices
ds_0 = ds.isel(time=slice(0, 200))
s3a_gateway_path = f's3a://{repo}/{branch}/precipitation_data.zarr'
task = ds_0.to_zarr(s3a_gateway_path, 
                    # zarr_version=3, 
                    mode='w', 
                    compute=False)

Without

zarr_version

, I get

PermissionError: Access Denied

. If I set

zarr_version=3

, I get

KeyError: 'zarr.json'

. Maybe I am setting the

s3a_gateway_path

incorrectly?

Hello everyone! Currently in lakeFS, we just use the

sizeBytes

and

modifiedTime

of a file when comparing local files with remote commits. However, I believe there may be some risks associated with this approach. For instance, if two clients have different system times and both edit the same file, let's say

a.txt

, the

sizeBytes

and

modifiedTime

could be the same on both clients, but the contents are different. Consequently, when I commit the changes on client A and then pull from the remote on client B, the modifications could be lost. So here are my questions: 1. Do you have any evidence or user cases to prove this situation is rare? 2. What is the probability of encountering this risk? Have any tests been conducted? 3. Why do we only use second-level precision for the

modifiedTime

and not something more precise like nanoseconds? (Unix time in nanoseconds cannot be represented by an int64 for dates only prior to the year 1678 or after 2262) Thanks a lot! :)

Hello wonderful people. I am really impressed with LakeFS and working getting it running in my Kubernetes cluster with MinIO as a backend. Consulting the documentation I now have a working Helm deployment where I set the MinIO specific configs (discover_bucket_region: false, force_path_style: true). This setup is working as expected when I create a user in the UI. However, for my setup I need headless CICD so using the UI as part of the setup process is a no-go. I dug into the forums and found the recommended solution of using “lakefs superuser --user-name admin”. When I run this in the K8 pod it successfully recognizes my config for MinIO passed in from the Helm chart, however it appears that this endpoint relies on a metadata provider that is part of the AWS SDK which was not implemented in MinIO. The following error is shown:

INFO  [2023-09-02T17:23:06Z]build/pkg/auth/service.go:189 pkg/auth.NewAuthService initialized Auth service           service=auth_service

WARNING[2023-09-02T17:23:06Z]build/pkg/cloud/aws/metadata.go:64 pkg/cloud/aws.(*MetadataProvider).GetMetadata.func1 Tried to to get AWS account ID for BI     error="InvalidParameterValue: Unsupported action GetCallerIdentity\n\tstatus code: 400, request id: 178124CA4D3D18C0"

Digging into the code it appears that the AWS package is trying to find the email of the AWS account, and since MinIO doesn’t have one, it fails. Relevant code is here and here. Before I fork LakeFS and try to dig in myself I thought I would see if anyone here had any ideas on a more elegant solution (aka supported) solution? TLDR: LakeFS + Minio + Headless = not supported?

Hi, I'm trying to use LakeFS helm chart to deploy it with 3 replicas in our EKS cluster which has 3availability zones. I have configured "podAntiAffinity" and it works. I'm unable to perform rollingupdates because of the issue explained here https://medium.com/xmglobal/how-can-affinity-rules-break-your-kubernetes-rolling-updates-3055eb2d478c. TO address this, Kubernetes added https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/, but I don't see a way to set those in the current Helm Chart (and I don't want to fallback to manual yamls as we hopefully will fluffy yamls soon). Someone has a way to deploy one-per-az using affinity? (I can't use preferred because I'm using karpenter as node allocator and based of testing, that will not enforce it hard enough). For now I'm just using 2 replicas as a workaround but 😕

Hi, when I want to try LakeFS through Try me without installing on the webpage, I get

Oh my! An error occurred: Unauthorized

when trying to register

It's also mentioned in the documentation that zero copy importing is only possible from the object storage service in which your installation stores its data. Does this also mean that you can't have two different repositories that would use a different object storage service?

Hi, lakeFS! I’d like to try out the garbage collection feature with your help. I set the GC policy to have 1 retention day as a default and for all my branches. A few days ago, I wrote many versions of a dataset on a branch and see high storage in my S3 bucket as intended in my test. My questions: 1. Do I understand it correctly that S3 data from the latest version/commit will be retained and data from all other older versions will be deleted per the GC policy? 2. I ran the

spark-submit

command as directed by the documentation. As far as I can tell, the command ran and finished without errors. But in S3, I don’t see the list of objects removed in

_lakefs/retention/gc/unified/<RUN_ID>/deleted/

and my storage didn’t go down, so I assume objects from old versions haven’t been deleted. Do you have any ideas about this? Note: I just set the GC policy after already writing all my versions - Is this why?

Hi team ! How to list all branch that can reach a given commit ? Is there a way to do that from the python sdk ?

Hi All, just a quick question on a race condition that i have noticed in commits. Say i have two processes, on uploads a bunch of files that need to go into a single commit, while this process is still going, process b start uploading a set of its own files. Now these are all staged in uncommited changes. And Process A is now able to commit extra files that have been uploaded by Process B, is this something that can be avoided?

what is this error means ? Any help Appreciated -No data: Error: No data at _$.list (https://lakefs.admin.cl1.dev.cxl.io/assets/index-7bf1aa94.js:71:102851) at async https://lakefs.admin.cl1.dev.cxl.io/assets/index-7bf1aa94.js:71:111901

Got fixed - Need to restart the cluster