Hey everyone. I'm new to lakeFS, and I'm trying to use it a pyspark job running in AWS Glue 4.0. We're processing data in Iceberg tables, so I've been following this setup. I'm running into this error:

An error occurred while calling o128.getDynamicFrame. io/lakefs/iceberg/LakeFSCatalog has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0

I've used the same libraries listed in the setup link above (lakefs-iceberg:0.1.2, iceberg-spark-runtime-3.3_2.12:1.3.0), and I've also tried using the same spark runtime version that was listed as a test dependency for

io.lakefs:lakefs-iceberg:0.1.2

in Maven. Am I missing something obvious, or is there a version of the lakefs-iceberg lib that was compiled with Java runtime 52.0?

Do you guys have any example known good working configuration for LakeFS with Azure CosmosDB? We have postgres working but were interested in this other technology currently get the website up but when trying to sign up get the error:

I have a weird message error when trying to do a merge using

lakefs_client

python sdk:

File "/path/lakefs_helper.py", line 102, in merge
    return self.lakefs.refs.merge_into_branch(
  File "/path/venv/lib64/python3.10/site-packages/lakefs_client/api/refs_api.py", line 869, in merge_into_branch
    return self.merge_into_branch_endpoint.call_with_http_info(**kwargs)
  File "/path/venv/lib64/python3.10/site-packages/lakefs_client/api_client.py", line 835, in call_with_http_info
    return self.api_client.call_api(
  File "/path/venv/lib64/python3.10/site-packages/lakefs_client/api_client.py", line 409, in call_api
    return self.__call_api(resource_path, method,
  File "/path/venv/lib64/python3.10/site-packages/lakefs_client/api_client.py", line 203, in __call_api
    raise e
  File "/path/venv/lib64/python3.10/site-packages/lakefs_client/api_client.py", line 196, in __call_api
    response_data = self.request(
  File "/path/venv/lib64/python3.10/site-packages/lakefs_client/api_client.py", line 455, in request
    return <http://self.rest_client.POST|self.rest_client.POST>(url,
  File "/path/venv/lib64/python3.10/site-packages/lakefs_client/rest.py", line 267, in POST
    return self.request("POST", url,
  File "/path/venv/lib64/python3.10/site-packages/lakefs_client/rest.py", line 224, in request
    raise ServiceException(http_resp=r)
lakefs_client.exceptions.ServiceException: (503)
Reason: Service Unavailable
HTTP response headers: HTTPHeaderDict({'content-length': '118', 'content-type': 'text/plain', 'date': 'Thu, 24 Aug 2023 23:47:57 GMT'})
HTTP response body: upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection termination

My helper function that do the merge is:

def merge(self, source_branch, dest_branch, metadata={}, message=None):
        try:
            # Check if there is a need to merge:
            diffs = self.diff(left_ref=dest_branch, right_ref=source_branch)
            if len(diffs) == 0:
                print(f"No changes detected. Skipping merge {source_branch} to {dest_branch}")
                return
            if message:
                merge = models.Merge(message=message, metadata=metadata)
            else:
                merge = models.Merge(metadata=metadata)
            return self.lakefs.refs.merge_into_branch(
                repository=self.repo_name,
                source_ref=source_branch,
                destination_branch=dest_branch,
                merge=merge,
            )
        except Exception as e:
            # traceback.print_exc()
            print(f"ERROR: Failed to merge {source_branch} to {dest_branch}")
            raise e

With self.lakefs as :

configuration = lakefs_client.Configuration()
        configuration.username = conf["access_key_id"]
        configuration.password = conf["secret_access_key"]
        configuration.host = conf["endpoint"]
        self.lakefs = LakeFSClient(configuration)

But then if I do the merge of that 2 exact same branch via the UI, it succeed ! Our self deployed server running : 107.0 And the lakefs_client is 0.107.0

I’m setting up an on-premise lakefs instance and so far everything works, except importing files from S3, which returns this error :

$ lakectl import \                                                                     
  --from <s3://jaffle_shop_repo/raw_payments.csv> \
  --to <lakefs://jaffle-shop-repo/main/raw_payments.csv> 
Import failed: import error: error on ingest: NoSuchKey: 
        status code: 404, request id: tx00000000000000a72c046-0064ec5ed8-b6658b2f-fra1b, host id:

the Lakefs repo uses

<s3://jaffle_shop_repo>

as the storage namespace, and is located at the root of the S3 bucket. It doesn’t seem like a credentials problem, since the

lakectl fs upload

command works to add data in the repo, and write this data in the S3 storage repo’s folder. I suspect that it’s just a uri path problem, tried several variations but none seem to work Any idea on what could be wrong?

hey 👋 quick question: how can I load a yaml file in Lua hooks? I don’t see a way to do that in the lua library reference

Hello, I'm new to lakeFS and would like to ask, does anyone have an example of writing a Zarr store using xarray with lakeFS? Here is my code trying to do that but the last

to_zarr()

step is getting errors.

import dask.array as da
import xarray as xr
import numpy as np

configuration = lakefs_client.Configuration()
configuration.username = access_key_id
configuration.password = secret_access_key
configuration.host = endpoint_url
client = LakeFSClient(configuration)

repo = "zarr-test"
branch = "zarr-store"
client.branches.create_branch(
    repository=repo,
    branch_creation=models.BranchCreation(
    name=branch,
    source="main"))

# Create random array
state = da.random.RandomState(1234)
shape = (180, 360, 400)
chunk_shape = (36, 72, 200)
nlats, nlons, ntimes = shape
arr = state.random(shape, chunks=chunk_shape)
arr = da.random.random(shape, chunks=chunk_shape)
ds = xr.Dataset(
    data_vars={
        "precipitation": xr.DataArray(arr, dims=('lat', 'lon', 'time'))
    },
    coords={
        "lat": xr.DataArray(np.linspace(-90, 90, num=nlats, endpoint=False), dims='lat'),
        "lon": xr.DataArray(np.linspace(-180, 180, num=nlons, endpoint=False), dims='lon'),
        "time": xr.date_range(start="2000-06-01", freq="D", periods=ntimes)
    },
    attrs={
        "description": "GPM IMERG test dataset"
    }
)

# Write the first 200 time slices
ds_0 = ds.isel(time=slice(0, 200))
s3a_gateway_path = f's3a://{repo}/{branch}/precipitation_data.zarr'
task = ds_0.to_zarr(s3a_gateway_path, 
                    # zarr_version=3, 
                    mode='w', 
                    compute=False)

Without

zarr_version

, I get

PermissionError: Access Denied

. If I set

zarr_version=3

, I get

KeyError: 'zarr.json'

. Maybe I am setting the

s3a_gateway_path

incorrectly?

Hello everyone! Currently in lakeFS, we just use the

sizeBytes

and

modifiedTime

of a file when comparing local files with remote commits. However, I believe there may be some risks associated with this approach. For instance, if two clients have different system times and both edit the same file, let's say

a.txt

, the

sizeBytes

and

modifiedTime

could be the same on both clients, but the contents are different. Consequently, when I commit the changes on client A and then pull from the remote on client B, the modifications could be lost. So here are my questions: 1. Do you have any evidence or user cases to prove this situation is rare? 2. What is the probability of encountering this risk? Have any tests been conducted? 3. Why do we only use second-level precision for the

modifiedTime

and not something more precise like nanoseconds? (Unix time in nanoseconds cannot be represented by an int64 for dates only prior to the year 1678 or after 2262) Thanks a lot! :)

Hello wonderful people. I am really impressed with LakeFS and working getting it running in my Kubernetes cluster with MinIO as a backend. Consulting the documentation I now have a working Helm deployment where I set the MinIO specific configs (discover_bucket_region: false, force_path_style: true). This setup is working as expected when I create a user in the UI. However, for my setup I need headless CICD so using the UI as part of the setup process is a no-go. I dug into the forums and found the recommended solution of using “lakefs superuser --user-name admin”. When I run this in the K8 pod it successfully recognizes my config for MinIO passed in from the Helm chart, however it appears that this endpoint relies on a metadata provider that is part of the AWS SDK which was not implemented in MinIO. The following error is shown:

INFO  [2023-09-02T17:23:06Z]build/pkg/auth/service.go:189 pkg/auth.NewAuthService initialized Auth service           service=auth_service

WARNING[2023-09-02T17:23:06Z]build/pkg/cloud/aws/metadata.go:64 pkg/cloud/aws.(*MetadataProvider).GetMetadata.func1 Tried to to get AWS account ID for BI     error="InvalidParameterValue: Unsupported action GetCallerIdentity\n\tstatus code: 400, request id: 178124CA4D3D18C0"

Digging into the code it appears that the AWS package is trying to find the email of the AWS account, and since MinIO doesn’t have one, it fails. Relevant code is here and here. Before I fork LakeFS and try to dig in myself I thought I would see if anyone here had any ideas on a more elegant solution (aka supported) solution? TLDR: LakeFS + Minio + Headless = not supported?

Hi, I'm trying to use LakeFS helm chart to deploy it with 3 replicas in our EKS cluster which has 3availability zones. I have configured "podAntiAffinity" and it works. I'm unable to perform rollingupdates because of the issue explained here https://medium.com/xmglobal/how-can-affinity-rules-break-your-kubernetes-rolling-updates-3055eb2d478c. TO address this, Kubernetes added https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/, but I don't see a way to set those in the current Helm Chart (and I don't want to fallback to manual yamls as we hopefully will fluffy yamls soon). Someone has a way to deploy one-per-az using affinity? (I can't use preferred because I'm using karpenter as node allocator and based of testing, that will not enforce it hard enough). For now I'm just using 2 replicas as a workaround but 😕

Hi, when I want to try LakeFS through Try me without installing on the webpage, I get

Oh my! An error occurred: Unauthorized

when trying to register

It's also mentioned in the documentation that zero copy importing is only possible from the object storage service in which your installation stores its data. Does this also mean that you can't have two different repositories that would use a different object storage service?

Hi, lakeFS! I’d like to try out the garbage collection feature with your help. I set the GC policy to have 1 retention day as a default and for all my branches. A few days ago, I wrote many versions of a dataset on a branch and see high storage in my S3 bucket as intended in my test. My questions: 1. Do I understand it correctly that S3 data from the latest version/commit will be retained and data from all other older versions will be deleted per the GC policy? 2. I ran the

spark-submit

command as directed by the documentation. As far as I can tell, the command ran and finished without errors. But in S3, I don’t see the list of objects removed in

_lakefs/retention/gc/unified/<RUN_ID>/deleted/

and my storage didn’t go down, so I assume objects from old versions haven’t been deleted. Do you have any ideas about this? Note: I just set the GC policy after already writing all my versions - Is this why?

Hi team ! How to list all branch that can reach a given commit ? Is there a way to do that from the python sdk ?

Hi All, just a quick question on a race condition that i have noticed in commits. Say i have two processes, on uploads a bunch of files that need to go into a single commit, while this process is still going, process b start uploading a set of its own files. Now these are all staged in uncommited changes. And Process A is now able to commit extra files that have been uploaded by Process B, is this something that can be avoided?

what is this error means ? Any help Appreciated -No data: Error: No data at _$.list (https://lakefs.admin.cl1.dev.cxl.io/assets/index-7bf1aa94.js:71:102851) at async https://lakefs.admin.cl1.dev.cxl.io/assets/index-7bf1aa94.js:71:111901

Got fixed - Need to restart the cluster

No worries - thanks

Hi Channel, I have a question about tagging. I have a Hive Metastore point to the main branch to a deltalake database, I want to tag the current database and want to create another database in HMS to point to this tag, is there a simple way to do it? thanks for any help

Hello, I didn't find an example of implementing upload, getfile using .net framework, can you help me

Hi, I am trying to access the underlying s3 storage of a repository using

aws --profile lakefs s3 ls s3://<bucket>/<repo> --recursive

. I have configured the

lakefs

profile with credentials from LakeFS, and the corresponding endpoint url. I also have

AuthFullAccess

,

FSFullAccess

and

RepoManagementFullAccess

permissions. However, I still get the following error

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Forbidden

Do you have an idea what I am missing?

Hi everyone, I'm simply following the quickstart tutorial. After I merged by branch denmark un main and I ran

SELECT * FROM READ_PARQUET('<lakefs://quickstart/main/lakes.parquet>');

I got a error

Error: Invalid Input Error: No magic bytes found at end of file '<lakefs://quickstart/main/lakes.parquet>'

Do you know what causes this and how to fix ?

Question around Garbage Collection: Let say I have

f1

in commit

c1

then I delete

f1

in commit

c2

Further commit happen unrelated to

f1

My branch head is at `c8`(and without

f1

)

f1

is commited file but not in the branch head. Does it mean that it will be deleted if I run garbage collection one year later (eg: beyond any retention rule) ?

Hi 👋 - Is there anyone I could chat to about some requirements I have and if LakeFS is a good fit?

Hey there! I’m trying to play with nextflow and LakeFS.. locally they seem to be working nicely together, however, when I configure nextflow to work with AWS batch, the instances seem to have the wrong credentials. Any good reference for this setup?

@Neil Murray :

Is there any limit to the size to file uploaded via web client?

There should be no limit. Obviously for more than a few hundred MiB you'll want to do something more reliable; lakeFS is also accessible using the S3 protocol, where you can even use multipart uploads.

@Neil Murray :

Is there any way to label or attach properties to data files (other than tagging)?

Yes, but! Objects support user metadata, this is a tested part of the API. However there is no current UI support to access it. Currently it only gets used to store MIME type. But if you need it, please make sure to ask us for it!

@Neil Murray

Any limit to number of concurrent users?

Nope. I don't think the lakeFS server even has a concept of "active users", other than just caching some properties.

hello! Is there any documentation or "best practices" when having multiple reads and writes to a bucket over time. Async. What i want is: 1. Create a branch (If possible a blank branch with no history) 2. Write some data to it and performing transformations etc 3. Commit to this branch and run pre-merge-hook on it 4. Merge into main branch. For now im trying something like this in python but sometimes i get errors of dirty branches and conflicts. Can i specify merge strategy on python merge?

is it possible to rename a branch ??

Is it possible to integrate LakeFS with active directory for RBAC?