I have a third-party CSV data that gets a new version published every month. This data needs to be put in database and lineage needs to be maintained. Wondering what is the best way to achieve this. Is this something Lakefs can do?

Hi team, I have one question about lakefs server setup. I deployed it on K8s cluster with postgres and S3. Somehow the deployment gets restarted and then we need to config the repo again. If I use the same repo name and storage namespace, it will complain that

storage namespace already in use

. I thought that if we use same postgres and S3 path to persist data and metadata, the lakefs server should be able to restore from previous deployment status. It seems that is not the case. Is there any setting to avoid configure the lakefs server again during redeployment?

Hi everyone, I'm planning to give a demo next week to some developers in my company since I'm really enthousiastic about the possibilities of LakeFS + Databricks! However, I'm still running into the same issue as before when trying to connect to my storage account with the lakefs:// file system. Maybe for some more context, I'm trying to follow this guide: (https://lakefs.io/blog/databricks-lakefs-integration-tutorial/)

• I've tried with a simple single line-csv file
• I've added all spark config variables necessary to the Compute cluster advanced options
• My storage account has no firewall in place
• The databricks subnet has no firewall in place
• I've added the latest versions of all required packages (see attached screenshot)

The client seems to work fine, so that makes me think that my credentials aren't the issue, but just the file system keeps throwing the following error: Caused by: io.lakefs.hadoop.shade.api.ApiException: Content type "text/html; charset=utf-8" is not supported for type: class io.lakefs.hadoop.shade.api.model.ObjectStats Any ideas as to what I could try or what causes the issue? Any help would be really appreciated!

@Ariel Shaqed (Scolnicov) what’s the status on the “Return a

stat

-like object on file up-/downloads containing basic version info” plan re: the lakeFS server? I saw that the RFC was merged, and that you commented on the idea. Basically for now, I can emulate this in the frontend by grabbing the object info as normal (via

objects_api.stat_object

), and then basically doing the equivalent of

git rev-parse $ref

with the requested revision. The

is object staged or committed

use case you asked for is e.g. integrity checking: Am I working with properly tracked/committed data, or a “dirty” untracked version staged on main by my colleague without my knowledge? This would matter more when pulling data using raw branch names instead of commit SHAs, but from my experience, that’s quite common actually.

Hi awesome team ! Got a new challenge today: LabelStudio + LakeFS : we have our LabelStudio server (self-deployed) getting image from LakeFS directly. All credentials are setup correctly but we are running against CORS issue, as mentioned here . Running Chrome in unsafe mode (

google-chrome-stable --disable-web-security --user-data-dir='test'

) solve the issue: the image show up in LabelStudio web UI, thus we are confident that credentials are correct. I am not super familiar with CORS. From what I understand, LakeFS S3 gateway need to be configure in someway in order for LabelStudio to show image that is not coming directly from itself ??? Anybody have any clue on this ?

What is the recommended way to handle multiple data sources that update at different frequencies? Say we have datasets D1, D2, D3 ... each that update at different times (say monthly, weekly, yearly etc.). Each dataset may contain multiple CSV files. Should each of them have their own branch or should they be just folders within a single branch or some other better way to handle them? Ultimately would like to have the

main

branch always point to the latest version of all these datasets. What is the best way to organize this with LakeFS?

Hi, is there a way to perform zero-copy of objects within LakeFS? (i.e. is copy_object api zero-copy/metadata-copy? or it makes an actual copy).

What is the equivalent of

git clone

for LakeFS to clone and manipulate the data locally? I have setup a LakeFS in docker with Minio S3 endpoint as the object store. Uploaded some files into LakeFS through its UI. Now, I do not see any option to organize the uploaded files (such as creating folders) in LakeFS UI nor see any option to

clone

the data to local machine to manipulate its hierarchy and commit back. What am I missing? What is the recommended toolset or workflow to organize the data hierarchies in LakeFS? I have some files in LakeFS UI and now want to be able to re-organize them into different folders and commit as new branch.

I have a question regarding programmatic access to LakeFS. I want users to be able to authenticate to LakeFS with python by utilizing OpenID connect and the specified claim name. My case here is that I would prefer that users do not create their own generated credentials, but instead get temporary credentials when they are calling a function. I am unable to find anything in the documentation, so any help is appreciated! As a reference, I want to be able to implement a flow similar to what is available in MinIO.

Hi, I'm using lakeFs filesystem (within Spark io.lakefshadoop lakefs assembly0.2.1 https://docs.lakefs.io/integrations/spark.html#lakefs-hadoop-filesystem) with presigned URLs and 2hours expiration time and sometimes I have error 500 from AWS (which from reading says its S3 throttling, I checked and for sure its not expired). java.io.IOException: Server returned HTTP response code: 500 for URL: https://xxxx.s3.eu-west-1.amazonaws.com/... Is this normal? Can we somehow configure the Spark LakeFS filesystem to retry those requests individually without relying on Spark task retry? So Spark doesn't fail the whole task in a "known" issue (its wasting task retries on that and sometimes it fails, also it takes around 20secs to happen).

Hi, I'm trying to connect to AWS and create repo in S3 bucket but when I run this:

docker run --pull always -p 8000:8000 -e LAKEFS_BLOCKSTORE_TYPE='s3' -e AWS_ACCESS_KEY_ID='<My Access Key ID>' -e AWS_SECRET_ACCESS_KEY='<My Secret Access Key>' treeverse/lakefs run --local-settings

I get this:

Using local-settings parameters configuration. This is suitable only for testing! It is NOT SUPPORTED for production.

time="2023-11-08T20:59:25Z" level=info msg="lakeFS run" func=cmd/lakefs/cmd.glob..func8 file="cmd/run.go:91" version=1.1.0

time="2023-11-08T20:59:25Z" level=info msg="initialized Auth service" func=pkg/auth.NewAuthService file="build/pkg/auth/service.go:188" service=auth_service

time="2023-11-08T20:59:25Z" level=warning msg="Tried to to get AWS account ID for BI" func="pkg/cloud/aws.(*MetadataProvider).GetMetadata.func1" file="build/pkg/cloud/aws/metadata.go:81" error="operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: bcfe8cca-10c1-4bee-b9c8-db359a3bf938, api error InvalidClientTokenId: The security token included in the request is invalid."

I'm 100% sure that

AWS_ACCESS_KEY_ID

and

AWS_SECRET_ACCESS_KEY

I'm sending as parameters to docker are correct but I'm not sure how to include security token.

Can lakeFS in the current release find difference between the same file in two different commits or branches? If I have json file in one commit, can I diff this file in another commit and see what has changed in this file?

is it possible to assign user's access to a single repo on lakefs?

and make them not able to write to other repos?

or is that something only for enterprise version?

Hi guys, still very new to lakeFS and just wanted to check if is it possible to run lakeFS command in sql like command ? Example, for creating branch…can we simply do ‘create branch test from main’ instead of using lakeFS client/ lakectl?

Hello :) I am new to lakeFS and finding a way to integrate with Hadoop cluster I’ve found several similar questions and the answer was “no” back then. I’m wondering if there are still no support for HDFS and if not, are there are any workarounds or examples for using HDFS as an underlying storage?

Hello! @Adrian Rumpold and I are wondering if the fact that a storage namespace is not emptied completely on repository deletion is intentional or a bug. In our concrete case, this breaks idempotence of repository creation calls, which we are currently exploring here: https://github.com/appliedAI-Initiative/lakefs-spec/pull/154 For a concrete problem, imagine the case “I want to run a Jupyter Notebook, in which a temporary repository gets created before start and deleted again after successful execution”. This notebook would not run twice in succession because the storage namespace is never cleaned up. Thanks!

Isn’t there file-level deduplication in lakeFS? I thought there might be, but then I uploaded the same file twice and got two different physical addresses.

Hey @Angela Bovo, as of today, lakeFS does not offer data dedeuplication in the sense you are describing, so this is the expected result. The dedeuplication offered by lakeFS happens at the branch level. That is, creating a branch is a metadata only operation, which causes no data to be copied.

Hey all! Recently spun up LakeFS for the first time using a helm chart, with a self hosted postgres DB and Ceph blockstore, all local. Really cool so far, very excited to play around with it some more heart lakefs One quick question, when spinning up the pod for the first time, I noticed it would halt after connecting to the DB. After adding in a http/https proxy to allow it to communicate with the internet, it spun up fine. I was wondering if there are some behind the scenes calls that are being made, and what exactly they are?

Hi all, I observe the lakeFS project for a while and so glad to see the 1.0 release with a lot of surprising features. I see that RBAC is now only available in cloud or enterprise version. I wonder if the enterprise version supports RBAC on the branch level? Following concrete scenario: in our company, there are data scientists in different groups. Some of them should only be able to access specific branches. The post shows a similar scenario but whether it is certainly branch-restricted is not clear. Would one have an idea if it is supported? If not, whether it is somewhen planed in the roadmap? Thanks.

Totally random question: can you share what tool is used to generate the illustrations in the docs? I really like the casual hand-drawn appearance! axolotl

Welcome!

trashcan for jys

joined #help.

hello, is it possible to use local storage files inside the lakeFS container? I am using a synology NAS mounted on a linux server.

Hello! Is it possible to import files from Synology NAS without using MinIO or any s3-compatible storage? Our data is stored as json file (in NAS) and cannot change this format due to business issues. NAS is mounted on a CentOS server (

:/mnt

), the same server that lakeFS docker container is operating.

sudo docker run \
  -v /mnt:/home/lakefs/ext_data \
  -p 8000:8000 \
  -e LAKEFS_DATABASE_TYPE=postgres \
...
  -e LAKEFS_AUTH_ENCRYPT_SECRET_KEY="lakefs" \
  -e LAKEFS_BLOCKSTORE_TYPE=local \
  -e LAKEFS_BLOCKSTORE_LOCAL_PATH="~/lakefs" \
  -e LAKEFS_BLOCKSTORE_LOCAL_IMPORT_ENABLED=true \
  -e LAKEFS_BLOCKSTORE_LOCAL_ALLOWED_EXTERNAL_PREFIXES="/mnt" \
  treeverse/lakefs:latest run

I can access all files in NAS via docker volume (

:/home/lakefs/ext_data

) but can't figure it out how to import files inside the LakeFS repository.

Hey! Quick question - I was working through the tutorial and merged denmark-lakes back into main. When I query the updated lakes.parquet, I get this. What did I mess up? Querying the lakes.parquet file in the denmark-lakes branch works fine. Also, hopefully you are enjoying reinvent. Another question - would the spatial extension for DuckDB work with lakeFS?

Good morning! I’m experimenting with new neat lakeFS Python SDK - very cool! @Niro I’m trying to list the repos in my lakeFS installation, how do I do that with the new SDK?

Morning guys, I am working with LakeFS python sdk and I have a couple of questions: 1. I have seen that one can use boto3 client to get the data from lakeFS, however I would like to know if I can use a client instance to pull all the data inside a repo as well 2. Do you have any integration with Huggingface datasets or pyarrow? 3. It seems that whenever I try to create a tag through the sdk, it gives me error (provided that I insert the latest commit id and the version number). Thanks to everyone for any answer.

Hello. I have some problem when I try to read file from lakefs repository. The repository is based on Azure Storage account. preinstall config is:

lakefsEndPoint = '<https://rntlj-151-236-193-133.a.free.pinggy.link/api>'

spark.conf.set("fs.lakefs.access.mode", "presigned")
spark.conf.set("fs.lakefs.impl", "io.lakefs.LakeFSFileSystem")
spark.conf.set("fs.lakefs.access.key", f"{lakefsAccessKey}")
spark.conf.set("fs.lakefs.secret.key", f"{lakefsSecretKey}")
spark.conf.set("fs.lakefs.endpoint", f"{lakefsEndPoint}")

spark.conf.set("spark.databricks.delta.logStore.crossCloud.fatal", "false")

installed libraries in the Databricks on Azure

io.lakefs:hadoop-lakefs-assembly:0.2.1 Maven
io.lakefs:lakefs-spark-client-312-hadoop3_2.12:0.10.0 Maven
lakefs-client PyPI

Attempt to read file:

repo_name = "test-rep-1" 
sourceBranch = "main" 

dataPath = 'tst_3a.csv'
dataPath = f"lakefs://{repo_name}/{sourceBranch}/{dataPath}"
print(dataPath)
print(f"Reading CSV from {dataPath}")
df = spark.read.csv(dataPath)
df.show()

causes error:

java.io.IOException: statObject
Py4JJavaError: An error occurred while calling o480.csv.
: java.io.IOException: statObject
	at io.lakefs.LakeFSFileSystem.getFileStatus(LakeFSFileSystem.java:764)
	at io.lakefs.LakeFSFileSystem.getFileStatus(LakeFSFileSystem.java:74)
	at org.apache.hadoop.fs.FileSystem.isDirectory(FileSystem.java:1777)
...
Caused by: io.lakefs.hadoop.shade.sdk.ApiException: Message: Content type "text/html; charset=utf-8" is not supported for type: class io.lakefs.hadoop.shade.sdk.model.ObjectStats
HTTP response code: 200
HTTP response body: <!DOCTYPE html>
<html lang="en">
  <head>
    <!-- Generated with Vite-->
...
</html>

HTTP response headers: {accept-ranges=[bytes], cache-control=[no-cache, no-store, no-transform, must-revalidate, private, max-age=0], content-length=[480], content-type=[text/html; charset=utf-8], date=[Thu, 30 Nov 2023 12:52:33 GMT], expires=[Thu, 01 Jan 1970 00:00:00 GMT], pragma=[no-cache], x-accel-expires=[0], x-frame-options=[SAMEORIGIN]}
	at io.lakefs.hadoop.shade.sdk.ApiClient.deserialize(ApiClient.java:925)
	at io.lakefs.hadoop.shade.sdk.ApiClient.handleResponse(ApiClient.java:1127)

and when i try to write parquet to the repositore i have the same error:

fileName = 'tst_3.csv'
dataPath = 'tst_3_df'

df = spark.read.csv(f'/{source_data}/{fileName}')
df.write.format("csv").save(f"lakefs://{repo_name}/{sourceBranch}/{dataPath}")

What can be a mistake ? And I use this instructions https: // lakefs.io/blog/databricks-lakefs-integration-tutorial/ (for Azure storage) but it doesn`t work for me.