To run garbage collection, is the assumption that I need to have a spark cluster set up?

Hey, we are using an S3 compatible bucket, and when we try to upload data with "--direct" flag, we get the following error

upload to backing store: MissingRegion: could not find region configuration

I believe this can be simply solved by setting ``blockstore.s3.region``, but what should we set it to when we are not using AWS directly?

Hi , quick question is there a way on lakefs python api , to direct get_objects to output file into a specific dir instead of root temp dir configured on the client object?

Any idea why I might be observing different behavior between the GUI and

lakectl

when performing multiple imports? Let's say I have S3 bucket 1 with cat.jpg at and S3 bucket 2 with dog.jpg at In GUI: • I import cat.jpg to main branch, merge _main_imported • Then, I import dog.jpg to main branch, merge _main_imported Output (expected): cat.jpg and dog.jpg are now both in main In CLI: •

lakectl import --from <s3://cat_bucket/> --to <lakefs://repo/main/>

and manually merge _main_imported •

lakectl import --from <s3://dog_bucket/> --to <lakefs://repo/main/>

. Now, when I go to compare branches, I can see that cat.jpg is being removed if I merge Output (not expected): only dog.jpg is now in main

Sorry for all the questions - but I got another one: is there some way to get a commit number before a commit is made?

Hi, I am getting this error when trying to read a csv file via lakefs repo

23/02/06 12:13:28 WARN FileStreamSink: Assume no metadata directory. Error while looking for metadata directory in the path: <lakefs://ragoldstandard/main/bronze_layer/sample.csv>.
java.io.IOException: statObject
	at io.lakefs.LakeFSFileSystem.getFileStatus(LakeFSFileSystem.java:779)
	at io.lakefs.LakeFSFileSystem.getFileStatus(LakeFSFileSystem.java:46)
	at org.apache.hadoop.fs.FileSystem.isDirectory(FileSystem.java:1777)
	at org.apache.spark.sql.execution.streaming.FileStreamSink$.hasMetadata(FileStreamSink.scala:54)
	at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:370)
	at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:274)
	at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:245)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:245)
	at org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:571)
	at jdk.internal.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
	at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: io.lakefs.hadoop.shade.api.ApiException: Content type "text/html; charset=utf-8" is not supported for type: class io.lakefs.hadoop.shade.api.model.ObjectStats
	at io.lakefs.hadoop.shade.api.ApiClient.deserialize(ApiClient.java:822)
	at io.lakefs.hadoop.shade.api.ApiClient.handleResponse(ApiClient.java:1020)
	at io.lakefs.hadoop.shade.api.ApiClient.execute(ApiClient.java:944)
	at io.lakefs.hadoop.shade.api.ObjectsApi.statObjectWithHttpInfo(ObjectsApi.java:1478)
	at io.lakefs.hadoop.shade.api.ObjectsApi.statObject(ObjectsApi.java:1451)
	at io.lakefs.LakeFSFileSystem.getFileStatus(LakeFSFileSystem.java:775)

Please can some help

Hi everyone! I was wondering where logs such as the ones mentioned in this issue https://github.com/treeverse/lakeFS/issues/4261 are stored? I am trying to contribute to that issue and I can't for the life of me trace them down after doing any kind of operation.

Hey, while uploading files, the process failed and we noticed the following errors:

time="2023-02-16T07:39:00Z" level=warning msg="error verifying credentials for key" func=pkg/gateway.AuthenticationHandler.func1 file="build/pkg/gateway/middleware.go:55" authenticator=sigv4 error=SignatureDoesNotMatch key=MASKED

Is it possible to see more detailed error of why this happened? Any thoughts what could cause that? CC @Conor Simmons for visibility

Hi everyone! I’m exploring LakeFS for our data versioning use case. I followed the tutorial to create a repo -> upload file -> commit to master. However, I’m unable to actually read the file from the MinIO bucket using Pandas.

import pandas as pd
import os
import s3fs

class S3FileSystemPatched(s3fs.S3FileSystem):
    def __init__(self, *k, **kw):
        super(S3FileSystemPatched, self).__init__(*k,
                                                  key = os.environ["AWS_ACCESS_KEY_ID"],
                                                  secret = os.environ["AWS_SECRET_ACCESS_KEY"],
                                                  client_kwargs={'endpoint_url': os.environ["AWS_S3_ENDPOINT"]},
                                                  **kw)
        print('S3FileSystem is patched')
s3fs.S3FileSystem = S3FileSystemPatched

data = pd.read_csv("<s3://example/master/test.csv>")

it throws

FileNotFoundError: example/master/test.csv

Is it possible to rename repositories? I was not able to do that via the ui

Is there a way to enable ssl on lakefs? the default instructions just enable the web on http

The lakeFS instructions state that it doesn’t support storage classes? Is that JUST on the PutObject operation? I ran a cp s3://lakefs-testing/main s3://lakefs-testing/main —recursive —storage-class STANDARD-IA and it seems to have copied the data to a different storage class. what is actually not supported?

Hi! I was looking to use the

gateways.s3.fallback_url

setting, but haven't been able to figure out exactly how to use it. Is there any documentation besides the configuration reference guide? In particular, I'm unsure how to pass authentication credentials to the fallback backend: If I specify credentials that don't exist in lakeFS but are valid on the fallback S3 endpoint, I get an

InvalidAccessKeyId

error returned by the

aws s3

command. If I specify the lakeFS credentials and try to access a bucket that would be handled by the fallback case, I get an

AccessDenied

error. Do the credentials need to be identical for both the lakeFS S3 endpoint and the fallback? Thanks for your help! 🙏

after making changes to a policy attached to a user's authorisation with RBAC, should that user have to re-auth for them to take effect? and/or, what's the error in this policy here to block the user from accessing the files (which they can do, see on the right)

I'm currently trying to configure lakefs to restrict users access to a single repo. If I take the sample from here (https://docs.lakefs.io/reference/authorization.html#additional-policies) and remove everything else I end up with the following error message

insufficient permissions: uM: insufficient permissions at ot (<http://localhost:8090/assets/index-7bf1aa94.js:71:93127>) at async T$.list (<http://localhost:8090/assets/index-7bf1aa94.js:71:100206>) at async <http://localhost:8090/assets/index-7bf1aa94.js:71:111901>

when accessing http://localhost:8090/repositories. What am I missing? EDIT: I can restrict write access, my issue though is that I don't even want the user to see repos he has no access to

Hi all, Having some trouble getting rclone to work nicely with Lakefs. I've manged to spin up Lakefs in a container and joined it to a network I had created earlier. I have rclone in another container on that same network so service names are resolved internally in Docker.

[lakefs]
env_auth = false
type = s3
provider = Other
endpoint = <http://lakefs:8000>
region = use-east-1
secret_access_key = $SECRET_ACCESS_KEY
access_key_id = $ACCESS_KEY_ID
force_path_style = true
no_check_bucket = true

I have to use

Other

as a provider because rclone will override

force_path_style

with

false

if I use the

AWS

provider as suggested on the docs. If I then run:

rclone ls -vv --dump=bodies lakefs:repo/branch/path

I get the following error

2023/03/02 17:11:13 Failed to lsd with 2 errors: last error was: MissingFields: Missing fields in request.
        status code: 400, request id: , host id:

Anyone have any idea how I go about resolving this? ``````

I'm referring to https://docs.lakefs.io/integrations/duckdb.html to try and work with some parquet files that I have in lakeFS. I'm trying to set the endpoint but the config's not taking (and if I try to query them it goes to S3)

drones.duckdb> SET s3_endpoint='<http://rmoff-test.us-east-2.lakefscloud.io|rmoff-test.us-east-2.lakefscloud.io>';
0 rows in set
Time: 0.000s
drones.duckdb> SELECT * FROM duckdb_settings() where name like 's3_e%';
+-------------+------------------+------------------------------------------+------------+
| name        | value            | description                              | input_type |
+-------------+------------------+------------------------------------------+------------+
| s3_endpoint | <http://s3.amazonaws.com|s3.amazonaws.com> | S3 Endpoint (default '<http://s3.amazonaws.com|s3.amazonaws.com>') | VARCHAR    |
+-------------+------------------+------------------------------------------+------------+
1 row in set
Time: 0.006s

Maybe this is more a duckdb question than lakeFS, but interested if anyone has seen this

Re OIDC - it says that the feature is deprecated in this part of the docs, but not in this part. I assume the first part is valid based on the update. What authentication options will be supported in the future releases? Thanks!

Write back from duckDB to lakeFS I get an ambiguous error when I try to write to a protected branch. I'm not familiar enough with the interface between the two tools here. Is lakeFS throwing a more verbose exception that duckDB is squashing, or is lakeFS just throwing a generic error that could be better ("Cannot write to a protected branch" for example)

D export database '<s3://drones03/main/drone-registrations/>' (format parquet);
 94% ▕████████████████████████████████████████████████████████▍   ▏ Error: Invalid Error: IO Error: Unexpected response during S3 multipart upload finalization

It works just fine writing to a branch - I'm just interested in where to follow up re. improving the error

I suspect this is a duckDB issue rather than lakeFS per se, but asking here since y'all so helpful anyway 🙂 Running DuckDB locally to work with data on lakeFS cloud works just great. However, in trying to use it against my local lakeFS install I'm hitting problems. Using

aws s3

I can connect to my lakeFS just fine using the S3 gateway:

$ aws s3 --endpoint-url <http://127.0.0.1:8000> ls <s3://drones03/main/>
2023-03-06 09:59:24     119663 Registations-P107-Active-2017.parquet

But if I use DuckDB it errors, I think because it is forcing the connection to HTTPS (which it's not)

D SET s3_endpoint='<http://127.0.0.1:8000>';
D select * from read_parquet('<s3://drones03/main/Registations-P107-Active-2017.parquet>');
Error: Invalid Error: IO Error: Connection error for HTTP HEAD to '<https://http>://127.0.0.1:8000/drones03/main/Registations-P107-Active-2017.parquet'

D SET s3_endpoint='127.0.0.1:8000';
D select * from read_parquet('<s3://drones03/main/Registations-P107-Active-2017.parquet>');
Error: Invalid Error: IO Error: SSLConnection error for HTTP HEAD to '<https://127.0.0.1:8000/drones03/main/Registations-P107-Active-2017.parquet>'

Is there a way around this?

yes, set the ssl off and access using path

I'm looking at RBAC. What's the default effect for an action with no explicit `allow`/`deny` ? i.e. if I create a user and don't grant it any

fs:*

(neither allow nor deny), is that the same as granting

deny

? Put another way, if I only want to allow certain actions, do I need to

deny

the others, or can I simply

allow

the ones that I want?

I'm trying to use the S3 gateway from R, but not having much luck. • It connects without error, but returns no objects when listing 'buckets' or objects within a specified bucket. • The same lakeFS server works fine with boto and

aws

CLI doing the same thing • The lakeFS server log for the HTTP interaction is the same for the R (doesn't work) and Python (does work) • The R code works fine listing buckets and objects with Minio. • R library and S3 HTTP code Any pointers to what to check next? thanks. Full details in thread

Hi, im trying to run the lakefs server but i keep getting this error when i try to login (i removed my email for privacy) Any thoughts?

Hi, Please is it possible to read data like a csv or load a json file directly from a lakefs repo?? if yes, is there any documentation around it

Hi lakefs team! I am getting an

500 Internal Server Error

that I reported as a bug some time ago and was already supposed to be solved. I have just added the new details with the error as a new comment in the related Github Issue: https://github.com/treeverse/lakeFS/issues/4909 Can someone help me reopening the issue?

Hi team, I am trying configure lakefs with azure blob storage using storage access key

sudo docker run \
  --name lakefs \
  -p 80:8000 \
  -e LAKEFS_DATABASE_TYPE="postgres" \
  -e LAKEFS_DATABASE_POSTGRES_CONNECTION_STRING=$LAKEFS_DATABASE_POSTGRES_CONNECTION_STRING \
  -e LAKEFS_AUTH_ENCRYPT_SECRET_KEY=$LAKEFS_AUTH_ENCRYPT_SECRET_KEY \
  -e LAKEFS_BLOCKSTORE_TYPE="azure" \
  -e LAKEFS_BLOCKSTORE_AZURE_STORAGE_ACCOUNT=$LAKEFS_BLOCKSTORE_AZURE_STORAGE_ACCOUNT \
  -e LAKEFS_BLOCKSTORE_AZURE_STORAGE_ACCESS_KEY=$LAKEFS_BLOCKSTORE_AZURE_STORAGE_ACCESS_KEY \

and I get the following error

time="2023-03-11T22:41:44Z" level=error msg="failed to get azure blob from container &{%!s(*generated.ContainerClient=&{<https://lakefs11673f4e.blob.core.windows.net/mds> {[0x1400120 0xc008045c00 0xc0080b01c0 0xc008077770 0xc00804ff38 0x13ffda0 0x13ffaa0 {0xc000159350}]}}) %!s(*exported.SharedKeyCredential=<nil>)} key &{%!s(*generated.BlobClient=&{<https://lakefs11673f4e.blob.core.windows.net/mds/dummy> {[0x1400120 0xc008045c00 0xc0080b01c0 0xc008077770 0xc00804ff38 0x13ffda0 0x13ffaa0 {0xc000159350}]}}) %!s(*generated.BlockBlobClient=&{<https://lakefs11673f4e.blob.core.windows.net/mds/dummy> {[0x1400120 0xc008045c00 0xc0080b01c0 0xc008077770 0xc00804ff38 0x13ffda0 0x13ffaa0 {0xc000159350}]}}) %!s(*exported.SharedKeyCredential=<nil>)}" func="pkg/logging.(*logrusEntryWrapper).Errorf" file="build/pkg/logging/logger.go:262" error="DefaultAzureCredential: failed to acquire a token.\nAttempted credentials:\n\tEnvironmentCredential: missing environment variable AZURE_TENANT_ID\n\tManagedIdentityCredential: no default identity is assigned to this resource\n\tAzureCLICredential: Azure CLI not found on path" host=68.219.229.172 method=POST operation_id=CreateRepository path=/api/v1/repositories request_id=94bca5e0-f1e3-49e0-ab44-20313230bbdf service_name=rest_api user=alex

time="2023-03-11T22:41:44Z" level=warning msg="Could not access storage namespace" func="pkg/api.(*Controller).CreateRepository" file="build/pkg/api/controller.go:1406" error="DefaultAzureCredential: failed to acquire a token.\nAttempted credentials:\n\tEnvironmentCredential: missing environment variable AZURE_TENANT_ID\n\tManagedIdentityCredential: no default identity is assigned to this resource\n\tAzureCLICredential: Azure CLI not found on path" reason=unknown service=api_gateway storage_namespace="<https://lakefs11673f4e.blob.core.windows.net/mds>"

Reading the above, it looks like lakefs tries to use my storage access key, then fails and tries to fall back on Service principal credentials. I verified that shared key works using python. What should I do?

hi Folks. I've been trying to run the everything bagel docker-compose which used to work. Yet recently, after configuring lakectl against lakeFS cloud, the bagel fails on setting lakeFS without notifying about it. the lakeFS container doesn't exist on

docker ps

. I've noticed that there is a lakeCTL command in the

lakefs-setup -> && lakectl repo create <lakefs://example>

<s3://example>

any idea if the things are connected? or how to fix it?

lakefs-setup:
  image: treeverse/lakefs:latest
  container_name: lakefs-setup
  depends_on:
    - postgres
    - minio-setup
  environment:
    - LAKEFS_AUTH_ENCRYPT_SECRET_KEY=some random secret string
    - LAKEFS_DATABASE_CONNECTION_STRING=<postgres://lakefs:lakefs@postgres/postgres?sslmode=disable>
    - LAKECTL_CREDENTIALS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
    - LAKECTL_CREDENTIALS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    - LAKECTL_SERVER_ENDPOINT_URL=<http://lakefs:8000>
    - LAKEFS_BLOCKSTORE_TYPE=s3
  entrypoint: ["/app/wait-for", "postgres:5432", "--", "sh", "-c",
    "lakefs setup --user-name docker --access-key-id AKIAIOSFODNN7EXAMPLE --secret-access-key wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY && lakectl repo create <lakefs://example> <s3://example>"
    ]

i'm playing around with DuckDB on the object page. Should something like this work? I'm getting no rows returned and from devtools it's looking for the wrong path.

I'm looking at the

branch revert

docs and trying to do a revert, but hitting an unclear error. I want to rollback the last commit (ideally without having to look up its specific ID). Is that possible?

$ lakectl branch revert <lakefs://quickstart/main> HEAD~1

Branch: <lakefs://quickstart/main>
Are you sure you want to revert the effect of commits HEAD~1: y
get commit from ref HEAD~1: not found
404 Not Found