Florentino Sainz
08/11/2023, 7:33 AMKartikey Mullick
08/11/2023, 5:59 PMindex.yaml
file
For reference, this is the relevant HCL script (Have removed the GKE and Cloud SQL modules)
I am sure I have specified either the chart or the repository arguments wrongly, but not sure what is the right one
provider "helm" {
kubernetes {
host = "${google_container_cluster.primary.endpoint}"
token = "${data.google_client_config.current.access_token}"
client_certificate = "${base64decode(google_container_cluster.primary.master_auth.0.client_certificate)}"
client_key = "${base64decode(google_container_cluster.primary.master_auth.0.client_key)}"
cluster_ca_certificate = "${base64decode(google_container_cluster.primary.master_auth.0.cluster_ca_certificate)}"
}
}
resource "helm_release" "my-lakefs" {
name = "my-lakefs"
chart = "charts/lakefs"
repository = "<https://github.com/treeverse/lakeFS>"
}
Oliver Dain
08/15/2023, 7:40 PMrclone
to work with the sample repo one gets by clicking "try without installing". I've setup my rclone config file like:
[lakefs]
type = s3
provider = AWS
env_auth = false
access_key_id = <redacted>
secret_access_key = <redacted>
endpoint = <https://big-zebra.us-east-1.lakefscloud.io>
no_check_bucket = true
where big-zebra.us-east-1...
is, I think, the correct endpoint for that sample repo (it's what shows in my address bar). But I get certificate errors:
$ rclone -vv ls lakefs:sample-repo/testing/
2023/08/15 12:35:09 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "-vv" "ls" "lakefs:sample-repo/testing/"]
2023/08/15 12:35:09 DEBUG : Creating backend with remote "lakefs:sample-repo/testing/"
2023/08/15 12:35:09 DEBUG : Using config file from "/home/oliver/.config/rclone/rclone.conf"
2023/08/15 12:35:09 DEBUG : fs cache: renaming cache item "lakefs:sample-repo/testing/" to be canonical "lakefs:sample-repo/testing"
2023/08/15 12:35:49 DEBUG : 2 go routines active
2023/08/15 12:35:49 Failed to ls: RequestError: send request failed
caused by: Get "<https://sample-repo.big-zebra.us-east-1.lakefscloud.io/?delimiter=&encoding-type=url&max-keys=1000&prefix=testing%2F>": x509: certificate is valid for *.<http://us-east-1.lakefscloud.io|us-east-1.lakefscloud.io>, <http://us-east-1.lakefscloud.io|us-east-1.lakefscloud.io>, not <http://sample-repo.big-zebra.us-east-1.lakefscloud.io|sample-repo.big-zebra.us-east-1.lakefscloud.io>
I see it's a wildcard cert for *.<http://us-east-1.lakefscloud.io|us-east-1.lakefscloud.io>
which seems like it should work, but it also looks like rclone
is adding the repo name (sample-repo
) to the hostname which seems incorrect. What am I doing wrong?Robin Moffatt
08/16/2023, 12:04 PMlocal
command, I get an error when I try to commit - any ideas?
lakectl local commit . -m "Remove employee data and add ratings information" --pre-sign=false
Getting branch: main
diff 'local:///Users/rmoff/work/orion' <--> '<lakefs://quickstart/7eaeafd3fb90df419ca26eca69ba2b153eae13bd3dccf80ba47cbed1260cb0c1/data/OrionStar/>'...
diff '<lakefs://quickstart/38df1480d835fe52dc6db31b87ef35b78403d0c9c621c74a6d032c686551abc5/data/OrionStar/>' <--> '<lakefs://quickstart/main/data/OrionStar/>'...
delete remote path: EMPLOYEE_ORGANIZATI~ ... done! [%0 in 8ms]
delete remote path: EMPLOYEE_ADDRESSES.~ ... done! [%0 in 8ms]
delete remote path: EMPLOYEE_DONATIONS.~ ... done! [%0 in 8ms]
delete remote path: EMPLOYEE_PAYROLL.csv ... done! [%0 in 8ms]
upload RATINGS.csv ... done! [6.86KB in 9ms]
upload RATINGS.csv failed: upload to backing store: MissingRegion: could not find region configuration
Error executing command.
yiming song
08/17/2023, 2:20 AMyiming song
08/17/2023, 2:26 AMMẫn Phạm
08/17/2023, 8:55 AMHT
08/21/2023, 2:44 AMimport s3fs
import smartfs
conf = smartfs.load_rclone_profile("lakefs")
fs = s3fs.S3FileSystem(
anon=False,
endpoint_url=conf["endpoint"],
key=conf["access_key_id"],
secret=conf["secret_access_key"],
)
## Comment/uncomment following to trigger the bug.
fs.find("<s3://test-repo/main/>",maxdepth=None,withdirs=False)
print(fs.find("<s3://test-repo/main/>",maxdepth=1,withdirs=True))
The content looks like this:
$ rclone ls lakefs:test-repo/main/
0 dir1/fileB
0 fileA
When I run the code above, the second find
should list 2 entries: fileA
, dir1
but somehow it is missing dir1
But if I comment out the first find
, then the second find
will list the 2 entries correctely.
I don't have an AWS account in order to check if it's a lakefs issue or s3fs issue.
Can someone with a real AWS account test this ?
Context:
Lakefs self hosted: 0.104.0
pip freeze:
aiobotocore==2.5.2
boto3==1.21.21
botocore==1.29.161
s3fs==2023.6.0
s3transfer==0.5.2
Edit: I was decribing the expected result slightly wrong.暗号
08/21/2023, 8:07 AMlakectl local init <lakefs://datasetest/test1/> E:\WorkProjects\data
lakectl local list
+-----------+------------+---------------+
| DIRECTORY | REMOTE URI | SYNCED COMMIT |
+-----------+------------+---------------+
+-----------+------------+---------------+
D:\Develops\lakefs\lakeFS_0.107.0_Windows_x86_64>lakectl local clone <lakefs://datasetest/test1/> E:\lakedata
EOF
Error executing command.
Jonas
08/22/2023, 8:53 AMfs:ListRepositories
by something like arn:lakefs:fs:::repository/{prefix}*
? We have a an internal structure with different teams who don't have to see all the repos by all other teams. The doc only states *
as possible resourceRich McLaughlin
08/22/2023, 1:59 PM2023-08-22T13:54:49.349664594Z time="2023-08-22T13:54:49Z" level=info msg="lakeFS run" func=cmd/lakefs/cmd.glob..func8 file="cmd/run.go:94" version=0.107.0
2023-08-22T13:54:49.349684195Z time="2023-08-22T13:54:49Z" level=fatal msg="Failed to open KV store" func=cmd/lakefs/cmd.glob..func8 file="cmd/run.go:102" error="connect failed: failed to connect to `host=/tmp user=lakefs database=`: dial error (dial unix /tmp/.s.PGSQL.5432: connect: no such file or directory)"
Connection string works in psql are we missing variables?Rich McLaughlin
08/23/2023, 8:23 AMversion: "3.5"
name: lakefs
services:
lakefs:
image: treeverse/lakefs:latest
ports:
- "8000:8000"
When i try to use the container on its own it fails with the following errors
2023-08-23T08:20:56.611Z ERROR - Container jigsaw-lakefs_0_61e8c142 for site jigsaw-lakefs has exited, failing site start
2023-08-23T08:20:56.620Z ERROR - Container jigsaw-lakefs_0_61e8c142 didn't respond to HTTP pings on port: 8000, failing site start. See container logs for debugging.
2023-08-23T08:20:56.641Z INFO - Stopping site jigsaw-lakefs because it failed during startup.
We have the WEBSITES_PORT set to 8000 and have tried specifying the ports in the start up command but just will not work.
Unfortunately we can not use docker compose option as we are not able to bind the website address to a vnet to secure the Postgress DB. Any help would be appreciated.Eirik Knævelsrud
08/23/2023, 10:58 AMlingyu zhang
08/24/2023, 11:51 AMMichael Gaebel
08/24/2023, 1:53 PMAn error occurred while calling o128.getDynamicFrame. io/lakefs/iceberg/LakeFSCatalog has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0
I've used the same libraries listed in the setup link above (lakefs-iceberg:0.1.2, iceberg-spark-runtime-3.3_2.12:1.3.0), and I've also tried using the same spark runtime version that was listed as a test dependency for io.lakefs:lakefs-iceberg:0.1.2
in Maven. Am I missing something obvious, or is there a version of the lakefs-iceberg lib that was compiled with Java runtime 52.0?Rich McLaughlin
08/24/2023, 8:16 PMHT
08/24/2023, 11:57 PMlakefs_client
python sdk:
File "/path/lakefs_helper.py", line 102, in merge
return self.lakefs.refs.merge_into_branch(
File "/path/venv/lib64/python3.10/site-packages/lakefs_client/api/refs_api.py", line 869, in merge_into_branch
return self.merge_into_branch_endpoint.call_with_http_info(**kwargs)
File "/path/venv/lib64/python3.10/site-packages/lakefs_client/api_client.py", line 835, in call_with_http_info
return self.api_client.call_api(
File "/path/venv/lib64/python3.10/site-packages/lakefs_client/api_client.py", line 409, in call_api
return self.__call_api(resource_path, method,
File "/path/venv/lib64/python3.10/site-packages/lakefs_client/api_client.py", line 203, in __call_api
raise e
File "/path/venv/lib64/python3.10/site-packages/lakefs_client/api_client.py", line 196, in __call_api
response_data = self.request(
File "/path/venv/lib64/python3.10/site-packages/lakefs_client/api_client.py", line 455, in request
return <http://self.rest_client.POST|self.rest_client.POST>(url,
File "/path/venv/lib64/python3.10/site-packages/lakefs_client/rest.py", line 267, in POST
return self.request("POST", url,
File "/path/venv/lib64/python3.10/site-packages/lakefs_client/rest.py", line 224, in request
raise ServiceException(http_resp=r)
lakefs_client.exceptions.ServiceException: (503)
Reason: Service Unavailable
HTTP response headers: HTTPHeaderDict({'content-length': '118', 'content-type': 'text/plain', 'date': 'Thu, 24 Aug 2023 23:47:57 GMT'})
HTTP response body: upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection termination
My helper function that do the merge is:
def merge(self, source_branch, dest_branch, metadata={}, message=None):
try:
# Check if there is a need to merge:
diffs = self.diff(left_ref=dest_branch, right_ref=source_branch)
if len(diffs) == 0:
print(f"No changes detected. Skipping merge {source_branch} to {dest_branch}")
return
if message:
merge = models.Merge(message=message, metadata=metadata)
else:
merge = models.Merge(metadata=metadata)
return self.lakefs.refs.merge_into_branch(
repository=self.repo_name,
source_ref=source_branch,
destination_branch=dest_branch,
merge=merge,
)
except Exception as e:
# traceback.print_exc()
print(f"ERROR: Failed to merge {source_branch} to {dest_branch}")
raise e
With self.lakefs as :
configuration = lakefs_client.Configuration()
configuration.username = conf["access_key_id"]
configuration.password = conf["secret_access_key"]
configuration.host = conf["endpoint"]
self.lakefs = LakeFSClient(configuration)
But then if I do the merge of that 2 exact same branch via the UI, it succeed !
Our self deployed server running : 107.0
And the lakefs_client is 0.107.0Bertrand Gallice
08/28/2023, 9:10 AM$ lakectl import \
--from <s3://jaffle_shop_repo/raw_payments.csv> \
--to <lakefs://jaffle-shop-repo/main/raw_payments.csv>
Import failed: import error: error on ingest: NoSuchKey:
status code: 404, request id: tx00000000000000a72c046-0064ec5ed8-b6658b2f-fra1b, host id:
the Lakefs repo uses <s3://jaffle_shop_repo>
as the storage namespace, and is located at the root of the S3 bucket.
It doesn’t seem like a credentials problem, since the lakectl fs upload
command works to add data in the repo, and write this data in the S3 storage repo’s folder.
I suspect that it’s just a uri path problem, tried several variations but none seem to work
Any idea on what could be wrong?Taha Sadiki
08/29/2023, 12:27 PMDieu M. Nguyen
08/29/2023, 9:01 PMto_zarr()
step is getting errors.
import dask.array as da
import xarray as xr
import numpy as np
configuration = lakefs_client.Configuration()
configuration.username = access_key_id
configuration.password = secret_access_key
configuration.host = endpoint_url
client = LakeFSClient(configuration)
repo = "zarr-test"
branch = "zarr-store"
client.branches.create_branch(
repository=repo,
branch_creation=models.BranchCreation(
name=branch,
source="main"))
# Create random array
state = da.random.RandomState(1234)
shape = (180, 360, 400)
chunk_shape = (36, 72, 200)
nlats, nlons, ntimes = shape
arr = state.random(shape, chunks=chunk_shape)
arr = da.random.random(shape, chunks=chunk_shape)
ds = xr.Dataset(
data_vars={
"precipitation": xr.DataArray(arr, dims=('lat', 'lon', 'time'))
},
coords={
"lat": xr.DataArray(np.linspace(-90, 90, num=nlats, endpoint=False), dims='lat'),
"lon": xr.DataArray(np.linspace(-180, 180, num=nlons, endpoint=False), dims='lon'),
"time": xr.date_range(start="2000-06-01", freq="D", periods=ntimes)
},
attrs={
"description": "GPM IMERG test dataset"
}
)
# Write the first 200 time slices
ds_0 = ds.isel(time=slice(0, 200))
s3a_gateway_path = f's3a://{repo}/{branch}/precipitation_data.zarr'
task = ds_0.to_zarr(s3a_gateway_path,
# zarr_version=3,
mode='w',
compute=False)
Without zarr_version
, I get PermissionError: Access Denied
. If I set zarr_version=3
, I get KeyError: 'zarr.json'
. Maybe I am setting the s3a_gateway_path
incorrectly?lingyu zhang
08/31/2023, 12:18 PMsizeBytes
and modifiedTime
of a file when comparing local files with remote commits. However, I believe there may be some risks associated with this approach. For instance, if two clients have different system times and both edit the same file, let's say a.txt
, the sizeBytes
and modifiedTime
could be the same on both clients, but the contents are different. Consequently, when I commit the changes on client A and then pull from the remote on client B, the modifications could be lost. So here are my questions:
1. Do you have any evidence or user cases to prove this situation is rare?
2. What is the probability of encountering this risk? Have any tests been conducted?
3. Why do we only use second-level precision for the modifiedTime
and not something more precise like nanoseconds? (Unix time in nanoseconds cannot be represented by an int64 for dates only prior to the year 1678 or after 2262)
Thanks a lot! :)Justin Pottenger
09/02/2023, 5:52 PMINFO [2023-09-02T17:23:06Z]build/pkg/auth/service.go:189 pkg/auth.NewAuthService initialized Auth service service=auth_service
WARNING[2023-09-02T17:23:06Z]build/pkg/cloud/aws/metadata.go:64 pkg/cloud/aws.(*MetadataProvider).GetMetadata.func1 Tried to to get AWS account ID for BI error="InvalidParameterValue: Unsupported action GetCallerIdentity\n\tstatus code: 400, request id: 178124CA4D3D18C0"
Digging into the code it appears that the AWS package is trying to find the email of the AWS account, and since MinIO doesn’t have one, it fails. Relevant code is here and here.
Before I fork LakeFS and try to dig in myself I thought I would see if anyone here had any ideas on a more elegant solution (aka supported) solution?
TLDR: LakeFS + Minio + Headless = not supported?Florentino Sainz
09/04/2023, 8:24 AMTruce Wallace
09/04/2023, 9:47 PMOh my! An error occurred: Unauthorized
when trying to registerTruce Wallace
09/05/2023, 8:29 AMDieu M. Nguyen
09/05/2023, 8:37 PMspark-submit
command as directed by the documentation. As far as I can tell, the command ran and finished without errors. But in S3, I don’t see the list of objects removed in _lakefs/retention/gc/unified/<RUN_ID>/deleted/
and my storage didn’t go down, so I assume objects from old versions haven’t been deleted. Do you have any ideas about this? Note: I just set the GC policy after already writing all my versions - Is this why?HT
09/06/2023, 1:31 AMYaphet Kebede
09/06/2023, 5:09 PMPrakash Kumar
09/08/2023, 4:11 PMPrakash Kumar
09/08/2023, 10:48 PM