Channels
announcements
blockers_for_windward
career-opportunities
celebrations
cuddle-corner
data-discussion
data-events
dev
events
general
help
iceberg-integration
lakefs-for-beginners
lakefs-hubspot-cloud-registration-email-automation
lakefs-releases
lakefs-suggestions
lakefs-twitter
linen-dev
memes-and-banter
say-hello
stackoverflow
test
Title
m
Matija Teršek
02/17/2023, 8:58 PMHey, while uploading files, the process failed and we noticed the following errors:
time="2023-02-16T07:39:00Z" level=warning msg="error verifying credentials for key" func=pkg/gateway.AuthenticationHandler.func1 file="build/pkg/gateway/middleware.go:55" authenticator=sigv4 error=SignatureDoesNotMatch key=MASKEDi
Iddo Avneri
02/17/2023, 9:15 PMHi!
We will take a look and get back to you.
Ib the meantime, can you describe how you are uploading the files?
c
Conor Simmons
02/17/2023, 9:30 PMIt's using the staging API following this discussion: https://lakefs.slack.com/archives/C016726JLJW/p1675461592588169?thread_ts=1675457296.031589&cid=C016726JLJW
i
Iddo Avneri
02/17/2023, 9:34 PMThank you for the context. We’ll look into it and get back.
👍 1
a
Amit Kesarwani
02/17/2023, 10:31 PM@Matija Teršek @Conor Simmons Based on the error message you are getting: “error verifying credentials for key”, lakeFS credentials are wrong. Can you double check your lakeFS access and secret key?
m
Matija Teršek
02/17/2023, 10:31 PMThe credentials are correct since the uploading process works for majority of the time.
staging*
a
Amit Kesarwani
02/17/2023, 10:34 PMStaging Python program makes multiple API calls. Do you know which API call (or line #) is failing and what API error response are you getting?
c
Conor Simmons
02/17/2023, 10:51 PMlakefs_client.objects.stage_objecta
Amit Kesarwani
02/17/2023, 11:10 PM@Conor Simmons Are you getting this error for certain types of files? Somebody reported similar error message when filename has unicode characters in it.
c
Conor Simmons
02/19/2023, 7:56 PMNot that I've noticed. The files are images (PNG) and JSON
m
Matija Teršek
02/22/2023, 10:49 AMAny updates @Amit Kesarwani?
j
Jonathan Rosenberg
02/22/2023, 1:03 PMHi @Matija Teršek
It looks like lakeFS is failing over the sigv4 header verification which is used by our lakeFS s3 gateway. I don’t see how this can happen from the provided code as it’s not using the S3 gateway.
Could you share the piece of code that didn’t work (and the line at which the error happened) and the stack trace (or the client error if you don’t have it)?
c
Conor Simmons
02/22/2023, 4:27 PM@Jonathan Rosenberg here is the stack trace
Traceback (most recent call last):
File "/home/conor/Luxonis/envs/lakefs/bin/luxonis_ml", line 33, in <module>
sys.exit(load_entry_point('luxonis-ml', 'console_scripts', 'luxonis_ml')())
File "/home/conor/Luxonis/luxonis-ml/src/luxonis_ml/luxonis_ml.py", line 156, in main
_dataset_push(args)
File "/home/conor/Luxonis/luxonis-ml/src/luxonis_ml/luxonis_ml.py", line 99, in _dataset_push
artifact.push(message=args.message)
File "/home/conor/Luxonis/luxonis-ml/src/luxonis_ml/ops/artifacts.py", line 196, in push
self.lakefs_client.objects.stage_object(self.name, self.dataset.branch, path, staged_object)
File "/home/conor/Luxonis/envs/lakefs/lib/python3.9/site-packages/lakefs_client/api/objects_api.py", line 1409, in stage_object
return self.stage_object_endpoint.call_with_http_info(**kwargs)
File "/home/conor/Luxonis/envs/lakefs/lib/python3.9/site-packages/lakefs_client/api_client.py", line 835, in call_with_http_info
return self.api_client.call_api(
File "/home/conor/Luxonis/envs/lakefs/lib/python3.9/site-packages/lakefs_client/api_client.py", line 409, in call_api
return self.__call_api(resource_path, method,
File "/home/conor/Luxonis/envs/lakefs/lib/python3.9/site-packages/lakefs_client/api_client.py", line 203, in __call_api
raise e
File "/home/conor/Luxonis/envs/lakefs/lib/python3.9/site-packages/lakefs_client/api_client.py", line 196, in __call_api
response_data = self.request(
File "/home/conor/Luxonis/envs/lakefs/lib/python3.9/site-packages/lakefs_client/api_client.py", line 463, in request
return self.rest_client.PUT(url,
File "/home/conor/Luxonis/envs/lakefs/lib/python3.9/site-packages/lakefs_client/rest.py", line 277, in PUT
return self.request("PUT", url,
File "/home/conor/Luxonis/envs/lakefs/lib/python3.9/site-packages/lakefs_client/rest.py", line 224, in request
raise ServiceException(http_resp=r)
lakefs_client.exceptions.ServiceException: (524)
Reason:
HTTP response headers: HTTPHeaderDict({'Date': 'Fri, 17 Feb 2023 10:30:30 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Set-Cookie': 'cf_use_ob=0; path=/; expires=Fri, 17-Feb-23 10:31:00 GMT', 'X-Frame-Options': 'SAMEORIGIN', 'Referrer-Policy': 'same-origin', 'Cache-Control': 'private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0', 'Expires': 'Thu, 01 Jan 1970 00:00:01 GMT', 'CF-RAY': '79add1c3dcb0f790-DEN', 'Server': 'cloudflare'})
HTTP response body: <!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]> <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]> <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>
<title><http://lakefs.example.com|lakefs.example.com> | 524: A timeout occurred</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/main.css" />
</head>
<body>
<div id="cf-wrapper">
<div id="cf-error-details" class="p-0">
<header class="mx-auto pt-10 lg:pt-6 lg:px-8 w-240 lg:w-full mb-8">
<h1 class="inline-block sm:block sm:mb-2 font-light text-60 lg:text-4xl text-black-dark leading-tight mr-2">
<span class="inline-block">A timeout occurred</span>
<span class="code-label">Error code 524</span>
</h1>
<div>
Visit <a href="<https://www.cloudflare.com/5xx-error-landing?utm_source=errorcode_524&utm_campaign=lakefs.example.com>" target="_blank" rel="noopener noreferrer"><http://cloudflare.com|cloudflare.com></a> for more information.
</div>
<div class="mt-3">2023-02-17 10:30:30 UTC</div>
</header>
<div class="my-8 bg-gradient-gray">
<div class="w-240 lg:w-full mx-auto">
<div class="clearfix md:px-8">
<div id="cf-browser-status" class=" relative w-1/3 md:w-full py-15 md:p-0 md:py-8 md:text-left md:border-solid md:border-0 md:border-b md:border-gray-400 overflow-hidden float-left md:float-none text-center">
<div class="relative mb-10 md:m-0">
<span class="cf-icon-browser block md:hidden h-20 bg-center bg-no-repeat"></span>
<span class="cf-icon-ok w-12 h-12 absolute left-1/2 md:left-auto md:right-0 md:top-0 -ml-6 -bottom-4"></span>
</div>
<span class="md:block w-full truncate">You</span>
<h3 class="md:inline-block mt-3 md:mt-0 text-2xl text-gray-600 font-light leading-1.3">
Browser
</h3>
<span class="leading-1.3 text-2xl text-green-success">Working</span>
</div>
<div id="cf-cloudflare-status" class=" relative w-1/3 md:w-full py-15 md:p-0 md:py-8 md:text-left md:border-solid md:border-0 md:border-b md:border-gray-400 overflow-hidden float-left md:float-none text-center">
<div class="relative mb-10 md:m-0">
<a href="<https://www.cloudflare.com/5xx-error-landing?utm_source=errorcode_524&utm_campaign=lakefs.example.com>" target="_blank" rel="noopener noreferrer">
<span class="cf-icon-cloud block md:hidden h-20 bg-center bg-no-repeat"></span>
<span class="cf-icon-ok w-12 h-12 absolute left-1/2 md:left-auto md:right-0 md:top-0 -ml-6 -bottom-4"></span>
</a>
</div>
<span class="md:block w-full truncate">Denver</span>
<h3 class="md:inline-block mt-3 md:mt-0 text-2xl text-gray-600 font-light leading-1.3">
<a href="<https://www.cloudflare.com/5xx-error-landing?utm_source=errorcode_524&utm_campaign=lakefs.example.com>" target="_blank" rel="noopener noreferrer">
Cloudflare
</a>
</h3>
<span class="leading-1.3 text-2xl text-green-success">Working</span>
</div>
<div id="cf-host-status" class="cf-error-source relative w-1/3 md:w-full py-15 md:p-0 md:py-8 md:text-left md:border-solid md:border-0 md:border-b md:border-gray-400 overflow-hidden float-left md:float-none text-center">
<div class="relative mb-10 md:m-0">
<span class="cf-icon-server block md:hidden h-20 bg-center bg-no-repeat"></span>
<span class="cf-icon-error w-12 h-12 absolute left-1/2 md:left-auto md:right-0 md:top-0 -ml-6 -bottom-4"></span>
</div>
<span class="md:block w-full truncate"><http://lakefs.example.com|lakefs.example.com></span>
<h3 class="md:inline-block mt-3 md:mt-0 text-2xl text-gray-600 font-light leading-1.3">
Host
</h3>
<span class="leading-1.3 text-2xl text-red-error">Error</span>
</div>
</div>
</div>
</div>
<div class="w-240 lg:w-full mx-auto mb-8 lg:px-8">
<div class="clearfix">
<div class="w-1/2 md:w-full float-left pr-6 md:pb-10 md:pr-0 leading-relaxed">
<h2 class="text-3xl font-normal leading-1.3 mb-4">What happened?</h2>
<p>The origin web server timed out responding to this request.</p>
</div>
<div class="w-1/2 md:w-full float-left leading-relaxed">
<h2 class="text-3xl font-normal leading-1.3 mb-4">What can I do?</h2>
<h3 class="text-15 font-semibold mb-2">If you're a visitor of this website:</h3>
<p class="mb-6">Please try again in a few minutes.</p>
<h3 class="text-15 font-semibold mb-2">If you're the owner of this website:</h3>
<p><span>The connection to the origin web server was made, but the origin web server timed out before responding. The likely cause is an overloaded background task, database or application, stressing the resources on your web server. To resolve, please work with your hosting provider or web development team to free up resources for your database or overloaded application.</span> <a rel="noopener noreferrer" href="<https://support.cloudflare.com/hc/en-us/articles/200171926-Error-524>">Additional troubleshooting information here.</a></p>
</div>
</div>
</div>
<div class="cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300">
<p class="text-13">
<span class="cf-footer-item sm:block sm:mb-1">Cloudflare Ray ID: <strong class="font-semibold">79add1c3dcb0f790</strong></span>
<span class="cf-footer-separator sm:hidden">•</span>
<span id="cf-footer-item-ip" class="cf-footer-item hidden sm:block sm:mb-1">
Your IP:
<button type="button" id="cf-footer-ip-reveal" class="cf-footer-ip-reveal-btn">Click to reveal</button>
<span class="hidden" id="cf-footer-ip">2601:281:8000:277:28c6:4566:2540:fbd9</span>
<span class="cf-footer-separator sm:hidden">•</span>
</span>
<span class="cf-footer-item sm:block sm:mb-1"><span>Performance & security by</span> <a rel="noopener noreferrer" href="<https://www.cloudflare.com/5xx-error-landing?utm_source=errorcode_524&utm_campaign=lakefs.example.com>" id="brand_link" target="_blank">Cloudflare</a></span>
</p>
<script>(function(){function d(){var b=a.getElementById("cf-footer-item-ip"),c=a.getElementById("cf-footer-ip-reveal");b&&"classList"in b&&(b.classList.remove("hidden"),c.addEventListener("click",function(){c.classList.add("hidden");a.getElementById("cf-footer-ip").classList.remove("hidden")}))}var a=document;document.addEventListener&&a.addEventListener("DOMContentLoaded",d)})();</script>
</div><!-- /.error-footer -->
</div>
</div>
</body>
</html>j
Jonathan Rosenberg
02/22/2023, 4:28 PMthanks let me have a look
c
Conor Simmons
02/22/2023, 4:29 PMIt failed on
lakefs_client.objects.stage_objectj
Jonathan Rosenberg
02/22/2023, 4:44 PMOut of curiosity, I can see that the endpoint you set for lakeFS is ‘lakefs.example.com’, is that correct?
c
Conor Simmons
02/22/2023, 4:44 PMNo I edited the stack trace to change it. Do you need to know it?
j
Jonathan Rosenberg
02/22/2023, 4:45 PMNo, just making sure you didn’t use it
👍 1
m
Matija Teršek
02/22/2023, 4:45 PMWe are self hosting via docker image and nginx reverse proxy for https.
👍 1
j
Jonathan Rosenberg
02/23/2023, 1:33 PMHi @Conor Simmons and @Matija Teršek
Just one more question regarding the lakeFS endpoint: do you suffix it with
/api/v1<http://lakefs.example.com/api/v1|lakefs.example.com/api/v1>c
Conor Simmons
02/23/2023, 1:40 PMNo
j
Jonathan Rosenberg
02/23/2023, 1:40 PMCan you make sure it doesn’t end with a
/or alternatively, suffix it with
/api/v1c
Conor Simmons
02/23/2023, 1:42 PMIt definitely doesn't end with a
/j
Jonathan Rosenberg
02/23/2023, 1:43 PMwhat’s the current status?
Is it working currently?
c
Conor Simmons
02/23/2023, 1:44 PMCurrently it's working yeah. The error happened seemingly randomly in a loop of many stages
j
Jonathan Rosenberg
02/23/2023, 1:49 PMI’m trying to figure out how the first error that you posted is related to the stack trace that you shared.
Is it possible that these are two different errors?
c
Conor Simmons
02/23/2023, 1:50 PMI believe so. I think the original error was on server side ( CC @Matija Teršek)
j
Jonathan Rosenberg
02/23/2023, 1:53 PMthe first error is a sig V4 verification error that lakeFS S3 gateway generates, while the stack trace error is a timeout between Cloudflare and lakeFS.
Can you describe your setup? How is lakeFS deployed? What is calling it?
Also, do you collect the lakeFS metrics?