https://lakefs.io/ logo
Join Slack
Powered by
Hey, while uploading files, the process failed and...
# help
m
Hey, while uploading files, the process failed and we noticed the following errors:
Copy code
time="2023-02-16T07:39:00Z" level=warning msg="error verifying credentials for key" func=pkg/gateway.AuthenticationHandler.func1 file="build/pkg/gateway/middleware.go:55" authenticator=sigv4 error=SignatureDoesNotMatch key=MASKED
Is it possible to see more detailed error of why this happened? Any thoughts what could cause that? CC @Conor Simmons for visibility
i
Hi! We will take a look and get back to you. Ib the meantime, can you describe how you are uploading the files?
c
i
Thank you for the context. We’ll look into it and get back.
👍 1
a
@Matija Teršek @Conor Simmons Based on the error message you are getting: “error verifying credentials for key”, lakeFS credentials are wrong. Can you double check your lakeFS access and secret key?
m
The credentials are correct since the uploading process works for majority of the time.
staging*
a
Staging Python program makes multiple API calls. Do you know which API call (or line #) is failing and what API error response are you getting?
c
lakefs_client.objects.stage_object
a
@Conor Simmons Are you getting this error for certain types of files? Somebody reported similar error message when filename has unicode characters in it.
c
Not that I've noticed. The files are images (PNG) and JSON
m
Any updates @Amit Kesarwani?
j
Hi @Matija Teršek It looks like lakeFS is failing over the sigv4 header verification which is used by our lakeFS s3 gateway. I don’t see how this can happen from the provided code as it’s not using the S3 gateway. Could you share the piece of code that didn’t work (and the line at which the error happened) and the stack trace (or the client error if you don’t have it)?
c
@Jonathan Rosenberg here is the stack trace
Copy code
Traceback (most recent call last):
  File "/home/conor/Luxonis/envs/lakefs/bin/luxonis_ml", line 33, in <module>
    sys.exit(load_entry_point('luxonis-ml', 'console_scripts', 'luxonis_ml')())
  File "/home/conor/Luxonis/luxonis-ml/src/luxonis_ml/luxonis_ml.py", line 156, in main
    _dataset_push(args)
  File "/home/conor/Luxonis/luxonis-ml/src/luxonis_ml/luxonis_ml.py", line 99, in _dataset_push
    artifact.push(message=args.message)
  File "/home/conor/Luxonis/luxonis-ml/src/luxonis_ml/ops/artifacts.py", line 196, in push
    self.lakefs_client.objects.stage_object(self.name, self.dataset.branch, path, staged_object)
  File "/home/conor/Luxonis/envs/lakefs/lib/python3.9/site-packages/lakefs_client/api/objects_api.py", line 1409, in stage_object
    return self.stage_object_endpoint.call_with_http_info(**kwargs)
  File "/home/conor/Luxonis/envs/lakefs/lib/python3.9/site-packages/lakefs_client/api_client.py", line 835, in call_with_http_info
    return self.api_client.call_api(
  File "/home/conor/Luxonis/envs/lakefs/lib/python3.9/site-packages/lakefs_client/api_client.py", line 409, in call_api
    return self.__call_api(resource_path, method,
  File "/home/conor/Luxonis/envs/lakefs/lib/python3.9/site-packages/lakefs_client/api_client.py", line 203, in __call_api
    raise e
  File "/home/conor/Luxonis/envs/lakefs/lib/python3.9/site-packages/lakefs_client/api_client.py", line 196, in __call_api
    response_data = self.request(
  File "/home/conor/Luxonis/envs/lakefs/lib/python3.9/site-packages/lakefs_client/api_client.py", line 463, in request
    return self.rest_client.PUT(url,
  File "/home/conor/Luxonis/envs/lakefs/lib/python3.9/site-packages/lakefs_client/rest.py", line 277, in PUT
    return self.request("PUT", url,
  File "/home/conor/Luxonis/envs/lakefs/lib/python3.9/site-packages/lakefs_client/rest.py", line 224, in request
    raise ServiceException(http_resp=r)
lakefs_client.exceptions.ServiceException: (524)
Reason:
HTTP response headers: HTTPHeaderDict({'Date': 'Fri, 17 Feb 2023 10:30:30 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Set-Cookie': 'cf_use_ob=0; path=/; expires=Fri, 17-Feb-23 10:31:00 GMT', 'X-Frame-Options': 'SAMEORIGIN', 'Referrer-Policy': 'same-origin', 'Cache-Control': 'private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0', 'Expires': 'Thu, 01 Jan 1970 00:00:01 GMT', 'CF-RAY': '79add1c3dcb0f790-DEN', 'Server': 'cloudflare'})
HTTP response body: <!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>


<title><http://lakefs.example.com|lakefs.example.com> | 524: A timeout occurred</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/main.css" />


</head>
<body>
<div id="cf-wrapper">



    <div id="cf-error-details" class="p-0">
        <header class="mx-auto pt-10 lg:pt-6 lg:px-8 w-240 lg:w-full mb-8">
            <h1 class="inline-block sm:block sm:mb-2 font-light text-60 lg:text-4xl text-black-dark leading-tight mr-2">

              <span class="inline-block">A timeout occurred</span>
              <span class="code-label">Error code 524</span>
            </h1>
            <div>
               Visit <a href="<https://www.cloudflare.com/5xx-error-landing?utm_source=errorcode_524&utm_campaign=lakefs.example.com>" target="_blank" rel="noopener noreferrer"><http://cloudflare.com|cloudflare.com></a> for more information.
            </div>
            <div class="mt-3">2023-02-17 10:30:30 UTC</div>
        </header>

        <div class="my-8 bg-gradient-gray">
            <div class="w-240 lg:w-full mx-auto">
                <div class="clearfix md:px-8">

<div id="cf-browser-status" class=" relative w-1/3 md:w-full py-15 md:p-0 md:py-8 md:text-left md:border-solid md:border-0 md:border-b md:border-gray-400 overflow-hidden float-left md:float-none text-center">
  <div class="relative mb-10 md:m-0">

    <span class="cf-icon-browser block md:hidden h-20 bg-center bg-no-repeat"></span>
    <span class="cf-icon-ok w-12 h-12 absolute left-1/2 md:left-auto md:right-0 md:top-0 -ml-6 -bottom-4"></span>

  </div>
  <span class="md:block w-full truncate">You</span>
  <h3 class="md:inline-block mt-3 md:mt-0 text-2xl text-gray-600 font-light leading-1.3">

    Browser

  </h3>
  <span class="leading-1.3 text-2xl text-green-success">Working</span>
</div>

<div id="cf-cloudflare-status" class=" relative w-1/3 md:w-full py-15 md:p-0 md:py-8 md:text-left md:border-solid md:border-0 md:border-b md:border-gray-400 overflow-hidden float-left md:float-none text-center">
  <div class="relative mb-10 md:m-0">
    <a href="<https://www.cloudflare.com/5xx-error-landing?utm_source=errorcode_524&utm_campaign=lakefs.example.com>" target="_blank" rel="noopener noreferrer">
    <span class="cf-icon-cloud block md:hidden h-20 bg-center bg-no-repeat"></span>
    <span class="cf-icon-ok w-12 h-12 absolute left-1/2 md:left-auto md:right-0 md:top-0 -ml-6 -bottom-4"></span>
    </a>
  </div>
  <span class="md:block w-full truncate">Denver</span>
  <h3 class="md:inline-block mt-3 md:mt-0 text-2xl text-gray-600 font-light leading-1.3">
    <a href="<https://www.cloudflare.com/5xx-error-landing?utm_source=errorcode_524&utm_campaign=lakefs.example.com>" target="_blank" rel="noopener noreferrer">
    Cloudflare
    </a>
  </h3>
  <span class="leading-1.3 text-2xl text-green-success">Working</span>
</div>

<div id="cf-host-status" class="cf-error-source relative w-1/3 md:w-full py-15 md:p-0 md:py-8 md:text-left md:border-solid md:border-0 md:border-b md:border-gray-400 overflow-hidden float-left md:float-none text-center">
  <div class="relative mb-10 md:m-0">

    <span class="cf-icon-server block md:hidden h-20 bg-center bg-no-repeat"></span>
    <span class="cf-icon-error w-12 h-12 absolute left-1/2 md:left-auto md:right-0 md:top-0 -ml-6 -bottom-4"></span>

  </div>
  <span class="md:block w-full truncate"><http://lakefs.example.com|lakefs.example.com></span>
  <h3 class="md:inline-block mt-3 md:mt-0 text-2xl text-gray-600 font-light leading-1.3">

    Host

  </h3>
  <span class="leading-1.3 text-2xl text-red-error">Error</span>
</div>

                </div>

            </div>
        </div>

        <div class="w-240 lg:w-full mx-auto mb-8 lg:px-8">
            <div class="clearfix">
                <div class="w-1/2 md:w-full float-left pr-6 md:pb-10 md:pr-0 leading-relaxed">
                    <h2 class="text-3xl font-normal leading-1.3 mb-4">What happened?</h2>
                    <p>The origin web server timed out responding to this request.</p>
                </div>

                <div class="w-1/2 md:w-full float-left leading-relaxed">
                    <h2 class="text-3xl font-normal leading-1.3 mb-4">What can I do?</h2>
                          <h3 class="text-15 font-semibold mb-2">If you're a visitor of this website:</h3>
      <p class="mb-6">Please try again in a few minutes.</p>

      <h3 class="text-15 font-semibold mb-2">If you're the owner of this website:</h3>
      <p><span>The connection to the origin web server was made, but the origin web server timed out before responding. The likely cause is an overloaded background task, database or application, stressing the resources on your web server. To resolve, please work with your hosting provider or web development team to free up resources for your database or overloaded application.</span> <a rel="noopener noreferrer" href="<https://support.cloudflare.com/hc/en-us/articles/200171926-Error-524>">Additional troubleshooting information here.</a></p>
                </div>
            </div>

        </div>

        <div class="cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300">
  <p class="text-13">
    <span class="cf-footer-item sm:block sm:mb-1">Cloudflare Ray ID: <strong class="font-semibold">79add1c3dcb0f790</strong></span>
    <span class="cf-footer-separator sm:hidden">&bull;</span>
    <span id="cf-footer-item-ip" class="cf-footer-item hidden sm:block sm:mb-1">
      Your IP:
      <button type="button" id="cf-footer-ip-reveal" class="cf-footer-ip-reveal-btn">Click to reveal</button>
      <span class="hidden" id="cf-footer-ip">2601:281:8000:277:28c6:4566:2540:fbd9</span>
      <span class="cf-footer-separator sm:hidden">&bull;</span>
    </span>
    <span class="cf-footer-item sm:block sm:mb-1"><span>Performance & security by</span> <a rel="noopener noreferrer" href="<https://www.cloudflare.com/5xx-error-landing?utm_source=errorcode_524&utm_campaign=lakefs.example.com>" id="brand_link" target="_blank">Cloudflare</a></span>

  </p>
  <script>(function(){function d(){var b=a.getElementById("cf-footer-item-ip"),c=a.getElementById("cf-footer-ip-reveal");b&&"classList"in b&&(b.classList.remove("hidden"),c.addEventListener("click",function(){c.classList.add("hidden");a.getElementById("cf-footer-ip").classList.remove("hidden")}))}var a=document;document.addEventListener&&a.addEventListener("DOMContentLoaded",d)})();</script>
</div><!-- /.error-footer -->


    </div>
</div>
</body>
</html>
j
thanks let me have a look
c
It failed on 
lakefs_client.objects.stage_object
j
Out of curiosity, I can see that the endpoint you set for lakeFS is ‘lakefs.example.com’, is that correct?
c
No I edited the stack trace to change it. Do you need to know it?
j
No, just making sure you didn’t use it
👍 1
m
We are self hosting via docker image and nginx reverse proxy for https.
👍 1
j
Hi @Conor Simmons and @Matija Teršek Just one more question regarding the lakeFS endpoint: do you suffix it with 
/api/v1
? e.g. 
<http://lakefs.example.com/api/v1|lakefs.example.com/api/v1>
c
No
j
Can you make sure it doesn’t end with a 
/
?
or alternatively, suffix it with 
/api/v1
and try again?
c
It definitely doesn't end with a 
/
. We have yet to reproduce the error so far
j
what’s the current status? Is it working currently?
c
Currently it's working yeah. The error happened seemingly randomly in a loop of many stages
j
I’m trying to figure out how the first error that you posted is related to the stack trace that you shared. Is it possible that these are two different errors?
c
I believe so. I think the original error was on server side ( CC @Matija Teršek)
j
the first error is a sig V4 verification error that lakeFS S3 gateway generates, while the stack trace error is a timeout between Cloudflare and lakeFS. Can you describe your setup? How is lakeFS deployed? What is calling it? Also, do you collect the lakeFS metrics?
5 Views
Open in Slack
PreviousNext
Powered by