I have a weird message error when trying to do a merge using

Question

I have a weird message error when trying to do a merge using `lakefs client` python sdk ``` File path lakefs helper py line 102 in merge return self lakefs refs merge into branch File path venv lib64 python3 10 site packages lakefs client api refs api py line 869 in merge into branch return self merge into branch endpoint call with http info kwargs File path venv lib64 python3 10 site packages lakefs client api client py line 835 in call with http info return self api client call api File path venv lib64 python3 10 site packages lakefs client api client py line 409 in call api return self call api resource path method File path venv lib64 python3 10 site packages lakefs client api client py line 203 in call api raise e File path venv lib64 python3 10 site packages lakefs client api client py line 196 in call api response data = self request File path venv lib64 python3 10 site packages lakefs client api client py line 455 in request return <http self rest client POST|self rest client POST> url File path venv lib64 python3 10 site packages lakefs client rest py line 267 in POST return self request POST url File path venv lib64 python3 10 site packages lakefs client rest py line 224 in request raise ServiceException http resp=r lakefs client exceptions ServiceException 503 Reason Service Unavailable HTTP response headers HTTPHeaderDict content length 118 content type text plain date Thu 24 Aug 2023 23 47 57 GMT HTTP response body upstream connect error or disconnect reset before headers retried and the latest reset reason connection termination``` My helper function that do the merge is ``` def merge self source branch dest branch metadata= message=None try Check if there is a need to merge diffs = self diff left ref=dest branch right ref=source branch if len diffs == 0 print f No changes detected Skipping merge source branch to dest branch return if message merge = models Merge message=message metadata=metadata else merge = models Merge metadata=metadata return self lakefs refs merge into branch repository=self repo name source ref=source branch destination branch=dest branch merge=merge except Exception as e traceback print exc print f ERROR Failed to merge source branch to dest branch raise e``` With self lakefs as ``` configuration = lakefs client Configuration configuration username = conf access key id configuration password = conf secret access key configuration host = conf endpoint self lakefs = LakeFSClient configuration ``` But then if I do the merge of that 2 exact same branch via the UI it succeed Our self deployed server running 107 0 And the lakefs client is 0 107 0

Lynn Rozen · Answer

Hi < HT> I ll try to reproduce it and update

Ariel Shaqed (Scolnicov) · Answer

Hi < HT> I m afraid we may need more info in order to help here These lines ```Reason Service Unavailable HTTP response headers HTTPHeaderDict content length 118 content type text plain date Thu 24 Aug 2023 23 47 57 GMT HTTP response body upstream connect error or disconnect reset before headers retried and the latest reset reason connection termination``` indicate either an error on a gateway could be a proxy reverse proxy load balancer K8s ingress controller or possibly in lakeFS itself But they hide the underlying cause Do you have logs from the lakeFS server itself at the time of failure If so I d be grateful if you could scrub them of sensitive data and send them Thanks

HT · Answer

So I manage to reproduce with a freshly Docker deployed locally The python code ``` from datetime import datetime import lakefs client from lakefs client import models from lakefs client client import LakeFSClient from livsdk smartfs import load rclone profile conf = load rclone profile locallakefs branch name = f test datetime now strftime Y m d H M S repo= test repo lakeFS credentials and endpoint configuration = lakefs client Configuration configuration username = conf access key id configuration password = conf secret access key configuration host = conf endpoint client = LakeFSClient configuration branch name = f test merge datetime now strftime Y m d H M S commit origin=client branches api create branch repository=repo branch creation=models BranchCreation name=branch name source= main with open dev shm test csv rb as f client objects api upload object repository=repo branch=branch name path= file csv content=f client commits api commit repository=repo branch=branch name commit creation=models CommitCreation message= Added a CSV file metadata= using python api Create a second branch branch second = branch name + bis client branches api create branch repository=repo branch creation=models BranchCreation name=branch second source=commit origin with open dev shm new csv rb as f client objects api upload object repository=repo branch=branch second path= new csv content=f client commits api commit repository=repo branch=branch second commit creation=models CommitCreation message= Added new file metadata= using python api merge merge = models Merge metadata= client refs merge into branch repository=repo source ref=branch name destination branch=branch second merge=merge ``` That crash with ```Traceback most recent call last File data hieu opt python venv fastai lib64 python3 10 site packages urllib3 connectionpool py line 790 in urlopen response = self make request File data hieu opt python venv fastai lib64 python3 10 site packages urllib3 connectionpool py line 536 in make request response = conn getresponse File data hieu opt python venv fastai lib64 python3 10 site packages urllib3 connection py line 461 in getresponse httplib response = super getresponse File usr lib64 python3 10 http client py line 1374 in getresponse response begin File usr lib64 python3 10 http client py line 318 in begin version status reason = self read status File usr lib64 python3 10 http client py line 287 in read status raise RemoteDisconnected Remote end closed connection without http client RemoteDisconnected Remote end closed connection without response``` And the lakefs docker log ```lakeFS 0 107 1 Up and running ^C to shutdown lakeFS running in quickstart mode Login at Access Key ID AKIAIOSFOLQUICKSTART Secret Access Key wJalrXUtnFEMI K7MDENG bPxRfiCYEXAMPLEKEY For more information on how to use lakeFS check out the docs at For support or any other question gt lt join our Slack channel Version 0 107 1 2023 08 28 04 17 31 http panic serving 172 17 0 1 52354 assignment to entry in nil map goroutine 1283 running net http conn serve func1 usr local go src net http server go 1854 +0xbf panic 0x1d836e0 0x57956d0 usr local go src runtime panic go 890 +0x263 build pkg graveler graveler go 2557 +0xa3e 0xc00614e240 0x57b0d80 0xc00c97a3f0 0xc000012438 0xc00e11104a 0x1e 0xc00e9843c0 build pkg graveler ref manager go 403 +0x8c build pkg graveler graveler go 2028 +0x79 go pkg mod github com cenkalti backoff v4 v4 2 1 retry go 18 +0x1b 0xc0009ae680 0x57a5758 0xc00614f440 0x0 0x0 0x0 go pkg mod github com cenkalti backoff v4 v4 2 1 retry go 88 +0x152 0x57a5758 0xc00614f440 0x4232eb 0x0 0x0 go pkg mod github com cenkalti backoff v4 v4 2 1 retry go 61 +0x65 go pkg mod github com cenkalti backoff v4 v4 2 1 retry go 49 go pkg mod github com cenkalti backoff v4 v4 2 1 retry go 38 0xc0001321c0 0x57b0d80 0xc00c97a3f0 0xc000012438 0xc00e11104a 0x1e 0xc00e9843c0 0x210a9cd 0x5 build pkg graveler graveler go 2025 +0x22a 0x57b0d80 0xc00c97a3f0 0xc000012438 0xc00e11104a 0x1e 0xc00e111029 0x1a 0xc00ef921d0 0xa build pkg graveler graveler go 2505 +0x425 0x57b0d80 0xc00001dfb0 0xc00e11101a 0x9 0xc00e11104a 0x1e 0xc00e111029 0x1a 0xc00ef921d0 build pkg catalog catalog go 1721 +0x4e9 0x57adc00 0xc00e10c930 0xc00ca3e900 0x0 0xc00f19c1c0 0x0 0xc00e11101a 0x9 0xc00e111029 build pkg api controller go 3905 +0x303 0x57adc00 0xc00e10c930 0x1c8b100 build pkg api lakefs gen go 24486 +0xbb 0x57adc00 0xc00e10c930 0xc00ca3e800 build pkg api lakefs gen go 24493 +0x7bc net http HandlerFunc ServeHTTP 0xc00db14bd0 0x57adc00 0xc00e10c930 0x7f995e8ce108 usr local go src net http server go 2122 +0x2f 0x7f9937331b98 0xc00001dda0 0x57adc00 build pkg api metrics middleware go 27 +0xfc net http HandlerFunc ServeHTTP 0x57adc30 0x7f9937331b98 0xc00001dda0 0x390000c0009af301 usr local go src net http server go 2122 +0x2f 0xc0008197a0 0xc00ca3e800 go pkg mod github com prometheus client golang v1 16 0 prometheus promhttp instrument server go 147 +0xc5 net http HandlerFunc ServeHTTP 0x57b0d80 0x57adc30 0xc0008197a0 0x5792340 usr local go src net http server go 2122 +0x2f 0x57adc30 0xc0008197a0 0xc00ca3e700 build pkg api auth middleware go 108 +0x4b0 net http HandlerFunc ServeHTTP 0x1f4cb00 0x57adc30 0xc0008197a0 0xc usr local go src net http server go 2122 +0x2f 0x57af9a0 0xc00ea9e000 0xffffffffffffff01 build pkg httputil logging go 87 +0x645 net http HandlerFunc ServeHTTP 0xc00ca3e300 0x57af9a0 0xc00ea9e000 0xc00cbf3090 usr local go src net http server go 2122 +0x2f 0x57af9a0 0xc00ea9e000 0xc00ca3e100 build pkg api serve go 170 +0x2e3 net http HandlerFunc ServeHTTP 0xc00dafd3e0 0x57af9a0 0xc00ea9e000 0xc0005ae2e8 usr local go src net http server go 2122 +0x2f 0x57af9a0 0xc00ea9e000 0xc00e111005 go pkg mod github com go chi chi v5 v5 0 0 chain go 31 +0x2c 0x57af9a0 0xc00ea9e000 0xc00ca3e100 go pkg mod github com go chi chi v5 v5 0 0 mux go 436 +0x1f9 net http HandlerFunc ServeHTTP 0x57b0cd8 0x57af9a0 0xc00ea9e000 0x66db4e0 usr local go src net http server go 2122 +0x2f 0x57af9a0 0xc00ea9e000 0xc00ca3e000 go pkg mod github com go chi chi v5 v5 0 0 mux go 87 +0x32a 0xc00ea9e000 0xc0009afad0 build cmd lakefs cmd run go 342 +0x102 net http HandlerFunc ServeHTTP 0x0 0x57af9a0 0xc00ea9e000 0x8663d4 usr local go src net http server go 2122 +0x2f net http serverHandler ServeHTTP 0x57a8750 0x57af9a0 0xc00ea9e000 0xc00ca3e000 usr local go src net http server go 2936 +0x316 net http conn serve 0xc00eea0480 0x57b0d80 0xc00e386ea0 usr local go src net http server go 1995 +0x612 created by net http Server Serve usr local go src net http server go 3089 +0x5ed```

HT · Answer

I went ahead and log an issue <https github com treeverse lakeFS issues 6494>

Ariel Shaqed (Scolnicov) · Answer

Thanks That looks like a nasty one

Ariel Shaqed (Scolnicov) · Answer

Just closing the loop thanks for the great bug report < Barak Amar> released the fix in 0 108 0

HT · Answer

Thanks for fixing the issue