Checkout failure: "Git fetch failed with exit code 128"

[carlossanlop]

Build

https://dev.azure.com/dnceng/public/_build/results?buildId=1706886&view=results

Build leg reported

Libraries Test Run release coreclr Linux_musl x64 Release

Pull Request

Rolling build for runtime

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

• Open the failing build above and investigate
• Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

1. Add the label "Known Build Error"
2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation

{
   "ErrorMessage" : "Git fetch failed with exit code: 128",
   "BuildRetry": true
}

Additional information about the issue reported

The Checkout dotnet/runtime@main to s step is the one that failed. Here is the end of the log output:

remote: Compressing objects:  29% (30070/103687)        
remote: Compressing objects:  30% (31107/103687)        
remote: Compressing objects:  31% (32143/103687)        
remote: Compressing objects:  31% (32322/103687)        
remote: Compressing objects:  32% (33180/103687)        
remote: Compressing objects:  33% (34217/103687)        
remote: Compressing objects:  33% (35247/103687)        
remote: Compressing objects:  34% (35254/103687)        
remote: Compressing objects:  35% (36291/103687)        
remote: Compressing objects:  36% (37328/103687)        
remote: Compressing objects:  37% (38365/103687)        
remote: Compressing objects:  37% (38926/103687)        
remote: Compressing objects:  38% (39402/103687)        
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: protocol error: bad pack header
##[error]Git fetch failed with exit code: 128
Finishing: Checkout dotnet/runtime@main to s

Report

Build	Definition	Step Name	Console log	Pull Request
1265061	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/65212/merge to s	Log	dotnet/aspnetcore#65212
1265135	dotnet/sdk	Checkout dotnet/sdk@refs/pull/52583/merge to s	Log	dotnet/sdk#52583
1264649	dotnet/sdk	Checkout dotnet/sdk@refs/pull/52658/merge to s	Log	dotnet/sdk#52658
1264650	dotnet/sdk	Checkout dotnet/sdk@refs/pull/52659/merge to s	Log	dotnet/sdk#52659
1264394	dotnet/runtime	Checkout dotnet/runtime@refs/pull/123539/merge to s	Log	dotnet/runtime#123539
1264053	dotnet/runtime	Checkout dotnet/runtime@refs/pull/123524/merge to s	Log	dotnet/runtime#123524
1263818	dotnet/sdk	Checkout dotnet/sdk@refs/pull/52362/merge to s	Log	dotnet/sdk#52362
1263656	dotnet/installer	Checkout dotnet/installer@refs/pull/20785/merge to s	Log	dotnet/installer#20785
1263697	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/65197/merge to s	Log	dotnet/aspnetcore#65197
2885619	dotnet-installer	Clone dotnet/installer	Log	#57102
1261802	dotnet/aspnetcore	Checkout dotnet/aspnetcore@release/2.3 to s	Log
1261239	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/5840/merge to s	Log	dotnet/arcade-services#5840
1260018	dotnet/sdk	Checkout dotnet/sdk@refs/pull/52487/merge to s	Log	dotnet/sdk#52487
1259798	dotnet/runtime	Checkout dotnet/runtime@refs/pull/123326/merge to s	Log	dotnet/runtime#123326
1259464	dotnet/sdk	Checkout dotnet/sdk@refs/pull/52571/merge to s	Log	dotnet/sdk#52571
1258443	dotnet/runtime	Checkout dotnet/runtime@refs/pull/123354/merge to s	Log	dotnet/runtime#123354
2882121	dotnet-sdk	Clone dotnet/sdk	Log	#56830
1256481	maestro-auth-test/helix-queue-insights-test	Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/1110/merge to s	Log	maestro-auth-test/helix-queue-insights-test#1110
1255851	dotnet/runtime	Checkout dotnet/runtime@refs/pull/123258/merge to s	Log	dotnet/runtime#123258
2880420	dotnet-sdk	Clone dotnet/sdk	Log	#56765
1255123	maestro-auth-test/helix-queue-insights-test	Checkout maestro-auth-test/helix-queue-insights-test@refs/pull/1106/merge to s	Log	maestro-auth-test/helix-queue-insights-test#1106
1255003	dotnet/runtime	Checkout dotnet/runtime@refs/pull/123229/merge to s	Log	dotnet/runtime#123229
1255067	dotnet/runtime	Checkout dotnet/runtime@refs/pull/123233/merge to s	Log	dotnet/runtime#123233
1253359	dotnet/runtime	Checkout dotnet/runtime@refs/pull/123182/merge to s	Log	dotnet/runtime#123182
1252144	dotnet/runtime	Checkout dotnet/runtime@refs/pull/123156/merge to s	Log	dotnet/runtime#123156
1251182	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/5773/merge to s	Log	dotnet/arcade-services#5773
1250943	dotnet/runtime	Checkout dotnet/runtime@refs/pull/123115/merge to s	Log	dotnet/runtime#123115
1250766	dotnet/runtime	Checkout dotnet/runtime@refs/pull/123108/merge to s	Log	dotnet/runtime#123108
1250034	dotnet/runtime	Checkout dotnet/runtime@refs/pull/122990/merge to s	Log	dotnet/runtime#122990
1248550	dotnet/sdk	Checkout dotnet/sdk@refs/pull/52031/merge to s	Log	dotnet/sdk#52031
1248206	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/5742/merge to s	Log	dotnet/arcade-services#5742
1246150	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/64969/merge to s	Log	dotnet/aspnetcore#64969
2873156	dotnet-dotnet	Checkout dotnet-dotnet@add-cfs to s
1245021	dotnet/sdk	Checkout dotnet/sdk@refs/pull/52335/merge to s	Log	dotnet/sdk#52335
2873233	dotnet-dotnet	Checkout dotnet-dotnet@add-cfs to s
1243388	dotnet/sdk	Checkout dotnet/sdk@refs/pull/51158/merge to s	Log	dotnet/sdk#51158
1241430	dotnet/maui	Checkout dotnet/maui@release/10.0.1xx-sr2 to s	Log
1241219	dotnet/maui	Checkout dotnet/maui@refs/pull/33335/merge to s	Log	dotnet/maui#33335
1241068	dotnet/maui	Checkout dotnet/maui@main to s	Log
1241029	dotnet/maui	Checkout dotnet/maui@refs/pull/33268/merge to s	Log	dotnet/maui#33268
1240952	dotnet/maui	Checkout dotnet/maui@refs/pull/33177/merge to s	Log	dotnet/maui#33177
1240781	dotnet/maui	Checkout dotnet/maui@refs/pull/32701/merge to s	Log	dotnet/maui#32701
1240787	dotnet/maui	Checkout dotnet/maui@refs/pull/31839/merge to s	Log	dotnet/maui#31839

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
6	17	43

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323960
Error message validated: Git fetch failed with exit code: 128
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:15:14 PM UTC

[ChadNedzlek]

So this is... interesting. I agree this seems like essentially a random network failure that is not the fault of the code. Also... it doesn't seem like there is anything to be done other than retry the build.

@ulisesh and @AlitzelMendez. This seems like another instance of a "known issue" that would really benefit from the ability to get retried automatically. I think Carlos did the right thing by opening the known issue, but his PR (and any future PR that hits this) is still going to be blocked, since the build doesn't run at all in this case, because the checkout failed. And it would be great to help folks that hit this.

[ulisesh]

I created the issue to track how often it was happening and measure the impact of the problem. It might be hard to get traction on networking problems but having the number of impacted builds has great value in my opinion.

In the last 24 hours, we had 37 builds failed with the same error. cc @markwilkie who might find this data interesting.

I think automatic retries for known issues will be the next feature after we are done with analyzing builds retroactively, we can discuss this if people think we should change priorities.

[markwilkie]

So 37 builds were caught with the known issues feature @ulisesh ? Could I get the query please? (super exciting)

[ulisesh]

I saw 37 hits in the office but now I only see 33 🤔. There are 33 failed jobs in 2 different builds. Sorry if I misused the terms but I still think it is great that we can track these build breaks 😄

This is the query:

KnownIssues
| where IssueId == 9009 and IssueRepository == "dotnet/arcade"

[ViktorHofer]

This happened 36 times in the last 24 hours. Should we add a retry (assuming that's possible)?