AZD deploy getting stuck on (Uploading deployment package) as of 2024/02/24

[OwenCross289]

• Make sure you've installed the latest version using instructions in the wiki

Output from azd version
Run azd version and copy and paste the output here:
azd version 1.12.0 (commit dc37b93)

Describe the bug
Deploying multiple services with AZD up to azure appservice the second service (add likely 3rd) gets stuck on

Deploying service ServiceName
Deploying service ServiceName (Uploading deployment package)
Deploying service ServiceName (Running build process)

To Reproduce
use an azure.yaml like this

name: deployment
metadata:
    template: azd-init@1.5.1
hooks:
  predeploy:
    shell: pwsh
    run: ./infra/scripts/run_pre_deployment_scripts.ps1
    interactive: true
services:
    Service.One:
        project: Src/Service.One/Service.One.csproj
        host: appservice
        language: csharp
    Service.Two:
        project: Src/Service.Two/Service.Two.csproj
        host: appservice
        language: csharp
    Service.Three:
        project: Src/Service.Three/Service.Three.csproj
        host: appservice
        language: csharp

Expected behavior
A clear and concise description of what you expected to happen.

Environment
Information on your environment:
* Language name and version: c#, sdk 9.0.200, OS Ubuntu 22.04
* IDE and version : No IDE, using CLI via Azure Devops Pipelines

Additional context
Add any other context about the problem here.

When splitting out the deployment from azd up to

azd provision
azd deploy Service.One
azd deploy Service.Two
azd deploy Service.Three

deployment succeeds.

Secondly this appears to be an azure api issue? as when I apply --debug the second service gets stuck in this loop

   GET https://management.azure.com/subscriptions/id/resourceGroups/rg-name/providers/Microsoft.Web/sites/app-service-two/deploymentStatus/id?api-version=2023-01-01&c=REDACTED&h=REDACTED&s=REDACTED&t=REDACTED
   Authorization: REDACTED
   User-Agent: azsdk-go-armappservice/v2.3.0 (go1.23.0 X:loopvar; linux),azdev/1.12.0 (Go go1.23.0 X:loopvar; linux/amd64)
   X-Ms-Correlation-Request-Id: c991d2fc466a494e01288331d5bc85b1

2025/02/25 13:00:58 main.go:50: Response: ==> REQUEST/RESPONSE (Try=1/338.11567ms, OpTime=338.139847ms) -- RESPONSE RECEIVED
   GET https://management.azure.com/subscriptions/id/resourceGroups/rg-name/providers/Microsoft.Web/sites/app-service-two/deploymentStatus/id?api-version=2023-01-01&c=REDACTED&h=REDACTED&s=REDACTED&t=REDACTED
   Authorization: REDACTED
   User-Agent: azsdk-go-armappservice/v2.3.0 (go1.23.0 X:loopvar; linux),azdev/1.12.0 (Go go1.23.0 X:loopvar; linux/amd64)
   X-Ms-Correlation-Request-Id: c991d2fc466a494e01288331d5bc85b1
   --------------------------------------------------------------------------------
   RESPONSE Status: 500 Internal Server Error
   Cache-Control: no-cache
   Content-Length: 199
   Content-Type: application/json; charset=utf-8
   Date: Tue, 25 Feb 2025 13:00:57 GMT
   Etag: "1DB8782DE251A35"
   Expires: -1
   Pragma: no-cache
   Strict-Transport-Security: REDACTED
   X-Aspnet-Version: REDACTED
   X-Cache: REDACTED
   X-Content-Type-Options: REDACTED
   X-Ms-Correlation-Request-Id: c991d2fc466a494e01288331d5bc85b1
   X-Ms-Failure-Cause: REDACTED
   X-Ms-Ratelimit-Remaining-Subscription-Global-Reads: REDACTED
   X-Ms-Ratelimit-Remaining-Subscription-Reads: REDACTED
   X-Ms-Request-Id: e9c64e61-9bde-4dc6-9ba0-1a6006ea6a00
   X-Ms-Routing-Request-Id: REDACTED
   X-Msedge-Ref: REDACTED
   X-Powered-By: REDACTED

2025/02/25 13:00:58 main.go:50: Retry: response 500
2025/02/25 13:00:58 main.go:50: Retry: End Try #2, Delay=2.109470897s
2025/02/25 13:01:00 main.go:50: Retry: =====> Try=3 for GET https://management.azure.com/subscriptions/id/resourceGroups/rg-name/providers/Microsoft.Web/sites/app-service-two/deploymentStatus/id?api-version=2023-01-01&c=REDACTED&h=REDACTED&s=REDACTED&t=REDACTED
2025/02/25 13:01:00 main.go:50: Request: ==> OUTGOING REQUEST (Try=1)
   GET https://management.azure.com/subscriptions/id/resourceGroups/rg-name/providers/Microsoft.Web/sites/app-service-two/deploymentStatus/id?api-version=2023-01-01&c=REDACTED&h=REDACTED&s=REDACTED&t=REDACTED
   Authorization: REDACTED
   User-Agent: azsdk-go-armappservice/v2.3.0 (go1.23.0 X:loopvar; linux),azdev/1.12.0 (Go go1.23.0 X:loopvar; linux/amd64)
   X-Ms-Correlation-Request-Id: c991d2fc466a494e01288331d5bc85b1

2025/02/25 13:01:00 main.go:50: Response: ==> REQUEST/RESPONSE (Try=1/551.66293ms, OpTime=551.687081ms) -- RESPONSE RECEIVED
   GET https://management.azure.com/subscriptions/id/resourceGroups/rg-name/providers/Microsoft.Web/sites/app-service-two/deploymentStatus/id?api-version=2023-01-01&c=REDACTED&h=REDACTED&s=REDACTED&t=REDACTED
   Authorization: REDACTED
   User-Agent: azsdk-go-armappservice/v2.3.0 (go1.23.0 X:loopvar; linux),azdev/1.12.0 (Go go1.23.0 X:loopvar; linux/amd64)
   X-Ms-Correlation-Request-Id: c991d2fc466a494e01288331d5bc85b1
   --------------------------------------------------------------------------------
   RESPONSE Status: 500 Internal Server Error
   Cache-Control: no-cache
   Content-Length: 199
   Content-Type: application/json; charset=utf-8
   Date: Tue, 25 Feb 2025 13:00:59 GMT
   Etag: "1DB8782DE251A35"
   Expires: -1
   Pragma: no-cache
   Strict-Transport-Security: REDACTED
   X-Aspnet-Version: REDACTED
   X-Cache: REDACTED
   X-Content-Type-Options: REDACTED
   X-Ms-Correlation-Request-Id: c991d2fc466a494e01288331d5bc85b1
   X-Ms-Failure-Cause: REDACTED
   X-Ms-Ratelimit-Remaining-Subscription-Global-Reads: REDACTED
   X-Ms-Ratelimit-Remaining-Subscription-Reads: REDACTED
   X-Ms-Request-Id: e82d179a-081a-404d-aa2f-157b2b882d86
   X-Ms-Routing-Request-Id: REDACTED
   X-Msedge-Ref: REDACTED
   X-Powered-By: REDACTED

2025/02/25 13:01:00 main.go:50: Retry: response 500
2025/02/25 13:01:00 main.go:50: Retry: End Try #3, Delay=4.725880328s

[UnMorrer]

I am running into a similar issue when deploying a streamlit application to Azure App Service.
After a while, the responses are status code 500 (instead of 202), BUT this seems to only affect the streamlit deployment (and not other Apps hosted on Azure App Service) for now.

[weikanglim]

Secondly this appears to be an azure api issue

Your observations does seem to be conclusive here, App Service isn't behaving the way we'd expect it to. The general troubleshooting strategy I'd recommend is to open up the App Service resource in Azure and navigate to deployment logs. There may be more detailed logs there.

If more help is needed at any point, an Azure support request may also be useful avenue.

[GABRIELNGBTUC]

We recently started trying to use azd to deploy to app service inside our pipelines and have been seeing this issue intermittently (azd provision being stuck for 5-30mins before succeeding).

We haven't tested running --debug yet but we believe the issue comes from azd because we have the same pipeline with the AzureFunction/AzureWebApp tasks and those consistently take ~1min to deploy the app to the app service resource. Only when using azd deploy do we see issues with an extremely long deployment time

[Menghua1]

@rajasekaran2003 In the latest round of manual testing of azd, there is a same issue.
Running the azd deploy command to deploy the web service fails, and only running azd deploy web alone will succeed. The specific information is as follows:

Environment:

• Template: todo-csharp-cosmos-sql, todo-java-mongo, todo-csharp-sql, todo-nodejs-mongo-terraform, todo-python-mongo-terraform, todo-java-postgresql-terraform.
• Azd version: azd version 1.13.0-pr.4623382 (commit d7dce27).
• OS: Mac, Windows, Codespaces.
• Branch: Staging.

[rajeshkamal5050]

@JeffreyCA can you help triage this?

[JeffreyCA]

I'm able to reproduce this issue on both the latest release and an older version of azd (1.11.1 from Jan 2025) using the todo-csharp-cosmos-sql template.

It looks like after deploying the app through zip deploy, polling the deploymentStatus endpoint is returning 500 InternalServerError, which suggests maybe a service-side issue. However, in the Azure portal (deployment center), the deployments appears to be successful:

Will start a thread internally to follow up with the App Service team.

[cecheta]

I've started experiencing this myself, and also noticed the 500 error from the deployment status endpoint

[vhvb1989]

@JeffreyCA , can we make azd to skip polling status from deploymentStatus after getting the 500 error?
I wonder if we are running PollUntilDone() which make us block the program and prevent us from taking any action after the 500.

So, maybe we could try calling Pool() and looking to the response. If there is a 500, we can print a warning to user like azd couldn't track deployment status. You can use Azure Portal for deployment status
Then we complete the operation