Skip to content

Do not delay exit waiting for telemetry - #56863

Open
nagilson wants to merge 1 commit into
dotnet:release/11.0.1xxfrom
nagilson:nagilson-otel-delay-fix
Open

nagilson wants to merge 1 commit into
dotnet:release/11.0.1xxfrom
nagilson:nagilson-otel-delay-fix

Conversation

@nagilson

Copy link
Copy Markdown
Member

Resolves #56815 (comment)

The initial fix was made in INTERNAL on dotnet/dotnet 11.0.100 for GA.

https://dev.azure.com/dnceng/internal/_git/dotnet-dotnet/pullrequest/67911?_a=files

Scenario Pre-patch Post-patch
--version, default NativeAOT path 2.11 s 0.21 s
No-op build, managed CLI + MSBuild logger 8.76 s 4.52 s
Build telemetry overhead versus opt-out 5.05 s 0.91 s

The tracerProvider often get blocked on pending HTTP requests from recent Azure Monitor Exporter changes. It is supposed to be a minimal shutdown delay. Pending requests often take 3-5 seconds to complete, which results in extreme exit delay.

The MS Build logger telemetry client calls ForceFlush, which is why the additional change is needed. The defaultTimeout remains as 5 seconds, because it informs network timeout for HTTP requests in options.Retry.NetworkTimeout - since telemetry POST completion time is often around 3-5 seconds, setting this lower would drop telemetry.

--

The DIFFERENCE between this PR and what went into GA:

  • Fixed a test only in dotnet/sdk
  • I went from 10 to 100 ms because 10ms may not give enough time to write blob storage on slower disks for short lived commands

When merging, prefer these changes.

Resolves dotnet#56815 (comment)

The initial fix was made in INTERNAL on dotnet/dotnet 11.0.100 for GA.

https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/authorize?client_id=499b84ac-1321-427f-aa17-267ca6975798&site_id=501454&response_mode=form_post&response_type=code+id_token&redirect_uri=https%3A%2F%2Fspsprodcus3.vssps.visualstudio.com%2F_signedin&nonce=6e955c3b-2d8e-4b63-a9e6-27ed4d3332c6&state=realm%3Ddev.azure.com%26reply_to%3Dhttps%253A%252F%252Fdev.azure.com%252Fdnceng%252Finternal%252F_git%252Fdotnet-dotnet%252Fpullrequest%252F67911%26ht%3D2%26protocol%3Dentra.web.3%26hid%3Db55de4ed-4b5a-4215-a8e4-0a0a5f71e7d8%26nonce%3D6e955c3b-2d8e-4b63-a9e6-27ed4d3332c6%26lltid%3D221510f4-2b26-40d5-988a-08b05ea44407&resource=499b84ac-1321-427f-aa17-267ca6975798&cid=6e955c3b-2d8e-4b63-a9e6-27ed4d3332c6&wsucxt=1&instance_aware=true

Scenario	Pre-patch	Post-patch
--version, default NativeAOT path	2.11 s	0.21 s
No-op build, managed CLI + MSBuild logger	8.76 s	4.52 s
Build telemetry overhead versus opt-out	5.05 s	0.91 s

The tracerProvider often get blocked on pending HTTP requests from recent Azure Monitor Exporter changes. It is supposed to be a minimal shutdown delay. Pending requests often take 3-5 seconds to complete, which results in extreme exit delay.

The MS Build logger telemetry client calls ForceFlush, which is why the additional change is needed. The defaultTimeout remains as 5 seconds, because it informs network timeout for HTTP requests in options.Retry.NetworkTimeout - since telemetry POST completion time is often around 3-5 seconds, setting this lower would drop telemetry.

--

The DIFFERENCE between this PR and what went into GA:
- Fixed a test only in dotnet/sdk
- I went from 10 to 100 ms because 10ms may not give enough time to write blob storage on slower disks for short lived commands

When merging, prefer these changes.
@nagilson
nagilson requested a review from a team as a code owner October 30, 2026 21:26
Copilot AI balanced review requested due to automatic review settings October 30, 2026 21:26
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 2 pipeline(s).
1 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.
@nagilson

Copy link
Copy Markdown
Member Author

Already approved in servicing

@nagilson

Copy link
Copy Markdown
Member Author

Once this is approved I will backport to 11.0.2xx or monitor flow to make sure it gets in.

@nagilson

Copy link
Copy Markdown
Member Author

Ah, ty 😉

@nagilson

Copy link
Copy Markdown
Member Author

/backport to release/11.0.2xx

@github-actions

Copy link
Copy Markdown
Contributor

Started backporting to release/11.0.2xx (link to workflow run)

@github-actions

Copy link
Copy Markdown
Contributor

@nagilson backporting to release/11.0.2xx was not run because the source pull request has not been merged. Please merge this pull request before requesting a backport.

@nagilson
nagilson enabled auto-merge October 30, 2026 21:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The local MSBuild flush timeout lacks regression coverage because existing tests do not exercise the non-CI, non-OTLP path and would not catch a reversion to Timeout.Infinite.

1 open issue
What changed in this PR

This PR addresses telemetry-related CLI exit delays reported in #56815 by bounding local trace persistence waits, including the MSBuild logger’s flush path.

Changes:

  • Applies a 100-ms local trace budget to successful shutdown and nonterminal flushes.
  • Adds exporter persistence tests and updates the shutdown timeout assertion.
  • Documents the bounded persistence behavior while retaining existing CI and failed-command budgets.
File Description
test/​Microsoft.DotNet.Cli.Telemetry.Tests/​AzureExporterLifecycleTests.cs Adds shutdown and force-flush persistence checks using a 100-ms budget.
test/​dotnet.Tests/​TelemetryTests/​TelemetryClientTests.cs Updates the expected successful local shutdown timeout.
src/​Cli/​dotnet/​Telemetry/​TelemetryClient.cs Bounds local trace shutdown and force-flush waits.
documentation/​project-docs/​telemetry.md Documents the local persistence budgets and MSBuild flush behavior.

Review effort: Balanced

Comment on lines +294 to +296
int timeout = s_isCIEnvironment || s_enableOtlpExporter
? s_shutdownTimeoutMs
: LocalTracePersistenceTimeoutMilliseconds;

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

3 participants