Azure Host Pool validation environments and agent versions

Hi @SebastianD-5208

Thank you for performing the additional testing and sharing the Event Viewer logs.

The fresh deployment is particularly insightful. Since a newly provisioned session host initially installs the latest production agent and then, within a few minutes, automatically upgrades itself to 1.0.15008.300 (validation), it suggests the update is being initiated by the Azure Virtual Desktop agent update service rather than by your Terraform deployment or DSC configuration.

At this point, I'd recommend checking a few additional items:

1. Confirm the Host Pool still reports Validation environment = No:

*az desktopvirtualization hostpool show *

*--resource-group <resource-group> *

*--name <hostpool-name> *

--query validationEnvironment

2. Verify that all session hosts exhibiting this behavior belong only to that host pool and aren't receiving configuration from another deployment process.
3. Review the RD Agent logs under:

• Applications and Services Logs > Microsoft > Windows > RemoteDesktopServices
• Microsoft-RDInfra-RDAgent (if present)

4. Compare the behavior with a host deployed into a different host pool (also configured with validationEnvironment = No) to determine whether this is isolated to a specific host pool or region.

Since you've now reproduced the behavior on a clean deployment, and the MSI installer is being invoked automatically after the production agent is installed, this no longer appears to be caused by Terraform configuration or the UseAgentDownloadEndpoint setting.

One question that would help narrow this down:

Which Azure region is the host pool deployed in? If multiple customers in the same region are observing the same automatic upgrade, it could indicate a regional rollout or an issue with the AVD agent update service.

Given the evidence you've collected, I believe this would be a good case for a Microsoft moderator to escalate to the Azure Virtual Desktop engineering team. The fact that a clean deployment consistently transitions from the latest production agent to a validation build despite validationEnvironment = No is not expected behavior and merits further investigation.

Please don't forget "Accept the Answer" if this information helped you. This will help us and others in the community as well.

Thanks for your reply. In regards to your questions:

1. Host Pool is still set on "validationEnvironment": false according to portal and az command.
2. I deployed two hosts together with the host pool and both showed the same update behaviour at the same time. The only other "deployment near" process running in this environment ist salt stack which did nothing according to the logs. (apart from that, none of our salt states comes close to anything I would suspect of triggering an update)
3. Unfortunately I cant attach the .evtx log from the RemoteDesktopServices. Here are some entries I found notable/interesting but I'm lacking the knowledge to further connect them to the root issue. In chronological order. (see below point 4)
4. I have just deployed the same setup (same terraform configuration in a slightly reduced version, no salt stacke here!) to an empty testing environment and the behaviour was the same. Hosts get deployed with the latest prod version of the agent and within one hour they update to 1.0.15008.300. Host pool has validation environment set to "No". We are talking about west europe here. Same behaviour (as Ive written some posts ago) was observed in host pool the recently got switched from "Yes" to "No" where we added some hosts. (and the already existsing hosts where on 1.0.15008.300)

Some warnings along those lines pop up:

unable to retrieve SessionHostConfigurationLastUpdateTime from registry: System.FormatException: String was not recognized as a valid DateTime.
   at System.DateTimeParse.Parse(String s, DateTimeFormatInfo dtfi, DateTimeStyles styles)
   at Microsoft.RDInfra.RDAgent.Service.RDAgentBrokerConfiguration.get_SessionHostConfigurationLastUpdateTime() in C:\__w\1\s\src\RDAgent\src\Service\AgentSettings.cs:line 3097

One Entry from the AgentBackgroundUpdater (with some entires removed):

Please note: This is a Windows 11 Enterprise multisession machine on version 10.0.26100 Build 26100. Dont know why this mesage states Windows 10.

GetUpdatedPackagesFromBrokerAsync Sending AgentCaps: ≤{"AgentVersion":"1.0.14669.400",
"OSDetails":{"OSVersion":"10.0.26100.8875",
"OSVersionFullDescription":"Windows 10 Enterprise multi-session"},
"SxSStackVersion":"rdp-sxs260519600",
"StackType":2,
"MonitoringVersion":"RDMonitoringAgentBootLoader_999.999.999.999",
"GenevaVersion":"46.24.3",
"WVDAgentManagerVersion":"",
"SessionHostConfigurationLastUpdateTime":"0001-01-01T00:00:00",
"AzureResourceId":"RESOURCEID",
"BootLoaderVersion":"1.0.8925.0",
"SessionHostTimeZone":"W. Europe Standard Time",
"FlightConfig":null,
"InRecoveryMode":false,
"RdrSxSVersion":"",
"SystemSettingsProxyVersion":"",
"SessionHostMonAgentVersion":"",
"Architecture":"X64"}≥

Still Source AgentBackgroundUpdater, as I said in chronological order

ProcessUpdatesAsync package info Version = 1.0.15008.300, Uri = ≤SOME mrsglobal URL≥, Flight info Microsoft.RDInfra.Shared.Common.FlightConfig

SetFlightContext set registry key to {"ConfigVersion":30555,"FlightSystemVersion":1}

OnBrokerResponseAsync: processing - ≤{"StatusCode":3,"StatusMessage":"IncorrectAgentVersion","Details":{"AgentExpectedVersion":"1.0.15008.300","AgentDownloadLocation":"SOME mrsglobal URL≥

assembly version is 1.0.14669.400

CheckAndApplyBypass EnableAgentVersionBypass : AgentByPassVersion is 1.0.15008.300

This is reported as ERROR: (rest is information)

CheckAndApplyBypass returning false on exception

From then in the logs read like the expected update messages for an agent update so I wont continue posting them.

I hope someone is able to find my mistake or we can escalate this. Thanks in advance.

Hello Sebastian,

Thanks for the detailed logs. This is now much clearer.

The important entries are:

StatusCode: 3

StatusMessage: IncorrectAgentVersion

AgentExpectedVersion: 1.0.15008.300

followed by:

ProcessUpdatesAsync package info Version = 1.0.15008.300

That indicates the AVD broker/background updater is explicitly instructing the session host to move from 1.0.14669.400 to 1.0.15008.300. This is no longer pointing to Terraform, DSC, Salt, or the initial installation package.

Microsoft’s agent documentation does note that multiple agent versions can exist across environments during rollout, and that not every validation build reaches production. However, your clean reproduction is different: a host pool with validationEnvironment=false installs the production build first, then the broker explicitly reports a validation build as the expected version.

The SessionHostConfigurationLastUpdateTime parsing warning is interesting, but I wouldn’t modify that registry value manually. The stronger evidence is the broker response specifying 1.0.15008.300.

At this point I think you have enough evidence for escalation:

• clean host pool created with validationEnvironment=false
• clean session hosts
• reproduced without Salt
• initial production agent 1.0.14669.400
• automatic update shortly afterward
• AgentBackgroundUpdater receives IncorrectAgentVersion
• broker explicitly expects 1.0.15008.300
• reproduced in West Europe

Microsoft also documents that once installed, the AVD Agent is updated automatically, so blocking Windows Installer or trying to pin the MSI manually would not be a supported workaround.

=========================================================================

Microsoft Moderator Vinodh247-1375: Could this be escalated to the Azure Virtual Desktop engineering team? The broker appears to be assigning validation agent 1.0.15008.300 to newly deployed session hosts even though the host pool reports validationEnvironment=false.

=========================================================================

I would preserve the AgentBackgroundUpdater logs, exact UTC timestamps, host pool/resource IDs, and one affected session host for Microsoft to correlate against the broker-side rollout configuration.

@SebastianD-5208 Please don't forget "Accept the Answer" if this information helped you. This will help us and others in the community as well.

Hi Allan, big thanks for your help so far! Is there anything I can do to get the attention of MS Mods here?