Share via


Invoke-AzureRMVMRunCommand takes a LONG TIME to run (20-30 minutes) whereas the script when run interactively in a VM takes 10 seconds

Question

Tuesday, April 2, 2019 5:11 AM

This command takes 20-30 minutes to run:

Invoke-AzureRMVMRunCommand -ResourceGroupName <RESOURGEGROUPNAME> -CommandId 'RunPowerShellScript' -scriptpath '2019-03-21-STOP_Services.ps1' -asJob -vmname <VMNAME>

############ STOP_Services.ps1 #############

get-service *veeam* | stop-service 
get-service *veeam* | set-service -startuptype disabled

get-service *vmic* | stop-service 
get-service *vmic* | set-service -startuptype disabled

######################################

If I run STOP_Services.ps1 after I RDP into a AZURE VM, it takes approx 5 seconds to run.

What can I do to make Invoke-AzureRMVMRunCommand execute faster ?

All replies (8)

Tuesday, April 2, 2019 5:22 AM

Hi, 

Can run the command with -debug and -verbose and share me the results?

Regards, 

Msrini


Tuesday, April 2, 2019 3:31 PM

PS C:\scripts> Invoke-AzureRMVMRunCommand -ResourceGroupName iidrtestrg -CommandId 'RunPowerShellScript' -scriptpath '2019-03-21-Disable-NOC-LMI-VEEAM.ps1' -vmname ii-deltek -verbose -debug
DEBUG: 11:08:30 AM - InvokeAzureRmVMRunCommand begin processing with ParameterSet 'DefaultParameter'.

Confirm
Continue with this operation?
[Y] Yes  [A] Yes to All  [H] Halt Command  [S] Suspend  [?] Help (default is "Y"): y
DEBUG: 11:08:32 AM - using account id '[email protected]'...

Confirm
Continue with this operation?
[Y] Yes  [A] Yes to All  [H] Halt Command  [S] Suspend  [?] Help (default is "Y"): y

Confirm
Are you sure you want to perform this action?

Performing the operation "Invoke" on target "ii-deltek".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): y

POWERSHELL has been hung for 22 minutes.


Tuesday, April 2, 2019 3:33 PM

Here are the contents of the 2019-03-21-Disable-NOC-LMI-VEEAM.ps1

#############################################################
##
## VERSION: 1.6
##
##########################################################

## stop-service -force dhcp
## set-service dhcp -startuptype disabled

net use H: \blscripts.file.core.windows.net\blscripts /u:AZURE\blscripts <ACCESSKEY> /persistent:yes 

$today = Get-Date -UFormat "%Y-%m-%d %R"
$server = [System.Net.DNS]::GetHostByName($Null).HostName

$logfile = "H:\BL-CSE_LOG-$server.txt"

write-output ">>>>>>>>>>> `n BL-CSE script will disable VEEAM, NOC, LMI, etc on this $server at $today " | out-file -append $logfile

get-service *veeam* | stop-service 
get-service *veeam* | set-service -startuptype disabled

Stop-Service VeeamBackupSvc
Set-Service VeeamBackupSvc -StartupType Disabled
Stop-Service VeeamBrokerSvc
Set-Service VeeamBrokerSvc -StartupType Disabled
Stop-Service VeeamCatalogSvc
Set-Service VeeamCatalogSvc -StartupType Disabled
Stop-Service VeeamDeploySvc
Set-Service VeeamDeploySvc -StartupType Disabled
Stop-Service VeeamCloudSvc
Set-Service VeeamCloudSvc -StartupType Disabled
Stop-Service VeeamDistributionSvc
Set-Service VeeamDistributionSvc -StartupType Disabled
Stop-Service VeeamMountSvc
Set-Service VeeamMountSvc -StartupType Disabled
Stop-Service VeeamNFSSvc
Set-Service VeeamNFSSvc -StartupType Disabled
Stop-Service VeeamTransportSvc
Set-Service VeeamTransportSvc -StartupType Disabled
Stop-Service VeeamEndpointBackupSvc
Set-Service VeeamEndpointBackupSvc -StartupType Disabled
 
get-service *saaz* | stop-service 
get-service *saaz* | set-service -startuptype disabled

Stop-Service SAAZappr
Set-Service SAAZappr -StartupType Disabled
Stop-Service SAAZapsc
Set-Service SAAZapsc -StartupType Disabled
Stop-Service SAAZDPMACTL
Set-Service SAAZDPMACTL -StartupType Disabled
Stop-Service SAAZMSMACTL
Set-Service SAAZMSMACTL -StartupType Disabled
Stop-Service SAAZRemoteSupport
Set-Service SAAZRemoteSupport -StartupType Disabled
Stop-Service SAAZScheduler
Set-Service SAAZScheduler -StartupType Disabled
Stop-Service SAAZServerPlus
Set-Service SAAZServerPlus -StartupType Disabled
Stop-Service SAAZWatchDog
Set-Service SAAZWatchDog -StartupType Disabled
Stop-Service ITSPlatformManager
Set-Service ITSPlatformManager -StartupType Disabled
 
get-service *logmein* | stop-service 
get-service *logmein* | set-service -startuptype disabled
get-service *lmi* | stop-service 
get-service *lmi* | set-service -startuptype disabled

Stop-Service LogMeIn
Set-Service LogMeIn -StartupType Disabled
Stop-Service LMIGuardianSvc
Set-Service LMIGuardianSvc -StartupType Disabled
Stop-Service LMIMaint
Set-Service LMIMaint -StartupType Disabled

### STOP HYPERV Services

get-service *vmic* | stop-service 
get-service *vmic* | set-service -startuptype disabled

Stop-Service vmicguestinterface
Stop-Service vmicheartbeat
Stop-Service  vmickvpexchange
Stop-Service  vmicrdv
Stop-Service  vmicshutdown
Stop-Service  vmictimesync
Stop-Service  vmicvss

## Stop SOPHOS
get-service *sophos* | stop-service 
get-service *sophos* | set-service -startuptype disabled

####################

get-service saaz*     | Out-String | Out-File -append $logfile 
get-service log*      | Out-String | Out-File -append $logfile 
get-service lmi*      | Out-String | Out-File -append $logfile 
get-service vmic*     | Out-String | Out-File -append $logfile 
get-service *sophos*  | Out-String | Out-File -append $logfile 
get-service *veeam*   | Out-String | Out-File -append $logfile 

write-output "This $server will SHUTDOWN in 10 seconds - BL CSE script" | out-file -append $logfile
start-sleep 60 

stop-computer -force


Tuesday, April 2, 2019 3:52 PM

Here's the output at 11:51 AM EST (approx 43 minutes of execution time)

DEBUG: AzureQoSEvent: CommandName - Invoke-AzureRmVMRunCommand; IsSuccess - False; Duration - 00:25:49.7183703; Exception - Microsoft.Azure.Commands.Compute.Common.ComputeCloudException: Long running
operation failed with status 'Failed'. Additional Info:'VM 'ii-deltek' has not reported status for VM agent or extensions. Please verify the VM has a running VM agent and can establish outbound
connections to Azure storage.'
ErrorCode: VMAgentStatusCommunicationError
ErrorMessage: VM 'ii-deltek' has not reported status for VM agent or extensions. Please verify the VM has a running VM agent and can establish outbound connections to Azure storage.
ErrorTarget:
StartTime: 4/2/2019 11:08:39 AM
EndTime: 4/2/2019 11:33:53 AM
OperationID: f4698365-bd06-4e3e-8606-669cd60dd403
Status: Failed > Microsoft.Rest.Azure.CloudException: Long running operation failed with status 'Failed'. Additional Info:'VM 'ii-deltek' has not reported status for VM agent or extensions. Please
verify the VM has a running VM agent and can establish outbound connections to Azure storage.'
   at Microsoft.Rest.ClientRuntime.Azure.LRO.AzureLRO`2.CheckForErrors()
   at Microsoft.Rest.ClientRuntime.Azure.LRO.AzureLRO`2.<StartPollingAsync>d__20.MoveNext()
End of stack trace from previous location where exception was thrown
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Rest.ClientRuntime.Azure.LRO.AzureLRO`2.<BeginLROAsync>d__15.MoveNext()
End of stack trace from previous location where exception was thrown
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Rest.Azure.AzureClientExtensions.<GetLongRunningOperationResultAsync>d__1`2.MoveNext()
End of stack trace from previous location where exception was thrown
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Rest.Azure.AzureClientExtensions.<GetLongRunningOperationResultAsync>d__0`1.MoveNext()
End of stack trace from previous location where exception was thrown
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Rest.Azure.AzureClientExtensions.<GetPostOrDeleteOperationResultAsync>d__8`1.MoveNext()
End of stack trace from previous location where exception was thrown
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Management.Compute.VirtualMachinesOperations.<RunCommandWithHttpMessagesAsync>d__23.MoveNext()
End of stack trace from previous location where exception was thrown
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Management.Compute.VirtualMachinesOperationsExtensions.<RunCommandAsync>d__37.MoveNext()
End of stack trace from previous location where exception was thrown
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Management.Compute.VirtualMachinesOperationsExtensions.RunCommand(IVirtualMachinesOperations operations, String resourceGroupName, String vmName, RunCommandInput parameters)
   at Microsoft.Azure.Commands.Compute.Automation.InvokeAzureRmVMRunCommand.<ExecuteCmdlet>b__0_0()
   at Microsoft.Azure.Commands.Compute.ComputeClientBaseCmdlet.ExecuteClientAction(Action action)
    End of inner exception stack trace
   at Microsoft.Azure.Commands.Compute.ComputeClientBaseCmdlet.ExecuteClientAction(Action action)
   at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord();
DEBUG: Finish sending metric.
DEBUG: 11:51:37 AM - InvokeAzureRmVMRunCommand end processing.
DEBUG: 11:51:37 AM - InvokeAzureRmVMRunCommand end processing.


Tuesday, April 2, 2019 6:27 PM

It seems like VM agent is not able to communicate to Azure storage. Can you enable Service endpoint to Azure Storage and check if that helps?

Regards, 

Msrini


Wednesday, April 3, 2019 6:39 PM

How do I enable the SERVICE ENDPOINT via powershell?


Wednesday, April 3, 2019 7:25 PM

You can use portal to configure it in a easy way. 

Reference: /en-us/azure/virtual-network/tutorial-restrict-network-access-to-resources

Regards, 

Msrini


Friday, April 19, 2019 5:58 PM

Any update on this issue?

If the proposed answer was useful please remember to mark it as the answer so others can easily find it.