Start-Job hangs Powershell on Windows Server 2016

Article
2019-10-02

Question

_{Wednesday, October 2, 2019 8:59 PM}

I'm not sure where one reports bugs and a phone call to support the folks on the other end suggested I send a letter. I figured this was better.

Occasionally when running Start-Job the operation hangs. You can replicate this by creating a loop that spins off a bunch of jobs with a ScriptBlock. I haven't tested it without a ScriptBlock. Using threads to spin the jobs off to keep the main thread running I see multiple jobs that failed to start out of a loop of 200.

I've worked with jobs in older versions of Windows Server and never seen this before. Thought someone might want to know.

Here's a code block you can use to replicate. Just adjust the code for where you want log files to be dumped. You'll know it failed by having a missing log file for a job.

$ChildScriptBlock = {
    $Parameters = $args[0]
    
    $log = '<your path to log files>\' + $Parameters["jobName"] + '.txt'
    "Child Script: Got " + $Parameters["P1"] + " and " + $Parameters["P2"] + " With fallback of " + $Parameters["Fallback"]>> $log

    
    
}

$ParentScriptBlock = {
    $ParentParameters = $args[1]
    $ChildScriptBlock = $ParentParameters["ChildScriptBlock"]
    $log = '<your path to log files>\' + $ParentParameters["jobName"] + '.txt'
    $ChildParameters = @{
        Fallback = "fallback";
    }
    foreach ($key in $ParentParameters.Keys)
    {
        if ($key -ne "ChildScriptBlock")
        {
            $ChildParameters[$key] = $ParentParameters[$key]
        }
    }
    $job = Start-Job -Name $ParentParameters["jobName"] -ScriptBlock $ChildScriptBlock -ArgumentList $ChildParameters
}

$ParentParameters = @{
    P1 = "Parameter 1";
    P2 = "Parameter 2";
    ChildScriptBlock = $ChildScriptBlock;
}

$runspacePool = [runspacefactory]::CreateRunspacePool()
$runspacePool.Open()
$taskList = @()
for ($jobnumber = 1; $jobnumber -lt 200; $jobnumber++)
{
    $ParentParameters["jobName"] = "job_$jobnumber"
    $newThread = [powershell]::Create().AddScript($ParentScriptBlock).AddParameter('ParentParameters', $ParentParameters)
    $newThread.RunspacePool = $runspacePool
    $handle = $newThread.BeginInvoke()
    $taskList += New-Object -TypeName psobject -Property @{ "Handle" = $handle; "Thread" = $newThread; }
    Start-Sleep 1
}

$taskList | Where-Object { $_.Handle.IsCompleted } | ForEach-Object {
    $_.Thread.EndInvoke($_.Handle)
    $_.Thread.Dispose()
    $_.Thread = $_.Handle = $Null
}

All replies (17)

_{Wednesday, October 2, 2019 9:16 PM}

Is there a question?

To report bugs please post in UserVoice.

https://devblogs.microsoft.com/powershell/understanding-the-powershell-uservoice/

\(ツ)_/

_{Wednesday, October 2, 2019 9:57 PM}

I should have added "Is anyone else seeing this?" but yeah thanks for that link. I'll take a look. It's not exactly easy to figure out how to report a bug.

_{Wednesday, October 2, 2019 10:07 PM}

PowerShell bugs are reported on UserVoice.

I have never seen what you are seeing. Be sure to check the job status. Jobs can seem hung it your code has certain coding mistakes.

The correct way to group jobs is to crate a child job within the main job and not by using a job to start a job.

\(ツ)_/

_{Wednesday, October 2, 2019 10:14 PM}

You are creating a runspace to run a job which is completely unnecessary. Why would you do this? Runspaces and jobs are not anywhere alike. A job is a separate process and a runspace is within the current process. It is a thread/task managed by the PS runspace manager.

Get-Runspace

You will see that your runspace is still running because it has no terminator. Be sure the runspace can execute correctly. There is no point in running a job in a runspace. It adds nothing except overhead.

\(ツ)_/

_{Wednesday, October 2, 2019 10:21 PM}

Another obvious coding error is that the scriptblock terminates immediately after starting the job. This also terminates the job. Doing this in a loop many times will make the whole thing appear to hang.

\(ツ)_/

_{Thursday, October 3, 2019 3:13 PM}

That's not the problem. Do this on Windows Server 2016 in either ISE or just a standard Powershell window:

$ScriptBlock = {
    # Do something simple or complex it doesn't matter
}
$arguments = "Whatever args you want"
for ($j = 1; $j -lt 200; $j++)
{
    Write-Host "Starting job $j"
    Start-Job -Name "Job_$j" -ScriptBlock $ScriptBlock -ArgumentList $arguments
    Write-Host "Finished starting job $j"
}

The larger you make the number of jobs you're going to spin off the greater likelihood it's going to hang at some point and you'll get the "Starting job" but never get to "Finished starting job" and if you put in proper code to detect whether the job you were trying to start got started, you'll see that Powershell hung before spinning off the job.

_{Thursday, October 3, 2019 3:19 PM}

You are creating a runspace to run a job which is completely unnecessary. Why would you do this? Runspaces and jobs are not anywhere alike. A job is a separate process and a runspace is within the current process. It is a thread/task managed by the PS runspace manager.

Get-Runspace

You will see that your runspace is still running because it has no terminator. Be sure the runspace can execute correctly. There is no point in running a job in a runspace. It adds nothing except overhead.

\(ツ)_/

This is for illustration purposes and the thread code was lifted mostly from Start-Parallel module in order to isolate the starting of jobs away from the parent thread so it doesn't hang and then I can see how prevalent the failing for jobs to start actually is. Out of 200 (well 199 technically) jobs the highest I saw fail to start in running it a handful of times was 6. Not a single time did it get all the way through starting all jobs.

The problem is Powershell locking up on starting a job and not my code. The original code that kept hanging was very pedestrian and didn't use runspaces.

_{Thursday, October 3, 2019 3:22 PM}

I see no point in any of this. The running of 200+ jobs does not cause any issues when I run it. As noted above bug reports need to be posted to UserVoice.

\(ツ)_/

_{Thursday, October 3, 2019 3:25 PM}

I did that on UserVoice. You did this on Windows Server 2016? I've done this before on 2012 without issue.

_{Thursday, October 3, 2019 3:51 PM}

You are creating a runspace to run a job which is completely unnecessary. Why would you do this? Runspaces and jobs are not anywhere alike. A job is a separate process and a runspace is within the current process. It is a thread/task managed by the PS runspace manager.

Get-Runspace

You will see that your runspace is still running because it has no terminator. Be sure the runspace can execute correctly. There is no point in running a job in a runspace. It adds nothing except overhead.

\(ツ)_/

This is for illustration purposes and the thread code was lifted mostly from Start-Parallel module in order to isolate the starting of jobs away from the parent thread so it doesn't hang and then I can see how prevalent the failing for jobs to start actually is. Out of 200 (well 199 technically) jobs the highest I saw fail to start in running it a handful of times was 6. Not a single time did it get all the way through starting all jobs.

The problem is Powershell locking up on starting a job and not my code. The original code that kept hanging was very pedestrian and didn't use runspaces.

Jobs are all run on separate threads. They are a completely separate copy of PS. Run 5 jobs and run Get-Process PowerShell and you will see that there are many copies of PowerShell running. Each is a separate thread that runs outside of the creating PowerShell process.

Runspaces run "inProcess" and jobs run "out of process". Look up those terms to understand the difference. Running jo0bs from a separate runspace solves no problems and adds a lot of extra overhead.

Either use jobs or runspaces but not both. Runspaces are useful when we want to control how many task threads are released at a time. The defauit is 8 but can be adjusted easily. Runspaces are not more efficient for most tasks which is wy MS hasn't created a CmdLet that runs a script in a runspace. Jobs are more efficient for most tasks. When we use runspaces we likely want to have reusable code tied to a runspace and rerun the runspace pool with new arguments. Once the pool is built we can just restart it to do a now large parallel task.

So running a job in a runspace still creates a separate copy of PowerShell but also adds a lot of overhead to the task and to the PowerShell process that created the runspace. The runspace also adds compute overhead to the local PS session. Jobs start and execute out of process then notify the calling session when complete. The job runs causing no overhead in the session that created the job.

I see many blogs and scripts that clearly don't understand how this works or make bad assumptions on what is more efficient. In the end - no matter what you do - a system can only run as many tasks in parallel as there are processors. Adding threads adds overhead because the threads have to be rescheduled frequently. The efficiency and performance are highly dependent on the processors being used. Some are very efficient at task switching and others are less efficient. All processes are threads so jobs can be less efficient because we cannot control concurrency as well as we can with runspaces. We can choose to release only the optimal number of tasks (threads) with runspaces but jobs are controlled by the system.

Jobs are best used for large long-running scripts. Runspace are efficient for many short scripts running in parallel. Adding a job to a runspace just subverts this so you end up with the worst of both worlds.

PS also has a persisted job that can run between sessions and be restarted with new data when needed. It is called a "Scheduled Job". This job is stored on disk and can run between boots of the system. It can also continue running when the user logs off.

See:

help scheduledjob

What can be useful is to create a scheduled job that create a set of runspaces and runs continuously in the background processing data as the data becomes available or when we need periodically collect data from remote locations. The runspaces can be set up as a queuing system that processes queued data in round-robin fashion. This can be designed to be very low overhead and to be very efficient when there is a load.

\(ツ)_/

_{Thursday, October 3, 2019 4:45 PM}

There is a ThreadJob module as well that comes with PS 6. It's like Jobs but uses threads. You can download it for PS 5.

_{Thursday, October 3, 2019 4:48 PM}

Not likely. Just use jobs or use runspaces but not both and you won't have issues.

\(ツ)_/

_{Thursday, October 3, 2019 5:25 PM}

Not likely. Just use jobs or use runspaces but not both and you won't have issues.

\(ツ)_/

I'm not sure if I'm not explaining clearly or what the disconnect is but you're completely missing the entire purpose here.

The problem I'm describing came from seeing Start-Job hang when just executing from a single thread. Everything else after was trying to work around what's clearly a bug.
The bulk of what's happening is going on *IN* the ScriptBlock being run by child processes and I'm trying to squeeze as much out of the system as a whole within limits that don't impair normal function. Keeping it in the same thread space does NOT accomplish this.
There's very little happening in the main thread so any overhead of standing up and tearing down new independent threads is NBD.
Maybe I'm missing it but using runspaces like in this code doesn't result in fully independent threads. At least that's what I read somewhere.
If the thread in runspaces isn't fully independent, then launching a job from one of these threads does create a fully forked process which is what I want.
If the thread in runspaces IS fully independent, then fine I'll stop launching a job and just use that as a workaround.

TL; DR; -- the main purpose is the bug of Powershell hanging on Start-Job and runspaces was just part of looking for a workaround.

_{Thursday, October 3, 2019 5:42 PM}

Side note -- I tried using Invoke-Command which treats the job as remote. This doesn't cause Powershell to hang but introduced new inexplicable problems.

Ultimately I need to be able to keep creating new jobs to maximize usage and start-job is the way to do it, but because of the bug I have no other choice than to launch in some other manner than just running Start-Job in the main thread.

It's worth noting that the issue appears even if I keep the number of concurrent jobs throttled.

_{Thursday, October 3, 2019 6:11 PM}

Side note -- I tried using Invoke-Command which treats the job as remote. This doesn't cause Powershell to hang but introduced new inexplicable problems.

Ultimately I need to be able to keep creating new jobs to maximize usage and start-job is the way to do it, but because of the bug I have no other choice than to launch in some other manner than just running Start-Job in the main thread.

It's worth noting that the issue appears even if I keep the number of concurrent jobs throttled.

You can create as many jobs as you have memory for (32767 or more) and there will be no issue. Running start-job in a runspace will not overcome these limits and will make managing the jobs more difficult.

There is no bug in 2016 so you must have other issues like memory or some Net Framework element that need updating or fixing.

Your issue belongs only in UserVoice. We cannot fix your system for you nor can I reproduce your issues.

Part of the issue here is that you are trying to move into highly technical areas of systems programming with no experience in systems programming or Windows systems coding and design. This is what is making you go in circles with this. Part of the problem is that you keep thinking you need a job in a runspace. A runspace IS a job but an in-process job. Runspaces will behave correctly if designed and managed correctly. If you are using many runspaces then you should use a RunspaceFactory that you can use to throttle the number of active runspaces. It would also be best to use events to mange and respond to the runspace states.

Jobs are simpler and managed by the system although many jobs will consume more memory and resources. A run space with a job will only be worse.

If you really think you have a bug then post in UserVoice. This is not the correct forum for you issue.

\(ツ)_/

_{Thursday, October 3, 2019 6:26 PM}

Not Maybe I'm missing it but using runspaces like in this code doesn't result in fully independent threads. At least that's what I read somewhere.

If the thread in runspaces isn't fully independent, then launching a job from one of these threads does create a fully forked process which is what I want.

If the thread in runspaces IS fully independent, then fine I'll stop launching a job and just use that as a workaround.

TL; DR; -- the main purpose is the bug of Powershell hanging on Start-Job and runspaces was just part of looking for a workaround.

A thread is never independent of the process that it runs in. You ae reading a lot of misleading things. A runspace runs in a RunSpaceFactory which only allows 8 threads at a time but allows you to schedule as many threads as the factory is configured for. The default factory is the one that PowerShell is running in. If you open PowerShell and type "Get-Runspace" you will see that. Adding many runspaces will slow down the main PowerShell instance and eat up memory. Using a new factory will help this to some degree but the process running all of this will reach some limits anyway. Using only jobs will run all jobs in independent processes that will not impact the parent PowerShell session/process.

Also stop thinking in threads. Runspaces are not threads like you get with "[system.Threading.Thread]::new()". A runspace is a complex object that can raise events and manage state and data. It may also release many threads. When using Start-Job in a runspace you are generating a lot of extra threads that all are charged to the current PS process. Failure to cleanup all of these threads is costly and cleaning up causes major delays in the parent processes. It is just a bad idea. Just put good and carefully tested code in the runspace or job and you won't have issues.

Understanding the purpose for these things is important. Just knowing they exist and following the leader when many PS coders are trying to find cool new things with as little experience as you have only adds to the misinformation.

Use only one - runspaces or jobs. There is almost never any reason to combine them ands when you do it must be carefully thought out and understood.

\(ツ)_/

_{Thursday, October 3, 2019 6:37 PM}

Here is something that can help you to understand the issues. It uses Invoke to the local system if remot9ing allows this and will show you the effect of jobs on the Windows system

get-process powershell
get-runspace

get-job |remove-job

invoke-command -ScriptBlock {start-job {sleep 60}} -ComputerName sbs01,sbs01 -asjob

get-process powershell
get-runspace

\(ツ)_/

Share via

Start-Job hangs Powershell on Windows Server 2016

Question

All replies (17)

Additional resources