It seems the semaphore count is directly related to the cache for SCHANNEL. There's a task running every 5 minutes which remove unusued cache/zombies connections.
If you want to limit the number of max connections for servers (20000 per default) you can change the key MaximumCacheSize in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\
and restart the computer to apply. But, you will have to wait the timeout of 5 minutes to drop cache above your MaxCacheSize.
Also check the values ServerCacheTime and ClientCacheTime, a value of 0 turns off TLS session caching on the server & client.