Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Question
Thursday, April 23, 2020 12:04 PM
We have a strange issue at the moment on our recently implemented Always On VPN solution. Users will be happily working fine without any issue then on a given day they will start to get the following error.
The issue generally resolves itself in a few hours but is happening all to often and causing frustration to users. The normal reboot of laptop and routers has no effect.
On the server for the user we get the following pair of error logs
On the client we get two RasClient errors Event 20227 code 812 and Event 20226 Code 631
Been through the following /en-us/windows-server/remote/remote-access/vpn/always-on-vpn/deploy/always-on-vpn-deploy-troubleshooting
The client and server configuration match and they are normally working it just seems for a period of time on given days it does not match.
Any Ideas??
All replies (29)
Friday, April 24, 2020 6:14 AM
Hi ,
Please Enable Microsoft-Windows-Security-Auditing on RRAS Server and then check if there are something related for us to troubleshooting.
This can be performed via local GPO on RRAS server as below:
Monitored the Security Events of VPN until a new VPN client was rejected.
Check Windows-Security-Auditing log to see if there are something related.
Best Regards,
Candy
Please remember to mark the replies as an answers if they help.
If you have feedback for TechNet Subscriber Support, contact [email protected]
Friday, April 24, 2020 8:59 AM
Hi Candy,
Thank you for the additional information. We have now enabled this additional logging and have the following result.
We had a pair of events on the server within the System log for a user at 09:14:23
We had the following Audit Failure message
Friday, April 24, 2020 9:54 AM
Hi ,
Have you configure NPS to accept and receive the RADIUS authentication requests sent by the VPN server? If yes, please ensure that your client configuration matches the conditions that are specified on the NPS server, like connection request policy.
Best Regards,
Candy
Please remember to mark the replies as an answers if they help.
If you have feedback for TechNet Subscriber Support, contact [email protected]
Friday, April 24, 2020 10:02 AM
Hi Candy,
Thanks we have checked this and both match. Users are able to connect fine one day and then another day get this issue for a period of time and then starts to work again.
Monday, April 27, 2020 12:16 PM
Can you Wireshark the traffic from both days and try to compare?
Sounds like you have a strange one on your hands.
Tuesday, April 28, 2020 8:03 AM
Hi sysadminjames,
Could you provide some more information around how to configure wireshark to get the best results and compare.
Tuesday, April 28, 2020 8:14 AM
Morning Leigh,
I am no expert but just filter the traffic to the destination IP of your AOVPN.
Open Wireshark > select Wifi/Lan adapter whichever one you are using > begin capture then dial VPN > You should see you Public IP appear in the destination column > right cliick that > apply as filter > selected.
Then restart you capture with the filter applied.
It's difficult because we do not know what we are looking for, but if you can get a trace from a good day, then one from a bad you can go down each line to see if you can find a difference.
What's your environment look like?
What OS?
How many servers etc..?
User and device tunnels?
Tuesday, April 28, 2020 11:26 AM
Hello All,
I don't think that a simple network trace will help much here. The image that Leigh posted earlier shows the "Keying Module Port" as 4500 and State as "EAP payload sent". The VPN traffic is encrypted by this point in the VPN connection establishment process.
IKEv2 starts out using UDP port 500 and one plain text message is sent in each direction to establish the cryptographic parameters for the next message exchange; if the NAT detection mechanism detects that NAT is in use (or if MOBIKE is enabled) then IKEv2 switches to port 4500. So the use of port 4500 and the mention of EAP means that the failure occurs after encryption has been negotiated for the message exchanges.
This is what a network trace of an IKEv2 connection establishment looks like (this trace is augmented with information from the Microsoft-Windows-WFP provider):
The SKF and SK IKEv2 payloads indicate encrypted fragment payload and encrypted payload respectively.
There are ways of "looking into" the encrypted packets to see what is going on (for example, by using the "IKEEXT Trace Provider" WPP ETW provider), but much of the data, toolset and know-how needed to do this is not widely available.
Gary
Tuesday, April 28, 2020 1:32 PM
Hi Gary,
Thanks for the detailed up and confirmation that digging into network traces would not be much help. We have spent a bit of time looking into these but not seeing any difference.
One of our system guys who was effected started to play around with his settings and has had some success in changing the authentication from EAP (PEAP) to machine certificates. He was able to connect straight away and not had the issue since. We have done this for another 2 users and have had the same result. We cannot seem to find a reason why but we are going to see if this continues to resolve the issue.
Tuesday, April 28, 2020 5:20 PM
Hi Leigh,
Are you using a device tunnel?
Tuesday, April 28, 2020 5:22 PM
Yes we have Device and User tunnel configured.
Tuesday, April 28, 2020 5:51 PM
Is he changing these settings on the device or user configuration ?
Tuesday, April 28, 2020 5:58 PM
Changed on both - Has continued to work without issue
Tuesday, April 28, 2020 7:49 PM
So looking over the documentation, I believe that the device tunnel should certainly be using the machine certificate. This is set on the NPS policy and in the VPN settings.
The user policy however I have set to use EAP (PEAP) protocols and that is what is also mentioned i the docs.
Presumably you have configured both tunnels to use the same certificate?
Wednesday, April 29, 2020 9:21 AM
We have the exact same problem
Out of interest, are you using any load balacing? Who is your firewall vendor?
Personally im using Cisco FTD firewall and Microsoft NLB
Wednesday, April 29, 2020 9:23 AM
So BenSBB are you using both device and user tunnel?
Is your User tunnel setup to use EAP (PEAP) ?
Wednesday, April 29, 2020 9:29 AM
Just trying to use user tunnel at the moment with PEAP
Device tunnel works fine with machine cert but i haven't rolled that out to users
Wednesday, April 29, 2020 10:17 AM
You get the same error as above, it will connect one day but then another it does not ?
What error code do you get in EventVwr?
Wednesday, April 29, 2020 11:04 AM
Yes exatly. It's intermittent, some users will connect fine and some will fail. Then after a few mins the ones that failed will be able to manually connect (but this defeats the purpose of "Always-On VPN") There doesn't seem to be any pattern to it.
The users are intermittently getting Error 812 - "The connection was prevented because of a policy configured on your RAS/VPN server. Specifically, the authentication method used by the server to verify your username and password may not match the authentication method configured in your connection profile. Please contact the Administrator of the RAS server and notify them of this error."
If they hammer the connect button or wait a few mins and try again it will eventually connect. Once the connection is made it works reliably
I'm getting the following errors on the RRAS server when the failure occurrs:
Event 20271:
The user <USER UPN> connected from <USER PUBLIC IP> but failed an authentication attempt due to the following reason: The connection was prevented because of a policy configured on your RAS/VPN server. Specifically, the authentication method used by the server to verify your username and password may not match the authentication method configured in your connection profile. Please contact the Administrator of the RAS server and notify them of this error.
Event 20255:
The following error occurred in the Point to Point Protocol module on port: VPN2-497, UserName: <Unauthenticated User>. The connection was prevented because of a policy configured on your RAS/VPN server. Specifically, the authentication method used by the server to verify your username and password may not match the authentication method configured in your connection profile. Please contact the Administrator of the RAS server and notify them of this error.
Wednesday, April 29, 2020 11:29 AM
Hi BenSBB,
What you explain is the exact issue we are having although it does not seem to matter how many times you try to connect it just fails and then at some point in the day it starts to work.
We are still running ok with users at the moment on machine certificates for user tunnel. So we now have a mixture of configuration.
I am working perfectly fine with Device Tunnel on Machine Certs and User Tunnel on EAP.
Might be worth trying with a user who is getting the problem the change we have made and see if you get the same results.
Wednesday, April 29, 2020 11:33 AM
Definitely sounds like a mismatch somewhere in your VPN properties on the client and the NPS policy that is configured.
Here is what I have, for my nps01 entry it is my internal domain not public one. If that makes sense? so nps01.domain.local
Wednesday, April 29, 2020 11:34 AM
Hello All,
In an earlier message Leigh posted an image from the event log. In one part of the image is the text: "Main Mode Filter ID: 77475".
The filter IDs are dynamic (any modifications to filters often causes recalculation of all filters and the assignment of new filter IDs) so that filter ID number is probably now passé. If you capture a new event, you could try examining the Main Mode Filter; issue the command "netsh WFP show state file=VPN.xml" and then search for that filter ID. It might look like this:
<item>
<providerContextKey>{474ad6f6-97b0-4c95-99b3-60e26da5646f}</providerContextKey>
<displayData>
<name>IKEv2 Server Main mode IPsec tunnel policy (v4) (* to *)</name>
<description/>
</displayData>
<flags/>
<providerKey/>
<providerData/>
<type>FWPM_IPSEC_IKEV2_MM_CONTEXT</type>
<ikeV2MmPolicy>
<softExpirationTime>0</softExpirationTime>
<authenticationMethods numItems="2">
<item>
<authenticationMethodType>IKEEXT_EAP</authenticationMethodType>
<eapAuthentication>
<flags numItems="1">
<item>IKEEXT_EAP_FLAG_REMOTE_AUTH_ONLY</item>
</flags>
</eapAuthentication>
</item>
<item>
<authenticationMethodType>IKEEXT_CERTIFICATE</authenticationMethodType>
<certificateAuthentication>
<inboundConfigType>3</inboundConfigType>
<outboundConfigType>IKEEXT_CERT_CONFIG_TRUSTED_ROOT_STORE</outboundConfigType>
<outboundTrustedRootStoreCriteria numItems="2">
<item>
<certData/>
<certHash/>
<eku>
<numEku>2</numEku>
<eku numItems="2">
<item>1.3.6.1.5.5.7.3.1</item>
<item>1.3.6.1.5.5.8.2.2</item>
</eku>
</eku>
<name/>
<flags/>
</item>
<item>
<certData/>
<certHash/>
<eku>
<numEku>1</numEku>
<eku numItems="1">
<item>1.3.6.1.5.5.7.3.1</item>
</eku>
</eku>
<name/>
<flags/>
</item>
</outboundTrustedRootStoreCriteria>
<flags/>
<localCertLocationUrl/>
</certificateAuthentication>
</item>
</authenticationMethods>
<initiatorImpersonationType>IKEEXT_IMPERSONATION_NONE</initiatorImpersonationType>
<ikeProposals numItems="18">
<item>
<cipherAlgorithm>
<algoIdentifier>IKEEXT_CIPHER_3DES</algoIdentifier>
<keyLen>0</keyLen>
<rounds>0</rounds>
</cipherAlgorithm>
<integrityAlgorithm>
<algoIdentifier>IKEEXT_INTEGRITY_SHA1</algoIdentifier>
</integrityAlgorithm>
<maxLifetimeSeconds>28800</maxLifetimeSeconds>
<dhGroup>IKEEXT_DH_GROUP_2</dhGroup>
<quickModeLimit>0</quickModeLimit>
</item>
<item>
<cipherAlgorithm>
<algoIdentifier>IKEEXT_CIPHER_3DES</algoIdentifier>
<keyLen>0</keyLen>
<rounds>0</rounds>
</cipherAlgorithm>
<integrityAlgorithm>
<algoIdentifier>IKEEXT_INTEGRITY_SHA_256</algoIdentifier>
</integrityAlgorithm>
<maxLifetimeSeconds>28800</maxLifetimeSeconds>
<dhGroup>IKEEXT_DH_GROUP_2</dhGroup>
<quickModeLimit>0</quickModeLimit>
</item>
<item>
<cipherAlgorithm>
<algoIdentifier>IKEEXT_CIPHER_3DES</algoIdentifier>
<keyLen>0</keyLen>
<rounds>0</rounds>
</cipherAlgorithm>
<integrityAlgorithm>
<algoIdentifier>IKEEXT_INTEGRITY_SHA_384</algoIdentifier>
</integrityAlgorithm>
<maxLifetimeSeconds>28800</maxLifetimeSeconds>
<dhGroup>IKEEXT_DH_GROUP_2</dhGroup>
<quickModeLimit>0</quickModeLimit>
</item>
<item>
<cipherAlgorithm>
<algoIdentifier>IKEEXT_CIPHER_AES_128</algoIdentifier>
<keyLen>0</keyLen>
<rounds>0</rounds>
</cipherAlgorithm>
<integrityAlgorithm>
<algoIdentifier>IKEEXT_INTEGRITY_SHA1</algoIdentifier>
</integrityAlgorithm>
<maxLifetimeSeconds>28800</maxLifetimeSeconds>
<dhGroup>IKEEXT_DH_GROUP_2</dhGroup>
<quickModeLimit>0</quickModeLimit>
</item>
<item>
<cipherAlgorithm>
<algoIdentifier>IKEEXT_CIPHER_AES_128</algoIdentifier>
<keyLen>0</keyLen>
<rounds>0</rounds>
</cipherAlgorithm>
<integrityAlgorithm>
<algoIdentifier>IKEEXT_INTEGRITY_SHA_256</algoIdentifier>
</integrityAlgorithm>
<maxLifetimeSeconds>28800</maxLifetimeSeconds>
<dhGroup>IKEEXT_DH_GROUP_2</dhGroup>
<quickModeLimit>0</quickModeLimit>
</item>
<item>
<cipherAlgorithm>
<algoIdentifier>IKEEXT_CIPHER_AES_128</algoIdentifier>
<keyLen>0</keyLen>
<rounds>0</rounds>
</cipherAlgorithm>
<integrityAlgorithm>
<algoIdentifier>IKEEXT_INTEGRITY_SHA_384</algoIdentifier>
</integrityAlgorithm>
<maxLifetimeSeconds>28800</maxLifetimeSeconds>
<dhGroup>IKEEXT_DH_GROUP_2</dhGroup>
<quickModeLimit>0</quickModeLimit>
</item>
<item>
<cipherAlgorithm>
<algoIdentifier>IKEEXT_CIPHER_AES_192</algoIdentifier>
<keyLen>0</keyLen>
<rounds>0</rounds>
</cipherAlgorithm>
<integrityAlgorithm>
<algoIdentifier>IKEEXT_INTEGRITY_SHA1</algoIdentifier>
</integrityAlgorithm>
<maxLifetimeSeconds>28800</maxLifetimeSeconds>
<dhGroup>IKEEXT_DH_GROUP_2</dhGroup>
<quickModeLimit>0</quickModeLimit>
</item>
<item>
<cipherAlgorithm>
<algoIdentifier>IKEEXT_CIPHER_AES_192</algoIdentifier>
<keyLen>0</keyLen>
<rounds>0</rounds>
</cipherAlgorithm>
<integrityAlgorithm>
<algoIdentifier>IKEEXT_INTEGRITY_SHA_256</algoIdentifier>
</integrityAlgorithm>
<maxLifetimeSeconds>28800</maxLifetimeSeconds>
<dhGroup>IKEEXT_DH_GROUP_2</dhGroup>
<quickModeLimit>0</quickModeLimit>
</item>
<item>
<cipherAlgorithm>
<algoIdentifier>IKEEXT_CIPHER_AES_192</algoIdentifier>
<keyLen>0</keyLen>
<rounds>0</rounds>
</cipherAlgorithm>
<integrityAlgorithm>
<algoIdentifier>IKEEXT_INTEGRITY_SHA_384</algoIdentifier>
</integrityAlgorithm>
<maxLifetimeSeconds>28800</maxLifetimeSeconds>
<dhGroup>IKEEXT_DH_GROUP_2</dhGroup>
<quickModeLimit>0</quickModeLimit>
</item>
<item>
<cipherAlgorithm>
<algoIdentifier>IKEEXT_CIPHER_AES_256</algoIdentifier>
<keyLen>0</keyLen>
<rounds>0</rounds>
</cipherAlgorithm>
<integrityAlgorithm>
<algoIdentifier>IKEEXT_INTEGRITY_SHA1</algoIdentifier>
</integrityAlgorithm>
<maxLifetimeSeconds>28800</maxLifetimeSeconds>
<dhGroup>IKEEXT_DH_GROUP_2</dhGroup>
<quickModeLimit>0</quickModeLimit>
</item>
<item>
<cipherAlgorithm>
<algoIdentifier>IKEEXT_CIPHER_AES_256</algoIdentifier>
<keyLen>0</keyLen>
<rounds>0</rounds>
</cipherAlgorithm>
<integrityAlgorithm>
<algoIdentifier>IKEEXT_INTEGRITY_SHA_256</algoIdentifier>
</integrityAlgorithm>
<maxLifetimeSeconds>28800</maxLifetimeSeconds>
<dhGroup>IKEEXT_DH_GROUP_2</dhGroup>
<quickModeLimit>0</quickModeLimit>
</item>
<item>
<cipherAlgorithm>
<algoIdentifier>IKEEXT_CIPHER_AES_256</algoIdentifier>
<keyLen>0</keyLen>
<rounds>0</rounds>
</cipherAlgorithm>
<integrityAlgorithm>
<algoIdentifier>IKEEXT_INTEGRITY_SHA_384</algoIdentifier>
</integrityAlgorithm>
<maxLifetimeSeconds>28800</maxLifetimeSeconds>
<dhGroup>IKEEXT_DH_GROUP_2</dhGroup>
<quickModeLimit>0</quickModeLimit>
</item>
<item>
<cipherAlgorithm>
<algoIdentifier>5</algoIdentifier>
<keyLen>0</keyLen>
<rounds>0</rounds>
</cipherAlgorithm>
<integrityAlgorithm>
<algoIdentifier>IKEEXT_INTEGRITY_SHA1</algoIdentifier>
</integrityAlgorithm>
<maxLifetimeSeconds>28800</maxLifetimeSeconds>
<dhGroup>IKEEXT_DH_GROUP_2</dhGroup>
<quickModeLimit>0</quickModeLimit>
</item>
<item>
<cipherAlgorithm>
<algoIdentifier>5</algoIdentifier>
<keyLen>0</keyLen>
<rounds>0</rounds>
</cipherAlgorithm>
<integrityAlgorithm>
<algoIdentifier>IKEEXT_INTEGRITY_SHA_256</algoIdentifier>
</integrityAlgorithm>
<maxLifetimeSeconds>28800</maxLifetimeSeconds>
<dhGroup>IKEEXT_DH_GROUP_2</dhGroup>
<quickModeLimit>0</quickModeLimit>
</item>
<item>
<cipherAlgorithm>
<algoIdentifier>5</algoIdentifier>
<keyLen>0</keyLen>
<rounds>0</rounds>
</cipherAlgorithm>
<integrityAlgorithm>
<algoIdentifier>IKEEXT_INTEGRITY_SHA_384</algoIdentifier>
</integrityAlgorithm>
<maxLifetimeSeconds>28800</maxLifetimeSeconds>
<dhGroup>IKEEXT_DH_GROUP_2</dhGroup>
<quickModeLimit>0</quickModeLimit>
</item>
<item>
<cipherAlgorithm>
<algoIdentifier>6</algoIdentifier>
<keyLen>0</keyLen>
<rounds>0</rounds>
</cipherAlgorithm>
<integrityAlgorithm>
<algoIdentifier>IKEEXT_INTEGRITY_SHA1</algoIdentifier>
</integrityAlgorithm>
<maxLifetimeSeconds>28800</maxLifetimeSeconds>
<dhGroup>IKEEXT_DH_GROUP_2</dhGroup>
<quickModeLimit>0</quickModeLimit>
</item>
<item>
<cipherAlgorithm>
<algoIdentifier>6</algoIdentifier>
<keyLen>0</keyLen>
<rounds>0</rounds>
</cipherAlgorithm>
<integrityAlgorithm>
<algoIdentifier>IKEEXT_INTEGRITY_SHA_256</algoIdentifier>
</integrityAlgorithm>
<maxLifetimeSeconds>28800</maxLifetimeSeconds>
<dhGroup>IKEEXT_DH_GROUP_2</dhGroup>
<quickModeLimit>0</quickModeLimit>
</item>
<item>
<cipherAlgorithm>
<algoIdentifier>6</algoIdentifier>
<keyLen>0</keyLen>
<rounds>0</rounds>
</cipherAlgorithm>
<integrityAlgorithm>
<algoIdentifier>IKEEXT_INTEGRITY_SHA_384</algoIdentifier>
</integrityAlgorithm>
<maxLifetimeSeconds>28800</maxLifetimeSeconds>
<dhGroup>IKEEXT_DH_GROUP_2</dhGroup>
<quickModeLimit>0</quickModeLimit>
</item>
</ikeProposals>
<flags numItems="1">
<item>00000080</item>
</flags>
<maxDynamicFilters>100</maxDynamicFilters>
<retransmitDurationSecs>1800</retransmitDurationSecs>
</ikeV2MmPolicy>
<providerContextId>9223372036854775814</providerContextId>
</item>
Check whether the sub-element "authenticationMethods" matches your expectations.
Gary
Wednesday, April 29, 2020 11:54 AM
I'm actually using the "automatic" setting for the tunnel, so it uses SSTP first, is there some way to troubleshoot that? I guess the IKEv2 topics mentioned aren't relevant for my issue
Wednesday, April 29, 2020 2:15 PM
Hello Ben,
For SSTP, the best way of understanding what is happening is probably to trace the Microsoft-Windows-RRAS ETW provider on the RRAS. I just traced a successful SSTP connection and this is what a very small section of the trace looks like:
There are other ETW providers that can be traced, but for your problem this type of trace will probably contain enough information to make good progress (or even resolve) the problem.
Gary
Wednesday, April 29, 2020 4:37 PM
The only difference to what you have posted in terms of images is the Type of VPN is IKEv2.
Wednesday, April 29, 2020 9:28 PM
Do I have this right though, if you are selecting machine certificate then you user tunnel is authenticating using the certificate found in certlm.msc > Personal > "computer template setup for vpn" (in my case this certificate was specifically for my device tunnel) ?
Thursday, April 30, 2020 8:58 AM
Thanks for the hint, I managed to get the tracing working after a bit of education on my part!
I'm not really seeing much thats pointing me in the right direction though :(
Here is what I see from "EventID 16000" of the trace:
Event Name Time MSec Process Name Rest
Microsoft-Windows-RRAS/EventID(16000) 33444.314 Process(1244) (1244) ThreadID="3,324" ProcessorNumber="0" DebugString="Error 812 while processing Access-Request" FormattedMessage="Error 812 while processing Access-Request "
Microsoft-Windows-RRAS/EventID(16000) 33444.316 Process(1244) (1244) ThreadID="3,324" ProcessorNumber="0" DebugString="Auth Protocol c227 returned error 812" FormattedMessage="Auth Protocol c227 returned error 812 "
So the error I noticed in the event viewer is received at msec 33444
I found the following that could be a clue but i can't find anything when googling them
Thursday, April 30, 2020 9:29 AM | 1 vote
Hello Ben,
The trace does not contain the answer to your problem, but it tells you where to look next. This is the key section:
The first few lines are RADIUS attributes being inserted into a RADIUS request. Type 26 is RADIUS_ATTRIBUTE_VENDOR_SPECIFIC, type 79 is RADIUS_ATTRIBUTE_EAP_MESSAGE, type 1 is RADIUS_ATTRIBUTE_USER_NAME and type 24 is RADIUS_ATTRIBUTE_STATE.
This all takes place 13.4 seconds into the trace. At 33.4 seconds into the trace (20 seconds later), the RRAS reports failure reason 0x75 (117 decimal). The description for this failure reason is "The remote RADIUS server did not respond to the local NPS proxy within an acceptable time period. Verify that the remote RADIUS server is available and functioning properly."
There are several ways of trying to analyse this further. A simple first step might just be to use a network sniffer (Wireshark or similar) to monitor the RADIUS traffic between the RRAS and NPS. It might be useful to continue to trace the Microsoft-Windows-RRAS events too, so that you can correlate the two traces.
Gary
Thursday, April 30, 2020 11:28 AM
Thanks to your help Gary it appears that i've found the cause of my issues
I ran a wireshark trace on the RRAS server and NPS server at the same time and after a few disconnects/reconnects managed to reproduce the issue.
When the issue occurrs I could not actually see that the RAS server was sending any data to the NPS server (I was filtering by the NPS IP in Wireshark). However, after doign a little more reading through the capture , I found that the traffic was actually going to my backup NPS server in MS Azure (configured with less priority in RRAS), which is not what I expected to see! The backup NPS server contains no log entries for the authentication so I have to assume the traffic is just not arriving. I checked on my firewall and didnt see any blocked traffic from the RAS to the secondary NPS (we have an IKEv2 tunnel to Azure). I removed the secondary NPS from teh configuration to identify it is indeed the problem and i've not seen any errors yet ... previously was seeing those 812 errors logged by a user every 5-10 mins or so.
I will schedule some time out of hours to try to understand why the second NPS doesn't work properly because I need to have it working, or at least another solution, as right now I have a single point of failure there. But for now at least it works properly from the user perspective so my users will be much happier
Hope this helps someone ... check your traffic between your RAS and NPS in finer detail!!