Share via

IPSec Troubleshooting

  • Article

Applies To: Windows Server 2003, Windows Server 2003 R2, Windows Server 2003 with SP1, Windows Server 2003 with SP2

Troubleshooting

What problem are you having?

  • Traffic is not being secured.

  • IPSec SA negotiation is failing.

  • IPSec SA negotiation is failing when using certificate authentication.

  • Some computers are unable to communicate with secure computers.

  • Local computer IPSec policy is not being used.

  • Unable to assign or unassign an IP Security policy.

  • DES is being used when 3DES is needed.

  • Unsecured traffic is being sent to a secure server.

  • Event Viewer is displaying error messages related to IPSec.

  • A policy location error message is displayed.

  • Event Viewer is displaying "Bad SPI" error message.

  • IP Security Monitor displays the error message "The IPSec server is unavailable or incompatible with the IPSec monitor."

Traffic is not being secured.

Cause:  An IPSec security association (SA) that secures traffic (also known as a hard SA) is not being established.

Solution:  This might be caused by either filter list misconfiguration or the enabling of unsecured communication with non-IPSec-aware computers. Examine the policies that are assigned to both IPSec peers and determine, based on the rules, filter lists, and filter action settings, whether IPSec requires traffic to be secured. Check to determine whether the Allow unsecured communication with non-IPSec-aware computer option on the Security Methods tab of the appropriate filter action is enabled. Use audit logging to determine the cause of the failed negotiation.

See also:  IPSec troubleshooting tools; Add, edit, or remove IPSec security methods

IPSec SA negotiation is failing.

Cause: An IPSec security association (SA) that secures traffic (also known as a hard SA) is not being established. If unsecured communication with non-IPSec-aware computers is enabled, the negotiation failure is expected.

Negotiation failure between two IPSec-aware computers commonly occurs when the default policies are used on two IPSec peers and one or both are not members of an Active Directory domain. Computers running Windows XP Home Edition do not support Active Directory domain membership or the Kerberos V5 authentication method. Default policies use the Kerberos V5 authentication method, which does not work for computers that are not Active Directory domain members.

Solution:  A negotiation failure between two IPSec-aware computers is most often caused by a mismatched key exchange method, authentication method, or security method. Examine the policies that are assigned to both IPSec peers and determine, based on the advanced settings, rules, filter lists, and filter action settings, whether IPSec should be securing the traffic. If it should, determine whether there is a mismatched key exchange, authentication method, or security method that is preventing a successful negotiation of a hard SA. Use audit logging to determine the cause of the failed negotiation.

See also:  IPSec troubleshooting tools; Add, edit, or remove IPSec security methods; Define IPSec authentication methods; Create key exchange security methods

IPSec SA negotiation is failing when using certificate authentication.

Cause:  The correct computer certificates are not installed on the IPSec peers.

Solution:  Install the correct computer certificates on the IPSec peers.

Cause:  An incorrect certification authority (CA) is selected in the authentication method.

Solution:  For the rules of each IPSec peer, select the CA that issued the computer certificate that is installed on the other IPSec peer.

Cause:  The certificate chain from the issuing CA to the root CA is broken.

Solution:  Install a computer certificate on each IPSec peer that follows a valid certificate chain up to a trusted root CA.

Cause:  The computer certificate on one or both of the IPSec peers has expired.

Solution:  Install a current computer certificate on each IPSec peer.

See also:  IPSec troubleshooting tools; Authentication methods; Define IPSec authentication methods; Special IPSec considerations; Using Certificates

Some computers are unable to communicate with secure computers.

Cause:  Secure computers are using a lockdown policy that requires client computers to initiate secure communications. The computers that are unable to communicate are not configured with a rule to initiate secure communication.

Solution:  Either allow unsecured incoming connection requests on the secure servers or configure additional rules for the computers that are unable to communicate so that they can initiate secure communication with the secure computers.

Cause:  Client computers are not IPSec-capable and secure computers are not allowing unsecured communications.

Solution:  Either install client computers that are IPSec-capable or reconfigure the secure computers to allow communication with computers that are not IPSec-capable.

Cause:  Some computers are configured with a policy that has a key exchange, authentication method, or security method that is mismatched with the policy configured for the secure computers.

Solution:  Examine the policies that are assigned to both IPSec peers and determine, based on the advanced settings, rules, filter lists, and filter action settings, whether IPSec should be securing the traffic. If it should, determine whether there is a mismatched key exchange, authentication method, or security method that is preventing a successful SA negotiation. Use audit logging to determine the cause of the failed negotiation.

See also:  Add, edit, or remove filter actions; IPSec troubleshooting tools; Add, edit, or remove IPSec security methods; Define IPSec authentication methods; Create key exchange security methods

Local computer IPSec policy is not being used.

Cause:  The computer is a member of an Active Directory domain and is receiving Active Directory-based IPSec policy.

Solution:  All domain members receive Active Directory-based IPSec policy, which overrides local IPSec policy settings. To prevent this, remove the computer from the domain.

Cause:  The IPSec service is not started.

Solution:  Start or restart the IPSec service.

See also:  Restart the IPSec services

Unable to assign or unassign an IP Security policy.

Cause:  You cannot assign or unassign policies when IP Security Policies are used for Active Directory.

Solution:  To assign Active Directory-based policies, use IP Security Policies within Group Policy.

See also:  Assign or unassign IPSec policy in Group Policy

DES is being used when 3DES is needed.

Cause:  The IPSec peer is a computer running Windows 2000 without either the Windows 2000 High Encryption Pack or Windows 2000 Service Pack 2 (or later) installed.

Solution:  Install either the Windows 2000 High Encryption Pack or Windows 2000 Service Pack 2 (or later) on the computer running Windows 2000.

Unsecured traffic is being sent to a secure server.

Cause:  The client computer is using the default response rule.

Solution:  When the client computer is configured with the default response rule, it sends unsecured packets when it either initiates communication or restarts communication after a delay long enough for the quick mode SA and dynamic filter list to time out. To prevent this, configure additional rules for client computers that always initiate secure communication to secure servers.

See also:  Security information for IPSec

Cause:  There might be a problem with either the IPSec Policy Agent or the Oakley service.

Solution:  View the system and security event logs for messages that might help in determining the origin of the problem.

A policy location error message is displayed.

Cause:  This error might occur if the IPSec Policy Agent is attempting to retrieve policy from an incorrect storage location.

Solution:  Examine both the IPSec Policy Agent events in Event Viewer and the IPSec Policy Agent log for messages that indicate from what location policy is being retrieved (the local registry or the domain). If HKLM\SOFTWARE\Policies\Microsoft\Windows\IPSec\GPTIPSecPolicy\DSIPSecPolicyFlags is set to 1, then IPSec policy is obtained from Active Directory. Otherwise, it is obtained from the registry. The local computer warns a user that domain policy is in effect when it starts. If the policy is supposed to be Active Directory-based, but cannot be obtained, you can delete HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Group Policy\History and then restart the computer.

Caution

  • Incorrectly editing the registry may severely damage your system. Before making changes to the registry, you should back up any valued data on the computer.

See also:  View Event Logs.

Event Viewer is displaying "Bad SPI" error message.

Cause:  This error might occur if a key lifetime value is set too low, or if the SA has expired but the sender continues to transmit data to the receiver.

Solution:   To determine and correct the problem, use IP Security Monitor to examine the number of rekeys. If the number of rekeys is very large compared to the amount of time the connections have been active, set the key lifetimes in the policy to be longer. Good values for high-traffic Ethernet connections are greater than both 50 megabytes and five minutes. This event is usually logged when an SA has been deleted on one IPSec peer, and the other IPSec peer sends secure traffic on the SA. This is usually a temporary situation that occurs on systems that are busy and fixes itself.

IP Security Monitor displays the error message "The IPSec server is unavailable or incompatible with the IPSec monitor."

Cause:  You are using the IP Security Monitor console to monitor IPSec on a computer running Windows 2000.

Solution:  The IP Security Monitor console can only be used to monitor IPSec on computers running Windows XP or a Windows Server 2003 operating system. To monitor IPSec on a computer running Windows 2000, use the IPSecmon command at the Windows 2000 command prompt on the computer that is being monitored.

See also:  IPSec troubleshooting tools