Understanding the Limitations of XCA Templates
XCA is a powerful tool for managing certificates, but it does not include a dedicated "IPSec Server" or "IPSec Client" certificate type in its default menu options. Attempting to select standard SSL Server or Client templates will result in authentication failures because the resulting certificates lack the necessary Extended Key Usage flags that Windows expects for IPsec negotiations. The key difference lies in the OID (Object Identifier) values assigned to the EKU extension. For IPSec, the system requires the Server Authentication OID (1.3.6.1.5.5.7.3.2) and the Client Authentication OID (1.3.6.1.5.5.7.3.1). You must manually inject these OIDs into the certificate request to ensure compatibility with Windows IPsec policies.Step 1: Creating the Local Certificate Authority
The first step is to generate a root CA that will sign your host certificates. Open XCA and navigate to Database > New Database. Save this database file securely, as it contains your private keys. Next, go to Keys > New Key to generate a RSA key pair for the CA. Set a strong password for the private key protection. Once the key is generated, move to Certificates > New Certificate. Select the newly created key and choose the Root CA template. Ensure the validity period is set appropriately, typically several years, since changing the root CA requires reissuing all subordinate certificates. When prompted for the Distinguished Name, enter a clear Common Name such as "MyLocalIPSecCA". Sign the certificate using the CA key. Finally, export the CA certificate by right-clicking it in the list and selecting Export > Base64 encoded. Save this asrootca.cer so you can install it on both target hosts later.
Step 2: Generating the Server Certificate with Correct EKU
For the host acting as the IPsec initiator or responder, you need a certificate configured for Server Authentication. In XCA, create a new key for the server host. Then, create a new certificate request using this key. Instead of choosing a standard template, select Custom or manually edit the attributes. Crucially, you must add the Extended Key Usage extension. Add the OID1.3.6.1.5.5.7.3.2 which corresponds to Server Authentication. You may also optionally add 1.3.6.1.5.5.7.3.1 for Client Authentication if you want bidirectional trust, but strictly speaking, Server Auth is required for the endpoint responding to connections. Submit this request to your local CA within XCA (using Sign Certificate) or export it as a CSR and sign it externally. Export the signed certificate as Base64 encoded (server.cer) and the private key separately.
Step 3: Generating the Client Certificate with Correct EKU
Repeat the process for the second host, which acts as the client or secondary peer. Create a new key and a new certificate request. This time, ensure the EKU extension includes the OID1.3.6.1.5.5.7.3.1 for Client Authentication. If you are using symmetric IPSec policies where both sides authenticate each other, you might include both OIDs here as well. Sign this certificate with your local CA. Export the signed certificate (client.cer) and its corresponding private key. Now you have two sets of credentials: one for the server side and one for the client side, both trusted by the same local CA.
Installing Certificates and Configuring Windows IPsec
On each Windows host, import therootca.cer into the Trusted Root Certification Authorities store under the Local Computer account. This ensures the OS trusts your custom CA. Next, import the respective host certificate (with its private key) into the Personal store. To configure IPsec, open Windows Defender Firewall with Advanced Security. Right-click Connection Security Rules and select New Rule. Choose Custom and proceed through the wizard. Under Authentication Method, select Use computer certificate and ensure the correct certificate is selected for each host. Apply the rule to the appropriate network profiles. Test the connection by pinging the remote host; if configured correctly, the initial ping may fail due to policy negotiation, but subsequent traffic should be encrypted.
Troubleshooting Common Issues
If authentication fails, check the Event Viewer under Applications and Services Logs > Microsoft > Windows > IPsecPolicyManager. Look for errors indicating certificate validation failures. Common causes include incorrect EKU OIDs, missing CA chain in the trust store, or firewall blocking UDP ports 500 and 4500. Ensure that the date and time on both hosts are synchronized, as certificate validation relies heavily on accurate timestamps.Frequently Asked Questions
Why does my IPSec tunnel drop immediately after establishment?
This usually indicates a mismatch in Phase 1 and Phase 2 proposals or an issue with the encryption algorithms. Verify that both hosts use identical ESP/AH settings in their IPsec policies. Additionally, check if NAT traversal is enabled if either host is behind a router.
Can I use self-signed certificates without a CA?
No, Windows IPsec requires a trusted certificate chain for certificate-based authentication. Self-signed certificates are not trusted by default unless manually added to the Trusted People store, which is less secure and harder to manage than using a local CA.
What happens if I forget the private key password in XCA?
If you lose the password for the private key associated with your certificate, the certificate becomes unusable. You cannot recover the key from the certificate itself. You must generate a new key pair and request a new certificate from your CA.
Do I need to renew the CA certificate often?
The CA certificate should have a long validity period, typically 5-10 years, to avoid frequent re-issuance of all host certificates. However, best practices suggest rotating the CA key periodically for enhanced security, though this requires updating trust on all connected hosts.
How do I verify the EKU OIDs are correct?
You can view the details of any exported certificate using the Windows Certificate Viewer. Double-click the .cer file, go to the Details tab, and scroll to Extended Key Usage. Confirm that the listed OIDs match 1.3.6.1.5.5.7.3.2 for servers and 1.3.6.1.5.5.7.3.1 for clients.