To configure host-to-host IPSec authentication without Active Directory, you must first establish a local Certificate Authority using XCA, then generate two distinct certificates: one for the CA itself and another for each endpoint. These certificates enable mutual authentication between Windows machines by verifying identity through trusted root signatures rather than pre-shared keys.
Installing and Configuring XCA
XCA is a free, open-source certificate manager that simplifies the creation of private keys and certificates on Windows systems where no enterprise PKI exists. Begin by downloading the latest stable release from the official XCA website. During installation, ensure you select the option to install the necessary cryptographic service providers if prompted.
Once installed, launch XCA as an administrator. The first critical step is to create a new database file. Navigate to File > New Database. Save this file in a secure location, such as C:\XCA\MyCA.xca, because losing this file means losing your entire trust chain. You will be prompted to set a password for the database master key. Choose a strong password and store it safely; without it, you cannot export private keys or modify the database.
Creating the Root Certificate Authority
With the database open, you need to generate the Root CA certificate. Go to Keys > New Key. In the dialog box, provide a meaningful name like LocalRootCA. Ensure the algorithm is set to RSA with a minimum bit length of 2048, though 4096 is recommended for long-term security. Click OK to generate the key pair.
Next, create the certificate request. Select the newly created key and go to Certificates > New Certificate. Set the Subject Name to match your organization or hostname, for example, CN=Local Root CA. Crucially, under the Extensions tab, locate the Basic Constraints extension. Check the box for CA:TRUE. This setting is mandatory; without it, the certificate cannot sign other certificates, and IPSec peers will reject it as invalid. Set the Validity Period to a reasonable duration, such as 10 years, since this is a self-signed root. Sign the certificate using the key you just generated. Your Root CA is now ready.
Generating Endpoint Certificates for IPSec
You now need two client certificates, one for each host involved in the IPSec tunnel. Repeat the process of creating a new key for each machine. Name them descriptively, such as HostA_Key and HostB_Key.
For each key, create a new certificate request. In the Subject Name, use the specific hostname of the machine, e.g., CN=HostA. Under the Extensions tab, you must add the Enhanced Key Usage (EKU) extension. Add both Client Authentication and Server Authentication OIDs. This dual usage allows the certificate to be used for initiating or responding to IPSec connections. Without these EKUs, Windows may refuse to present the certificate during negotiation.
When signing these certificates, do not use the Root CA key directly in the same session if you want to simulate a real-world hierarchy. Instead, import the Root CA certificate into the "Trusted Root" section of XCA's database, then select it as the signer. This ensures the chain of trust is properly established within the tool. Export the resulting certificates as .cer files for the public part and .pfx files for the private parts, ensuring you protect the PFX files with passwords.
Exporting Keys Correctly
Right-click the Root CA certificate and select Export > Export Certificate. Save it as a Base-64 encoded X.509 (.CER) file. This file will be installed on all hosts as the trusted root.
For the endpoint certificates, right-click the certificate and choose Export > Export Private Key. Select the PFX format. Include the full certificate chain if possible, but at minimum, include the private key and set a robust export password. Send the CER file and the corresponding PFX file securely to their respective target machines.
Configuring Windows IPSec Policies
On each Windows host, open Group Policy Management and edit the Local Group Policy. Navigate to Computer Configuration > Windows Settings > Security Settings > IP Security Policies on Local Computer.
Create a new IP Security Policy. Assign rules that specify the remote peer's IP address. In the rule settings, navigate to the Methods tab. Select Use Internet Key Exchange (IKE) and click Properties. Under the Authentication Methods tab, remove "Pre-Shared Key" and select Certificate Authentication. Ensure "Negotiate Public Key" is checked.
Install the Root CA .CER file into the Trusted Root Certification Authorities store on both machines. Then, import the endpoint .PFX file into the Personal store of the local computer account. Finally, assign the policy to the network adapters. When connecting, Windows will automatically negotiate the IPSec tunnel using the certificates you created.
Troubleshooting Common Issues
If the connection fails, check the Event Viewer under Applications and Services Logs > Microsoft > Windows > IKEEXT. Look for errors indicating certificate validation failures. Common causes include mismatched hostnames in the certificate Subject vs. the actual machine name, or missing EKU extensions. Ensure the time synchronization between hosts is accurate, as certificate validity checks rely heavily on system time.
Frequently Asked Questions
Can I use the same certificate for both machines?
No, each machine must have its own unique private key and certificate. Sharing certificates compromises security and prevents proper identification of the endpoint during the IKE handshake.
What happens if the Root CA expires?
If the Root CA certificate expires, all subordinate certificates signed by it become invalid. You would need to generate a new Root CA, re-sign all endpoint certificates, and redistribute them to all hosts.
Do I need to install the certificate in the User store?
For host-to-host IPSec, the certificate must be installed in the Local Computer Personal store, not the Current User store. IPSec policies operate at the system level and require access to the computer's identity.
Why did my IPSec negotiation fail with error 1380?
Error 1380 typically indicates that the certificate presented does not meet the required criteria, often due to missing Extended Key Usage extensions. Verify that both Client and Server Auth OIDs are present in the certificate extensions.
Is XCA safe for production environments?
XCA is suitable for small-scale or lab environments where Active Directory is unavailable. For large enterprises, consider deploying an internal Enterprise PKI using Windows Server AD CS for centralized management and automation.