LetsEncrypt Gives 403/404 Error During Certificate Renewal on New Server

Let's Encrypt certificate renewal fails with 403 Forbidden or 404 Not Found errors when the web server cannot properly serve the ACME challenge files. This typically happens because the .well-known/acme-challenge directory is blocked by firewall rules, missing in the document root, or restricted by access control configurations on the new server.

Understanding the ACME Challenge Mechanism

To issue or renew a certificate, Let's Encrypt must verify that you control the domain. The standard method for this is the HTTP-01 challenge. During this process, the Let's Encrypt validation server attempts to download a specific file from your website at http://yourdomain.com/.well-known/acme-challenge/<token>. If the server returns a 403 error, it means the request reached the server but was denied permission. A 404 error indicates that the server could not find the requested resource at all. When migrating to a new server, these errors often stem from configuration mismatches between the old and new environments. The new server might not have the correct directory structure, or the web server software (such as Nginx, Apache, or IIS) may be configured differently than before. It is crucial to ensure that the path where the ACME client places the challenge file matches exactly where the web server expects to serve it.

Checking Web Server Configuration for .well-known Access

The most common cause of this issue is an incorrect configuration in the web server settings. You need to ensure that requests to the .well-known directory are allowed and served correctly. For Nginx servers, check your site configuration file (usually located in /etc/nginx/sites-available/). Ensure there is a location block that allows access to the challenge directory:

location ~ /\.well-known\/acme-challenge\/ {
    allow all;
    root /var/www/html;
}

Replace /var/www/html with your actual document root. If you have a generic catch-all location block that denies access to hidden files (starting with a dot), you must explicitly exempt .well-known from this rule. For Apache servers, check your .htaccess file or virtual host configuration. Ensure that AllowOverride is enabled and that no Deny from all directives are blocking the .well-known folder. You can add the following to your configuration:

<Directory "/var/www/html/.well-known">
    AllowOverride None
    Require all granted
</Directory>

If you are using IIS on Windows, ensure that the "Request Filtering" module does not block hidden segments. Go to the IIS Manager, select your site, double-click "Request Filtering," go to the "Hidden Segments" tab, and remove .well-known if it is listed.

Verifying Directory Structure and Permissions

Even with correct server configuration, the physical directory must exist and be accessible. Run the following commands to create the necessary directory and set proper permissions:

sudo mkdir -p /var/www/html/.well-known/acme-challenge
sudo chown -R www-data:www-data /var/www/html/.well-known
sudo chmod -R 755 /var/www/html/.well-known

Note that the user www-data is typical for Debian/Ubuntu systems. For CentOS/RHEL, the user might be apache or nginx. Adjust the ownership command accordingly. Incorrect permissions can lead to 403 errors because the web server process lacks read access to the challenge file.

Testing the Challenge Endpoint Manually

Before attempting another renewal, test if the endpoint is reachable. Use curl to simulate what Let's Encrypt does:

curl -I http://yourdomain.com/.well-known/acme-challenge/test

Create a dummy file named test inside the acme-challenge directory with some content. If you receive a 200 OK status code, the server is configured correctly. If you get a 403 or 404, revisit the web server configuration steps above.

Troubleshooting Firewall and Reverse Proxy Issues

Sometimes the web server is configured correctly, but external tools block the request. If you are using a reverse proxy like Cloudflare, ensure that SSL/TLS mode is set to "Flexible" or "Full" rather than "Strict" during the renewal process if the origin server doesn't have a valid cert yet. Also, check that your firewall allows inbound traffic on port 80 (HTTP). Port 443 (HTTPS) is not used for the HTTP-01 challenge. If you are using a load balancer, ensure it forwards requests for .well-known directly to the backend server without stripping the path. Some load balancers require specific routing rules to pass the query parameters and paths correctly.

Automating Renewal Checks

Once you resolve the immediate error, configure your ACME client (like Certbot) to handle future renewals gracefully. Enable the dry-run option to test the renewal process without actually changing certificates:

certbot renew --dry-run

This command simulates the renewal and will alert you if any configuration issues persist. It is a safe way to verify that your setup works before relying on automatic cron jobs for production certificates.

Frequently Asked Questions

Why do I get a 403 error only on the new server?

New servers often have stricter default security settings or different web server versions. The previous server might have had loose permissions or a different configuration that inadvertently allowed access to hidden directories. Always audit the new server's security policies.

Can I use the DNS-01 challenge instead?

Yes, if HTTP challenges continue to fail due to complex infrastructure, switch to DNS-01. This method requires updating your domain's DNS records via an API, which bypasses web server configuration entirely. Certbot supports this with plugins for major DNS providers.

Does Cloudflare interfere with Let's Encrypt renewal?

Cloudflare can interfere if its caching or proxy settings hide the real server response. Ensure that the ".well-known" path is excluded from Cloudflare's cache and that the proxy status is set to "DNS Only" for this specific path during renewal.

What should I do if the .well-known directory is missing?

Create the directory manually as shown in the troubleshooting section. Most ACME clients do not create this directory automatically if it does not exist. Ensure the parent directory has the correct ownership so the web server can write the challenge file into it.

How do I check if my firewall is blocking port 80?

Use the command sudo ufw status on Ubuntu or sudo iptables -L on CentOS to view active rules. Ensure that incoming traffic on port 80 is allowed. You can also test externally using a tool like telnet yourdomain.com 80 to verify connectivity.