When the root DNSSEC key signing key (KSK) is rolled over, systemd-resolved can start returning SERVFAIL with a log line like DNSSEC validation failed for question . IN DNSKEY: missing-key — even on machines that shipped the new key months earlier. The cause is that systemd-resolved carries its own compiled-in root trust anchor and, unlike most full recursive resolvers, it never performs automatic RFC 5011-style trust anchor updates. The fix is to point the resolver at the updated key file shipped by your distribution via a .positive trust anchor file.
Why the resolver fails despite having the new key on disk
Distributions such as Ubuntu ship the current root zone trust anchor in the dns-root-data package, typically as a DS record set at /usr/share/dns/root.ds. This file was updated well in advance of the KSK-2017 to KSK-2024 rollover precisely so that resolvers could pick it up. However, systemd-resolved does not read that file automatically. The resolved.conf(5) man page documents that the trust anchor for the root domain is built into the resolver binary itself, and additional anchors must be configured explicitly through dnssec-trust-anchors.d(5).
Furthermore, the documentation is explicit that systemd-resolved does not update its trust anchor database from DNS servers at runtime. It only removes anchors when they are detected as revoked. So even a machine that has been continuously online since the new key was published will never learn the new KSK on its own. When the rollover completes and the old key is removed from the root zone's DNSKEY RRset, the resolver can no longer validate the chain of trust, treats the root DNSKEY query as having a missing key, and returns SERVFAIL for everything it tries to validate.
This is a known design trade-off: a minimal stub resolver avoids the complexity of RFC 5011 state management, but that shifts the burden of keeping the trust anchor current onto the distribution packaging — and onto you if the wiring between the two is missing.
Diagnosing the problem
You can confirm this failure mode quickly:
- Check the journal for the
missing-keymessage:journalctl -u systemd-resolved | grep -i dnssec. - Test validation directly with
resolvectl query example.comand observe the SERVFAIL. - Verify whether the updated key is present on disk:
cat /usr/share/dns/root.ds. If it contains the new KSK-2024 DS record (key tag 38696) but the resolver still fails, the file is simply not being consumed.
Emergency workaround
To restore DNS immediately, drop a trust anchor file under /run/dnssec-trust-anchors.d/. Files there are volatile and disappear on reboot, which makes this a safe stopgap:
sudo mkdir -p /run/dnssec-trust-anchors.d
echo '. IN DS 38696 8 2 683D2D0ACB8C9B712A1948B27F741219298D0A450D612C483AF444A4C0FB2B16' | \
sudo tee /run/dnssec-trust-anchors.d/root.positive
sudo systemctl restart systemd-resolved
Configuring any explicit root trust anchor disables the built-in one, so once this file is loaded, validation should succeed again immediately. Alternatively, copying the distribution's own file works too, as long as its format matches what the .positive loader expects.
Permanent fix
For a lasting solution, link the package-provided trust anchor into one of the persistent directories that systemd-resolved scans — /etc/dnssec-trust-anchors.d/, /run/dnssec-trust-anchors.d/, and /usr/lib/dnssec-trust-anchors.d/, searched in that order. A symlink in /etc is the cleanest choice because it survives reboots and automatically tracks future updates to the package:
sudo apt update && sudo apt install -y dns-root-data
sudo mkdir -p /etc/dnssec-trust-anchors.d
sudo ln -sf /usr/share/dns/root.ds /etc/dnssec-trust-anchors.d/root.positive
sudo systemctl restart systemd-resolved
Because the symlink points at the package-managed file, the next time the root key rolls over and dns-root-data is updated, your resolver picks up the new anchor without any manual intervention — provided the package keeps the file at the same path and in a compatible format.
Prevention and alternatives
- Monitor rollover announcements. Root KSK rollovers are announced years in advance by IANA. Put the rollover date on your calendar if you manage machines running systemd-resolved.
- Consider a full recursive resolver. Software such as Unbound or BIND implements RFC 5011 automatic trust anchor rollover and would have transitioned to KSK-2024 on its own. Pointing
systemd-resolvedat it (or replacing it) removes this class of failure entirely. - Verify after package updates. After upgrading
dns-root-data, runresolvectl query example.comand confirm the answer comes back with DNSSEC validation succeeding. - Watch for distribution fixes. Some distributions have been wiring this symlink into their packaging automatically; if your release adds it, the manual symlink can be removed to avoid conflicts.
Frequently Asked Questions
Why did being online since the new key was published not help?
Because systemd-resolved does not implement RFC 5011. It never queries DNS to refresh its trust anchors at runtime; it only drops anchors it detects as revoked. The new key on disk was irrelevant because the resolver never looked at it.
Does adding a .positive file for the root disable the built-in anchor?
Yes. The documentation states that configuring an explicit root trust anchor disables the built-in one. That is exactly what you want during a rollover — you are replacing the stale compiled-in key with the current one.
Where does systemd-resolved look for trust anchor files?
In /etc/dnssec-trust-anchors.d/, /run/dnssec-trust-anchors.d/, and /usr/lib/dnssec-trust-anchors.d/, in that order. Files must carry the .positive (or .negative) suffix, and a same-named file in an earlier directory overrides one in a later directory.
Is the /run workaround safe to keep long-term?
No. Files under /run are wiped on every boot, so your fix would silently vanish after a restart and validation would break again. Use the /etc symlink for anything persistent.
What is the DS record in the workaround actually doing?
It is the SHA-256 digest of the KSK-2024 root zone key (key tag 38696). By presenting it as a trust anchor, you tell the resolver to accept the root zone's DNSKEY RRset when it hashes to that digest, restoring the chain of trust.
Will this happen again at the next rollover?
Not if you use the symlink to the package-managed file, since dns-root-data is updated ahead of scheduled rollovers. If you stick with the built-in anchor and no explicit file, then yes — the same failure will recur whenever the root KSK changes again.