Strange timeouts connecting to registry.gitlab in CI/CD pipelines usually stem from DNS resolution failures, network proxy misconfigurations, or SSL certificate verification errors. By systematically verifying your Ubuntu system’s DNS settings, checking for interfering proxies, and ensuring proper authentication tokens, you can resolve these connectivity issues and restore smooth container image pulls.
Verifying DNS Resolution on Ubuntu
The most common cause of intermittent timeouts when accessing external registries like registry.gitlab.com is incorrect or slow DNS resolution. On Ubuntu systems, the Domain Name System (DNS) configuration is managed through the systemd-resolved service. If your server recently changed networks or if a local DNS cache has become corrupted, lookups for GitLab domains may fail or hang.
To diagnose this, first check the current status of the DNS resolver using the command systemctl status systemd-resolved. Ensure the service is active and running. Next, verify that your system can actually resolve the hostname by running dig registry.gitlab.com or nslookup registry.gitlab.com. If these commands return no answer or time out, your system cannot reach public DNS servers. In such cases, you may need to adjust your network interface configuration. For users migrating servers or dealing with complex network setups, reviewing Fixing DNS Issues When Moving Headless Ubuntu 22.04.5 LTS Server to a New Network provides essential context on how network changes impact resolution stability.
If the default resolvers are unreliable, consider configuring specific upstream DNS servers. Edit the /etc/systemd/resolved.conf file to set DNS=8.8.8.8 1.1.1.1 under the [Resolve] section. After saving the changes, restart the resolver with sudo systemctl restart systemd-resolved. This forces the system to use reliable public DNS providers, which often resolves timeout issues caused by local ISP DNS failures.
Checking Proxy Configuration Interference
Corporate environments or custom network setups often require HTTP/HTTPS proxies. If your Ubuntu environment has proxy variables set globally but they do not support HTTPS traffic to GitLab, Docker will fail to pull images with a timeout error. The CI/CD runner might be inheriting these environment variables incorrectly.
Inspect your current proxy settings by echoing the relevant environment variables: echo $http_proxy, echo $https_proxy, and echo $no_proxy. If these are set, ensure that registry.gitlab.com is included in the no_proxy list. This tells the system to bypass the proxy for GitLab traffic. A misconfigured proxy can intercept SSL handshakes, leading to silent failures or long timeouts before the connection is dropped.
In Docker configurations, proxy settings must also be defined explicitly. If you are using a custom Docker daemon setup, check the /etc/systemd/system/docker.service.d/http-proxy.conf file. Ensure that the proxy URLs are correct and that the daemon is reloading its configuration after any changes. Restarting the Docker service with sudo systemctl restart docker applies these new settings. If you are using a VPN to access internal resources, refer to Setting up VPN with Ubuntu using local DC as DNS server with Homelab to understand how VPN tunnels might route traffic unexpectedly, potentially blocking direct access to public registries.
Validating Authentication and Token Permissions
While authentication errors typically result in a 401 Unauthorized response, some network appliances or firewalls may drop packets silently instead of returning an error code, mimicking a timeout. This often happens when personal access tokens (PATs) have expired or lack the necessary scopes to read container repositories.
Ensure your CI/CD variables contain a valid Personal Access Token with the read_container_registry scope. You can test the token directly from the command line using curl -H "Authorization: Bearer YOUR_TOKEN" https://registry.gitlab.com/api/v4/projects. If this returns a JSON object, your authentication is working correctly at the API level. If it times out, the issue is likely network-related rather than credential-related.
Additionally, check if your GitLab instance uses a self-signed certificate or an internal CA. If so, the Docker daemon on your Ubuntu runner must trust this certificate. Copy the CA certificate to /etc/docker/certs.d/registry.gitlab.com/ca.crt and restart the Docker service. Without this step, TLS handshakes will fail, causing the connection to hang until the timeout limit is reached.
Docker Daemon and Storage Driver Issues
Sometimes the timeout is not about connectivity but about the Docker daemon’s ability to process requests efficiently. High disk I/O or storage driver conflicts can cause delays that appear as network timeouts.
Check the Docker daemon logs for errors related to storage drivers using journalctl -u docker.service. Look for messages indicating overlay filesystem issues or permission denied errors. If you see frequent I/O wait warnings, consider moving your Docker data directory to a faster storage device. Furthermore, ensure that your firewall rules on Ubuntu are not blocking outbound connections on port 443. Use ufw status to verify that outgoing HTTPS traffic is allowed. Blocking standard ports can lead to mysterious timeouts that are difficult to diagnose without inspecting low-level network traces.
Frequently Asked Questions
Why does my CI/CD pipeline work locally but fail in the runner?
This discrepancy usually indicates that the runner environment lacks the necessary proxy configurations or DNS settings present in your local machine. Check the runner’s environment variables and ensure they match your local network requirements. Additionally, verify that the runner has access to the same DNS servers as your local development environment.
How can I tell if the timeout is caused by DNS or the network?
Use the ping or dig commands from the runner host. If DNS resolution succeeds but the connection still times out, the issue is likely network-related, such as a firewall blocking port 443. If DNS fails entirely, focus on correcting your /etc/resolv.conf or systemd-resolved settings.
Does Docker caching affect timeout errors?
No, Docker caching stores images locally and does not influence network connectivity timeouts. However, if you are pulling a new tag frequently, caching helps reduce bandwidth usage. Timeouts occur during the initial download phase, regardless of whether layers are cached.
Can a self-signed certificate cause a timeout?
Yes, if the Docker daemon is configured to strictly verify TLS certificates and the server presents a self-signed cert not in its trust store, the handshake will fail. Depending on the client implementation, this may result in an immediate error or a timeout while waiting for a response that never comes. Installing the CA certificate resolves this.
What is the best way to debug persistent DNS issues on Ubuntu?
The most effective method is to use systemd-resolve --status to view detailed resolver information. Compare the active DNS servers with known good public resolvers like Google’s 8.8.8.8. If the active servers are unresponsive, manually update /etc/systemd/resolved.conf to point to reliable alternatives and restart the service.