Use curl -w with the %{url_effective} format specifier to capture the final URL after redirects. This approach allows you to distinguish between the requested endpoint and the actual destination, providing transparency into HTTP redirections within Bash scripts.
The Problem with Redirects
When using curl in a script, it is common to encounter situations where the server responds with a 3xx status code, indicating that the resource has moved. By default, curl follows these redirects automatically if the -L flag is used, or it may display headers without following them if omitted. In many automation scenarios, knowing the exact URL that was ultimately accessed is critical for logging, debugging, or conditional logic. The standard output of curl typically includes the response body and headers, but it does not explicitly print the final resolved URL unless specifically configured to do so.
This lack of explicit output can make troubleshooting difficult. For instance, if a script fails because it expects data from one domain but receives a redirect to another, the error message might only show the initial request URL. Understanding this discrepancy requires capturing the effective URL separately from the main response content. This is particularly relevant when dealing with complex web architectures involving load balancers, CDNs, or authentication proxies that alter the path dynamically.
Using the Write-out Format
The solution lies in curl's write-out feature, controlled by the -w or --write-out option. This option accepts a string containing special variables that curl expands and writes to standard output at the end of the transfer. One of the most useful variables for this specific use case is %{url_effective}. This variable contains the last URL used after any redirects have been followed. It effectively tells you exactly which address the server finally served the content from.
To implement this, you append the format string to your curl command. A basic structure looks like this:
curl -w "%{url_effective}\n" -o /dev/null https://example.comIn this example, the -o /dev/null argument discards the actual response body, which is often unnecessary if you only care about the URL. The -w flag ensures that the effective URL is printed to the terminal. The \n adds a newline character for clean formatting. This method is non-intrusive and works across all versions of curl that support the write-out feature, which is virtually every modern Linux distribution.
Integrating with Bash Variables
Once you have the output formatted correctly, the next step is to capture it within a Bash script for further processing. You can achieve this using command substitution, which allows you to store the result of the curl command in a variable. This makes the URL available for comparison, logging, or passing to other commands.
RESPONSE=$(curl -s -w "\n%{url_effective}" https://api.example.com/data)
LAST_LINE=$(echo "$RESPONSE" | tail -n 1)
URL_EFFECTIVE=$(echo "$LAST_LINE" | grep -v "^$" | tail -n 1)However, a cleaner approach is to separate the response body and the write-out data entirely. By using -o /dev/null, you ensure that the stdout contains only the write-out string. This simplifies parsing significantly.
FINAL_URL=$(curl -s -w "%{url_effective}\n" -o /dev/null https://api.example.com/data)
echo "The request resolved to: $FINAL_URL"This pattern is robust and easy to read. It isolates the metadata (the URL) from the payload (the data), preventing issues where the response body might contain text that looks like a URL or interferes with the write-out string. If you need both the body and the URL, you can use -o to save the body to a file and still capture the URL via -w.
Handling Errors and Edge Cases
While %{url_effective} is powerful, it behaves differently depending on the success of the request. If the connection fails completely before any HTTP transaction occurs, %{url_effective} will be empty. Additionally, if the server returns an error status (4xx or 5xx) but still provides a valid URL, the variable will contain that URL. It is important to check the exit code of curl to determine if the request succeeded.
You can combine the URL capture with error handling to create a resilient script snippet. For example, checking if the URL is empty can indicate a network failure rather than a redirect issue. This distinction helps in diagnosing whether the problem is with the DNS resolution, the network connectivity, or the application logic itself.
For more advanced scripting needs, such as managing background processes or handling interrupts, consider reviewing related techniques for signal management. See Troubleshooting SIGINT in Bash Scripts for insights on keeping your scripts stable during long-running operations.
Advanced Formatting Options
Beyond just the URL, curl allows you to include multiple pieces of information in the write-out string. You can add timing data, HTTP codes, and content lengths simultaneously. This is useful for performance monitoring tools integrated into your Bash workflows.
curl -w "HTTP Code: %{http_code}\nTime Total: %{time_total}s\nURL: %{url_effective}\n" https://example.comThis single command provides a comprehensive summary of the request. When combined with Bash's ability to parse multi-line output, you can extract specific metrics for logging systems. Remember to escape special characters in your format string if you are embedding it within larger heredocs or complex variable expansions.
Frequently Asked Questions
Why is my url_effective variable empty?
An empty url_effective usually indicates that the connection failed before any HTTP request was sent, such as a DNS lookup failure or a timeout. Ensure that the network is reachable and the domain name is correct before assuming the URL was resolved.
Can I get the original URL instead of the effective one?
No, curl does not provide a built-in variable for the originally requested URL in the write-out format. However, you can simply store the input URL in a Bash variable before running curl and compare it with %{url_effective} to detect redirects.
How do I suppress the response body while keeping the URL?
Use the -o /dev/null option to discard the response body. This directs the actual content to nowhere, leaving only the output generated by the -w flag visible on the terminal or captured in a variable.
Does this work with HTTPS certificates?
Yes, %{url_effective} works regardless of whether the connection uses HTTP or HTTPS. It reflects the final URL after any redirects, including those that change the protocol, provided the connection is established successfully.
What if I want to log this to a file instead of printing it?
You can redirect the entire curl command's output to a file using shell redirection operators like > or >>. Since -o /dev/null handles the body, the file will contain only the formatted write-out string, making it easy to append logs.