To retrieve the original filename from a download URL using
wget or
cURL, you must rely on HTTP headers rather than parsing the URL itself. The most reliable method is extracting the value of the
Content-Disposition header, which explicitly specifies the name the server intends for the file. If this header is absent, you can fall back to deriving the name from the last segment of the URL path after removing query parameters.
Understanding Content-Disposition Headers
When a web server sends a file, it often includes metadata in the response headers that tells the client how to handle the content. One of these headers is
Content-Disposition. This header can take two primary forms:
inline, suggesting the browser should display the content directly, and
attachment, which prompts the user to save the file. When used with
attachment, it almost always contains a
filename parameter that defines the exact name to be used when saving.
For example, a typical header might look like this:
Content-Disposition: attachment; filename="report_2023.pdf"
This mechanism is robust because it allows the server to rename files dynamically without changing the URL structure. For instance, multiple users downloading the same report might receive different filenames based on their session data, yet the URL remains constant. Therefore, relying on the URL path alone is insufficient for accurate automation.
Extracting Filename with cURL
The
cURL utility provides powerful options for inspecting and manipulating HTTP responses. To catch the filename, you need to request only the headers without downloading the entire body of the file. This is achieved using the
-I (or
--head) flag. By combining this with
-s for silent mode, you prevent progress meters from cluttering your output, making it easier to parse programmatically.
You can pipe the output through
grep to find the specific line containing the disposition information. However, simply grepping for "filename" might yield false positives if other headers contain similar text. A more precise approach involves using regular expressions to extract the value between quotes or equals signs.
Here is a robust shell snippet to extract the filename using
cURL:
url="https://example.com/download.php?id=123"
filename=$(curl -sI "$url" | grep -i 'content-disposition' | sed 's/.*filename[^=]*=\(.*\)/\1/' | sed 's/^[""\']//;s/[""\']$//')
echo "Filename: $filename"
In this command,
sed performs two critical steps. First, it isolates the part of the string after
filename=. Second, it strips any surrounding quotation marks that are commonly present in MIME headers. This ensures that the variable
$filename contains a clean string ready for use in subsequent script commands.
Extracting Filename with wget
While
wget is excellent for downloading files, it does not have a direct single-command equivalent to
cURL's
-I flag for just fetching headers in all versions. However, you can achieve similar results by running
wget with the
--server-response option (often abbreviated as
-S). This prints the remote server's response to standard error.
To capture this output into a variable, you must redirect standard error to standard output using
2>&1. Then, you can process the stream exactly as you would with
cURL.
url="https://example.com/download.php?id=456"
headers=$(wget --server-response -O /dev/null "$url" 2>&1)
filename=$(echo "$headers" | grep -i 'content-disposition' | sed 's/.*filename[^=]*=\(.*\)/\1/' | sed 's/^[""\']//;s/[""\']$//')
echo "Filename: $filename"
Note that we use
-O /dev/null to ensure no actual file data is written to disk during this inspection phase. This makes the operation fast and efficient. It is important to note that
wget behavior can vary slightly across versions, so testing this snippet in your specific environment is recommended.
Fallback Strategies
Not all servers provide the
Content-Disposition header correctly. Some may omit it entirely, while others might use non-standard formatting. In such cases, you need a fallback strategy. The most common alternative is to parse the URL itself. You can extract the last component of the URL path, assuming it represents the filename.
However, URLs often include query strings (e.g.,
?v=1.2) or trailing slashes that complicate this extraction. You should remove query parameters first using tools like
awk or
sed, then extract the basename. If both the header and the URL parsing fail, consider checking the
Content-Type header to infer the file extension, although this will not give you the base name.
For more advanced scenarios involving complex redirection chains, you might need to follow redirects manually to see the final destination URL before applying these extraction techniques. This is particularly relevant when dealing with services that use short links or API endpoints that redirect to cloud storage providers like AWS S3 or Azure Blob Storage.
Handling Special Characters and Spaces
Filenames often contain spaces, special characters, or even unicode characters. When capturing these values in a shell script, proper quoting is essential to prevent word splitting or globbing issues. Always enclose your variables in double quotes when passing them to other commands.
Additionally, some servers encode filenames in the header using percent-encoding (URL encoding). For example, a space might appear as
%20. If your downstream processes require a human-readable name, you may need to decode these sequences. Tools like
printf or specialized utilities like
python3 -c can help decode these strings if necessary.
It is also worth noting that some servers provide both a
filename* parameter (RFC 5987 encoded) and a legacy
filename parameter. Modern browsers prefer
filename*, but older scripts might only check for
filename. Ensure your parsing logic checks for both variants to maximize compatibility.
Frequently Asked Questions
What if the Content-Disposition header is missing?
If the header is missing, you must parse the URL path as a fallback. Extract the last segment of the URL after the final slash, ignoring any query parameters starting with a question mark. This method is less reliable because servers do not guarantee the URL path matches the desired filename.
How do I handle filenames with spaces in bash scripts?
Always wrap your extracted filename variable in double quotes whenever you use it in commands. For example, use
"$filename" instead of
$filename. This prevents the shell from splitting the string at spaces and treating each part as a separate argument.
Can I use wget to download and rename automatically?
Yes,
wget has an
-O option that allows you to specify the output filename directly. However, this requires you to know the filename beforehand or extract it separately first. You cannot combine the download and automatic renaming into a single atomic step without prior knowledge of the target name.
Why does my regex fail on some servers?
Regex failures usually occur due to variations in header formatting. Some servers put the filename in quotes, others do not. Some use
= while others use
= followed by a space. Make sure your
sed or
grep patterns account for optional whitespace and quote characters around the filename value.
Is there a difference between curl -I and curl -L?
-I fetches only headers, while
-L follows redirects. If the download URL redirects to another location that sets the
Content-Disposition header, you must use
-L combined with
-I (i.e.,
curl -sIL) to see the headers of the final destination. Without following redirects, you might miss the correct filename provided by the final server.