WSL wget Command Downloads HTML Instead of CSV

Understanding the User-Agent Mismatch

When you run wget from the WSL terminal, it sends a default HTTP header identifying itself as "Wget/1.21" (or similar version). Many modern websites, especially those serving dynamic content or protected data like CSV exports, check this header. If the request does not look like it comes from a legitimate browser such as Chrome, Firefox, or Edge, the server responds with an HTML page containing an error message, a login form, or a CAPTCHA challenge rather than the requested resource.

This issue frequently confuses users who are accustomed to using curl or wget on native Linux servers where such restrictions might be less common or configured differently. It is important to note that this is unrelated to network connectivity issues, which are covered in our guide on Connection Refused to WSL(Linux) Host. Instead, this is purely an application-layer protocol negotiation problem between your client and the web server.

Solution 1: Spoofing the User-Agent String

The most effective and immediate fix is to modify the HTTP headers sent by wget to include a User-Agent string that resembles a popular web browser. This tricks the server into thinking the request is coming from a human user browsing the site normally.

To do this, use the --user-agent flag followed by a string commonly recognized by web servers. For example, a generic Chrome user-agent string works well for most cases:

wget --user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" https://example.com/data.csv -O output.csv

Here, the -O output.csv flag ensures that the downloaded content is saved to a file named output.csv regardless of the filename suggested by the server. Without this flag, wget might name the file based on the URL path, which could lead to confusion if the server redirects the request.

Solution 2: Using curl as an Alternative

If wget continues to present difficulties, switching to curl is a viable alternative. curl is pre-installed in most WSL distributions and offers robust header manipulation capabilities. You can achieve the same result by passing the -A (or --user-agent) flag:

curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" -o output.csv https://example.com/data.csv

This approach is particularly useful if you are already familiar with curl syntax or if you need to add additional headers such as cookies or authentication tokens, which are often required when downloading protected CSV files.

Verifying the Downloaded Content

After executing the command with the spoofed user-agent, it is crucial to verify that the downloaded file is indeed a CSV and not an HTML error page. You can do this by checking the file size and content type.

Use the file command to inspect the file type:

file output.csv

If the output indicates "ASCII text" or "CSV", the download was successful. If it says "HTML document" or "SGML document", the server still rejected the request. In such cases, try a different user-agent string, perhaps one mimicking Firefox or Edge, as some servers have specific whitelists of allowed browsers.

Common Pitfalls and Troubleshooting

Even with the correct user-agent, you may encounter issues if the website requires session cookies or JavaScript rendering. wget and curl do not execute JavaScript, so if the CSV link is generated dynamically via a script after login, these tools will fail. In such scenarios, you must first log in via a browser, export the cookies, and then pass them to wget using the --load-cookies option.

Additionally, ensure that your WSL environment has the necessary permissions to write to the target directory. While rare, permission errors can sometimes manifest as unexpected behaviors if the system attempts to write to a read-only location. For more information on general WSL configuration and installation problems, refer to our article on Troubleshooting: Windows Subsystem for Linux (WSL) Installation Issues.

Frequently Asked Questions

Why does wget return an HTML file instead of the expected data?

This happens because the web server identifies the default wget user-agent as a bot or scraper and serves an error page or login form instead of the actual data. The server expects requests to appear as if they are coming from a standard web browser to prevent automated abuse.

How can I check what user-agent wget is sending by default?

You can see the exact headers being sent by running wget with the -d (debug) flag. Look for the line starting with USER-AGENT in the debug output to confirm the string being transmitted to the server.

Will spoofing the user-agent violate any terms of service?

While technically possible, always review the website's Terms of Service before automating downloads. Some sites explicitly prohibit programmatic access regardless of the method used. Spoofing is generally acceptable for personal use or testing, but aggressive scraping may lead to IP bans.

What should I do if the CSV file is password-protected?

If the download requires authentication, you need to provide credentials using the --http-user and --http-password flags in wget, or use -u username:password in curl. Ensure that the connection uses HTTPS to protect your credentials during transmission.

Can I save the user-agent setting permanently for all wget commands?

Yes, you can set a default user-agent by adding the --header="User-Agent: Mozilla/5.0..." to your ~/.wgetrc configuration file. This way, every subsequent wget invocation will automatically include the specified browser-like identifier without needing extra flags.