Unblock Websites with CURL through Reverse SSH Tunnel Proxy

Web Scraping Setup with Socks5 Proxy in Home Network

Introduction

This article provides a step-by-step guide on how to set up a web scraping environment using a Socks5 proxy within a home network, making it accessible to a remote Virtual Private Server (VPS) via reverse SSH tunneling.

Prerequisites

1. Python: Ensure Python 3.x is installed on both the local device and the remote VPS.
2. Scrapy: Install Scrapy, a popular web scraping framework for Python.
3. Socks5 Proxy: Install a Socks5 proxy server on a separate device within the home network.
4. SSH Client: Install an SSH client on the local device, such as OpenSSH or PuTTY.

Setting Up the Socks5 Proxy

Configure the Socks5 proxy server to listen for connections on a specific port (e.g., 1080).

Configuring the Local Device

1. Install the Socks5 proxy client on the local device.
2. Configure the Socks5 proxy client to connect to the Socks5 proxy server using the appropriate hostname, port, and authentication credentials (if required).

Setting Up the Reverse SSH Tunneling

1. On the remote VPS, create a local port forwarding rule using the SSH client. For example:

ssh -N -f -R localhost:8080:localhost:1080 user@vps_ip_address

Replace "localhost:1080" with the Socks5 proxy server's listening port, and "user@vps_ip_address" with your VPS username and IP address.

2. On the local device, configure Scrapy to use the Socks5 proxy by modifying the settings.py file:

DOWNLOAD_HANDLERS = {
        'socks5': 'scrapy.downloadermiddlewares.socks.SocksMiddleware',
        'http': None,
        'https': None,
    }
    HTTPCACHE_ENABLED = False
    HTTPCACHE_DIR = 'scrapy_cache'
    HTTPCACHE_IGNORE_HTTP_CODES = [500]
    HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
    HTTPCACHE_EXPIRATION_SECS = 0
    SOCKS_PROXY_TYPE = 5
    SOCKS_PROXY_HOST = '127.0.0.1'
    SOCKS_PROXY_PORT = 8080

Replace "127.0.0.1:8080" with the local IP address and port of the SSH tunnel on the local device.

With the Socks5 proxy set up and reverse SSH tunneling configured, your web scraping setup should now be able to access the internet anonymously through the proxy, making it an invaluable tool for data extraction in a home network environment.

References

• Books:
  • Web Scraping with Python (2019) by Gaëtan Renard
• Articles:
  • Scrapy Documentation: Socks middleware
  • SSH Tunneling: SSH Tunneling Example
• Online Resources:
  • Scrapy Tutorial: Scrapy Tutorial
  • SSH Tunneling: SSH Tunneling for Remote Access