Using PacketStream Residential Proxies with Scrapy

Route Scrapy requests through PacketStream's HTTPS proxy, verify the residential exit, and add country or sticky-session routing.

Route Scrapy requests through a residential exit without replacing your spiders or pipelines. This setup uses Scrapy’s proxy middleware and a local TLS bridge.

Configure

Install Scrapy and set your PacketStream credentials:

python -m pip install scrapy
export PACKETSTREAM_USER='your_username'
export PACKETSTREAM_AUTH_KEY='your_auth_key'

Scrapy’s standard proxy middleware does not make a TLS connection to an HTTPS proxy. Install socat with your operating system’s package manager, then start this loopback-only TLS bridge in another terminal:

socat 'TCP4-LISTEN:31110,bind=127.0.0.1,reuseaddr,fork' \
  'OPENSSL:proxy.packetstream.io:31111,verify=1,commonname=proxy.packetstream.io'

Keep the bridge running while the spider runs, then stop it with Ctrl+C. If local port 31110 is already in use, replace 31110 with the same unused loopback port in both the TCP4-LISTEN value above and the Python proxy URL below.

Create packetstream_exit.py:

import os
from urllib.parse import quote

import scrapy


class PacketStreamExitSpider(scrapy.Spider):
    name = "packetstream_exit"
    custom_settings = {"DOWNLOAD_TIMEOUT": 30}

    def start_requests(self):
        username = quote(os.environ["PACKETSTREAM_USER"], safe="")
        auth_key = quote(os.environ["PACKETSTREAM_AUTH_KEY"], safe="")
        proxy = f"http://{username}:{auth_key}@127.0.0.1:31110"
        yield scrapy.Request("https://ipinfo.io", meta={"proxy": proxy})

    def parse(self, response):
        yield response.json()

Run it with scrapy runspider packetstream_exit.py -O -. Scrapy sends plaintext only to the local bridge. socat verifies the gateway certificate and carries the traffic to proxy.packetstream.io:31111 over TLS. Scrapy does not include a standard authenticated SOCKS5 downloader.

Verify your exit

The item written to standard output contains the ipinfo.io response. Inspect its ip and country fields before changing the spider’s start URL.

Country targeting and sticky sessions

Apply the modifiers before URL-encoding the password:

auth_key = quote(
    f'{os.environ["PACKETSTREAM_AUTH_KEY"]}_country-US_session-crawl42',
    safe="",
)

Keep the same local proxy URL and HTTPS bridge.

Troubleshooting

Set meta["proxy"] on every request path that must use PacketStream, including requests created in callbacks. Percent-encode both credential components before placing them in the local proxy URL.
Keep the loopback bridge running, use the loopback port selected above (31110 by default) from Scrapy, and leave upstream certificate verification enabled.
Scrapy and its downloader can reuse connections. A second request does not guarantee a new exit.

See proxy troubleshooting for authentication, TLS, connection, and rotation checks.

Using PacketStream residential proxies with Scrapy

Configure

Verify your exit

Country targeting and sticky sessions

Troubleshooting

Related