Residential proxies for web scraping

Set up a scraper in Python

You need the username and auth key from the Proxy Access page of the dashboard. Keep them in environment variables:

export PACKETSTREAM_USER='your_username'
export PACKETSTREAM_AUTH_KEY='your_auth_key'

This scraper uses the HTTPS gateway on port 31111. It picks a country and retries a few times with backoff, with a new exit on each attempt. It stops on wrong credentials and on pages that are gone:

import os
import time
from urllib.parse import quote

import requests

USER = quote(os.environ["PACKETSTREAM_USER"], safe="")
AUTH_KEY = os.environ["PACKETSTREAM_AUTH_KEY"]


def proxy_url(country=None, session=None):
    password = AUTH_KEY
    if country:
        password += f"_country-{country}"
    if session:
        password += f"_session-{session}"
    return f"https://{USER}:{quote(password, safe='')}@proxy.packetstream.io:31111"


def fetch(url, country="US", attempts=3):
    proxy = proxy_url(country)
    for attempt in range(1, attempts + 1):
        try:
            # A new Session opens a new proxy connection, so each
            # attempt gets a new exit selection.
            with requests.Session() as session:
                response = session.get(
                    url, proxies={"http": proxy, "https": proxy}, timeout=30
                )
        except requests.exceptions.ProxyError as error:
            if "Tunnel connection failed: 407" in str(error):
                raise  # Wrong username or auth key: retrying won't help.
            response = None  # No exit was available: try again.
        except requests.RequestException:
            response = None
        if response is not None:
            if response.ok:
                return response
            if response.status_code in (404, 410):
                # The page is gone: retrying won't help.
                response.raise_for_status()
        time.sleep(2 ** attempt)
    raise RuntimeError(f"gave up after {attempts} attempts: {url}")


page = fetch("https://example.com/products?page=1")
print(page.status_code, len(page.content), "bytes")

The same proxy URL works in most HTTP clients. For other languages, see language integrations. For Scrapy, use the Scrapy guide. For pages that need JavaScript, use a headless browser such as Playwright.

Rotate or stick?

Use one session label per job, and a new label for each job that runs at the same time. Browsers and HTTP libraries reuse connections, so a new request does not always mean a new exit. Read rotation and sticky sessions for the details.

Keep the cost down

You pay for every byte that goes through the gateway, including error pages and retries. Three things help most:

1. Measure your pages. Page size decides the bill. The bandwidth calculator turns pages and page size into GB and dollars.
2. Skip what you don’t parse. An HTTP client that fetches only the HTML uses much less than a browser that also loads images, fonts and scripts.
3. Limit retries. Retry in one place in your code, with a small limit. Retries in several layers multiply traffic.

How to estimate proxy bandwidth and how to plan proxy concurrency cover this in more depth.

Honest limits

• Sites can still block you. A residential IP is not a pass. Sites can rate-limit you, block you, or show a CAPTCHA. We don’t solve CAPTCHAs.
• The proxy doesn’t run JavaScript. It forwards your traffic. If a page builds its content with JavaScript, your browser or client must run it.
• Country only. You can pick the country. You can’t pick a city, region or ISP.
• Exits are real homes. Speed varies, and a Packeter can go offline. Use timeouts and retries.
• Only collect what you may. Collect public data, follow each site’s terms, and don’t collect personal data unlawfully. Our terms forbid getting around logins or other access controls.

Need clean text, not raw HTML?

The Fetch API reads one public page and returns it as Markdown, for $0.0025 per page that returns content. It does not run JavaScript. It uses the same balance as your proxy traffic.