Where Residential Proxy Bandwidth Goes After the First HTML

A residential proxy bill is a byte bill. The job is usually one HTML document, one JSON payload, or one public product observation. Those two numbers rarely match.

Teams comparing a cURL collector to a headless browser often conclude that “the browser is slower” or that concurrency is the leak. The more common leak is a byte mix nobody measured: redirects, images, scripts, fonts, and a new proxy connection for work that did not need a new identity.

This is an attribution teardown of one job, not a forecast. If you still need a planning formula before a first run, use the bandwidth estimate. If rotation looks wrong because the HTTP pool is reusing sockets, that is connection pooling. Neither of those posts tells you which bucket ate last week’s gigabytes.

Name the document you actually wanted

Write the unit of work down before the first measured request:

job        = one expected observation
wanted     = the HTML, JSON, or API body the parser reads
attempt    = one proxy-backed HTTP exchange
connection = one TCP/TLS session to proxy.packetstream.io
success    = destination responded AND the wanted body passed validation

PacketStream meters usage at the gateway. Application counters (size_download, a HAR total, or Chromium’s encoded transfer length) are a useful split of your client, not a billing replica. Protocol overhead, tunnel setup, compression, and retries can all move the dashboard number. Use the dashboard as the calibration source after a bounded pilot.

Keep credentials out of process arguments. Use PACKETSTREAM_USER and PACKETSTREAM_AUTH_KEY, then write them to the private cURL config shown below.

Four buckets that are not “the HTML”

Most of the extra gigabytes on a residential-proxy job fall into four buckets. They look similar on a single “bytes transferred” chart.

A fifth leak—retries of invalid 200s, challenge pages, and parser rejects—belongs with observability and retries. Count those attempts, but do not average them into the success path.

Do not mix a direct HTTP sample with a full browser sample. A browser may load JavaScript, fonts, stylesheets, images, analytics, and background requests that cURL never asks for. That is expected, not a defect in residential proxies.

Measure the HTTP job first

cURL can report body bytes, header bytes, request bytes, uploaded body bytes, and redirect count without printing the body. This setup runs in a subshell, creates the config and response body with mode 0600, keeps the auth key out of cURL’s process arguments, and removes both files when the measurement ends:

(
set -euo pipefail
umask 077

measurement_dir=$(mktemp -d "${TMPDIR:-/tmp}/packetstream-bandwidth.XXXXXX")
curl_config="$measurement_dir/curl.conf"
body_file="$measurement_dir/wanted-body"
trap 'rm -f -- "$curl_config" "$body_file"; rmdir -- "$measurement_dir"' EXIT

printf 'proxy = "https://proxy.packetstream.io:31111"\nproxy-user = "%s:%s"\n' \
  "$PACKETSTREAM_USER" "$PACKETSTREAM_AUTH_KEY" >"$curl_config"

curl --disable --config "$curl_config" \
  --silent --show-error \
  --connect-timeout 15 \
  --max-time 120 \
  --location \
  --output "$body_file" \
  --write-out '{"http_code":%{http_code},"download_body_bytes":%{size_download},"download_header_bytes":%{size_header},"request_bytes":%{size_request},"upload_body_bytes":%{size_upload},"redirects":%{num_redirects},"url":"%{url_effective}","namelookup":%{time_namelookup},"connect":%{time_connect},"starttransfer":%{time_starttransfer},"total":%{time_total}}\n' \
  'https://example.com/public-page'

wanted_body_bytes=$(wc -c <"$body_file")
printf 'wanted_body_bytes=%s\n' "$wanted_body_bytes"
)

The transfer record prints first. The wanted-body size prints next, before the subshell removes the private files.

The 15-second connection limit and 120-second total limit are starting points for this small public-page pilot. Set both from the workload’s measured behavior. Keep the total deadline long enough for a valid response, but short enough that a stalled proxy, destination, or response stream cannot retain the private measurement files indefinitely.

Then compare three variants of the same public URL, from the same runtime:

1. First hop only. Run the command once without --location. A redirect reports its 3xx status, zero followed redirects, and the original effective URL.
2. Follow redirects. Run the command as shown with --location. This reaches the document you will parse and reports the number of followed hops and final effective URL.
3. Separate processes. Run one URL per cURL process. Each process opens a new proxy connection. Do not use --no-keepalive as the control; it disables TCP keepalive probes, not HTTP connection reuse. PacketStream automatic rotation selects a residential exit on a new connection, not on every application request object.

Record download_body_bytes, download_header_bytes, and request_bytes separately. cURL’s size_download excludes headers. If the follow run downloads more body bytes than the wanted file contains, use a trace or one controlled request per hop to isolate intermediate response bodies. Do not label that delta as headers.

--write-out sizes are still client counters. They do not include every byte the gateway meters. For the bounded GET pilot above, define client-observed bytes as download_body_bytes + download_header_bytes + request_bytes, aggregated over the exact dashboard window. Keep upload_body_bytes as a separate diagnostic counter. The dashboard delta is calibration for that client and connection policy, not a constant you can reuse elsewhere.

The recommended HTTPS proxy is proxy.packetstream.io:31111. SOCKS5 on 31113 and HTTP on 31112 are different contracts; do not mix them in one attribution table.

Then measure the browser job, as a browser

If the job needs a real browser—client-rendered markup, a public flow a direct GET cannot reproduce—attribute resource types, not a single HAR total.

Playwright’s requestfinished event tells you that a request completed, but it does not expose wire bytes. For Chromium, open a CDP session, send Network.enable, attach the event listeners, and only then navigate. Join each request ID to Network.requestWillBeSent and Network.responseReceived so you can group the result by resource type:

document
stylesheet
script
image
font
xhr / fetch
media
other

Use Network.loadingFinished.encodedDataLength as the completed hop’s total. Redirects reuse a request ID, so finalize the previous hop from requestWillBeSent.redirectResponse.encodedDataLength before replacing that ID’s metadata. For a request that ends with Network.loadingFailed, retain the encoded length reported with responseReceived plus later Network.dataReceived.encodedDataLength chunks and mark the result as partial. That partial value is a lower bound, not a successful zero-byte transfer.

A Chromium DevTools HAR is also suitable when it includes response._transferSize; do not substitute the decoded response.bodySize. If your tooling exposes only decoded body length, label that value as an approximation and do not call it transferred bytes.

Choose the cache policy before the run. For a cold-load sample, send Network.setCacheDisabled with cacheDisabled: true before navigation and create the Playwright context with serviceWorkers: 'block'. For a warm-cache sample, record requestServedFromCache, response.fromDiskCache, response.fromPrefetchCache, and response.fromServiceWorker separately because they do not represent a new network transfer in the same way.

For each type record request count, encoded transferred bytes, failed or partial request count, and whether the parser read the body. The question is not “did the page look right.” It is “which types are required for a valid observation.”

A practical split on one SKU list:

Blocking unused types can cut billed bytes. Overblocking can produce a 200 with an empty observation that looks successful. Validate the result, not the screenshot.

Route the browser through the HTTPS proxy the same way the Playwright setup post does. One browser context is one place connection reuse can hide; treat the context’s lifetime as part of the unit of work.

Connection churn is billed work

Automatic rotation selects a residential exit when the client opens a new proxy connection. Several HTTP requests on one pooled connection can share that exit. Several short-lived processes can pay gateway TCP, TLS, and CONNECT setup on every item.

That handshake is not the HTML document. It still traverses the gateway.

A useful comparison, same destination, same country targeting if you use it:

policy A = one process, keep-alive on, N items
policy B = N separate processes, one item each

If policy B’s dashboard usage is materially higher and the wanted bodies are the same size, the delta is connection churn plus whatever retry policy you attached to each new process. Do not “fix” that by buying more gigabytes.

A sticky session (AUTH_KEY_session-<non-sensitive-label>) keeps new connections on the same residential exit for a related workflow. It does not make payloads smaller. Continuity depends on the supplying Packeter remaining connected; a disconnect fails the session instead of silently switching identity. Recover the workflow, then measure it again. Do not advertise a clock as a bandwidth feature.

Country targeting (AUTH_KEY_country-US) does not change the byte buckets. It can change retry behavior if a requested country has no available exit: the request fails rather than substituting another country. Record that as availability, not as “mystery GB,” and do not retry it as if it were a large HTML download.

A compact ledger for one job

Store the split next to the result:

{
  "job_id": "catalog:sku-42:US:2026-08-25",
  "client": "curl",
  "proxy_endpoint": "https://proxy.packetstream.io:31111",
  "connection_policy": "reuse",
  "http_code": 200,
  "valid_result": true,
  "wanted_body_bytes": 184320,
  "download_body_bytes": 196608,
  "download_header_bytes": 2048,
  "request_bytes": 412,
  "upload_body_bytes": 0,
  "redirects": 1,
  "redirect_body_bytes": 12288,
  "unused_asset_bytes": 0,
  "attempts": 1
}

Only populate redirect_body_bytes from a per-hop trace or a controlled comparison, not from the redirect count alone. For a browser job, replace unused_asset_bytes: 0 with the per-type totals. Omit credentials, sensitive session labels, and destination account data.

Aggregate every ledger row whose request falls inside the same dashboard window, then compute:

window_wanted_body_bytes   = sum(wanted_body_bytes)
window_redirect_body_bytes = sum(redirect_body_bytes)
window_unused_asset_bytes  = sum(unused_asset_bytes)
window_client_bytes        = sum(download_body_bytes + download_header_bytes + request_bytes)

wanted_share     = window_wanted_body_bytes / window_dashboard_bytes
redirect_share   = window_redirect_body_bytes / window_dashboard_bytes
asset_share      = window_unused_asset_bytes / window_dashboard_bytes
unexplained      = max(0, window_dashboard_bytes - window_client_bytes)
unexplained_share = unexplained / window_dashboard_bytes

Each numerator now covers the same bounded window and every share uses the same denominator. The named shares can overlap with client-observed bytes, so do not add them together as if they partition the bill. unexplained is expected to be non-zero. It includes protocol overhead and anything the client did not log. If it dominates the window, your application counters are incomplete, not proof that the gateway is padding usage.

A wanted share that stays high as you add items is a healthy HTTP collector. A wanted share that collapses when you switch to a browser, without a corresponding jump in valid observations, is an asset problem. A wanted share that collapses when you move from pooled requests to separate processes, with unchanged bodies, is connection churn.

Change the bucket you named

Raising concurrency does not shrink a document, drop an image, or merge two connections. It usually multiplies the same mix.

Prepaid residential bandwidth on PacketStream is currently $1.00/GB with a $50 minimum, no subscription, and a purchased balance that does not expire—confirm on pricing before you purchase. The rate does not tell you whether last week’s gigabytes were HTML.

What this teardown will not tell you

It will not forecast next month. That is a sample plus a formula, calibrated later.

It will not tell you whether automatic rotation is “working.” Rotation is a connection event. Measure connections, then bytes.

It will not pick a client for you. Direct HTTP is usually cheaper when it can collect a valid observation. A browser is justified when the observation does not exist without one. Attribute both; do not average them.

Run the HTTP split until the wanted body, the redirect hops, and the dashboard window agree closely enough to trust. Only then add the browser. The extra gigabytes almost always have names.