How to Plan Proxy Concurrency for a Web Scraping Project

Concurrency is the number of tasks in progress at the same time. Increasing it can raise throughput, but only until another part of the system becomes the constraint. Beyond that point, extra workers create longer queues, more retries, higher memory use, and more pressure on the destination without producing more valid data.

A good concurrency plan begins with permission and workload measurements, then adds limits at every scarce resource. The objective is stable, responsible collection—not the largest worker count a machine can start.

Separate concurrency from request rate

Concurrency and rate are related, but they are not interchangeable.

• Concurrency is the number of in-flight tasks.
• Rate is the number of tasks started or completed per unit of time.
• Latency is the time one task remains in flight.

A rough planning relationship is:

concurrency ≈ target completions per second × average task duration in seconds

If an allowed workload aims for two completed requests per second and a representative request takes 1.5 seconds, the starting estimate is three in-flight requests. Real systems need headroom for variation, but the formula exposes the inputs instead of treating worker count as a guess.

Do not use the fastest sample. Measure a representative range of pages and keep a high percentile for timeout and capacity planning.

Start with the destination’s limits

Before sizing infrastructure, define what the destination permits. Review its terms, published API or crawling guidance, robots rules where applicable, and rate-limit signals.

Use the lowest limit required by those constraints. Residential proxies change the network path; they do not make an inappropriate request acceptable or turn a destination limit into a target to evade.

Group work by destination rather than applying one global rate across unrelated sites. A collector that behaves well toward one origin can still overwhelm another if their capacity and policies differ.

Measure one complete unit of work

“One request” may not be the right unit. A browser page can load many resources. A product observation may require several public pages. A sticky workflow may need to restart from the beginning after a disconnect.

For a small pilot, record:

• total duration;
• proxy requests;
• response bytes;
• redirects;
• browser subresources when applicable;
• parser time;
• database write time;
• retry attempts;
• final validation result.

Measure the complete unit that the business consumes. A fast HTML response is not useful if the parser or storage layer becomes the bottleneck immediately afterward.

Add layered concurrency limits

One semaphore at the top of the application is rarely enough. Use independent limits for resources that can saturate separately.

A typical hierarchy includes:

global job limit
├── per-destination limit
├── per-country limit
├── HTTP connection-pool limit
├── browser-process or page limit
└── storage-write limit

The global limit protects the service. The destination limit controls request pressure. Country limits keep one localized workload from occupying every worker. Browser and storage limits protect expensive local resources.

Keep the configuration visible and versioned. If five worker processes each allow 20 requests, the real maximum is 100—not 20.

Use rate limiting and concurrency control together

A semaphore caps how many tasks are active. It does not control how quickly new tasks start when responses are fast.

A token bucket or scheduled release controls starts over time. Combining the two provides a maximum in-flight count and a maximum start rate:

start job only when:
  a concurrency slot is available
  and
  a rate token is available

Add small random variation to scheduled retries so many workers do not wake at the same instant. Keep retry traffic inside the same limits as initial traffic; otherwise a failure wave can exceed the intended rate.

Build backpressure into every stage

Backpressure makes producers slow down when consumers cannot keep up. Without it, a scheduler can create millions of queued tasks while workers, parsers, or storage fall behind.

Use bounded queues between stages. When a queue reaches its limit, pause or reject new production rather than letting memory or database rows grow without control.

For a collection pipeline:

1. the scheduler emits only into a bounded work queue;
2. fetchers stop claiming when the parse queue is full;
3. parsers stop claiming when the write queue is full;
4. writers commit before acknowledging completed work.

Queue age is an important signal. A low CPU graph can look healthy while scheduled work waits hours to begin.

Account for connection reuse

PacketStream automatic rotation selects a residential exit for each new proxy connection. An HTTP client’s connection pool may reuse a connection for several requests.

That means increasing application concurrency can open more proxy connections, but the exact relationship depends on the client’s pool settings and response lifecycle. Do not assume one worker equals one connection or one request equals one new exit.

During the pilot, observe both logical requests and connections. Define the maximum pool size intentionally, close response bodies correctly, and decide whether the pool belongs to a job, worker, destination, or country.

Use sticky sessions when a related workflow needs to retain one exit for up to 60 minutes. If its Packeter disconnects, the session fails; replay the dependent workflow safely instead of increasing concurrency to compensate.

Treat unavailable countries as capacity signals

Country targeting uses a two-letter ISO country code in the auth key:

AUTH_KEY_country-GB

If no exit is available in the requested country, the request fails. Do not fall back silently to another country, and do not retry in an unbounded loop.

Place country-targeted jobs in their own bounded queue. A short retry window may handle transient availability; after that, mark the job unavailable and reschedule it according to the product requirement. This prevents one country from consuming the entire global retry budget.

Estimate bandwidth before raising the limit

More concurrency can consume prepaid bandwidth faster even when total work does not change, because errors and retries can arrive in a tighter burst.

Estimate:

hourly bytes = successful units per hour
             × requests per unit
             × average bytes per request
             × retry multiplier

Compare application measurements with PacketStream dashboard usage during a controlled run. Include browser assets, redirects, uploads when material, and workflow replays. The proxy bandwidth estimation guide provides a fuller model.

Ramp with a stop condition

Increase concurrency in measured stages rather than jumping from a laptop test to the intended maximum.

At each stage, hold the workload long enough to observe:

• valid completions per minute;
• request and end-to-end duration percentiles;
• response status distribution;
• retry multiplier;
• parser validation rate;
• transferred bytes per completed unit;
• queue age;
• browser, CPU, memory, and storage saturation.

Define a stop condition before the run. Stop increasing when valid throughput no longer rises proportionally, retries or invalid results increase, the destination signals throttling, or a local resource approaches its operating limit.

There is no universal PacketStream concurrency number that replaces this measurement. The appropriate value depends on the destination, client, page weight, country, session model, parser, and infrastructure.

Prefer adaptive reductions over automatic increases

It is safer for a system to reduce work automatically than to increase it without review. On sustained throttling, timeouts, storage pressure, or parser failures, lower the rate and concurrency for the affected group.

Automatic increases can hide mistakes by pushing more traffic into a broken parser or an unavailable country. Require a stable observation window and an explicit ceiling, or keep increases as an operator-controlled change.

Size for valid output

The useful metric is not requests started. It is validated, timely observations completed within the destination’s rules and the project’s budget.

Start with one measured unit, calculate an initial in-flight estimate, add per-destination and per-resource limits, enforce a start rate, and connect every stage with bounded queues. Then ramp until another worker stops improving valid throughput.

That approach makes concurrency a controlled capacity decision. It protects the destination, keeps proxy usage predictable, and gives the team a clear reason for every limit in the system.