How to Build Idempotent Web Scraping Jobs with Residential Proxies

Distributed web scraping jobs are rarely executed exactly once. A worker can time out after a destination responded, lose its queue lease after writing data, or restart while another worker is already retrying the same item. Residential devices can disconnect, and a multi-step sticky workflow may need to begin again.

Idempotency makes those repeated attempts safe. The goal is not to prevent every duplicate request. It is to ensure that processing the same logical job more than once does not create duplicate records, publish the same event twice, or leave a half-finished result.

Give every logical job a stable identity

A queue message ID usually identifies one delivery, not the underlying work. Retries and reschedules may create new messages for the same target.

Create a deterministic job key from the fields that define one expected observation. Depending on the project, those fields might include:

• normalized source URL or source record ID;
• requested country;
• collection window or scheduled date;
• parser version;
• data product or customer scope;
• workflow type.

An example job record:

{
  "job_key": "catalog:sku-42:US:2026-04-23:v3",
  "url": "https://example.com/public-product/sku-42",
  "country": "US",
  "parser_version": 3,
  "scheduled_for": "2026-04-23T15:00:00Z"
}

Hash the canonical representation if the natural key is too long, but retain the source fields for debugging. Avoid including volatile values such as the attempt number or queue delivery ID in the logical key.

Separate the job from its attempts

One job can have several attempts. Model them separately.

The job describes the desired outcome and current state. Each attempt records what happened during one execution: start time, worker, response status, transferred bytes, parse result, and failure category.

job
├── attempt 1: timeout
├── attempt 2: Packeter disconnected
└── attempt 3: collected and committed

This structure preserves operational evidence without publishing three copies of the same observation. It also makes the retry multiplier visible when estimating proxy bandwidth.

Do not place the proxy username or auth key in either record. Store a redacted endpoint, requested country, and non-sensitive session metadata instead.

Define a small state machine

A clear state machine prevents workers from interpreting the same row differently. A basic model might use:

• pending: ready to be claimed;
• running: owned by a worker until a lease expires;
• retryable: safe to attempt again after a delay;
• complete: result committed successfully;
• failed: terminal outcome requiring review or a new job.

Transitions should be conditional. A worker may mark a job complete only if it still owns the active lease. A retry scheduler should move a job only from retryable, not overwrite a result that another worker already completed.

Keep failure categories specific enough to drive policy. Authentication failures, unavailable country exits, destination throttling, network timeouts, and parser errors should not all receive the same retry schedule.

Use bounded leases, not permanent locks

Queue consumers and database workers can disappear. A permanent running flag leaves abandoned work stuck forever.

Use a lease with an owner and expiration:

{
  "state": "running",
  "lease_owner": "worker-17",
  "lease_expires_at": "2026-04-23T15:03:00Z"
}

The worker renews the lease while it is healthy. Another worker can claim the job only after expiration. Completion should compare the lease owner or a generation value so an old worker cannot overwrite newer work after waking up late.

Set the lease from measured job duration plus a reasonable margin. A very short lease creates unnecessary overlap; a very long lease delays recovery.

Make the result write atomic

The most dangerous failure happens between writing the result and acknowledging the queue message. If the acknowledgement is lost, the queue correctly redelivers the job.

Protect the result with a unique constraint on the logical job key or observation key. Then write the result and update the job state in one database transaction when possible.

Common patterns include:

• insert once with a unique key;
• upsert the same observation deterministically;
• compare-and-swap using a job generation;
• write to a staging row, validate it, then promote it atomically.

Avoid “check then insert” without a uniqueness constraint. Two workers can both pass the check before either inserts.

Keep side effects behind the commit

A successful scrape may trigger an alert, publish a message, update a search index, or notify another service. Those side effects can duplicate even when the database row does not.

One reliable approach is a transactional outbox:

1. commit the observation;
2. insert an outbox event in the same transaction;
3. let a separate dispatcher publish the event;
4. give the downstream consumer its own idempotency key.

If the dispatcher publishes an event and crashes before marking it sent, it may publish again. The stable event key lets the consumer recognize that replay.

Retry only work that is safe to repeat

Public, read-only retrieval is often safe to retry, but the complete workflow may contain actions that are not. A browser script might submit a form, change account state, or trigger a download with side effects. Review each step before applying a generic retry wrapper.

For safe requests, use exponential backoff with jitter, cap the attempt count, and enforce a total time budget. Stop retrying deterministic failures such as invalid credentials or a parser that cannot understand a known response format.

Respect destination terms, rate limits, and access controls. A retry system should reduce accidental pressure, not turn an inappropriate request into a persistent one.

Replay sticky workflows as a unit

A sticky session can retain the same residential exit for related work for up to 60 minutes. If its Packeter disconnects, the session fails.

When several steps depend on shared state, represent the entire workflow as one replayable unit. Store intermediate values in a staging area, then publish only after every required step passes validation.

If the session fails midway:

1. discard or quarantine the incomplete staging result;
2. create a new session identifier;
3. restart the dependent sequence from its safe beginning;
4. commit under the same logical job key.

Do not merge half of one session with half of another unless the data model explicitly proves that the steps are independent.

Make parsers deterministic

The same saved response should produce the same normalized record for a given parser version. That makes a failed parse reproducible without another proxy request.

Useful practices include:

• store a permitted response sample or content fingerprint for debugging;
• version parser rules;
• normalize whitespace, URLs, currencies, and timestamps consistently;
• distinguish missing data from an explicit empty value;
• validate required fields before commit;
• quarantine surprising schema changes.

When the parser changes, decide whether the old job should be reprocessed from saved input or recollected as a new observation. Include the parser version in the key when different versions are intended to coexist.

Measure duplicate work

Idempotency protects correctness, but repeated attempts still consume time and bandwidth. Track:

• attempts per completed job;
• jobs completed after lease recovery;
• duplicate result writes rejected by the database;
• outbox events deduplicated downstream;
• bytes transferred per logical job;
• failure categories that trigger the most retries.

A rising duplicate rate may indicate leases that are too short, workers that acknowledge too late, overlapping schedulers, or timeouts that do not match observed destination behavior.

Design for at-least-once execution

“Exactly once” is usually a property created by several layers working together, not a promise the queue can make alone. Assume a job can arrive twice.

Give it a stable identity, record each attempt, claim it with a bounded lease, enforce uniqueness at the result store, commit side effects through a replay-safe mechanism, and retry only operations that are safe to repeat.

With those controls in place, residential proxy failures and worker restarts become routine state transitions rather than data-quality incidents. The system can recover without publishing duplicates—and without spending more bandwidth than the retry policy allows.