← Blog · · 6 min read

Web Scraping Data Quality: Validation, Deduplication, and Change Detection

A scraper is only useful when its output can be trusted. A job may finish with a perfect success rate and still produce duplicate records, stale values, missing fields, or prices in the wrong currency. Transport success tells you that a response arrived. Data quality tells you whether the result is fit for its intended use.

The safest design treats collection, parsing, validation, and publication as separate stages. Each stage should preserve enough evidence to explain what happened without retaining more data than the project needs.

Define quality before writing the collector

Start with a written contract for one valid record. Describe the fields, types, allowed units, null behavior, and freshness requirements. A product record, for example, might require:

  • a stable source URL or source identifier;
  • a non-empty title;
  • a price represented as a decimal amount plus an explicit currency;
  • a collection timestamp in UTC;
  • an availability value from a small controlled set;
  • the source country or locale used for the request.

Separate required fields from optional ones. If a page legitimately omits a review score, null is more honest than 0. If a currency cannot be identified, reject or quarantine the record instead of guessing from a symbol that may be used by multiple currencies.

The contract should also define invariants. A discount price should not exceed the regular price without an explanation. A percentage should stay within its valid range. A collection timestamp should not be older than the job that produced it.

Preserve raw evidence long enough to debug

Parsers change more often than network clients. When practical and permitted, keep a bounded raw snapshot or a content hash alongside the parsed result. This lets you determine whether an unexpected value came from the source, a parser regression, or a later transformation.

A useful provenance record can include:

{
  "source_url": "https://example.com/items/42",
  "collected_at": "2025-09-25T14:03:12Z",
  "requested_country": "US",
  "http_status": 200,
  "parser_version": "product-v7",
  "content_sha256": "..."
}

Avoid placing proxy credentials in provenance, logs, or captured commands. Store an exit IP or approximate geography only when it is genuinely needed for diagnostics, protect it like operational data, and apply a retention policy.

Validate shape and meaning

Schema validation catches structural failures: a missing field, a number encoded as an object, or an invalid timestamp. Semantic validation catches values that are structurally valid but implausible.

Run both layers before publishing:

  1. Parse the response into a candidate record.
  2. Validate required fields and types.
  3. Normalize units, whitespace, identifiers, and timestamps.
  4. Apply domain rules and reasonable bounds.
  5. Send failures to a review queue with a reason code.

Do not discard every imperfect record into one generic error bucket. missing_price, unknown_currency, selector_not_found, and blocked_response point to different remedies. Stable reason codes also make quality trends measurable.

Deduplicate at more than one level

Duplicate URLs are not the same as duplicate entities. Tracking parameters, mobile paths, translated routes, and alternate product pages can all refer to the same underlying item.

Use several layers of deduplication:

  • normalize URLs before scheduling work;
  • honor a stable source identifier when one exists;
  • derive a deterministic record key from the fields that identify the entity;
  • hash normalized content to detect identical payloads;
  • make storage writes idempotent so a retry cannot create a second record.

Keep the original source URL even when you use a normalized key. It remains useful evidence when a merge rule turns out to be too aggressive.

Retries deserve special attention. A timed-out request may have completed at the destination or downstream service even if the worker never received the acknowledgement. Design the pipeline so replaying the same job is safe. The retry and session recovery guide covers how to bound retry behavior without silently duplicating work.

Detect meaningful change, not formatting noise

Comparing raw HTML creates alerts for timestamps, generated element IDs, reordered attributes, and other changes that do not affect the record. Compare normalized structured data instead.

For each accepted record, calculate a fingerprint from the fields that matter to the use case. Exclude collection metadata such as collected_at. When the fingerprint changes, store a field-level diff:

{
  "changed": ["price", "availability"],
  "before": {"price": "29.00", "availability": "in_stock"},
  "after": {"price": "27.50", "availability": "out_of_stock"}
}

Not every change should trigger the same action. A small price movement may be routine, while a currency change, a 90% drop, or a sudden disappearance across many records may indicate a parsing problem. Use thresholds to route suspicious changes for review rather than automatically treating them as truth.

Measure the pipeline, not just request success

A useful quality dashboard includes rates for:

  • valid records;
  • required-field failures;
  • duplicates before and after parsing;
  • parser failures by template or domain;
  • unusually large changes;
  • stale records;
  • manual-review acceptance and rejection.

Break metrics down by parser version, page type, destination, and requested country. A global average can hide one broken template. Compare a new parser against a fixed test corpus before deploying it, then roll it out gradually.

Sample the accepted records too

Reviewing only failures creates a blind spot: the most dangerous output is often a plausible-looking record that passed every automated rule.

Take a small, representative sample of accepted records and compare them with their source evidence. Include common pages, edge cases, different locales, and high-impact changes. Track findings as new tests so the same failure becomes easier to catch next time.

Keep collection behavior explicit

Country targeting should be part of the job specification when location changes the content. With PacketStream, append a two-letter ISO country code to the auth key, such as AUTH_KEY_country-US. If no exit is available in that country, the request fails instead of silently returning another location. Preserve that failure rather than falling back to data from the wrong market.

Automatic rotation selects a residential exit for each new proxy connection. A connection pool may therefore reuse an exit across several HTTP requests. If identity continuity matters within a workflow, use a named sticky session for up to 60 minutes and restart the workflow deliberately if its Packeter disconnects.

These transport choices should be recorded as job configuration, not inferred later from the data.

Publish only after a quality gate

A reliable production flow looks like this:

  1. schedule normalized, deduplicated work;
  2. collect with bounded timeouts and retries;
  3. preserve limited provenance;
  4. parse with a versioned parser;
  5. validate structure and meaning;
  6. compare with the previous accepted record;
  7. quarantine anomalies;
  8. publish idempotently;
  9. sample accepted output.

The result is not merely cleaner data. It is a system that can explain its decisions, recover from parser changes, and improve without refetching everything. Before scaling that system, estimate its transfer requirements with the proxy bandwidth planning guide.

More articles Try PacketStream