Soft Blocks: When a 200 OK Is Really a Block Page

A hard block is easy to handle. The destination answers 403 or 429, your client sees a failure, and the job takes the failure path.

A soft block answers 200 OK. The body is a challenge page, an “access denied” notice, a login wall, a consent interstitial, or a page shell with the data missing. Nothing in the transport layer looks wrong, so a naive pipeline parses it, finds no price or an empty result list, and writes that as fact.

That is worse than an error. An error costs you one record. A soft block that passes as success costs you the record and quietly corrupts the dataset around it: prices drop to null, products look discontinued, search rankings look empty.

This guide covers how to recognize soft blocks, how to label them so they stay out of your data, and how to respond without making the situation worse for you or the site.

What a soft block looks like

Soft blocks come in a few recurring shapes:

• Challenge or interstitial pages. A short HTML page that runs a script, shows a CAPTCHA, or asks the visitor to wait. The status is often 200, sometimes 403 or 503.
• Denial notices served as 200. A templated “access denied” or “unusual traffic” message with the site’s normal header and footer, so it still looks like a page.
• Redirects to somewhere else. The request for a product page ends, after redirects, on a login page, a regional home page, or a generic landing page.
• Consent or region walls. A cookie-consent or “not available in your region” page sits in front of the content.
• Degraded pages. The full template renders, but the parts you need are missing or replaced: no prices, “0 results,” or placeholder text where reviews should be.

The last shape is the hardest, because the page can look structurally correct. A site may also serve a degraded page for reasons that have nothing to do with your traffic, such as an outage, a layout change, or an item that really is gone. Detection tells you the response is not the page you expected. It does not always tell you why.

Signals that separate real pages from soft blocks

No single signal is reliable. Combine cheap checks, and decide based on what you expect from the page, not on a universal list of bad words.

Final URL

Compare the URL after redirects with the URL you requested. A product request that ends on /login, /consent, or the site root did not return the product. Record the final URL for every response so you can group failures by where they landed.

Size against a baseline

Track the body size of accepted pages per template. A product page that is usually 180 KB and suddenly arrives at 6 KB deserves a closer look. Use a range learned from your own accepted responses rather than a fixed threshold, and keep it per site and per template.

Required content

This is the strongest signal. Each template you parse should declare what a real page must contain: a product title, a price block, at least one result row, a canonical link that matches the item. If the page lacks the required content, it is not a valid page, whatever its status code says.

Write this as part of the parser contract, not as an afterthought. The data quality guide describes the same idea for records; here it applies one step earlier, to the response.

Known markers

Block and challenge pages tend to reuse a small set of titles and phrases per site. Keep a short, per-site list of markers you have actually observed, such as a page title or a distinctive element, and treat a match as evidence, not proof. Generic word lists (“blocked,” “robot”) produce false positives on real pages that happen to discuss those words.

Repetition across different URLs

Real pages for different items differ. Soft blocks are templates. If many unrelated URLs return bodies with the same normalized hash or the same title within a short window, you are probably looking at one block page served many times.

Content type

A JSON endpoint that suddenly returns text/html is almost never returning your data.

A response classifier

The code below classifies a response before any parser touches it. It returns a label and a reason. It does not retry or change anything about the request; that decision belongs to the caller.

import hashlib
import re
from dataclasses import dataclass, field
from urllib.parse import urlsplit


@dataclass
class PageExpectation:
    """What a real page from one template must look like."""
    required_patterns: list[str]          # regexes a real page must contain
    min_bytes: int = 0                    # from your own accepted baseline
    content_type: str = "text/html"
    known_block_markers: list[str] = field(default_factory=list)
    allowed_final_paths: tuple[str, ...] = ()  # empty means "same path"


@dataclass
class Verdict:
    label: str    # "ok", "hard_block", "soft_block", "suspect"
    reason: str
    body_sha256: str


def classify(requested_url, final_url, status, content_type, body, expect):
    digest = hashlib.sha256(body).hexdigest()
    text = body.decode("utf-8", errors="replace")

    if status == 407:
        # Proxy credentials or configuration, not the destination.
        raise RuntimeError("proxy authentication failed")
    if status in (401, 403, 429):
        return Verdict("hard_block", f"http_{status}", digest)
    if status >= 400:
        return Verdict("suspect", f"http_{status}", digest)

    if not content_type.startswith(expect.content_type):
        return Verdict("soft_block", "unexpected_content_type", digest)

    requested_path = urlsplit(requested_url).path
    final_path = urlsplit(final_url).path
    allowed = expect.allowed_final_paths or (requested_path,)
    if final_path not in allowed:
        return Verdict("soft_block", "redirected_elsewhere", digest)

    for marker in expect.known_block_markers:
        if marker in text:
            return Verdict("soft_block", "known_block_marker", digest)

    missing = [p for p in expect.required_patterns
               if not re.search(p, text)]
    if missing:
        # A page that is also unusually small is very likely a block;
        # a full-size page missing content may be a layout change.
        if len(body) < expect.min_bytes:
            return Verdict("soft_block", "small_and_missing_content", digest)
        return Verdict("suspect", "missing_required_content", digest)

    return Verdict("ok", "passed", digest)

Two details matter more than the code itself.

First, the classifier separates soft_block from suspect. A small page with a known marker is a block. A full-size page that lost its price selector may be a block, or it may be a site redesign that just broke your parser. Those need different responses, and mixing them hides parser regressions inside block metrics.

Second, nothing labeled anything other than ok reaches the parser’s output table. Store the verdict, the reason, the final URL, the status, the size, and the body hash with the job record. Keep a bounded sample of raw bodies per reason when your data policy allows it, so a person can confirm what the classifier saw.

Feed the body hashes into a counter as well. If one suspect hash shows up across hundreds of unrelated URLs in an hour, promote it to a known marker for that site after someone has looked at a sample.

How to respond

Classification is only useful if each label leads to a defined action.

A rising soft-block rate is the site telling you something about your traffic. Treat it like a 429: lower concurrency for that destination, spread requests over a longer window, and reconsider how often you need the data. Do not treat it as a signal to retry the same URL as fast as possible from a new exit. That usually raises the block rate, multiplies your bandwidth, and pushes against the site’s policy rather than working within it.

Some practical rules:

• Budget retries per job, not per attempt. A soft block consumes an attempt from the same budget as a timeout. The retry and backoff guide covers how to bound that budget.
• Do not overwrite good data with a blocked fetch. A soft-blocked response for an item you already have should leave the last accepted record in place and mark it stale, not replace it with nulls.
• Pause on a sustained rate. A circuit breaker per destination, opened when the soft-block rate crosses a threshold you choose, protects both your dataset and the site.
• Respect walls that are policy, not friction. A login wall or a region restriction is the site deciding who may see the page. Collect public data that the site makes available, and follow its terms.

Where PacketStream fits

With PacketStream residential proxies, your client receives exactly what the destination sent, so soft-block handling lives in your code. Two proxy behaviors affect how you read the results:

• Rotation is per connection. Automatic rotation selects a residential exit for each new proxy connection, not each HTTP request. If your client reuses pooled connections, a run of soft blocks can come from a single exit. Record the exit IP or approximate geography per response when you diagnose block patterns; the connection pooling guide explains why.
• Country targeting changes what the page should look like. A request with a _country- modifier gets the destination’s local version of the page, and region walls or consent pages differ by country. Build per-country expectations when you collect from more than one market. See country targeting for the modifier and its behavior.

The PacketStream Fetch API does part of this classification for you, with limits worth knowing:

• A site’s non-2xx answer comes back as a target_error, and when the site answers 403, 429, or 503, Fetch asks again up to twice from different IP addresses within its 25-second limit. A fetch that still fails is free.
• A short page that asks for JavaScript returns 422 javascript_required, and a page with no readable text returns 422 empty_content. Neither is billed. Many script-based challenge pages fall into the first case because Fetch does not run JavaScript.
• Fetch does not solve CAPTCHAs and does not judge whether readable content is the page you wanted. A denial notice or degraded page served as 200 with ordinary text comes back as page content and is billed like any other page. Run the same required-content check on Fetch output that you would run on a raw response.

The Fetch errors and limits page lists every error code and which ones are worth retrying.

Measure it

Add soft blocks to the same dashboard as transport errors, not to a separate log nobody reads. Useful series per destination and template:

• verdict counts over time (ok, hard_block, soft_block, suspect);
• top reasons within each verdict;
• distinct body hashes among non-ok responses;
• final-URL paths for redirected responses;
• the share of stored records that are stale because their refresh was blocked.

The observability guide covers the transport-side metrics that sit next to these.

The goal is not a scraper that never sees a block page. It is a pipeline that recognizes one, keeps it out of the data, and slows down when the site asks it to.