Web Crawler

A web crawler is an automated program that discovers web pages by requesting URLs and following links between them. It starts from one or more seed URLs, extracts eligible links, and adds unseen destinations to a queue.

Crawlers power search indexing, site audits, link checks, archival work, and content inventories. A focused crawler may stay within one host or path, while a broad crawler can traverse many sites. URL normalization and duplicate detection prevent fragments, tracking parameters, redirects, or equivalent paths from creating endless repeated work.

Crawling is about discovery. Web scraping is about extracting selected facts from fetched content. A crawler may pass each discovered page to a scraper, but a scraper can also work from a fixed list of URLs without following any links. Keeping those responsibilities separate makes request limits, parsing failures, and data completeness easier to reason about.

A polite crawler identifies itself when appropriate, reads the site’s robots.txt, limits request frequency, avoids trap-like infinite URL spaces, and backs off when a server reports overload or rate limits. Site terms and applicable law still govern access and use; crawler directives are not a substitute for them.

Related terms: web scraping and robots.txt.