Redirect chains and crawler budget

One redirect is cheap: a request, a status, a location, and a second request. A chain of four is four chances to lose a crawler, four round trips before any content is read, and a set of links that point at addresses the site no longer considers canonical. Every hop looks insignificant in isolation, which is why chains accumulate.

This article is about what a redirect costs a crawler, the shapes that produce chains, and how to measure the chain you have. It uses this site's own redirects as the worked example, because they are deliberately one hop each.

What a redirect costs

A redirect is a complete request that returns no content. For a person, it is a flash of latency. For a crawler, it is a request spent on a URL that produced no page, plus a second request for the target. A chain of N redirects costs N requests before the content request, and each one is an opportunity for a timeout, a loop, or a hop limit.

Most crawlers cap how many redirects they will follow in one fetch. The number varies; the practical rule does not: the safe design is one hop to the final URL. That is also the only design whose cost does not grow when the site moves again.

The shapes that produce chains

• The protocol, host and slash stack. http to https, then www to apex, then a missing trailing slash. Three hops where one would do, and the third is often added by a framework rather than a person.
• The legacy domain. A site moves to a new domain and redirects the old domain to the new homepage, but the old deep links redirect to old paths that then redirect again. The old domain should map directly to the final path.
• The temporary redirect left in place. A 302 from a migration becomes permanent in practice, and a chain builds on top of it.
• The redirect to a canonical that redirects. The redirect target is not the final URL; the canonical disagrees with the redirect; the crawler follows to a third address.
• The UA-specific redirect. A mobile or bot-specific branch sends crawlers somewhere different from people, which is both a chain and a cloaking risk.
• The loop. A to B, B to A. Easy to create with two rules that each think they are normalizing the other.

How this site does it

The proxy has a small table of legacy paths and one rule for each, and every rule targets the final URL rather than an intermediate one. On 2026-10-29 I measured five of them with a redirect count:

One of the five is a doubled-locale URL that was published by mistake, one is an alias in the catalogue, one is a path that moved when the writing path changed, one is a protocol upgrade, and one is a hostname the site used before it had a domain. They are different stories with the same target property: each goes straight to the final URL.

The test suite asserts the exact status and destination for each. That is not decoration: a redirect is a promise, and a promise that is not tested drifts the first time a rule is reordered.

Measuring a chain

# 1. Hops and final URL for one address
curl -sS -L -o /dev/null -w "%{http_code} %{num_redirects} %{url_effective}\n" \
  http://your-domain.com/old-path/

# 2. Each individual hop, in order
curl -sS -L -o /dev/null -D - http://your-domain.com/old-path/ \
  | grep -iE "^HTTP/|^location:"

# 3. Find internal links that still point at legacy URLs
curl -sS https://your-domain.com/ | grep -oE 'href="[^"]+"' \
  | grep -E 'geo-scanner|/insights/|/zh/zh/'

# 4. Check the sitemap lists final URLs, not redirectors
curl -sS https://your-domain.com/sitemap.xml | grep -c '<loc>'

The third command is the one that prevents chains from coming back. A redirect is for addresses the site cannot edit — other people's links, old emails, printed material. An internal link should point at the final URL directly; otherwise the site pays the hop on every crawl and every click.

Batching redirects into one hop

The fix is mechanical once the chain is measured: every rule should target the final URL, not the next rule’s source. If a legacy host redirects to the apex, the legacy deep paths should map to the apex deep paths in the same rule rather than to the old deep paths. If http and www and slash normalization are separate branches, they should collapse into one computed target.

Two cautions. First, do not remove a redirect just because a chain is gone; keep it as long as the old address can receive traffic. Second, do not build UA-specific redirects to solve a bot problem — that is how a redirect becomes a cloaking question.

What redirects do not buy

A clean redirect does not guarantee the target is crawled or cited, and a redirect does not make the old URL's content relevant again. It preserves the path: a link that worked, a bookmark that resolves, a crawler that reaches the page in one request instead of four. That is a maintenance property, and it is worth keeping even when nothing is measured.

The standard from the rest of this series applies: make the path one hop, make it tested, and make the site's internal links stop depending on it.