fetch-any

A ladder, not a fetch

Most of the web answers a plain curl with a 403, a bot challenge, or a 200 that is an empty shell waiting for JavaScript. fetch-any stops pretending there is one right way. It climbs:

1. curl with browser headers — a real user-agent, an Accept line, a Google referer. Clears the lazy blocks.
2. r.jina.ai — a reader proxy that renders and returns text. Clears most JavaScript shells.
3. The Wayback Machine — the latest archived snapshot, raw (id_). Clears hard blocks, at the cost of freshness.
4. Real Chrome — only if the chrome-devtools CLI is installed. Clears almost everything, slowly.

Each rung has to return a 200 and more than 400 bytes of text to count — a 200 with nothing in it is a miss, not a hit. Text comes out on stdout, already stripped of script, style, nav and footer. The route that worked goes to stderr so it never contaminates the output. Exit 0 on success, 1 when every rung failed.

Install and use

curl -fsSLo ~/bin/fetch-any https://caelum.codes/fetch-any/fetch-any
chmod +x ~/bin/fetch-any

fetch-any https://example.com/some/article > article.txt
# [fetch-any] route=jina        ← on stderr

Read it before you run it. It is fifty lines:

#!/usr/bin/env bash
# fetch-any <url> — get a page's text by whatever route works.
# Ladder: curl w/ browser headers → r.jina.ai → Wayback → Chrome (real browser, via chrome-devtools CLI if present).
# Prints text to stdout; exit 0 on success, 1 if every route failed. Route used goes to stderr.
set -u
U="${1:?usage: fetch-any <url>}"
UA='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36'
HDRS=(--compressed -A "$UA" -H 'Accept: text/html,application/xhtml+xml,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.9' -H 'Referer: https://www.google.com/')
html2text() { python3 -c '
import sys,re,html
s=sys.stdin.buffer.read().decode("utf-8","replace")
s=re.sub(r"(?is)<(script|style|noscript|svg|nav|footer|header).*?</\1>","",s)
s=re.sub(r"(?i)<br\s*/?>|</p>|</div>|</li>|</h[1-6]>|</tr>","\n",s)
s=re.sub(r"<[^>]+>","",s); s=html.unescape(s)
s=re.sub(r"[ \t]+"," ",s); s=re.sub(r"\n\s*\n+","\n\n",s)
print(s.strip())'; }
ok() { [ -n "$1" ] && [ "$(printf %s "$1" | wc -c)" -gt 400 ]; }

# 1. curl with browser headers
body=$(curl -sL --max-time 20 "${HDRS[@]}" -w '\n__CODE__%{http_code}' "$U")
code=${body##*__CODE__}; body=${body%__CODE__*}
if [ "$code" = 200 ] && ok "$body"; then echo "[fetch-any] route=curl" >&2; printf %s "$body" | html2text; exit 0; fi

# 2. jina reader
body=$(curl -s --max-time 30 -H 'X-Return-Format: text' "https://r.jina.ai/$U" -w '\n__CODE__%{http_code}')
code=${body##*__CODE__}; body=${body%__CODE__*}
if [ "$code" = 200 ] && ok "$body"; then echo "[fetch-any] route=jina" >&2; printf '%s\n' "$body"; exit 0; fi

# 3. Wayback (latest snapshot; id_ = raw)
snap=$(curl -s --max-time 20 "https://archive.org/wayback/available?url=$U" | jq -r '.archived_snapshots.closest.url // empty' 2>/dev/null)
if [ -n "$snap" ]; then
  raw=$(printf %s "$snap" | sed -E 's#(/web/[0-9]+)/#\1id_/#')
  body=$(curl -sL --max-time 30 "${HDRS[@]}" "$raw")
  if ok "$body"; then echo "[fetch-any] route=wayback ($snap)" >&2; printf %s "$body" | html2text; exit 0; fi
fi

# 4. Real Chrome via chrome-devtools-mcp CLI, if installed
if command -v chrome-devtools >/dev/null 2>&1; then
  body=$(chrome-devtools navigate_page --url "$U" >/dev/null 2>&1 && chrome-devtools evaluate_script --function 'document.body.innerText' 2>/dev/null)
  if ok "$body"; then echo "[fetch-any] route=chrome" >&2; printf '%s\n' "$body"; exit 0; fi
fi

echo "[fetch-any] all routes failed for $U (last http=$code). Use the Chrome MCP: navigate + get_page_text." >&2
exit 1

Known edges

• Rung 2 sends the URL to a third party. r.jina.ai sees every URL that gets past rung 1. Do not point this at anything private or authenticated.
• Rung 3 is stale by definition. A Wayback hit is whatever was archived last, which may be years old, and the script does not tell you the date beyond the snapshot URL on stderr. Read the stderr line.
• The 400-byte floor is a heuristic. A genuinely short page (a status endpoint, a one-line notice) will be treated as a miss and the ladder keeps climbing. Lower ok() if that is your case.
• The HTML-to-text pass is a regex, not a parser. It drops <nav>, <header> and <footer> wholesale; a site that puts its article inside a <header> will come back empty and fall through to rung 2.
• It is not polite. It sends a browser user-agent and a fake referer. That is the point, and it is also the reason not to loop it over a whole site.