Most scraping projects don't fail because you can't download HTML. They fail because raw HTML is unusable. Fetch a modern page with curl and you get tens of thousands of characters of <div> soup: navigation menus, cookie banners, inline styles, tracking scripts, and a few hundred words of actual content buried somewhere in the middle. Feeding that into an LLM burns tokens on boilerplate, and writing a per-site parser means your scraper breaks every time the site ships a redesign.
The hard part of scraping was never the download — it's the cleanup. Markdownify removes that part entirely: one HTTP GET returns the readable content of any URL as clean, GitHub-flavored markdown, with no headless browser, no CSS selectors, and no per-site rules. This tutorial walks through scraping a single page, crawling a set of pages, and turning the results into usable data.
The endpoint is GET https://fetchapi.tech/v1/markdownify and takes a single query parameter, url. It returns JSON rather than a raw blob, which matters when you're scraping at scale:
| Field | Meaning |
|---|---|
content |
The main article/page content as clean markdown |
title |
Page title, when the extractor can find one |
description |
Meta description, when present |
url |
The source URL, echoed back |
word_count |
Words in content — useful for token budgeting |
char_count |
Characters in content |
Under the hood it uses content extraction tuned for readability, so it keeps article text, headings, lists, tables, blockquotes, and links while dropping navigation, ads, footers, and comments. Code blocks survive with their fences intact, which makes it ideal for scraping documentation.
Here's the whole thing, no API key required on the free tier:
curl -s "https://fetchapi.tech/v1/markdownify?url=https://example.com"
The response is real, clean markdown wrapped in JSON:
{
"content": "# Example Domain\n\nThis domain is for use in documentation examples without needing permission. Avoid use in operations.\n\n[Learn more](https://iana.org/domains/example)",
"title": "",
"description": "",
"url": "https://example.com",
"word_count": 20,
"char_count": 167
}
Try it on something heavier. Scraping the Wikipedia article on web scraping returns roughly 3,800 words of markdown in a single call — article structure, headings, links, and references all preserved, with zero HTML overhead.
In a pipeline you usually want only the content field. Pipe the JSON through jq to strip everything else:
curl -s "https://fetchapi.tech/v1/markdownify?url=https://docs.python.org/3/tutorial/introduction.html" \
| jq -r '.content' > python-intro.md
That one line gives you a clean markdown file you can drop into a RAG index, a git repo, or a note-taking app. Because the output is markdown, you also get structure for free — # headings become natural chunk boundaries, and [text](url) links stay machine-readable.
A scraper needs targets. Two reliable discovery strategies:
Sitemaps. Most sites publish /sitemap.xml. Fetch it, filter for the sections you care about, and you have a crawl list:
curl -s "https://example.com/sitemap.xml" \
| grep -oE '<loc>[^<]+</loc>' \
| sed -E 's/<\/?loc>//g' \
| grep '/blog/' > urls.txt
Link harvesting. Extract every link from a hub page's markdown. Because Markdownify already converted the page, you don't need an HTML parser — a regex is enough:
curl -s "https://fetchapi.tech/v1/markdownify?url=https://news.ycombinator.com" \
| jq -r '.content' \
| grep -oE '\]\(https?://[^)]+\)' \
| sed -E 's/^\]\(//; s/\)$//' \
| sort -u > urls.txt
Combine both, deduplicate, and you have a queue. Always filter that queue through robots.txt before you start fetching.
Single curl calls don't scale past a few dozen pages. Here's a compact Python crawler that adds the three things every production scraper needs: bounded concurrency, local caching, and retries with backoff.
import hashlib
import json
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path
import requests
BASE = "https://fetchapi.tech/v1/markdownify"
CACHE = Path("cache")
CACHE.mkdir(exist_ok=True)
def scrape(url: str) -> dict | None:
"""Fetch one URL as markdown, caching the raw JSON on disk."""
key = hashlib.sha256(url.encode()).hexdigest()[:16]
cached = CACHE / f"{key}.json"
if cached.exists():
return json.loads(cached.read_text())
for attempt in range(4):
try:
resp = requests.get(BASE, params={"url": url}, timeout=30)
if resp.status_code == 429:
time.sleep(2 ** attempt) # back off on rate limits
continue
if resp.status_code == 400:
return {"url": url, "error": resp.json().get("detail")}
resp.raise_for_status()
data = resp.json()
cached.write_text(json.dumps(data))
return data
except requests.RequestException:
time.sleep(2 ** attempt)
return {"url": url, "error": "failed after retries"}
def crawl(urls: list[str], workers: int = 4) -> list[dict]:
results = []
with ThreadPoolExecutor(max_workers=workers) as pool:
futures = {pool.submit(scrape, u): u for u in urls}
for fut in as_completed(futures):
r = fut.result()
if r:
results.append(r)
print(f"{r.get('word_count', 0):>6} words {futures[fut]}")
return results
if __name__ == "__main__":
urls = [u.strip() for u in Path("urls.txt").read_text().splitlines() if u.strip()]
pages = crawl(urls)
with open("corpus.jsonl", "w") as f:
for page in pages:
f.write(json.dumps(page) + "\n")
print(f"\nScraped {len(pages)} pages -> corpus.jsonl")
Three details make this robust. Caching the raw JSON means you can re-process your corpus without re-fetching, so a schema change costs nothing. The retry loop handles transient 5xx and rate-limit responses instead of silently dropping pages. And keeping workers at four is deliberate: aggressive parallelism gets you blocked, so a small pool with a polite delay is faster in practice than fifty threads.
Scraping real sites means dealing with pages that resist extraction. The API returns a 400 with a JSON detail field when it can't fetch or extract, so check for it explicitly rather than assuming content exists:
curl -s -o /dev/null -w "%{http_code}\n" \
"https://fetchapi.tech/v1/markdownify?url=https://this-domain-does-not-exist-12345.com"
# 400
{"detail": "Could not fetch URL — page may be blocked or unreachable"}
A few patterns to expect:
word_count is tiny on a page you know is long, treat it as a rendering problem, not a content problem.content should trigger a fallback path — log the URL, skip it, and move on. Never write null into your corpus.A simple quality gate keeps a crawl honest: discard any page where word_count is below a threshold, and record the count in your manifest so you can audit what got dropped.
The output is already shaped for downstream use. Because headings are markdown headings, you can split a document into semantic chunks without a tokenizer:
def chunk_by_heading(markdown: str) -> list[dict]:
chunks, current = [], {"heading": None, "body": []}
for line in markdown.splitlines():
if line.startswith("## "):
if current["body"]:
chunks.append({"heading": current["heading"], "text": "\n".join(current["body"])})
current = {"heading": line[3:].strip(), "body": []}
else:
current["body"].append(line)
if current["body"]:
chunks.append({"heading": current["heading"], "text": "\n".join(current["body"])})
return chunks
Each chunk now carries the section it belongs to, which is exactly what you want when you feed a RAG pipeline or an AI agent. Use word_count from the API response to budget tokens before you embed — roughly 1.3 tokens per word — so you can skip or split oversized documents without parsing them first.
| Approach | Setup | Output | Maintenance |
|---|---|---|---|
Raw curl + regex |
None | Messy HTML | Breaks per site |
| Per-site CSS selectors | High | Structured fields | Breaks on redesign |
| Headless browser (Playwright) | High | Rendered HTML | Slow, heavy |
| Markdownify API | One HTTP call | Clean markdown | Site-agnostic |
The trade-off is explicit: you give up pixel-perfect field control in exchange for a scraper that works on any site without maintenance. For content ingestion, documentation harvesting, and feeding LLMs, that's the right trade.
Automated fetching comes with obligations. Respect robots.txt, keep concurrency low, cache so you don't refetch pages you already have, identify your crawler with a clear user agent, and honour the site's terms of service and copyright. Scrape public content you have a legitimate right to use, and cite sources when you redistribute. A scraper that's polite and cache-first is both easier to run and easier to justify.
Scraping with Markdownify collapses a multi-step pipeline — download, parse, clean, convert — into a single GET that returns JSON with clean markdown and a word count. Start with the first curl in this post, use sitemaps and link harvesting to build a queue, then drop the Python crawler in when you need scale. Pair Markdownify with OGSnap for link metadata and the YouTube Transcript API for video content, and your ingestion layer covers text, links, and video with one key.
Check the docs for endpoint details, and if you're building on the output, see our guide to Markdownify best practices for LLM content extraction for chunking and token-budgeting patterns.