← Back to blog

YouTube Data Extraction Tutorial for Data Scientists: Build Datasets from Video Transcripts & Metadata

YouTube is the largest spoken-language corpus ever recorded — and one of the most underused data sources in applied machine learning. Every minute, hundreds of hours of tutorials, conference talks, lectures, and commentary are uploaded, and most of it exists in a form that is notoriously difficult to turn into a dataset: video. Text is easy to scrape, tokenize, and embed. Video is not. That gap is exactly why so few public datasets are built from YouTube content, despite the platform's scale.

For a data scientist, a YouTube video is really two datasets waiting to be separated:

1. The transcript — a timestamped text corpus, ready for NLP, topic modeling, and LLM fine-tuning.
2. The metadata — title, channel, upload date, view count, and duration, which give you structured features for regression, forecasting, and classification tasks.

The friction has always been extraction. The official YouTube Data API v3 requires OAuth setup and has quota limits measured in units. Scraping caption tracks yourself means parsing protobuf payloads that break whenever YouTube changes its frontend. The FetchAPI suite removes that friction: two plain HTTP GET endpoints — the YouTube Transcript API and the Video Info API — return clean JSON with no OAuth, no SDK, and no quota math. The free tier works directly from curl.

This tutorial walks through building a real, analysis-ready YouTube dataset: pulling metadata and transcripts, merging them into tidy data with pandas, and scaling the pipeline to hundreds of videos.

Step 1: Pull video metadata with the Video Info API

The GET /v1/info endpoint accepts a YouTube URL or video ID and returns the video's metadata as JSON. Here's the simplest possible call:

curl -s "https://fetchapi.tech/v1/info?url=https://www.youtube.com/watch?v=dQw4w9WgXcQ" | jq

Response:

{
  "video_id": "dQw4w9WgXcQ",
  "title": "Rick Astley - Never Gonna Give You Up (Official Video) (4K Remaster)",
  "channel": "Rick Astley",
  "channel_url": "https://www.youtube.com/channel/UCuAXFkgsw1L7xaCfnd5JJOw",
  "duration_seconds": 213,
  "duration_formatted": "3:33",
  "view_count": 1812505285,
  "upload_date": "20091025",
  "thumbnail": "https://i.ytimg.com/vi/dQw4w9WgXcQ/maxresdefault.jpg",
  "has_transcript": true
}

For dataset building, the important fields are upload_date (ISO YYYYMMDD, so it sorts correctly as a string), view_count (a target variable if you ever want to predict popularity), and has_transcript (a filter flag so you can skip videos with no captions before calling the transcript endpoint).

Step 2: Pull the transcript as timestamped JSON

The GET /v1/transcript endpoint returns the full caption track as an array of timestamped segments:

curl -s "https://fetchapi.tech/v1/transcript?url=https://www.youtube.com/watch?v=dQw4w9WgXcQ" | jq '.transcript[0:2]'

Response:

{
  "transcript": [
    {
      "start": 21.79,
      "end": 25.95,
      "text": "We're no strangers to love. You know the rules and so do",
      "start_formatted": "0:21",
      "end_formatted": "0:25"
    },
    {
      "start": 25.95,
      "end": 29.11,
      "text": "love. You know the rules and so do I. I feel commitments from what I'm",
      "start_formatted": "0:25",
      "end_formatted": "0:29"
    }
  ],
  "video_id": "dQw4w9WgXcQ",
  "title": "Rick Astley - Never Gonna Give You Up (Official Video) (4K Remaster)",
  "duration_seconds": 213,
  "view_count": 1812505285,
  "upload_date": "20091025"
}

Two details matter for research use. First, the segment boundaries come straight from the caption track, so start and end give you sub-second alignment between text and audio — useful if you ever want to build a speech-text benchmark or retrieve the audio snippet around a quote. Second, the same response also carries the metadata fields, so one call per video gives you both datasets. If you only need raw text for a language model, pass &format=text and the API returns the joined plain-text transcript instead of segments.

Step 3: Merge everything into a tidy dataset with pandas

The cleanest research format is tidy data: one row per caption segment, with metadata columns repeated per video. Joining the two endpoints on video_id gives you exactly that. Here's a compact Python pipeline:

import requests
import pandas as pd

BASE = "https://fetchapi.tech/v1"

def fetch_video(video_id: str) -> dict | None:
    """Return metadata + transcript rows for one video."""
    meta = requests.get(f"{BASE}/info", params={"url": video_id}, timeout=15).json()
    if not meta.get("has_transcript"):
        return None
    tr = requests.get(f"{BASE}/transcript", params={"url": video_id}, timeout=30).json()
    rows = []
    for seg in tr["transcript"]:
        rows.append({
            "video_id": meta["video_id"],
            "title": meta["title"],
            "channel": meta["channel"],
            "upload_date": meta["upload_date"],
            "duration_seconds": meta["duration_seconds"],
            "view_count": meta["view_count"],
            "start": seg["start"],
            "end": seg["end"],
            "text": seg["text"],
        })
    return rows

# One video first, to validate the schema
rows = fetch_video("dQw4w9WgXcQ")
df = pd.DataFrame(rows)
print(df[["video_id", "start", "text"]].head())

Once the schema looks right, scale to your full sample — a list of video IDs from a channel, a playlist, or a search you've curated:

video_ids = ["dQw4w9WgXcQ", "jNQXAC9IVRw", "kJQP7kiw5Fk"]  # your sample

frames = []
for vid in video_ids:
    rows = fetch_video(vid)
    if rows:
        frames.append(pd.DataFrame(rows))
    time.sleep(1)  # be polite: 1 req/s is plenty for research batches

corpus = pd.concat(frames, ignore_index=True)
corpus.to_parquet("youtube_corpus.parquet", index=False)

Storing as Parquet keeps the timestamps, text, and metadata in one typed file that loads in milliseconds later — much nicer for iteration than re-hitting the API every time you start a notebook.

What to do with the dataset

Once you have a transcript corpus joined to metadata, a wide range of analyses opens up:

• NLP over talks and lectures. Run topic modeling or embedding-based clustering over conference-talk transcripts to map the landscape of a field — no manual curation required. Timestamps let you cite the exact minute a claim appears.
• Clickbait detection. You now have both the title and the actual content. Train a classifier that predicts whether a title's claims are supported by the transcript — a genuinely useful dataset that doesn't exist off the shelf.
• Popularity forecasting. view_count as a target, with title length, channel, upload date, duration, and transcript lexical features as predictors. Great for teaching regression with real, messy data.
• Longitudinal language analysis. Because every video carries an upload_date, you can track vocabulary, speaking rate, or readability over time for a channel or topic area.
• LLM fine-tuning corpora. Transcripts are cleaner than most web-scraped text: no navigation boilerplate, no cookie banners, one speaker stream. Just filter carefully and respect the uploader's license.

Scaling and robustness tips

A few habits keep the pipeline stable past the first hundred videos:

• Filter before you fetch. Use the has_transcript flag from /v1/info to skip caption-less videos — music videos and vlogs often have none, and calling /v1/transcript on them wastes a request.
• Cache aggressively. Save each video's raw JSON to disk (raw/{video_id}.json) before you parse it. When you change your analysis, you reparse locally instead of re-downloading.
• Retry with backoff. Any public API can return a transient 5xx. A two-line retry wrapper with time.sleep(2 ** attempt) makes batch jobs resilient.
• Resume, don't restart. Write successful video_ids to a done.txt and skip them on the next run, so a mid-batch failure doesn't force a full re-fetch.

Ethics and terms of service

YouTube data work comes with obligations. FetchAPI only serves public videos with captions, and you should keep it that way: don't use it to bypass privacy settings or paywalls, respect the uploader's copyright and license when redistributing or fine-tuning on transcripts, cite your sources in any released dataset, and stay within the platform's terms of service. A good rule of thumb: if your dataset would embarrass the creator if they read the "how it was built" section, rethink the design.

Wrap-up

With two GET requests you can turn any YouTube video into analysis-ready data: /v1/info for metadata and features, /v1/transcript for timestamped text. No OAuth dance, no quota spreadsheet, no brittle scrapers — just JSON you can load straight into pandas. The free tier is enough to prototype a real corpus today, so the fastest way to start is to run the first curl in this post and see what your own data looks like.

Next time you need a text corpus, skip the generic news scrape and try mining YouTube — the data is richer, timestamped, and (thanks to the caption track) already transcribed. Check the docs for endpoint details, and explore the rest of the FetchAPI suite — Markdownify for web content, OGSnap for link metadata, and DiffCheck for change tracking — when your research pipeline needs web data too.