What Changed: Web Scraping Becomes a Single API Call
Web scraping has gotten harder. Pages render through JavaScript, throttle automated traffic, and serve different markup depending on the user’s region. Even when you do get the HTML, it’s rarely the shape you wanted—so most projects bolt on a cleanup pipeline to reach clean text, a structured record, a transcript, or a screenshot.
Firecrawl’s /scrape endpoint collapses those steps into a single API call. You give it a URL and specify the desired formats, and it renders the page in a real browser on the server side—handling JavaScript, proxies, redirects, and caching—then returns the result in the shape you asked for. This article walks through the endpoint end to end: formats, parameters, costs, structured extraction, batch operations, and interactive pages.
How It Works: Formats, Pricing, and Caching
Every scrape starts at 1 credit. A few options increase the cost:
| Feature | Credit Cost |
|---|---|
| Base scrape (any single format) | 1 |
JSON extraction (formats: [{type: "json"}]) |
+4 (5 total) |
Enhanced proxy (proxy: "enhanced") |
same 1 credit — no surcharge |
PDF parsing (parsers: ["pdf"]) |
+1 per PDF page |
Audio extraction (formats: ["audio"], REST only) |
+4 (5 total) |
| Cached result | 1 (caching saves time, not credits) |
As the official documentation notes, caching reduces latency but does not reduce the bill. The max_age parameter controls the cache window in milliseconds—pass 0 to force a fresh scrape, or 86400000 for a one-day window. The response’s cache_state field tells you whether it was a hit or miss.
Requesting Multiple Formats
You can request several formats at once. Here’s a Python example that gets markdown, HTML, raw HTML, links, screenshot, summary, and images from arXiv.org:
from firecrawl import Firecrawl
from dotenv import load_dotenv
load_dotenv()
app = Firecrawl()
url = "https://arxiv.org"
data = app.scrape(
url,
formats=['html', 'rawHtml', 'links', 'screenshot', 'summary', 'images']
)Each format serves a different purpose:
markdown: Clean Markdown of the main page content—ideal for LLM pipelines.html: Cleaned HTML with scripts and styles stripped.rawHtml: Unmodified HTML as received from the server.links: Flat list of all hyperlinks on the page.screenshot: Signed PNG URL of the rendered page (supportsfullPage).summary: AI-generated 1–3 paragraph summary.images: All image URLs.json: Structured data extracted by an LLM (covered next).audio: MP3 extracted from supported video URLs (REST only).
Structured Data Extraction: Prompt vs. Schema
The json format runs an LLM over the page and returns a structured dict. This is where the /scrape endpoint shines for product builders who need to extract specific fields—product names, prices, reviews—without writing custom parsers.
Prompt-Only Mode
Pass a natural language prompt, and the LLM chooses the output shape. Fast to write, but field names can drift between runs. Example pulling top stories from Hacker News:
url = "https://news.ycombinator.com"
data = app.scrape(
url,
formats=[
"markdown",
{
"type": "json",
"prompt": "Extract the top 3 stories on the page. For each story, include its title, the URL it links to, the points score, and the author who submitted it."
}
]
)
print(data.json)The output might look like:
{
"stories": [
{
"title": "BYOMesh – New LoRa mesh radio offers 100x the bandwidth",
"url": "https://example.com/story1",
"points": 209,
"author": "nullagent"
}
]
}Schema-Driven Mode
For production, pass a Pydantic model to lock field names and types. This guarantees every run returns the exact same shape. The Firecrawl blog demonstrates nested schemas:
from pydantic import BaseModel, Field
class IndividualArticle(BaseModel):
title: str = Field(description="The title of the news article")
subtitle: str = Field(description="The subtitle")
url: str = Field(description="The URL")
author: str = Field(description="The author")
date: str = Field(description="The publication date")
read_duration: int = Field(description="Estimated reading time in minutes")
topics: list[str] = Field(description="List of topics")
class NewsArticlesSchema(BaseModel):
news_articles: list[IndividualArticle] = Field(description="Extracted articles")
data = app.scrape(
url,
formats=[
{
"type": "json",
"schema": NewsArticlesSchema,
"prompt": "Extract the top 5 stories as news articles."
}
]
)
print(data.json)Both modes cost 5 credits (1 base + 4 for JSON). Use prompt-only for exploration; switch to schema when downstream code depends on field names.
Practical Use Cases: Batch, Actions, and Specialized Formats
Batch Scraping
When you need hundreds or thousands of URLs, use batch_scrape() (sync) or start_batch_scrape() (async). Submit multiple URLs, then poll or set a callback for results. Ideal for SEO audits, price monitoring, or content aggregation.
Interactive Pages with Actions
Some pages require user interaction before revealing content. The actions parameter lets you define a series of steps—click, type, wait, screenshot—that Firecrawl executes on the rendered page. The official tutorial shows a login example: navigate to a test site, fill credentials, click login, then capture the post-login screenshot.
Specialized Formats
- Audio: Extract MP3 from YouTube links (REST only, +4 credits).
- PDF Parsing: Parse PDF pages with
parsers=["pdf"]; each page costs +1 credit. - Screenshot: Capture full-page screenshots with
fullPage: true. - Branding: Extract brand colors, fonts, and style from a page.
- Question and Highlights: Answer specific questions or return relevant passages—reduces token consumption up to 100x.
Location and Proxy Controls
The location parameter lets you request from a specific country (e.g., "country": "DE" for German results). The proxy parameter has three modes: basic (default, cheapest), enhanced (stronger anti-blocking), and auto (basic first, falls back to enhanced only if needed). The proxy_used field in metadata tells you which backend ran.
Limitations and Trade-offs
No tool is perfect. Here are the key limitations to consider:
- Caching costs credits: Even cached results cost 1 credit. Caching saves latency, not money. Use
max_ageto balance freshness vs. speed, but know that every hit still counts. - Prompt drift: Without a schema, field names can change between runs. For production extraction, always define a Pydantic model.
- Audio is REST-only: The audio format cannot be used via SDKs—you must call the REST API directly.
- Interactive complexity: While
actionshandle simple workflows (click, type, wait), complex multi-step flows (e.g., sign-in with CAPTCHA) may require custom session handling outside Firecrawl. - Lockdown Mode: For security-sensitive environments, set
lockdown=Trueto force cache-only scraping. If the URL isn’t cached, it errors rather than fetching live—useful for strict outbound controls. - PDF page costs: Each PDF page consumes an extra credit. Extracting a 50-page PDF costs 51 credits (1 base + 50 pages).
Concrete Takeaway: How to Get Started
If you’re new to Firecrawl, the easiest path is the REST API via cURL:
curl -X POST https://api.firecrawl.dev/v2/scrape \
-H "Authorization: Bearer $FIRECRAWL_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "formats": ["markdown"]}'For application code, use the Python or Node SDK as shown above. If you live in an MCP-compatible client like Claude Desktop or Cursor, install the Firecrawl MCP server and let the model call firecrawl_scrape directly—no code needed.
Key decisions for product builders:
- One-off exploration → prompt-only JSON with cURL.
- Production data pipeline → schema-driven extraction via SDK.
- High volume → batch scraping with
start_batch_scrape(). - Interactive pages → use
actionsparameter. - Cost-sensitive → avoid unnecessary JSON extraction (it’s 5x the base cost) and use caching to reduce latency, even if credits are still consumed.
Firecrawl’s /scrape endpoint isn’t a silver bullet—you’ll still need custom logic for highly complex flows or strict compliance requirements—but for the vast majority of “give me clean data from this URL” tasks, it has lowered the barrier significantly. Start with a simple scrape, add formats as needed, and graduate to schemas when your pipeline demands consistency.
Sources
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.