Ask an agent to scrape a paginated website and the most common failure is not a parsing error — it is a tidy summary of page one’s thirty items followed by a confident “done.” Two questions hide behind that failure: how do you merge every page’s results into a consistent shape, and how do you reliably know whether more pages exist? Firecrawl’s July 15 tutorial answers both with one minimal loop: turn each page into schema-conforming structured JSON, keep incrementing the page number, and stop when a page comes back with an empty array. The author used it to walk all 61 pages of Hacker News and collect just under 1,040 stories — without ever knowing the total page count in advance.
Page One Is Not “Done”: the Missing Verifiable Stop Condition
The essence of the problem is evidence of completion. Given nothing more than a natural-language instruction like “scrape all the pages,” an agent has no reliable way to know that dozens more pages exist; the list it sees has content and length, and nothing on the surface distinguishes “finished” from “first page.” Hacker News is the canonical example: every page holds a fixed 30 stories, and a single scrape of the same URL always returns the same 30 items. Stopping is therefore not a parsing problem but a process problem — unless there is a programmatically verifiable rule, the agent stays on page one.
Firecrawl’s rule is almost plain: the page that returns zero results is the last page. Its value is that you never need the total page count up front — the count is an output the loop discovers, not an input you have to look up first.
Structured JSON First, Not a Wall of Markdown
A stop condition can only shrink to a one-line check if every page’s output has a fixed shape. Omit formats and scrape defaults to markdown — fine for human eyes, but the page collapses into a wall of text where titles, scores, and comment counts blur together, with no stable fields for a program to read. The deeper difference is that Firecrawl returns structured JSON rather than raw HTML — no site-specific parser per website — and that is exactly what makes “the same loop for any paginated site” a defensible claim.
The json format comes in two strengths: prompt-only, where the model decides the fields, or prompt plus schema, where the shape is guaranteed. For a loop, use the schema — prompt-only output cannot carry a programmatic stop judgment like “check the array length.”
A Schema Shrinks the Stop Condition to One Line
In the Node SDK you define the schema with Zod; it is converted to JSON Schema as the wire format before the request goes out. For Hacker News the shape is a stories array where each item carries title, url, points, and comments. The returned result.json matches the schema exactly — not approximately, but down to field names and types.
With that guarantee, data collection and the stop judgment live in the same loop:
while (true) {
const rows = (await scrapePage(page)).json.stories;
if (rows.length === 0) break;
stories.push(...rows.map((row) => ({ ...row, page })));
page++;
}Each row is stamped with the page it came from before it is pushed. That small habit buys auditability: any record can be traced back to the page where it appeared, which is exactly what you want when reconciling or deduplicating later. When the loop ends, stories is written to stories.json and the task closes with a single file.
The Hacker News Run: 61 Pages, Just Under 1,040 Stories
Hacker News pagination is as simple as the pattern gets: the ?p= parameter in the URL increments with the page number, and every page holds a fixed 30 stories. Wire the loop from the previous section to that rule and you have the tutorial’s complete implementation. In the author’s own test run, the loop walked all 61 pages and collected just under 1,040 stories — and 61 was never configured; the loop ran until the first page returned an empty array, and 61 is simply the last content page it completed.
The part worth pausing on is that the page count is found, not assumed. Any run that peeks at the first few pages and decides “this is probably all of it” is guessing, and on a paginated site a wrong guess is indistinguishable from a truncated dataset. Handing completion over to a rule gives the agent, for the first time, evidence of “done” that can be checked.
Offset Pagination: Just Swap the Increment
Not every site paginates with page numbers. Older forums and directories commonly use offset pagination: a start parameter that increments by the page size — start=0, start=25, start=50. The rewrite is minimal: replace page++ with start += PAGE_SIZE, and the stop condition — empty results means stop — does not change at all.
That points to the more general layer of the tutorial: the same loop applies to any paginated site, and only two things ever change — how the URL is assembled and which fields the schema carries. The stop condition is shared across every paginated site; site-specific differences stay quarantined inside the URL and the schema, which is why this pattern is worth lifting straight into production code.
Quotas and Boundaries: When This Loop Is the Wrong Tool
Get the quota rules straight first. The scrape endpoint can be used keyless: through an official client — SDK, CLI, or MCP server — no API key is required, subject to a per-IP rate limit. crawl, extract, and map, however, require an API key, and registering in the dashboard unlocks those endpoints and raises the limits. In other words, the single-page loop in this post can run anonymously, but anything site-wide treats registration as a prerequisite.
Choosing the right entrance matters more. The tutorial draws the division of labor explicitly: to traverse an entire site, use /crawl — link discovery and traversal are built-in capabilities, never a manual loop’s job. For pagination where the URL does not change — infinite scroll or click-to-load — use interact, which holds a real browser session where the data only appears after actions. The loop here fits list pages whose URL changes with a page number or an offset; three situations, three entrances, and forcing this loop onto the other two is the wrong design.
Sources
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.