Data & research

Agentic Business Scraping

A crew of agents that goes and finds information on businesses — who exists, where, in what trade, who runs it and how to reach them — by discovering the directories that hold it, reading only what each one permits, and pulling it into a single indexed database with a source attached to every field.

6 collection agents Missouri + 8 states robots.txt gated Source on every field

See it work

The problem

There is no single list of the small businesses in a region, and certainly not one that says who owns them or who is close to selling up. That information exists, but it is scattered across hundreds of small, inconsistent directories, each with its own markup and its own rules about what a scraper may touch.

Missouri and its eight neighbours means hundreds of separate chamber-of-commerce and business directories, each one its own scraping problem, and each one liable to change its front-end without notice. Writing a scraper per site by hand does not finish; that is the job the agents exist to do.

The harder problem isn't finding the data — it's trusting it. A wrong phone number or a falsely merged business record is worse than an empty field, and at this scale nobody can check every row by hand.

What it does

Business index

Every business found across Missouri and its eight neighbours — IA, IL, KY, TN, AR, OK, KS, NE — filterable by state, category and succession score.

Owner candidates

Candidates surfaced by agents as likely nearing a sale or retirement, each one carrying a source and a confidence grade.

Provenance on every row

Click a business and see field-by-field provenance — where a phone number, an address or an owner name actually came from. A missing field shows NULL rather than a guess.

Collection agents

Six Claude Code subagents cover the collection phases — among them source-discovery, tos-reviewer, chamber-scraper and entity-resolver — parallelising within a stage, deliberately not across stages. How that runs is below.

How the agent scraping works

The point of doing this with agents rather than a hand-written scraper is that nobody has to write a scraper per site, and nobody has to notice when a site changes. The crew runs in stages, and each stage is a different agent doing one job. Agents run in parallel within a stage — many sources at once — but never across stages, because a later stage is always reasoning about what an earlier one produced, and overlapping them means resolving records that are still arriving.

1 · An agent goes looking for the sources

A source-discovery agent hunts for the directories that list businesses in a given area — chambers of commerce, trade bodies, local business listings — instead of being handed a list of URLs by a person. Each one it finds is registered as a data source, so every row pulled later can be traced back to where it came from. 191 sources are registered so far.

2 · An agent checks what each source allows

Before anything is read, a terms-of-service agent checks the site's robots.txt and the source is gated on that, and only on that. 43 of the first 91 chamber sources publish a blanket Disallow: / and are never scraped — no workaround, no exception. That constraint costs coverage and is kept anyway.

3 · Scraping agents read the pages

A scraping agent works out the shape of each site for itself rather than following a selector somebody hard-coded. Two different directory front-ends are detected per source, because they share almost no markup. Where a selector returns nothing, the column is written NULL — the agent never fills a gap with a plausible-looking guess, because a wrong phone number is worse than a missing one.

4 · An agent checks the haul against what the source claimed

One of the two front-ends publishes its own result count, so parsed-versus-reported can be verified exactly. The other doesn't, so a category returning exactly 50 rows gets logged as a count mismatch rather than assumed complete. Every load is checked against a scrape-run log, which is how a truncated pull gets caught instead of quietly becoming a hole in the database.

5 · An entity-resolver agent works out who is who

The same business turns up in four directories under four slightly different names. An entity-resolution agent matches those into one record — but two business records are never auto-merged. Likely duplicates go to a review table with a pending status for a human to confirm, because a false merge silently drops a business from the pipeline and nobody ever notices. That checkpoint is deliberate, not a gap.

6 · Agents surface the candidates, and show their working

From the assembled records, agents surface the owner candidates worth a call — 4,861 of them so far — each carrying a source and a confidence grade rather than a bare name. Every field on every row keeps the source it came from, so clicking a business shows field-by-field provenance instead of a number you are asked to take on trust.

The data discipline is the actual product.

An agent crew that cuts corners at this scale produces a database nobody can act on, because no single row can be trusted and no human can check 56,655 of them by hand. So the rules above — NULL over a guess, no silent merge, robots.txt obeyed, every load reconciled — are not politeness. They are the thing that makes the output worth having.

The numbers

What we'd do next

The review table for likely-duplicate records keeps growing faster than a human can clear it by hand. That queue, and widening coverage past the 43 chambers that currently block scraping under robots.txt, is where the next stretch of work goes.