How we turn an announcement into a job estimate

1. How we find projects

We poll a source registry (config/sources.yaml) rather than scraping arbitrarily — adding a new source is a YAML entry, not new code. The registry currently mixes three kinds of feeds:

• Trade press RSS — Data Center Dynamics, Data Center Frontier, Construction Dive, Utility Dive, Semiconductor Digest, and keyword-filtered wire services (PR Newswire, Business Wire).
• GDELT DOC 2.0 — a broad local-news net — a keyword query (“data center,” “chip fab,” “transmission line,” etc.) run against GDELT's global news index, with a Massachusetts focus during the pilot (e.g. “Holyoke data center,” “Westfield hyperscale,” “ISO New England transmission”).
• Newsroom scrapers — hyperscaler and chipmaker newsrooms without RSS (Meta, Google, Microsoft, AWS, Oracle, NVIDIA, TSMC, Intel) plus state economic-development sites, using registry-driven CSS selectors so a new site is configuration, not code.

We store the raw article text as collected and only filter/classify it downstream — nothing is discarded at ingest time because it looked irrelevant.

2. How facts get extracted

Each stored article is run through an LLM extraction step that pulls structured fields — company, project type, location, capex, power (MW), phase, and any job numbers the announcement itself claims — along with a confidence score for the extraction. Low-signal or non-announcement articles (e.g. unrelated news that merely mentions a keyword) are filtered out at this stage, not at ingest.

3. How duplicate mentions become one project

The same project is usually announced, re-announced, and covered by multiple outlets. We block candidate matches by county, then score them on a composite 0–1 basis blending company fuzzy match, project-name fuzzy match, and capex proximity (config/dedup.yaml):

Capex counts as a full proximity match within a 20% relative difference, decaying linearly to zero at 3x that tolerance (covering typical rounding across re-announcements, e.g. “$10B” becoming “$10.2B” in a later article). Missing fields on either side drop that component and renormalize the remaining weights, rather than penalizing to zero — many announcements omit a project name.

Scores ≥ 0.90 auto-merge into the existing project. Scores in [0.65, 0.90) are ambiguous — a new project is still created so the data isn't lost, but the match is queued to a review queue for human/LLM adjudication. Scores below 0.65 are treated as a genuinely distinct project. The thresholds are deliberately conservative: a false auto-merge silently corrupts the dataset, while a missed merge just creates a duplicate that review can catch later.

4. How job demand is estimated

Every demand number on this site carries a basis: claimed or modeled.

• Claimed figures come straight from the announcement itself — the largest claimed construction or permanent job count across every article linked to a project. Claimed always takes precedence over modeled when it's available.
• Modeled figures are computed when no claim exists, from two inputs: capex (via a peak-workers-per-$1B ratio) for construction-phase headcount, and power in MW (via a staff-per-100MW ratio) for operational-phase headcount. A phase curve then scales the construction peak down for earlier phases (announced → permitted → groundbreaking → under construction), and the total is split across specific occupations (SOC codes) using a per-project-type trade mix.

The ratios below are versioned and cited in config/ratios/v1.yaml. Entries marked NEEDS_HUMAN_SOURCING are extrapolated beyond what the working spec states outright and are flagged for replacement with real BLS/industry sourcing before being treated as authoritative — we show them anyway, with the flag, rather than hide the assumption.

Occupation split (hyperscale data center — construction phase)

Each phase's total headcount is divided across occupations by fixed fractions that sum to 1.0. The hyperscale-datacenter split (also reused, with small variations, for colo and the “other” fallback) is modeled on the spec's named occupations plus standard heavy-commercial-construction crew composition — NEEDS_HUMAN_SOURCING, not independently cited:

Known simplification: the operational-phase (permanent staff) headcount is split using the same construction trade-mix proportions as the build phases, because the ratio config doesn't yet have a distinct operational-occupation split. In reality, construction crews and a facility's permanent staff have different compositions — electricians dominate a build, while a data center's permanent staff skews toward technicians and network administrators. We flag this rather than silently assume it away; a future ratio-config revision will add a dedicated ops split.

5. Honest limitations

• PR job claims are often inflated. Independent research puts real job effects at roughly a third of the naive headline. That's why we show claimed and modeled side by side, with sources, rather than trusting either alone.
• Location precision varies. Geocoding uses the Census gazetteer/geocoder; it resolves full street addresses precisely, but many announcements only name a city or county, so the best available precision is often a county centroid, not an exact site. Some projects have no coordinates at all yet and simply won't appear on the map.
• Modeled ratios are directional placeholders where marked NEEDS_HUMAN_SOURCING above — seeded from the working spec, not yet independently verified against BLS or industry data for every project type.
• These are planning signals, not guarantees. A job estimate reflects an announcement's claimed intent or a modeled projection at a point in time — not a hiring commitment, and not a substitute for confirming an actual opening.