This post is the architecture writeup of how we hit sub-1-second effective freshness on shared targets like Bovada with marginal infrastructure cost of $12/month, plus a residential Pixel and a few GitHub Actions runners that are free. It's not a fancy stack. It is a tight one.
Every sportsbook aggregator starts the same way: one VM polls each book at the fastest cadence the book's WAF tolerates. For Bovada that's roughly 2 seconds. For Pinnacle's guest API, similar. For DraftKings, you don't poll directly from a datacenter IP because they return HTTP 403 (they fingerprint your network ASN as not-a-real-customer).
At a 2-second cadence, average freshness is 1.5 seconds (random arrival within the polling window plus a small processing tail). Worst-case is 2.5 seconds. Customers running line-shopping or +EV scanners notice the difference between 1-second and 2-second freshness. The 1-second floor is the prize.
You can't get there by polling faster from one IP. The WAF will rate-limit you, then 403 you. The math says you need more polling sources, not faster ones from a single source.
ParlayAPI runs the following ingest fleet. Numbers are current as of this post.
| Node | Region | Role | Cost / mo |
|---|---|---|---|
| Pixel-4a-01 | US residential | Books that geofence datacenter IPs | $0 |
| gh-actions-01 | GitHub Actions | 1-min cron, ephemeral runner | $0 |
| do-nyc1-01 | NJ/NYC datacenter | US-East poller | $4 |
| do-sfo3-01 | SF datacenter | US-West poller | $4 |
| do-ams3-01 | EU datacenter | EU-region poller | $4 |
| gcp-relay-1 | GCP us-central1 | Outbound proxy for the primary box | already deployed |
Total marginal cost: $12/month. Plus the residential Pixel and the GitHub Actions runners, which don't bill us anything.
Each node holds a 32-byte HMAC secret. When a node fetches a public bookmaker page, it doesn't parse the response locally; it signs and POSTs the raw body to a single ingest endpoint on the collector:
POST /v1/node/ingest
X-Node-Id: do-nyc1-01
X-Timestamp: 1779000000
X-Signature: sha256=<hmac_sha256(secret, "{ts}.{body_sha256}")>
Content-Type: application/json
{
"source": "bovada",
"url": "https://www.bovada.lv/services/sports/event/coupon/events/A/description/baseball/mlb",
"fetched_at_utc": "2026-05-19T22:00:00Z",
"status_code": 200,
"body_b64": "<base64-encoded response>",
"body_sha256": "abc123...",
"fetch_duration_ms": 84,
"error": null
}The collector verifies the signature against a ±60-second skew window, persists the envelope to disk for audit, and a separate consumer task re-runs the source-specific parser on each new file. The node is dumb. The parser is downstream. Adding a new node is mostly minting a new HMAC secret.
When multiple nodes target the same source (e.g. all three DigitalOcean droplets polling Bovada), they shouldn't double-fetch. The coordinator solves this with Redis sorted-set bookkeeping:
rr:bovada:owners HASH node_id -> last_heartbeat_ts
rr:bovada:slot:{i} STRING current owner for slot i (TTL = slot_dur)rr:bovada:slot:{slot_index} with a TTL of slot_dur.Result: with 3 nodes at 3s base interval, the API sees a fresh Bovada read every 1s. When a node drops out (Pixel reboots, GitHub Actions timeout), the remaining nodes recompute slot_dur on the next tick. Effective freshness degrades proportionally instead of going dark.
If all mesh nodes go silent, the collector falls back to its own polling (which never depended on the mesh; the mesh is additive). Effective freshness reverts to the per-source baseline of ~2-3 seconds.
That fail-safe was tested unexpectedly during a recent gateway outage: the reverse SSH tunnel between our primary api host and the public gateway dropped for ~15 minutes. The mesh nodes kept polling and writing to the collector through the entire incident; not one data sample was lost. Customer-visible 502s were entirely a public-facing routing problem, not a data-layer problem. Post-mortem here.
The architecture isn't a secret. It's documented in our public MESH_ARCHITECTURE.md. The competitive moat isn't the design; it's the operational discipline of running it. Every aggregator could deploy this. The ones that don't are choosing convenience over freshness.
Three things we intentionally avoid:
POST /v1/node/ingest on the collector; the collector verifies signatures with a ±60s skew window and the consumer task re-parses each capture.docs/MESH_ARCHITECTURE.md in the public ParlayAPI repository.