A distributed scraper across 50+ job portals that turns 1,000+ raw listings into 500-750+ deduplicated leads a day.
Fifty-odd company job portals, no two built the same way: static HTML, JavaScript-rendered SPAs, paginated JSON APIs. The crawler detects which it is dealing with and dispatches the right strategy, so adding a source does not mean writing a new scraper.
Volume was never the hard part. Deduplication was. The same role gets posted to a company board, an aggregator, and a recruiter feed under three different titles, and counting it three times makes the output useless. A three-layer Redis dedupe collapses 1,000+ daily listings down to 500-750+ genuinely distinct leads, which PostgreSQL then tracks through their lifecycle. The Claude API scores each one for relevance and drafts the outreach. APScheduler runs the whole cycle unattended.