A site-adaptive crawler holding about a 95% success rate across mixed, hostile sources.
The extraction layer underneath the job pipeline, kept separate because the problem generalises. It fingerprints a target before crawling it and routes to the matching strategy: Scrapy for static HTML, Playwright or Selenium for JavaScript-rendered pages, direct endpoint reads for paginated APIs.
Proxy rotation, rate limiting, and exponential-backoff retries keep it alive against sources that would rather it were not. It holds about a 95% success rate across its targets, and a deduplicating ETL layer normalises everything into PostgreSQL.