Not every site gives up its data the same way. Some are server-rendered and crawl over plain HTTP (wget); some hydrate with JavaScript or sit behind an anti-bot challenge and need a real browser (browser); some are blocked outright or behind a hard paywall (blocked). This map classifies real sites across four verticals and six countries, so you can see the cost of a pipeline before you build it. The crawlability of a site tracks its business model — ad-supported pages open up for SEO, subscription pages lock down — more than its country.
Snapshot from exploratory probes; verdicts marked for reconfirmation from a production crawler node are refined over time. Need a specific site assessed? Talk to us.
Loading crawlability data…