Proxies for SEO tools: how many addresses Screaming Frog, A-Parser and Key Collector burn

Most SEO desktop tools accept a proxy list, and most people paste the first free list they find. The crawl runs for twenty minutes, the tool reports captchas and timeouts, and the report is half empty. This page counts what each tool needs and where a free list is enough.

Where the addresses go

Two kinds of load hide behind the word "SEO tool". A site crawler such as Screaming Frog or Netpeak Spider hits one domain thousands of times. A SERP tool such as Key Collector, A-Parser or Rank Tracker hits Google and Yandex, which count requests per address and answer with a captcha after a few dozen. The first job needs a handful of addresses and patience. The second needs volume.

Site crawlers

Screaming Frog and Netpeak Spider crawl one site at a time. Ten addresses at two requests a second each finish a 50,000-page site in about forty minutes without a block from anything short of Cloudflare's strictest mode. A free list works here, if you check it right before the run, because a crawl that loses half its addresses mid-way still finishes, only slower.

Sitebulb and SEO PowerSuite's WebSite Auditor behave the same way. Set the pause to half a second, cap the threads at the number of addresses, and let it run.

SERP tools

Key Collector, A-Parser, Rank Tracker and ScrapeBox ask search engines directly, and search engines are the one target where a public list fails inside the hour. Google limits an address to somewhere near fifty queries a day before the captcha. Yandex is stricter. A project of 2,000 keywords checked in two engines to a depth of three pages is 12,000 queries, which at fifty per address is 240 addresses that all survive the day. A free list of a hundred gives you twenty that work, and half of those are already flagged by Google because everyone else pulled the same list.

At that volume a pool is the only thing that holds. RemProxy has a page for SEO software with the tools listed by name, a thread calculator and a sample that sorts the pool by speed: a pool for SEO software. Thousands of addresses, rotated so that each one touches Google a handful of times a day.

Tools that need no proxies

Ahrefs, Semrush, Serpstat and Topvisor run on their own infrastructure. You pay for their data, and a proxy list does nothing for them. The same goes for Google Search Console and Yandex Webmaster. Buy addresses only for the tools that run on your machine.

Settings that save the list

• Threads equal to the number of addresses, never more. Two threads on one address is the fastest way to a ban.
• Pause between requests to the same engine from the same address: ten seconds for Google, fifteen for Yandex.
• Ban timeout of ten minutes instead of removal. Most search-engine blocks lift within the hour.
• Rotation per request for SERP tools, per session for crawlers that keep cookies.
• A test run of a hundred queries before the full project, to see the failure rate per address.

Grab the current list from the homepage for a crawler. For a rank check of any size, do the math above first.