Back to Case Studies
Security

AI Crawlability Checker: Auditing Any Website Without Opening an SSRF Hole

A free tool that tests whether 16 AI crawlers can reach and read a site. The hard part was letting strangers make our server fetch arbitrary URLs safely.

Executive Summary

  • Challenge: Site owners could not tell whether ChatGPT, Claude or Perplexity could read their pages. A checker has to fetch any URL a visitor types in, which is a textbook server-side request forgery (SSRF) risk.
  • Solution: A zero-dependency Node.js proxy that validates the resolved IP address inside the socket's own DNS lookup, re-checks every redirect, and caps time, size and rate.
  • Outcome: A public AI Crawlability Checker that reports robots.txt rules, CDN blocks and JavaScript-only content for 16 AI crawlers in one pass.

The Challenge

AI assistants now send real traffic, and they decide what to cite by what their crawlers can fetch. Yet most site owners have no idea which AI bots their robots.txt allows, whether their CDN silently blocks them, or whether their content exists before JavaScript runs. Those are three different failures, and each needs a different fix.

Answering those questions means our server fetches a URL chosen by an anonymous visitor, around twenty times per audit, using different user-agents. Done naively, that lets anyone use the tool to reach internal services, cloud metadata endpoints, or the private network the server runs on.

Constraints & Requirements

  • •Accept any public URL from an anonymous visitor
  • •Never reach private, loopback, link-local or metadata addresses
  • •No third-party dependencies in the fetch path
  • •Clear results for non-technical site owners, with a fix for every failed check

System Considerations

What had to be true

  • — Per-crawler requests with each bot's real user-agent, compared against a normal browser request
  • — robots.txt evaluation per crawler, separating training, search and user-triggered bots
  • — Raw-HTML analysis: visible word count, H1, text-to-HTML ratio, landmarks and JSON-LD

Non-negotiables

  • — IP validation must happen at connect time, not before it
  • — Every redirect hop is validated as if it were a new request
  • — Hard limits on response size, response time, redirects and request rate

Architecture Approach

The server uses only Node.js built-ins: http, https, dns, net and zlib. The common SSRF mistake is to resolve the hostname, check the IP, and then make the request, which resolves the name again. An attacker-controlled DNS server can answer with a public IP the first time and 127.0.0.1 the second (DNS rebinding). We pass a custom lookup function to the request itself, so the address that gets checked is the address the socket connects to.

The block list uses net.BlockList and covers loopback, RFC 1918 private ranges, carrier-grade NAT, link-local (including the 169.254.169.254 cloud metadata address), benchmarking and multicast ranges, plus IPv6 unique-local, link-local and NAT64. Only http and https on ports 80, 443, 8080 and 8443 are allowed, and URLs with embedded credentials are rejected. Redirects are followed manually, up to five, and each hop goes through the same checks.

Responses are decompressed as a stream and cut off at 3 MB. Requests time out after 10 seconds, and each IP is limited to 120 proxied fetches per minute, roughly six full audits. The static site sends a strict Content Security Policy with no third-party scripts.

To detect CDN blocks, the tool fetches the page once as a normal browser and once per crawler with that crawler's published user-agent. A crawler counts as blocked if it gets an error status the browser did not, or a challenge or CAPTCHA page the browser did not. We wrote up what the results mean in Training Bots vs. Answer Bots.

Trade-offs & Decisions

Prioritized

  • Safety of the fetch path over coverage of unusual ports and protocols
  • Zero dependencies, so there is no supply-chain surface in the request path
  • Plain-language results with a concrete fix for every check

Intentionally Not Optimized

  • Verified-IP testing: requests come from our server, so CDNs that admit bots only from published IP ranges may treat the real crawler differently. The report says so.
  • Distributed rate limiting: limits live in process memory, which is enough for a single instance
  • JavaScript rendering: we deliberately analyse raw HTML only, because that is what most AI crawlers see

Outcome

The checker is live and free at aicrawlcheck.mecverse.com. It shares its hardened fetch layer with our SEO and link preview checker, so a fix in one tool reaches both.

Each failed check comes with a specific fix, whether that is a robots.txt rule, a CDN bot setting, or moving content into server-rendered HTML, so a site owner can act on the report without a developer translating it.

16 AI crawlers checked in one audit

0 third-party dependencies in the fetch path

Every redirect hop re-validated, up to 5

If your product fetches URLs for users, treat it as a security feature first and a product feature second.