Web Technician Online

robots.txt fetch checker

Fetch a site’s robots.txt for inspection.

Submit public URLs only. Do not include login tokens, passwords, private links or customer information. This performs a bounded GET request from our server; it does not log in to the target.

Result

Your result will appear here.

On submission, this URL goes to our server, which sends a bounded GET request to the public target and queries Google Public DNS for its IPv4 address. No credentials or browser cookies are forwarded.

What this tool can tell you

This tool uses the site’s server checker to obtain a real HTTP response from a checked and pinned public IPv4 address. HTTPS hostname and chain validation stay enabled. Private/reserved addresses and unusual ports are rejected; IPv6-only endpoints are not supported. A rate or connection limit is shown as a limit or failure, never a fabricated diagnosis.

The status/header/certificate checks inspect the first response using GET. Redirect and robots checks follow at most five redirects within a 20-second total budget. Headers are limited to 32 KB; robots text to 512 KiB. Response cookies are omitted. TLS trust does not certify overall security or revocation status.

Shows observed responses and limits. A failed connection from this server does not prove the target is unavailable to everyone.