Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.
An MCP (Model Context Protocol) server that provides tools for parsing, analyzing, and generating robots.txt files with a focus on AI bot detection. ------|-------------| | Fetch and analyze a robots.txt from any URL. Shows which AI bots are blocked/allowed. | | Analyze pasted robots.txt content for AI bot blocking status. | | Generate a robots.txt with specified blocked bots, sitemaps, and…
Verification confirms publisher identity (repo ownership), not code safety. The security scan covers known CVEs and suspicious install scripts — it cannot prove the absence of malicious code.
An MCP (Model Context Protocol) server that provides tools for parsing, analyzing, and generating robots.txt files with a focus on AI bot detection. ------|-------------| | Fetch and analyze a robots.txt from any URL. Shows which AI bots are blocked/allowed. | | Analyze pasted robots.txt content for AI bot blocking status. | | Generate a robots.txt with specified blocked bots, sitemaps, and custom rules. | | List all known AI bots with user-agents, companies, and descriptions. | | Check if a…