polite-crawler
A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.
Description as published in the official MCP registry.
Maintain this server? Get the badge
What we measured
| Endpoint | None listed (runs locally) | registry |
|---|---|---|
| GitHub stars | 0 | github.com/mark1russell7/crawler |
| Last push | 2026-10-13 | GitHub API |
| Licence | MIT | GitHub API |
| Registry version | 0.1.0 · active · updated 2026-10-13 | registry |
| Package | npm: polite-crawler | registry |
Packages
npm: polite-crawler
Exists · metadata read 2026-10-14
Latest version: 0.1.0. Last published: 2026-10-13.
Deprecated: no.
Weekly downloads: unavailable · read 2026-10-14. downloads include bots and CI.
Signed reports
No signed reports yet. A report carries a signed decision record from the reporter's gate, so it shows that the gate allowed a call, not just an opinion; it does not prove which server answered. How to file one.
Connect
npx -y polite-crawlerCommands are built from the registry entry. Check the publisher's documentation before giving any server access to your data.
Related servers
- AI Visibility Audit — Can AI crawlers read and cite your site? One-call audit: robots.txt AI-bot rules, llms.txt, JSON-LD, meta tags, sitemap. Score plus…
- ActableSite AI Crawler Monitor Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt…
- Flarum Forum Scraper - Discussion Crawler Tool Search or monitor any Flarum-powered forum for discussions, replies, participants, dates…
- oassis — web scraping, crawling and browser control for AI agents Scrape to markdown, map and crawl sites, drive a real browser. Pay per call, no API key…
- AI Crawler Index {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}}…
- Crawler IP Verifier - real Googlebot or fake {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}}…
Is this your server?
Show the result of our latest check where your users look first. The badge is a live image: it changes when our check does, and it links back to this page.
[](https://openforallofus.com/tools/io.github.mark1russell7/polite-crawler)<a href="https://openforallofus.com/tools/io.github.mark1russell7/polite-crawler"><img src="https://openforallofus.com/badge/io.github.mark1russell7/polite-crawler.svg" alt="Agora: MCP handshake check" height="20"></a>The badge reports one thing: the result of our last handshake check. A server with no remote endpoint shows as "runs locally". How we check.
Agents can read this page as data: MCP endpoint https://openforallofus.com/api/mcp, tool get_tool with name io.github.mark1russell7/polite-crawler.