polite-crawler

by GitHub Actions · Search & Web · MCP server

polite-crawler is an MCP server by GitHub Actions. A polite web crawler for research and AI agents. It fetches a source once, keeps it, and reads it many times: robots.txt obeyed, one request per site, full-text search. The MCP server is the package polite-crawler-mcp. Its npm package was downloaded 842 times last week.

Category: Search & Web
Author: GitHub Actions
GitHub stars: Not on GitHub
This week: Tracking since 20 Oct 2026
Weekly downloads: 842
Language: JavaScript
License: MIT (open source)
Last update: 2026-10-20 (4 days ago)
Links: Source npm

crawler
web-crawler
polite-crawler
robots.txt
rfc9309
scraper
web-scraping
fetch
cache
mcp
model-context-protocol
claude
+13 more

How to install polite-crawler

polite-crawler is published on npm as polite-crawler, and it depends on @modelcontextprotocol/sdk. These steps assume it runs as a local stdio server, which is the usual setup.

Check the README. Confirm the start command and note any API keys or environment variables it needs (pass them with -e KEY=value or an env block). You need Node.js for npx.

Add it to Claude Code from a terminal:

claude mcp add polite-crawler -- npx -y polite-crawler

Or add it to Cursor or Claude Desktop in .cursor/mcp.json or claude_desktop_config.json:

{
  "mcpServers": {
    "polite-crawler": {
      "command": "npx",
      "args": [
        "-y",
        "polite-crawler"
      ]
    }
  }
}

Restart the client and check it. Run claude mcp list, or open the client's MCP settings, and confirm polite-crawler and its tools are listed. The install guide covers VS Code and Codex.

Also listed in

Maintain this project? Get it featured ($99/mo, pinned at the top of Search & Web) or see what featuring includes.

Report a problem with this listing

Frequently asked questions

What is polite-crawler?

polite-crawler is an MCP server by GitHub Actions in the Search & Web category. A polite web crawler for research and AI agents. It fetches a source once, keeps it, and reads it many times: robots.txt obeyed, one request per site, full-text search. The MCP server is the package polite-crawler-mcp. Its npm package was downloaded 842 times last week.

Is polite-crawler free and open source?

Yes. polite-crawler is published under the MIT license, so you can run, modify and self-host it at no cost. If it connects to a paid service, that service's own pricing and API terms still apply. Check the repository for the exact license text before using it commercially.

What are the alternatives to polite-crawler?

Other Search & Web projects on MCP Harbor include Scrapling, TrendRadar, repomix. Each alternative links to its own page with stars, license and recent activity. The Search & Web category ranks all 746 by stars, and you can re-sort it by growth this week.

Alternatives to polite-crawler

ScraplingTrending★ 89,376

D4Vinci · Python+1,131 stars this week (+1.3%)

🕷️ A Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

TrendRadar★ 63,148

sansan0 · Python+171 stars this week (+0.3%)

⭐AI-driven public opinion & trend monitor with multi-platform aggregation, RSS, and smart alerts.🎯 告别信息过载，你的 AI 舆情监控助手与热点筛选工具！聚合多平台热点 + RSS 订阅，支持关键词精准筛选。AI 智能筛选新闻 + AI 翻译 + AI 分析简报直推手机，也支持接入 MCP 架构，赋能 AI 自然语言对话分析、情感洞察与趋势预测等。支持 Docker ，数据本地/云端自持。集成微信/飞书/钉钉/Telegram/邮件/ntfy/bark/slack 等渠道智能推送。

repomix★ 29,005

yamadashy · TypeScript+131 stars this week (+0.5%)

📦 Repomix is a powerful tool that packs your entire repository into a single, AI-friendly file. Perfect for when you need to feed your codebase to Large Language Models (LLMs) or other AI tools like Claude, ChatGPT, DeepSeek, Perplexity, Gemini, Gemma, Llama, Grok, and more.

open-seoTrending★ 24,721

every-app · TypeScript+829 stars this week (+3.5%)

Open source alternative to Semrush and Ahrefs

apify-mcp-serverTrending★ 12,865

apify · TypeScript+1,179 stars this week (+10.1%)

The Apify MCP server enables your AI agents to extract data from social media, search engines, maps, e-commerce sites, or any other website using thousands of ready-made scrapers, crawlers, and automation tools available on the Apify Store.

Horizon★ 10,982

Thysrael · Python+347 stars this week (+3.3%)

📡 Your own AI-powered news radar. Generates daily briefings in English & Chinese. | 用 AI 构建你专属的新闻雷达

All 746 Search & Web MCP servers →

Data last updated 25 October 2026. Stars and downloads are refreshed every day from GitHub and npm.