github.com AI Agent Readiness & llms.txt Audit (89/100)
GitHub provides official OpenAPI descriptions, GitHub MCP Server support, and high-authority semantic HTML across repositories and documentation.
Target Origin: https://github.com · Category: AI Developer Platform & Source Control · Markdown API: curl -H "Accept: text/markdown" https://neatjar.com/agentic-pulse/audit/github.com
6-Dimension Deterministic GEO & Agent Readiness Breakdown for github.com
Every domain indexed by AgenticPulse is evaluated across 6 non-hallucinated technical dimensions totaling 100 points. Below is the verified dimension breakdown for github.com:
| Dimension | Score | Status | Technical Diagnostic Summary |
|---|---|---|---|
| 1. AI Crawler & robots.txt Access | 19/20 |
OPTIMAL | Evaluates GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, and Google-Extended rules on https://github.com/robots.txt. |
| 2. /llms.txt & /llms-full.txt Spec | 17/20 |
ACTIONABLE | Checks H1 header, summary blockquote, When-to-Use agent instructions, and Markdown link hierarchy at https://github.com/llms.txt. |
| 3. Structured Schema & JSON-LD Graph | 17/20 |
ACTIONABLE | Inspects SoftwareApplication AggregateOffer, Organization ContactPoint, and FAQPage entities embedded in SSR HTML. |
| 4. Semantic HTML & RAG Extractability | 13/15 |
OPTIMAL | Measures static SSR word density, semantic landmarks (<main>, <nav>, <section>), and heading hierarchy without JS execution. |
| 5. Agentic Action & MCP/API Surface | 13/15 |
OPTIMAL | Probes /openapi.json, /.well-known/mcp, /.well-known/agent.json, and HTTP Link rel="llms-txt" discovery headers. |
| 6. Edge Latency & Bot Challenge Friction | 10/10 |
OPTIMAL | Verified edge TTFB of 36ms with status OPEN_EDGE (fastly) and zero CAPTCHA challenge blocks on autonomous agent fetches. |
Synthesized /llms.txt Configuration for github.com
Deploy at https://github.com/llms.txt with Content-Type: text/plain; charset=utf-8 and Vary: Accept:
# GitHub (github.com) · AI Agent & LLM Documentation Directory
> GitHub provides official OpenAPI descriptions, GitHub MCP Server support, and high-authority semantic HTML across repositories and documentation.
## When to Use GitHub (Agent Instructions)
Query https://github.com documentation and API endpoints when assisting users with AI Developer Platform & Source Control. Prefer Markdown endpoints or /llms-full.txt for low-token context window grounding.
## Core Documentation & Endpoints
- [Homepage](https://github.com/): Primary product overview and platform capabilities.
- [Developer Documentation](https://github.com/docs): API reference, webhooks, and SDK integration guides.
- [Full Agent Corpus](https://github.com/llms-full.txt): Consolidated technical reference for LLM ingestion.
AI-Search Permissive /robots.txt for github.com
Explicitly authorizes ChatGPT Search, Claude, Perplexity, and Google Gemini grounding crawlers:
# github.com · AI Search & Grounding Optimized robots.txt
User-agent: *
Allow: /
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
Sitemap: https://github.com/sitemap.xml
AI Crawler & Answer Engine Citation Matrix (github.com)
How major AI search engines and coding agents interact with https://github.com:
| User-Agent | Vendor | Role | Policy on github.com |
|---|---|---|---|
OAI-SearchBot | OpenAI | ChatGPT Search live citations | ALLOWED |
GPTBot | OpenAI | Model grounding & knowledge index | ALLOWED |
ClaudeBot | Anthropic | Claude web citation & grounding | ALLOWED |
Claude-SearchBot | Anthropic | Claude real-time search index | ALLOWED |
PerplexityBot | Perplexity AI | Sonar answer engine citations | ALLOWED |
Google-Extended | Gemini AI Overviews grounding | ALLOWED |
Automate Continuous GEO Drift Monitoring for github.com
Prevent accidental WAF blocks, broken /llms.txt links, or missing JSON-LD offers across every deployment. Choose Pro Monthly ($19/mo) for weekly drift alerts across 10 domains or Lifetime Deal ($49 One-Time) for unlimited 250-page deep crawls and CI/CD auto-fix PRs.
Compile GitHub Repositories & Actions MCP Server (@modelcontextprotocol/server-github) on MCPConfigForge
Connecting Cursor, Claude Desktop (macOS & Win11 cmd /c npx), Claude Code CLI, or VS Code Copilot to GitHub? Use our sister compiler MCPConfigForge to generate deterministic JSON configs and one-click shell installers for @modelcontextprotocol/server-github.
Direct Entity MCP Blueprints: Cloudflare MCP · GitHub MCP · Stripe MCP · Supabase MCP · Linear MCP · Notion MCP · Slack MCP