check_crawlers
Whether robots.txt blocks the crawlers that cite you.
GET https://geondex.com/api/v1/crawlers- Cost: Free
- MCP tool:
check_crawlers - Auth:
Authorization: Bearer gx_…(see Authentication) - Try it: Open check_crawlers in the playground
Which of the crawlers AI engines actually cite from are blocked by your domain's robots.txt — an agent can read robots.txt but not which user-agents matter for citations as opposed to the ones that only train. Free, cached one hour.
Request
Query parameters
domain·string· required — Your site, e.g. ttsensei.com (no scheme). 3–253 characters
Example request
curl -s "https://geondex.com/api/v1/crawlers?domain=ttsensei.com" -H "Authorization: Bearer gx_YOUR_KEY"Response
200 with the envelope every tool returns: a one-line summary, the full data, and credits: { charged, remaining }. On a free call charged is 0 and remaining is null.
Fields of data
domain·stringhasRobotsTxt·boolean— No robots.txt at all means allow-all, not blockedcrawlers·object[]crawlers[].agent·stringcrawlers[].engine·stringcrawlers[].kind·"citation" | "training"crawlers[].blocked·booleancrawlers[].advisoryOnly·boolean· optional — A robots.txt disallow is advisory for this agent — it fetches regardlessblockedCitation·object[]— Citation crawlers genuinely blocked; excludes advisory-onlyblockedCitation[].agent·stringblockedCitation[].engine·stringblockedCitation[].kind·"citation" | "training"blockedCitation[].blocked·booleanblockedCitation[].advisoryOnly·boolean· optional — A robots.txt disallow is advisory for this agent — it fetches regardlessblockedTraining·object[]blockedTraining[].agent·stringblockedTraining[].engine·stringblockedTraining[].kind·"citation" | "training"blockedTraining[].blocked·booleanblockedTraining[].advisoryOnly·boolean· optional — A robots.txt disallow is advisory for this agent — it fetches regardless
Example response
Real-shaped, checked against the output schema by a test. Long lists are shortened.
{
"summary": "ttsensei.com allows all citation crawlers (training-only crawlers blocked: GPTBot, CCBot).",
"data": {
"domain": "ttsensei.com",
"hasRobotsTxt": true,
"crawlers": [
{
"agent": "OAI-SearchBot",
"engine": "ChatGPT",
"kind": "citation",
"blocked": false
},
{
"agent": "ChatGPT-User",
"engine": "ChatGPT",
"kind": "citation",
"blocked": false
},
{
"agent": "Claude-SearchBot",
"engine": "Claude",
"kind": "citation",
"blocked": false
},
{
"agent": "Claude-User",
"engine": "Claude",
"kind": "citation",
"blocked": false
},
{
"agent": "PerplexityBot",
"engine": "Perplexity",
"kind": "citation",
"blocked": false
},
{
"agent": "Perplexity-User",
"engine": "Perplexity",
"kind": "citation",
"blocked": false,
"advisoryOnly": true
},
{
"agent": "GPTBot",
"engine": "ChatGPT",
"kind": "training",
"blocked": true
},
{
"agent": "ClaudeBot",
"engine": "Claude",
"kind": "training",
"blocked": false
},
{
"agent": "CCBot",
"engine": "Common Crawl",
"kind": "training",
"blocked": true
},
{
"agent": "Google-Extended",
"engine": "Gemini",
"kind": "training",
"blocked": false
}
],
"blockedCitation": [],
"blockedTraining": [
{
"agent": "GPTBot",
"engine": "ChatGPT",
"kind": "training",
"blocked": true
},
{
"agent": "CCBot",
"engine": "Common Crawl",
"kind": "training",
"blocked": true
}
]
},
"credits": {
"charged": 0,
"remaining": null
}
}Errors
Every error has the body {"error":{"code":"…","message":"…"}} and is not charged.
| HTTP | When |
|---|---|
400 | Bad input |
401 | Missing or invalid key |
402 | Out of credits |
404 | Tool not available on this server |
429 | Rate limited |
502 | Upstream failure |
Every error.code and what to do about it: Errors & rate limits.