Crawler Analytics
AI bots detected in the access log, classified as training crawls (broad, steady) or retrieval fetches (bursty, concentrated) — the latter is the signal we're being cited live.
Hits per bot
coloured by inferred classTraining vs retrieval
share of hitsPer-bot detail
6 bots · 617 hits| Bot | Vendor | Hits | Pages | Class | Conf. | Burst | Concentration | Span |
|---|---|---|---|---|---|---|---|---|
| Bytespider | ByteDance | 556 | 107 | training | 59% | 21% | 31% | 3.9h |
| meta-externalagent | Meta | 28 | 20 | training | 69% | 29% | 3% | 3.4h |
| Amazonbot | Amazon | 15 | 15 | training | 68% | 33% | 0% | 2.2h |
| PerplexityBot | Perplexity | 7 | 7 | retrieval | 0% | 100% | 0% | 1s |
| ChatGPT-User | OpenAI | 7 | 5 | retrieval | 6% | 29% | 4% | 2.2h |
| ClaudeBot | Anthropic | 4 | 2 | training | 60% | 50% | 0% | 1.9h |
Retrieval hot pages
live-citation signalClassification reasoning
how each call was madeconfig class 'training' (prior 0.15); 31% path concentration across 107 path(s); 21% of hits in a 5-min window → training (score 0.20).
config class 'training' (prior 0.15); 3% path concentration across 20 path(s); 29% of hits in a 5-min window → training (score 0.15).
config class 'training' (prior 0.15); 0% path concentration across 15 path(s); 33% of hits in a 5-min window → training (score 0.16).
config class 'search' (prior 0.50); 0% path concentration across 7 path(s); 100% of hits in a 5-min window → retrieval (score 0.50).
config class 'retrieval' (prior 0.90); 4% path concentration across 5 path(s); 29% of hits in a 5-min window → retrieval (score 0.53).
config class 'training' (prior 0.15); 0% path concentration across 2 path(s); 50% of hits in a 5-min window → training (score 0.20).