Check three places: your robots.txt file, which can block crawlers by name; your page code, which can carry a noindex tag; and Google Search Console, which shows pages Google could not index and why. AI crawlers such as GPTBot follow the same robots rules. A technical SEO audit from CICOR Marketing checks all three and reports exactly which crawlers are blocked and where.