.Google.com's John Mueller responded to a concern regarding why Google.com indexes pages that are refused from creeping through robots.txt as well as why the it's safe to ignore the related Look Console reports about those creeps.Bot Web Traffic To Query Parameter URLs.The person asking the concern documented that crawlers were generating links to non-existent query criterion URLs (? q= xyz) to web pages along with noindex meta tags that are actually also blocked out in robots.txt. What urged the inquiry is actually that Google is actually crawling the hyperlinks to those web pages, acquiring shut out through robots.txt (without seeing a noindex robotics meta tag) after that obtaining shown up in Google Explore Console as "Indexed, though obstructed through robots.txt.".The person talked to the complying with inquiry:." Yet below's the big inquiry: why would Google.com index webpages when they can't even see the material? What's the advantage during that?".Google's John Mueller affirmed that if they can't crawl the web page they can not view the noindex meta tag. He also makes an interesting mention of the internet site: search operator, recommending to neglect the results given that the "ordinary" users will not view those results.He wrote:." Yes, you are actually correct: if we can't crawl the page, we can't observe the noindex. That stated, if our experts can't crawl the pages, at that point there is actually certainly not a great deal for our company to mark. Thus while you could find a few of those web pages with a targeted website:- query, the normal customer will not view them, so I definitely would not fuss over it. Noindex is also alright (without robots.txt disallow), it merely suggests the Links will definitely find yourself being actually crawled (and also end up in the Explore Console file for crawled/not indexed-- neither of these statuses cause problems to the rest of the website). The integral part is actually that you don't create all of them crawlable + indexable.".Takeaways:.1. Mueller's answer validates the restrictions in operation the Website: hunt progressed search driver for diagnostic factors. Among those explanations is actually considering that it is actually certainly not attached to the frequent search mark, it's a separate thing altogether.Google's John Mueller talked about the web site search operator in 2021:." The short response is that an internet site: query is actually certainly not indicated to be complete, nor made use of for diagnostics reasons.A site question is a specific type of search that limits the end results to a particular site. It is actually generally merely the word web site, a digestive tract, and afterwards the internet site's domain.This question limits the results to a certain website. It is actually not suggested to become a thorough compilation of all the web pages coming from that web site.".2. Noindex tag without using a robots.txt is actually fine for these type of conditions where a bot is connecting to non-existent pages that are actually receiving found out through Googlebot.3. URLs with the noindex tag will definitely generate a "crawled/not indexed" entry in Look Console and also those won't possess a damaging result on the rest of the internet site.Read the inquiry and also respond to on LinkedIn:.Why would certainly Google.com mark pages when they can't even observe the information?Included Graphic by Shutterstock/Krakenimages. com.