.Google.com's John Mueller answered a concern regarding why Google marks pages that are refused from crawling by robots.txt as well as why the it is actually secure to neglect the related Look Console reports regarding those creeps.Crawler Web Traffic To Inquiry Guideline URLs.The individual asking the inquiry recorded that bots were actually producing web links to non-existent question parameter Links (? q= xyz) to webpages along with noindex meta tags that are actually also obstructed in robots.txt. What triggered the inquiry is actually that Google.com is actually creeping the links to those web pages, obtaining obstructed by robots.txt (without envisioning a noindex robotics meta tag) then acquiring reported in Google Look Console as "Indexed, though obstructed by robots.txt.".The person inquired the observing question:." However listed here is actually the large inquiry: why would Google mark webpages when they can not even find the material? What's the conveniences during that?".Google.com's John Mueller affirmed that if they can't creep the page they can't view the noindex meta tag. He also creates an interesting reference of the internet site: hunt driver, suggesting to disregard the outcomes since the "normal" customers will not view those results.He created:." Yes, you are actually correct: if our experts can not crawl the web page, our experts can't view the noindex. That claimed, if our team can't crawl the pages, after that there is actually certainly not a whole lot for our company to mark. Therefore while you may observe a number of those webpages along with a targeted web site:- concern, the typical consumer will not find them, so I wouldn't bother it. Noindex is actually likewise great (without robots.txt disallow), it merely indicates the URLs will find yourself being crept (as well as find yourself in the Browse Console file for crawled/not catalogued-- neither of these conditions induce problems to the rest of the web site). The essential part is that you don't create them crawlable + indexable.".Takeaways:.1. Mueller's response verifies the constraints in using the Internet site: search progressed hunt driver for diagnostic causes. One of those reasons is actually because it's not connected to the normal search mark, it's a separate point altogether.Google.com's John Mueller commented on the web site hunt driver in 2021:." The short answer is actually that a web site: concern is actually not implied to be full, neither made use of for diagnostics purposes.A site inquiry is a specific kind of search that restricts the results to a certain site. It's generally only the word web site, a bowel, and after that the web site's domain.This question limits the end results to a certain internet site. It's certainly not meant to become a detailed compilation of all the webpages coming from that web site.".2. Noindex tag without making use of a robots.txt is actually alright for these sort of circumstances where a crawler is linking to non-existent pages that are actually obtaining found through Googlebot.3. Links with the noindex tag are going to produce a "crawled/not catalogued" item in Browse Console and also those will not possess an adverse result on the remainder of the internet site.Review the concern as well as respond to on LinkedIn:.Why would Google mark webpages when they can't also find the information?Featured Picture by Shutterstock/Krakenimages. com.