Uncrawled URLs in search results
Why Webmasters Stack Exchange recommends it
Webmasters Stack Exchange users link it as the authoritative source for the point that robots.txt blocks crawling but does not stop a URL from appearing in search results.
Recommended by 0 people across 0 different months, 2011 to 2017.
What it is
A short 2009 video from Google's Matt Cutts explaining why a URL blocked by robots.txt can still show up in Google's search results. He clarifies that disallowing crawling is different from preventing indexing, since Google can list a URL it never fetched based on links pointing to it.
Cited in 1 Wikipedia article, including Robots.txt.
What people said
Cites Kevin Peno · 2011-04 · Webmasters Stack Exchange · on “Robots denied by domain is still listed in search results”
Based on my research into the subject, I've found that there isn't a 100% guarenteed way to prevent indexing and caching of data, but you can come pretty darn close (assuming you want to deal with increased bot traffic). Here's how I've interpreted the information. One would think that the robots.txt file is used to define robots information site-wide and meta tags are used for page specific details. I think the spirit behind the the 2 are exactly this but this is not the case in practice. Don't create a robots.txt [LINK] file This works with all major search providers to prevent content from appearing on the SERP, but does [LINK] not [LINK] prevent [this video] indexing. This also prevents bots
Cites DisgruntledGoat · 2013-11 · Webmasters Stack Exchange · on “Does Google crawl page when G+ logo is embedded?”
You appear to have many misconceptions about how search engines and ranking works, so I would suggest reading through the various Google help files. Google itself has an SEO starter guide [LINK] . To answer some basic points: PageRank is based purely on links to your site and nothing else. If your site is new then Googlebot will not be visiting very often anyway, so there is no reason to block it. The number of visitors to your site (or visitor increases) is not a ranking factor at all . Google cannot possibly know how many people are visiting your site besides the ones it sends directly. robots.txt prevents crawling of a page, but the URL may appear in search results [this video] under certain c
Cites Sasa1234 · 2014-10 · Webmasters Stack Exchange · on “Robots.txt vs. Noindex Tags”
According to matt cutts, even though we block the page using Robots.txt , it is better using noindex tag. ( Source : [this video] ) If it is blocked by Robots.txt, how Google bot read whether it has noindex tag or not? Because as I know if it blocked by Robots.txt, Google will not read any thing in that post.
Cites Wooz · 2015-12 · Webmasters Stack Exchange · on “Using robots.txt to deny access to MediaWiki special pages using substring matching”
I consider there will be none but internal links to such pages. So I am certainly none of the rare cases which Google would be interested to list: [this video]
Cites verstappen_doodle · 2017-09 · Webmasters Stack Exchange · on “How to prevent CDN content URLs being indexed by Google”
Well, robots.txt prevent crawling and meta robots tag in HTML (or) X-Robots-Tag HTTP header prevents indexing (and other functionalities available too). So, even when a URL is disallowed in robots.txt, it can be indexed in Google if it is referenced somewhere using an anchor tag. So, my question is, how can I prevent public CDN URL being indexed by Google? For example, I upload an image on Facebook which is private. But, the CDN URL holding the data is actually public, so anyone with the complete link can access the content. How does Facebook prevent these URLs being indexed? Example textual URL: [LINK] Now, I posted the above URL here. Will google index this URL? Update: Yes, this is indexe