Wikipedia talk:Searching/External search engines

From Wikipedia, the free encyclopedia

You have new messages (last change).

This page relates to external searches.

Yahoo!'s Content Acquistion Program includes Wikipedia, which should boost traffic through Yahoo!. Yahoo approved our datafeed on May 25 2004.

The frequency with which Wikipedia is spidered by search engines varies between articles. Why articles have not yet been spidered is a frequently asked question. The discussion below attempts to answer this, and includes other discussions relating to external searches of Wikipedia.

See also: Searching, Google test, What Google liked, Send in the clones & MediaWiki:Googlesearch, MediaWiki:Notitlematches

Contents

[edit] Yahoo Search

You might be interested to read this [1].

"Additional CAP partners include The New York Public Library, one of the most renowned libraries in the country; Project Gutenberg, the Web's oldest producer of free electronic books; University of Michigan's OAIster project, which provides hard to find academic collections; UCLA's Cuneiform Digital Library Initiative (CDLI) with content documenting Babylonian history back to 3500 B.C.; Wikipedia, a free, multilingual online encyclopedia with articles in more than 50 languages; and the National Science Digital Library (NSDL), the National Science Foundation's online library, with more than 250 collections that improve the way Americans learn about science, technology, engineering, and mathematics. The OYEZ, CDLI and NSDL projects are all federally funded in part or in whole by the National Science Foundation."

Dori | Talk 17:36, Mar 2, 2004 (UTC)

I am surprised this has generated so few responses. Wikipedia is under Yahoo!'s Content Acquistion Program... does this mean they will index us a lot... or something else? Pete/Pcb21 (talk) 23:18, 2 Mar 2004 (UTC)
Jimbo has commented on the mailing list:
"Yes, I was in negotiations with Yahoo about this last week.
We have a contract in which we supply them with an XML feed (which I will have Jason construct) and they stick us in their index. They make no promises as to the placement of our urls, of course, as that's entirely up to their editorial department. But of course we have absolutely maximum quality content, so it is thought by all that we will rank very high in their index.
The area that this will benefit us most is when news breaks on some topic about which there is little information on the net -- an area in which we excel anyway.
I tried to get their PR person to feature us more prominently in the press release, but alas, she didn't listen to me.
--Jimbo"
Pete/Pcb21 (talk) 23:27, 2 Mar 2004 (UTC)
This sounds incredible! It also sounds like we will need to be even more vigilant on RC, especially concerning articles being highlighted by Yahoo! news. :-) Jwrosenzweig 23:32, 2 Mar 2004 (UTC)


[edit] Google search not finding main entries

Back in October, I wrote new entries for Carl Spaatz and Lyman Lemnitzer. Today I decided to do a search and see if there were any mentions of their names that were not linked back to the main entries. I did, in fact, find two such mentions. But I also found that a google search under "Spaatz" or "Lemnitzer" failed to provide a hit on either of the main entries for these men. Obviously both names were mentioned several times in the relevant entry. Other entries with links to these entries were listed (such as List of people associated with World War II). Google even had the links from my user page which post-dated the creation of these entries. So why doesn't google pick up on them? MK 15:34 (EST) 30 November 2003

One month isn't that long for Google to find something, particularly if the pages that link to it have a low page rank. Angela 23:37, 30 Nov 2003 (UTC)

[edit] Saving a page, but it doesn't appear in Google

What am I doing wrong. I have started, written and saved a page. I then log out, clear the computer of cookies, and do a google search for the page I have written. Google finds it, but always opens it in the edit mode, rather than as a completed document. Is this normal or am I doing something wrong? Ragussa 13:25, 10 May 2004 (UTC)

you're not doing anything wrong. Google just doesn't update it's links that frequently. Give it a little while, and it'll show up on Google just fine. theresa knott 13:38, 10 May 2004 (UTC)
The other part of this is that Google seems to index the edit pages for non-existent articles. Don't know why, there's a meta tag that says not to. -- Cyrius|&#9998 13:47, May 10, 2004 (UTC)
Anyone have any stats on how long it takes, on average, for a newpage to get indexed, and what variables affect the time taken. The Village Pump was indexed on the 8th May (two days ago as I write), but is a very frequently updated page. Clements Markham I started on April 7 and is now #1 google hit for the name. Ranulph Fiennes, on the other hand, I started on April 13, and it appears not to have been indexed yet. Pete/Pcb21 (talk) 13:51, 10 May 2004 (UTC)
I believe "more worthy" pages are indexed more frequently, where "worthy" is a combination of frequently-changing, well internally connected, and well-linked-to from outside the site. Of these the last probably carries the greatest weight. The Fiennes article's version on Nationmaster has been indexed: [2] -- Finlay McWalter | Talk 14:58, 10 May 2004 (UTC)
According to Google's_FAQ_page, depending on when the page was submitted and the timing of its web crawls, it may take 6 to 8 weeks for a new page to be added to Google's index. GUllman 21:37, 10 May 2004 (UTC)

[edit] Wikipedia Talk namespace and Google

moved from the Reference desk by IMSoP 17:26, 6 May 2004 (UTC)

Why is it that Wikipedia Talk pages are invisible to Google, even to Google searches within the Wikipedia domain?

Please give specific examples. I've seen many WP pages returned by both a general google search, as well as ones limited to WP. Google, unlike some other search engines, does suppress certain inherently transient pages such as VfD. Niteowlneils 06:10, 5 May 2004 (UTC)

So far as I know, they aren't. I've seen them come up in search results. RickK 03:42, 5 May 2004 (UTC)

Out of interest, how is the suppression of VfD done? Is it a X-archive:No type thingy (I know I've probably got that slightly wrong, but a tecchy willl know what I mean) or some other method? Article link will do. --bodnotbod 15:26, May 6, 2004 (UTC)

Specific example: The Talk page for the article egg white definitely contains the words albumin, albumen, Eiweiß, and Wikipedia, but when I searched in Google for "albumin albumen eiweiß wikipedia" only one page turned up, and it was most certainly not the page I was attempting to access via Google.

Let me explain how google interacts with wikipedia. First off, most of those words weren't in that talk page until they were added today. Google only crawls wikipedia occasionally (theoretically weekly, but that's very variable). And google's search function can't report results for pages which its crawl function hasn't visited. It certainly hasn't had time to see those particular ones. Moreover, google doesn't crawl all the pages in the entire wikipedia at any one time (infact, it doesn't necessarily crawl all the pages ever). Google's algorithm for figuring out what to crawl, and how frequently, while related to pagerank, is secret, variable, and frequently site-specific. So we don't know what they chose to crawl, and what they chose not to. If you simply search for "talk" you'll see that google has crawled and indexed lots of talk pages. Another thing: talk pages are very unlikely to be linked to from a source outside the wikipedia - while folks will link readily to an article (from their website, their blog, or whatever). Google generally values links from outside much more highly than internal ones, and this may well explain why lots of talk pages are deemed "unimportant" by google, and thus not crawled at all. You wouldn't be the first person to express frustration at wikipedia using the fallback of the google search engine, rather than mediawiki's own search function - but we're (perpetually) hardware-poor, and the built-in search function isn't something we can afford to enable. -- Finlay McWalter | Talk 22:11, 5 May 2004 (UTC)
Folks, should I add a page (like Wikipedia:Google issues, linked to from Wikipedia:Searching) which explains stuff like this (as this isn't the first time I've answered this kind of question)? -- Finlay McWalter | Talk 22:17, 5 May 2004 (UTC)
Sounds good to me, but it's you that has to put the work in ;op --bodnotbod 00:10, May 7, 2004 (UTC)
Oh, the last crawl appears to be May 3rd. Wikipedia:Searching says the crawl is monthly, not weekly, but that might be out of date. -- Finlay McWalter | Talk 22:25, 5 May 2004 (UTC)
Is there any place that has a discussion of what it would take to have our own search engine with an index that gets updated perhaps once a day? How much hardware would it take to enable the built-in search function? nroose Talk 18:42, 30 May 2004 (UTC)

[edit] Google/Wikipedia search engine problems

I have a question about the current Google/Wikipedia search engine, or comment. Namely, it seems to produce very inconsistent, incomplete, or paradoxical responses to inquiries. A few examples:

  • Oftentimes, I will read an article, and then do a search on the title of that article (EXACTLY as it is in the title, verbatim, down to caps even) and it fails to be found by Google to be on wikipedia. I find this very strange. Sometimes, nothing is found; other times, other articles, that only very indirectly link to the article, are produced. For instance, if one does a search on "Modular group" or even "Modular group Gamma", one doesn't get a link to the article entitled "Modular group Gamma", instead one gets a link to a user page, where "Modular group Gamma" is listed among several hundred pages created and/or edited by the user. What is going on here?
  • Another example: If you do a search on "Gauss" (or even "Carl Gauss"), then you won't get the article on Gauss the person for at least a couple pages (if that, I gave up after a while), you get lots of articles with "Gauss" as a keyword, or linking to "Gauss", but not to the article on Gauss himself. This seems very strange.
  • Many times, when looking for a specific article, (to see if it's there) I do a search and get absolutely nothing. But then, I say, "well, Google has failed me in the past, let me try directly" and I type in the actual URL of what should be the article page, and up in comes!! There it is!

This is what I find most disconcerting about the search engine. Someone will look something up, not get any results, and just assume that it is not present in the wikipedia. They won't know the little tricks about following other search results, going to more "meta-" pages (e.g. in math, going to major mathematical pages and looking around), or typing in URLs directly. This doesn't give a bad impression to newcomers, but it certainly fails to take advantage of everything that IS here. And it's a major inconvenience to people who use the wiki.

I would like to know if I am the only user that this happens to. I only bring it up in the village pump because it has been a common, persistent, recurring problem for me ever since I started (or ever since the Google/wikipedia page came up). It's not just an isolated incident with a few searches. Revolver 15 Nov 2003

I think this is because google is confused about www.wikipedia.org, en.wikipedia.org, and en2.wikipedia.org. I suspect it will settle down some in the following weeks. Also, google will often (some say always) be inconsistent on results for a website that (like this) changes often - different search servers at google (all which _appear_ to be www.google.com) are looking at subtly different sets of crawl-data. So sometimes (especially during the "googledance", when they progressively update these database-copies) two identical queries will produce different results. And as to the "right link being way down the search", that's a function of google's (secret, and utterly arcane) pagerank algoritm - there's not much we can do about that, as manipulating google's rankings (for good or ill) is notoriously difficult. -- Finlay McWalter 20:51, 15 Nov 2003 (UTC)
I also had similar experiences. I have a theory for this. According to my theory, Google/ whatever might consider a page for searching only some time after the article is written, probably because it allows time(which seems to be around a month or so) for vetting by enough number of people. But the links to user pages and individual words in articles comes into the realm of search because in most cases the user page and the words that link exist much prior to the creation of the article. However, having said that, one of pages which had been there for sometime in the third page in Google suddenly disappeared for me totally, even after including the word wikipedia. I can't figure this one out. KRS 05:06, 16 Nov 2003 (UTC)
I duuno about that. I've seen articles I've written pop up in the index within a day or two, but they have been ones that were listed (I think) for a day or so on the main page. -- Viajero 13:39, 16 Nov 2003 (UTC)
Google only shows an article when it rescans that portion of the site. An article with few or no links may take a lot of time to be found and indexed on Google. An article listed on the main page will be found and indexed very quickly. To help, it's useful to link to articles from their parent subjects and to link to related articles, so Google and other search engines can follow the web of links between related items. Jamesday
Google supposedly has automatic functions to remove pages that "spam the engine", by creating synthetic cross-links or by posting many identical pages on different domains. Because of the license model, there are many fairly identical copies WP pages on non-WP sites. And of course WP has heavy internal cross-links. Could this be causing a 'false positive' in the Google spam-killer? Anjouli 07:28, 18 Nov 2003 (UTC)

[edit] Google Indexing Update

It seems to me it's been a while since Google updated its Wikipedia index. Is that our fault (i.e., did we accidentally tell its robots to go away in one of our files), or is it their fault, or is it my psychotic delusion? -- Someone else 11:22, 16 Nov 2003 (UTC)

  • Having a look at the stats for this month [3] (second to last table) the googlebot has made over 90,000 visits and accounts for 1.25% of all hits. Going by percentage and comparing with previous months this is about normal. However going from www. to en. seems to have affected the rankings of wikipedia pages in google. Also having both en. and en2. addresses doesn't help. But it will hopefully settle down in a month or two. -- Popsracer 11:54, 16 Nov 2003 (UTC)

[edit] Google Appliance?

Maybe this is obvious and has been discussed before, but have we considered using an appropriate Google Search Appliance [4]. This is actual hardware that would need to be purchased that would sit in the racks of our servers and could be setup to index the entire Wikipedia every day. I don't know how expensive this solution is or whether "we" can afford it, but it looks like an ideal solution to the problem. Any comments? -- FrankH 17:24, 30 May 2004 (UTC)

[edit] Atomz Search?

Has anyone explored the Atomz Search application? (see also their FAQ). I use the free Express version for my personal site, and it works very well, is extraordinarily customizable, easy to integrate into a site, and you control when content is indexed. I've seen it on many other pro sites as well. It's a pay service for sites with more than 500 pages (heh), and of course there's no pricing on their website -- it's ye olde "contact our sales staff" routine. It may be too pricey for a site as large and index-intensive as this one, but it should at least be worth exploring..... especially since the search application in hosted on their servers, not ours. And who knows, they may be willing to negotiate a deal with a site as prominent as ours is becoming. Perhaps Jimbo or someone else with an idea of how much we would be willing to spend to have a reliable internal search mechanism could contact them...? --Catherine - talk 19:31, 17 May 2004 (UTC)

[edit] Google results: Mirrors vs Wikipedia

I'm sure this must have cropped up before, but I can't find it; can anyone point me to a relevant discussion? Anyway, I did a search on Google today for Lucifer cipher, and in the top 10 results were no less than 7 mirrored copies of the Lucifer (cipher) page, but not the Wikipedia article itself, which surfaces at position 70. This seems to happen a lot for various articles, and is somewhat annoying (especially since the mirrored pages are out of date and advert-laden). Anything Wikipedia can do? Feel free to point me to the previous discussions... — Matt 13:42, 18 May 2004 (UTC)

Many people are PO'ed about this, and I have no idea how it could be fixed (short of someone at google taking some action). If those pages were respecting the GFDL to the letter, they would link to the exact Wikipedia article, which should raise the pagerank of the Wikipedia article, and eventually bring it to the top, but this does not seem to be happening. Dori | Talk 14:06, May 18, 2004 (UTC)
[From one of the PO'd peeps]. This is a relatively recent problem (roughly since thefreedictionary.com came along) but must be costing us traffic, and is a long term threat to the continued growth of the GFDL corpus. What is odd is that, despite at least some of the mirrors linking to WP and thus making it probably the most linked to version of the page, WP comes so low. Are the other sites so good at search exchange optimization/google-breaking? Has WP somehow fallen foul of a negative points score due to being seen as a link farm somehow? Pete/Pcb21 (talk) 15:42, 18 May 2004 (UTC)
They have as many links as we do (as a mirror), but it's just that they were up when wikipedia wasn't and google spidered them first. Dori | Talk 16:22, May 18, 2004 (UTC)


[edit] How does Google index Wikipedia?

Wikipedia would seem to be part of the "deep Web" and hence inaccessible to Google. That is, there isn't any static page that links to all the other pages (or a static tree of such links). So how does Google's spider find articles? Does it watch special:newpages, or does it have a Wikipedia-specific search procedure (perhaps based on special:allpages), or what? The speed with which new Wikipedia articles get indexed is astonishing.... Dpbsmith 16:11, 9 Jun 2004 (UTC)

New articles will be found because other pages have links to them. Usually when someone creates a new page, they link to that page from a pre-existing article, which is already in Google. - DropDeadGorgias (talk) 16:18, Jun 9, 2004 (UTC)
Unless someone with access to the apache logs undertakes a detailed study, we really don't know how google spiders wikipedia. I understand that google maintains customised crawler preferences for the top websites (tuning things like search depth, frequency, and which things to ignore) but I've no evidence that they've done this for wikipedia. I agree with DropDeadGorgias' suggestion - ephemeral things like special:newpages and special:recent_changes change too quickly to be of much use to the crawler (which visits most sites no more frequently than weekly). -- Finlay McWalter | Talk 16:34, 9 Jun 2004 (UTC)
I have realized, that lonely pages did not get indexed. So, google is another reason to make special:lonelypages shorter.
uhm, first of all: to make special:lonelypages work. Or are there orphans no more?
You might use User:Topbanana/Reports/Nothing_links_to_this_article as the temporary alternative. And the other items on User:Topbanana/Reports give a lot of work for those who like cleanup work - spelling mistakes, missing interwiki, most wanted articles etc. andy 22:25, 11 Jun 2004 (UTC)


[edit] Countermeasures: Page rank

I wasn't sure about something, and wanted to bring it up here. Basically, mirror versions are appearing much higher in google than we are. The explanation people give for this is that they're somehow manipulating the pagerank system. My question is not "how" (I'm not technical enough to really grasp), but rather "Could we do this too?". [[User:Meelar|Meelar (talk)]] 13:44, 2004 Jul 21 (UTC)

One key way they seem to do this is by including a series of phrases such as (for an article called Stuff): "What is Stuff? Information about Stuff. Stuff definition..." — Chameleon My page/My talk 15:07, 21 Jul 2004 (UTC)

How could we do this without putting it in the article text? [[User:Meelar|Meelar (talk)]] 19:42, 2004 Jul 21 (UTC)

They often include it in the page <title>, and presumably also in the meta tags. It could also be incorporated in small text at the bottom of the article. — Chameleon My page/My talk 20:24, 21 Jul 2004 (UTC)
By the way, the book "Google Hacks" includes some basic SEO tips in the final chapters. (Just thought I'd mention it.) Lucky Wizard 02:09, 30 Jul 2004 (UTC)
I read somewhere on Wikipedia that at least once someone wrote to google about a mirror having a higher rank than the real wiki, and the people at google fixed it. Not sure if this is possible for the entire wikipedia, though. -- Chris 73 | Talk 22:16, 21 Jul 2004 (UTC)
Good idea, how about a few GOOGLEBOMBS too?
I propose we collectively draft an official letter to Google on the matter, as well as working at our end to boost our ranking to the level it deserves to be at. — Chameleon My page/My talk 10:44, 22 Jul 2004 (UTC)
Such a thing is not unheard of; for special things, Google will give preference to certain sites over others, and these things are built in. See: UPC search, definition search. So it's not like Google would just dismiss it out of hand, and in fact, I think they might enjoy more integration. But how about WE get that integration, instead of one of the cheap ripoffs? I support this and think it should be done quickly. --Golbez 05:16, 23 Jul 2004 (UTC)
Google has a feature that allows you to only search websites that have to do with certain subjects, like Mac, linux, U.S. Government, etc. I think that we should ask them to do a similar thing for Wikipedia. But is telling the difference between Wikipedia and a mirror really that hard? Search results that are from Wikipedia look like: ARTICLE NAME HERE - From Wikipedia, the free encyclopedia. [[User:Mike Storm|Mike Storm (Talk)]] 02:09, 23 Jul 2004 (UTC)
The real problem here is that potential users of Wikipedia and thus possible long term contributors to the knowledge base are regularly diverted away from what is the real source of the information in the first place, and to which they maybe one day would otherwise contribute. However, as a follow up to my suggestion that we might think about a PD search engine to slaughter Google, it looks like the cathedral once again is out to do for the bazaar: [5] Conceptually nice, however the thing seems to be down or broken a lot at the moment. However, I think this shows that the cathedral has had enough of the antics of the bazaar and has decided to act already. Sjc 10:57, 23 Jul 2004 (UTC)

Wikipedia:Database download gives the technical reasons we are almost assured a low Google ranking: because we're database-bound, crawlers are restricted to one access per second. Our mirrors are typically flat HTML, so can be crawled much faster.

I suggest that there's not much point worrying about our Google ranking until we are confident we have the server power (enough Squid frontends, I would guess) to handle the traffic. Remember that the deal with Yahoo doubled our load in a week - David Gerard 10:40, 23 Jul 2004 (UTC)

I just found my article on effeminacy on the free dictionary.com. I don't see where they referenced wikipedia nor myself. I wish I could get credit for all my hard work.WHEELER 23:58, 27 Jul 2004 (UTC)

Wikipedia is referenced at the bottom, below the stuff like "free dictionary browser". Lucky Wizard 02:09, 30 Jul 2004 (UTC)

I would encourage everyone to submit the articles they care most about to DMOZ, the basis directory for Google and other search engines. This may eventually ameliorate some of the problems related to searchability. -- Stevietheman 17:39, 29 Jul 2004 (UTC)

[edit] Finding non-WP sources of information

Is there a trick to googling information that is not Wikipedia-derived? I try searching "foo -wikipedia -gnu" but it still finds thefreedictionary, etc. even though thefreedictionary has the word wikipedia in it. I look online for information about something and I have to wade through 2 pages of non-obvious Wikipedia clones before I find something written by someone else.  :-) - Omegatron 17:06, Aug 10, 2004 (UTC)

[edit] Lack of Wiki hits in Google

Is anyone bothered by the fact that the 'pedia no longer appears anywhere near the top of searches in Google? When I put in Syagrius or magister militum I get any number of sites containing copies of the wiki text, but not this site itself - in these two cases I gave up looking. The info on these sites is presumably copied at some moment in time and therefore "frozen", and is therefore less likely to be accurate. Please forgive me if this is a subject that has been raised before, but I couldn't find any mention of it. Djnjwd 23:02, 22 Aug 2004 (UTC)

It has been mentioned before, but what can we do? We can't force google to put us top. Theresa Knott 00:08, 23 Aug 2004 (UTC)
Yes, you can. Go to google and put in a word. Look at the top-right of the screen; that word is hotlinked. It takes you to selected definitions from certain sites. Would it be too much to ask Google that they give us the same consideration? Maybe I will, but it'd be nice if someone official did it. --Golbez 03:19, 23 Aug 2004 (UTC)
We can send google quality complaints. Basically, these other sites optomize for google and we don't; that's why they kill us in the google rank →Raul654 00:15, Aug 23, 2004 (UTC)
Is it true that a possible reason for the low Google rating is low reliability? The other dictionary sites are more consistently, available than Wikipedia. I think uptime has been pretty good for a while now, but the site still rates relatively poorly... David Remahl 00:22, 23 Aug 2004 (UTC)
Similarly, latency might figure in, if relevence is considered equal. I think our only real hope is to enforce our license so that people can get from any mirrored page to the "live" page. We may consider modifying our license slightly to ensure that the link is prominent (many are at the bottom of long articles in a tiny font). Derrick Coetzee 00:51, 23 Aug 2004 (UTC)
I hope something like this is done. Google ranking wouldn't matter all that much if when people had read a mirrored article once they knew where it came from originally, and that the mirror was inferior, and came here in future. — Trilobite (Talk) 00:56, 23 Aug 2004 (UTC)
I'm wondering if we could legally sue the people who own these domains (do a detailed WhoIs to find who it's registered to). I'm not sure if Wikipedia could get a team of lawyers, but is it actually possible? — Ilγαηερ (Tαlκ, cοηtrιbs) 02:17, 23 Aug 2004 (UTC)
Well, you certainly don't need to start with suing. There is a standard license enforcement sequence, starting with polite requests, to more sternly worded, then threatening legal action. I know it has worked with a number of sites. I don't know where on wikimedia, but this has another place it is being discussed actively, somewhere on meta I'm sure. - Taxman 02:45, Aug 23, 2004 (UTC)
Wikipedia:Mirrors and forks has a non-compliance process, including a Wikipedia:Standard GFDL violation letter. -- Chris 73 Talk 02:58, Aug 23, 2004 (UTC)
But even the compliant sites shouldn't be above wiki. Suing aside, this is sort of a problem. — Ilγαηερ (Tαlκ, cοηtrιbs) 04:26, 23 Aug 2004 (UTC)
Why not? From Google's perspective thefreedictionary.com is a better site than Wikipedia. It has virtually the same content, faster response time, better use of tooltips/metatags/etc... Yeah, Wikipedia is becoming well-known and has a gazillion links to its homepage, but links to specific pages aren't that common, so the clones don't lose out from this perspective either. AFAIK Google doesn't have a weighting for being the "original". Pcb21| Pete 09:15, 23 Aug 2004 (UTC)

There's discussion on this ongoing at Wikipedia:Send in the clones. [[User:Meelar|Meelar (talk)]] 06:38, 2004 Aug 23 (UTC)

There is also a valuable discussion about problems with Google on Wikipedia:External search engines. Some of those discussions began last year, and it looks as though each intake of new editors asks the same questions - and gets the same answers! May I suggest these pages plus this current Parish Pump discussion are somehow consolidated (by an administrator?) and placed on the Community Portal page with a heading like 'Wikipedia and search engine difficulties'. That way we have somewhere to keep an eye on it. It might be noted in whatever welcome material we sent new editors to draw their attention it.

There are also the regular pages Search engines and Google which so far as I can see do not touch on this problem. Apwoolrich 13:20, 25 Aug 2004 (UTC)

I tend to believe that there's also a software (Google compatibility) issue involved. I did a Google search for the first paragraph of our Cohortative mood article - it's a vanity thing, and Firefox make such searches very simple - and got three results: 2 from thefreedictionary.com (which does link to the Wikipedia article, although the article has been since moved) and one from wikiverse.org (which doesn't). This means that not only Google does not rank the Wikipedia article highly, it is also entirely unaware of its existence (the same can be asserted using a Google cache query). Worse still, particularly when Wikipedia's search is disabled, is that, naturally, Google is also unaware of the article when doing a Wikipedia-specific search. The article is also not particularly new; presumably, Google scans the Web every 30 days, and the article is seven months old. One reason for this (and for other issues) is possibly Wikipedia's Crawl-delay value set at robots.txt. While it not particularly high (in fact, it is minimal), Wikipedia is pretty big, which might discourage even usually-reliable Google. -- Itai 14:31, 25 Aug 2004 (UTC)

Retrieved from "http://en.wikipedia.org../../../s/e/a/Wikipedia_talk%7ESearching_External_search_engines_baaa.html"