Stop words

Not to be confused with Safeword.

In computing, stop words are words which are filtered out before or after processing of natural language data (text).[1] Though stop words usually refer to the most common words in a language, there is no single universal list of stop words used by all natural language processing tools, and indeed not all tools even use such a list. Some tools specifically avoid removing these stop words to support phrase search.

Any group of words can be chosen as the stop words for a given purpose. For some search engines, these are some of the most common, short function words, such as the, is, at, which, and on. In this case, stop words can cause problems when searching for phrases that include them, particularly in names such as "The Who", "The The", or "Take That". Other search engines remove some of the most common words—including lexical words, such as "want"—from a query in order to improve performance.[2]

Hans Peter Luhn, one of the pioneers in information retrieval, is credited with coining the phrase and using the concept. [3]

See also

References

  1. ↑ Rajaraman, A.; Ullman, J. D. (2011). "Data Mining". Mining of Massive Datasets (PDF). pp. 1–17. doi:10.1017/CBO9781139058452.002. ISBN 9781139058452.
  2. ↑ Stackoverflow: "One of our major performance optimizations for the "related questions" query is removing the top 10,000 most common English dictionary words (as determined by Google search) before submitting the query to the SQL Server 2008 full text engine. It’s shocking how little is left of most posts once you remove the top 10k English dictionary words. This helps limit and narrow the returned results, which makes the query dramatically faster".
  3. ↑ Luhn, H. P. (1959). Keyword-in-Context Index for Technical Literature (KWIC Index). Yorktown Heights, NY: International Business Machines Corp.

External links

This article is issued from Wikipedia - version of the Wednesday, January 20, 2016. The text is available under the Creative Commons Attribution/Share Alike but additional terms may apply for the media files.