Google Ngram Viewer

The Google Ngram Viewer or Google Books Ngram Viewer is an online viewer, initially based on Google Books, that charts frequencies of any word or short sentence using yearly count of n-grams found in the sources printed between 1800 and 2012^[1][2][3][4] in American English, British English, French, German, Spanish, Russian, Hebrew, and Chinese.^[1][5] Lacking an independent corpus, Italian words are counted by their use in other languages. It can search for a single word, including a misspelling, or a phrase, or gibberish.^[5] The n-grams are matched by case-sensitive spelling, comparing exact uppercase letters,^[2] and plotted on the graph, if found in 40 or more books.^[6] It now supports searches for parts of speech and wildcards.

It was developed by Jon Orwant and Will Brockman and released in mid-December 2010.^[1][3] It was inspired by a prototype (called "Bookworm") created by Jean-Baptiste Michel and Erez Aiden from Harvard's Cultural Observatory and Yuan Shen from MIT.

Researchers have analyzed the Google Ngram database of books written in American or British English. Research based on the ngram database has included the finding of correlations between the emotional output and significant events in the 20th century such as World War II^[7] or to check and challenge popular trend statements such as the secularisation or economisation of modern societies.^[8] In this sense, the viewer represents a valuable research tool for digital humanities.

Operation and restrictions

Commas delimit user-entered search-terms, indicating each separate word or phrase to find.^[6] The Ngram Viewer returns a plotted line chart within seconds of the user pressing the Enter key or the "Search" button on the screen.

As an adjustment for more books having been published during some years, the data is normalized, as a relative level, by the number of books published in each year.^[6]

Google populated the database from over 5 million books published up to 2008. Accordingly, as of May 2012, no data will match beyond the year 2008. Due to limitations on the size of the Ngram database, only matches found in over 40 books are indexed in the database; otherwise the database could not have stored all possible combinations.^[6]

Typically, search-terms cannot end with punctuation, although a separate full stop, or period, can be searched.^[6] Also, an ending question mark (as in "Why?") will cause a 2nd search for the question mark separately.^[6]

Omitting the periods in abbreviations will allow a form of matching, such as using "R M S" to search for "R.M.S." versus "RMS".

Corpora

The corpora used for the search are composed of total_counts, 1-grams, 2-grams, 3-grams, 4-grams, and 5-grams files for each language. The file format of each of the files is tab-separated data. Each line has the following format:^[9]

• total_counts file

year TAB match_count TAB page_count TAB volume_count NEWLINE

• Version 1 ngram file (generated in July 2009)

ngram TAB year TAB match_count TAB page_count TAB volume_count NEWLINE

• Version 2 ngram file (generated in July 2012)

ngram TAB year TAB match_count TAB volume_count NEWLINE

The Google Ngram Viewer uses match_count to plot the graph.

As an example, a word "Wikipedia" from the Version 2 file of the English 1-grams is stored as follows:^[10]

The graph plotted by the Google Ngram Viewer using this data is here.

Criticism

The dataset has been criticized for its reliance upon inaccurate OCR and for including large numbers of incorrectly dated and categorized texts.^[11][12] Although the authors claim that the results are reliable from 1800 onwards, poor OCR and insufficient data mean that frequencies given for other languages such as Chinese may only be accurate from 1970 onwards, with earlier parts of the corpus showing no results at all for common terms, and data for some years containing more than 50% noise.^[13][14]

See also

• Lexical analysis
• Culturomics

References

External links

• Google Ngram Viewer