Classic monolingual WSD

Classic monolingual Word Sense Disambiguation evaluation tasks uses WordNet as its sense inventory and is largely based on supervised / semi-supervised classification with the manually sense annotated corpora:^[1]

• Classic English WSD uses the Princeton WordNet as it sense inventory and the primary classification input is normally based on the SemCor corpus.
• Classical WSD for other languages uses their respective WordNet as sense inventories and sense annotated corpora tagged in their respective languages. Often researchers will also tapped on the SemCor corpus and aligned bitexts with English as its source language

Sense inventories

During the first Senseval workshop the HECTOR sense inventory was adopted. The reason for adopting a previously unknown sense inventory was mainly to avoid the use of popular fine-grained word senses (such as WordNet), which could make the experiments unfair or biased. However, given the lack of coverage of such inventories, since the second Senseval workshop the WordNet sense inventory has been adopted. WSD exercises require a dictionary, to specify the word senses which are to be disambiguated, and a corpus of language data to be disambiguated. WordNet is the most popular example of sense inventory. The reason for adopting the HECTOR database during Senseval-1 was that the WordNet inventory was already publicly available.^[2]

Task Description

Comparison of methods can be divided in 2 groups by amount of words to test. The difference consists in the amount of analysis and processing:

• all-words task implies disambiguating all the words of the text
• lexical sample consists in disambiguating some previously chosen target words.

It is assumed that the former one is more realistic evaluation, although with very laborious testing of results. Initially only the latter was used in evaluation but later the former was included.

Lexical sample organizers had to choose samples on which the systems were to be tested. A criticism of earlier forays into lexical-sample WSD evaluation is that the lexical sample had been chosen according to the whim of the experimenter (or, to coincide with earlier experimenters' selections). For English Senseval, a sampling frame was devised in which words were classified according to their frequency (in the BNC) and their polysemy level (in WordNet). Also, inclusion POS-tagging problem was a matter of discussion and it was decided that samples should be words with known part of speech and some indeterminants (for ex. 15 noun tasks, 13 verb tasks, 8 adjectives, and 5 indeterminates).

For comparison purposes, known, yet simple, algorithms named baselines are used. These include different variants of Lesk algorithm or most frequent sense algorithm.

Evaluation measures

During the evaluation of WSD systems two main performance measures are used:

• Precision: the fraction of system assignments made that are correct
• Recall: the fraction of total word instances correctly assigned by a system

If a system makes an assignment for every word, then precision and recall are the same, and can be called accuracy. This model has been extended to take into account systems that return a set of senses with weights for each occurrence.

See also

• Word sense disambiguation
• Other variants of WSD evaluations
• Word sense
• WordNet
• SemEval

References