<?xml version="1.0" standalone="yes"?>
<Paper uid="W03-0412">
  <Title>PhraseNet: Towards Context Sensitive Lexical Semantics/</Title>
  <Section position="5" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
4 Evaluation and Application
</SectionTitle>
    <Paragraph position="0"> In this section we provide a first evaluation of PhraseNet.</Paragraph>
    <Paragraph position="1"> We do that in the context of a learning task.</Paragraph>
    <Paragraph position="2"> Learning tasks in NLP are typically modelled as classification tasks, where one seeks a mapping g : X ! c1;:::;ck, that maps an instance x 2 X (e.g., a sentence) to one of c1;:::;ck - representing some properties of the instance (e.g., a part-of-speech tag of a word in the context of the sentence). Typically, the raw representation - sentence or document - are first mapped to some feature based representation, and then a learning algorithm is applied to learn a mapping from this representation to the desired property (Roth, 1998). It is clear that in most cases representing the mapping g in terms of the raw representation of the input instance - words and their order - is very complex. Functionally simple representations of this mapping can only be formed if we augment the information that is readily available in the input instance with additional, more abstract information. For example, it is common to augment sentence representations with syntactic categories - part-of-speech (POS), under the assumption that the sought-after property, for which we seek the classifier, depends on the syntactic role of a word in the sentence rather than the specific word. Similar logic can be applied to semantic categories. In many cases, the property seems not to depend on the specific word used in the sentence - that could be replaced without affecting this property - but rather on its 'meaning'.</Paragraph>
    <Paragraph position="3"> In this section we show the benefit of using PhraseNet in doing that in the context of Question Classification.</Paragraph>
    <Paragraph position="4"> Question classification (QC) is the task of determining the semantic class of the answer of a given question.</Paragraph>
    <Paragraph position="5"> For example, given the question: &amp;quot;What Cuban dictator did Fidel Castro force out of power in 1958?&amp;quot; we would like to determine that its answer should be a name of a person. Our approach to QC follows that of (Li and Roth, 2002).</Paragraph>
    <Paragraph position="6"> The question classifier used is a multi-class classifier which can classify a question into one of 50 fine-grained classes.</Paragraph>
    <Paragraph position="7"> The baseline classifier makes use of syntactic features like the standard POS information and information extracted by a shallow parser in addition to the words in the sentence. The classifier is then augmented with standard WordNet or with PhraseNet information as follows.</Paragraph>
    <Paragraph position="8"> In all cases, words in the sentence are augmented with additional words that are supposed to be semantically related to them. The intuition, as described above, is that this provides a level of abstract - we could have potentially seen an equivalent question, where other &amp;quot;equivalent&amp;quot; words occur.</Paragraph>
    <Paragraph position="9"> For WordNet, for each word in a question, all its hypernyms are added to its feature based representation (in addition to the syntactic features). For PhraseNet, for each word in a question, all the words in the corresponding conset wordlist are added (where the context is supplied by the question).</Paragraph>
    <Paragraph position="10"> Our experiments compare the three pruning operations described above. Training is done on a data set of 21,500 questions. Performance is evaluated by the precision of classifying 1,000 test questions, defined as follows: Precison = # of correct predictions# of predictions (2) Table 2 presents the classification precision before and after incorporating WordNet and PhraseNet information into the classifier. By augmenting the question classifier with PhraseNet information, even in this preliminary stage, the error rate of the classifier can be reduced by 12%, while an equivalent use of WordNet information reduces the error by only 5.7%.</Paragraph>
    <Paragraph position="11">  tion Question classification precision and error rate reduction compared with the baseline error rate(15.8%) by incorporating WordNet and PhraseNet(PN) information. 'Baseline' is the classifier that uses only syntactic features. The classifier is trained over 21,500 questions and tested over 1000 TREC 10 and 11 questions.</Paragraph>
  </Section>
class="xml-element"></Paper>