<?xml version="1.0" standalone="yes"?>
<Paper uid="A97-1027">
  <Title>Dutch Sublanguage Semantic Tagging combined with Mark-Up Technology</Title>
  <Section position="6" start_page="185" end_page="185" type="evalu">
    <SectionTitle>
3 Evaluation &amp; Results
</SectionTitle>
    <Paragraph position="0"> Before a large scale validation involving &amp;quot;a gold standard&amp;quot; and various statistical metrics (e.g. see (Hripcsak et al., 1995)) is set up and conducted, a modest formative evaluation (Hirschman and Thompson, 1995) allowed to rapidly assess the functionality of the application from the point of view of the actual user. A limited validation test has been set up. A sample of 100 Dutch sentences of varying length and syntactic complexity was selected. All the words in the dictionary covering the 100 sentences were manually tagged with LSP semantic word class labels. The medical doctor supervising the medical registration activities was asked to provide some  61 On 21/1/87 your patient has been operated in our cardiovascular surgery unit.</Paragraph>
    <Paragraph position="1"> 62 Pre-operative diagnosis: coronary sclerosis.</Paragraph>
    <Paragraph position="2"> 63 Operative procedure: quintuple coronary bypass.</Paragraph>
    <Paragraph position="3"> 64 Reconstruction of the left arteria mammaria on the LAD.</Paragraph>
    <Paragraph position="4"> 65 Venal jump graft from the aorta to the diagonalis, further to the LAD.</Paragraph>
    <Paragraph position="5"> 66 Venal jump graft from the aorta to the first branch of the circumflexus, further to the second branch of the circumflexus, till the RDP .</Paragraph>
    <Paragraph position="6"> 67 Single venal bypass from the aorta to the AVSulcusbranch. null 68 After the procedure, the patient has been admitted to the Intensive Care unit.</Paragraph>
    <Paragraph position="7"> 69 Enclosed you can find the operation report.</Paragraph>
    <Paragraph position="8"> viewpoint of a medical encoder (using ICD-9-CM),  and to evaluate the system's responses.</Paragraph>
    <Paragraph position="9"> For all the 100 sentences, pseudo-HTML code was generated. The recall was 100 % (all the labels concerned were flagged). The precision ranged from 66% to 100 % depending on the label combination. Nevertheless, these figures are temporary as examination of the sentences showed that very few words had more than one semantic label so that the medical subselection stage did not have a big impact. A larger test set needs to be processed in order to provide more conclusive results. Probably, recall will drop while precision could raise. Nevertheless, the experience did prove to be valuable as the collaborating doctor, who had never heard of NLP before, said he was &amp;quot;positively surprised and impressed&amp;quot; by the capabilities of the system. He also judged the tool to be an interesting utility and consented in setting up a larger experiment to measure exactly the impact of the tool on the daily routine of the medical encoders. The evaluation procedure of this large test will be organised to comply as much as possible with the evaluation criteria recently proposed by Friedman and Hripcsak (Friedman and Hripcsak, 1997).</Paragraph>
  </Section>
class="xml-element"></Paper>