<?xml version="1.0" standalone="yes"?>
<Paper uid="P05-1070">
  <Title>Instance-based Sentence Boundary Determination by Optimization for Natural Language Generation</Title>
  <Section position="6" start_page="570" end_page="571" type="evalu">
    <SectionTitle>
5 Evaluations
</SectionTitle>
    <Paragraph position="0"> To evaluate the quality of our sentence boundary decisions, we implemented a baseline system in which boundary determination of the aggregation module is based on a threshold of the maximum number of propositions allowed in a sentence (a simplified version of the second strategy in Section 2. We have tested two threshold values, the average (3) and maximum (6) number of propositions among corpus instances. Other sentence complexity measures, such as the number of words and depth of embedding are not easily applicable for our comparison because they require the propositions to be realized first before the boundary decisions can be made.</Paragraph>
    <Paragraph position="1"> We tune the relative weight of our approach to best fit our system's capability. Currently, the weights are empirically established to W d</Paragraph>
    <Paragraph position="3"> =3and SBC =3. Based on the output generated from both systems, we derive four evaluation metrics: 1. Dangling sentences: We define dangling sentences as the short sentences with only one proposition that follow long sentences. This measure is used to verify our claim that because we use global instead of local optimization, we can avoid generating dangling sentences by making more balanced sentence boundary decisions. In contrast, the baseline approaches have dangling sentence problem when the input proposition is 1 over the multiple of the threshold values. The first row of Table 2 shows that when the input proposition length is set to 7, a pathological case, among the 200 input proposition sets randomly generated, the base-line approach always produce dangling sentences (100%). In contrast, our approach always generates more balanced sentences (0%).</Paragraph>
    <Paragraph position="4"> 2. Semantic group splitting. Since we use an instance-based approach, we can maintain the semantic cohesion better. To test this, we randomly generated 200 inputs with up to 10 propositions containing semantic grouping of both the number of bedrooms and number of bathrooms. The second row, Split Semantic Group, in Table 2 shows that our algorithm can maintain semantic group much better than the baseline approach. Only in 1% of the output sentences, our algorithm generated number of bedrooms and number of bathrooms in separate sentences. In contrast, the baseline approaches did much worse (61% and 21%).</Paragraph>
    <Paragraph position="5"> 3. Sentence realization failure. This measure is used to verify that since we also take a sentence's lexical and syntactical realizability into consideration, our sentence boundary decisions will result in less sentence realization failures.  An realization failure occurs when the aggregation module failed to realize one sentence for all the propositions grouped by the sentence boundary determination module. The third row in Table 2, Realization Failure, indicates that given 200 randomly generated input proposition sets with length from 1 to 10, how many realization happened in the output. Our approach did not have any realization failure while for the baseline approaches, there are 56% and 72% outputs have one or more realization failures.</Paragraph>
    <Paragraph position="6"> 4. Fluency. This measure is used to verify our claim that since we also optimize our solutions based on boundary cost, we can reduce incoherence across multiple sentences. Given 200 randomly generated input propositions with length from 1 to 10, we did a blind test and presented pairs of generated sentences to two human subjects randomly and asked them to rate which output is more coherent. The last row, Fluency, in Table 2 shows how often the human subjects believe that a particular algorithm generated better sentences. The output of our algorithm is preferred for more than 59% of the cases, while the baseline approaches are preferred 4% and 8%, respectively. The other percentages not accounted for are cases where the human subject felt there is no significant difference in fluency between the two given choices. The result from this evaluation clearly demonstrates the superiority of our approach in generating coherent sentences.</Paragraph>
  </Section>
class="xml-element"></Paper>