<?xml version="1.0" standalone="yes"?>
<Paper uid="A94-1002">
  <Title>Practical Issues in Automatic Documentation Generation</Title>
  <Section position="3" start_page="7" end_page="9" type="intro">
    <SectionTitle>
2 User-Centered Design
</SectionTitle>
    <Paragraph position="0"> User-needs analysis is a common practice in the development of computer-human interface systems and other end-user software. Particularly in developing a large scale, practical system, the needs of the user must be studied if the resulting system is to be accepted and effectively used by the users. In this section, we describe the user-needs analysis and system development methodology that we are using in our ongoing development of PLANDoc.</Paragraph>
    <Paragraph position="1"> Our analysis combined two complementary approaches. First, we interviewed a variety of different groups of people involved in the telephone network planning task. Our goal was to identify potential users of PLANDoc and to solicit their views on how such a system could be most helpful. Second, we collected a set of manually-written narratives to inform the development of the generator, providing insights on report form and content, vocabulary and sentence structure. In this section we describe how user interviews and corpus analysis shaped the design of the documentation generator. But first we provide some brief background information on the problem domain.</Paragraph>
    <Section position="1" start_page="7" end_page="7" type="sub_section">
      <SectionTitle>
2.1 Problem Setting
</SectionTitle>
      <Paragraph position="0"> Voice and data service is carried to telephone customers through a complex network of routes consisting of copper or fiber cables supplemented by additional equipment such as Digital Loop Carrier (DLC) systems and fiber multiplexors. It is the telephone network planning engineer's job is to derive a capacity expansion (relief) plan specifying when, where, and how much new copper, fiber, multiplexing and other equipment to install in a route to avoid facilities exhaustion. This activity is an integral part of telephone operations. New installations are costly, but at the same time facilities exhaustion can lead to a disruption in telephone service. Currently, about 1,000 planning engineers in 8 regional and independent telephone companies produce a total of about 15,000 route studies per year.</Paragraph>
      <Paragraph position="1"> The engineer uses PLAN to compute an optimum, cost-effective base relief plan needed to meet forecast demand over the next twenty years. The base plan, however, may not always be realizable or desirable due to political, economical, practical and other factors known to the engineer but not to the computer.</Paragraph>
      <Paragraph position="2"> The engineer uses PLAN's Interactive Refinement Module that allows 'what-if' modeling to explore the effects of various changes to the base plan. For example, an engineer might explore requesting a DLC activation for a given site, or changing a fiber activation time. After comparing the effects of different refinement scenarios, the engineer ultimately decides on a realizable relief plan to recommend to management for project authorization.</Paragraph>
      <Paragraph position="3"> Overall interaction with PLAN thus includes an automatically generated base plan, a sequence of refinements to explore the effects of different changes to the base, and a final proposed plan which may indude elements of the base plan along with a selected set of refinements.</Paragraph>
    </Section>
    <Section position="2" start_page="7" end_page="7" type="sub_section">
      <SectionTitle>
2.2 Interviews
</SectionTitle>
      <Paragraph position="0"> With the help of Bencore Planning and Engineering staff 3 we formulated an initial proposal for PLANDoc and drafted preliminary target narratives. We then conducted a series of interviews with planning engineers, managers, auditors and PLAN support staff from several regional telephone companies in their home offices and at two PLAN training courses 4. The work experience of the engineers we interviewed ranged from beginner to expert. Our 3Many thanks to M. Horwath, D. Imhoff and L.</Paragraph>
      <Paragraph position="1"> Tenet.</Paragraph>
      <Paragraph position="2"> 4 Some of the helpful regional Planning and Engineering personnel included P. MeNeill, J. Brunet, P. King, D. Kelly, I. MeNeill, T. Smith, C. Lowe, and G. Giles, goal was to determine how engineers actually used the PLAN system, whether they would find an automated documentation facility to be helpful, and, if so, what the form and content of the narratives should be.</Paragraph>
      <Paragraph position="3"> We learned that novice planners often run 'bozo' refinements just to develop a feel for the process, while experienced planners sometimes run refinements they know will be suboptimal just for the record, i.e., for the benefit of managers, auditors and regulators who might ask &amp;quot;did you try such and such?&amp;quot;. More critical to the need for documentation, we also learned that experienced planners keep handwritten notes on paper, listing their refinements and why they tried them; they asked for a way to enter their notes on-line to keep track of their reasoning. Inexperienced planners asked to see narratives written by experienced planners in order to learn from them; unfortunately few such narratives exist.</Paragraph>
      <Paragraph position="4"> Finally, all planners welcomed the idea of having the computer generate narratives that they could include in their documentation packages, especially if they could add to the narratives themselves.</Paragraph>
      <Paragraph position="5"> These findings shaped the content of PLANDoc narratives and the design of the system. Specifically, they indicated that planners may not want all refinements that they tried to appear in the narrative. For example, novice planners do not want to include their 'bozo' refinements, while experienced planners do want to include the suboptimal refinements they ran to show that their final refinements were superior. Thus, PLANDoc includes a facility that lets the planner select a subset of refinements to be included in the final narrative. Planners made it clear that they use knowledge not included in PLAN to make their decisions (e.g., corporate strategies) and they wanted a way to record that knowledge on-line, while they were working. This gave rise to PLANDoc's facility to prompt for manually-written engineer's notes at crucial points. We instituted only two user-visible changes to PLAN's original, successful interface, one to prompt for engineering notes and another to allow the engineer to request a narrative and select a subset of refinements to be included.</Paragraph>
      <Paragraph position="6"> Both options are presented using familiar PLAN interface commands and screen formats. Reports are generated off-line.</Paragraph>
    </Section>
    <Section position="3" start_page="7" end_page="7" type="sub_section">
      <SectionTitle>
2.3 Corpus Analysis
</SectionTitle>
      <Paragraph position="0"> We also arranged for an experienced retired planning engineer, Jim Phillips, who is also a PLAN expert, to write a corpus of target narratives based on PLAN runs of actual routes. Based on the findings from our interviews and on the target narratives, we arrived all from Pacific Bell, R. Riggs, D. Spiegel, S. Sweat, L. Doane, R. Tufts, and R. Ott, all from Southwestern Bell, S. Wasalinko from NYNEX, and C. Lazette from Ameritech.</Paragraph>
      <Paragraph position="1">  at the report format shown in Figure 1. It consists of two parts, a tabular summary of route input data and a narrative that integrates machine-generated text with the engineer's manually-entered notes.</Paragraph>
      <Paragraph position="2"> Our corpus of target narratives provided information on what should be included in the report and its overall structure. Thus, it directly influenced development of both the Lexicalizer and Content Planner modules of PLANDoo. An analysis of PLAN's menu of refinement actions and the sentences in the target narratives allowed us to specify a set of 31 different possible message types for refinement sentences including, for example, fiber extensions to CSAs (Carrier Serving Areas), or DLC (Digital Loop Carrier) equipment activations or denials for CSAs.</Paragraph>
      <Paragraph position="3"> We then systematically categorized the sentences in our corpus to reveal all the different phrasings for each message type. This categorization showed that there was tremendous variety in the possible sentences for each message type with respect to sentence structure and lexical choice. Indeed, our first implementation of PLANDoc's sentence generator 5, resuited in more than 150 paraphrases for some message classes.</Paragraph>
      <Paragraph position="4"> The target narratives also informed the design of PLANDoo's Content Planner. Our analysis revealed that choosing a specific paraphrase for use in a summary depends on what has already been mentioned (i.e., the choice is based on previous discourse). Furthermore, the narratives provided examples of how multiple messages were frequently combined to form complex compound sentences. In order to avoid a combinatorial explosion from combining many different sentences forms, we needed to specify a bounded sublanguage for PLAN's domain that ensured the sentence variety needed to maintain discourse coherence and fluency while enabling the construction of complex sentences. Before discussing this problem, we provide an actual sample of some PLANDoc output in Figure 2.</Paragraph>
    </Section>
    <Section position="4" start_page="7" end_page="9" type="sub_section">
      <SectionTitle>
2.4 Sample PLANDoc Output
</SectionTitle>
      <Paragraph position="0"> At present, the tabular Input Summary generator 6 and the textual Refinements Summary generator of the PLANDoc system are fully implemented. Fig- null RUNID: REG1 Run-ID REG1 started at the BASE plan. This saved refinement activated DLC for CSAs 3122, 3130, 3134, 3208 and 3420 in the third quarter of 1994. It demanded that PLAN use DLC system IDLC272 for all placements in CSA 3122. The 20 year PWE was $2110.1K, a $198.6K savings over the BASE plan and the 5 year IFC was $1064.0K, a $64.5K penalty over the BASE plan.</Paragraph>
      <Paragraph position="1"> Engineer's note: These CSA's are beyond 28 kf and need range extenders to provide service on copper. Moving them to 1994 will negate a job adding a reg bay to the office.</Paragraph>
      <Paragraph position="2"> RUNID: 3234-2 This saved refinement included all DLC changes in Run-ID REG1E. It requested the activation of DLC for CSA 3234 in the second quarter of 1994 and for CSA 3233 in the fourth quarter of 1994. DLC systems DLC96SS and DLC96M2 were used for all placements in CSAs 3233 and 3234. For this refinement, the 20 year route PWE was $1925.3K, a $383.4K savings over the BASE plan and the 5 year IFC was $833.9K, a $165.6K savings over the BASE plan.</Paragraph>
      <Paragraph position="3"> Engineer's note: I didn't need to demand the activation of these systems in the refinement as they were activated at this time in the BASE plan. The 'idlc272' was demanded because of the high demand. The non-integrated systems in CSA 3234 because it is a business area.</Paragraph>
      <Paragraph position="4">  ure 2 is an abbreviated sample of a Refinements Summary generated by PLANDoc. The incorporated Engineering Notes were entered manually by the Planning Engineer at run time and automatically integrated into the narrative by PLANDoc.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>