<?xml version="1.0" standalone="yes"?>
<Paper uid="X96-1004">
  <Title>Some Technology Transfer: Observations from the TIPSTER Text Program</Title>
  <Section position="4" start_page="29" end_page="31" type="metho">
    <SectionTitle>
6. Users
</SectionTitle>
    <Paragraph position="0"> The development of new technology to support a user task must be a collaborative process. It is the user who knows what task has to be accomplished, but it is the technologist who understands what the technology can do and how it can be configured to potentially help. The contributions of both groups are necessary. In the case of the TIPSTER Program most of the leadership has been taken by the technologists, albeit technologists with considerable experience in analytical tasks and knowledge of present and future user requirements. When the technologist is leading, an important part of any technology transfer project, we have found, is the establishment and maintenance of trust between the technologist and the user.</Paragraph>
    <Paragraph position="1"> There are many facets to building and maintaining this trust. It requires understanding the user's style what do they require from someone in order to trust them? Style will vary widely among different user organizations, from those who feel most comfortable with detailed project plans and schedule charts, to those who hate view graphs and only trust round table discussions with an occasional hand drawn diagram on the back of a sheet of paper. Some feel best with developers sitting in their midst, some like their developers working in hi-tech labs at the developers spaces. Some like formal reviews, others like people to stop by and chat, and so forth.</Paragraph>
    <Paragraph position="2"> It is important for the technologist to communicate that he/she is promoting the user's ends, not the technologist's ends. The technology is being brought to serve the user's task and not solely to bring credit to the technology's developer. The technologist must understand the user's job well enough to explain clearly how the technology could possibly help.</Paragraph>
    <Paragraph position="3"> Demonstrations with the user's data, mocked up user interfaces, examples of similar projects, and descriptions of the results of testing the technology against tasks similar to the user's, can all be used to support the contention that the technology could be made helpful to the user. At the same time, the technologist, being an outsider, cannot prescribe how the task should be done with the new technology. It is the user's purview to control his/her own work style, after all. At the same time, if the technologist simply lets the user dictate how the job should be done, as often as not the user will make inappropriate use of the technology, either expecting too little or too much of its capabilities. The best solution seems to be for the technologist to offer as many reasonable choices as possible, for the application of the technology, allowing the user to choose or fine tune the actual work flow.</Paragraph>
    <Paragraph position="4"> It is important also to educate the user about the technology. Better informed users will make better decisions about what technology to purchase and how to use it, so that the educational aspects of any work with the user will pay dividends far beyond the success or failure of any particular project. In the case of TIPSTER, the existence of an established and well  recognized program has been helpful in gaining user trust, but also the program workshops and evaluation conferences, as well as the proceedings coming out of them, have been useful in helping to explain to users what the technology does and how it works. The demonstration projects themselves were conceived of as partly having an educational mission; nothing would be so effective in persuading people that the technology &amp;quot;was for real,&amp;quot; that it had something to offer them, and a bit about how it worked, than seeing it and trying it in an operational mode. If only one or two of these projects can succeed in making a large, positive contribution to a user's task, the returns should be substantial in making it clear to people that the technology is ready to be deployed and for what kinds of tasks.</Paragraph>
    <Paragraph position="5"> The mission of educating the user about the technology means being as honest as possible about what it can and cannot do, explaining clearly how the technology will fit into the user's environment, discussing requirements on their time for development, evaluation, and training, and making good faith estimates of maintenance efforts, should a system become operational. In any project involving the joint investment of resources, a clear agreement worked out ahead of time concerning the obligations and resources coming from each party is important. Resources in this case means not only funding, but also the personnel time necessary to do things such as manage the project, develop requirements, review interface designs and so on. Generally, we have found that unless users are sufficiently concerned about the improvements in their task to invest time in learning about and supporting the application development, they are a &amp;quot;high risk&amp;quot; user, that is, one who is likely to back out before a project is completed or who will not use and support the software once it is completed.</Paragraph>
    <Paragraph position="6"> We have found that written agreements between technologists and customers, signed at an appropriate management level, even when operating within the same agency, covering expectations of funding and personnel resources as well as the criteria for success of a project, are extremely helpful in preventing disagreements and disappointments over the course of an applications development.</Paragraph>
    <Paragraph position="7"> As with any job, the task of bringing an application to the operational environment will more likely be successful if the people who are involved work as a unified team. On the part of the project leader, frequently the TIPSTER representative, this requires including in the process from the beginning all those people who will be affected by the application. It means having representatives from the user group, from a couple levels of their management, from the developers and their management, from the IS (Information Services) staff or the equivalent, cognizant of each other, of the goals and progress of the project, and aware of their respective roles and obligations. At the same time, the project leader must be aware that most of these people have other jobs to do and their time should be asked for only when necessary. This translates into carefully planned communication of information to those who need it, when they need it, well focused meetings, and the presentation to the user only of quality products (even interim products like prototype GUIs) which have been thoroughly reviewed and are free of obvious mistakes.</Paragraph>
    <Paragraph position="8"> To the extent that the project leader can also communicate the excitement of developing a new capability and create a team which enjoys working together, these factors will cause people to put forth their best effort to make the project successful.</Paragraph>
    <Paragraph position="9"> 7. Risks and Evaluations In the end, however, it is well to remember that every project to transfer a new technology to the work place is a risk. While a great deal can and should be done to reduce the risk, there is never any guarantee that such a project will succeed in all or even most of its goals. As with any other endeavor, no failures means both that nothing will be learned and that nothing very difficult has been tried. So, some setbacks, at least, in transferring this technology to the user are necessary. It was problems, some of them not solved the first time, which taught us what I have recorded in this short discussion. At the same time, even an apparently failed project can have good results, if the causes of the failure are understood and the knowledge incorporated into later planning. For this reason, testing and evaluation of not only the research, but also applications is crucial.</Paragraph>
    <Paragraph position="10"> Evaluation incorporates all aspects of the technology transfer project: (1) management, (2) relations with the user and IS staff, (3) development costs and schedule, (4) user interface, (5) work flow, (6) robustness of the software, (7) throughput, (8) accuracy, (9) ease of integration, (10) ease of maintenance, (11) use of the architecture. Evaluation methods differ for each. For project management issues (1,2,3), appropriate oversight by the management of the project leader is required; self evaluation at the end of the project, by both the project leader and the developer can be helpful. For usability issues (4,5) we have less experience. However, a current TIPSTER project is using a combination of interviews with user/testers after their sessions with the prototype and observations of the user/tester sessions to try to determine how easy to use the application is. Performance issues (6,7,8) can be evaluated by observation of the system for failures, for the speed of various tasks, and by comparison of  human generated output with machine output.</Paragraph>
    <Paragraph position="11"> Software design issues (9,10,11) are currently being evaluated through examination of the design in the TIPSTER Engineering Review Board, as well as by monitoring of the integration and maintenance phases by the project leader.</Paragraph>
    <Paragraph position="12"> Much of the risk of a technology insertion project can be mitigated by two strategies - paying careful attention to the user issues addressed above and paying careful attention to the steps of technology transfer detailed above. Understanding how far along one's technology is on the path toward deployment and carefully evaluating the technology at each stage in its progress can prevent many mistakes. A number of difficulties I have observed in technology transfer projects have occurred because integration has occurred before component functions were sufficiently well researched and understood. Another common difficulty is failure to work out the best uses of the technology before an application is chosen and a user organization signed up. Both of these failings occur because needed steps in the progression of technology from research to the user environment have been short-changed.</Paragraph>
    <Paragraph position="13"> A third strategy can also reduce the risk of technology transfer projects, which can be described as &amp;quot;start small and out of the way.&amp;quot; This strategy does not reduce the chances of failure, but reduces the risk that failure will be disastrous. It has been important for TIPSTER to have demonstration projects which showed that TIPSTER technologies could do something useful and important. However, there are many important tasks that are at the same time &amp;quot;small,&amp;quot; that is, well defined, and &amp;quot;out of the way,&amp;quot; that is, limited in the number of users they affect.</Paragraph>
    <Paragraph position="14"> These have proven excellent opportunities to demonstrate the new technology without taking on all the additional problems of a very large and complicated development. These small-scale and limited applications have also simplified dealings with the user, since there were fewer of them immediately involved.</Paragraph>
  </Section>
class="xml-element"></Paper>