<?xml version="1.0" standalone="yes"?>
<Paper uid="W02-1502">
  <Title>The Grammar Matrix: An Open-Source Starter-Kit for the Rapid Development of Cross-Linguistically Consistent Broad-Coverage Precision Grammars</Title>
  <Section position="8" start_page="3" end_page="3" type="evalu">
    <SectionTitle>
6 Evaluation and Evolution
</SectionTitle>
    <Paragraph position="0"> The matrix itself is not a grammar but a collection of generalizations across grammars. As such, it cannot be tested directly on corpora from particular languages, and we must find other means of evaluation. We envision overall evaluation of the matrix based on case studies of its performance in helping grammar engineers quickly start new grammars and in helping them scale those grammars up. Evaluation in detail will based on automatable deletion/substitution metrics, i.e., tools that determine which types from the matrix get used as is, which get used with modifications, and which get ignored in various matrix-derived grammars. Furthermore, if the matrix evolves to include defeasible constraints, these tools will check which constraints get overridden and whether the value chosen is indeed common enough to be motivated as a default value. This evaluation in detail should be paired with feedback from the grammar engineers to determine why changes were made.</Paragraph>
    <Paragraph position="1"> The main goal of evaluation is, of course, to improve the matrix over time. This raises the question of how to propagate changes in the matrix to grammars based on earlier versions. The following three strategies (meant to be used in combination) seem promising: (i) segregate changes that are important to sync to (e.g., changes that affect MRS outputs, fundamental changes to important analyses), (ii) develop a methodology for communicating changes in the matrix, their motivation and their implementation to the user community, and (iii) develop tools for semi-automating resynching of existing grammars to upgrades of the matrix.</Paragraph>
    <Paragraph position="2"> These tools could use the type hierarchy to predict where conflicts are likely to arise and bring these to the engineer's attention, possibly inspired by the approach under development at CSLI for the dynamic maintenance of the LinGO Redwoods tree-bank (Oepen et al., 2002).</Paragraph>
    <Paragraph position="3"> Finally, while initial development of the matrix has been and will continue to be highly centralized, we hope to provide support for proposed matrix improvements from the user community.</Paragraph>
    <Paragraph position="4"> User feedback will already come in the form of case studies for the library as discussed in Section 5 above, but also potentially in proposals for modification of the matrix drawing on experiences in grammar development. In order to provide users with some cross-linguistic context in which to develop and evaluate such proposals themselves, we intend to provide some sample matrix-derived grammars and corresponding testsuites with the matrix. A user could thus make a proposed change to the matrix, run the testsuites for several languages using the supplied grammars which draw from that changed matrix, and use [incr tsdb()] to determine which phenomena have been affected by the change. It is clear that full automation of this evaluation process will be difficult, but at least some classes of changes to the matrix will permit this kind of quick cross-linguistic feedback to users with only a modest amount of additional infrastructure. null</Paragraph>
  </Section>
class="xml-element"></Paper>