A pre-release version of the MERGE grammar was used for evaluating the performance of indexing. MERGE is the adaptation of the English Resource Grammar [CSLI2002] for TRALE [Meurers and Penn2002] (a HPSG parsing system built on top of ALE). This preliminary grammar has 13 rules with 2 daughters each, 4 unary rules, and 136 lexical entries. The type hierarchy contains 1157 types, with 144 introduced features.
For performance measurements, a test set containing 1970 sentences of lengths between 2 and 16 words7.1 was used. This corpus was automatically generated from a smaller ``seed'' corpus of 196 sentences. The seed sentences consist of 98 sentences included as test sentences in this release of MERGE, 86 sentences extracted from the Wall Street Journal (Penn Tree Bank v. 2) annotated parse trees, and 12 hand-built sentences. The sentences in the test corpus were generated by replacing nouns in the seed sentences with noun phrases or with words that are both noun and verb and by using the seed sentences as subordinated clauses.
Two versions of MERGE were employed during the experimental evaluation. The first version uses a Prolog encoding for typed feature structures that does not support the representation of type constraints, while the second version uses an extended encoding of Prolog terms that allows for the representation of type constraints. Since many unification failures are caused by the type constraints, the parsing times for the second version are faster than for the first one, as it will be seen in the following sections. Also, the lack of constraints in the first version causes overgeneration, therefore more sentences are recognized as grammatically correct by the non-constrained version (only 1112 out of the initial 1970 sentences are considered correct by the constrained grammar).
The motivation for evaluating both an unconstrained and a constrained version of the same grammar resides in the need for presenting the performance of the proposed indexing methods in two extreme cases. The unconstrained grammar serves as a larger scale grammar (where more edges are added to the chart and more unifications take place). The constrained version is a more typical TFSG. The performance of the indexing methods on a real large-scale TFSG is expected to be in between these two extremes.