In the second task, we are interested in your judgement of the pairwise-relative quality of the parser-generated dependency sets for a given pair of sentences. As in the first task, for each sentence:
Unlike the first task, in this task we are asking you to compare parser-generated dependencies for two different sentences. For each sentence pair, we would like your judgement about which of the two sentences has the higher-quality set of parser-generated dependencies, as judged by your own linguistic sense of the severity of any errors therein. This means that, where the parser and ground truth conflict, you should not assume that the ground truth is correct.
As per your preference, you may change the notation used for the lexical categories.
To aid in seeing potential errors, we use scoring method B from the first task to highlight differences between the ground truth and the predictions. We follow the same notational conventions as in the first task, including the use of orange colouring and dashed edges to help identify dependencies considered to be erroneous by the scoring method.
For each pair of sentences, select the sentence that you believe has the higher-quality set of parser-generated dependencies, where quality is judged independently of the ground truth.
More precisely:
Ties are not permitted. In making your judgements, consider the (likely) semantics corresponding to the generated dependencies.
NB: We are asking for your professional opinion. If for any sentence you believe that the ground truth is incorrect, and/or that the parser has in fact found the correct syntactic analysis, you should disregard the ground truth. It is included here primarily to aid in seeing potential errors.
At any point, you may leave or close this window and return to this URL later; each selection is automatically saved and will be restored upon returning. (The selections are associated with your email address, judge4@main.study.)
When you are done, hit the Verify button at the bottom of the page to confirm that you have evaluated all sentences.