Professor of Computational Linguistics
University of Toronto, Department of Computer Science
Research
Computational analysis of arguments, issue framing, and ideology
Determining the ideology of a text: We develop methods to automatically determine the ideological position of a political text. For example, one might wish to take a newspaper editorial or a blog and classify it as socialist, conservative, or Green. In practice, much of the research has taken speeches by members of a legislature. One might expect, a priori, that methods based solely on the vocabulary used in a text would not be effective, because the members of a legislature, regardless of ideology, are all discussing the same topics — e.g., the legislation before them or the issues of the day — and hence would all be using the same topic-derived vocabulary. The ideology expressed in a text would thus be apparent only at the sentence- and text-meaning levels. Nonetheless, one might hypothesize that different ideological frameworks lead to sufficiently different ways of talking about a topic that vocabulary can be a discriminating feature.
Prior research used support-vector machines with word features to classify political texts — in particular, legislative speech — by ideology. Our own work on this topic led us to hypothesize that such classifiers are sensitive not to expressions of ideology but rather to expressions of attack and defence, opposition and government. We tested this hypothesis on speeches in the Canadian Parliament by training on one parliament and testing on another in which party roles have been interchanged, and we found that the performance of the classifier completely disintegrates. But removing the factor of government–opposition status, as in the European Parliament, enables a more-ideological classification. Our results cast doubt on the generality of the results of research that uses words as features in classifying the ideology of speech in legislative settings — and possibly in political speech more generally — and suggest that the language of attack and defence, of government and opposition, may dominate and confound any sensitivity to ideology in these kinds of classifiers.
References: Hirst et al (2014a, 2014b), Hirst and Feng (2016).
Classification of argumentation schemes: Argumentation schemes are structures or templates for forms of arguments. The arguments need not be deductive or inductive; on the contrary, most argumentation schemes are for presumptive or defeasible arguments. For example, argument from cause to effect is a commonly used scheme in everyday arguments. Given the text of an argument with premises and conclusion identified, we classify it as an instance of one of five common schemes, using features specific to each scheme.In our experiments, we focused on the five most frequently used schemes, and conducted two kinds of classification. Baseline is 50% in both cases. In one-against-others classification, we achieved over 90% best average accuracies for two schemes, with other three schemes in the 60s to 70s; and in pair-wise classification, we obtained 80% to 90% best average accuracies for most scheme pairs. The poor performance of our classification system on other experimental setups is partly due to the lack of training examples or to insufficient world knowledge.
Reference: Feng and Hirst (2011).
Issue framing: Framing is generally conceptualized as a communication process to present an object or an issue from a particular perspective or point of view. For example, ‘economic benefits’ can be used as a generic frame for various issues; but the frame ‘marriage is about more than procreation’ is specific to the issue of gay marriage. We focus on identifying generic frames at the sentence level in the Media Frames Corpus. We employ both uni- and bi-directional LSTMs and gated recurrent networks to automatically learn frame representations.Given a text about a controversial issue, our goal is to classify each sentence that expresses a frame relating to the issue. (Previous approaches to generic frame classification operated only at the document level.) Our approach achieves at least a 14-point improvement over classifiers trained with topics and other strong baseline models.
References: Hirst etal (2014b), Hirst and Feng (2016), Naderi and Hirst (2016, 2017b).
Reputation defence strategies: Criticisms and persuasive attacks pose threats to reputation or face, and they are common in all social interactions. Allegations are often made against organizations (e.g., companies and governments) and individuals (e.g., medical practitioners and politicians), and various argumentation tactics and persuasive strategies are used in response to these allegations to attempt to defend the respondent’s reputation and thereby save face.We propose a new task of automatically detecting reputation defence strategies in the field of computational argumentation.
We cast the problem as relation classification, where given a reputation threat and adefence responding to it, we determine the particular reputation defence strategy. We annotate a dataset of parliamentary questions and answers with reputation defence strategies. We then propose a model based on supervised learning to address the detection of these strategies, and report promising experimental results. Our proposed model incorporating information regarding threats to reputation can predict reputation defence language with high confidence. Further experiments and evaluations on different datasets show that the model is able to generalize to new utterances and can predict the language of reputation defence in a new dataset.
References: Naderi and Hirst (2017a, 2018a, 2018b).
Automatically evaluating the quality of an argument: With a large group of researchers in Germany, the U.S., and Ireland, led by Henning Wachsmuth, we looked at the problem of how to automatically assess the quality of an argument or argumentation. While different quality dimensions have been approached in natural language processing, a common understanding of argumentation quality is still missing.
Our first paper presented the first holistic work on computational argumentation quality in natural language. We comprehensively surveyed the diverse existing theories and approaches to assess logical, rhetorical, and dialectical quality dimensions, and we derived a systematic taxonomy from these. In addition, we developed a corpus with 320 arguments, annotated for all 15 dimensions in the taxonomy. Our results established a common ground for research on computational argumentation quality assessment.
Our second paper studied the extent to which the different view of argumentation quality in argumentation theory and in practical assessment approaches match empirically. We found that most observations on quality phrased spontaneously are in fact adequately represented by theory. Even more, relative comparisons of arguments in practice correlate with absolute quality ratings based on theory. Our results clarified how the two views can learn from each other.
References: Wachsmuth et al (2017a, 2017b).
See also our research on digitizing and analyzing parliamentary proceedings.
References
Feng, Vanessa Wei and Hirst, Graeme. “Classifying arguments by scheme.” Proceedings, 49th Annual Meeting of the Association for Computational Linguistics, Portland, Oregon, June 2011, 987–996. [PDF]
Hirst, Graeme; Riabinin, Yaroslav; Graham, Jory; Boizot-Roche, Magali; and Morris, Colin. “Text to ideology or text to party status?” In: Kaal, Bertie; Maks, E. Isa; and van Elfrinkhof, Annemarie M.E. (editors), From Text to Political Positions: Text analysis across disciplines, Amsterdam: John Benjamins, 2014a, 93–115. [PDF]
Hirst, Graeme; Feng, Vanessa Wei; Cochrane, Christopher; and Naderi, Nona. “Argumentation, ideology, and issue framing in parliamentary discourse.” Proceedings of the Workshop on Frontiers and Connections between Argumentation Theory and Natural Language Processing, Forlì-Cesena, Italy, July 2014b. CEUR Workshop Proceedings, volume 1341. [PDF]
Hirst, Graeme and Feng, Vanessa Wei. “Automatic exploration of argument and ideology in political texts.” In: Mohammed, Dima and Lewiński, Marcin (editors), Argumentation and Reasoned Action: Proceedings of the First European Conference on Argumentation, Lisbon, 9-12 June 2015, volume II, College Publications (Studies in Logic and Argumentation, volume 52), 2016, 493-504. [PDF]
Naderi, Nona and Hirst, Graeme. “Argumentation mining in parliamentary discourse.” In: Baldoni, Matteo; Baroglio, Cristina; Bex, Floris; Grasso, Floriana; Green, Nancy; Namazi-Rad, Mohammad-Reza; Numao, Masayuki; Suarez, Merlin Teodosia (editors),
Principles and Practice of Multi-Agent Systems, Lecture Notes in Computer Science, volume 9935, 2016, 16–25. [PDF]
Naderi, Nona and Hirst, Graeme. “Recognizing reputation defence strategies in critical political exchanges.” Proceedings, 11th Conference on Recent Advances in Natural Language Processing, Varna, September 2017a, 527–535. [PDF]
Naderi, Nona and Hirst, Graeme. “Classifying frames at the sentence level in news.” Proceedings, 11th Conference on Recent Advances in Natural Language Processing, Varna, September 2017b, 536–542. [PDF]
Naderi, Nona and Hirst, Graeme. “Automatically labeled data generation for classification of reputation defence startegies.”In: Fišer, Darja; Eskevich, Maria; and de Jong, Franciska (eds). Proceedings of LREC2018 Workshop ParlaCLARIN: Creating and Using Parliamentary Corpora, Miyazaki, Japan, May 2018a, 48–54. [PDF]
Naderi, Nona and Hirst, Graeme. “Using context to identify the language of face-saving.” Proceedings, 5th Workshop on Argument Mining, Brussels, November 2018b, 111–120. [PDF]
Wachsmuth, Henning; Naderi, Nona; Hou, Yufang; Bilu, Yonatan; Prabhakaran, Vinod; Alberdingk Thijm, Timothy; Hirst, Graeme; and Stein, Benno. “Computational argumentation quality assessment in natural language.” 15th Conference of the European Chapter of the Association for Computational Linguistics, Valencia, April 2017a, 176–187. [PDF]
Wachsmuth, Henning; Naderi, Nona; Habernal, Ivan; Hou, Yufang; Hirst, Graeme; Gurevych, Iryna; and Stein, Benno. “Argumentation quality assessment: Theory vs. practice.” 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, August 2017b, volume 2, 250–255. [PDF]