Assessing the Impact of Political Involvement on the Reliability and Validity of Virtualized Real-time-response Measurement
Abstract
How does political involvement impact on data quality of virtualized Real-Time-Response (RTR) Measurement? The article addresses this issue, drawing on a large field study (n = 5660) conducted during the 2017 German general election campaign. Since it is unclear how heterogeneous characteristics of individuals influence data quality of RTR-Measurement from audiences surveyed outside the lab, we assess the impact of political involvement on the reliability and validity of virtualized real-time-response data. We show that political involvement shapes reliability and validity systematically but to a degree that does not compromise established standards of data quality. Thus, we conclude that virtualized RTR outside the laboratory with limited means of control is an appropriate method to survey heterogeneous samples in large N-field studies and therefore offers new paths of data collection.
Appendix A1: Measurement Instructions to the participants

Please note the following:
– The Debat-O-Meter records your ratings exact to the second, which means one rating is recorded per second.
– Why you rate a person and what exactly you consider good or bad is left entirely to you.
– If no button gets pressed, no data gets transmitted, which will be interpreted as a “neutral” rating.
– Please do not use the device when the host is speaking.
Appendix A2: Description of the sample
Age | Gender | Education | Political interest | ||||
---|---|---|---|---|---|---|---|
Category | Frequency (%) | Category | Frequency (%) | Category | Frequency (%) | Category | Frequency (%) |
<20 | 403 (9.9) | Female | 1715 (42.5) | No degree | 19 (0.5) | No interest | 14 (0.3) |
20–29 | 1035 (25.3) | Male | 2319 (57.5) | Still in school | 95 (2.3) | Weak | 199 (4.9) |
30–39 | 924 (22.6) | Lower secondary school | 288 (7.1) | Medium | 1285 (31.6) | ||
40–49 | 653 (15.9) | Mid-level secondary school | 1040 (25.5) | Strong | 1670 (41.1) | ||
50–59 | 611 (15.0) | Higher-level secondary school | 1215 (30.2) | Very strong | 893 (23.8) | ||
60–69 | 327 (8.0) | Tertiary education | 1422 (34.9) | ||||
>70 | 139 (3.2) |
References
Bachl, M. 2013. “Die Wirkung des TV-Duells auf die Bewertung der Kandidaten und die Wahlabsicht.” In Das TV-Duell in Baden-Württemberg 2011. Inhalte, Wahrnehmungen und Wirkungen, edited by M. Bachl, F. Brettschneider, and S. Ottler, 171–98. Wiesbaden: Springer VS.10.1007/978-3-658-00792-8_8Search in Google Scholar
Bachl, M. 2014. Analyse rezeptionsbegleitend gemessener Kandidatenbewertungen in TV-Duellen Erweiterung etablierter Verfahren und Vorschlag einer Mehrebenenmodellierung. Berlin: Universität Hohenheim.Search in Google Scholar
Benoit, W. L., G. J. Hansen, and R. M. Verser. 2003. “A Meta-Analysis of the Effects Viewing U.S. Presidential Debates.” Communication Monographs 70 (4): 335–50, https://doi.org/10.1080/0363775032000179133.Search in Google Scholar
Biocca, F., P. David, and M. West. 1994. “Continuous Response Measurement (CRM): A Computerized Tool for Research on the Cognitive Processing of Communication Messages.” In Measuring Psychological Responses to Media Messages, edited by A. Lang, 15–64. Hillsdale: Erlbaum.Search in Google Scholar
Bortz, J., and N. Döring. 2006. Forschungsmethoden und Evaluation für Human-und Sozialwissenschaftler., 4., überarbeitete Auflage. Heidelberg: Springer, https://doi.org/10.1007/978-3-540-33306-7.Search in Google Scholar
Boyd, T. C., and G. D. Hughes. 1992. “Validating Realtime Response Measures.” NA-Advances in Consumer Research 19.Search in Google Scholar
Boydstun, A. E., R. A. Glazier, M. T. Pietryka, and P. Resnik. 2014. “Real-Time Reactions to a 2012 Presidential Debate A Method for Understanding Which Messages Matter.” Public Opinion Quarterly 78 (Special issue): 330–43, https://doi.org/10.1093/poq/nfu007.Search in Google Scholar
Campbell, A., P. Converse, W. Miller, and D. Stokes. 1960. The American Voter. New York, NY: Wiley.Search in Google Scholar
Druckman, J. N. 2003. “The Power of Television Images: The First Kennedy-Nixon Debate Revisited.” The Journal of Politics 65 (2): 559–71, https://doi.org/10.1111/1468-2508.t01-1-00015.Search in Google Scholar
Eulau, H., and P. Schneider. 1956. “Dimensions of Political Involvement.” Public Opinion Quarterly 20 (1): 128–42, https://doi.org/10.1086/266603.Search in Google Scholar
Faas, T., and J. Maier. 2011. “Medienwahlkampf. Sind TV-Duelle nur Show und damit nutzlos?” In Der unbekannte Wähler? Mythen und Fakten über das Wahlverhalten der Deutschen, edited by E. Bytzek, and S. Roßteutscher, 99–114. Frankfurt/New York: Campus.Search in Google Scholar
Faas, T., J. Maier, and M. Maier. 2017. Merkel gegen Steinbrück: Analysen zum TV-Duell vor der Bundestagswahl 2013. Wiesbaden: Springer-Verlag.10.1007/978-3-658-05432-8Search in Google Scholar
Fenwick, I., and M. D. Rice. 1991. “Reliability of Continuous Measurement Copy-Testing Methods.” Journal of Advertising Research 31 (1): 23–9, http://psycnet.apa.org/psycinfo/1991-26085-001.Search in Google Scholar
Green, D., B. Palmquist, and E. Schickler. 2002. Partisan Hearts and Minds. Political Parties and the Social Identity of Voters. New Haven, CT: Yale University Press.Search in Google Scholar
Hallonquist, T., and J. G. Peatman. 1947. “Diagnosing your Radio Program, or the Program Analyzer at Work.” In Education on the Air. Yearbook of the Institute for Education by Radio, edited by Institute for Education by Radio, 463–74. OH: Ohio State University Press.Search in Google Scholar
Hallonquist, T., and E. E. Suchmann. 1944. “Listening to the Listener. Experiences with the Lazarsfeld-Stanton Program Analyzer.” In Radio Research 1942-1943, edited by P. F. Lazarsfeld, and F. Stanton, 265–334. New York: Arno Press.Hughes.Search in Google Scholar
Holbrook, T. M. 2002. “Presidential Campaigns and the Knowledge Gap.” Political Communication 19 (4): 437–54, https://doi.org/10.1080/10584600290109997.Search in Google Scholar
Hughes, G. D., and R. Lennox. 1990. “Realtime Response Research: Construct Validation and Reliability Assessment.” In Enhancing Knowledge Development in Marketing, edited by W. Bearden, et al.., 284–8. Chicago, IL: American Marketing Association.Search in Google Scholar
Lazarsfeld, P. F., B. Berelson, and H. Gaudet. 1944. The People’s Choice. New York, NY: Columbia University Press.Search in Google Scholar
Maier, J. 2007a. “Eine Basis für rationale Wahlentscheidungen? Die Wirkungen des TV-Duells auf politische Kenntnisse.” In Schröder gegen Merkel. Wahrnehmung und Wirkung des TV-Duells 2005 im Ost-West-Vergleich, edited by M. Maurer, C. Reinemann, J. Maier, and M. Maier, 129–43. Wiesbaden: VS Verlag.10.1007/978-3-531-90709-3_7Search in Google Scholar
Maier, J. 2007b. “Erfolgreiche Überzeugungsarbeit. Urteile über den Debattensieger und die Veränderung der Kanzlerpräferenz.” In Schröder gegen Merkel. Wahrnehmung und Wirkung des TV-Duells 2005 im Ost-West-Vergleich, edited by M. Maurer, C. Reinemann, J. Maier, and M. Maier, 91–109. Wiesbaden: VS Verlag.10.1007/978-3-531-90709-3_5Search in Google Scholar
Maier, J., and T. Faas. 2011. “‘Miniature Campaigns’ in Comparison: The German Televised Debates, 2002–09.” German Politics 20 (1): 75–91.10.1080/09644008.2011.554102Search in Google Scholar
Maier, J., T. Faas, and M. Maier. 2014. “Aufgeholt, aber nicht aufgeschlossen: Ausgewählte Befunde zur Wahrnehmung und Wirkung des TV-Duells 2013 zwischen Angela Merkel und Peer Steinbrück.” Zeitschrift für Parlamentsfragen 45 (1): 38–54.10.5771/0340-1758-2014-1-38Search in Google Scholar
Maier, J., T. Faas, B. Rittberger, J. Fortin-Rittberger, K. A. Josifides, S. Banducci, P. Bellucci, M. Blomgren, I. Brikse, and K. Chwedczuk-Szulc. 2018. “This Time It’s Different? Effects of the Eurovision Debate on Young Citizens and its Consequence for EU Democracy–Evidence from a Quasi-Experiment in 24 Countries.” Journal of European Public Policy 25 (4): 606–29.10.1080/13501763.2016.1268643Search in Google Scholar
Maier, J., J. F. Hampe, and N. Jahn. 2016a. “Breaking Out of the Lab Measuring Real-Time Responses to Televised Political Content in Real-World Settings.” Public Opinion Quarterly 80 (2): 542–53.10.1093/poq/nfw010Search in Google Scholar
Maier, J., M. Maier, M. Maurer, C. Reinemann, and V. Meyer, eds. 2009. Real-Time Response Measurement in the Social Sciences. Methodological Perspectives and Applications. Peter Lang.Search in Google Scholar
Maier, J., M. Maurer, C. Reinemann, and T. Faas. 2007. “Reliability and Validity of Real-Time Response Measurement: A Comparison of Two Studies of a Televised Debate in Germany.” International Journal of Public Opinion Research 19 (1): 53–73.10.1093/ijpor/edl002Search in Google Scholar
Maier, J., B. Rittberger, and T. Faas. 2016b. “Debating Europe: Effects of the “Eurovision Debate” on EU Attitudes of Young German Voters and the Moderating Role Played by Political Involvement.” Politics and Governance 4 (1): 55–68.10.17645/pag.v4i1.456Search in Google Scholar
Maier, M., and J. Strömbäck. 2009. “Advantages and Limitations of Comparative Audience Responses to Televised Debates: A Comparative Study of Germany and Sweden.” In Real-Time Response Measurement in the Social Sciences. Methodological Perspectives and Applications, edited by J. Maier, M. Maier, M. Maurer, C. Reinemann, and V. Meyer, 97–116. Frankfurt: Peter Lang.Search in Google Scholar
Maurer. 2009. Sagen Bilder mehr als tausend Worte? https://doi.org/10.5771/1615-634x-2009-2-198.Search in Google Scholar
Maurer, M., and C. Reinemann. 2003. Schröder gegen Stoiber: Nutzung, Wahrnehmung und Wirkung der TV-Duelle. Wiesbaden: Westdeutscher Verlag.10.1007/978-3-322-80456-3Search in Google Scholar
Maurer, M., and C. Reinemann. 2015. “Do Uninvolved Voters Rely on Visual MessageElements? A Test of a Central Assumption of the ELM in the Context of Televised Debates.” Politische Psychologie 4 (2): 235–51.Search in Google Scholar
Maurer, M., C. Reinemann, J. Maier, and M. Maier, eds. 2007. Schröder gegen Merkel. Wahrnehmung und Wirkung des TV-Duells 2005 im Ost-West-Vergleich. Wiesbaden: VS Verlag.Search in Google Scholar
McKinney, M. S., and D. B. Carlin. 2004. “Political Campaign Debates.” In Handbook of Political Communication Research, edited by L. L. Kaid, 203–34. Mahwah: Lawrence Erlbaum Associates.Search in Google Scholar
Metz, T., U. Wagschal, T. Waldvogel, M. Bachl, L. Feiten, and B. Becker. 2016. “Das Debat-O-Meter: ein neues Instrument zur Analyse von TV-Duellen. ZSE Zeitschrift für Staats- und Europawissenschaften.” Journal for Comparative Government and European Policy 14 (1): 124–49, doi:https://doi.org/10.5771/1610-7780-2016-1-124.10.5771/1610-7780-2016-1-124Search in Google Scholar
Mullinix, K. J. 2015. “Presidential Debates, Partisan Motivations, and Political Interest.” Presidential Studies Quarterly 45 (2): 270–88.10.1111/psq.12187Search in Google Scholar
Nagel, F. 2012. Die Wirkung verbaler und nonverbaler Kommunikation in TV-Duellen. Wiesbaden: VS Verlag.10.1007/978-3-531-93497-6Search in Google Scholar
Otto, L., M. Maier, and I. Glogger. 2015. “Image- or Issue-Orientation? How the Presentation Modality Influences the Perception of Candidates in Televised Debates.” Politische Psychologie 4 (2): 215–34.Search in Google Scholar
Papastefanou, G. 2013. Reliability and Validity of RTR Measurement Device. (Working Paper No. 27). Mannheim: GESIS - Leibniz-Institut für Sozialwissenschaften.Search in Google Scholar
Petty, R. E., and J. T. Cacioppo. 1986. Communication and Persuasion: Central and Peripheral Routes to Attitude Change. New York, NY: Springer.10.1007/978-1-4612-4964-1Search in Google Scholar
Range, J. 2017. “Wissens-und Partizipations-Gaps: Führte das TV-Duell 2013 zu einer politischen und kognitiven Mobilisierung?” In Merkel gegen Steinbrück, edited by T. Faas, J. Maier, and M. Maier, 75–86. Wiesbaden: Springer VS.10.1007/978-3-658-05432-8_5Search in Google Scholar
Reinemann, C., J. Maier, T. Faas, and M. Maurer. 2005. “Reliabilität und Validität von RTR-Messungen.” Publizistik 50 (1): 56–73.10.1007/s11616-005-0118-4Search in Google Scholar
Reinemann, C., and M. Maurer. 2005. “Unifying or Polarizing? Short-Term Effects and Postdebate Consequences of Different Rhetorical Strategies in Televised Debates.” Journal of Communication 55 (4): 775–94.10.1111/j.1460-2466.2005.tb03022.xSearch in Google Scholar
Reinemann, C., and M. Maurer. 2010. “Leichtgläubig und manipulierbar? Die Rezeption persuasiver Wahlkampfbotschaften durch politisch Interessierte und Desinteressierte.” In Information – Wahrnehmung – Emotion, edited by T. Faas, K. Arzheimer, and S. Roßteutscher, 239–57. Wiesbaden: VS Verlag für Sozialwissenschaften.10.1007/978-3-531-92336-9_12Search in Google Scholar
Schill, D., R. Kirk, and A. E. Jasperson. 2016. Political Communication in Real Time: Theoretical and Applied Research Approaches. New York, NY/London: Routledge/ Taylor & Francis Group.10.4324/9781315669083Search in Google Scholar
Schwerin, H. 1940. “An Exploratory Study of the Reliability of the “Program Analyzer”.” Journal of Applied Psychology 24 (6): 742–5.10.1037/h0058363Search in Google Scholar
Waldvogel, T., and T. Metz. 2020. “Measuring Real-Time Responses in Real-Life Settings.” International Journal of Public Opinion Research, https://doi.org/10.1093/ijpor/edz050.Search in Google Scholar
Wagschal, U. 1999. Statistik für Politikwissenschaftler. München/Wien: Oldenbourg.10.1515/9783486791204Search in Google Scholar
Weiber, R., and D. Mühlhaus. 2014. Strukturgleichungsmodellierung. Heidelberg: Springer.10.1007/978-3-642-35012-2Search in Google Scholar
© 2021 Walter de Gruyter GmbH, Berlin/Boston
Articles in the same Issue
- Frontmatter
- Editors’ Note
- Articles
- Debate Reaction Ideal Points: Political Ideology Measurement Using Real-Time Reaction Data
- Heuristic Synchronization of Real-Time Response Data
- Assessing the Impact of Political Involvement on the Reliability and Validity of Virtualized Real-time-response Measurement
- Tracing Policy-relevant Information in Social Media: The Case of Twitter before and during the COVID-19 Crisis
- Election and Public Administration Research
- Statistical Considerations on Political Responsivity in Italy Analyzed Over the 70-Year Time Period from 1948 to 2018
- The Results of Exit Polls in Kansas to Verify Voting Machine Counts in the November 2016 Election
- External Peer Challenge in Local Government: The Role of Spatial Spillover and Past Performance
Articles in the same Issue
- Frontmatter
- Editors’ Note
- Articles
- Debate Reaction Ideal Points: Political Ideology Measurement Using Real-Time Reaction Data
- Heuristic Synchronization of Real-Time Response Data
- Assessing the Impact of Political Involvement on the Reliability and Validity of Virtualized Real-time-response Measurement
- Tracing Policy-relevant Information in Social Media: The Case of Twitter before and during the COVID-19 Crisis
- Election and Public Administration Research
- Statistical Considerations on Political Responsivity in Italy Analyzed Over the 70-Year Time Period from 1948 to 2018
- The Results of Exit Polls in Kansas to Verify Voting Machine Counts in the November 2016 Election
- External Peer Challenge in Local Government: The Role of Spatial Spillover and Past Performance