Home Assessing the Impact of Political Involvement on the Reliability and Validity of Virtualized Real-time-response Measurement
Article
Licensed
Unlicensed Requires Authentication

Assessing the Impact of Political Involvement on the Reliability and Validity of Virtualized Real-time-response Measurement

  • Thomas Waldvogel EMAIL logo , Thomas Metz , Uwe Wagschal , Bernd Becker , Linus Feiten and Samuel Weishaupt
Published/Copyright: May 18, 2021
Become an author with De Gruyter Brill

Abstract

How does political involvement impact on data quality of virtualized Real-Time-Response (RTR) Measurement? The article addresses this issue, drawing on a large field study (n = 5660) conducted during the 2017 German general election campaign. Since it is unclear how heterogeneous characteristics of individuals influence data quality of RTR-Measurement from audiences surveyed outside the lab, we assess the impact of political involvement on the reliability and validity of virtualized real-time-response data. We show that political involvement shapes reliability and validity systematically but to a degree that does not compromise established standards of data quality. Thus, we conclude that virtualized RTR outside the laboratory with limited means of control is an appropriate method to survey heterogeneous samples in large N-field studies and therefore offers new paths of data collection.


Corresponding author: Thomas Waldvogel, Political Science, Albert-Ludwigs-Universitat Freiburg, Freiburg im Breisgau, Germany, E-mail:

Appendix

Appendix A1: Measurement Instructions to the participants

Please note the following:

  • – The Debat-O-Meter records your ratings exact to the second, which means one rating is recorded per second.

  • – Why you rate a person and what exactly you consider good or bad is left entirely to you.

  • – If no button gets pressed, no data gets transmitted, which will be interpreted as a “neutral” rating.

  • – Please do not use the device when the host is speaking.

Appendix A2: Description of the sample

Age Gender Education Political interest
Category Frequency (%) Category Frequency (%) Category Frequency (%) Category Frequency (%)
<20 403 (9.9) Female 1715 (42.5) No degree 19 (0.5) No interest 14 (0.3)
20–29 1035 (25.3) Male 2319 (57.5) Still in school 95 (2.3) Weak 199 (4.9)
30–39 924 (22.6) Lower secondary school 288 (7.1) Medium 1285 (31.6)
40–49 653 (15.9) Mid-level secondary school 1040 (25.5) Strong 1670 (41.1)
50–59 611 (15.0) Higher-level secondary school 1215 (30.2) Very strong 893 (23.8)
60–69 327 (8.0) Tertiary education 1422 (34.9)
>70 139 (3.2)

References

Bachl, M. 2013. “Die Wirkung des TV-Duells auf die Bewertung der Kandidaten und die Wahlabsicht.” In Das TV-Duell in Baden-Württemberg 2011. Inhalte, Wahrnehmungen und Wirkungen, edited by M. Bachl, F. Brettschneider, and S. Ottler, 171–98. Wiesbaden: Springer VS.10.1007/978-3-658-00792-8_8Search in Google Scholar

Bachl, M. 2014. Analyse rezeptionsbegleitend gemessener Kandidatenbewertungen in TV-Duellen Erweiterung etablierter Verfahren und Vorschlag einer Mehrebenenmodellierung. Berlin: Universität Hohenheim.Search in Google Scholar

Benoit, W. L., G. J. Hansen, and R. M. Verser. 2003. “A Meta-Analysis of the Effects Viewing U.S. Presidential Debates.” Communication Monographs 70 (4): 335–50, https://doi.org/10.1080/0363775032000179133.Search in Google Scholar

Biocca, F., P. David, and M. West. 1994. “Continuous Response Measurement (CRM): A Computerized Tool for Research on the Cognitive Processing of Communication Messages.” In Measuring Psychological Responses to Media Messages, edited by A. Lang, 15–64. Hillsdale: Erlbaum.Search in Google Scholar

Bortz, J., and N. Döring. 2006. Forschungsmethoden und Evaluation für Human-und Sozialwissenschaftler., 4., überarbeitete Auflage. Heidelberg: Springer, https://doi.org/10.1007/978-3-540-33306-7.Search in Google Scholar

Boyd, T. C., and G. D. Hughes. 1992. “Validating Realtime Response Measures.” NA-Advances in Consumer Research 19.Search in Google Scholar

Boydstun, A. E., R. A. Glazier, M. T. Pietryka, and P. Resnik. 2014. “Real-Time Reactions to a 2012 Presidential Debate A Method for Understanding Which Messages Matter.” Public Opinion Quarterly 78 (Special issue): 330–43, https://doi.org/10.1093/poq/nfu007.Search in Google Scholar

Campbell, A., P. Converse, W. Miller, and D. Stokes. 1960. The American Voter. New York, NY: Wiley.Search in Google Scholar

Druckman, J. N. 2003. “The Power of Television Images: The First Kennedy-Nixon Debate Revisited.” The Journal of Politics 65 (2): 559–71, https://doi.org/10.1111/1468-2508.t01-1-00015.Search in Google Scholar

Eulau, H., and P. Schneider. 1956. “Dimensions of Political Involvement.” Public Opinion Quarterly 20 (1): 128–42, https://doi.org/10.1086/266603.Search in Google Scholar

Faas, T., and J. Maier. 2011. “Medienwahlkampf. Sind TV-Duelle nur Show und damit nutzlos?” In Der unbekannte Wähler? Mythen und Fakten über das Wahlverhalten der Deutschen, edited by E. Bytzek, and S. Roßteutscher, 99–114. Frankfurt/New York: Campus.Search in Google Scholar

Faas, T., J. Maier, and M. Maier. 2017. Merkel gegen Steinbrück: Analysen zum TV-Duell vor der Bundestagswahl 2013. Wiesbaden: Springer-Verlag.10.1007/978-3-658-05432-8Search in Google Scholar

Fenwick, I., and M. D. Rice. 1991. “Reliability of Continuous Measurement Copy-Testing Methods.” Journal of Advertising Research 31 (1): 23–9, http://psycnet.apa.org/psycinfo/1991-26085-001.Search in Google Scholar

Green, D., B. Palmquist, and E. Schickler. 2002. Partisan Hearts and Minds. Political Parties and the Social Identity of Voters. New Haven, CT: Yale University Press.Search in Google Scholar

Hallonquist, T., and J. G. Peatman. 1947. “Diagnosing your Radio Program, or the Program Analyzer at Work.” In Education on the Air. Yearbook of the Institute for Education by Radio, edited by Institute for Education by Radio, 463–74. OH: Ohio State University Press.Search in Google Scholar

Hallonquist, T., and E. E. Suchmann. 1944. “Listening to the Listener. Experiences with the Lazarsfeld-Stanton Program Analyzer.” In Radio Research 1942-1943, edited by P. F. Lazarsfeld, and F. Stanton, 265–334. New York: Arno Press.Hughes.Search in Google Scholar

Holbrook, T. M. 2002. “Presidential Campaigns and the Knowledge Gap.” Political Communication 19 (4): 437–54, https://doi.org/10.1080/10584600290109997.Search in Google Scholar

Hughes, G. D., and R. Lennox. 1990. “Realtime Response Research: Construct Validation and Reliability Assessment.” In Enhancing Knowledge Development in Marketing, edited by W. Bearden, et al.., 284–8. Chicago, IL: American Marketing Association.Search in Google Scholar

Lazarsfeld, P. F., B. Berelson, and H. Gaudet. 1944. The People’s Choice. New York, NY: Columbia University Press.Search in Google Scholar

Maier, J. 2007a. “Eine Basis für rationale Wahlentscheidungen? Die Wirkungen des TV-Duells auf politische Kenntnisse.” In Schröder gegen Merkel. Wahrnehmung und Wirkung des TV-Duells 2005 im Ost-West-Vergleich, edited by M. Maurer, C. Reinemann, J. Maier, and M. Maier, 129–43. Wiesbaden: VS Verlag.10.1007/978-3-531-90709-3_7Search in Google Scholar

Maier, J. 2007b. “Erfolgreiche Überzeugungsarbeit. Urteile über den Debattensieger und die Veränderung der Kanzlerpräferenz.” In Schröder gegen Merkel. Wahrnehmung und Wirkung des TV-Duells 2005 im Ost-West-Vergleich, edited by M. Maurer, C. Reinemann, J. Maier, and M. Maier, 91–109. Wiesbaden: VS Verlag.10.1007/978-3-531-90709-3_5Search in Google Scholar

Maier, J., and T. Faas. 2011. “‘Miniature Campaigns’ in Comparison: The German Televised Debates, 2002–09.” German Politics 20 (1): 75–91.10.1080/09644008.2011.554102Search in Google Scholar

Maier, J., T. Faas, and M. Maier. 2014. “Aufgeholt, aber nicht aufgeschlossen: Ausgewählte Befunde zur Wahrnehmung und Wirkung des TV-Duells 2013 zwischen Angela Merkel und Peer Steinbrück.” Zeitschrift für Parlamentsfragen 45 (1): 38–54.10.5771/0340-1758-2014-1-38Search in Google Scholar

Maier, J., T. Faas, B. Rittberger, J. Fortin-Rittberger, K. A. Josifides, S. Banducci, P. Bellucci, M. Blomgren, I. Brikse, and K. Chwedczuk-Szulc. 2018. “This Time It’s Different? Effects of the Eurovision Debate on Young Citizens and its Consequence for EU Democracy–Evidence from a Quasi-Experiment in 24 Countries.” Journal of European Public Policy 25 (4): 606–29.10.1080/13501763.2016.1268643Search in Google Scholar

Maier, J., J. F. Hampe, and N. Jahn. 2016a. “Breaking Out of the Lab Measuring Real-Time Responses to Televised Political Content in Real-World Settings.” Public Opinion Quarterly 80 (2): 542–53.10.1093/poq/nfw010Search in Google Scholar

Maier, J., M. Maier, M. Maurer, C. Reinemann, and V. Meyer, eds. 2009. Real-Time Response Measurement in the Social Sciences. Methodological Perspectives and Applications. Peter Lang.Search in Google Scholar

Maier, J., M. Maurer, C. Reinemann, and T. Faas. 2007. “Reliability and Validity of Real-Time Response Measurement: A Comparison of Two Studies of a Televised Debate in Germany.” International Journal of Public Opinion Research 19 (1): 53–73.10.1093/ijpor/edl002Search in Google Scholar

Maier, J., B. Rittberger, and T. Faas. 2016b. “Debating Europe: Effects of the “Eurovision Debate” on EU Attitudes of Young German Voters and the Moderating Role Played by Political Involvement.” Politics and Governance 4 (1): 55–68.10.17645/pag.v4i1.456Search in Google Scholar

Maier, M., and J. Strömbäck. 2009. “Advantages and Limitations of Comparative Audience Responses to Televised Debates: A Comparative Study of Germany and Sweden.” In Real-Time Response Measurement in the Social Sciences. Methodological Perspectives and Applications, edited by J. Maier, M. Maier, M. Maurer, C. Reinemann, and V. Meyer, 97–116. Frankfurt: Peter Lang.Search in Google Scholar

Maurer. 2009. Sagen Bilder mehr als tausend Worte? https://doi.org/10.5771/1615-634x-2009-2-198.Search in Google Scholar

Maurer, M., and C. Reinemann. 2003. Schröder gegen Stoiber: Nutzung, Wahrnehmung und Wirkung der TV-Duelle. Wiesbaden: Westdeutscher Verlag.10.1007/978-3-322-80456-3Search in Google Scholar

Maurer, M., and C. Reinemann. 2015. “Do Uninvolved Voters Rely on Visual MessageElements? A Test of a Central Assumption of the ELM in the Context of Televised Debates.” Politische Psychologie 4 (2): 235–51.Search in Google Scholar

Maurer, M., C. Reinemann, J. Maier, and M. Maier, eds. 2007. Schröder gegen Merkel. Wahrnehmung und Wirkung des TV-Duells 2005 im Ost-West-Vergleich. Wiesbaden: VS Verlag.Search in Google Scholar

McKinney, M. S., and D. B. Carlin. 2004. “Political Campaign Debates.” In Handbook of Political Communication Research, edited by L. L. Kaid, 203–34. Mahwah: Lawrence Erlbaum Associates.Search in Google Scholar

Metz, T., U. Wagschal, T. Waldvogel, M. Bachl, L. Feiten, and B. Becker. 2016. “Das Debat-O-Meter: ein neues Instrument zur Analyse von TV-Duellen. ZSE Zeitschrift für Staats- und Europawissenschaften.” Journal for Comparative Government and European Policy 14 (1): 124–49, doi:https://doi.org/10.5771/1610-7780-2016-1-124.10.5771/1610-7780-2016-1-124Search in Google Scholar

Mullinix, K. J. 2015. “Presidential Debates, Partisan Motivations, and Political Interest.” Presidential Studies Quarterly 45 (2): 270–88.10.1111/psq.12187Search in Google Scholar

Nagel, F. 2012. Die Wirkung verbaler und nonverbaler Kommunikation in TV-Duellen. Wiesbaden: VS Verlag.10.1007/978-3-531-93497-6Search in Google Scholar

Otto, L., M. Maier, and I. Glogger. 2015. “Image- or Issue-Orientation? How the Presentation Modality Influences the Perception of Candidates in Televised Debates.” Politische Psychologie 4 (2): 215–34.Search in Google Scholar

Papastefanou, G. 2013. Reliability and Validity of RTR Measurement Device. (Working Paper No. 27). Mannheim: GESIS - Leibniz-Institut für Sozialwissenschaften.Search in Google Scholar

Petty, R. E., and J. T. Cacioppo. 1986. Communication and Persuasion: Central and Peripheral Routes to Attitude Change. New York, NY: Springer.10.1007/978-1-4612-4964-1Search in Google Scholar

Range, J. 2017. “Wissens-und Partizipations-Gaps: Führte das TV-Duell 2013 zu einer politischen und kognitiven Mobilisierung?” In Merkel gegen Steinbrück, edited by T. Faas, J. Maier, and M. Maier, 75–86. Wiesbaden: Springer VS.10.1007/978-3-658-05432-8_5Search in Google Scholar

Reinemann, C., J. Maier, T. Faas, and M. Maurer. 2005. “Reliabilität und Validität von RTR-Messungen.” Publizistik 50 (1): 56–73.10.1007/s11616-005-0118-4Search in Google Scholar

Reinemann, C., and M. Maurer. 2005. “Unifying or Polarizing? Short-Term Effects and Postdebate Consequences of Different Rhetorical Strategies in Televised Debates.” Journal of Communication 55 (4): 775–94.10.1111/j.1460-2466.2005.tb03022.xSearch in Google Scholar

Reinemann, C., and M. Maurer. 2010. “Leichtgläubig und manipulierbar? Die Rezeption persuasiver Wahlkampfbotschaften durch politisch Interessierte und Desinteressierte.” In Information – Wahrnehmung – Emotion, edited by T. Faas, K. Arzheimer, and S. Roßteutscher, 239–57. Wiesbaden: VS Verlag für Sozialwissenschaften.10.1007/978-3-531-92336-9_12Search in Google Scholar

Schill, D., R. Kirk, and A. E. Jasperson. 2016. Political Communication in Real Time: Theoretical and Applied Research Approaches. New York, NY/London: Routledge/ Taylor & Francis Group.10.4324/9781315669083Search in Google Scholar

Schwerin, H. 1940. “An Exploratory Study of the Reliability of the “Program Analyzer”.” Journal of Applied Psychology 24 (6): 742–5.10.1037/h0058363Search in Google Scholar

Waldvogel, T., and T. Metz. 2020. “Measuring Real-Time Responses in Real-Life Settings.” International Journal of Public Opinion Research, https://doi.org/10.1093/ijpor/edz050.Search in Google Scholar

Wagschal, U. 1999. Statistik für Politikwissenschaftler. München/Wien: Oldenbourg.10.1515/9783486791204Search in Google Scholar

Weiber, R., and D. Mühlhaus. 2014. Strukturgleichungsmodellierung. Heidelberg: Springer.10.1007/978-3-642-35012-2Search in Google Scholar

Received: 2020-11-26
Accepted: 2021-04-26
Published Online: 2021-05-18
Published in Print: 2021-06-25

© 2021 Walter de Gruyter GmbH, Berlin/Boston

Downloaded on 28.9.2025 from https://www.degruyterbrill.com/document/doi/10.1515/spp-2020-0014/html
Scroll to top button