Home Validation of Assessment Vignettes and Scoring Rubric of Multicultural and International Competency in Faculty Teaching
Article
Licensed
Unlicensed Requires Authentication

Validation of Assessment Vignettes and Scoring Rubric of Multicultural and International Competency in Faculty Teaching

  • Sheila J. Henderson EMAIL logo , Ruth A. Horton , Paul K. Saito and Kumea Shorter-Gooden
Published/Copyright: July 29, 2014
Become an author with De Gruyter Brill

Abstract

The purpose of this research was to develop a new tool for assessing multicultural and international competency in faculty teaching through vignette scenarios of university classroom critical incidents – across disciplines of clinical and forensics psychology, business, and education. Construct and content validity of the initial draft vignettes and the associated scoring rubric was established by a panel of experts who approved the vignettes as consistent with current teaching models and multicultural and international competency models. Each vignette presented a critical incident needing resolution – a rupture in the relationship between the instructor and students vis-à-vis a diversity issue. These hypothetical ruptures took place in university classrooms in four different disciplines: clinical and forensic psychology, business, and education. The authors recruited ten faculty participants for the online study, who were personally known to the authors as interested in competency-related issues and who were teaching in graduate programs. Each participant responded to three questions for each vignette, which prompted their written feedback on (a) process and content issues surrounding the critical incident as well as (b) coaching for the hypothetical instructor on how to resolve the incident. Five scorers rated the participants’ written responses according to a five-point scoring rubric. In the data analysis, once statistical conclusion validity and interscorer reliability was established (Krippendorff’s α = 0.887), content validity was again explored by assessing whether the written responses were indeed captured by the guiding models. The results, which indicated preliminary validity and reliability of the vignettes and the rubric, suggest that the vignettes are worthy of further testing. These vignettes can also be used as discussion tools to stimulate dialogue in competency trainings for faculty.

References

Angelides, P., & Gibbs, P. (2004). Supporting the continued professional development of teachers through the use of vignettes. Teacher Education Quarterly, Fall, 111121. Retrieved from http://www.teqjournal.org/backvols/2006/33_4/18angelides&gibbs.pdfSearch in Google Scholar

Banks, J. A., McGee Banks, C. A., Cortez, C. E., Hahn, C. L., Merryfield, M. W., Moodley, K. A., … Parker, W. C. (2005). Democracy and diversity: Principles and concepts for educating citizens in a global age. Center for Multicultural Education, College of Education University of Washington, Seattle. Retrieved from http://education.washington.edu/cme/DemDiv.pdfSearch in Google Scholar

Barter, C., & Renold, E. (2000). “I wanna tell you a story”: Exploring the application of vignettes in qualitative research with children and young people. International Journal of Social Research Methodology, 3(4), 307323. doi:10.1080/13645570050178594Search in Google Scholar

Bennett, J. (1993). Toward ethnorelativism: A developmental model of intercultural sensitivity. In R. M.Paige (Ed.), Education for the intercultural experience (pp. 2171). Yarmouth, ME: Intercultural Press.Search in Google Scholar

Bhawuk, D., & Brislin, R. (1992). The measurement of intercultural sensitivity using the concepts of individualism and collectivism. International Journal of Intercultural Relations, 16, 413436. doi:10.1016/0147–1767(92)90031-O.Search in Google Scholar

Borrayo, E. (2008). Challenging dialogues: Working through resistance toward multicultural understanding. In J.White & S.Henderson (Eds.), Building multicultural competency: Development, training, and practice (pp.155166). Lanham, MD: Rowman & Littlefield Publishers.Search in Google Scholar

Burgoyne, S., Placier, P., Thomas, M., Welch, S., RuffinC., Flores, L. Y., … Miller, M. (2007). Interactive theater and self-efficacy. New Directions for Teaching and Learning, 111, 2126. doi:10.1002/tl.282.Search in Google Scholar

Cabrera, A. F., Colbeck, C. L., & Terenzini, P. T. (2001). Developing performance indicators for assessing classroom teaching practices and student learning. Research in Higher Education, 42(3), 327352. doi:10.1023/A:1018874023323Search in Google Scholar

Chapman, V. G., & Inman, M. (2009). A Conundrum: Rubrics or creativity/metacognitive development?Education Digest: Essential Readings Condensed For Quick Review, 75(2), 5356.Search in Google Scholar

Constantine, M. G., Gloria, A. M., & Ladany, N. (2002). The factor structure underlying three self-report multicultural counseling competence. Cultural Diversity and Ethnic Minority Psychology, 8(4), 334345. doi: 10.1037/1099-9809.8.4.335Search in Google Scholar

Constantine, M., & Ladany, N. (2000). Self-report multicultural counseling competency scales: Their relation to social desirability attitudes and multicultural case conceptualization ability. Journal of Counseling Psychology, 47, 155164.10.1037/0022-0167.47.2.155Search in Google Scholar

Cooper, B., & Gargan, A. (2009). Rubrics in education: Old term, new meanings. Phi Delta Kappan, 91(1), 5455.10.1177/003172170909100109Search in Google Scholar

Cronbach, L. J., & Meehl, R. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281302.10.1037/h0040957Search in Google Scholar

D’Andrea, M., Daniels, J., & Heck, R. (1991). Evaluating the impact of multicultural counseling training. Journal of Counseling & Development, 70, 143150. doi:10.1002/j.1556-6676.1991.tb01576.xSearch in Google Scholar

Dovidio, J., & Gaertner, S. (1986). Prejudice, discrimination, and racism. Orlando, FL: Academic Press.Search in Google Scholar

Fantini, A. (2007). Exploring and assessing intercultural competence. Retrieved from http://csd.wustl.edu/Publications/Documents/RP07-01.pdfSearch in Google Scholar

Finch, J. (1987). The vignette technique in survey research. Sociology, 21(1), 105114. doi:10.1177/0038038587021001008Search in Google Scholar

Hammer, M., Bennett, M., & Wiseman, R. (2003). Measuring intercultural sensitivity: The intercultural development inventory. International Journal of Intercultural Relations, 27, 421443. doi:10.1016/S0147-1767(03)00032-4Search in Google Scholar

Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1, 7789. doi:10.1080/19312450709336664Search in Google Scholar

Hays, D. (2008). Assessing Multicultural competence in counselor trainees: A review of instrumentation and future directions. Journal of Counseling & Development, 86, 95101. doi:10.1002/j.1556-6678.2008.tb00630.xSearch in Google Scholar

Isaeva, T. E. (2005). Classification of the college instructor’s professional and personalcompetencies. Russian Education and Society, 49(9), 3141.10.2753/RES1060-9393490904Search in Google Scholar

Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 173.10.1111/jedm.12000Search in Google Scholar

Kelley, C., & Meyers, J. (1995). Cross-cultural adaptability index. Minneapolis, MN: National Computer Systems.Search in Google Scholar

Kitano, M. (1996). What a course will look like after multicultural change. In A.Morey & M.Kitano (Eds.), Multicultural course transformation in higher education: A broader truth. Boston, MA: Allyn & Bacon.Search in Google Scholar

Krathwohl, D. (2002). A revision of Bloom’s taxonomy. Theory into Practice, 41(4), 212218. doi:10.1207/s15430421tip4104_2Search in Google Scholar

LaFromboise, T., Coleman, H., & Hernandez, A. (1991). Development and factor structure of the cross-cultural counseling inventory-revised. Professional Psychology: Research and Practice, 22, 380388. doi:10.1037/0735-7028.22.5.380Search in Google Scholar

Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2004). Practical resources for assessing and reporting intercoder reliability in content analysis research projects. Retrieved from: http://ils.indiana.edu/faculty/hrosenba/www/Research/methods/lombard_reliability.pdfSearch in Google Scholar

Miles, M. (1990). New methods for qualitative data collection and analysis: Vignettes and prestructured cases. International Journal of Qualitative Studies in Education, 3(1), 3751. doi:10.1080/0951839900030104Search in Google Scholar

Mobley, M., Worthington, R. L., & Soth, A. M. (2006). The Missouri multicultural counseling self-efficacy scale: A preliminary investigation. Paper presentation at the Twenty-third Annual Teachers College Winter Roundtable on Cross-Cultural Psychology and Education, New York, NY.Search in Google Scholar

Mueller, J. (2010). Authentic assessment toolbox. Retrieved from http://jonathan.mueller.faculty.noctrl.edu/toolbox/index.htmSearch in Google Scholar

Odgers, T., & Giroux, I. (2006). Internationalizing faculty: A phased approach to transforming curriculum design and instruction. Paper presented at Internationalizing Canada’s Universities: Practices, Challenges, and Opportunities symposium. Retrieved from http://www.viu.ca/internationalization/staff/internationalizingcurriculum.aspSearch in Google Scholar

Olebe, M., & Koester, J. (1989). Exploring the cross-cultural equivalence of the behavioral assessment scale for intercultural communication. International Journal of Intercultural Relations, 13(3), 333347. doi:10.1016/0147-1767(89)90016-3Search in Google Scholar

Olson, C., Green, M., & Hill, B. (2005). Building a strategic framework for comprehensive internalization. Washington, DC: American Council on Education.Search in Google Scholar

Olson, C., Green, M., & Hill, B. (2006). Advancing comprehensive internationalization: What institutions can do and what students should learn.Washington, DC: American Council on Education.Search in Google Scholar

Olson, C., & Kroeger, K. (2001). Global competency and intercultural sensitivity. Journal of Studies in International Education, 5(2), 116137. doi:10.1177/102831530152003Search in Google Scholar

Paschall, M., Fishbein, D., Hubal, R., & Eldreth, D. (2004). Psychometric properties of virtual reality vignette performance: A novel approach for assessing adolescents’ social competency skills. Health Education Research, 20(1), 6170. doi:10.1093/her/cyg103Search in Google Scholar

Platt, J. J. & Laszloffy, T. A. (2013). Critical patriotism: Incorporating nationality into MFT education and training. Journal of Marital and Family Therapy, 39(4), 441456.10.1111/j.1752-0606.2012.00325.xSearch in Google Scholar

Ponterotto, J., Gretchen, D., Utsey, S., Rieger, B., & Austin, R. (2002). A revision of the multicultural counseling awareness scale. Journal of Multicultural Counseling and Development, 30, 1120. doi:10.1002/j.2161-1912.2002.tb00489.xSearch in Google Scholar

Ponterotto, J. G., Rieger, B. P., Barrett, A., & Sparks, R. (1994). Assessing multicultural counseling competence: A review of instrumentation. Journal of Counseling and Development, 72(3), 316322. doi:10.1002/j.1556-6676.1994.tb00941.xSearch in Google Scholar

Pope-Davis, D. B., Coleman, H. L. K., Liu, W. M. & Toporek, R. L. (2003). Handbook of multicultural competencies in counseling and psychology. Thousand Oaks, CA: Sage.Search in Google Scholar

Queener, J., & Smith, S. (2008). Managing resistance: The difference between success and failure of multicultural competency initiatives. In J.White & S. J.Henderson (Eds.), Building multicultural competency: Development, training, and practice (pp.143154)). Lanham, MD: Rowman & Littlefield Publishers.Search in Google Scholar

Rankin, S., & Reason, R. (2008). Transformational tapestry model: A comprehensive approach to transforming campus climate. Journal of Diversity in Higher Education, 1(4), 262274. doi:10.1037/a0014018Search in Google Scholar

Roberson, L., Kulik, C. T., & Pepper, M. B. (2002). Assessing instructor cultural competence in the classroom: An instrument and a development process. Journal of Management Education, 26, 4055. doi:10.1177/105256290202600104Search in Google Scholar

Ruben, B. (1976). Assessing communication competency for intercultural adaptation. Group and Organization Studies, 1, 334354. doi:10.1177/105960117600100308Search in Google Scholar

Ruben, B., & KealeyD. (1979). Behavioral assessment of communication competency and the prediction of cross-cultural adaptation. International Journal of Intercultural Relations, 3, 1548. doi:10.1016/0147–1767(79)90045-2Search in Google Scholar

Schoenberg, N., & Ravdal, H. (2000). Using vignettes in awareness and attitudinal research. International Journal of Social Research Methodology, 3(1), 6374. doi:10.1080/136455700294932Search in Google Scholar

SheuH. B., Rigali-OilerM., & Lent, R. W. (2012). Multicultural counseling self-efficacy scale – racial diversity form: Factor structure and test of a social cognitive model. Psychotherapy Research, 22(5), 527542. doi:10.1080/10503307.2012.683344Search in Google Scholar

Sinicrope, C., Norris, J., & Watanabe, Y. (2008). Understanding and assessing competence: A summary of research, theory, and practice (Technical report for the Foreign Language Evaluation Project). Unpublished manuscript, University of Hawaii at Manoa, Hawaii. Retrieved from http://www.hawaii.edu/sls/uhwpesl/26(1)/Norris.pdfSearch in Google Scholar

Smith, L., Constantine, M., Graham, S., & Dize, C. (2008). The territory ahead for multicultural competence: The “spinning” of racism. Professional Psychology: Research and Practice, 39(3), 337335. doi:10.1037/0735–7028.39.3.337Search in Google Scholar

Sodowsky, G., Taffe, R., Gutkin, T., & Wise, S. (1994). Development of the multicultural counseling inventory: A self-report measure of multicultural competencies. Journal of Counseling Psychology, 41(2), 137148. doi:10.1037/0022-0167.41.2.137Search in Google Scholar

Spanierman, L. B., Oh, E., Heppner, P. P., Neville, H. A., Mobley, M., Wright, C. V., & Navarro, R. (2011). The multicultural teaching competency scale: Development and initial validation. Urban Education, 46(3), 440464. doi:10.1177/0042085910377442Search in Google Scholar

Sue, D. W., Arredondo, P., & McDavis, R. (1992). Multicultural counseling competencies and standards: A call to the profession. Journal of Counseling and Development, 70, 477486. doi:10.1002/j.2161-1912.1992.tb00563.xSearch in Google Scholar

Sue, D. W., Bernier, Y., Durran, A., Feinberg, L., Pedersen, P., Smith, E., & Vasquesz-Nuttal, E. (1982). Position paper: Cross cultural counseling competencies. The Counseling Psychologist, 10, 4552. doi:10.1177/0011000082102008Search in Google Scholar

Sue, D. W., Capodilupo, C., Torino, G., Bucceri, J., Holder, A., Nadal, K., & Esquilin, M. (2007). Racial microaggressions in everyday life: Implications for clinical practice. American Psychologist, 62(4), 271286. doi:10.1037/0003-066X.62.4.271Search in Google Scholar

Suskie, L. (2004). Assessing student learning: A common sense guide. Boston, MA: Anker Publishing Company.Search in Google Scholar

West, P. (1982). Reproducing naturally occurring stories: Vignettes in survey research. Working Paper. Aberdeen: MRC Medical Sociology Unit. Retrieved from http://www.sphsu.mrc.ac.uk/library/other%20reports/WP-May-1982.pdfSearch in Google Scholar

White, J. & Henderson, S. J. (Eds.). (2008). Building multicultural competency: Development, training and practice. Lanham, MD: Rowman & Littlefield Publishers.Search in Google Scholar

Wilks, T. (2004). The use of vignettes in qualitative research into social work values. Qualitative Social Work, 3(1), 7887. doi:10.1177/1473325004041133Search in Google Scholar

Wise, L. (1993). Scoring rubrics for performance tasks: Lessons learned from job performance assessment in the military. Paper presented at the Annual Meeting of the National Council of Measurement in Education, Atlanta, GA. Retrieved from http://www.eric.ed.gov/ERICWebPortal/search/detailmini.jsp?_nfpb=true&_&ERICExtSearch_SearchValue_0=ED358114&ERICExtSearch_SearchType_0=no&accno=ED358114Search in Google Scholar

Worthington, R. L., Mobley, M., Franks, R. P., & Tan, J. A. (2000). Multicultural counseling competencies: Verbal content, counselor attributions, and social desirability. Journal of Counseling Psychology, 47, 460468. doi:10.1037/0022-0167.47.4.460Search in Google Scholar

Published Online: 2014-7-29
Published in Print: 2016-3-1

©2016 by De Gruyter

Downloaded on 23.9.2025 from https://www.degruyterbrill.com/document/doi/10.1515/mlt-2014-0002/html
Scroll to top button