Abstract
The purpose of this research was to develop a new tool for assessing multicultural and international competency in faculty teaching through vignette scenarios of university classroom critical incidents – across disciplines of clinical and forensics psychology, business, and education. Construct and content validity of the initial draft vignettes and the associated scoring rubric was established by a panel of experts who approved the vignettes as consistent with current teaching models and multicultural and international competency models. Each vignette presented a critical incident needing resolution – a rupture in the relationship between the instructor and students vis-à-vis a diversity issue. These hypothetical ruptures took place in university classrooms in four different disciplines: clinical and forensic psychology, business, and education. The authors recruited ten faculty participants for the online study, who were personally known to the authors as interested in competency-related issues and who were teaching in graduate programs. Each participant responded to three questions for each vignette, which prompted their written feedback on (a) process and content issues surrounding the critical incident as well as (b) coaching for the hypothetical instructor on how to resolve the incident. Five scorers rated the participants’ written responses according to a five-point scoring rubric. In the data analysis, once statistical conclusion validity and interscorer reliability was established (Krippendorff’s α = 0.887), content validity was again explored by assessing whether the written responses were indeed captured by the guiding models. The results, which indicated preliminary validity and reliability of the vignettes and the rubric, suggest that the vignettes are worthy of further testing. These vignettes can also be used as discussion tools to stimulate dialogue in competency trainings for faculty.
References
Angelides, P., & Gibbs, P. (2004). Supporting the continued professional development of teachers through the use of vignettes. Teacher Education Quarterly, Fall, 111–121. Retrieved from http://www.teqjournal.org/backvols/2006/33_4/18angelides&gibbs.pdfSearch in Google Scholar
Banks, J. A., McGee Banks, C. A., Cortez, C. E., Hahn, C. L., Merryfield, M. W., Moodley, K. A., … Parker, W. C. (2005). Democracy and diversity: Principles and concepts for educating citizens in a global age. Center for Multicultural Education, College of Education University of Washington, Seattle. Retrieved from http://education.washington.edu/cme/DemDiv.pdfSearch in Google Scholar
Barter, C., & Renold, E. (2000). “I wanna tell you a story”: Exploring the application of vignettes in qualitative research with children and young people. International Journal of Social Research Methodology, 3(4), 307–323. doi:10.1080/13645570050178594Search in Google Scholar
Bennett, J. (1993). Toward ethnorelativism: A developmental model of intercultural sensitivity. In R. M.Paige (Ed.), Education for the intercultural experience (pp. 21–71). Yarmouth, ME: Intercultural Press.Search in Google Scholar
Bhawuk, D., & Brislin, R. (1992). The measurement of intercultural sensitivity using the concepts of individualism and collectivism. International Journal of Intercultural Relations, 16, 413–436. doi:10.1016/0147–1767(92)90031-O.Search in Google Scholar
Borrayo, E. (2008). Challenging dialogues: Working through resistance toward multicultural understanding. In J.White & S.Henderson (Eds.), Building multicultural competency: Development, training, and practice (pp.155–166). Lanham, MD: Rowman & Littlefield Publishers.Search in Google Scholar
Burgoyne, S., Placier, P., Thomas, M., Welch, S., RuffinC., Flores, L. Y., … Miller, M. (2007). Interactive theater and self-efficacy. New Directions for Teaching and Learning, 111, 21–26. doi:10.1002/tl.282.Search in Google Scholar
Cabrera, A. F., Colbeck, C. L., & Terenzini, P. T. (2001). Developing performance indicators for assessing classroom teaching practices and student learning. Research in Higher Education, 42(3), 327–352. doi:10.1023/A:1018874023323Search in Google Scholar
Chapman, V. G., & Inman, M. (2009). A Conundrum: Rubrics or creativity/metacognitive development?Education Digest: Essential Readings Condensed For Quick Review, 75(2), 53–56.Search in Google Scholar
Constantine, M. G., Gloria, A. M., & Ladany, N. (2002). The factor structure underlying three self-report multicultural counseling competence. Cultural Diversity and Ethnic Minority Psychology, 8(4), 334–345. doi: 10.1037/1099-9809.8.4.335Search in Google Scholar
Constantine, M., & Ladany, N. (2000). Self-report multicultural counseling competency scales: Their relation to social desirability attitudes and multicultural case conceptualization ability. Journal of Counseling Psychology, 47, 155–164.10.1037/0022-0167.47.2.155Search in Google Scholar
Cooper, B., & Gargan, A. (2009). Rubrics in education: Old term, new meanings. Phi Delta Kappan, 91(1), 54–55.10.1177/003172170909100109Search in Google Scholar
Cronbach, L. J., & Meehl, R. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281–302.10.1037/h0040957Search in Google Scholar
D’Andrea, M., Daniels, J., & Heck, R. (1991). Evaluating the impact of multicultural counseling training. Journal of Counseling & Development, 70, 143–150. doi:10.1002/j.1556-6676.1991.tb01576.xSearch in Google Scholar
Dovidio, J., & Gaertner, S. (1986). Prejudice, discrimination, and racism. Orlando, FL: Academic Press.Search in Google Scholar
Fantini, A. (2007). Exploring and assessing intercultural competence. Retrieved from http://csd.wustl.edu/Publications/Documents/RP07-01.pdfSearch in Google Scholar
Finch, J. (1987). The vignette technique in survey research. Sociology, 21(1), 105–114. doi:10.1177/0038038587021001008Search in Google Scholar
Hammer, M., Bennett, M., & Wiseman, R. (2003). Measuring intercultural sensitivity: The intercultural development inventory. International Journal of Intercultural Relations, 27, 421–443. doi:10.1016/S0147-1767(03)00032-4Search in Google Scholar
Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1, 77–89. doi:10.1080/19312450709336664Search in Google Scholar
Hays, D. (2008). Assessing Multicultural competence in counselor trainees: A review of instrumentation and future directions. Journal of Counseling & Development, 86, 95–101. doi:10.1002/j.1556-6678.2008.tb00630.xSearch in Google Scholar
Isaeva, T. E. (2005). Classification of the college instructor’s professional and personalcompetencies. Russian Education and Society, 49(9), 31–41.10.2753/RES1060-9393490904Search in Google Scholar
Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73.10.1111/jedm.12000Search in Google Scholar
Kelley, C., & Meyers, J. (1995). Cross-cultural adaptability index. Minneapolis, MN: National Computer Systems.Search in Google Scholar
Kitano, M. (1996). What a course will look like after multicultural change. In A.Morey & M.Kitano (Eds.), Multicultural course transformation in higher education: A broader truth. Boston, MA: Allyn & Bacon.Search in Google Scholar
Krathwohl, D. (2002). A revision of Bloom’s taxonomy. Theory into Practice, 41(4), 212–218. doi:10.1207/s15430421tip4104_2Search in Google Scholar
LaFromboise, T., Coleman, H., & Hernandez, A. (1991). Development and factor structure of the cross-cultural counseling inventory-revised. Professional Psychology: Research and Practice, 22, 380–388. doi:10.1037/0735-7028.22.5.380Search in Google Scholar
Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2004). Practical resources for assessing and reporting intercoder reliability in content analysis research projects. Retrieved from: http://ils.indiana.edu/faculty/hrosenba/www/Research/methods/lombard_reliability.pdfSearch in Google Scholar
Miles, M. (1990). New methods for qualitative data collection and analysis: Vignettes and prestructured cases. International Journal of Qualitative Studies in Education, 3(1), 37–51. doi:10.1080/0951839900030104Search in Google Scholar
Mobley, M., Worthington, R. L., & Soth, A. M. (2006). The Missouri multicultural counseling self-efficacy scale: A preliminary investigation. Paper presentation at the Twenty-third Annual Teachers College Winter Roundtable on Cross-Cultural Psychology and Education, New York, NY.Search in Google Scholar
Mueller, J. (2010). Authentic assessment toolbox. Retrieved from http://jonathan.mueller.faculty.noctrl.edu/toolbox/index.htmSearch in Google Scholar
Odgers, T., & Giroux, I. (2006). Internationalizing faculty: A phased approach to transforming curriculum design and instruction. Paper presented at Internationalizing Canada’s Universities: Practices, Challenges, and Opportunities symposium. Retrieved from http://www.viu.ca/internationalization/staff/internationalizingcurriculum.aspSearch in Google Scholar
Olebe, M., & Koester, J. (1989). Exploring the cross-cultural equivalence of the behavioral assessment scale for intercultural communication. International Journal of Intercultural Relations, 13(3), 333–347. doi:10.1016/0147-1767(89)90016-3Search in Google Scholar
Olson, C., Green, M., & Hill, B. (2005). Building a strategic framework for comprehensive internalization. Washington, DC: American Council on Education.Search in Google Scholar
Olson, C., Green, M., & Hill, B. (2006). Advancing comprehensive internationalization: What institutions can do and what students should learn.Washington, DC: American Council on Education.Search in Google Scholar
Olson, C., & Kroeger, K. (2001). Global competency and intercultural sensitivity. Journal of Studies in International Education, 5(2), 116–137. doi:10.1177/102831530152003Search in Google Scholar
Paschall, M., Fishbein, D., Hubal, R., & Eldreth, D. (2004). Psychometric properties of virtual reality vignette performance: A novel approach for assessing adolescents’ social competency skills. Health Education Research, 20(1), 61–70. doi:10.1093/her/cyg103Search in Google Scholar
Platt, J. J. & Laszloffy, T. A. (2013). Critical patriotism: Incorporating nationality into MFT education and training. Journal of Marital and Family Therapy, 39(4), 441–456.10.1111/j.1752-0606.2012.00325.xSearch in Google Scholar
Ponterotto, J., Gretchen, D., Utsey, S., Rieger, B., & Austin, R. (2002). A revision of the multicultural counseling awareness scale. Journal of Multicultural Counseling and Development, 30, 11–20. doi:10.1002/j.2161-1912.2002.tb00489.xSearch in Google Scholar
Ponterotto, J. G., Rieger, B. P., Barrett, A., & Sparks, R. (1994). Assessing multicultural counseling competence: A review of instrumentation. Journal of Counseling and Development, 72(3), 316–322. doi:10.1002/j.1556-6676.1994.tb00941.xSearch in Google Scholar
Pope-Davis, D. B., Coleman, H. L. K., Liu, W. M. & Toporek, R. L. (2003). Handbook of multicultural competencies in counseling and psychology. Thousand Oaks, CA: Sage.Search in Google Scholar
Queener, J., & Smith, S. (2008). Managing resistance: The difference between success and failure of multicultural competency initiatives. In J.White & S. J.Henderson (Eds.), Building multicultural competency: Development, training, and practice (pp.143–154)). Lanham, MD: Rowman & Littlefield Publishers.Search in Google Scholar
Rankin, S., & Reason, R. (2008). Transformational tapestry model: A comprehensive approach to transforming campus climate. Journal of Diversity in Higher Education, 1(4), 262–274. doi:10.1037/a0014018Search in Google Scholar
Roberson, L., Kulik, C. T., & Pepper, M. B. (2002). Assessing instructor cultural competence in the classroom: An instrument and a development process. Journal of Management Education, 26, 40–55. doi:10.1177/105256290202600104Search in Google Scholar
Ruben, B. (1976). Assessing communication competency for intercultural adaptation. Group and Organization Studies, 1, 334–354. doi:10.1177/105960117600100308Search in Google Scholar
Ruben, B., & KealeyD. (1979). Behavioral assessment of communication competency and the prediction of cross-cultural adaptation. International Journal of Intercultural Relations, 3, 15–48. doi:10.1016/0147–1767(79)90045-2Search in Google Scholar
Schoenberg, N., & Ravdal, H. (2000). Using vignettes in awareness and attitudinal research. International Journal of Social Research Methodology, 3(1), 63–74. doi:10.1080/136455700294932Search in Google Scholar
SheuH. B., Rigali-OilerM., & Lent, R. W. (2012). Multicultural counseling self-efficacy scale – racial diversity form: Factor structure and test of a social cognitive model. Psychotherapy Research, 22(5), 527–542. doi:10.1080/10503307.2012.683344Search in Google Scholar
Sinicrope, C., Norris, J., & Watanabe, Y. (2008). Understanding and assessing competence: A summary of research, theory, and practice (Technical report for the Foreign Language Evaluation Project). Unpublished manuscript, University of Hawaii at Manoa, Hawaii. Retrieved from http://www.hawaii.edu/sls/uhwpesl/26(1)/Norris.pdfSearch in Google Scholar
Smith, L., Constantine, M., Graham, S., & Dize, C. (2008). The territory ahead for multicultural competence: The “spinning” of racism. Professional Psychology: Research and Practice, 39(3), 337–335. doi:10.1037/0735–7028.39.3.337Search in Google Scholar
Sodowsky, G., Taffe, R., Gutkin, T., & Wise, S. (1994). Development of the multicultural counseling inventory: A self-report measure of multicultural competencies. Journal of Counseling Psychology, 41(2), 137–148. doi:10.1037/0022-0167.41.2.137Search in Google Scholar
Spanierman, L. B., Oh, E., Heppner, P. P., Neville, H. A., Mobley, M., Wright, C. V., & Navarro, R. (2011). The multicultural teaching competency scale: Development and initial validation. Urban Education, 46(3), 440–464. doi:10.1177/0042085910377442Search in Google Scholar
Sue, D. W., Arredondo, P., & McDavis, R. (1992). Multicultural counseling competencies and standards: A call to the profession. Journal of Counseling and Development, 70, 477–486. doi:10.1002/j.2161-1912.1992.tb00563.xSearch in Google Scholar
Sue, D. W., Bernier, Y., Durran, A., Feinberg, L., Pedersen, P., Smith, E., & Vasquesz-Nuttal, E. (1982). Position paper: Cross cultural counseling competencies. The Counseling Psychologist, 10, 45–52. doi:10.1177/0011000082102008Search in Google Scholar
Sue, D. W., Capodilupo, C., Torino, G., Bucceri, J., Holder, A., Nadal, K., & Esquilin, M. (2007). Racial microaggressions in everyday life: Implications for clinical practice. American Psychologist, 62(4), 271–286. doi:10.1037/0003-066X.62.4.271Search in Google Scholar
Suskie, L. (2004). Assessing student learning: A common sense guide. Boston, MA: Anker Publishing Company.Search in Google Scholar
West, P. (1982). Reproducing naturally occurring stories: Vignettes in survey research. Working Paper. Aberdeen: MRC Medical Sociology Unit. Retrieved from http://www.sphsu.mrc.ac.uk/library/other%20reports/WP-May-1982.pdfSearch in Google Scholar
White, J. & Henderson, S. J. (Eds.). (2008). Building multicultural competency: Development, training and practice. Lanham, MD: Rowman & Littlefield Publishers.Search in Google Scholar
Wilks, T. (2004). The use of vignettes in qualitative research into social work values. Qualitative Social Work, 3(1), 78–87. doi:10.1177/1473325004041133Search in Google Scholar
Wise, L. (1993). Scoring rubrics for performance tasks: Lessons learned from job performance assessment in the military. Paper presented at the Annual Meeting of the National Council of Measurement in Education, Atlanta, GA. Retrieved from http://www.eric.ed.gov/ERICWebPortal/search/detailmini.jsp?_nfpb=true&_&ERICExtSearch_SearchValue_0=ED358114&ERICExtSearch_SearchType_0=no&accno=ED358114Search in Google Scholar
Worthington, R. L., Mobley, M., Franks, R. P., & Tan, J. A. (2000). Multicultural counseling competencies: Verbal content, counselor attributions, and social desirability. Journal of Counseling Psychology, 47, 460–468. doi:10.1037/0022-0167.47.4.460Search in Google Scholar
©2016 by De Gruyter
Articles in the same Issue
- Frontmatter
- Executive Editors’ Comment
- Editors’ Comments: Lighting Multicultural Candles through Lifelong Learning
- Ethnic Identity, Multiculturalism, and Their Interrelationships: Differences between Jewish and Arab Students
- Immigration Reform and Education: Demystifying Mythologies about Latina/o Students
- Validation of Assessment Vignettes and Scoring Rubric of Multicultural and International Competency in Faculty Teaching
- Understanding Sources of Self-Efficacy of Chinese Students Learning English in an American Institution
- Culture, Leadership, and Activism: Translating Fink’s Taxonomy of Significant Learning into Pedagogical Practice
- Service Learning for At-Risk Student Populations: The Contextual Dynamism of Implementation
Articles in the same Issue
- Frontmatter
- Executive Editors’ Comment
- Editors’ Comments: Lighting Multicultural Candles through Lifelong Learning
- Ethnic Identity, Multiculturalism, and Their Interrelationships: Differences between Jewish and Arab Students
- Immigration Reform and Education: Demystifying Mythologies about Latina/o Students
- Validation of Assessment Vignettes and Scoring Rubric of Multicultural and International Competency in Faculty Teaching
- Understanding Sources of Self-Efficacy of Chinese Students Learning English in an American Institution
- Culture, Leadership, and Activism: Translating Fink’s Taxonomy of Significant Learning into Pedagogical Practice
- Service Learning for At-Risk Student Populations: The Contextual Dynamism of Implementation