Educational implications of assessing learning outcomes with multiple choice questions and short essay questions

Kwon, Hyo-Jin; Chae, Su Jin; Park, Joo Hyun; Hyo-Jin Kwon; Su Jin Chae; Joo Hyun Park

doi:10.3946/kjme.2023.266

Abstract

Purpose

This study investigates the characteristics of different item types to assess learning outcomes and explore the educational implications that can be obtained from the results of learning outcome assessments.

Methods

Forty-five second-year premedical students participated in this study. Multiple choice question (MCQ) and short essay question (SEQ) scores and pass rates for 10 learning outcomes were analyzed. Descriptive statistics and correlation analysis were used to analyze the data.

Results

The correlation analysis indicated that there was a significant correlation between SEQs and pass rate but there was no significant correlation between MCQs and pass rate. Some students with identical scores on the MCQs had different scores on the SEQs or on the learning outcomes.

Conclusion

This study showed that students’ achievement of learning outcomes can be assessed using various types of questions in outcome-based education.

Keywords: Educational assessment, Educational measurement, Examination, Learning, Medical student

Introduction

Traditionally, student assessments in medical schools have primarily relied on written exams, which focus on knowledge-based learning objectives that professors deem important [1]. These exams are mainly used as summative assessments to determine student progress and graduation rather than to provide feedback on the current state of their learning [2]. Thus, as an appropriate assessment method for this purpose, medical schools have been using multiple choice questions (MCQs) [3], which have become a powerful assessment tool because they ensure assessment reliability and fairness [4,5]. However, as MCQs are an assessment method that emphasizes recall and comprehension, essay-type questions were introduced as an alternative assessment approach for assessing higher-order cognitive skills [6,7].

In outcome-based education, which is a major issue in contemporary medical education, assessment can provide information about a student’s current performance level and feedback on students’ weaknesses [8]. However, few studies on outcome-based education have evaluated whether students achieved the intended learning outcomes and whether the assessment results were used to provide feedback and learning guidance.

The University of Ulsan College of Medicine implemented an outcome-based assessment for its new curriculum introduced in the 2022 academic year. The system’s primary objective is to evaluate whether students have successfully achieved each course’s desired learning outcomes and determine their pass or fail status accordingly. To be recognized as having completed the course, all learning outcomes must be successfully achieved, and those who have not must repeat the course. Consequently, there was a need for a comprehensive assessment of each course’s set learning outcomes, and the university opted to use both MCQs and short essay questions (SEQs) for assessment.

This study aims to explore the educational significance found in the assessment system of University of Ulsan College of Medicine, which implements an outcome-based assessment, and to discuss the education implications obtained from the results of the assessments using various question types.

Methods

1. Participants and educational conditions

Forty-five participants in their second-year premedical program at University of Ulsan College of Medicine who had enrolled in the course “Cells and Metabolism” volunteered for this study. University of Ulsan College of Medicine implemented a pass/fail outcome-based assessment system for the second-year premedical program in 2022. “Cells and Metabolism” is a course that integrates several basic medical fields, such as biochemistry and genetics, and includes 10 learning outcomes. The main contents of these outcomes are related to cell structure, cell function, gene expression, genome, properties and metabolism of carbohydrates, breakdown and synthesis of monosaccharides, electron transport system, properties and metabolism of lipids, metabolism of amino acids and nucleic acids, and integrative human metabolism. To assess the achievement of each learning outcome, there were 6–18 MCQs and 1 SEQ per outcome. The assessment was conducted using a computer-based test consisting of 107 questions, with 97 MCQs and 10 SEQs (Table 1). Based on the assessment results, students’ achievement of the 10 learning outcomes was graded as either pass or fail, following predefined criteria.

2. Data collection and analysis

The scores of the MCQs and SEQs were converted to t-scores and standardized to a mean of 50 and standard deviation of 10. We also calculated the achievement rate of each student’s learning outcome, which is the percentage of the 10 predetermined learning outcomes passed by the students. Descriptive statistics were used to obtain basic information on academic achievement, and correlation analyses were conducted to explore the relationship between the assessment methods. The data were analyzed using IBM SPSS Statistics ver. 27.0 for Windows (IBM Corp., Armonk, USA).

3. Ethics statement

This study was exempted by the Institutional Review Board at Asan Medical Center (exemption no., 2022-1527).

Results

1. Achievement results by learning outcomes

The results of the number of questions, MCQ scores, SEQ scores, and pass rates for the 10 learning outcomes are shown in Table 1. The MCQ scores ranged from 3.56 to 12.18, while the SEQ scores ranged from 1.87 to 4.11. On average, the pass rate for the learning outcomes was 0.74, indicating that students passed approximately seven out of 10 learning outcomes. The learning outcomes for “genome” (pass rate=0.87) and “properties and metabolism of carbohydrates” (pass rate=0.84) were achieved by a majority of students, but “electron transport system” (pass rate=0.47) and “cell function” (pass rate=0.64) were less successful than the other learning outcomes.

Table 2 presents the correlation analysis results of MCQ scores, SEQ scores, and pass rate. Correlation analysis revealed a slightly stronger correlation between MCQs and SEQs (r=0.631, p<0.01). Further, SEQs and the pass rate had a positive correlation (r=0.474, p<0.01), while the correlation between MCQs and the pass rate was not significant [9].

2. Comparison of pass rate and SEQ scores based on MCQ scores

The top and bottom 25% (11 students each), selected based on the MCQ score quartiles, were analyzed for their MCQ and SEQ scores and pass rates; the results are presented in Table 3. The top and bottom 25% groups had average pass rates of 86% and 50%, respectively. Among students with low MCQ scores, both L3 and L4 had a score of 34.7 points. However, the content and number of the learning outcomes they passed and their SEQ scores varied. It is noteworthy that L10 had lower SEQ scores compared to other similarly situated students.

The top 25% of the group included students with identical MCQ scores but different SEQ scores. For example, H1 and H2 obtained identical MCQ scores, while their SEQ scores differed by more than seven points. Further, the content and number of the learning outcomes they passed varied. While H7 had high MCQ scores, their SEQ scores were extremely low, resulting in a low pass rate of 10%.

Discussion

This study sought the educational implications of using MCQs and SEQs to assess learning outcomes in outcome-based education. The findings and their implications are discussed below.

First, we found that SEQs had a significant correlation with MCQs and pass rates in learning outcomes. These results are consistent with research indicating that SEQs assess higher-order cognitive domains compared with MCQs [6,7,10]. Thus, SEQs should be leveraged more extensively in assessing student achievement of learning outcomes. While MCQs have been commonly used for their objectivity, additional methods that can accurately assess student learning should be explored. Moreover, continuous training and evaluation of SEQ raters should be implemented to ensure assessment reliability and objectivity [11].

Second, certain students (e.g., L10, H7) tended to score lower on the SEQs compared to others. SEQs assess higher-order cognitive skills, such as problem-solving and abstract thinking [6,10]. Therefore, if two students have the same score on the MCQs, the student with the lower score on the SEQs may not have a lower level of higher-order cognitive skills compared with their counterparts. To address this, these students should be trained on how to approach SEQs or provided with feedback to support the development of their higher-order cognitive skills.

Third, the content and number of learning outcomes passed by students varied, even when they obtained the same SEQ scores (e.g., L3 and L4, and H1 and H2). Learning outcomes for a course represent the essential elements that students must fulfill in their learning process. By assessing which learning outcomes have been achieved and which have not, you can provide appropriate feedback on students’ learning progress. And we enable targeted interventions to improve their learning and ensure full achievement of the course learning outcomes [1,12]. Furthermore, it is crucial to acknowledge that students achieving the same score may exhibit disparate learning statuses. Therefore, it is imperative to employ assessment methods that comprehensively evaluate students across multiple dimensions.

This study’s limitation is that it focused only on a single course at University of Ulsan College of Medicine. However, the findings are valuable for demonstrating the feasibility of evaluating students’ achievement of learning outcomes using MCQs and SEQs within the context of outcome-based education. Further, the specific information about student achievement from these assessment methods can be useful for remediation. Future research should explore and develop diverse assessment methods capable of evaluating learning outcomes and positively impacting educational activities.

Acknowledgments

None.

Notes

Funding

This study was supported by the University of Ulsan (no., 2022-0696).

Conflicts of interest

No potential conflict of interest relevant to this article was reported.

Author contributions

Conception or design of the work: SJC, HJK, JHP; data collection, data analysis: HJK; Interpretation, drafting the article: HJK, SJC; critical revision of the article: SJC, JHP; and final approval of the version to be published: SJC.

Table 1.

Content and Assessment Results for Each Learning Outcome

Content of learning outcomes	No. of MCQs	No. of SEQs	MCQ scores mean±SD	SEQ scores mean±SD	Pass rate
1. Cell structure	8	1	6.29±1.24	3.47±1.34	0.76
2. Cell function	12	1	6.09±2.67	2.09±1.64	0.64
3 Gene expression	18	1	12.18±2.47	3.04±1.55	0.78
4. Genome	6	1	3.56±1.54	4.11±1.16	0.87
5. Properties and metabolism of carbohydrates	6	1	3.84±1.08	4.02±1.47	0.84
6. Breakdown and synthesis of monosaccharides	12	1	7.92±2.02	2.27±1.51	0.71
7. Electron transport system	6	1	3.96±1.48	2.27±2.35	0.47
8. Properties and metabolism of lipids	12	1	8.67±2.03	3.09±1.79	0.78
9. Metabolism of amino acids and nucleic acids	9	1	5.12±2.01	2.58±1.79	0.73
10. Integrative human metabolism	8	1	4.36±1.45	1.87±1.69	0.78
Total	97	10	61.98±11.45	28.80±9.63	0.74

Data are presented as number, mean±standard deviation, or %.

MCQs: Multiple choice questions, SEQs: Short essay questions.

Table 2.

Correlations for MCQ Scores, SEQ Scores, and Pass Rate

	MCQ scores	SEQ scores	Pass rate
MCQ scores	1	0.631^**	0.112
SEQ scores		1	0.474^**
Pass rate			1

MCQ: Multiple choice question, SEQ: Short essay question.

^** p<0.01.

Table 3.

Learning Outcomes of Top and Bottom 25% of the Group Based on MCQ Scores

Achievement	Students	MCQ scores	SEQ scores	Pass (○)/fail (X) for the 10 learning outcomes										Pass rate (%)
Achievement	Students	MCQ scores	SEQ scores	1	2	3	4	5	6	7	8	9	10	Pass rate (%)
Low	L1	42	11	X	X	○	○	○	X	X	X	○	○	50
	L2	42.5	20	X	X	X	○	○	○	X	X	○	○	50
	L3	43.5	20	X	X	○	○	○	X	X	X	○	○	50
	L4	43.5	11	○	X	○	X	○	X	X	X	X	○	40
	L5	44	26	X	○	X	○	X	X	X	X	○	○	40
	L6	45	18	○	X	X	○	○	X	X	X	○	○	50
	L7	48	17	X	○	○	○	○	X	X	X	X	○	50
	L8	48.5	25	○	○	○	○	X	○	X	○	X	X	60
	L9	50.5	27	○	X	X	○	X	X	○	○	○	○	60
	L10	51	16	○	X	○	○	○	X	X	X	X	○	50
	L11	52	21	○	X	○	○	○	○	X	X	X	X	50
High	H1	71	32	X	X	○	○	○	○	X	○	○	○	70
	H2	71	25	○	○	○	○	○	○	X	○	○	○	90
	H3	72.5	42	○	○	○	○	○	○	○	○	○	○	100
	H4	73	35	○	○	○	○	○	○	○	○	○	X	90
	H5	74.5	28	○	○	○	○	○	X	○	○	○	○	90
	H6	75.5	3	○	○	○	○	○	○	○	○	○	○	100
	H7	75.5	42	X	X	X	X	X	X	○	X	X	X	10
	H8	78	41	○	○	○	○	○	○	○	○	○	○	100
	H9	82	47	○	○	○	○	○	○	○	○	○	○	100
	H10	84	43	○	○	○	○	○	○	○	○	○	○	100
	H11	86	43	○	○	○	○	○	○	○	○	○	○	100
Mean		61.98	28.8											86.36

MCQ: Multiple choice question, SEQ: Short essay question.

References

1. Im SJ. Assessment in outcome-based education. Korean Med Educ Rev 2013;15(1):25-30.

2. Carraccio C, Wolfsthal SD, Englander R, Ferentz K, Martin C. Shifting paradigms: from Flexner to competencies. Acad Med 2002;77(5):361-367.

3. Epstein RM. Assessment in medical education. N Engl J Med 2007;356(4):387-396.

4. Hift RJ. Should essays and other “open-ended”-type questions retain a place in written summative assessment in clinical medicine? BMC Med Educ 2014;14:249.

5. Herrero JI, Lucena F, Quiroga J. Randomized study showing the benefit of medical study writing multiple choice questions on their learning. BMC Med Educ 2019;19(1):42.

6. Palmer EJ, Devitt PG. Assessment of higher order cognitive skills in undergraduate education: modified essay or multiple choice questions?: research paper. BMC Med Educ 2007;7:49.

7. Schuwirth LW, van der Vleuten CP. Different written assessment methods: what can be said about their strengths and weaknesses? Med Educ 2004;38(9):974-979.

8. Yu T. Analysis of the US competency-based education policies and implications for competency-based education in Korea. J Educ Technol 2019;35(2):289-311.

9. Gogtay NJ, Thatte UM. Principles of correlation analysis. J Assoc Physicians India 2017;65(3):78-81.

10. Khan MU, Aljarallah BM. Evaluation of Modified Essay Questions (MEQ) and Multiple Choice Questions (MCQ) as a tool for assessing the cognitive skills of undergraduate medical students. Int J Health Sci (Qassim) 2011;5(1):39-43.

11. Shumway JM, Harden RM; Association for Medical Education in Europe. AMEE guide no. 25: the assessment of learning outcomes for the competent and reflective physician. Med Teach 2003;25(6):569-584.

12. Seong TJ, Lee B, Ahn S, Gwak YL. Effects of computerized formative assessment with the cognitive diagnosis model on achievement and affective attitude for math. J Educ Eval 2019;32(1):1-25.