The development of an evaluation tool to assess professional behavior and clinical competencies from the graduates’ perspective

Kusmiati, Mia; Bahari, Rafidah; Sanip, Suhaila; Hamid, Noor Aini Abdul; Emilia, Ova; Mia Kusmiati; Rafidah Bahari; Suhaila Sanip; Noor Aini Abdul Hamid; Ova Emilia

doi:10.3946/kjme.2020.148

Korean J Med Educ > Volume 32(1); 2020 > Article

Kusmiati, Bahari, Sanip, Hamid, and Emilia: The development of an evaluation tool to assess professional behavior and clinical competencies from the graduates’ perspective

Original Research

Korean J Med Educ 2020; 32(1): 1-11.

Published online: March 1, 2020

DOI: https://doi.org/10.3946/kjme.2020.148

The development of an evaluation tool to assess professional behavior and clinical competencies from the graduates’ perspective

Mia Kusmiati^1,²

, Rafidah Bahari²

, Suhaila Sanip³

, Noor Aini Abdul Hamid⁴

, Ova Emilia⁵

¹Department of Medical Education, Faculty of Medicine, Universitas Islam Bandung, Bandung, Indonesia

²Department of Psychiatry, Faculty of Medicine, University of Cyberjaya, Cyberjaya, Malaysia

³Department of Medical Education, Faculty of Medicine & Health Sciences, University Sains Islam Malaysia, Kualalumpur, Malaysia

⁴Department of Medical Education, International Medical School, Management & Science University, Shah Alam, Malaysia

⁵Department of Medical Education and Bioethic, Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia

Corresponding Author: Mia Kusmiati (https://orcid.org/0000-0003-2858-4960) School of Medicine, Bandung Islamic University, Tamansari Street No. 22, Bandung-40116, Indonesia Tel: +62.224203368 (ext. 6905) Fax: +62.224231213 email: dr.mia74@gmail.com

Received July 11, 2019 Revised August 27, 2019 Accepted December 27, 2019

This is an open-access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Purpose

This study was designed to develop an evaluation tool for assessing professional behavior and clinical competencies from the graduates’ perspective.

Methods

This study employed mixed method in a sequential exploratory design. The semi-structured interviews were conducted with three graduates from different cohorts. The qualitative analysis of the interviews found six emerging themes for professional behavior and clinical competencies development. These themes were then developed into a 55-item questionnaire. The questionnaire was then distributed to 84 medical graduates for exploratory factor analysis (EFA) from February to April 2019. The quantitative data were analyzed using IBM SPSS ver. 21.0 (IBM Corp., Armonk, USA) for principal axis factoring. After conducting EFA, we proceeded with confirmatory factor analysis (CFA) with another 120 graduates to validate the tool.

Results

Eighty-four graduates completed the questionnaire for EFA. Upon completion of EFA, 35 out of 55 items of the questionnaire were found to be valid and reliable. The most appropriate fit was seven factors, which explained 58.18% of variance between them after 15 iterations with Cronbach’s α of 0.916. The personal satisfaction factor was noted to be weak. It was therefore added to patient management factor due to its similar intention. The final EFA factor after the modification was six. The CFA found that 34 out of 35 items was valid and reliable that representation of the latent variables.

Conclusion

The questionnaire has achieved the desired construct validity score and can be used as an evaluation tool to assess professional behavior and clinical competencies from the graduates’ perspective.

Keywords: Professional competence, Clinical competence, Graduate, Evaluation study

Introduction

Medical professionalism is a normative system that organizes and delivers health care. It calls upon its group members to jointly declare (“profess”) what the public and individual patients expected, based on shared competency standards and ethical values. These standards and values ensure that all medical professional will deliver quality care to their patients [1]. Professionalism is understood to exist or develop in varying degrees as a characteristic or attribute that is identifiable within individuals. In other words, professionalism can be conceptualized and assessed at different levels: individual, interpersonal and institutional/societal [2].

The professional behavior and clinical competency of a doctor reflect a range of personal and interpersonal qualities, attributes, behaviors, commitments, and values [3-6]. Doctors are expected to be accountable, altruistic, committed to excellence, compassionate, respectful, responsive, sensitive to diversity, and ethically sound [3,6].

Many evaluation tools have been developed to assess a doctor’s professional behavior. For example, Professional Mini Evaluation Exercise assesses the medical residency’s professional behavior from the clinical supervisor’s perspective [7,8]. Our search of the literature, however, could not find a tool that was developed to assess professional behavior and clinical competencies development from a graduate perspective, although they are one of the stakeholders in medical education [9]. As self-assessment is an essential factor in enhancing and improving professional ability [10,11], graduates can provide valuable input to the faculty for improving a medical education program because they have undergone the curriculum and have the means to assess its effectiveness for their medical practice. Therefore, there is a need to develop a tool to assess professional behavior and clinical competencies development from the graduates’ perspectives.

We have published our work on graduates’ performance evaluation from the perspectives of the preceptors and patients [11,12]. In this article, we will elaborate on the graduates’ perspective on their professional behavior and clinical competencies development. The graduates’ perspective of professional behavior and clinical competencies is important in helping Universitas Islam Bandung to evaluate its educational outcome achievement as the graduates can further elaborate on the behaviors and competencies that were needed from their workplace-based experience.

This study aimed to develop a tool for assessing professional behavior and clinical competencies development from the graduates’ perspective.

Methods

The development of the questionnaire involved three aspects: (1) the study participants; (2) the factor analysis, with 20% of target population as the sample size [13] or between 10:1 to 20:1 subject to factor ratio [14,15]; and (3) the unstandardized parameter and their standard error of mean [16].

1. Study design

Using a mixed-method sequential exploratory design, a qualitative study preceded the tool (questionnaire) development [17]. It was followed by exploratory factor analysis (EFA). The results of the EFA guided questionnaire revision. The revised questionnaire was then distributed for confirmatory factor analysis (CFA). Fig. 1 summarizes the steps involved in the study.

2. Sampling strategy and study participants

A total of 207 graduates were involved in the study. The qualitative study used purposive sampling while EFA and CFA used random sampling strategies. The qualitative study involved three graduates for semistructured interviews. The EFA involved 84 graduates while the CFA involved another 120 graduates.

3. Study procedures

The qualitative study involved three medical graduates from batches 2006, 2007, and 2008. Upon receiving written consent, they were interviewed between 45 to 60 minutes per person. Examples of the questions asked to the graduates were as shown in Fig. 2. The interview questions were formulated based on the literature review of the topic and the research objectives. It was reviewed many times by the research team to ensure its clarity. The qualitative study preceded tool development because it was necessary to explore the graduates’ perception of their professional behavior and clinical competencies development first. All interview transcripts were transcribed verbatim. Thematic analysis of the transcripts found seven emerging themes. The themes were (1) professional behavior, (2) humanity, (3) patient management, (4) clinical skills competence, (5) personal satisfaction, (6) cognitive competence, and (7) interpersonal skill. The description of the theme is as listed in Table 1 [18,19].

The themes that emerged from the interviews were then developed into a 55-item questionnaire based on the seven steps recommended by Stenfors-Hayes et al. [16]. The seven steps were familiarization, condensation, comparison, grouping, articulating, labelling, and contrasting. The 55-items questionnaire consists of eight items for professionalism, 11 items for humanity, five items for patient management, 12 items for clinical skill competence, four items for personal satisfaction, six items for cognitive competence, and nine items for interpersonal skill. Table 2 illustrated the items for each theme and the source of references [1-3,5,20-23].

4. Ethical approval

The Health Research Ethics Committee, Faculty of Medicine, Bandung Islamic University has approved this observational study (approval letter no., 005/Ethic committee, FK/VI/2017). The informed consent was obtained from all individual participants included in the study.

Results

1. Exploratory factor analysis

The 55-item questionnaire was later distributed to 84 medical graduates using with a participant to factor ratio of 12:1 [15]. It was distributed for 3 months from February to April 2019. Prior to conducting EFA, Kaiser-Myer-Olkin (KMO) counting and Bartlett’s test of sphericity were performed to obtain sampling adequacy [24]. The KMO counting showed an adequate sample has been achieved if it has a value above 0.5 or 0.6. After obtaining an adequate sample size with a KMO value of 0.546 (>0.5) and Bartlett’s test of sphericity of p<0.001 (χ²=3,682.84), the extraction method was conducted to acquire and describe the data structure [13,14] using IBM SPSS ver. 21.0 (IBM Corp., Armonk, USA). Confirmation of the data extraction that was able to be loaded in the same factor or scale was done using principal axis factoring of varimax rotation with Kaiser normalization.

The varimax rotation method with Kaiser normalization on EFA extracted seven factors based on eigenvalue of more than one (Table 3, Fig. 3). These seven factors were retained initially in the questionnaire and accounted for 58.18% of the total variance. In Table 4, the matrix pattern of the varimax rotation method with Kaiser normalization showed the grouping of the items based on a loading value of >0.5. For example, question 11, 12, 13, 15, and 16 were categorized into factor 4 due to their loading values. Other items were classified in the same manner.

Thirty-five items were retained as latent/potential variables (loading value of >0.5 except item Q3 which had a value of 0.498). Q3 item was retained in the questionnaire because it provided the purpose to assess professional capacity. Twenty items were removed because their loading values were less than 0.5.

The calculation of internal consistency was conducted using Cronbach’s α coefficient [9,10] and interpreted according to the five categories recommended by Guilford and Fruchter [25] in 1978. The 35-item questionnaire had a very high internal consistency as evidenced by Cronbach’s α of 0.916 (Table 5) [14,26,27]. However, upon completion of EFA, one of the seven factors was noted to be weak and was added to another factor. Therefore, only 35 items within six factors were retained in the questionnaire, as demonstrated in Table 6. The list of questions for the revised questionnaire used in CFA is shown in Table 7 [28-31].

2. Confirmatory factor analysis

CFA is a part of structural equation modelling. Upon completion of EFA, CFA was then conducted on another 120 graduates to validate the construct of the evaluation tool. The sample size for CFA was based on participant to factor ratio of 20:1. The CFA was conducted using LISREL ver. 8.8 program for Windows (Scientific Software International Inc., Lincolnwood, USA). The CFA results were presented in Fig. 3. The t-loading value of all items in each group of factors was counted and was stated as “valid” if it has a value above 1.96 (at 95% confidence interval). The highest t-loading value was deemed as the most reliable indicator that represents its latent variable.

Fig. 3 showed that 34 out of 35 items (indicator variable) were valid and reliable in representing the six latent variables. Humanity variable was represented by item C2-C7. Item C7 was the strongest indicator with a t-value of 12.22. Cognitive competence variable was represented by item C8-C10. The strongest indicator was item C10 with t-value of 7.31. Clinical skills competence was represented by items C11 to C16 with item C11 being the strongest indicator with a t-value of 9.14. Professional behavior variable was represented by item C17 to C25. Item C20 was the strongest indicator with t-value of 8.62. Patient management variable was represented by item C26-C33. Item C29 was the strongest indicator with t-value of 9.88. Inter-personal skill variable was represented by item C34-C35. Item C34 was a stronger indicator with t-value of 8.98.

Discussion

David Kern stated that the process of curriculum development and evaluation requires an assessment of the stakeholders’ needs, in order to determine what aspects should be included in the curriculum [26]. This study was conducted to develop an evaluation tool for assessing the development of professional behavior and clinical competencies from the graduates’ perspective. The data collected from this tool will be used by UNISBA to evaluate its current medical curriculum.

The seven factors obtained from conducting EFA with 35-item questionnaire indicated high communality in which almost all items have the value of more than 0.7 [27]. The factor analysis yielded nine items for professional behavior, seven items for humanity, seven items for patient management, six items for clinical skill competence, one item for personal satisfaction, three items for cognitive competence, and two items for interpersonal skill.

Five of the seven factors identified in this study were strong factors because they have three or more items except for interpersonal skill factor and personal satisfaction factor. The one item for personal satisfaction was then inserted to patient management factor because it shared similar intention. The interpersonal skill factor remained as it is with two items. Therefore, the final extracted factors were six.

The factor which had the most items was factor 1 (professional behavior). All nine items in factor 1 were retained. The item with the highest loading value was item 34 (0.802) in factor 2 (humanity). Items included in patient management factor were aspects and attitudes that the graduates perceived they must have. These factors were extracted from the semi-structured interviews. As doctor-patient communication is one of the aspects known to increase patient’s adherence to medical treatment and enhanced clinical outcomes [11,32], its inclusion in the questionnaire was deemed necessary.

It was surprising that only two items were retained in the interpersonal skill factor [3], although this skill was important for medical practitioners [7]. A possible explanation could be due to the paternalistic way of managing patient in our country [33].

Most of the factors identified in this study were similar to the competency domains of the CanMEDS Physician Competency Framework [28,29] and the Accreditation Council for Graduate Medical Education [30] competencies except for humanity factor (Table 7) [28-31]. Humanity was chosen as one of the factors because it represented the vision and mission of UNISBA’s Medical School. Therefore, this factor might not be of interest to other institutions unless they have similar vision and mission.

1. Study limitation

Our study has several limitations. Firstly, the sample size of EFA was small due to a limited number of graduates available for the study. Second, the interview was conducted in Bahasa Indonesia. There could be a misinterpretation of meaning when the emerging themes were translated into English. Third, the original 55-item and 35-item questionnaires were distributed to the graduates in Bahasa Indonesia. There could also be a misinterpretation of meaning when the factors extracted from the literature was translated from English to Bahasa Indonesia for the questionnaire, and when it was translated again into English for publication.

2. Conclusion

Validity and reliability are essential psychometric properties of an evaluation tool. A high index of validity and reliability of an evaluation tool indicated that the tool was well constructed. The validity and reliability index obtained by this tool signified that this tool is valid and reliable in assessing the professional behavior and clinical competencies development from the graduates’ perspective. It also indicated that the evaluation tool can be valid and reliable to be used for collecting data as a source of information for curriculum evaluation exercise. Future work will investigate the similarities and differences of perspectives by all stakeholders in evaluating UNISBA’s medical program.

Acknowledgments

We would like to thank the Bandung Islamic University School of Medicine Alumni Association (Ikatan Alumni Medical School of UNISBA) for their support in the curriculum evaluation process. Our special gratitude goes to the Chief of Alumni Association, Dr. Surya Santosa MD.

Notes

Funding

None.

Conflicts of interest

No potential conflict of interest relevant to this article was reported.

Author contributions

Design of work, data collection, data analysis, and drafting the article: MK; final approval of the version to be published: RF, SS; and critical revision of the article: NA, OE.

Fig. 1.

Summary of the Steps Involved in the Study

Fig. 2.

Examples of the Questions for Semi-Structured Interview with the Graduates

UNISBA: Universitas Islam Bandung.

Fig. 3.

Result of Confirmatory Factor Analysis of Six Components on the Graduate's Instrument

(A) Item C1–C16. (B) Item C17–C35. hum: Humanity, cog: Cognitive competence, skl: Clinical skill competence, pro: Professional behavior, Pxm: Patient management, ips: Interpersonal skill, df: Degrees of freedom, RMSEA: Root mean square error of approximation.

Table 1.

The Description of the Themes

Theme	Description
Professional behavior	A set of quality and attributes of a person behavior that is observable and relates to medical practice [18,19].
Humanity	The qualities of being a human that relates to a physician role [19].
Patient management	The way physician act in managing a patient.
Clinical skills competence	Clinical skills deemed necessary by the graduates for medical practice.
Interpersonal skill	Skills thatcontribute to successful dealing in maintaining appropriate relationship with patient and other colleagues [18].
Cognitive competence	Knowledge required by the physician for medical practice.
Personal satisfaction	Any means that bringgratification, pleasure, or contentment to the graduates during medical practice.

Table 2.

Items Supporting Each Theme and the Source of References

Theme	Items	Source of references
Professional behavior	Adequate experience, doctor-patient communication, problem-solver, adequate knowledge, time management, analytical, adequate skill, adequate competence	Arnold [1] (2002)
Professional behavior		Epstein and Hundert [20] (2002)
Humanity	Comforting, humble, religious values, respect, religious, character, continually improving, lifelong learner, counselor, discipline, empathy, hardworking	Epstein and Hundert [20] (2002)
Humanity		Mueller [5] (2009)
Patient management	Wholeheartedness, psychological support, confidence, proper management, patient education	Arnold [1] (2002)
		Campbell et al. [21] (2007)
		Mueller [5] (2009)
Clinical skills competence	Active listener, subject expert, nasogastric tube insertion, intravenous cannulation, circumcision, performing minor surgery, handling trauma patient, injections, burns, poisoning & emergency, hospital admission, clinical decision	Epstein and Hundert [20] (2002)
Clinical skills competence		Hodges et al. [2] (2011)
Interpersonal skill	Subordinate communication, cooperation, peer communication, community perspective, personal instinct, opportunity for improvement, self-confidence, lifelong learning, humble	Hodges et al. [2] (2011)
		Haque et al. [22] (2016)
		Adam et al. [3] (2012)
Cognitive competency	Knowledge about disease, knowledge about treatment, referral, number of cases learnt, knowledge about management, cases variation	Epstein and Hundert [20] (2002)
Cognitive competency		Mueller [5] (2009)
Personal satisfaction	Personal satisfaction, patient perspective, job satisfaction, medical knowledge satisfaction	Epstein and Hundert [20] (2002)
Personal satisfaction		Bahaziq and Crosby [23] (2011)

Table 3.

Total Variance Explained Based on the Initial Eigenvalue >1

Component	Initial eigenvalue			Rotation sums of square loadings
Component	Total	% of variance	Cumulative %	Total	% of variance	Cumulative %
1	14.040	25.527	25.527	5.740	10.437	10.437
2	4.497	8.176	33.703	5.417	9.848	20.285
3	3.970	7.219	40.921	4.673	8.496	28.780
4	2.734	4.971	45.892	4.400	8.001	36.781
5	2.644	4.807	50.699	3.353	6.097	42.878
6	2.146	3.902	54.601	3.081	5.602	48.480
7	1.968	3.579	58.180	2.202	4.003	52.483

Table 4.

Results of Principle Axis Factoring with Orthogonal Varimax Rotation of 55 Items (n=84)

Item		Factor 1	Factor 2	Factor 3	Factor 4	Factor 5	Factor 6	Factor 7^a)
Q1.	Knowledge about diseases that I gained during my education supported my work in the field.	0.276	0.220	0.145	-0.001	-0.051	0.480	0.049
Q2.	Knowledge about kind and dose of remedy that I learned helped me towards medical practice.	0.092	0.113	0.197	0.120	0.176	0.477	-0.080
Q3.	Physician should make a referral when it is beyond their capacity.	0.221	0.498	0.054	-0.048	0.171	0.141	0.027
Q4.	The cases learned during clerkship are useful and applicable at workplace of medical practice.	0.212	0.643	0.060	0.255	0.096	0.288	-0.115
Q5.	Management of trauma I learned during medical education helps me in the field.	0.033	0.074	0.201	0.115	-0.096	0.646	0.128
Q6.	The variation of cases during the clerkship was sufficient.	0.178	0.214	-0.117	0.194	0.223	0.550	0.017
Q7.	To be a good doctor we should be able to be an active listener.	-0.029	0.223	-0.054	-0.021	0.373	0.226	-0.089
Q8.	Learning during the medical doctor program makes me master medical subjects optimally.	0.465	0.361	-0.041	0.338	-0.005	0.262	-0.146
Q9.	I have adequate skill in inserting nasogastric tube.	0.094	0.357	-0.018	0.363	0.067	0.200	0.080
Q10.	I am quite skilled in performing the insertion of the iv catheter.	0.048	-0.014	0.003	0.397	0.115	0.240	-0.015
Q11.	I am capable to perform circumcision.	0.059	-0.068	0.221	0.681	0.192	0.342	0.247
Q12.	I am skilled in performing minor surgery at workplace of medical practice.	0.168	0.234	0.267	0.634	-0.090	-0.165	0.222
Q13.	I am skilled in handling patient with trauma.	0.237	0.164	0.070	0.711	0.046	0.191	0.093
Q14.	I am skilled in injecting intra-muscular, intravenous, or sub-cutan.	0.194	0.422	0.030	0.465	0.124	0.208	0.013
Q15.	I can handle cases of burn injury.	0.279	0.045	0.036	0.656	0.186	-0.150	-0.028
Q16.	I can handle cases of poisoning and emergency.	0.521	-0.079	0.068	0.528	-0.017	-0.297	0.025
Q17.	I know well when patient needs to be hospitalized or not.	0.397	0.310	0.025	0.208	0.322	0.023	0.068
Q18.	I can make clinical decisions well.	0.544	0.225	0.182	0.305	0.305	-0.061	0.143
Q19.	I am pleased going through my medical profession.	0.627	0.285	0.177	0.055	0.389	-0.034	-0.045
Q20.	I feel my patients are quite satisfied with my performance.	0.092	-0.075	-0.036	0.591	0.564	0.008	0.008
Q21.	I am satisfied to be a medical doctor.	0.660	0.023	0.226	0.049	-0.007	0.126	0.053
Q22.	I am satisfied with my medical education.	0.350	-0.200	-0.060	0.189	0.117	0.128	-0.020
Q23.	Clerkship experience makes me competent as a medical doctor.	0.484	0.247	-0.028	0.095	0.043	0.066	0.141
Q24.	I can communicate well with patient and her/his family.	0.539	0.421	0.046	0.086	0.432	0.020	0.057
Q25.	My medical education makes me become a good problem-solver.	0.501	0.108	0.224	0.263	0.218	0.246	-0.069
Q26.	My medical knowledge is appropriate and applicable in medical practice.	0.579	0.109	0.225	0.257	0.047	0.203	-0.228
Q27.	My medical education made me manage my time properly.	0.661	0.048	0.249	0.166	0.044	0.102	0.311
Q28.	My medical education enables me to analyze every medical problem that I find.	0.621	0.020	0.001	0.025	-0.026	0.116	0.323
Q29.	Clinical experience during the clerkship has supported me to be a skilled doctor.	0.453	0.230	0.369	0.149	-0.147	0.169	-0.026
Q30.	My education is sufficient to achieve competence as a physician.	0.496	-0.029	0.374	0.087	-0.135	0.426	-0.012
Q31.	Making the patient feel comfortable is a compulsory attitude that must be had by a Muslim doctor.	-0.042	0.785	0.134	0.066	-0.031	-0.067	0.014
Q32.	In serving patients I prioritize the attitude of being humble & not conceited.	-0.064	0.567	0.522	0.068	0.112	-0.158	0.020
Q33.	I work on my medical profession in accordance to the values of Islam.	0.076	0.543	0.553	-0.100	-0.004	0.061	0.028
Q34.	I have shown politeness and respect to the patient and his family.	0.185	0.802	0.193	-0.001	0.034	-0.018	0.065
Q35.	Understanding about Islam that I learned during my medical education has built my character as a Muslim doctor.	0.042	0.414	0.653	-0.037	-0.267	0.101	0.167
Q36.	My improvisation ability increased when I am running the medical practice.	0.092	0.389	0.074	0.210	0.229	-0.226	0.475
Q37.	I am always updated with my medical knowledge and skill.	0.155	0.223	0.153	0.145	-0.034	0.180	0.697^b)
Q38.	I always give counseling, information, and education to the patients.	-0.096	0.474	0.494	0.002	0.084	0.166	0.232
Q39.	My education has established me to be a disciplined doctor.	0.216	-0.030	0.570	0.021	-0.096	0.275	0.475
Q40.	I always empathize toward my patients.	0.085	0.037	0.352	0.008	-0.032	-0.004	0.359
Q41.	As a Muslim doctor, I have to give my best effort for the optimal outcome of the patient and submit myself to God.	0.051	0.602	0.011	0.045	0.175	0.160	0.279
Q42.	I am considered as a good and skilled professional by other health workers at my workplace.	0.268	-0.161	0.206	0.370	0.455	0.143	0.357
Q43.	I am capable of cooperating in the team well.	0.049	0.144	0.247	0.261	0.760	0.022	-0.052
Q44.	I am capable of communicating with hospital staff and other health care professions.	0.388	0.213	0.079	0.074	0.551	0.070	0.039
Q45.	The community appraises me fairly well.	0.112	0.001	0.539	0.210	0.412	0.153	0.210
Q46.	My instinct as a doctor was trained during internship.	0.291	0.491	0.135	0.008	0.246	-0.005	0.313
Q47.	The examinations during medical education provide me opportunities for better performance.	0.217	-0.061	0.267	0.005	0.059	0.545	0.240
Q48.	I have a good self-confidence.	0.169	0.096	0.214	0.481	0.489	0.071	-0.001
Q49.	My medical education made me always self-aware about life-long learning.	0.244	0.133	0.501	0.230	0.121	0.258	0.013
Q50.	I do not appraise myself exclusively.	-0.053	0.429	0.144	0.290	0.152	0.179	-0.168
Q51.	I serve my patients wholeheartedly.	0.261	0.023	0.720	0.018	0.046	0.254	0.146
Q52.	I give support to the patient psychologically to encourage his recovery.	0.137	0.146	0.538	0.066	0.219	0.112	-0.034
Q53.	My confidence comes after handling 3–4 patients.	0.027	-0.144	0.177	-0.079	-0.329	0.250	-0.110
Q54.	I can manage my patients properly.	0.548	-0.166	0.336	0.219	0.229	0.157	0.263
Q55.	My explanation to the patient & his family makes them adhere to the treatment.	0.183	0.113	0.558	0.245	0.101	-0.017	-0.012

^a) Factor 1: Professional behavior, Factor 2: Humanity, Factor 3: Patient management, Factor 4: Clinical skill competence, Factor 5: Interpersonal skill, Factor 6: Cognitive competence, and Factor 7: Personal satisfaction.

^b) Item in factor 7 was inserted to factor 3 (due to only 1 item).

Table 5.

Result of Internal Consistency Cronbach’s α of Six Factors (n=84)

Factor name	Cronbach’s α
Professional behaviors	0.875
Humanity	0.849
Patient management	0.832
Clinical skills competence	0.814
Interpersonal skill	0.785
Cognitive competence	0.624

Table 6.

Subscales and Items of 35-Item Extracted Instrument following Factor Analysis of the Medical Graduate Results (n=84)

The item (Q)	Factor name	Amount of item retained
18, 19, 21, 24, 25, 26, 27, 28, 54	Professional behavior	9
3, 4, 31, 32, 33, 34, 41	Humanity	7
35, 39, 45, 49, 51, 52, 55, 37^a)	Patient management	8
11, 12, 13, 15, 16, 20	Clinical skills competence	6
43, 44,	Interpersonal skill	2
5, 6, 47	Cognitive competence	3
Total item		35

^a) Inserted to factor 3 (patient management); factor 7 (personal satisfaction) was deleted.

Table 7.

Comparison of Factors Extracted in This Study with CanMEDS and ACGME’s Competency Framework

Factors extracted	CanMEDS Physician Competency Framework [28,29]	ACGME’s competencies [30,31]
Cognitive competence	Medical expert, scholar	Medical knowledge
Interpersonal skill	Communicator, collaborator, manager	Inter-personal & communication skills
Professional behavior	Professional	Professionalism
Patient management	Health advocate	Patient care
Clinical skill competence	-	Practice-based learning & improvement, systems based practice
Humanity	-	-

ACGME: The Accreditation Council for Graduate Medical Education.

References

1. Arnold L. Assessing professional behavior: yesterday, today, and tomorrow. Acad Med 2002;77(6):502-515.

2. Hodges BD, Ginsburg S, Cruess R, et al. Assessment of professionalism: recommendations from the Ottawa 2010 Conference. Med Teach 2011;33(5):354-363.

3. Adam J, Bore M, McKendree J, Munro D, Powis D. Can personal qualities of medical students predict in-course examination success and professional behaviour?: an exploratory prospective cohort study. BMC Med Educ 2012;12:69.

4. Lynch DC, Surdyk PM, Eiser AR. Assessing professionalism: a review of the literature. Med Teach 2004;26(4):366-373.

5. Mueller PS. Incorporating professionalism into medical education: the Mayo Clinic experience. Keio J Med 2009;58(3):133-143.

6. Cruess R, McIlroy JH, Cruess S, Ginsburg S, Steinert Y. The professionalism mini-evaluation exercise: a preliminary investigation. Acad Med 2006;81(10 Suppl):S74-S78.

7. Tsugawa Y, Tokuda Y, Ohbu S, et al. Professionalism mini-evaluation exercise for medical residents in Japan: a pilot study. Med Educ 2009;43(10):968-978.

8. Ruhe V, Boudreau JD. The 2011 Program Evaluation Standards: a framework for quality in medical education programme evaluations. J Eval Clin Pract 2013;19(5):925-932.

9. Dilmore TC, Rubio DM, Cohen E, et al. Psychometric properties of the mentor role instrument when used in an academic medicine setting. Clin Transl Sci 2010;3(3):104-108.

10. Yeo S, Chang BH. Students’ self-assessment of achievement of terminal competency and 4-year trend of student evaluation on outcome-based education. Korean J Med Educ 2019;31(1):39-50.

11. Kusmiati M, Hamid NA, Sanip S, Emilia O. Development of an instrument for preceptor evaluation of medical graduates’ performance: the psychometric properties. Med Sci Educ 2019;29(4):935-940.

12. Kusmiati M, Bahari R, Hamid NA, Sanip S, Emilia O. Validation of patient perception instruments for junior doctor performance: a factor analysis. Glob Med Health Commun 2019;7(1):71-80.

13. Stalmeijer RE, Dolmans DH, Wolfhagen IH, Muijtjens AM, Scherpbier AJ. The development of an instrument for evaluating clinical teachers: involving stakeholders to determine content validity. Med Teach 2008;30(8):e272. e277.

14. Boerebach BC, Lombarts KM, Arah OA. Confirmatory factor analysis of the system for evaluation of teaching qualities (SETQ) in graduate medical training. Eval Health Prof 2016;39(1):21-32.

15. Anthoine E, Moret L, Regnault A, Sébille V, Hardouin JB. Sample size used to validate a scale: a review of publications on newly-developed patient reported outcomes measures. Health Qual Life Outcomes 2014;12:176.

16. Stenfors-Hayes T, Hult H, Dahlgren LO. What does it mean to be a good teacher and clinical supervisor in medical education? Adv Health Sci Educ Theory Pract 2011;16(2):197-210.

17. Pluye P, Hong QN. Combining the power of stories and the power of numbers: mixed methods research and mixed studies reviews. Annu Rev Public Health 2014;35:29-45.

18. Lesser CS, Lucey CR, Egener B, Braddock CH 3rd, Linas SL, Levinson W. A behavioral and systems view of professionalism. JAMA 2010;304(24):2732-2737.

19. Kasule OH. Medical professionalism and professional organizations. J Taibah Univ Med Sci 2013;8(3):137-141.

20. Epstein RM, Hundert EM. Defining and assessing professional competence. JAMA 2002;287(2):226-235.

21. Campbell EG, Regan S, Gruen RL, et al. Professionalism in medicine: results of a national survey of physicians. Ann Intern Med 2007;147(11):795-802.

22. Haque M, Zulkifli Z, Haque SZ, et al. Professionalism perspectives among medical students of a novel medical graduate school in Malaysia. Adv Med Educ Pract 2016;7:407-422.

23. Bahaziq W, Crosby E. Physician professional behaviour affects outcomes: a framework for teaching professionalism during anesthesia residency. Can J Anaesth 2011;58(11):1039-1050.

24. Worthington RL, Whittaker TA. Scale development research: a content analysis and recommendations for best practices. Couns Psychol 2006;34(6):806-838.

25. Guilford JP, Fruchter B. Fundamental statistics in psychology and education. 6th ed. New York, USA: McGraw-Hill; 1978.

26. Sweet LR, Palazzi DL. Application of Kern’s six-step approach to curriculum development by global health residents. Educ Health (Abingdon) 2015;28(2):138-141.

27. Shirali G, Shekari M, Angali KA. Assessing reliability and validity of an instrument for measuring resilience safety culture in sociotechnical systems. Saf Health Work 2018;9(3):296-307.

28. Frank JR. The CanMEDS 2005 Physician Competency Framework: better standards, better physicians, better care. Ottawa, Canada: Royal College of Physicians and Surgeons of Canada; 2005.

29. Frank JR, Snell L, Sherbino J. CanMEDS 2015 Physician Competency Framework. Ottawa, Canada: Royal College of Physicians and Surgeons of Canada; 2015.

30. Swing SR. The ACGME outcome project: retrospective and prospective. Med Teach 2007;29(7):648-654.

31. Malik MU, Diaz Voss Varela DA, Stewart CM, et al. Barriers to implementing the ACGME outcome project: a systematic review of program director surveys. J Grad Med Educ 2012;4(4):425-433.

32. Costello AB, Osborne J. Best practices in exploratory factor analysis: four recommendations for getting the most from your analysis. Pract Assess Res Eval 2005;10(1):7.

33. Claramita M, Nugraheni MD, van Dalen J, van der Vleuten C. Doctor-patient communication in Southeast Asia: a different culture? Adv Health Sci Educ Theory Pract 2013;18(1):15-31.