Reliability and validity test of the Korean version of Noe's evaluation

Article information

Korean J Med Educ. 2017;29(1):15-26
Publication date (electronic) : 2017 February 28
doi : https://doi.org/10.3946/kjme.2017.49
Department of Medical Education and Medical Humanities, Kyung Hee University School of Medicine, Seoul, Korea
Corresponding Author: Oh Young Kwon (http://orcid.org/0000-0003-0817-2256) Department of Medical Education and Medical Humanities, Kyung Hee University School of Medicine, 26 Kyungheedae-ro, Dongdaemun-gu, Seoul 02447, Korea Tel: +82.2.961.9102 Fax: +82.2.969.6958 email: koy04@naver.com
Received 2016 July 4; Revised 2016 October 5; Accepted 2016 December 8.

Abstract

Purpose

This study aims to demonstrate whether the functionalities between the widely used questionnaire in other countries and the Korean-translated version show similarity. Also, it intends to verify their reliability and validity.

Methods

The original questionnaire was first developed by professor Noe at the University of Minnesota with 29 items named as “mentoring functions” to identify the participants’ psychological and career-development functions. Using the Korean-translated version of the original questionnaire, the study was conducted on total 288 Kyung Hee University medical students, ranging from the first-year students to the third-year students on December 2015. In order to investigate if the survey form could be applied to the Korean participants, exploratory factor analysis and confirmatory factor analysis were conducted, using varimax rotation. Cronbach α statistics and the figures of standardized regression weights were analyzed respectively to indicate construct, convergent, and discriminant validities.

Results

The result of exploratory factor analysis shows there are a total of three functions, including the additional “friendship function” (Eigen value, 1.152; significant level if higher than 1.0) with its high emphasis. The result of confirmatory factor analysis also demonstrates the similarity (p=0.000). As the whole reliability scale of the three functions is significantly high (Cronbach α , 0.971), each reliability scales of the three functions are shown to be high (0.814–0.955) as well.

Conclusion

The model, with its three functions, proved the significant statistics regarding the reliability and validity. According to this ground, the conclusion is that the adapted questionnaire used in this study could be applied to Korean medical students.

Introduction

Medical students are under significant pressure because they not only have to master their major studies and practical techniques, but also set their career paths and look for jobs at the same time. According to the study [1], although medical students have various problems with their academics and career paths, it is shown that 67% of them do not find any colleagues or professors who can provide sincere advice and feedback. Thus, their burdens in the academics and futures are cumulative.

According to a study [2] conducted in the United States, most medical schools have mentoring programs (79%) or advising programs (100%). To follow this trend, Korean medical schools are adopting similar mentoring programs.

In mentoring programs, mentors are defined as doctors and medical professors who are relatively experts in their fields, and mentees are medical students who are less experienced. Mentoring is a series of activities [3] where mentors pass down their tips and know-how to their mentees and encourage them. Therefore, mentoring includes interactions between mentors and mentees, and it eventually brings positive effects for both of them. Concretely, it is expected that mentors [4] can help their mentees develop teamwork, communication, and personal-relationship skills through mentoring programs. Also, mentees can not only decide their career paths more wisely, but also reduce their pressure by sharing their personal issues with their mentors. Particularly, medical students can develop a deep understanding of a doctor as a professional job and strengthen their network through mentoring systems, because in large part, the mentees follow the same career directions as their doctor mentors did. With mentoring programs, both mentors and mentees can maintain their networks and develop their career even after graduation. Due to these benefits, accepting mentoring programs in many Korean medical schools is a rising trend.

However, there is a lack of development for valid measurements in order to prove its effectiveness and its domestic application. Therefore, developing appropriate measuring systems is significant and necessary to accumulate empirical research for the mentoring field. For these reasons, this study intends to develop an adapted Korean-version questionnaire, which [5] was originally created by professor Noe at the University of Minnesota in 1988. Finally, the purpose of this study is to establish a foothold to afford better mentoring programs to Korean medical students.

Subjects and methods

1. Subjects

The participants of this study were current Kyung Hee University medical students in their first, second, and third years. Of 288 total students, except those who responded falsely (four students) and did not respond, there were 174 male students and 114 females.

2. Measurement tool

The questionnaire that professor Noe created in 1988 at the University of Minnesota is composed of 29 items. However, this study was conducted with the Korean version (Appendix 1). All the items were measured using a 5-point Likert-type scale ranging from 1 (strongly disagree) to 5 (strongly agree). The original questionnaire was used after receiving prior consent from Professor Noe by email. Each item corresponds to an individual function, but the overall functions are largely divided into psychosocial (15 items) and career development functions (14 items). This classification of the functions is cited from the classification of mentoring functions [6], by Kathy Kram at Boston University. According to the classification of the original questionnaire, the psychosocial functions include acceptance and confirmation, role model, counseling and friendship, whereas the career development functions include coaching, protection, exposure and visibility, sponsorship, and challenging assignments. The survey questionnaire was provided as supplementary material to this article.

3. Methods

The sample estimation process should be conducted robustly in order to enhance the quality of studies. Larger samples are superior to smaller samples in many ways. Researchers can minimize the probability of committing errors, increase the generalizability of their findings, and maximize the accuracy of their conclusion through employing larger samples. According to researchers, 10 cases for each item should be sufficient to perform an exploratory factor analysis [7]. This study has a sufficient number of subjects with a total of 288 valid subjects.

This study used the Kaiser-Meyer-Olkin (KMO) test and Bartlett’s test of sphericity to measure the adequacy of samples in terms of the distribution of values for the execution of factor analysis. Exploratory factor analysis was used to examine whether the application of the adapted questionnaire to the Korean medical students can correspond to the same categorization of these functions. In the exploratory factor analysis, the principal factor method and the varimax method were used. This study conducted the confirmatory factor analysis to examine the model validity of the factors that are tied to the same group from the exploratory factor analysis. Also, since it is important to retain the reliability and validity in the confirmatory factor analysis, Cronbach α statistic was calculated and construct, convergent, and discriminant validity were conducted. The confirmatory factor analysis was performed with the use of chi-square, root mean square error of approximation (RMSEA) and comparative fit index to examine the fitness and correspondence of the model as a whole. Standard regression weights were used for the construct validity. The convergent validity was examined by using the construct reliability and variance extracted index. The discriminant validity was examined by using coefficient of determination and variance extracted index. This study used SPSS version 21.0 (IBM Corp., Armonk, USA) and Amos 20.0 (IBM Corp.), respectively, and the significance level was set for 0.05.

Results

1. General characteristics of the participants

The participants of this study were composed of 60.4% male students and 39.6% of female students. The number of people in each year is as follows (Table 1).

Demographic Characteristics of the Samples (n=288)

2. Result of exploratory factor analysis

In this study, the KMO test and Bartlett’s test of sphericity were used to measure the adequacy of the collected samples prior to the execution of exploratory for its suitability in the factor analysis. The outcome for the KMO test was 0.963, so the conformity was confirmed to perform the factor analysis. Also, as the stage to figure out common factors and to indicate the suitability of the factor analysis model, Bartlett’s test of sphericity generated 7,490.33 (0.000), showing the statistical significance. Since the variables are correlated with each other, this study can use the principal factor method. According to these grounds, the principal factor method, which uses varimax rotation, was performed as the factor extraction method. The table of the rotational component matrix is shown in the Table 2. The factor analysis was conducted with the principal component analysis using of varimax rotation to group common factors within 29 items. The result of the exploratory factor analysis within the 29 items of the original questionnaire was different from the result of this Korean version research form. The original questionnaire was classified into two parts: psychosocial and career development functions.

The Result of Factor Structure

However, this new study showed different outcomes; “friendship functions” was added. As a result, the researchers of this study decided to group the 12 items except two items in “psychosocial functions,” because the items were originally in the same category. With regard to the factor 2, the 12 items except one item were overlapped with the original questionnaire, so we decided to call them “career development functions” as the original ones are named. The last three items were classified as “friendship functions” in the original questionnaire as sub-functions. Therefore, we emphasize “friendship functions” as new functions of this study. The total explanation power of the three factors was 67.819% and each Eigen values were higher than 1.0 (Table 2).

3. Result of confirmatory factor analysis

Previously, through the exploratory factor analysis, we determined that the two categories the original research form proposed were inappropriate in South Korea. Accordingly, the confirmatory factor analysis was performed to examine the suitability of the three factor structures of the mentoring program functions that were discovered in the exploratory factor analysis. The fit index obtained in the confirmatory factor analysis is shown in the Table 3. The result of the structure equation analysis showed chi-square statistics, the degree of freedom (df), and p-value. Generally, goodness of fit is interpreted as statistically significant if p-value is higher than 0.05 and p-value gets higher when the statistics of chi-square gets smaller. The probability of the statistics of chi-square also increases when the size of samples gets higher. In other words, since the p-value is influenced by the chi-square and the size of samples, it cannot be said that the goodness of fit is lower in a situation where the size of samples is high and the data with many variables [8]. Therefore, it is important to make a decision regarding the goodness of fit with other various indexes.

Structure Model Fit Indices

There is RMSEA that is not affected by the size of samples. RMSEA is statistics that equals chi-square adjusted by using df and size of sample. Mostly, if RMSEA is lower than 0.08, it demonstrates that the goodness of fit is considerably high. Especially, within a 90% confidence interval, it demonstrates higher goodness of fit [9] when the upper value is lower than 0.08. The confirmatory factor analysis in this research demonstrated a favorable RMSEA (0.101), showing in 0.095 to 0.106 range in the 90% confidence interval. Also the confirmatory factor analysis demonstrated a reasonable comparative fit index as 0.853. A standard coefficient estimation shown in Fig. 1 demonstrates statistical significant in the level of significance 0.05. Standard coefficient estimation of the sub-function of three factors indicated 0.80 to 0.83.

Fig. 1.

Three-Factor Model of Mentoring Function

4. Result of reliability analysis

Cronbach α (Table 4) regarding the 29 items and the three factors were calculated to examine the reliability that measured the mentoring programs. As a result of the internal consistency of the scales, Cronbach α statistics for the factor 1, 2, and 3 were 0.955, 0.954, and 0.814, respectively. The overall Cronbach α statistic including all the 29 items is 0.971, demonstrated competent level of the reliability.

Reliability of Factors

5. Result of validity analysis

Construct validity was secured because standardized regression weights (Table 5) in all items were higher than 0.5. Construct reliability (Table 6) was confirmed because the internal consistency of the observed variables was higher than 0.7. Convergent validity was retained since all variance extracted indices was higher than 0.5. Regarding discriminant validity, a comparison of variance extracted indices (Table 6) and coefficients of determination (Table 7) shows that three of them are higher and three of them are lower, therefore it secured partial discriminant validity.

Standardized Regression Weights Regarding the Three Factors

Construct Reliability and Variance Extracted Index

Coefficient of Determination among the Factors

Discussion

The original Noe’s questionnaire “Mentoring Functions Item” consisted of 29 total items. Although the questionnaire was created in 1988, a lot of research [10,11] has been performed continuously based on this survey form. In various countries, other studies [12,13,14,15] investigating the function of mentoring programs have been published as well.

The questionnaire used in this study has been used widely in abroad and its validity has been proved. Also, a lot of research conducted in companies or schools in Korea have proven its validity. However, that was not the case for Korean medical students. Therefore, it was necessary to prove if the questionnaire can be applied to medical students. According to the results of factor analysis, the mentoring functions of Kram [6], which Noe quoted, were originally classified into two functions. However, we discovered a total of three functions with an emphasis on the friendship functions through our research conducted on Korean medical students. The order of Cronbach α statistics is as follows from the highest to the lowest: psychosocial functions (0.955), career development functions (0.954), and the friendship functions (0.814). This indicates that the overall reliability is high.

Thus, this adapted questionnaire used in this study is suitable for Korean students to measure the satisfaction of the mentoring programs. According to the study conducted in the United States [16], Noe’s questionnaire is the most frequently used questionnaire for evaluating mentoring programs, and it examines two functions–psychosocial functions and career development functions.

Kram [6] classified psychosocial and career development functions as the main functions of the mentoring programs. However, since then, many non-Korean studies propose that highly emphasized “role model functions” should be the third functions. After Kram classified the mentoring functions into two parts, many follow-up researchers [5] have maintained the same point of views. However, the study conducted by Scandura [17] added the “role model functions,” resulting in total three functions. This result shows some differences, because this study emphasizes “friendship functions,” not the role model functions.

The reliability and validity of each tri-functional questionnaires reach beyond satisfactory level. We have come to the conclusion that the research form can be applied to Korean medical students for the reason mentioned above. It is widely known that medical students are under so much pressure [18] because of the heavy workload, long period of study, and the pass/fail system. The fact that mentoring programs can be helpful for the medical students [19,20] has been proved through research. Considering the specific circumstances of medical school, it is important to develop measurement tools for Korean medical students to implement qualified mentoring programs.

This study confronts two limitations as follows. First, bilingual participants in Korean and English are ideal for comparing the original and the adapted questionnaire, but this study is far from that ideal process. The reason is that the researchers think it will give extra burden to medical students, who struggle with heavy workload although the students were bilingual. Second, the participants of the study are chosen only from Kyung Hee University, not from other universities. Due to the specific characteristics of mentoring programs that only Kyung Hee University possess, it is necessary to collect extra samples from other universities. In spite of these limitations, this study holds a significant meaning in that it provides an opportunity to adopt Korean version questionnaire about mentoring programs which are widely used in various countries. The purpose of the study is achieved sufficiently, since it proves its high reliability and validity in different aspects.

Through this study, we want to make two proposals. First, to secure the criterion-related validity of the study, it is necessary to use other studies with different measurement tools as criteria. This study cannot prove the criterion-related validity, due to the fact that there are no Korean measurement tools in which the reliability and validity have been qualified. Hence, the development of the measurement tools for mentoring programs should be implemented in follow-up studies. Second, this study is focused only on medical students. However, considering that interest in mentoring programs for residents and faculty members is increasing [21], measurement tools for mentoring programs for these people in follow-up studies should be developed.

In conclusion, the questionnaire used in this study proves its reliability and validity. Therefore, we are looking forward to applying this adapted questionnaire to other Korean medical schools regarding the research of mentoring programs.

Acknowledgements

None.

Notes

Funding

None.

Conflicts of interest

None.

References

1. Shin HI, Jeon WT. I'm not happy, but I don't care": help-seeking behavior, academic difficulties, and happiness. Korean J Med Educ 2011;23:7–14.
2. Fornari A, Murray TS, Menzin AW, Woo VA, Clifton M, Lombardi M, Shelov S. Mentoring program design and implementation in new medical schools. Med Educ Online 2014;19:24570.
3. Sanfey H, Hollands C, Gantt NL. Strategies for building an effective mentoring relationship. Am J Surg 2013;206:714–718.
4. Bae J, Ahn J, Pang J. The growth and challenges in mentoring participation: focused on students’ evaluation of KBU SMILE mentoring program. Soc Work Pract Res 2014;11:95–144.
5. Noe RA. An investigation of the determinants of successful assigned mentoring relationships. Pers Psychol 1988;41:457–479.
6. Kram KE. Phases of the mentor relationship. Acad Manag J 1983;26:608–625.
7. Nunnally JC. Psychometric theory 2nd edth ed. New York, USA: McGraw-Hill; 1978.
8. Joo YJ, Kim NY, Cho HK. Test development and verifying the validity and reliability for measuring a effectiveness of e-learning course in cyber university. J Korean Assoc Inf Educ 2008;12:109–120.
9. Lee HS, Rhim JH. Structural equation modeling with AMOS 6.0 Seoul, Korea: Bobmunsa; 2007. p. 32–35.
10. Son SJ, Kuchinke KP. The moderating role of trust in formal mentoring relationships in Korea. Asia Pac J Hum Resour 2016;54:57–58.
11. Ono K, Kato K. Succession of the mentor role and the experience of proteges among Japanese workers. Paper presented at: Annual Meeting of the International Society for the Systems Sciences; 2003 July 6; Crete, Greece.
12. Fowler JL, O'Gorman JG. Mentoring functions: a contemporary view of the perceptions of mentees and mentors. Br J Manag 2005;16:51–57.
13. Özkalp E, Kirel Ç, Sungur Z, Cengiz AA. Mentoring revisited from the perspective of mentors in learning process: the case study of Anadolu University in Turkey. Paper presented at: University Forum for Human Resource Development; 2006 May 31; Tilburg, Netherlands.
14. Kaderli R, Muff B, Stefenelli U, Businger A. Female surgeons' mentoring experiences and success in an academic career in Switzerland. Swiss Med Wkly 2011;141:w13233.
15. Gong R, Chen SY, Lee SL. Does mentoring work? The mediating effect of mentoring in China. Soc Behav Pers 2011;39:807–824.
16. Allen TD, Eby LT, O’Brien KE, Lentz E. The state of mentoring research: a qualitative review of current research methods and future research implications. J Vocat Behav 2008;73:343–357.
17. Scandura TA. Mentorship and career mobility: an empirical investigation. J Organ Behav 1992;13:169–174.
18. Ramanan RA, Phillips RS, Davis RB, Silen W, Reede JY. Mentoring in medicine: keys to satisfaction. Am J Med 2002;112:336–341.
19. Frei E, Stamm M, Buddeberg-Fischer B. Mentoring programs for medical students: a review of the PubMed literature 2000-2008. BMC Med Educ 2010;10:32.
20. Sambunjak D, Straus SE, Marusic A. A systematic review of qualitative research on the meaning and characteristics of mentoring in academic medicine. J Gen Intern Med 2010;25:72–78.
21. Chen MM, Sandborg CI, Hudgins L, Sanford R, Bachrach LK. A multifaceted mentoring program for junior faculty in academic pediatrics. Teach Learn Med 2016;28:320–328.

Appendix

Appendix 1.

Survey on Mentoring Program

kjme-2017-49-app1.pdf

Article information Continued

Fig. 1.

Three-Factor Model of Mentoring Function

Table 1.

Demographic Characteristics of the Samples (n=288)

Characteristic No. (%)
Gender
 Male 174 (60.4)
 Female 114 (39.6)
Grade
 1 90 (31.2)
 2 101 (35.1)
 3 97 (33.7)

Table 2.

The Result of Factor Structure

Item Factor 1 (psychosocial function) Factor 2 (career development function) Factor 3 (friendship function) Cronbach α
n 8 0.779 0.955
n 5 0.769
n 6 0.738
n 1 0.717
n 12 0.715
n 7 0.712
n 2 0.712
n 10 0.674
n 9 0.666
n 3 0.661
n 14 0.650
n 4 0.644
n 13 0.620
n 15 0.582
n 18 0.824 0.954
n 19 0.812
n 21 0.792
n 17 0.753
n 16 0.723
n 24 0.704
n 20 0.698
n 23 0.670
n 25 0.643
n 11 0.629
n 26 0.610
n 22 0.540
n 29 0.744 0.814
n 27 0.680
n 28 0.510
Eigen value 16.357 2.159 1.152
Kaiser-Meyer-Olkin 0.963
Bartlett’ test of sphericity
 Chi-square 7,490.33
 df (p-value) 406 (0.000)

df: Degree of freedom.

Table 3.

Structure Model Fit Indices

df χ2 p-value CFI RMSEA RMSEA (90% CI)
LO HI
Factor model 406 7,490.33 0.000 0.853 0.101 0.095 0.106

df: Degree of freedom, CFI: Comparative fit index, RMSEA: Root mean square error of approximation, LO: Lower limit, HI: Upper limit.

Table 4.

Reliability of Factors

Factor Cronbach α
Total 0.971
Factor 1 (psychosocial function) 0.955
Factor 2 (career development function) 0.954
Factor 3 (friendship function) 0.814

Table 5.

Standardized Regression Weights Regarding the Three Factors

Item Factor Estimate
n 8 Factor 1 0.790
n 5 Factor 1 0.794
n 6 Factor 1 0.763
n 1 Factor 1 0.712
n 12 Factor 1 0.829
n 7 Factor 1 0.753
n 2 Factor 1 0.799
n 10 Factor 1 0.800
n 9 Factor 1 0.765
n 3 Factor 1 0.783
n 14 Factor 1 0.738
n 4 Factor 1 0.744
n 13 Factor 1 0.801
n 15 Factor 1 0.821
n 18 Factor 2 0.779
n 19 Factor 2 0.805
n 21 Factor 2 0.804
n 17 Factor 2 0.790
n 16 Factor 2 0.729
n 24 Factor 2 0.850
n 20 Factor 2 0.834
n 23 Factor 2 0.821
n 25 Factor 2 0.824
n 11 Factor 2 0.722
n 26 Factor 2 0.815
n 22 Factor 2 0.787
n 29 Factor 3 0.677
n 27 Factor 3 0.746
n 28 Factor 3 0.869

Table 6.

Construct Reliability and Variance Extracted Index

Factor Construct reliability Variance extracted index
Psychosocial function 0.952 0.657
Career development function 0.971 0.702
Friendship function 0.812 0.592

Table 7.

Coefficient of Determination among the Factors

Factor Coefficient of determination
Factor 1 & Factor 1 0.689
Factor 1 & Factor 3 0.640
Factor 1 & Factor 3 0.640