Skip Navigation
Skip to contents

JEEHP : Journal of Educational Evaluation for Health Professions

OPEN ACCESS
SEARCH
Search
Advanced Search
mobile menu button

Articles

Page Path
HOME > J Educ Eval Health Prof > Volume 14; 2017 > Article
Brief report
 Smart device-based testing for medical students in Korea: satisfaction, convenience, and advantages
Eun Young Lim1orcid, Mi Kyoung Yim2orcid, Sun Huh3*orcid

DOI: https://doi.org/10.3352/jeehp.2017.14.7
Published online: April 24, 2017

1Division of Educational Evaluation, Korea Institute for Curriculum and Evaluation, Seoul, Korea

2Research and Development Division, Korea Health Personnel Licensing Examination Institute, Seoul, Korea

3Department of Parasitology and Institute of Medical Education, Hallym University College of Medicine, Chuncheon, Korea

*Corresponding email: shuh@hallym.ac.kr

Editor: A Ra Cho, The Catholic University of Korea, Korea

• Received: April 13, 2017   • Accepted: April 22, 2017

© 2017, Korea Health Personnel Licensing Examination Institute

This is an open-access article distributed under the terms of the Creative Commons Attribution License <http://creativecommons.org/licenses/by/4.0/>, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

prev next
  • 34,316 Views
  • 293 Download
  • 7 Web of Science
  • 11 Crossref
  • 7 Scopus
Full Article

Download PDF Download PDF

  • The aim of this study was to investigate respondents’ satisfaction with smart device-based testing (SBT), as well as its convenience and advantages, in order to improve its implementation. The survey was conducted among 108 junior medical students at Kyungpook National University School of Medicine, Korea, who took a practice licensing examination using SBT in September 2015. The survey contained 28 items scored using a 5-point Likert scale. The items were divided into the following three categories: satisfaction with SBT administration, convenience of SBT features, and advantages of SBT compared to paper-and-pencil testing or computer-based testing. The reliability of the survey was 0.95. Of the three categories, the convenience of the SBT features received the highest mean (M) score (M= 3.75, standard deviation [SD]= 0.69), while the category of satisfaction with SBT received the lowest (M= 3.13, SD= 1.07). No statistically significant differences across these categories with respect to sex, age, or experience were observed. These results indicate that SBT was practical and effective to take and to administer.
  • Keywords: Computers; Personal satisfaction; Republic of Korea; Medical students; Tablets
In 2020, the Korean medical licensing examination will be administered as a smart device-based test using a tablet PC as the user interface [1]. Therefore, medical schools have attempted to implement smart device-based testing (SBT) in the administration of practice licensing examinations and have investigated students’ satisfaction with SBT, as well as its effectiveness and convenience, in order to improve students’ adaptation to it. The purpose of this study was to investigate satisfaction with SBT and its perceived convenience among medical students at Kyungpook National University, Korea.
This was a cross-sectional observational study.
The evaluation survey included all 108 junior medical students at Kyungpook National University, Korea. They took a practice licensing examination using SBT and filled out an evaluation survey in September 2015. The evaluation survey for SBT consisted of three categories: satisfaction with SBT (two items), convenience of SBT features (13 items), and advantages of SBT (13 items). The category assessing the advantages of SBT was composed of two parts: one compared respondents’ satisfaction with SBT to their satisfaction with paper-and-pencil testing, and the other part contained a similar comparison with computer-based testing (CBT). The following 5-point Likert scale was used for the survey: 1, strongly disagree; 2, disagree; 3, neutral; 4, agree; and 5, strongly agree. Table 1 presents the content of the three categories, the number of items, and the reliability of the survey. For determining the reliability of the survey, internal consistency (Cronbach α) was used. The Cronbach α coefficient of the survey was 0.905, and the range of this coefficient for each item was 0.807 to 0.936; this indicated strong internal consistency and a high level of reliability. Table 2 presents the respondents’ background in terms of gender, age, and experience with CBT and SBT. The percentage of respondents with CBT experience was 96.3%, and the corresponding value for SBT was 17.6%. This indicates that most of the students had some experience with CBT, but not many students had experience with SBT.
The results of the evaluation of SBT were summarized using descriptive statistics in SPSS for Windows ver. 23.0 (IBM Corp., Armonk, NY, USA). The responses regarding satisfaction, convenience, and advantages were compared according to sex, age, experience with CBT, and experience with SBT. Two-way repeated-measures analysis of variance test was conducted for a comparative analysis using DBSTAT ver. 5.0 (DBSTAT Co., Chuncheon, Korea).
Informed consent was obtained from the subjects. The study design was approved by the institutional review board of Hallym University (HIRB-2015-092).
Table 3 shows the descriptive statistics of the three categories and the two subcategories regarding the advantages of SBT. The category of the convenience of SBT received the highest mean (M) score (M= 3.75, standard deviation [SD]= 0.69). The next highest mean was that of the category assessing the advantages of SBT compared to paper-and-pencil testing (3.43). The lowest mean was that of the category evaluating satisfaction with SBT (M= 3.13, SD= 1.07). The means among the three categories were significantly different (P< 0.0001). Supplementary files of the coding books and integrated data for each category are presented in Supplement 1. Appendix 1 presents the results of all the items. The range of the 28 items was 2.83 to 4.23. Satisfaction with SBT was assessed using two items: satisfaction with using a tablet PC as a testing tool (M=2.92, SD=1.24) and lack of difficulty in carrying out the testing according to the advice for test-takers presented on the tablet PC (M=3.13, SD=1.26). The percentage of negative responses (strongly disagree and disagree) was higher than 30% for the former item; this implies that further improvement in SBT administration is necessary.
The category evaluating the convenience of SBT features was composed of 13 items. The range of the items in this category was 2.83 to 4.23. The highest mean score (M= 4.23, SD= 0.74) was found for the item assessing whether the function of identifying items that were not marked before submitting the answers was convenient, and the next highest mean score was found for the following item: ‘it was convenient to have a ‘check’ function for later review’ (M=4.18, SD= 0.77). This indicates that students recognized that the features that enabled them to review and correct their answers were the most convenient. The lowest mean was for the item assessing whether it was convenient to use the functions of zoom in and zoom out when viewing an image or replaying a video file (M= 2.83, SD= 1.29). The next lowest mean was of the following item: ‘the loading time was adequate before the start of video playback’ (M= 3.25, SD= 1.29). Therefore, our results indicate that the features related to video need to be improved.
The results for the advantages of SBT compared to paper-and-pencil testing were as follows: The range of mean scores was 2.91 to 3.89, indicating that the advantages of SBT were positively evaluated by the students. The highest mean was for the item assessing whether the absence of separate answer-card marking was helpful in managing the testing time (M= 3.89, SD= 1.06). The next highest mean was for the following item: ‘it was convenient to select the correct answer on the tablet PC screen rather than marking an optical mark reader (OMR) answer card’ (M= 3.85, SD= 1.06). The lowest mean was for the following item: ‘tablet PC-based testing was more convenient than paper-and-pencil testing’ (M= 2.91, SD= 1.09). The next lowest mean was for the following item: ‘it was good to be able to see materials such as photographs, sounds, and video clips’ (M=3.07, SD= 1.09). The range of the advantages of SBT compared to CBT was 2.94 to 3.34. The highest mean was for the following item: ‘there was no noise or heat from the desktop computer, so the examination environment was comfortable’ (M= 3.34, SD= 1.02). The lowest mean was for the following item: ‘the testing instrument was smart, and it helped me to focus more on the testing’ (M=2.94, SD=1.14).
No statistically significant differences were observed in the mean scores for satisfaction with SBT, its convenience, and its advantages according to sex (P=0.2979), age (P=0.7126), experience with CBT (P= 0.5024), or experience with SBT (P= 0.2640) (Fig. 1).
SBT has also been administered in several other medical schools in Korea, where it has been known as ubiquitous-based testing (UBT). UBT was established to be a satisfactory and convenient testing tool in one of these medical schools [2]. In another medical school, students had a positive attitude towards the UBT interface. The factors of having experience with CBT and being male were associated with favorable responses towards UBT [3]. In this study, students thought that SBT could improve the effectiveness of taking a test, reduce extra work such as marking answers on an OMR sheet, and make the testing environment more pleasant than that in the computer lab. As the means of the items showed positive patterns for the satisfaction and convenience of SBT, most items had a high rate for ‘agree’ and ‘strongly agree.’ In conclusion, the results of the survey indicate a positive attitude toward SBT. However, the technical features of SBT, such as the zoom in and zoom out functions when viewing an image and playing video clips, could be improved further.

Authors’ contributions

Conceptualization: SH. Data curation: MKY. Formal analysis: EYL. Funding acquisition: SH. Methodology: EYL MKY SH. Project administration: SH. Visualization: EYL MKY SH. Writing–original draft: EYL. Writing–review & editing: EYL MKY SH.

Conflict of interest

No potential conflict of interest relevant to this article was reported.

Funding

This work is supported by the National Health Personnel Licensing Examination Board of Korea (Fundref ID: 10.13039/501100003647) research fund (RE02-1604-00).

Supplement 1. Data file is available from https://doi.org/10.7910/DVN/GT2IZY
Supplement 2. Audio recording of the abstract.
jeehp-14-07-abstract-recording.avi
Fig. 1.
Average responses in each category on a 5-point Likert scale on items assessing satisfaction with smart device-based testing, as well as its convenience and advantages, among medical students in Korea according to gender.
jeehp-14-07f1.gif
Table 1.
The contents, number of items, and Cronbach α coefficient of each category in the survey
Categories Content No. of items Cronbach a
Satisfaction of SBT Satisfaction of students taking SBT 2 0.807
Convenience of SBT features Degree of convenience of SBT features 13 0.916
Advantages of SBT Advantage of SBT compared to a paper testing and pencil testing and a computer based testing. 13 0.936
 1) Compared to a paper and pencil testing 7 0.905
 2) Compared to a computer based testing 6 0.892
Total 28 0.905

SBT, smart device-based testing.

Table 2.
The number of subjects based on their background
Background Frequency (%)
Gender
 Male 70 (64.8)
 Female 38 (35.2)
 Total 108 (100.0)
Age (yr)
 20-24 2 (1.9)
 25-30 93 (86.1)
 ≥31 13 (12.0)
 Total 108 (100.0)
Experience of taking computer-based testing
 No 4 (3.7)
 Yes 104 (96.3)
 Total 108 (100.0)
Experience of taking smart device-based testing
 No 89 (82.4)
 Yes 19 (17.6)
 Total 108 (100.0)
Table 3.
The descriptive statistics of 3 categories and 2 subcategories (N = 108)
Mean ± standard deviation
Satisfaction of SBT 3.13±1.07
Convenience of SBT 3.75±0.69
Advantages of SBT 3.33±0.83
 1) Compared to a paper and pencil testing 3.43±0.89
 2) Compared to a computer based testing 3.20±0.87

SBT, smart device-based testing.

Appendix 1.
Frequency and mean of each item in the survey given to medical students in Korea after smart device-based testing in 2015
Item Mean ± standard deviation Frequency
Strongly disagree Disagree Neutral Agree Strongly agree
Satisfaction with SBT
 1. I was satisfied with using the tablet PC as the testing tool. 2.92 ±1.24 19 20 30 29 10
 2. There was no difficulty in carrying out the test according to the advice for test takers presented on the tablet PC. 3.13 ±1.26 15 18 20 30 16
Convenience of SBT
 3. Function of the remaining testing time notification was more convenient than an announcement of the remaining testing time. 3.35 ±1.26 11 19 20 37 21
 4. The selection and correction function of the correct answer was convenient. 3.80 ±1.00 3 10 19 50 26
 5. It was convenient to see one item at a time on the screen. 3.85 ±1.01 2 11 19 45 31
 6. The function of identifying items that were not marked before submitting the answers was convenient. 4.23 ±0.74 0 3 11 52 42
 7. It was convenient to be able to see the previous items, the next items, and all items. 3.97 ±0.86 2 4 17 57 28
 8. It was convenient to be able to see the solved items, unsolved items, checked items, and the items noted for later review selectively 3.91 ±1.03 3 9 17 45 34
 9. It was convenient to have a 'check'function for later review. 4.18±0.77 0 3 15 50 40
 10. The'check wrong answers'function to check the wrong answer option was convenient. 3.95 ±0.95 1 7 24 40 36
 11.The font size was appropriate. 3.85 ±0.96 4 6 16 58 24
 12.The typeface was appropriate. 3.93 ±0.85 3 2 19 60 24
 13. The loading time was adequate before the start of video playback. 3.25 ±0.94 5 12 55 26 11
 14. It was convenient to use then functions of zoom in and zoom out when viewing an image and replaying a video file. 2.83 ±1.29 22 22 27 26 11
 15.The overall screen configuration was appropriate. 3.58 ±0.91 4 6 34 51 13
Advantages of SBT
 Comparison with paper-and-pencil testing
  16. It was convenient to select the correct answer on the tablet PC screen rather than marking the optical mark reader answer 3.85 ±1.03 5 6 18 50 29
  17. The absence of separate answer-card marking was helpful in managing the testing time. 3.89 ±1.06 5 6 19 44 34
  18.The psychological burden (tension) decreased because there was no separate answer-card marking. 3.75±1.16 7 10 17 43 31
  19.The font size was large and clear, minimizing eye fatigue. 3.35 ±1.08 9 10 36 40 13
  20. It was good to be able to see materials such as photographs, sounds, and video clips. 3.07 ±1.09 11 18 40 30 9
  21.Tablet PC-based testing was more convenient than paper-and-pencil testing. 3.21 ±1.25 14 18 21 41 14
  22.The test taker's ability is assessed more accurately by tablet PC-based testing than by paper-and-pencil testing. 2.91 ±1.09 16 16 43 28 5
 Comparison with desktop-based testing
  23.The touch-screen method was more convenient than the mouse-click method for selecting the correct answer. 3.33 ±1.12 10 12 31 42 13
  24. There was less eye strain in the case of the tablet PC than in the case of the desktop PC monitor. 3.22 ±1.09 9 14 42 30 13
  25. There may be a decrease in cheating behavior during tablet PC-based testing as compared to desktop PC-based testing. 3.24 ±1.03 9 10 45 34 10
  26. The testing instrument was smart, and it helped me to focus more on the test. 2.94 ±1.14 16 15 46 21 10
  27. There was no noise or heat from the desktop computer, so the examination environment was comfortable. 3.34 ±1.02 8 11 34 46 9
  28. Tablet PC-based testing is an improved testing method compared to the existing desktop PC-based testing method. 3.12 ±1.09 10 18 39 31 10

SBT, smart device-based testing.

Figure & Data

References

    Citations

    Citations to this article as recorded by  
    • (NON)COMPUTER-ORIENTED TESTING IN HIGHER EDUCATION: VIEWS OF THE PARTICIPANTS OF THE EDUCATIONAL PROCESS ON (IN)CONVENIENCE USING
      Volodymyr Starosta
      OPEN EDUCATIONAL E-ENVIRONMENT OF MODERN UNIVERSITY.2024; (16): 173.     CrossRef
    • Survey of dental students’ perception of ubiquitous-based test (UBT)
      Hyoung Seok Shin, Jae-Hoon Kim
      The Journal of The Korean Dental Association.2024; 62(5): 270.     CrossRef
    • Development and application of a mobile-based multimedia nursing competency evaluation system for nursing students: A mixed-method randomized controlled study
      Soyoung Jang, Eunyoung E. Suh
      Nurse Education in Practice.2022; 64: 103458.     CrossRef
    • Effects of School-Based Exercise Program on Obesity and Physical Fitness of Urban Youth: A Quasi-Experiment
      Ji Hwan Song, Ho Hyun Song, Sukwon Kim
      Healthcare.2021; 9(3): 358.     CrossRef
    • Development, Application, and Effectiveness of a Smart Device-based Nursing Competency Evaluation Test
      Soyoung Jang, Eunyoung E. Suh
      CIN: Computers, Informatics, Nursing.2021; 39(11): 634.     CrossRef
    • Evaluation of Student Satisfaction with Ubiquitous-Based Tests in Women’s Health Nursing Course
      Mi-Young An, Yun-Mi Kim
      Healthcare.2021; 9(12): 1664.     CrossRef
    • How to Deal with the Concept of Authorship and the Approval of an Institutional Review Board When Writing and Editing Journal Articles
      Sun Huh
      Laboratory Medicine and Quality Assurance.2020; 42(2): 63.     CrossRef
    • Evaluation of usefulness of smart device-based testing: a survey study of Korean medical students
      Youngsup Christopher Lee, Oh Young Kwon, Ho Jin Hwang, Seok Hoon Ko
      Korean Journal of Medical Education.2020; 32(3): 213.     CrossRef
    • Presidential address: Preparing for permanent test centers and computerized adaptive testing
      Chang Hwi Kim
      Journal of Educational Evaluation for Health Professions.2018; 15: 1.     CrossRef
    • Journal Metrics of Infection & Chemotherapy and Current Scholarly Journal Publication Issues
      Sun Huh
      Infection & Chemotherapy.2018; 50(3): 219.     CrossRef
    • The relationship of examinees’ individual characteristics and perceived acceptability of smart device-based testing to test scores on the practice test of the Korea Emergency Medicine Technician Licensing Examination
      Eun Young Lim, Mi Kyoung Yim, Sun Huh
      Journal of Educational Evaluation for Health Professions.2018; 15: 33.     CrossRef

    Figure
    • 0
    Smart device-based testing for medical students in Korea: satisfaction, convenience, and advantages
    Image
    Fig. 1. Average responses in each category on a 5-point Likert scale on items assessing satisfaction with smart device-based testing, as well as its convenience and advantages, among medical students in Korea according to gender.

    Fig. 1.

    Smart device-based testing for medical students in Korea: satisfaction, convenience, and advantages
    Categories Content No. of items Cronbach a
    Satisfaction of SBT Satisfaction of students taking SBT 2 0.807
    Convenience of SBT features Degree of convenience of SBT features 13 0.916
    Advantages of SBT Advantage of SBT compared to a paper testing and pencil testing and a computer based testing. 13 0.936
     1) Compared to a paper and pencil testing 7 0.905
     2) Compared to a computer based testing 6 0.892
    Total 28 0.905
    Background Frequency (%)
    Gender
     Male 70 (64.8)
     Female 38 (35.2)
     Total 108 (100.0)
    Age (yr)
     20-24 2 (1.9)
     25-30 93 (86.1)
     ≥31 13 (12.0)
     Total 108 (100.0)
    Experience of taking computer-based testing
     No 4 (3.7)
     Yes 104 (96.3)
     Total 108 (100.0)
    Experience of taking smart device-based testing
     No 89 (82.4)
     Yes 19 (17.6)
     Total 108 (100.0)
    Mean ± standard deviation
    Satisfaction of SBT 3.13±1.07
    Convenience of SBT 3.75±0.69
    Advantages of SBT 3.33±0.83
     1) Compared to a paper and pencil testing 3.43±0.89
     2) Compared to a computer based testing 3.20±0.87
    Item Mean ± standard deviation Frequency
    Strongly disagree Disagree Neutral Agree Strongly agree
    Satisfaction with SBT
     1. I was satisfied with using the tablet PC as the testing tool. 2.92 ±1.24 19 20 30 29 10
     2. There was no difficulty in carrying out the test according to the advice for test takers presented on the tablet PC. 3.13 ±1.26 15 18 20 30 16
    Convenience of SBT
     3. Function of the remaining testing time notification was more convenient than an announcement of the remaining testing time. 3.35 ±1.26 11 19 20 37 21
     4. The selection and correction function of the correct answer was convenient. 3.80 ±1.00 3 10 19 50 26
     5. It was convenient to see one item at a time on the screen. 3.85 ±1.01 2 11 19 45 31
     6. The function of identifying items that were not marked before submitting the answers was convenient. 4.23 ±0.74 0 3 11 52 42
     7. It was convenient to be able to see the previous items, the next items, and all items. 3.97 ±0.86 2 4 17 57 28
     8. It was convenient to be able to see the solved items, unsolved items, checked items, and the items noted for later review selectively 3.91 ±1.03 3 9 17 45 34
     9. It was convenient to have a 'check'function for later review. 4.18±0.77 0 3 15 50 40
     10. The'check wrong answers'function to check the wrong answer option was convenient. 3.95 ±0.95 1 7 24 40 36
     11.The font size was appropriate. 3.85 ±0.96 4 6 16 58 24
     12.The typeface was appropriate. 3.93 ±0.85 3 2 19 60 24
     13. The loading time was adequate before the start of video playback. 3.25 ±0.94 5 12 55 26 11
     14. It was convenient to use then functions of zoom in and zoom out when viewing an image and replaying a video file. 2.83 ±1.29 22 22 27 26 11
     15.The overall screen configuration was appropriate. 3.58 ±0.91 4 6 34 51 13
    Advantages of SBT
     Comparison with paper-and-pencil testing
      16. It was convenient to select the correct answer on the tablet PC screen rather than marking the optical mark reader answer 3.85 ±1.03 5 6 18 50 29
      17. The absence of separate answer-card marking was helpful in managing the testing time. 3.89 ±1.06 5 6 19 44 34
      18.The psychological burden (tension) decreased because there was no separate answer-card marking. 3.75±1.16 7 10 17 43 31
      19.The font size was large and clear, minimizing eye fatigue. 3.35 ±1.08 9 10 36 40 13
      20. It was good to be able to see materials such as photographs, sounds, and video clips. 3.07 ±1.09 11 18 40 30 9
      21.Tablet PC-based testing was more convenient than paper-and-pencil testing. 3.21 ±1.25 14 18 21 41 14
      22.The test taker's ability is assessed more accurately by tablet PC-based testing than by paper-and-pencil testing. 2.91 ±1.09 16 16 43 28 5
     Comparison with desktop-based testing
      23.The touch-screen method was more convenient than the mouse-click method for selecting the correct answer. 3.33 ±1.12 10 12 31 42 13
      24. There was less eye strain in the case of the tablet PC than in the case of the desktop PC monitor. 3.22 ±1.09 9 14 42 30 13
      25. There may be a decrease in cheating behavior during tablet PC-based testing as compared to desktop PC-based testing. 3.24 ±1.03 9 10 45 34 10
      26. The testing instrument was smart, and it helped me to focus more on the test. 2.94 ±1.14 16 15 46 21 10
      27. There was no noise or heat from the desktop computer, so the examination environment was comfortable. 3.34 ±1.02 8 11 34 46 9
      28. Tablet PC-based testing is an improved testing method compared to the existing desktop PC-based testing method. 3.12 ±1.09 10 18 39 31 10
    Table 1. The contents, number of items, and Cronbach α coefficient of each category in the survey

    SBT, smart device-based testing.

    Table 2. The number of subjects based on their background

    Table 3. The descriptive statistics of 3 categories and 2 subcategories (N = 108)

    SBT, smart device-based testing.

    SBT, smart device-based testing.

    Table 1.

    Table 2.

    Table 3.

    JEEHP : Journal of Educational Evaluation for Health Professions
    © Korea Health Personnel Licensing Examination Institute.
    TOP