SYSTEMATIC REVIEW OF THE USABILITY EVALUATION METHODOLOGY OF HEALTH INFORMATION SYSTEMS
DOI:
https://doi.org/10.31603/pharmacy.v8i1.6625Keywords:
Information Systems, Health, Systematic Review, Methods, UsabilityAbstract
A good information system will support the implementation of the tasks and functions of the organization. This systematic review aims to identify and characterize the methods and attributes in usability evaluation and identify measurement techniques in usability evaluation. This systematic review uses the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) methodology. The search was carried out using two databases, namely PubMed and Science Direct, by limiting information systems related to health published from 2015 to 2021. Articles that met the criteria were evaluated for methods, attributes, measurement techniques, and usability criteria used. Article quality was assessed using a modified Downs and Black checklist. The search identified 132 articles. The results of abstract screening and duplication-checking obtained 128 articles. The screening was carried out through a complete review of the inclusion and exclusion criteria, obtained 9 articles that met the criteria. There are five methods used for usability evaluation, namely Think aloud (TA), Heuristic evaluation (HE), System Usability Scale (SUS), User viewpoint, and Cognitive Walkthrough Method. Six problem definitions used in usability evaluation are ISO, Nielsen, user-based, TAUCHI, 19 item usability questions, and successful completion of scenarios. The measurement technique is done by qualitative and quantitative analysis. Effectiveness, efficiency, and satisfaction are criteria that are always assessed from almost all articles reviewed. The heuristic evaluation method with quantitative measurement techniques using an evaluator in evaluating usability based on scenarios then proceeding with qualitative measurement techniques to deepen problems and provide solutions is the most ideal usability evaluation measurement technique.
References
Abran, A., Khelifi, A., Suryn, W., & Seffah, A. (2003). Usability Meanings and Interpretations in ISO Standards. Software Quality Journal, 11(4), 325–338. https://doi.org/10.1023/A:1025869312943
Agharezaei, Z., Khajouei, R., Ahmadian, L., & Agharezaei, L. (2020). Compliance with design principles: A case study of a widely used laboratory information system. Eastern Mediterranean Health Journal, 26(12), 1456–1464. https://doi.org/10.26719/emhj.20.029
Atashi, A., Khajouei, R., Azizi, A., & Dadashi, A. (2016). User Interface Problems of a Nationwide Inpatient Information System: A Heuristic Evaluation. Applied Clinical Informatics, 07(01), 89–100. https://doi.org/10.4338/ACI-2015-07-RA-0086
Bastien, J. M. C. (2010). Usability testing: A review of some methodological and technical aspects of the method. International Journal of Medical Informatics, 79(4), e18–e23. https://doi.org/10.1016/j.ijmedinf.2008.12.004
Brown, N., Eghdam, A., & Koch, S. (2019). Usability Evaluation of Visual Representation Formats for Emergency Department Records. Applied Clinical Informatics, 10(03), 454–470. https://doi.org/10.1055/s-0039-1692400
Downs, S. H., & Black, N. (1998). The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. Journal of Epidemiology & Community Health, 52(6), 377–384. https://doi.org/10.1136/jech.52.6.377
Ebnehoseini, Z., Tara, M., Meraji, M., Deldar, K., Khoshronezhad, F., & Khoshronezhad, S. (2018). Usability Evaluation of an Admission, Discharge, and Transfer Information System: A Heuristic Evaluation. Open Access Macedonian Journal of Medical Sciences, 6(11), 1941–1945. https://doi.org/10.3889/oamjms.2018.392
Ependi, U., Kurniawan, T. B., & Panjaitan, F. (2019). System Usability Scale Vs Heuristic Evaluation: A Review. Simetris: Jurnal Teknik Mesin, Elektro dan Ilmu Komputer, 10(1), 65–74. https://doi.org/10.24176/simet.v10i1.2725
Farrahi, R., Rangraz Jeddi, F., Nabovati, E., Sadeqi Jabali, M., & Khajouei, R. (2019). The relationship between user interface problems of an admission, discharge and transfer module and usability features: A usability testing
method. BMC Medical Informatics and Decision Making, 19(1), 172. https://doi.org/10.1186/s12911-019-0893-x
Farzandipour, M., Sadeqi Jabali, M., Nickfarjam, A. M., & Tadayon, H. (2021). Usability evaluation of selected picture archiving and communication systems at the national level: Analysis of users’ viewpoints. International Journal of Medical Informatics, 147, 104372. https://doi.org/10.1016/j.ijmedinf.2020.104372
Hartson, H. R., Andre, T. S., & Williges, R. C. (2003). Criteria For Evaluating Usability Evaluation Methods. International Journal of Human-Computer Interaction, 15(1), 145–181. https://doi.org/10.1207/S15327590IJHC1501_13
Henriksen, K., & Albolino, S. (2010). Towards a safer healthcare system. Quality and Safety in Health Care, 19(Suppl 3), i1–i2. https://doi.org/10.1136/qshc.2010.045849
Horsky, J., McColgan, K., Pang, J. E., Melnikas, A. J., Linder, J. A., Schnipper, J. L., & Middleton, B. (2010). Complementary methods of system usability evaluation: Surveys and observations during software design and development cycles. Journal of Biomedical Informatics, 43(5), 782–790. https://doi.org/10.1016/j.jbi.2010.05.010
Ichsani, Y. (2018). Usability Performance Evaluation of Information System with Concurrent Think-Aloud Method as User Acceptance Testing: A Literature Review. Proceedings of the International Conference on Science and Technology (ICOSAT 2017). International Conference on Science and Technology (ICOSAT 2017) - Promoting Sustainable Agriculture, Food Security, Energy, and Environment Through Science and Technology for Development, Ancol, Indonesia. https://doi.org/10.2991/icosat-17.2018.26
Indyanty Wuryaning Lestari, E., Putri Yuniarti, T., & Loekqijana Agrawati, A. (2014). Pengembangan Laboratory Information System untuk Memperbaiki Waktu Tunggu Layanan. Jurnal Kedokteran Brawijaya, 28(1), 53–57. https://doi.org/10.21776/ub.jkb.2014.028.01.12
Isfahani, S. S., Khajouei, R., Jahanbakhsh, M., & Mirmohamadi, M. (2014). The evaluation of hospital laboratory information management systems based on the standards of the American National Standard Institute. Journal of Education and Health Promotion, 3, 5.
Kamel Ghalibaf, A., Jangi, M., Mazaheri Habibi, M. R., Zangouei, S., & Khajouei, R. (2018). Usability evaluation of obstetrics and gynecology information system using cognitive walkthrough method. Electronic Physician, 10(4), 6682–6688. https://doi.org/10.19082/6682
Khajouei, R., & Farahani, F. (2020). A combination of two methods for evaluating the usability of a hospital information system. BMC Medical Informatics and Decision Making, 20(1), 84. https://doi.org/10.1186/s12911-020-1083-6
Khajouei, R., Hajesmaeel Gohari, S., & Mirzaee, M. (2018). Comparison of two heuristic evaluation methods for evaluating the usability of health information systems. Journal of Biomedical Informatics, 80, 37–42. https://doi.org/10.1016/j.jbi.2018.02.016
Kruger, R., Brosens, J., & Hattingh, M. (2020). A Methodology to Compare the Usability of Information Systems. In M. Hattingh, M. Matthee, H. Smuts, I. Pappas, Y. K. Dwivedi, & M. Mäntymäki (Eds.), Responsible Design, Implementation and Use of Information and Communication Technology (Vol. 12067, pp. 452–463). Springer International Publishing. https://doi.org/10.1007/978-3-030-45002-1_39
Marghescu, D. (2009). Usability Evaluation of Information Systems: A Review of Five International Standards. In W. Wojtkowski, G. Wojtkowski, M. Lang, K. Conboy, & C. Barry (Eds.), Information Systems Development (pp. 131–142). Springer US. https://doi.org/10.1007/978-0-387-68772-8_11
Nguyen, L., Bellucci, E., & Nguyen, L. T. (2014). Electronic health records implementation: An evaluation of information system impact and contingency factors. International Journal of Medical Informatics, 83(11), 779–796. https://doi.org/10.1016/j.ijmedinf.2014.06.011
Nielsen, J., & Landauer, T. K. (1993). A mathematical model of the finding of usability problems. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI ’93, 206–213. https://doi.org/10.1145/169059.169166
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W.,
Mayo-Wilson, E., McDonald, S.,Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. Systematic Reviews, 10(1), 89. https://doi.org/10.1186/s13643-021-01626-4
Rangraz Jeddi, F., Nabovati, E., Bigham, R., & Farrahi, R. (2020). Usability evaluation of a comprehensive national health information system: A heuristic evaluation. Informatics in Medicine Unlocked, 19, 100332. https://doi.org/10.1016/j.imu.2020.100332
Sagar, K., & Saha, A. (2017). A systematic review of software usability studies. International Journal of Information Technology. https://doi.org/10.1007/s41870-017-0048-1
Seffah, A., Donyaee, M., Kline, R. B., & Padda, H. K. (2006). Usability measurement and metrics: A consolidated model. Software Quality Journal, 14(2), 159–178. https://doi.org/10.1007/s11219-006-7600-8
Tristantia, A. D. (2018). Evaluasi Sistem Pelaporan Insiden Keselamatan Pasien Di Rumah Sakit. Jurnal Administrasi Kesehatan Indonesia, 6(2), 83. https://doi.org/10.20473/jaki.v6i2.2018.83-94
Weichbroth, P. (2018). Usability attributes revisited: A time-framed knowledge map. 1005–1008. https://doi.org/10.15439/2018F137
Wronikowska, M. W., Malycha, J., Morgan, L. J., Westgate, V., Petrinic, T., Young, J. D., & Watkinson, P. J. (2021). Systematic review of applied usability metrics within usability evaluation methods for hospital electronic healthcare record systems: Metrics and Evaluation Methods for eHealth Systems. Journal of Evaluation in Clinical Practice, 27(6), 1403–1416. https://doi.org/10.1111/jep.13582
Yuniarto, D., Subiyakto, A., Abd. Rohman, A. B., & Marliana, R. R. (2019). Assessment of Readiness and Usability of Information Systems Use. Jurnal Online Informatika, 4(1), 1. https://doi.org/10.15575/join.v4i1.256
Zahiri Esfahani, M., Khajouei, R., & Baneshi, M. R. (2018). Augmentation of the think aloud method with users’ perspectives for the selection of a picture archiving and communication system. Journal of Biomedical Informatics, 80, 43–51. https://doi.org/10.1016/j.jbi.2018.02.017
