Article Sidebar

Submitted
Feb 4, 2022
Published
Apr 30, 2022
Download
Pdf
Statistic
Read Counter : 940 Download : 694

Downloads

Download data is not yet available.

Main Article Content

Abstract

A good information system will support the implementation of the tasks and functions of the organization. This systematic review aims to identify and characterize the methods and attributes in usability evaluation and identify measurement techniques in usability evaluation. This systematic review uses the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) methodology. The search was carried out using two databases, namely PubMed and Science Direct, by limiting information systems related to health published from 2015 to 2021. Articles that met the criteria were evaluated for methods, attributes, measurement techniques, and usability criteria used. Article quality was assessed using a modified Downs and Black checklist. The search identified 132 articles. The results of abstract screening and duplication-checking obtained 128 articles. The screening was carried out through a complete review of the inclusion and exclusion criteria, obtained 9 articles that met the criteria. There are five methods used for usability evaluation, namely Think aloud (TA), Heuristic evaluation (HE), System Usability Scale (SUS), User viewpoint, and Cognitive Walkthrough Method. Six problem definitions used in usability evaluation are ISO, Nielsen, user-based, TAUCHI, 19 item usability questions, and successful completion of scenarios. The measurement technique is done by qualitative and quantitative analysis. Effectiveness, efficiency, and satisfaction are criteria that are always assessed from almost all articles reviewed. The heuristic evaluation method with quantitative measurement techniques using an evaluator in evaluating usability based on scenarios then proceeding with qualitative measurement techniques to deepen problems and provide solutions is the most ideal usability evaluation measurement technique.

Keywords

Information Systems Health Systematic Review Methods Usability

Article Details

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

References

  1. Abran, A., Khelifi, A., Suryn, W., & Seffah, A. (2003). Usability Meanings and Interpretations in ISO Standards. Software Quality Journal, 11(4), 325–338. https://doi.org/10.1023/A:1025869312943
  2. Agharezaei, Z., Khajouei, R., Ahmadian, L., & Agharezaei, L. (2020). Compliance with design principles: A case study of a widely used laboratory information system. Eastern Mediterranean Health Journal, 26(12), 1456–1464. https://doi.org/10.26719/emhj.20.029
  3. Atashi, A., Khajouei, R., Azizi, A., & Dadashi, A. (2016). User Interface Problems of a Nationwide Inpatient Information System: A Heuristic Evaluation. Applied Clinical Informatics, 07(01), 89–100. https://doi.org/10.4338/ACI-2015-07-RA-0086
  4. Bastien, J. M. C. (2010). Usability testing: A review of some methodological and technical aspects of the method. International Journal of Medical Informatics, 79(4), e18–e23. https://doi.org/10.1016/j.ijmedinf.2008.12.004
  5. Brown, N., Eghdam, A., & Koch, S. (2019). Usability Evaluation of Visual Representation Formats for Emergency Department Records. Applied Clinical Informatics, 10(03), 454–470. https://doi.org/10.1055/s-0039-1692400
  6. Downs, S. H., & Black, N. (1998). The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. Journal of Epidemiology & Community Health, 52(6), 377–384. https://doi.org/10.1136/jech.52.6.377
  7. Ebnehoseini, Z., Tara, M., Meraji, M., Deldar, K., Khoshronezhad, F., & Khoshronezhad, S. (2018). Usability Evaluation of an Admission, Discharge, and Transfer Information System: A Heuristic Evaluation. Open Access Macedonian Journal of Medical Sciences, 6(11), 1941–1945. https://doi.org/10.3889/oamjms.2018.392
  8. Ependi, U., Kurniawan, T. B., & Panjaitan, F. (2019). System Usability Scale Vs Heuristic Evaluation: A Review. Simetris: Jurnal Teknik Mesin, Elektro dan Ilmu Komputer, 10(1), 65–74. https://doi.org/10.24176/simet.v10i1.2725
  9. Farrahi, R., Rangraz Jeddi, F., Nabovati, E., Sadeqi Jabali, M., & Khajouei, R. (2019). The relationship between user interface problems of an admission, discharge and transfer module and usability features: A usability testing
  10. method. BMC Medical Informatics and Decision Making, 19(1), 172. https://doi.org/10.1186/s12911-019-0893-x
  11. Farzandipour, M., Sadeqi Jabali, M., Nickfarjam, A. M., & Tadayon, H. (2021). Usability evaluation of selected picture archiving and communication systems at the national level: Analysis of users’ viewpoints. International Journal of Medical Informatics, 147, 104372. https://doi.org/10.1016/j.ijmedinf.2020.104372
  12. Hartson, H. R., Andre, T. S., & Williges, R. C. (2003). Criteria For Evaluating Usability Evaluation Methods. International Journal of Human-Computer Interaction, 15(1), 145–181. https://doi.org/10.1207/S15327590IJHC1501_13
  13. Henriksen, K., & Albolino, S. (2010). Towards a safer healthcare system. Quality and Safety in Health Care, 19(Suppl 3), i1–i2. https://doi.org/10.1136/qshc.2010.045849
  14. Horsky, J., McColgan, K., Pang, J. E., Melnikas, A. J., Linder, J. A., Schnipper, J. L., & Middleton, B. (2010). Complementary methods of system usability evaluation: Surveys and observations during software design and development cycles. Journal of Biomedical Informatics, 43(5), 782–790. https://doi.org/10.1016/j.jbi.2010.05.010
  15. Ichsani, Y. (2018). Usability Performance Evaluation of Information System with Concurrent Think-Aloud Method as User Acceptance Testing: A Literature Review. Proceedings of the International Conference on Science and Technology (ICOSAT 2017). International Conference on Science and Technology (ICOSAT 2017) - Promoting Sustainable Agriculture, Food Security, Energy, and Environment Through Science and Technology for Development, Ancol, Indonesia. https://doi.org/10.2991/icosat-17.2018.26
  16. Indyanty Wuryaning Lestari, E., Putri Yuniarti, T., & Loekqijana Agrawati, A. (2014). Pengembangan Laboratory Information System untuk Memperbaiki Waktu Tunggu Layanan. Jurnal Kedokteran Brawijaya, 28(1), 53–57. https://doi.org/10.21776/ub.jkb.2014.028.01.12
  17. Isfahani, S. S., Khajouei, R., Jahanbakhsh, M., & Mirmohamadi, M. (2014). The evaluation of hospital laboratory information management systems based on the standards of the American National Standard Institute. Journal of Education and Health Promotion, 3, 5.
  18. Kamel Ghalibaf, A., Jangi, M., Mazaheri Habibi, M. R., Zangouei, S., & Khajouei, R. (2018). Usability evaluation of obstetrics and gynecology information system using cognitive walkthrough method. Electronic Physician, 10(4), 6682–6688. https://doi.org/10.19082/6682
  19. Khajouei, R., & Farahani, F. (2020). A combination of two methods for evaluating the usability of a hospital information system. BMC Medical Informatics and Decision Making, 20(1), 84. https://doi.org/10.1186/s12911-020-1083-6
  20. Khajouei, R., Hajesmaeel Gohari, S., & Mirzaee, M. (2018). Comparison of two heuristic evaluation methods for evaluating the usability of health information systems. Journal of Biomedical Informatics, 80, 37–42. https://doi.org/10.1016/j.jbi.2018.02.016
  21. Kruger, R., Brosens, J., & Hattingh, M. (2020). A Methodology to Compare the Usability of Information Systems. In M. Hattingh, M. Matthee, H. Smuts, I. Pappas, Y. K. Dwivedi, & M. Mäntymäki (Eds.), Responsible Design, Implementation and Use of Information and Communication Technology (Vol. 12067, pp. 452–463). Springer International Publishing. https://doi.org/10.1007/978-3-030-45002-1_39
  22. Marghescu, D. (2009). Usability Evaluation of Information Systems: A Review of Five International Standards. In W. Wojtkowski, G. Wojtkowski, M. Lang, K. Conboy, & C. Barry (Eds.), Information Systems Development (pp. 131–142). Springer US. https://doi.org/10.1007/978-0-387-68772-8_11
  23. Nguyen, L., Bellucci, E., & Nguyen, L. T. (2014). Electronic health records implementation: An evaluation of information system impact and contingency factors. International Journal of Medical Informatics, 83(11), 779–796. https://doi.org/10.1016/j.ijmedinf.2014.06.011
  24. Nielsen, J., & Landauer, T. K. (1993). A mathematical model of the finding of usability problems. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI ’93, 206–213. https://doi.org/10.1145/169059.169166
  25. Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W.,
  26. Mayo-Wilson, E., McDonald, S.,Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. Systematic Reviews, 10(1), 89. https://doi.org/10.1186/s13643-021-01626-4
  27. Rangraz Jeddi, F., Nabovati, E., Bigham, R., & Farrahi, R. (2020). Usability evaluation of a comprehensive national health information system: A heuristic evaluation. Informatics in Medicine Unlocked, 19, 100332. https://doi.org/10.1016/j.imu.2020.100332
  28. Sagar, K., & Saha, A. (2017). A systematic review of software usability studies. International Journal of Information Technology. https://doi.org/10.1007/s41870-017-0048-1
  29. Seffah, A., Donyaee, M., Kline, R. B., & Padda, H. K. (2006). Usability measurement and metrics: A consolidated model. Software Quality Journal, 14(2), 159–178. https://doi.org/10.1007/s11219-006-7600-8
  30. Tristantia, A. D. (2018). Evaluasi Sistem Pelaporan Insiden Keselamatan Pasien Di Rumah Sakit. Jurnal Administrasi Kesehatan Indonesia, 6(2), 83. https://doi.org/10.20473/jaki.v6i2.2018.83-94
  31. Weichbroth, P. (2018). Usability attributes revisited: A time-framed knowledge map. 1005–1008. https://doi.org/10.15439/2018F137
  32. Wronikowska, M. W., Malycha, J., Morgan, L. J., Westgate, V., Petrinic, T., Young, J. D., & Watkinson, P. J. (2021). Systematic review of applied usability metrics within usability evaluation methods for hospital electronic healthcare record systems: Metrics and Evaluation Methods for eHealth Systems. Journal of Evaluation in Clinical Practice, 27(6), 1403–1416. https://doi.org/10.1111/jep.13582
  33. Yuniarto, D., Subiyakto, A., Abd. Rohman, A. B., & Marliana, R. R. (2019). Assessment of Readiness and Usability of Information Systems Use. Jurnal Online Informatika, 4(1), 1. https://doi.org/10.15575/join.v4i1.256
  34. Zahiri Esfahani, M., Khajouei, R., & Baneshi, M. R. (2018). Augmentation of the think aloud method with users’ perspectives for the selection of a picture archiving and communication system. Journal of Biomedical Informatics, 80, 43–51. https://doi.org/10.1016/j.jbi.2018.02.017