Abstract

Background: Online health interventions are becoming increasingly frequent. However, to prove effective and satisfy the specific needs of cancer patients, the standardized steps of development are crucial. This includes structured usability testing to identify potential usability issues in the patient-specific context early during the development process of a new program.

Methods: Usability of a newly developed online stress management program was prospectively assessed in patients with solid tumors undergoing systemic treatment. In an academic computer-lab facility, each patient was asked to fulfill 16 tasks, which covered key components of the program including website navigation, login-in to secure area, filling-in forms, accessing audio files, and contacting the trial team. Usability problems during these tasks were identified via the think-aloud method and video recording and categorized. General usability was tested with the System Usability Scale (SUS).

Results: A total of 165 tasks from 11 patients were analyzed. Overall usability was high (mean System Usability Scale score 83.6) exceeding the pre-defined cut-off of 70. Participants solved 97% (160/165) of all tasks, the majority (76%) independently. A total of 122 specific usability problems were identified, predominantly concerning website functionality (50.8%) and navigation (29.5%).

Conclusions: Structured usability testing of a novel online intervention in the target population of cancer patients allowed for identification and subsequent correction of a significant number of usability problems. This crucial step allowed for a patient-friendly, self-explanatory online program with enhanced user-specific functionality, navigation and terminology before embarking on the subsequent randomized trial.

Keywords

Cancer, internet-based, online, healthcare, usability, technical implications

Introduction

The use of internet-based health care interventions is growing rapidly enabling certain aspects of mental health care to be delivered to the patient without the need for face-to-face interactions. Internet-based cognitive behavioral therapy for common mental health problems such as anxiety disorders and depression can provide effective, acceptable and practical health care for those who otherwise might remain untreated [1]. Internet interventions can also fill an important gap in cancer care. Cancer patients and their caregivers frequently use the Internet as a source of information [2, 3] and appropriately designed online tools can augment and increase the availability of psychosocial care by making participation convenient, confidential and less stigmatizing [2, 4]. Nevertheless, problems with high dropout rates [5, 6] and low level of engagement have been reported with some internet interventions [7]. The usability of an internet intervention is a key aspect that determines whether it will be used by the patient or not [7]. The few existing guidelines stress the importance of conducting formalized usability testing of internet-based health care interventions in the target population, hereby assessing whether the end user can work with the webpage during specific tasks [2]. Usability is defined as ‘‘the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use’ (ISO 9241-11) [8]. In formalized usability testing the observed usability problems are grouped to identify flaws within the system, ultimately leading to design improvements that remove these barriers [9].

Aim of our study

Usability testing was conducted as part of the development process of the web-based stress management program for newly diagnosed cancer patients undergoing treatment “STREAM” (STRess Aktiv Mindern; Active Stress Reduction). The aim was to improve the final website (www.stress-aktiv-mindern.ch) specifically for use by cancer patients in a subsequent randomized trial. Here we describe the usability testing process, and identify key aspects of online intervention tools that are relevant for the development process of other online interventions for cancer patients.

Patients and Methods

Cancer patients (Table 1) who were undergoing systemic anti-cancer treatment at the Medical Oncology outpatient department of the University Hospital Basel were invited to participate in this study. The usability trial was conducted at the computer laboratory of the Center of Human-Computer Interaction of the Department of Psychology at the University of Basel. The Ethics Committee northwest/central Switzerland (EKNZ) approved the study and informed consent was obtained from all participant.

Table 1. Information on socio-demographics, medical history, internet skills and usage

^a) Patients might undergo more than one treatment
^b) 0 = several times per month, 1 = once a week, 2 = several times per week, 3 = daily

Patients first completed a pre-test questionnaire that assessed socio-demographic data, medical history, and computer skills. Patients then executed 16 tasks (for an overview see Table 2) on the website using the ‘think-aloud’ method. This method encourages patients to think aloud while solving a problem, thereby giving observers an insight into the participant’s cognitive processes. A task designed to familiarize patients with the think-aloud method was also included. The 16 tasks covered the most important steps within the public area of the website (including the website overview, registration, and login function) and included a sample module of the secured area of the website that covered website navigation, filling-in forms, use of audio files, and contacting the trial team. Literature suggests that the majority of usability problems and flaws can be identified with as few as eight to ten subjects [9]. Overall usability was assessed with the validated System Usability Scale (SUS) questionnaire [10]. All usability tasks were videotaped and the recordings were used to assess usability. A coding manual for the analyses of behavior and performance was created by consensual expert judgment and later applied by these experts to each participant and task.

Table 2. Overview of usability problems and implications

^a Classification of problem severity: (I) Major problems that have a large impact on the user’s interaction and are experienced by many users = Immediate changes needed; (II) Medium problems experienced by only a few users but with a large impact on the user interaction or experienced by many users but with a small impact on the user interaction = Should be changed

Effectiveness was measured by task success and characterized by the degree of help needed (“some help” and “a lot of help”). Problems were categorized in terms of terminology, navigation, content, functionality, and ‘others’. The severity of each specific usability problem was rated by a usability expert based on the impact each problem had on the user [9]. Major problems were defined as those that had a large impact on the user’s interaction such as creating significant delay and frustration or had an impact on a persons’ workflow and were experienced by many users. Medium problems were those experienced by only a few users that had a large impact on the user interaction, or those experienced by many users but with a small impact on the user interaction. Efficacy was assessed by measuring the time-on-task and the time for navigating to the right place for task completion. Self-reported data concerning satisfaction with the STREAM tool were collected using a Likert Scale (1–6) and after every task.

Results

Data from 11 participants (Table 1) who solved 165 tasks (Table 2) were analyzed. Data analyses according to pre-specified age groups (<65/ ≥65 years) did not reveal any significant differences (data not shown).

Overall usability

The mean SUS score was 83.6 indicating that the overall usability of the STREAM web-based stress management program clearly exceeded the pre-defined cut-off for good overall usability of 70 [11].

Effectiveness and efficacy

Participants solved 97% (160/165) of all tasks (Table 2). Thereof, 76% (121) tasks were solved independently, 16% (26) with some help, and 8% (13) with a lot of help. The mean time spent on tasks was 39 minutes 47 seconds (SD: 78: 03; range 26: 13–64: 47 minutes).

Specific usability problems

A total of 122 specific usability problems were identified (Table 2). These predominantly concerned website functionality (50.8%) and navigation (29.5%).

Satisfaction

Participants indicated they were satisfied with the platform with an overall rating of 4.91 (on a scale 1–6). They described the intervention as clear, structured, and professional. Moreover, 73% (8/11) of the participants indicated that they would continue to use the program themselves and all participants stated they would recommend the platform to other cancer patient.

Discussion and implications

Our results show that structured usability testing with the target population is an important step during the standardized development of online health interventions. Our online stress management program STREAM is aimed at cancer patients who are undergoing active treatment. The overall usability of the STREAM website was rated as good and well above the pre-defined cut-off for usability; however, our analysis identified 122 specific usability problems.

A multidisciplinary team consisting of an oncologist, psychologists, human-computer interaction researchers, and software engineering specialists analyzed and subsequently solved these problems. The solutions to these problems were all relatively straightforward. Therefore, the crucial step is to first identify the problems, and this is greatly facilitated by evaluating the usability of the tool by the target patient population. Interestingly, usability in terms of solving tasks independently (effectiveness), the time spent on tasks (efficacy), and user satisfaction did not differ between young (<65 years) and older (≥65 years) patients. The likely explanation for this is that participants in both age groups had a similar frequency and duration of Internet use (Table 1). The specific usability problems identified in this analysis allow some general recommendations: First, it is essential to introduce simple but specific wording and use it consistently throughout the program. Second, users should be able to view the entire page without using the scroll function. To enable this, text should be concise and written in simple to understand language. Third, the intuitive use of a webpage is essential and this will solve the majority of minor usability problems (Table 2). Finally, a close collaboration with the software engineering specialist is extremely important to find good and affordable implementation solutions. A limitation of this study is that the testing was done in the laboratory and may not reflect the use of the program at home. If problems occurred during the use of the online program, participants were able to ask for assistance. Second, the small sample size may also limit the generalizability of our results. However, it is important to note that usability tests are qualitative methods that aim to reveal the most important issues that may arise during a patient’s interaction with a webpage.

In conclusion, our study highlights the importance of conducting a professional usability test with the target population during the development of an online intervention, as recommended by current guidelines [2]. This preparative step allowed for identifying several important but easy to resolve usability problems by integrating the end user (cancer patients) with the development of the STREAM online program. It influenced the development process and enabled us to implement a revised version of this tool prior to launching the randomized controlled trial (clinicaltrials.gov NCT02289014) assessing the efficacy and feasibility [12, 13] of the STREAM tool for newly diagnosed cancer patients.

Authorship

Grossert A: Conceptualization, Data Curation, Formal Analysis, Investigation, Methodology, Validation, Visualization, Writing –Original Draft Preparation, Writing – Review & Editing

Heinz S: Conceptualization, Data Curation, Formal Analysis, Investigation, Methodology, Validation, Visualization, Writing –Original Draft Preparation, Writing – Review & Editing

Müller L: Data Curation, Formal Analysis, Investigation, Writing – Review & Editing

Gaab J: Writing – Review & Editing

Urech C: Writing – Review & Editing, Financial support

Berger Th: Writing – Original Draft Preparation, Writing – Review & Editing

Hess V: Conceptualization, Formal Analysis, Methodology, Validation, Writing – Original Draft Preparation, Writing – Review & Editing, Financial support

Acknowledgements

This study was supported by the Swiss National Science Foundation (PP00P3_139155/1 to VH; PP00P1_144824 to TB) and Swiss Cancer Research (KFS-3260-08-2013). We thank Sebastian Westhues and Laurin Stoll of YooApplications AG Basel for their innovative software solutions. We thank Jamie Ashman of Prism Ideas for language editing of the manuscript. We also thank the patients and their families for participating in this study.

Abbreviations

STREAM   STRess Aktiv Mindern; Active Stress Reduction

EKNZ   Ethics Committee northwest/central Switzerland

SUS   System Usability Scale

Competing interests

The authors declare that they have no conflicts of interest.

Funding Information

The Swiss National Science Foundation (PP00P3_139155/1 to VH; PP00P1_144824 to TB) and Swiss Cancer Research (KFS-3260-08-2013) supported this study.

References

1. Andrews G, Cuijpers P, Craske MG, McEvoy P et al. (2010) Computer therapy for the anxiety and depressive disorders is effective, acceptable and practical health care: a meta-analysis. PloS one. 5(10): e1319610.1371/journal.pone.0013196. [Crossref]
2. Leykin Y, Thekdi SM, Shumay DM, Munoz RF et al. (2012) Internet interventions for improving psychological well-being in psycho-oncology: review and recommendations. Psycho-oncology. 21(9): 1016-102510.1002/pon.1993. [Crossref]
3. van de Poll-Franse LV, van Eenbergen MC. (2008) Internet use by cancer survivors: current use and future wishes. Supportive care in cancer: official journal of the Multinational Association of Supportive Care in Cancer. 16(10): 1189–119510.1007/s00520-008-0419-z. [Crossref]
4. Owen JE, Klapow JC, Roth DL, Nabell L et al. (2004) Improving the effectiveness of adjuvant psychological treatment for women with breast cancer: the feasibility of providing online support. Psycho-oncology. 13(4): 281–29210.1002/pon.733. [Crossref]
5. David N, Schlenker P, Prudlo U, Larbig W (2013) Internet-based program for coping with cancer: a randomized controlled trial with hematologic cancer patients. Psycho-oncology. 22(5): 1064-107210.1002/pon.3104. [Crossref]
6. Carpenter KM, Stoner SA, Schmitz K, McGregor BA et al (2012) An online stress management workbook for breast cancer. Journal of behavioral medicine. 10.1007/s10865-012-9481-6. [Crossref]
7. Gorlick A, Bantum EO, Owen JE. (2014) Internet-based interventions for cancer-related distress: exploring the experiences of those whose needs are not met. Psycho-oncology. 23(4): 452-45810.1002/pon.3443. [Crossref]
8. ISO 9241-11, Ergonomic requirements for office work with visual display terminals (VDTs), Part 11: Guidance on usability, vol. 2017. Geneva, Switzerland: International Standardization Organization (ISO); 1998.
9. Tullis T, Albert B (2013) Measuring the User Experience. Collecting, Analyzing, and Presenting Usability Metrics. Waltham, USA: Morgan Kaufmann;
10. Kortum PT, Bangor A (2013) Usability Ratings for Everyday Products Measured With the System Usability Scale. International Journal of Human–Computer Interaction. 29(2): 67-7610.1080/10447318.2012.681221.
11. Measuring Usability with the System Usability Scale (SUS) http: //www.measuringu.com/sus.php. 2017/1/11.
12. Urech C, Grossert A, Alder J, Scherer S et al. (2018) Web-Based Stress Management for Newly Diagnosed Patients With Cancer (STREAM): A Randomized, Wait-List Controlled Intervention Study. Journal of clinical oncology: official journal of the American Society of Clinical Oncology. 36(8): 780-78810.1200/JCO.2017.74.8491. [Crossref]
13. Grossert A, Urech C, Alder J, Gaab J et al. (2016) Web-based stress management for newly diagnosed cancer patients (STREAM-1): a randomized, wait-list controlled intervention study. BMC cancer. 16(1): 83810.1186/s12885-016-2866-0. [Crossref]