Summative Evaluation Usability in Medical Device

Introduction

Medical practice is increasingly utilizing medical devices for patient observation and treatment. Errors resulting from the inadequate usability of these devices have become a growing concern. Many medical devices are designed without sufficient attention to ease of use, making them difficult to operate, particularly for patients and non-experts. As healthcare evolves, a wider range of individuals with varying levels of expertise are using these devices, further complicating the issue.

Making medical devices easy to use isn’t just about making them work; it’s about designing them in a way that anyone can understand and use without struggle. It’s like building a car with simple controls so that anyone can drive it safely, even if they’re not a mechanic.

For this concern regarding Usability Engineering (HUMAN FACTORS ENGINEERING), of medical device manufacturers have to evaluate the design parameters of the device to ensure easy use of the device by the user end.

IEC 62366-1 outlines a usability engineering process aimed at ensuring the usability-related risks of medical devices are acceptable. This standard is intended to assist medical device manufacturers. Through the usability process, manufacturers identify use scenarios that could result in hazardous situations or harm.

Usability is a crucial factor in the design of medical devices, closely linked with safety, labeling, and risk management. During the design and development stages, manufacturers conduct formative evaluations of the medical device, followed by summative evaluations after the design and development phases are completed.

• Formative Evaluation: User interface evaluation is performed to identify design strengths and weaknesses, as well as to uncover any unexpected usage errors.
• Summative Evaluation: Evaluating the user interface at the end of development aims to provide objective evidence that it can be used safely.

Summative evaluation is typically conducted for specific, hazard-related use scenarios. Depending on the product and its various use cases, this can involve a single test event or multiple tests. If, during the summative evaluation, it is found that certain risk control measures are insufficient to mitigate specific risks, the usability process must be revisited, and formative evaluation should be repeated.

What Is Summative Evaluation?

Summative evaluation, as specified by IEC 62366, is a methodology used to assess the usability of medical devices. It aims to identify and minimize user-related errors and associated risks by simulating actual use scenarios with the intended user population.

Summative user testing offers an indirect assessment of a design’s usability. Testers are assigned tasks, and their performance is used to measure the usability of the design elements. This process focuses more on evaluating the design than improving it. Summative evaluation is also known as quantitative user testing.

Summative user testing is performed after the product is launched on the market. It involves a group of more than ten testers, and the data collected focuses on the quantity rather than the quality of the design, such as how many users can use the device without encountering any obstacles.

What Questions Do They Answer?

Summative or quantitative testing addresses questions like “How many?” or “How much?” Quantitative data, often in the form of metrics, answers questions such as “What are the task completion rates?” or “What are the task times required?” to determine if the tasks were easy to perform.

Summative user testing involves engaging a large group of users to gather data that assesses the product’s design usability. This type of study enables the development team to determine the percentage of users who encountered difficulties with specific actions on the platform. The statistical insights obtained from this process guide the redesign of the platform to better meet user needs.

When Is It Performed?

When the product is already completed and launched in the market.

What Is The Aim Of The Manufacturer?

Summative user testing allows the manufacturer to determine if the design elements validated during formative user testing are effective for a broader audience. The quantitative data collected offers statistically significant insights into how well the design works for the target user base.

How Is The Redesign Process Started After A Summative Evaluation?

A redesign process begins with evaluating the existing design, which is why Summative testing is the first step. This stage involves assessing design elements and collecting data to identify user difficulties. In the second step, qualitative testing is conducted to determine new design elements for the platform. After redesigning the platform, both tests are repeated to ensure the new design is effective. This approach clarifies the roles of Summative and Formative testing: Summative testing gathers extensive data to evaluate the design, while Formative testing quickly collects data during the design process.

Why Perform A Summative Evaluation?

• This method provides empirical, reliable data, making it suitable for testing hypotheses.
• The core usability metrics—effectiveness, efficiency, and user satisfaction—can be evaluated.
• Additionally, it allows for the detection of more complex usability flaws that less formalized methods might overlook.
• A properly conducted summative test can simulate the real use of a product.
• It can be used to support marketing claims with empirical evidence.

References

1. EVS-EN 62366-1:2015