As a designer transitioning from industrial design to user experience design, I focus on human-centered design concepts to address contemporary social issues. My expertise includes UI/UX, front-end development, and industrial product design. I prioritize logic and innovation in design process.
“This study investigates how different chatbot explanation modes (textual, rules, and mixed) impact trust, comprehension, and mental effort in healthcare. Overall, the mixed mode was found to balance clarity and engagement, optimizing user experience for non-experts.”
Healthcare chatbots aim to tackle limited resources, rising chronic diseases, and unequal access to care. However, they often fail to process complex information and provide clear explanations for non-experts, causing misunderstandings and reduced trust. Therefore, it is essential to evaluate and optimize explanation modes – textual, rules, and mixed – to improve user trust and comprehension while reducing cognitive load. Enhancing these modes is key to making chatbots more reliable, user-friendly, and effective for non-experts managing their health.
This study aims to design and evaluate suitable chatbot explanation modes for non-experts. Three different modes are proposed, namely Textual mode, Rules mode and Mixed mode, to evaluate the impact of different explanation formats on users’ trust, comprehension and mental effort.
The research plan follows the Design Thinking methodology, combaining user-centered design principles to create and test healthcare chatbot prototypes. It begins with understanding user needs through participatory design which concducted a focus group to defines requirements and create three chatbot prototypes. Formative testing and pilot studies refine the main features of prototypes, followed by final evaluation to figure out the answer, ensuring they meet user needs effectively.
Focus Group
Participants provided key feedback on healthcare chatbots, highlighting areas for improvement. Trustworthiness is crucial; current outputs are too complex, and clear use of specialized terminology is needed. Guidance and topic direction are essential for understanding user descriptions. Users prefer personalized responses over generic ones and expect chatbots to show empathy to enhance comfort. Additionally, they desire instant feedback and integration of diverse information. Past negative experiences, such as incorrect or repetitive information, have affected trust. Overall, chatbots should focus on clear output, professional guidance, emotional support, and immediate feedback to meet user needs.
Inspired by cognitive load theory, I ideated solutions to minimize users’ mental effort. I recognized the potential of rule-based learning to improve decision-making and align with user preferences for structured information. Besides, information transparency is also important.
The Main Features in 3 Explanation Modes
Textual for a natural conversational flow basically using NLP mode, Rules for structured guidance always showed an “IF-Then” format in the explanation, while Mixed combines the both, it can give a natural reply with emotion and logical structure like “If-Then” as well.
So in the end, I opted for prototype version 2, applying the Wizard of Oz approach to the experiment.
In the evaluation methodology, I used a within-subject experiment design, where 28 participants interacted with three different chatbot prototypes: Mixed, Rules and Textual Mode. The participants were asked to complete three tasks with each prototype, which involved easy and difficult tasks.
Following the tasks, participants filled out four questionnaires, assessing Mental Effort, Perceived Trust and Perceived Usefulness, and Comprehension. The interaction was assisted by a wizard behind the scenes to ensure the dialogue followed the designed map. This setup allowed us to closely monitor how each mode influenced the users’ experience and gather data for analysis.
The evaluation used both quantitative and qualitative methods. Quantitatively, a one-way within-subjects design compared three prototypes. Descriptive analysis included mean scores for Perceived Trust and Mental Effort, median scores for Perceived Usefulness, and contingency tables for Comprehension. Normality was tested using the Shapiro-Wilk test, followed by parametric (ANOVA) or non-parametric tests (Friedman’s, Wilcoxon) based on p-values. Qualitatively, semi-structured interviews were conducted using an inductive, experiential approach in six steps to understand user interactions.
Quantitative Results: The mixed mode outperformed both textual and rules modes in Perceived Trust and Mental Effort, indicated by lower mean scores and statistical significance. For Comprehension, the Rules mode was most effective, particularly for those with lower numerical skills, though no significant differences were found.
Qualitative Analysis: Thematic analysis revealed four key themes: Users prefer simple, clear explanations for effective information retrieval. The mixed mode, blending structured logic and human interaction, was favored but can still cause information overload. Trust increased with clear, consistent explanations, and emotional engagement further boosted trust. Structured formats like ‘if-then’ statements helped reduce mental effort.
The mixed mode ranked highest in trust due to its combination of sentiment analysis with structured statements, enhancing user confidence. For comprehension, the rules mode proved most effective, particularly for individuals with lower numerical skills, aligning with human learning theory. Regarding mental effort, the mixed mode was again preferred, as it reduced cognitive load by incorporating multiple text formats. Overall, the mixed mode offers a balanced approach between trust and mental effort, while the rules mode excels in enhancing comprehension.
Theoretical: The study supports cognitive load theory, showing that mixed mode improves trust and comprehension with less effort, and challenges the effectiveness of traditional textual modes.
For UX Designers: Prioritize mixed mode for complex tasks, balance emotional support with structured logic, and offer user control to enhance the experience.
Practical Use: The findings can guide the design of chatbots in telemedicine, health education, and assistive technologies for a more user-friendly experience.
