UX Research Project –

AR Social Chatbot Design: The Effects of Captions Positions on Users’ Emotional Engagement & Social Presence

This study examined how caption positioning in AR social chatbots impacts emotional engagement and social presence. Using a design thinking approach, 2 AR prototypes (dynamic vs. fixed caption position) were tested with 21 participants through A/B testing and interviews. While emotional engagement showed no significant change, the dynamic caption position enhanced users’ sense of presence, particularly social awareness. The research fills gaps in AR design, offering insights and design recommendations for optimising captions in AR chatbots.

📚 The Context

Research indicates that social relationships are crucial to happiness (Diener and Seligman, 2002), and friendships play an important role in psychological well-being (Erikson, 1959). However, a study by the Department for Culture, Media & Sport (2022) found that 79% of people in the UK have experienced loneliness, highlighting the need for stronger social connections. Virtual friendships in social chatbots have been proposed as a solution to this problem (Søraker, 2012) and AR, as a low-threshold information device (Almaiah et al., 2023), could be a suitable technology.

🧐 Research Questions

What is the effect of the presence of captions and their positions in AR social chatbot designs on users’…?
1. Emotional engagement
2. Social presence

📌 The Objectives

Design & A/B test 2 prototypes
Deliver design suggestions
Provide new knowledge to help create better UX for building social relationships in AR

Prototype Development

The prototype was created using the Design Thinking framework.

👀 Empathy & Define: Literature Reviews

I started the empathy and define stages of the design thinking process mainly through literature reviews and identified 3 key insights, which suggest the importance of improving UX of AR social chatbots:

The direction was also informed by similar research in VR and an existing product called ‘Replika’.

💡 Insight 1

AR could be effective in improving social presence.

–

💡 Insight 2

While AR captions add to the experience, they can also affect usability in specific contexts.

–

💡 Insight 3

There is a lack of knowledge of caption positions in the context of mobile AR.

–

🧠 Ideation Session: Workshops & Focus Groups

The co-ideation workshop was held to design the UIs of the AR chatbot.

6 participants were given various pre-defined UI elements and asked to use some or all of the elements to create a paper wireframe of the AR chatbot. A focus group was also conducted to gather more insights.

The participants ended up creating 2 interfaces: one was fixed on the bottom of the screen and the other one stuck to the head of the avatar while following the users’ eyes, which then became the wireframes of the designs for further A/B testing.

The paper UI elements used in the workshop

The second workshop, which included a card-sorting activity and a focus group, was held to understand the appropriate conversation styles/tones for the chatbot.

The same participants were given several sentences from 3 conversations designed based on existing studies. The participants were asked to sort the sentences by how they feel in terms of emotions (Positive/Neutral/Negative) and senses of realness (AI-like/Human like).

The sentences categorised in the neutral area were used to create the final conversation. Also, the participants suggested to change the topic of the conversation to ‘Food’ or ‘Travelling’ as they are easier to talk about.

👩🏻‍💻 Design: Prototyping & Formative Testing

With the findings from the ideation session, I revised the conversation flow and translated the wireframes to the hi-fi prototypes.

Two rounds of formative testing was conducted afterwards. The results confirmed that the two different conversation topics do not affect the experience and also informed some minor changes in the UI design.

The final prototypes are as below:

Prototype 1 – Dynamic Caption Position (DC)

Prototype 2 – Fixed Caption Position (FC)

Data Collection

The study used convenience sampling, recruiting 21 participants aged 18+ with basic English proficiency and no hearing or speaking disabilities. The testing session began with informed consent and a background questionnaire, followed by A/B testing of 2 AR prototypes and a semi-structured exit interview.

The measurements include:

The Positive and Negative Affect Schedule (PANAS-SF) (Watson et al., 1988): to evaluate participants’ positive and negative emotions
The Holistic Presence Questionnaire (HPQ) (Toet et al, 2021): to evaluate the holistic presence in AR settings
The Two Dimensions of the Social Presence Measure (Kreijns et al, 2020): to help evaluate social presence in AR from ‘awareness’ and ‘proximity’ dimensions.

Data Analysis

Two normality tests were conducted on the quantitative data to determine descriptive statistics and guide further analysis. Based on the results, a Wilcoxon Signed Rank Test was used for PANAS and social presence data, while a Paired Sample T-test was applied to holistic presence data. For the interview data, I performed an inductive thematic analysis with a critical orientation, using a constructionist approach to interpret underlying meanings and identify themes through latent coding.

Testing Results

Quantitative Data

❤️ Emotional Engagement (PANAS)

Positive Affect: No significant differences.
Negative Affect: No significant differences.

🌟 Holistic Presence

Prototype DC was slightly higher than FC.

👥 2 Dimensions of the Social Presence

Overall: No significant differences.
Awareness: Prototype DC was slightly higher than FC.
Proximity: No significant differences.

Prototype DC

Qualitative Data

5 themes and the effects of limitations were concluded from the exit interviews.

‘Users’ mental models’ and ‘Users’ behavioural tendencies’, are related to the research question regarding how the caption positions affect users’ sense of social presence. The 3 other themes: ‘Realness affected by the relatability of the avatar’, ‘The impact of captions’, and ‘Captions positions are worth considering when designing AR chatbot experiences’, can be used as guidance for designers.

**Orange: Key Themes / Blue: Code derived from quotes of the participants*

Key Design Suggestions

Mix-reality designers should not take it for granted to apply the same principles for VR to AR.
Designers may prioritise a dynamic caption position when designing AR social chatbots.
The dynamic captions be placed in a position that would not distract users from having eye contact with the avatar (e.g. being near the avatar’s eyes) to ensure social presence

Additional design recommendations

Captions should be available if users are having a long or Designers conversation with the chatbot.
Designers may add human-like body movements (like nodding or hand gestures) and modal particles to the AR avatar to enhance the realness.
Designers may allow the users to choose an accent that they are familiar with for the AR avatar.
Designers may provide users the freedom to display or hide the captions.
Designers may design an evolving change of tones in the AR’s speaking to make it more natural