← All projects

A/B Test Experiment Plan

Insight
Timer

Can gamification increase young adults' use of a meditation and mindfulness app, and improve their mood along with it?

Title slide: Insight Timer Experiment Plan Presentation video to be added
Group presentation of the experiment plan.
Role
UX Researcher
Team
Group of 4
Duration
Spring 2022

Collaborators

Adaeze Ezekwerre · Annie Doctrow · Lesley Pace · Marissa Villano

INFO 615 — Dr. Jina Huh-Yoo

Goal

  • As consequences of the COVID-19 pandemic linger on, increase young adult utilization of the meditation and mindfulness app Insight Timer by introducing gamification

The Problem

As a result of the COVID-19 pandemic, the cost to protect physical health resulted in declined mental health. Consequences such as isolation, job loss, school closures, lack of childcare, restricted access to mental health care, and greater exposure likelihood put young adults at a higher risk of facing mental health issues during the pandemic (Panchal, et al 2021). Already at a higher risk of poor mental health even prior to the pandemic, Panchal et al write that fifty-six percent of young adults reported symptoms of anxiety and or depressive disorder during the pandemic (2021).

To help combat these symptoms, virtual meditation and mindfulness resources are gaining popularity as free or low-cost solutions. A longitudinal observational study by Stecher et al that examined app-based meditation habits and the associated mental health benefits found that routine use of the app allowed for users to form reflexive meditation habits and that the users associated their routine use with greater perceived mental health benefits (2021).

Furthermore, with the key being routine use of the app, a systematic review of 1,017 intervention studies targeting engagement of adult web and or mobile users suggests that "gamification can increase engagement in online programs and enhance related outcomes, such as learning and possibly health behavior" (Looyestyn 2017).

User Personas

Two user personas side by side. Christina, 24, a full-time paralegal who rents an apartment in the city with a friend, is scheduled to work business hours but is often held late for demanding cases and clients, uses technology extensively, is newly experiencing symptoms of anxiety due to COVID-19 exposure likelihood at work, and wants to exercise mindfulness more often to ease anxiety rather than only at peak symptoms. Her quote reads, I don't know if I can make it, I have to work late now. Alex, 20, a full-time student who lives on campus in a single dorm room, has a flexible schedule given a hybrid course system with no set study hours, uses technology extensively, has a history of depression but is experiencing heightened symptoms due to isolation, and wants to develop meditation exercises into a routine without feeling like a chore. His quote reads, Going to pass for today, but maybe next time?
Christina and Alex, the two young adult personas guiding the experiment.

Hypothesis

  • If we apply gamification to Insight Timer, then the added engagement will drive user utilization of the app, which will positively impact the user session daily frequency.
  • Additionally, if the user session daily frequency increases over time, we expect to see our secondary metric, daily mood, to reflect general improvement over the same period of time.

Methodology

An A/B test comparing the current app against a gamified version of it.

Test Cells

Experiment type
A/B test
Test Cell A
The current version of Insight Timer
Test Cell B
The current version of Insight Timer with gamification

Metrics

Primary
Session daily frequency
Secondary
  • Daily mood tracking feature
  • Demographics

Data Collection & Analysis

Duration
Longitudinal data collected over the course of 6 months
Exposure
10% of users will receive the added gamification
Sources
  • Quantitative data collected from unmoderated user logs
  • Quantitative and qualitative data collected through self-reporting
Analysis
An independent t-test will be used to analyze the data

Recruitment

Test cells are randomly selected so they represent the portion of the user base the experiment sets out to learn about.

Segmentation

Age
Young adults, defined as individuals between 18 and 29
Location
The United States
Rationale
This age group is expected to use technology similarly and to share common values and expectations
Behavior
Not segmented, given the exploratory and global nature of the experiment
Experience level
All levels included: new users, existing users and current routine users

Sample Size

Share of users
10%
Why this size
  • Gamification is not required to keep using the app, so there is no urgency to roll the feature out more widely
  • Mental health varies widely from person to person, and only scale separates a real trend in the secondary metric from coincidence

Study Window

Period
June 2022 through December 2022
Export cadence
Daily user data exported weekly
Week definition
Monday at 12:00AM to Sunday at 11:59PM local time, for a clean cut-off
Why six months
  • Enough time for a habit to form and also to break
  • Long enough to surface seasonal effects, such as Seasonal Affective Disorder in winter months driving users to seek more mindfulness or meditation

Potential Failures

Two ways the experiment could return a result we would not want to act on.

  • Gamification may not have the desired impact and could result in the opposite effect, leading to change aversion, loss of results and usage decline
  • The project may result in novelty effect, which could indicate that the feature can only have temporary success

Lessons Learned

Five takeaways from designing the experiment, each one a change we would make running it again.

  • More design cells

    A wider set of cells would give a full view of impact per demographic breakout.

  • Longer test windows

    More time on test would account for change aversion and the novelty effect.

  • Secondary research

    It builds the proof of concept and the reasoning for pursuing a hypothesis.

  • Methods beyond gamification

    Evaluating other approaches widens the scope of what could drive usage.

  • Metrics that fit

    Metrics must match the hypothesis, and measuring impact can take unconventional forms.

Conclusion

Future Goals

  • Testing on additional age groups
  • Testing sustainability

Contributions

  • Applying gamification to a mindfulness app to drive increased engagement and routine use
  • A potential solution for improving young adults' mental health during the COVID-19 pandemic and beyond

Experiment Plan

The full plan submitted for the course, covering the data collection plan, analysis and results framework in detail.

Insight Timer Experiment Plan Abstract, introduction, data collection plan, analysis and expected results. Download .docx →