Identifying E-Commerce Product Stay & Churn Reasons Using Mixed-method Research
Duration
Jan 2024 - Feb 2024
Methods
User Interviews, Jobs-to-be-done Framework, Survey, Co-design Workshop
Tools
Qualtrics, Calendly, Google Meet, Excel, Miro, Figjam
Overview
As the founding researcher at Koupon.ai, I established UX research practices from zero and continue to drive proactive and strategic research in the startup company which contributes to reduced risks and increased business profit. In this project, I worked with another researcher Yolanda, and led the end-to-end process from scoping to reporting, utilizing a mixed-methods approach involving interviews and surveys to understand why users adopt, stay or churn our product and collaborate with the cross-functional team to uplift the retention rate.
The Challenge
Koupon.ai offers a website and an app that connects users with Amazon deals and coupons to help them save money on their purchases. First launched in August 2023, Koupon.ai has attracted many new users to try the products and improved significantly through research and iterations. However, the team had no clue about what feature to build in the next step to keep the users engaged. After the initial fast growth during the holiday season, the team was concerned about the low retention rate of the acquired users.
The Process
0. Clarifying the context
The objective seemed clear - to find out what caused the low retention rate. However, as the lead researcher, I wanted to ensure the research goal was truly clear and aligned. Also working at a startup company with limited resources, I needed to prioritize the research that could address the most critical issue of the product. Therefore, when scoping the research, I had several follow-up conversations with the CEO, PM, and designers to understand the full context of the research need and I learned that:
-
Since the launch, we hadn't really understood who the actual users were and what user needs were met/unmet by Koupon.
-
No significant change was observed - The retention rate had been low since the product launch. This became a burning issue as the increase of new users had slowed down and retaining users became more important.
-
It was not just about the business - from previous research, we heard from some users who were frustrated using Koupon and were less interested in continuing using Koupon.
In alignment with the product goal to improve user retention, I reckoned that it was better to look into the problem from two sides: what made people come and stay, as well as what made people leave and churn.
.png)
Therefore, we decided to break the project into two phases. In phase one, we would focus on our power users who adopted our product and had been using it regularly for a while. And in phase two, we would move on to understanding what we failed to do for the churned users. We prioritized the power user study because it was a low-effort, high-impact attempt to keep our happy users and continue to attract other lookalike people.
While the team was hoping to receive user personas as the research artifact, we realized it was not the best option to achieve the research goal and present the insights; instead, I suggested using the Jobs-to-be-done framework to be focused on the underlying user needs and not be distracted or biased by the common peripheral information in personas, such as portrait images, demographics. etc.
Since the JTBD framework was novel to some team members, I did a short introduction of the framework in the weekly team meeting and shared reasons why using this framework would help us gain the most from the research, the team all agreed on this approach.
1. Phase One - Power User Study
To capture the rich context of power users' life situations, goals, needs, and pain points, we chose the good old qualitative method - 1:1 in-depth interviews.
Step 1 - User Interviews
We started by defining recruitment criteria, developing a screener survey, distributing the survey, and scheduling interviews. Our target number of participants was 5-6 and we intentionally scheduled 8 so in the case of 3 no-shows we were able to secure 5 interviews and have a good saturation at the end.
Step 2 - Analyze and Categorize Insights
Since Jobs-to-be-done is a defined and structured framework, I argued for a top-down approach to analyze the interview findings. We first broke down and transferred interview transcripts into sticky notes on Miro following the flow of interviews, and then utilized the Jobs-to-be-done "Froce Diagram" to categorize the stickies into Push, Pull, Anxieties, and Inertia to see which contributes to the user adoption of our product and which hindered that.
Categorization
Step 3 & 4 - Draft and Finalize the Jobs-to-be-done Statements
It seemed that the JTBD statements would just magically come out of the interview insights but that wasn't true. The main challenge we had was to determine how precise/general those jobs should be. On the extremely precise side, you can have a JTBD statement like When I look for deals on electronics, I want to only see those on the newest models so that I can save time and effort" where you have a very specific need under a very specific situation. On the contrary, you can have an extremely broad JTBD statement like "When I shop, I want to save money so I can have money for other things and have a better life" where it seems to be true for everyone and applicable to every product.
After a massive study on the Jobs-to-be-done theory, including books and case studies online, I realized there just wasn't a one-size-fits-all solution to this problem. Every business is different and we have different research goals from using the Jobs-to-be-done theory. Therefore, I led the focus back to our own objectives and our own product. And I set the criteria that the JTBD statements should be precise enough to be unique to our business and broad enough to uncover the root functional and emotional needs of our power users.
This rule effectively guided us to review and modify our JTBD statements to make them really useful to our product.
Step 5 - Prioritize the Jobs-to-be-done Statements
At this stage, we had over 20 Jobs-to-be-done items derived from the research. Not only would they be a huge information overload to our stakeholders but not all of them were of the same importance to our product. Therefore, we reviewed each JTBD statement based on two factors - salience to users and impact on the product, and we prioritized those that showed more salience to users from the interview findings and those we anticipated to have a higher and more immediate impact on the product. As a result, we went down from 23 JTBD statements to 4 high-level themes containing 14 JTBD statements that mattered most to users and the product at this moment.
So what? - Co-design Session
I created a FigJam board for the co-design session as it offered easier collaboration with designers who used Figam.
In my research philosophy, the value of research heavily depends on what stakeholders can take away from it and what they decide to do based on it. So in this project, not only did we create an easy-to-follow presentation, but we also brought the team straight into a 2-hour co-design session where we engaged the CEO, PM, and designers to share thoughts on research insights, and based on each JTBD statement we brainstormed.
By doing this, we came up with 14 feasible ideas that were added to the product roadmap, which would improve our product to grow and retain power users.
Now that we know why people liked us, we need to figure out why some don't.
2. Phase Two - Churned User Study
Step 1 - Archival Research
To gauge what might have made users churn and form our hypothesis before conducting the survey, I looked into the website intercept survey which I created earlier to regularly collect user feedback. In particular, I analyzed the qualitative answers to open-ended questions such as why they gave a certain NPS score and what they thought should be improved.
So later I formed four main hypotheses based on the emerging themes in the previous survey and team members' perceptions.
Step 2, 3, & 4 - Survey Design to Data Analysis
Since the objective of the survey was to understand churn reasons, we designed survey questions to cover why respondents stopped using Koupon as well as the contexts relevant to the churn such as frequency of Amazon shopping, Koupon usage, satisfaction with Koupon, etc.
Among the 69 received responses, I did data cleaning and removed 9 incomplete responses and 3 unthoughtful responses that showed evidence of being speeders, straightliners, and giving no/nonsense answers in the open-ended question. Exporting the raw data of 57 valid responses from Qualtrics to a spreadsheet, I then used descriptive statistics (e.g. frequency, mean, mode) to quantify user feedback and ran a thematic analysis on the qualitative data from the open-ended questions.
Step 5 - Synthesize and Report
From answering the hypotheses...
x
xx
To uncovering deeper insights...
I tapped into graphs to make the data more readable and compelling to stakeholders.
Product aspect 1
Product aspect 2
Product aspect 3
Product aspect 4
Product aspect 5
To display the data in a meaningful way, I used a heatmap to visualize the satisfaction scores and support the insight.
So what? - Product and Marketing Efforts to Reduce Churn
Based on the survey, we uncovered that:
The most critical churn reason by the survey respondents was that it was hard for them to find deals that interested them.
We took a holistic view of the survey results, further broke down this problem, and found its roots in four aspects:
-
flawed personalized recommendation
-
low product variety
-
unengaging first-time user experience
-
general usability issues
Working with UX designers and engineers, we are exploring the following solutions:
-
technically advance the recommendation system so users see more relevant deals
-
bring product variety to the next level by including more e-commerce sites besides Amazon
-
introducing more handholding flow for first-time users
-
plan continuous usability research to constantly iterate and improve the ease of use
An unexpected but truly enlighting finding from the survey was that:
Many survey respondents stopped using Koupon simply because they forgot. It not only reveals the big challenge of habit-forming for the product but also presents users' frustration on paying extra because they forgot to check deals on Koupon.
When sharing this finding with the team, I referred to the famous hook model from the book Hooked by Nir Eyal. Besides the product team, I particularly invited the marketing lead to together have a discussion on how we might provide triggers to keep users engaged in the habit-forming loops so that users remember to come back to Koupon and save on their purchases.
The Impact
Next steps: As the team continues to design and ship iterations and new features inspired by the research, the next step will be to conduct testing of key changes and to keep track of the results of these changes.
My Takeaways
1. Make stakeholders the hero
When I was working in the marketing field, we always said that we needed to make our customers the heroes of our brand story to build meaningful connections and make sure the brand message was well received. In this project, I found this rule was also true in research communication. By bringing my stakeholders to a co-creation session right after the research readout, I gave them the opportunity to share their amazing ideas and be the heroes. The implementation of research insights was so smooth as the stakeholders were excited to push their ideas into the roadmap.
2. No one-size-fits-all solution
3. Advocate for the appropriate research methods and proactively communicate with stakeholders on the whys