Improving the CVR of SmartAsset’s Matching Service by 95% through insight-driven A/B testing
Project Timeline: 1+ years
My Role: UX Researcher & Front-End Developer
Team: Product Manager, CRO Specialist, and Product Designers
Tools: SurveyMonkey, Google Surveys, UserTesting, Optimizely, Figma
Background
SmartAsset is the leading marketplace that connects consumers with financial advisors. Before 2021, our 25-30 panel questionnaire (aka "the funnel") attracted over 30,000 consumers each month, but had a high drop-off rate with a CVR of 2.9%. The CRO team was tasked with improving the CVR, targeting both usability and user experience.
Goal: Improve CVR by 100% through ongoing A/B testing of SmartAsset’s Financial Advisor Matching Service
Research Process
As lead researcher, I conducted comprehensive research to identify user pain points and opportunities for optimization:
1. User Research:
A 12-question email survey to validated consumers (n = 422) identified two key user groups:
Pre-Retirees: Primary audience qualified for financial advisor matching
Getting My Bearings: Secondary group with unmet needs, indicating a potential product opportunity
Deliverables: Personas, journey maps, and insights into user goals and challenges
Primary: The Pre-Retiree
Secondary: Getting My Bearings
2. Usability Testing:
Remote task-based testing (n = 20) revealed usability issues including (see image below):
Long loading screens lasting up to 12 seconds without any feedback or cues (1)
Inconsistent CTA placements, often below the fold on mobile (2)
Small clickable areas, resulting in slower user progress (3)
Inappropriate keyboard triggers for text inputs (4)
Deliverables: Prioritized list of UX/UI issues (annotated screen below)
Optimization Process
I owned the process of creating and implementing A/B testing concepts in Optimizely using HTML, CSS, and JS. Insights from research informed these concepts, leading to measurable improvements. Examples of key wins are highlighted below:
Loading Screen: Reduced duration by 50%, increasing CVR by 7%.
CTA Positioning: Standardized CTA placement, boosting CVR by 3%.
Button Design: Enlarged clickable areas for better usability, raising CVR by 2.8%.
Default Keyboard: Optimized keyboard triggers, achieving a 1% CVR lift.
Outcome
Success! We achieved a 95% increase in CVR (new baseline: 5.5%) through research-driven optimizations in 2021. Insights shaped the 2022 and beyond roadmap, including new question types, branching, improved matching algorithms, and an experience tailored for secondary users.
2023 Update: Continued funnel improvements yielded a 4.68% CVR lift in Q1, equating to an estimated $585k monthly revenue gain. Key initiatives included:
UI Accessibility: Expanded clickable areas, visual feedback for CTAs, and accessible color updates.
Expectation Management: Reordered questions for a more logical and user-friendly flow.
Learnings
Thorough research is essential for driving meaningful improvements in A/B testing.
Fundamental UX/UI best practices (e.g., button positioning, accessibility, and page speed) can significantly enhance user experience and conversion rates.