A personalized diagnostic product that helps SAT takers to reach target scores efficiently
Joint venture
Sanli, a leading private educational company in Shanghai, China
Product brief
A B2B product eco-system to conduct SAT placement tests with personalized analysis for students who struggle with starting to prepare for the SAT due to the lack of understanding of this test.
Duration
2019.10 - 2020.2
What makes this project unique?
This project not only helps students understand how to prepare for the SAT by informing their weaknesses but also collecting learning-related data to enrich their personal learning profile for better tracking their learning efficiency.
Deliveries
User Research & Interviews
Journey Mapping
Wireframing & User Testing
Screen Flows
Visual Design
Interaction Design
What is SAT?
SAT stands for Scholastic Assessment Test. It is a standardized test required to be taken by students seeking admission to undergraduate schools in the US.
EXECUTIVE SUMMARY: A 5-MINUTE BRIEFING
WHAT DOES KIDAPTIVE DO?
My company, Kidaptive, provides B2B service that combines and gives meaning to data from a variety of learning contexts with a cloud-based data management, assessment, and reporting backend engine.
Research paper shows the positive learning results for one of Kidaptive's clients.
WHAT ARE THE BUSINESS PROBLEMS?
Since the SAT prep cram school business has become a competitive market Our partner Sanli, one of the leading private educational companies in China, has two main problems want to solve via Kidaptive's strength.
PROBLEM 1
Offline education creates lots of discrete content and labor work for assessment and administration.
PROBLEM 2
Lacking strong proofs to convert potential customers (students) due to general instructions and materials which could be provided by other competitors as well.
WHO ARE THE USERS WE ARE HELPING?
To solve the business problems in an efficient way, I mapped out this project's primary users to understand their personal goals and things that stop them from achieving those goals.
WHAT ARE THE SOLUTIONS?
Diagnostic
A diagnostic that unifies different kinds of assessment questions with a template system to automatically sense a student's starting point with less manual work.
DIAGNOSTIC TEMPLATE SYSTEM
Create consistent displays for the 84 passages and 900+ questions in the centralize question management system and future updates with 8 templates in 3 different responsive sizes.
Before and after. Modularizing a raw question from the QMS with the template system.
ONLINE SUBMISSION & AUTO-GRADING
Students can take the diagnostic online when they are free to accommodate their busy school life. Also, free consultants from manual work via system auto-grading for submissions.
Submitting the reading section on the SAT diagnostic with a digital answer sheet.
Report portal
A centralize information station offers a consultant to easily access learning-related information from a single source of truth, enabling fast communication with parents and students with less discrepancy.
Bulk displays diagnostic results and related student's info to synchronize data into one place.
Personalized report
For consultants, a personalized report helps them better communicate with the students and get their buy-in with an in-depth analysis. For students, the report guides them to start the first step of test prep.
GOAL & SCAFFOLDING TIMELINE
This graph shows the delta between students' current positions and where they want to get to. Also, how many classes they could take to achieve the ultimate goal of scaffold and reduce overwhelming students.
Student's learning trajectory projecting with targets and class recommendations as scaffoldings.
STRENGTH & WEAKNESS ANALYSIS
This session prioritizes SAT topics for students to focus on and practice as a novice based on the importance of the topic and their mater level. The recommendations change based on students' performances.
An aggregate calculation of a student's sub score based on topics with focus recommendations.
WHAT ARE THE RESULTS?
At the time of this writing, we have deployed and launched a beta version for user testing and stakeholder review. In sum, we see good results from:
BUSINESS PERSPECTIVE
Kidaptive received $ 1M USD as the fund of a starting point of the joint venture from the stakeholder.
USER PERSPECTIVE
We received lots of positive feedback from students, consultants, and stakeholders during the preliminary stage and user testing.
“This is quite new. I haven’t seen this before. The quick improvement will be popular among students without time.”— Student
Student comments on the early build of strength & weakness feature in the personalized report
WHAT IS NEXT?
Once we get the green light from the partner, we are going to launch the product and track in depth data analysis and collect user feedback to examine the outcome and keep on iterating.
EVALUATE DATA WITH SUCCESS METRICS
Track the delta of new student's conversion rate, NPS, and the time span of a consultant for collecting student info.
DESIGN NEW FEATURES
Based on the feedback and roadmap to iterate on the assessment system with adaptive questions for efficient learning.
Now, If you have more time, let's dive deep into the detail journey of crafting this product.
This "behind the scenes" includes design decisions my team and I made to build a design over time that balances user & business needs. Feel free to reach out and learn more by shooting me a message. Enjoy!
FRAME THE DESIGN CHALLENGE
FIRST, SEE EYE TO EYE WITH STAKEHOLDERS
Aiming to solve both business problems and user's pain points, I facilitated my team to bring the stakeholders on to the same page by anchoring this project's vision with an HMW statement.
“How might we enable personalized and data-driven learning to alleviate traditional cram school SAT prep?”
KEY DRIVERS WITH THE REALITY
As the assessment and information are fragmented or paper-based in Sanli's current system, the ability to digitalize and centralize data is important to address the HMW statement. So,I based my designs to deliver upon the following key action items.
DIGITALIZE EXPERIENCE
A digitalize experience that unifies different kinds of assessment questions with layout templates, providing a single point of entrance to prepare for SAT.
PERSONALIZE RECOMMENDATION
A data analysis display that informs a student's weak spots and how to reach their goals with a personalized recommendation, allowing efficient preparation.
CENTRALIZE INFORMATION
Offering company staffs easy access to learning-related information, enabling fast communication with students & parents within different environments.
DEEP DIVE INTO THE PROBLEMS
HOW DOES THE CURRENT TEST PREP LOOK LIKE FROM A USER PERSPECTIVE?
I conducted four semi-structured interviews with students who prepare the SAT test in China and another six interviews with the consultants to understand the context of SAT preparation and test-taking experience in China.
“As a high school student, when I started my SAT prep, I participated in an intensive SAT prep program so that I can understand how to prepare SAT efficiently.”— Student
All four students we interviewed have engaged in SAT prep programs while preparing SAT
SYNTHESIZE WITH JOURNEY MAP
To consolidate the insights from the interview, I mapped out a journey map of students, parents, and consultants with various touch-points and the related tasks. This helped me to expose pain points and opportunities with various personas.
BALANCE USER NEEDS, BUSINESS GOALS, AND TECHNICAL CONSTRAINS
The journey map uncovered various areas of opportunities, such as creating an adaptive practice test to reduce students' time to spend. However, for this led-gem product, my PM and I kept in mind the business goals and technical constraints and decided to optimize the user experiences in the two stages: Conduct a diagnostic test and Discuss results and plans with a digital diagnostic and in-depth personalized report due to three main reasons.
Two stages that our team decided to focus on given the user pain points, business goals, and technical constrains.
FOR USERS
In the current experience, students and consultants have significant needs and pain points in the two stages.
FOR BUSINESS
The outcome of the Discussion results and plans stage is related to metrics such as key engagement, user conversion rate, and revenue.
DEAL WITH TECHNICAL CONSTRAINS
The current offline teaching system is lacking of data to be analyzed. There is a need to find a data input.
SET UP THE STRUCTURE
MAP THE DATA STRUCTURE
To consolidate solutions into an eco system, I collaborated with my PM and engineers to came up with a system architecture that focused on how the data should flow:
1. Collect data via diagnostic.
2. Analyze data with Kidaptive's backend engine.
3. Generate a personalized report.
4. Store student's report data via a report portal.
CREATE DETAIL USER FLOWS
With three rounds of iterations, we carved out the user flow that aligns with the data structure to narrow down the solutions. The revised and simplified user flow includes four main parts: Student diagnostic onboarding, Diagnostic test experience, Personalized report display, Report storage and review
User flow V1
FEEDBACK
Onboarding has too many nitty gritty questions and steps.
User flow V2
FEEDBACK
Account setting and email verification doesn't fit into the use case of the diagnostic (One time, staff-guided.)
User flow V3
FEEDBACK
Diagnostic & authorization needs to be separated into two applications.
DESIGN DECISION
Examine necessary questions and separate onboarding questions into two sets (before diagnostic and after diagnostic)
DESIGN DECISION
Authorize diagnostic with verification code inputting by consultants.
DESIGN DECISION
Move authorization feature to report portal to better distinguish the application usage between students and consultants.
The final user flow broken down into four main parts.
ITERATIONS ON SOLUTIONS
DECONSTRUCT EXISTING DIAGNOSTIC
To shift the paper-based SAT experience to a digital one. Me and my team inspected the current online test interface elements, try to understand how they work, and the intentions behind each of the feature. In sum, we found out the interface elements fit into two main categories: Content presentation and Test behavior
Deconstruct existing designs from different online diagnostic tools to categorize elements.
MODULAR CONTENT PRESENTATION WITH TEMPLATE SYSTEM
To display different kinds of content and questions coherently, I modular a collection of reusable templates, guided by clear standards with the following steps:
STEP 1: IDENTIFY QUESTION TYPES
Categorize questions into 13 different types from a paper based mock test and the questions management system.
STEP 2: DEFINE BASIC COMPONENTS
Extract common components from the different question types.
STEP 3: CREATE WIREFRAMES TO GET FEEDBACK
Design possible layouts to discuss with learning team and engineer team to receive feedback from different perspectives.
FINAL CONTENT PRESENTATION
The system includes eight templates with three different responsive sizes that can cover 84 passages and 900+ questions in our current QMS and more.
OPTIMIZE TEST BEHAVIORS DURING AND BEYOND DIAGNOSTIC
I sketched and brainstormed various possibilities with a senior designer to create mid-fidelity wireframes. The outcome was focus on Personal data input, Test navigation, and Submission.
IS THE DESIGN MAKING SENSE? VALIDATE WITH USABILITY TESTING
Given the constrains as a B2B product, the end users are hard to reach. Hence, I applied guerrilla usability testings with alternative users among the core interactions (Here uses Test navigation as an example).
SET UP TESTING GOALS
Aim to provide a close to reality test-taking experience for SAT prep students by iterating on the test navigation design with the following timeline.
FRAME THE HYPOTHESIS WITH TASKS
For students taking the test, they could complete the navigation tasks with the Diagnostic interface.
CONDUCT THE TESTS
To validate the hypothesis, we examined interviewees' behaviors. Also, we kept track of three main metrics: Functionality (success or not), Efficiency (how much time does it take), and Satisfaction (feedback from interviewees).
A note from the usability test with the three main factors.
ITERATE ON THE INTERACTIONS
Based on the tests, my team and I iterated on the interactions with two tasks (Review marked question and Submit a section using "review answers" button) that had low completion rates. Also, we regrouped the information hierarchy for the header and footer to increase a user's focus on the test.
PAIN POINTS
1. Kidaptive's logo in the middle is taking a lot of attention for the users.
2. Users can’t manage time since the timer is invisible at the beginning.
3. 3 out of 4 users didn’t aware the “Review answers” button can lead them to submit the test.
4. 2 out of 4 users couldn’t find the marked questions after clicking “Marked for review”.
ITERATIONS
1.a Make the timer visible and easy to focus.
1.b Regroup information hierarchy by removing the logo.
2.a Organize marked questions into “Review” for faster and clearer navigation with the “Marked for review” button.
2.b Increase the chance for users to discover the “submit” shortcut in “Review” while navigating through marked questions or unanswered questions.
PAIN POINTS
5. It takes extra time for users to close the full page when reviewing the unanswered questions.
ITERATIONS
3. Use a modal to minimize the time span switching between the test screen and review questions screen by clicking the backdrop area.
RESULTS BASED ON THE METRICS
After iterated on the design, we conducted second-round testing and found a significant improvement of the two tasks we were tackling based on the metrics.
SUBMIT A SECTION USING "REVIEW ANSWERS" SHORTCUT
REVIEW MARKED QUESTIONS
TIME SPEND TO MARK QUESTIONS AND SUBMIT A SECTION
+75%
Task completion rate
+50%
Task completion rate
-33%
time spent
DESIGN HANDOVER WITH LOVE AND CARE
DEAL WITH EDGE CASES
Based on the discussion with my team's developers, I provide not only the interfaces of the happy path but the edge cases in the design handover for better design consistency and to reduce confusion. This allows developers to use their time wisely and make me one of their favorite designers at all times.
REVIEW ANSWERS AT DIFFERENT STATUS
For the “review” shortcut, the text and icons will have different formats based on the status of the users.
FINAL DELIVERY
ONBOARDING
DIAGNOSTIC
REPORT PORTAL
PERSONALIZED REPORT
TAKEAWAY
For this B2B project, it has been hard to reach out and get clear information from either the partner and the users. Hence, I learned when and how and what to be flexible in the design process. Also, how to accommodate different stakeholders' needs. Take the fight that matters for the end-user experience.
Furthermore, I learned to care about the details and recognized they are as important as the big picture. The subtlety of interaction, visual design, and development...etc are the essential parts to complete the project. None of them could be neglected.
More importantly, if you want to try something no one has tried in the company. Just do it and push yourself to the limit. I am glad that I made all the interviews and testing happened. They all became important keys to unlock new pieces of evidence for the iterations.
Last, thanks for all the amazing people involved in this project. I couldn't finish this project without you. Special thanks to my mentor and best partner Hyemi Do.