Joshua Lee

Case 03 · QANDA

The feature I killed, and the one I launched

A Series C study app was about to build, for American students, a feature Korean students loved. The assumption underneath it had crossed a border without anyone checking whether it survived the trip.

Role
Product Manager Intern, US launch.
Timeframe
October to December 2024.
Product
AI study tools for 5,000+ students on web and mobile.
Company
QANDA (Cramify), Series C AI edtech.

In 30 seconds

  1. I killed a planned feature. 200+ US interviews showed the Korean demand for summary notes did not transfer. An estimated 6 engineering weeks saved.
  2. I built what students asked for. 131 hand-coded survey answers pointed to exam-like practice. Mock Exam lifted completion 24% in an A/B test.
  3. I let data overrule plans. Cutting MVP scope raised task completion 36%; fixing AI Tutor's answers raised retention 30%.

01The assumption

QANDA, a Series C AI edtech company, was launching in the US with a roadmap learned from Korean students. The next planned feature was a done-for-you summary note, a proven hit in Korea. I tested whether that demand crossed the border.

One assumption, two markets

What the plan assumed, what US students said, and what we did about it.

Korea: proven demand Done-for-you comprehensive summary notes A well-known study format. Students are overwhelmed, so give them one clean document.
US: 200+ student interviews "Tell me whether I know it" Flashcards, quizzes, self-testing. Active recall is a different job from reading a better document.

Killed the planned summary feature, an estimated 6 engineering weeks. Built Mock Exam instead: completion +24% in an A/B test.

A summary note answers what is on the exam. The students were asking whether they would pass it.

02What we built instead

An unused survey had 131 open-ended answers. I hand-coded every one, and the top unmet need was practice that feels like the real exam. I wrote the PRD and acceptance criteria for Mock Exam, aligned 3 engineers, a designer and the CSO, set success and guardrail metrics, and ran the A/B test.

From 131 answers to one feature

  1. Code 131 open-ended responses One to four keywords each, rolled into themes.
  2. Rank 4 student needs Ranked by how often each came up.
  3. Decide Exam-like practice The top need with nothing in the product serving it.
  4. Ship Mock Exam PRD and acceptance criteria, A/B tested with success and guardrail metrics set first. Completion +24%

03Fixing the right problem

AI Tutor was underused, and the easy answer was more promotion. Splitting the question showed awareness was already 85%; the real problem was answer quality. Separately, funnel analysis justified cutting real-time transcription from the MVP (task completion +36%), and a 4-cohort segmentation lifted the top cohort's retention 20%.

Why was AI Tutor underused?

The easy answer was more promotion. I split the question in three first.

AI Tutor is underused
NeedDo students want help on these questions?
Awareness · not it85% of students already knew it existed. More promotion would not help.
Answer quality · the problemOn application questions, answers copied the slides word for word.
Fix answer quality, not promotion→First on the impact/effort roadmap→Retention +30%

Where it got to.

+24%

Mock Exam completion, A/B tested.

+30%

Retention after AI Tutor shipped first.

6 weeks

Engineering time saved by the kill (est.).

04What I would do differently

List the assumptions on day one. The Korean insight was right in Korea; nobody had written down which beliefs were tested in the US.

Follow a kill with a memo. The argument won in meetings, but nothing recorded why.