A/B Testing Proposal
To learn more about user behaviors: Why do 60.2% users start from new blank document?
This A/B test can help us distinguish Type C users from Type A and Type B: A) Users just want to start from a new blank template. Users can’t find the template they need in the Template Picker, so they start from new blank template. B) Users enter into the new blank editing page unintentionally. C) Users want to start from an existing template, but they don’t know where to open the Template Picker, so they click “+ Document” button and enter into the new blank page accidentally.
we can measure the percentage of people who start from blank template in both current design and new design. If the percentage is lower in the new design, then we can infer that in current design, the button of opening the Template Picker is not very intuitive, so some people have trouble reaching it.
To find out whether Template Preview can help users find the template they need more easily.
We will measure the time from when they open an existing template in the editing page to when they close that window, and name this period of time as “Engaging Time”. If the “Engaging Time” is less than 30s, users probably opened the wrong template, and if they keep the template open after 30s, they probably found the right template and will start editing it.
We will calculate the ratio of users who close the template within 30s in both current design and new design. If the ratio is lower in the new design, we can say that the Template Preview function can help users find the template they need more easily, and our new design is a successful one.