Refining Insights

The Yale Center for Customer Insights (YCCI) Discovery Projects’ comprehensiveness and impact has been well-documented. During my YCCI project, where we partnered with one of the world’s leading financial services companies, I had a chance to conduct primary and secondary research, generate insights, test and learn, and develop insights around motivations for purchasing experiences.

Each of these phases taught me something new about working with ambiguity and conducting research. But the phase that taught me the most about the nature of customer insights was not what I expected – “test and learn.”

Our approach was to treat “test and learn” as a process – rather than a one-time experiment. We developed a workflow guided by four re-testing strategies:

1. Reduce the scope of each test

2. Ground insights within the customer journey

3. Don’t throw out your insights. Refine them with research.

4. Increase the scope of what’s testable

With this approach, our team was able to deliver deeper insights, more actionable results, and a compelling narrative for a well-known fin-tech company to transform their experiential marketing platform.

The Challenge

Our project was to help understand the motivations behind why customers purchase experiences (e.g. cooking classes, concerts, “meet & greets”, sports).

We conducted research to generate hypotheses about how consumer beliefs and goals impact their choices and affect purchasing behavior. We also designed a suite of A/B tests that we thought would comprehensively test these insights. Our initial results came back, almost entirely insignificant. We realized we would need to re-test, re-test, and re-test.

At first glance, there was no indication that our insights were useful. Our interviews and the results from our tests were saying different things. But contradictions are powerful in insight generation. They help us unpack unquestioned concepts and refine our understanding of insights.

“Test and Learn” as a process

We quickly realized we needed to re-think how to “test and learn.” In a sense, we thought of “test and learn” as a project within a project. It would have its own workflow and needed to be repeatable. We created templates for tests and developed macros-enabled workbooks to quickly analyze data. With each iteration, we would implement our strategies to help guide us toward actionable insights. We developed four strategies for re-testing:

 Reduce the scope of each test

Tolstoy’s novel, Anna Karenina, opens with the now-immortal sentence, “All happy families are alike; each unhappy family is unhappy in its own way.” This principle holds true to testing. If we made too many changes in our A/B test, we wouldn’t get useful information.

We reduced scope of our tests to get information which would be useful in creating more tests. For instance, we varied the location of a heading tag to see if it had any effect on our results. Seeing that the tag location had little to no effect, we were able to focus instead on the copy of our adjustments. This small test allowed us to refine our scope, and ultimately create significant tests in later rounds.

Ground insights within the customer journey.

Our tests were designed for a digital experience platform, which has different phases of the customer journey: exploration, consideration, purchase, etc. Certain goals and nudges make more sense at different stages of the customer journey. In our testing, we needed to align our insights to the stage of the customer journey.

To make these adjustments, we first spent time deeply understanding the context in which a user would be when using the platform. We found that our insights applied to different stages of the customer journey. By mapping our insights to the context of a customer, we were able to find significant results.

Don’t throw out your insights. Refine them with research.

Initial insights are compelling because they often resonate with multiple interviewees or yourself as a researcher. When our results came back negative, it was tempting to get rid of our initial insights and try to generate new ones. It was more useful to think of insights as “coarse” and “refined”. Before abandoning an insight, revisit it from different perspectives. There can be multiple levels of meaning in a story or a phrase that seems straightforward.

We recorded all our interviews. By having a record of our interviews, we were able to listen to the interviews again through the lenses of the insights we were testing. If an interviewee talked about “location,” but our tests for location weren’t successful, we thought about what other concepts “location” represent. By taking time to analyze and re-define words, we were able to refine our insights. We replaced general terms with specific ones. We added context. We cross-referenced interviews. We did more secondary research online.

Increase the scope of what’s testable

Once you follow the first strategy and narrow your tests enough to yield useful information, think of expanding the scope of what you can test. In our case, we thought about how we could redesign the orientation of the entire experience purchasing platform. This led us to useful tests that we otherwise would have not considered.

When it comes to testing, things aren’t as fixed as they might seem. Everything can be a variable, and testing some of the fundamental assumptions about your project can be useful. By doing so after we gained some useful information from earlier tests, we could be more sophisticated about what assumptions we were re-testing.

Conclusion

By approaching “test and learn” as a process, we were able to come up with deep insights which help further our understanding of customer behavior. The sequential aspect of test and learn made it easier for us to develop a compelling narrative to share our results with the company. And because our strategies were evidence-based and actionable, some have already been implemented on their platform. Through this process, we were able to fundamentally understand what it means to generate an insight.