Another step completed!

I have just gone through the third of the 12 weeks of the CXL Scholarship Program, studying the Growth Marketing Minidegree, and the subjects are more and more exciting and interesting!

In my first text, I covered the organization of the course and what I learned in the “Growth Marketing Foundations” section, the first of the minidegree. In the second text, I wrote about the first part of the “Running growth experiments” section, in which I learned the basics about research and conversion and the importance of good experiments. This week, in my third text, I finish the section about experimentation, analyzing two of the missing courses: A / B Testing Mastery and Statistics Fundamentals for testing, with Ben Labay.

The first course, A / B Testing Mastery, taught by Ton Wesseling, is a complete lecture on how to deal with the A / B testing process in general, considering from the most objective aspects, such as technical guidance on how to apply the technique , to more subjective aspects, like knowing when is the best time to use this type of tool.

The A / B Testing Power

A / B testing is today one of the most powerful techniques available to digital marketers: a simple way to measure performance by comparing metrics of two or more variables simultaneously. Almost anyone is able to run this type of tests using simple interface tools, often free of charge. However, the ease of the procedure brings with it positive and negative characteristics. The simplicity of the technique can sometimes lead to incorrect applications and misinterpretations, compromising the analytical process.

Statistical accuracy in the application of the technique is crucial. The instructor suggests that at least 1000 events (conversions or another parameter for which you want to optimize) are needed so that the results can offer some security. This does not mean, of course, that working with lower numbers lower makes it impossible to work with this tool. However, by understanding the statistics behind the procedure, it is possible to analyze and quantify the risks of each decision made oriented by the results.

In addition to the technical understanding of the tool, it is important to master the business in order to be able to identify the best time to implement this type of experiment. There are several situations in which this type of test is the best option. One of them, for example, is when there is high traffic and few conversions, common problem in the digital universe. Keeping focus on important variables (such as conversion, for example) helps to determine when there is any inefficiency in our business. For pages with high traffic and low conversion, it would be possible, for example, to apply an A / B test with using landing pages changing some element such as text, color or image and analyzing if the behaviour of the user changes.

Small differences can have big effects. With A / B Tests, it is possible to experiment different scenarios in which small changes are made. If the marketing team has a hypothesis that the end user could react better to a different Call To Action (CTA), we have the power to test whether this hypothesis is verified over time. It is important to make data-driven decisions, not opinion-based ones. Failure to make data oriented decisions creates a high opportunity cost for any company, as it opens the space for other market players to operate with competitive advantage.

A / B tests are not just for the purpose of improving sales indicators. Conversion can take different forms, such as the number of subscribers to an email list, the number of followers on a YouTube channel, the number of shares of a particular post, among others. In addition, it is possible to use the technique to improve a product, applying tests and making adjustments in order to make the product easier to understand, with increased usability, and other possible improvements.

How to solve a problem with A / B Tests

We have seen that A / B Tests are a powerful tool. However, the correct use is crucial for the technique to fulfill its purpose and assist in solving a specific problem. And it is exactly with a focus on the problem that we started to build a solution. The first step is to identify the issue that we seek to resolve. When defined, we build hypotheses and, from there, it is possible to create a comparative version.

To illustrate the situation, we can imagine a case where an e-commerce sales page has a lower conversion rate than other products. The problem is given. The hypotheses can be diverse: the text of the product description is not adequate, the positioning of the elements on the page disturbs the user, the images do not represent the product well, among others. With a defined hypothesis, different versions are created for the same page. Therefore, if the hypothesis chosen is that the text needs to be improved, two pages of the same product are launched simultaneously: one with the current text (A) and another with the improved text (B).

The next step is data generation. With both pages running simultaneously, conversion data is generated for both the first and second pages. Thus, after sufficient data collection, which will vary according to the situation, there will be three main possibilities:

• Conversion of page A is better than that of page B
• Conversion of page B is better than that of page A
• No statistical relevance to conclude which page converts better

With the result, the quantitative part of Test A / B is completed. The next step, qualitative part, depends on user feedback, which can be obtained through online surveys, telephone surveys or, even, following the reaction of users on social networks and other communication channels with the company. Depending on the objective, this may the most important part of the A / B Test.

Importance of statistical rigor

The second course, Statistics Fundamentals for Testing, taught by Ben Labay, is fast and to the point, completing Wessling’s course. It addresses basic subjects about data management, and some of the most important concepts in statistics, such as mean, median, variance, standard deviation and confidence intervals.

The understanding of these concepts is essential for the technical part of the application of A / B Tests to be performed with excellence, avoiding procedural errors that can generate false positive or false negative, which can result in complications of time and money, wasting time and other precious resources.

Next week…

The next section in the Growth Marketing Minidegree is titled “Data and Analytics”. The next text should deal with some of the main management tools used by Growth Marketer, such as Google Analytics and Google Tag Manager. I’m already looking forward to applying this week’s knowledge to these new tools!

See you next week!