False Results

Digital Marketing - Study Notes:

False positives and false negatives

A by-product of testing is the possibility of false positives or false negatives.

They arise because sometimes we read data incorrectly and draw invalid conclusions and recommendations from it. For example, if the data incorrectly overestimates performance, it can lead to a false positive result. When the data incorrectly points to poor performance, it might be a false negative. In essence, a false positive or false negative is an error in data where the result wrongly points to an outcome or condition.

In testing, false positives and false negatives are known as type I and type II errors:

• A type I error is a false positive, where the data shows a positive result for the test when the true result was a negative.
• A type II error is a false negative, where the data shows a negative result when a positive result was the true outcome of the test.

Double-check your results

In general, analysts should always remind themselves that the data and the conclusions they’re coming to might be false to some degree. So, it’s a good idea to test the data to further validate your conclusions or recommendations.

This is particularly important if you’re a beginner in terms of data analytics. In the early days of your analytics career, you need to double-check that you're reading the data correctly and that you’re not being swayed by what seem to be positive or negative results, but which, on closer inspection, are in fact the opposite.

Example

Experienced analysts are more familiar with false negatives and false positives and should proactively step in and assist non-analysts or beginner analysts if they come to the wrong conclusions. For example, they might say something like: “It might seem that display ads don't drive sales or conversions but if you look closer at the increase in brand search conversions over the same period, you can correlate display's influence in driving awareness of the brand to the increase in people looking for the brand online.”

In this instance, using direct conversion data to measure a display campaign would undoubtedly result in a false negative. The conversion data in Google Analytics is likely to show that the campaign hasn't been successful because people didn't click the banners and convert.

However, the success of the campaign should be measured by looking at other data sets because the purpose of display ads is to increase influence and awareness. With that in mind, brand search is a more suitable metric for measuring success. And it’s less likely to result in a false negative.

Testing for false positives or false negatives

So, how can analysts test for false positives or false negatives?

An analyst can:

• Use common sense when interpreting results.
• Trust, but verify.
• Review historical performance and look out for anomalies in the trends.
• Choose appropriate metrics to measure success.