Getting started with your Optimization Strategy
This post is an edited transcript of our recent conversation with Nick So, Director Optimization at WiderFunnel on our Masters of Growth Podcast.
Use a framework for generating hypotheses
The first thing that you’ll need if you want to optimize is a framework for coming up with ideas on elements to test and modify. In other words, to come up with decent hypotheses, you need to have the right mix of creativity and data comprehension strategies.
It’s not a matter of just churning out ideas like spaghetti, throwing them at a wall and hoping they stick. You’ll need a sense for which insights are useful, both in the ideation phase and in the analytical phase.
It is very helpful if you have some sort of framework or process in place, so when you gather data whether user research, data analytics, or results from past experiments, you can turn them into actionable insights and generate new strategy points.
One example of this kind of framework is the WiderFunnel Infinity Optimisation Process:
The folks at WiderFunnel divide the optimization approach into two parts: the creative phase and the validation phase. In the creative phase, the idea is to search for insights from analytics, look at user research, persuasion, and marketing principles, and consider consumer behavior, taking into account the business contacts and business goals. All this information is collected and sorted, then plugged into the next phase for validation.
The validation phase is all about experimentation: everything should be confirmed or dismissed by data. All these insights from the creative side are examined and validated using actual tests and hypotheses.
The third part of this framework is the LIFT model, which may be familiar to CRO enthusiasts. It basically weighs the positive characteristics of a page against the negative characteristics.
What the LIFT Model aims to do is to translate all these different data points, from marketing psychology or data analytics to user research, and to filter all those different elements into a common language that everyone across the organization can comprehend. When a designer says, “I don’t like how this looks,” the statement is not yet actionable, even if it might be justified. What does it tell us? But the LIFT Model allows us to understand the sentiment better: perhaps the element was unclear to the user, distracting from the value proposition. So the model takes away the subjectivity marketing and forces people to be accountable for their ideas.
Don’t try to test too many things at once
Once you’ve developed a few hypotheses for testing, it is not uncommon for people to get excited about the experimentation and start running several different tests at the same time. If this is you — think about the quality of your hypotheses. If you’re trying to test too many variables at once, you could be increasing the risk of false positives. Keep an eye on statistical significance and sample sizes: that way you can avoid making mistakes by being overzealous.
Start your Optimization Program today!
Try iridion PREMIUM for 30 days free. No credit card required!
3 Things You Should Do When Optimizing
Look out for the cowboys
Fact is, there are a lot of tall-tale-tellers working in optimization right now. Plenty of people who will promise you the moon and then fail to deliver. Perhaps it is true of every industry, or at least spaces that are rather ‘new to the game’, but it is certainly true of optimization. To put it differently: not all conversion rate optimizers are created equally. When you’re in the market for CRO help, the questions you ask and the way they are answered make all the difference.Don’t believe in a ‘magic bullet’ (or: the anti Don Draper approach)
No one can predict the future. Let’s repeat that once again: no one can predict the future. Guessing what a customer will want is not an effective approach for optimizing. Likewise, if your potential optimizer starts the conversation by talking about testing right off the bat or pitching you ‘the best strategy’, you might have to look elsewhere. A successful optimizer will start by examining your data, understanding your customers and what they want, and most of all, asking you questions. If the person you’re interviewing isn’t doing these things, it might be a warning sign.Explore the process together
Before you seriously consider hiring an optimizer, ask her flat out, “What’s your process?” and watch the response. If you see uncertainty (“what do you mean, process?”), then you know you can look elsewhere. Some CRO folks do have a process, but they get caught up in the technical aspects or in a one-size-fits-all approach. Before you interview optimizers, try coming up with a few scenarios to test a potential CRO’s agility. For example, you might choose data from a website that doesn’t yet have enough traffic to optimize efficiently, for example. An effective optimizer will recognize this and say frankly, it’s too early to optimize or something similar. But someone who is inexperienced or not observant might suggest tactics already. Another test scenario could involve discussing a poorly performing landing page, together with the suspicion that the value offer is not connecting with the consumer. Iff the CRO suggests sitting in a room with a lot of marketers, brainstorm and start testing, he or she is getting the cart before the horse. What is the actual hypothesis here? But if she suggests meeting with salespeople and drilling down what customers actually want, she’s probably the right person for the job. In short: Look for someone who talks first about understanding the customer, the overall scenario and data before suggesting tactics or solutions.Start structuring your Experiments today!
Try iridion for free as long as you want by signing up now!
Create your free Optimization Project
More Uplift, More Insight – 4 unseen factors in successful optimization
Every optimizer knows that a test can end with outstanding uplift one day, and the next day you’re left wondering why the last test failed. Conversion-optimization is more drudgery than the superhero images would have us believe.
Sooner or later, there are important questions that an optimizer needs to answer:
- “How can I separate bad ideas from good, early on?
- “How can I increase my rate of success?
- “How can I improve results and uplift?
Consider an ocean voyage… but on a stormy sea. It takes a stable craft, reliable navigational equipment, a hard-working crew and a captain who knows what he’s doing.
We have been at the business of sailing the “Sea of Conversion” for many years. What we know for sure is that every detail is important and every aid is relevant. To cross that sea with certainty, we’ve put together the most essential components for a successful voyage.
1. Build the boat – identify weak points and generate measures for optimization
The Alpha and Omega of any successful experiment comprises the ideas and the hypotheses at its foundation. Every successful experiment draws on a good idea or a good hypothesis at its heart. To tackle the right issue, it’s important to ask the right question.
Qualitative analysis leads to new ideas
To generate measures for optimization, important factors in addition to knowledge on the metrics for the site are creativity, empathy and a sound understanding of user behavior. Models can help guarantee continual high performance, making sure that input goes beyond mere subjective opinion and that user purchasing behavior is always being questioned.
The “seven levels of conversion” framework
Why should the seven levels framework be used?
The framework allows a simpler pick-up on the user’s decision-making process, thereby identifying weak points. You can see through the user’s eyes and comprehend a website visit from that perspective.
How do I benefit?
The weak points that are identified by using the framework offer new sources for optimization measures. If the framework is used regularly it helps to derive the right ideas or hypotheses for the particular website and target group.
We have digitalized the seven levels framework for the first time and we are making it accessible through Iridion. It has never been easier to use the framework.
2. Sail in the right direction – prioritize optimization measures the right way
Regardless of whether the issue concerns a sophisticated hypothesis or a quickly-formulated optimization idea, sooner or later the question will arise: How do you objectively evaluate and determine what’s worth testing?
Usually, assumptions that appear logical form the foundation for deciding what needs to be tested:
- How much traffic does the particular site see?
- Which sites have the highest drop out rate?
- What’s easy to implement?
- Or will the HIPPO (highest paid person’s opinion) simply once more determine how to go forward?
In reality, decisions are often made based on these or similar criteria. To reach well-founded decisions, however, it is important to use an evaluation system that is as objective as possible and one that incorporates all influencing factors that are relevant to the optimization measures.
Systems that can help in prioritizing are, for example:
- The PIE Framework by WiderFunnel
- CHPL-Scoring by konversionsKRAFT
What good will it do me?
Resources are often limited, and it is therefore particularly important to do testing only where a good cost-benefit relationship exists. The scoring provides a precise tool to isolate measures that are really worth the expense of an A/B test.
Tip:
With a system like that, personal HIPPO preferences can also be included and possibly weighted more highly. The chances of making all stakeholders happy and reaching a consensus increase enormously because of this system.
We have integrated CHPL scoring, which has been proven in practice, into Iridion and made it accessible to everyone.
3. Keep the logbook current – control your experiments
It isn’t easy, getting an overview of all of the information relevant to an experiment. But knowing what happened in all of your past and present experiments at any given time is essential in order to make meaningful decisions or to pass knowledge gained along to the right people.
The following questions should be answered in order to continually cultivate information that relates to the CRO process:
- What’s being tested?
- What is the current state of the current tests?
- Which tests were successful and which were not?
- Which hypotheses / ideas or weak points were tested?
- How do I quickly set up a results report?
Iridion helps answer these and other questions at any time.
Tools that can help organize tests (but are not directly associated with Iridion)
- Excel to prepare test results
- Powerpoint/Keynote for results reports
- Dropbox or Fileserver for Wireframes, concepts and other supplements.
- The particular testing tool for all test-related data and visualizations
- A PM Tool that coordinates and allocates the particular test status
- Google Docs for collaborative work
These solutions are not centralized and are very hard to automate, so Iridion puts everything into one solution.
Iridion makes it possible to obtain an overview at any time, and to grasp all important information prepared in one view.
4. Maintain knowledge and make it reusable – maintain the logbook
Information that results from testing is often developed with difficulty only. It is important that knowledge gained be put to use and retained after implementing the change. Every test can lead to an improvement in the foundation for follow-up decisions.
A typical scenario
“Two years ago Mr. Schulz, the responsible person at that time, had already tested a similar scenario. Unfortunately, Mr. Schultz is no longer working with us and knowledge acquired at that time is no longer available. What now?
A “CRO” Wiki
Quality and number of A/B tests are not the only factors crucial to guaranteeing continued forward movement. The application of knowledge already obtained is a relevant influencing variable to achieve a greater probability of success and an advantage over the competition.
Why do I need this?
Optimization always includes a learning curve. Optimization measures can therefore go through a number of cycles. If care is taken to avoid loss of available knowledge and, instead, that knowledge is used meaningfully, then hypotheses and test concepts can be better tuned to a particular target group.
The following tools help make knowledge permanently accessible:
- Internal Wiki for knowledge storage for test concepts (time consuming)
- Excel as storage for test results (not fit for teams)
- A PM tool can maintain and characterize knowledge in the ticket system (often ill-suited for test results)
- PDF’s with test reports on the server (not easily searchable)
Whether there is a significant uplift or not, every test still provides knowledge that should be recorded. Iridion makes the development of a CRO-Wiki possible.
Upshot – holding to a course isn’t easy
Every phase of the trip makes use of various methods and tools. Which ones– and how many of those are suitable– are often difficult to determine. In the end, the efficiency of the measures should increase and the outcome to the conversion optimization should be influenced positively.
We found it inconvenient, having to refer back to a hodgepodge of various tools during our daily work in order to master our optimization procedure.
We developed a tool that displays the complete flow of work centrally, in one solution– to simplify the CRO every day and to design it as efficiently as possible – Iridion.
Learn more about the tool here – Iridion – Welcome to our world.
Start structuring your Experiments today!
Try iridion for free as long as you want by signing up now!
Two out of Three of all A/B tests are not valid
I admit that this statement launches terror. So first let me calm you down. It isn’t that the concepts are bad or that A/B testing is unreliable in general. The erroneous results are much more likely to be traced back to something that can actually be avoided– namely by making sure to that test runtimes are adequate.
Results are subject to great fluctuations right at the start of testing and they only stabilize during the course of time and approach actual conversion rates. If we consider the following example of an A/B test with four variations, we see that the results of the variations converge again after a certain time.
Source:
http://www.qubit.com/sites/default/files/pdf/mostwinningabtestresultsareillusory_0.pdf
If tests are shut down too early, results are frequently displayed that actually show only the momentary perspective. The true outcome – like an unforeseeable twist in a movie – can’t even be predicted to any degree.
With this article, I would like to hand you the tools in order for you not to shut down your test too early and to make certain that it belongs to the one-third of tests that provide valid results.
How can I also verify uplifts for my variations with certainty?
Let’s imagine that we we’d like to find out whether there is a difference between men and women’s shoes sizes and to prove that using a test. If we now draw only upon a few men and a few women for the sample, we take the risk that we will not discover that men on average have larger shoe sizes than women. It could well be that, by chance, we have selected for our test men tending to have smaller feet and women with larger feet, and have already come to a conclusion with a sample that is too small.
The larger the sample, the greater also the probability that the sample “stabilizes” and we can verify the actual difference between shoe sizes in men and women. A larger sample ensures that we obtain a reliable image of the actual shoe sizes for men and women and that we also find the real, existing difference through the test.
The sample in an online business
In online business we are naturally interested in whether our created variation is better than the previous variation (control variation).
Example: Uplift with coupons
Let us now simply assume that users in one variation get a 10€ discount on their current order, which is not available in the control variation.
The hypothesis:
As a result of the additional monetary incentive of 10€, the motivation to carry out an order is increased, whereby the number of orders in turn also increases.
The result:
After a 30-day test runtime the testing tool features a significant uplift of 3%. Had we let the test run for 14 days only, we would have determined an uplift of exactly 1%, which would not have been significant.
The insight: Longer = more validity! But how long, exactly?
In principle, it can be said that the longer a test runs, the higher the probability of verifying a true difference with the test. In our case, that men have a larger shoe size than women.
If this effect – as with our illustration of shoe sizes – is very clear, then even a smaller sample will produce valid results. If, however, the effect is very slight, as with the example of the website with an uplift of 3%, then the sample must be many time larger in order to verify the effect with a particular degree of certainty.
If our sample is too small, we take the risk that we will not discover the existing difference, even though there is one. How large the sample must be at a minimum in order to also be able to verify the effect can be determined with the help of the statistical power calculation:
The statistical power calculation
The statistical power is the probability of being able to verify an uplift that also actually exists by means of an experiment.
The greater the power, the more likely it is to be able to significantly verify an uplift that actually exists by means of an A/B test.
As a rule, we refer to a powerful test if it has a power of at least 80%. This means that the probability of verifying an uplift that actually exists is 80%. In the reverse, there is also still a 20% risk that we will not verify an uplift that actually does exist. Perhaps you have already heard of the so-called “beta error” or a “false negative” (type 2 error).
It is as if Christopher Columbus had a 20% risk of sailing past America and therefore not discovering a new land that was actually there.
If the power of the experiment is too low, however, we not only run the danger of not verifying real uplifts. Even worse, we shut down an experiment because it features a significant winner that, in reality, is none at all. If such a circumstance arises, we refer to an “alpha error” or a “false positive.”
In this case Columbus would have thought he landed in India, although he discovered a new continent.
If we shut down an experiment too soon, that is, as soon as the testing tool shows a significant uplift, the error rate is at 77%. This means that the probability that the measured effect has arisen purely by chance is 77%.
Ton Wesseling: You should know that stopping a test once it’s significant is deadly sin number 1 in A/B-testing land. 77% of the A/A-tests (same page against same page) will reach significance at a certain point (Source).
Perhaps you are familiar with the following situation from your everyday optimizer world:
The test carried out contributed an uplift of 10% onto the orders. The results are significant after ten days (confidence > 95%) and the test is shut down. In reality, the type of result that you can only hope for.
In order not to lose time, the test is shut down immediately, the concept is quickly implemented by the IT department and the original is replaced with the new one. Now, all you have to do is wait until the uplift also translates into the numbers. 🙂
But what if the 10% uplift simply doesn’t appear under real conditions and the numbers show a very different picture? Namely, it can be that the uplift will not to be seen at all! In principle, the numbers remain exactly as they were previously – that is, unchanged.
This can be due to the fact that the test has simply been shut down too early. If the test had just been allowed to run somewhat longer, it would have been determined that the conversion rate for both variations realigned. Also, the significance that was determined earlier is gone. This effect is clarified by the following graphics:
What happened in the test?
If we had carried out a power calculation before the test we would have discovered that our test had to run for one month before we reached a power of 80%. However, the test was then shut down after only ten days because the result was significant. In your defense, the testing tool also had already referred to a winner, for which reason you shut down the test with the best of knowledge and belief. The crux of the matter: the statistical power at that point in time was just 20% and the testing tool also did not show this.
Calculation for the minimal test runtime for valid results
In order to discover how long a test has to run at a minimum to obtain valid results, we must carry out a calculation considering the statistical power even before the start of the test:
These factors determine the minimal test runtime:
Conversions / month – this is the metric on which you would like to optimize. As a rule, it is the orders. For exact planning, you should, when possible only consider the conversions here that were also previously on the test page.
For example, let’s carry out a test on the product detail page. Within one month 3,000 visitors have placed an order. Of these 3000 users, however, only 2,600 were previously on the product detail page. Four hundred orders were placed either directly from the shopping cart or via the category page. In this case, the conversions / month = 2,600.
The conversion rate for the test page – in this case the issue is the ratio of the persons who were on the test page and purchased, to those who did not purchase.
Example: The product detail page was visited by, in total, 15,000 users within the month. This means that the conversion rate for the test page = 2,600 conversions / 15,000 visitors on the product detail page (17.3%).
The number of variations for the test – this information is important because the runtime is extended depending on the number of variations. The more variations included in the test, the longer the test runtime.
Confidence – How certain would you like to be about your test results? Confidence indicates to what extent you are prepared to take the risk of chance effects. In this regard, you should select the confidence that you otherwise also use to interpret your test results. As a rule, this is 95%. This means that the risk of finding a random effect that, in reality, does not exist is only 5%.
Power – Power expresses the probability with which you can verify an actual, existing uplift by means of an experiment. The value of 80% is often the goal.
Expected uplift – This is the effect on the conversion rate that we expect from the test concept. Here, you should enter a value that is realistic based on your previous testing experience for such a test. The closer you are to this value with your estimate, the better you will also estimate the power of the experiment and obtain valid data. If the issue involves a test, the outcome of which you cannot estimate at all, I recommend estimating an uplift that you would like to achieve as a minimum in order to then speak of a successful test, one that you would also like to roll-out in follow-up.
Tools for calculating power
Because testing tools, as a rule, unfortunately do not carry out power calculations or offer it to their users, it becomes incumbent on you to carry this through. There are a number of tools available on the web to calculate the statistical power. One of the best known is gPower.
These tools are usually somewhat difficult to interpret and are not suited to the workday-world CRO. Fortunately, for us optimizers, there are also tools that are clearly better tuned to our CRO process to calculate test runtimes.
It can even work more elegantly: Iridion, developed by konversionsKRAFT itself, hands optimizers the opportunity to do a power test with its help from two perspectives. In addition to the “normal” test runtime calculation, you can also analyze whether it is even possible to achieve a valid result in a predetermined test runtime.
After having input the appropriate information for the experiment, an individual recommendation is given on how long the experiment must at least run, or whether you have to reduce the number of variations or extend the test runtime in order to obtain valid results.
Successful test planning in five steps
- Define how much uplift your variation must at least achieve so that you can refer to a successful test ––> That is, that you would also like to build the variation and roll it out.
- Determine how certain you wish to be of really discovering an actual uplift (power). In other words, to what extent are you ready to discover an uplift that exists it reality?
- Plan a realistic number of variations for your experiment. The less traffic you have available, the fewer variations you should use in your experiment.
- Calculate the minimal test runtime and the minimal number of conversions and visitors for your test and keep this firmly in view.
- Do not shut the test down until you have fulfilled the minimal test requirements.
Conclusion:
You would like your test to belong to the one-third of tests that provide valid results? By having a plan that is thought-through before the testing, you rule out the possibility that your experiment will not provide valid results due to too short a runtime.
The validity of the test result is assured through the help of a test runtime calculation. How certain this should be exactly, of course, depends on individual judgment. The level of the desired power and the accepted error probability can be established individually per test.
Find a good middle path between validity and business interests.
Of course, a high level of validity is associated, as a rule, with longer test runtimes and thus higher costs. Eventually, everyone has to decide for him- or herself at what point the highest profitability of A/B testing is achieved and a low residual risk of error is taken into account in order to be able to thereby save time and cash. This small window of maximum profitability from A/B testing is handled by the U-model of validity.
Start structuring your Experiments today!
Try iridion for free as long as you want by signing up now!
Create your free Optimization Project
The E-Commerce paradox
Looking for more Conversion Whiteboards?
Then follow us on our YouTube channel!Start structuring your Experiments today!
Try iridion for free as long as you want by signing up now!
Create your free Optimization Project
Why you’re ending your A/B testing too early
- Statistical influencing factors (e.g. sample size, number of variations, measuring method, error tolerance, etc.)
- External influencing factors (e.g. traffic source, time-to-purchase, frequency of usage, campaigns, season etc.)
- Economic influencing factors (e.g. budgets, positive/negative effects, costs-benefit relationship, etc.).
Aspect 1: the statistical test power
In this context, a traffic duration calculator is usually used in order to be able to predict a runtime. To avoid getting lost in the dry basics of statistics: In principle, the issue in this case concerns a mathematically modified form of the calculation of the test power. The test power indicates how probable it is to prove a difference between a null hypothesis (“there is no difference”) and a specific, alternative hypothesis (“there is a difference”). If the test power is high, a difference exists; if it is low, there is none. The greater the test power, the more likely it is to prove an actual effect (uplift). If this uplift is very big, a short test runtime is adequate for a good test power. If, however, the effect is small, then (much) more time is correspondingly required. To ultimately calculate the power, three things are needed:- The ?-error (significance level, usually 95%),
- The sample size and
- the expected uplift (effect)
Aspect 2: What are all of the influences on a test?
Is the number of samples large enough, in other words: If (very) much traffic is available, then – from the point of view of statistics – a test does not need to run for a particularly long time. This is precisely so when few variations are tested and the expected effect (also called impact) is high. The problem in this case is that, in the online world, you have no influence on the type of test participant. The selected sample is thus not representative at times. There are numerous examples that can be shown where variations run particularly well or poorly during the initial days, but the change (completely) in due course – despite a large sample at the start. Three aspects that have a determinative effect are the different behavioral patterns, type of usage, and surroundings variables. Thus, a series of factors can have an influence, such as:- The day of the week (weekend vs. weekday),
- Situation (work place vs. couch vs. commuting),
- Mood (good weather vs. bad),
- Prior knowledge (existing customers vs. new),
- Motivation (promotion versus recommendation),
- and many others
Aspect 3: Stated with some exaggeration – how much will the truth cost me?
The economic aspect finally also determines how long a test should run with positive, negative or possibly no effect. Is it monetarily justifiable to allow a variation with significant uplift to continue running?The control burns cash, so to speak, doesn’t it?Even worse:
The variation is significantly worse, wouldn’t it be better if I shut it down, to avoid burning cash?One more classic:
The variations converge. Does it make sense to let the test continue to run?Although the answers to these questions can be deduced based on statistics and external influencing factors, this is always a question of judgment. The question is, Where should the focus be placed? On the most rapid test result possible that is shown to be adequate to reach a valid decision:
I want to know whether the variation is betterOr does the result need to be as accurate as possible and be safe from statistical coincidence as well as it can be:
I want to know by exactly how much the variation is betterThe calculation can determine this even in the preliminary stage: At 80% test power and one-tailed testing, the test will clearly lead to a result more quickly, for which exact reason test tools in the most favorable price range apply this combination. At 95% and two-tailed testing, it takes correspondingly longer but, in turn, is more precise. Example shows the influence of the test power and testing on the sample size (and thus on the test runtime.) Just calculating the test runtime alone allows economic aspects to provide the decisive tip of the scales. For example, based on the expected uplift and available traffic, the test must run so long that it is uneconomical to even carry out the test.
Then what is the right strategy for the test runtime?
If there is an awareness of all of these influencing factors, then you should create an appropriate test runtime formula or a stop-test rule. Unfortunately, there is no panacea for this. Aside from the “hard” statistical factors, all other factors depend on the respective test situation. In the final analysis, the test hypothesis can “only” fail to function or not feature adequate contrast. There are several indicators that can help for your own formula under the pretext of the representative sample:- In order to be able to display as many different usage cycles and types as possible in the test, it should run over entire weeks (e.g. Mon. to Mon.).
- As many as possible various users should take part in the test, e.g. existing customers and new customers (various traffic sources, campaigns such as newsletters, TV spots, etc.). Segmentation here is important in order to interpret the results and to be able to gain deeper insight.
- The test should run a longer period of time, even if high traffic is available. The traffic can be reduced under certain circumstances (traffic allocation), in order to be able to test for a longer period of time (reduce costs for traffic volumes).
- Don’t stop the test “too early.” The statistical and external influences kept in mind require test time, This is equally true for positive and negative results! Just because the tool displays 95% significance or higher after a short time does not mean that you can now immediately stop and celebrate.
- Consider the time to purchase and possible adaptation factors (existing customers). The test should at least include one cycle.
Conclusion
This article should not lead to the impression that the test runtime is arbitrary and cannot be planned – to the contrary. If you have become aware of the aspects presented here, then an estimate of how long a test should run can be made, even in the preliminary stage. The more experience collected during testing, the more likely the “expected uplift” value necessary for the calculation and additional external factors can be estimated. My rule of thumb for an average test runtime and a stop-test strategy:A test should run at least two to four weeks and include at least 1,000 to 2,000 conversions per variation. Activities (news letters, TV spots, sales, etc.) should take place during this time. There should be as many different channels displayed as possible (total traffic mix, either through targeting in the preliminary stage or segmentation in follow-up). If the test has achieved a minimal statistical significance of 95% (two-tailed, that is positive and negative) and if it is stable, then stop the test.What have you experienced regarding test runtime or a stop-test strategy? I would appreciate your feedback! Links Calculators
Start structuring your Experiments today!
Try iridion for free as long as you want by signing up now!
Create your free Optimization Project
No results? Five inconspicuous testing errors that foul-up growth
- How is the conversion rate developing?
- How satisfied are you with how the conversion rate is developing?
- Do you feel you can control the conversion rate?
- Redesigns and similar projects often block the path to meaningful, incremental optimization
- Know-how from outside is an important factor for the success of the internal optimization team
- Despite numerous A/B tests, there is little or no measured effect in linear progress
- A large portion of those in charge in the market are not satisfied with the development of their conversion rate
#1 The “I am deceiving myself with bad results” error
The tools are not entirely blameless here. It now takes just a few hours until the results of a test are already significant, according to the tool. “High Five! Winning variation found!”the tool announces with a small trophy icon. Can that be? We have already explained the problem many times. Independent of the purely mathematically calculated significance value (CTBO/CTBB), there are requirements for the test runtime, so that results actually also become valid. A great influence on test results is the traffic source: visitors from newsletters are often existing customers who react differently to any variation than do new customers, There are TV-adverts, varying weather, at the start of the month more money is spent than at the end. The influences are thus very complex and also not to be grasped, even with the most sophisticated segmentation. Whoever runs his test too short will obtain only a short, random section. Therefore (very simplified): Fourteen days are a good minimum (!) test runtime. An approximately representative sample should be reached during this time. Tests during extreme SALES events should be avoided – or the test should be repeated outside of the event. For those who want more agility, better to run more tests in parallel. Why does this testing error foul-up growth? Very simple: The “winning variation” in reality was often no winner at all. Had the test been run only a few days longer, the result would have “converged.” Whoever stops testing too early may be ensnared in a statistical artifact. Ergo: Resources for the test were wasted, incorrect results communicated (the disappointment can be very great!) and, in the worst case, a change that provided no benefit at all was rolled-out.#2 The irrelevance error
Much too often I see test ideas for enterprises with subjects like “We want to know what happens if we move this box from the left to the right.”What good will that do? Who among us has once put him- or herself in the role of a user and thought, “So, if the box were now on the right side, with the accessories, then I would buy in this shop… ?” Similar is true for tests with different button colors and other details. Such tests will not yield results that will significantly and permanently influence growth. I see it this way: A test variation must be strong enough to actually influence user behavior. Changes that do not meet this challenge also cannot generate measurable results. From the perspective of the optimizer, an A/B test measures a change in the website. This, however, is a fatal fallacy. In actuality, the testing tool measures the consequence of a change in user behavior. A variation that does not change user behavior also cannot provide a result in the A/B test. Why does this testing error foul-up growth? Very simple: Let us assume that a shop generates 20,000 orders per month. For a good test with valid results we need at least two weeks runtime and at least 1,000 conversions per variation. An A/B test with five variations already needs 5,000 orders. If we play it safe, maybe this is 10,000 orders. We have no more than two to four test slots per month; that is a maximum of 20 to 50 test slots per year. What percentage of tests delivers valid results? How high is the average uplift? If you calculate this, it will quickly become clear that the valuable test slots should not be sacrificed for banalities. The more effective the test hypothesis, the more uplift can be generated. In actuality there are, in fact, entirely different limiting factors for the number of good test slots…# 3 Absent agility
Agility appears to have become a buzzword. The commercial effect of absent agility can easily be calculated in the form of opportunity costs. In fact, it is very easy: Whoever can carry out double as many successful optimizing sprints has twice as much success. The general conditions thereby are very conservative: The issue concerns an online business with 20 million euro turnover and a 15% profit margin. Why does this testing error foul-up growth? The most frequent question asked is: “Is a 25% global and cumulative uplift per year through optimization even possible?” Posing this question, however, is basically justified – but shifts the perspective. For most of those in charge the conversion rate is a fixed constant – a jump from 3% to 4% seems possible through massive price lowering or traffic retrenchment. It is thereby overlooked that it is the website that does the selling. Whoever holds only traffic, assortment and competition responsible for their own conversion rate has, in my opinion, overlooked the greatest optimization leverage. The right question must be:“What must we do in order to achieve a 25% uplift through optimization?”In 2013, according to their annual report, Amazon carried-out almost 2000 A/B tests – an immensely high figure that certainly is only even technically possible with extremely high traffic. How much knowledge would Amazon have gained from this? How many tests resulted in uplift? How much of the knowledge is a strategic competitive advantage? The answers should intensify the focus onto your own organization and lead to the following questions: Why does the organization limit the agile implementation of optimization? Who has an interest in that? What can we do? These answers often lead to initial ideas that can break the spell…
# 4 The technical error
Whoever does a lot of testing knows the feeling: poor test results ruin the mood.For conversion optimizers, all of a sudden A/B tests have a greater influence on wellbeing than the weather… 🙂 A frequent phenomenon: If the test runs well, it was a brilliant test idea. If the test runs poorly, there must be a problem with the testing tool. Quite seriously: Even well-run tests in fact suffer more frequently from technical problems than is thought. It is generally known that store times are critical to a positive UX and thereby to the conversion rate. What happens during a test with code injection?- The code is loaded.
- At the end of the loading procedure the DOM tree is manipulated.
- The front end is changed while the user already sees the page.
- The result: a palpable delay in page assembly and flickering effects – a poor user experience.
#5 The error of incorrect KPIs
Often cited example: I stand in front of a display window and consider whether the shop has suitable products at a suitable price. Dozens of implicit signals let my brain decide in seconds whether I enter the store or not. The first impression has an influence on my decision. Let us assume the display window does not look particularly promising. Very similar to a poor landing page. If the storeowner would polish-up his display window, let’s say to achieve for a high-quality impression, this would possibly tend to move me to enter the store. However, because he only optimized the display window (that is, his landing page) and the rest of the store is just as ramshackle and bad as before, I will merely be more disappointed. The higher micro-conversion subsequently has a contrary effect. We could also call it a case of expectations management or consistency. But, unfortunately, this error is often repeated online: Why does this foul-up growth? Because in the most frequent case supposed improvements are rolled-out that even damage the conversion rate. Unnecessary resources for tests are wasted, time is lost, and the overall result is deteriorated. Above all, medium and small enterprises that do not have enough conversions for A/B testing optimize on clicks and a reduction in the bounce rate. They are well advised to nevertheless measure the total conversion in tests and to keep an eye on this. The only consolation: knowledge from landing page tests can be used and (hopefully) applied successfully in the further course of the funnel. Supplement: The same is true in E-commerce for the relationship between orders (gross turnover) and returns (net turnover). Much too often, unfortunately, the latter is not measured or evaluated – also here the non-fulfillment of expectations can sink the overall profit margin. Bonus tip:#6 The “We have no business plan for optimization” error
Finally a very simple situation. Above, I spoke of an annual +25% conversion rate uplift and showed what happens to the contrary if only 5% is reached. A mental game:- Optimization costs resources, time, and money. Optimization slots are limited and thereby valuable.
- The goal of optimization is a result measurable in business – an ROI.
- If costs and benefits are established and also controllable – why is there no business or investment plan for optimization measures?
He who has no goal will never reach itSorry, if this gets a bit brief now. I believe very simply that many enterprises do not reach growth through optimization because they optimize haphazardly without a plan or a goal.
Conclusion
Optimization takes place in many areas and the results are validated with A/B testing. But, unfortunately, there are plenty of influencing factors in the form of these testing errors that sink the ROI for the measures– entirely unnecessarily. Anyone who masters these five topics alone and composes a business plan can count on sustainable and effective growth.Start structuring your Experiments today!
Try iridion for free as long as you want by signing up now!
Create your free Optimization Project
The U-Model of Validity
Looking for more Conversion Whiteboards?
Then follow us on our YouTube channel!Start structuring your Experiments today!
Try iridion for free as long as you want by signing up now!
Create your free Optimization Project
Checklist: 15 simple things for trouble-free A/B tests
#1 No deployments during the test run
Changes or bug fixes, even if they appear small and simple, can lead to complications as the test plays-out. This is above all important for <tag> based testing tools. If there is a roadmap with a definite deployment schedule, tests can be carried out before or after. If a deployment is not to be impeded during a test runtime, talk to your developer ahead of time. With a copy of the running test and a play-out, e.g. onto a staging server, possible conflicts can be recognized early on and the test can be adjusted. You would like to estimate the test runtime? Our rule of thumb for an average test runtime:My colleague Manuel Brückmann gets into this topic in detail in his article Why you’re shutting down your A/B-test too soon.A test should run at least 2 – 4 weeks and contain at least 1,000 – 2,000 conversions per variation. Special events should take placed during this period (newsletter, TV-spots, sales, etc.). The greatest possible number of channels should be displayed (total traffic mix, either through targeting in the preliminary stage or segmentation in follow-up). If the test has reached a minimum statistical significance of 95% (two-tailed, that is positive as well as negative) and if this is stable, then the test can be stopped.
#2 Be aware of contrast
The combination of different adaptations often makes contrast-rich testing possible. Too high a contrast makes a subsequent analysis of the most effective leverage difficult, however. Additionally, expenses for implementation and quality assurance will rise. My recommendation would therefore be: Check a big adjustment in a first test. Undertake further adjustments in following sprints and validate.#3 Be aware of the worst-case scenario
Often, with a wide product assortment, there are divergent illustrations in the template, or the header changes through progression of the site, e.g. UVPs are presented bolder in the checkout. This can lead to bigger or smaller problems in development as well as in quality assurance. By setting-up a worst-case scenario in the preliminary stage (e.g. in the form of an impediment backlog) special cases can accordingly be observed as early as development. With this list, quality assurance can offer an optimal check of all possible scenarios. Ask your customers about special cases (special sizes, particular product categories such as, e.g., accessories, etc.). This makes it easier for both sides the intercept deviations.#4 Agility vs. perfection
Often, A/B tests are expected to be implemented to complete perfection. But after a number of weeks of tuning cycles, development, quality assurance and test runtime, however, the result is unsuccessful. We like to say,“a test is not a deployment”. Important: This is not intended to indicate a reduction in the quality of test implementation or quality assurance. Instead, the issue concerns implementing minimal details, the effect of which, however, are questionable on a visitor. Should you be confronted by this problem at the next occasion, simply pose the following question: “Would I not buy this product because, e.g., the font size, line spacing or margin width isn’t right?”#5 Be aware of lateral entry
A lateral entry into a test should always be noted. SEA, SEO or established customers should under circumstances be shown a changed page or be redirected to another page. In the former case, serious errors could arise, caused by the play-out of the test.#6 Take note of the test starting point and any discrepancy in the URL structure
A look should be given to a possible discrepancy in the URL structure, above all when carrying-out multi-page or funnel tests. Should the structure unexpectedly change, in the worst case the targeting of the testing tool might no longer be effective. This results in participants no longer being able to see the test variation. For external providers, for e.g. methods of payment, conversion and revenue can be lost – through a missing return path on the thank you page.#7 No test without targeting
Excluding certain devices / browsers is not necessary if quality assurance can guarantee error-free test play-out onto all end-user devices with the associated browsers. In the rarest of cases, however, is this actually possible. A whitelist, a listing of all end-user devices as well as browsers that are to be checked by quality assurance, can exclude potential sources of error. It is advantageous if this list is already defined before development release. The current hit rates from an existing web analytics / reporting tool, such as Google analytics, assist in this regard.
The somewhat hidden custom-targeting in Visual Website Optimizer. The upper target, for example, for the exclusion of mobile end-user devices. The lower target comprises a regular print out, whereby only desired browsers (whitelist) are permitted into the test.
#8 Exclusion from measurement
Who are the daily users on a site? Aside from your customers, these are often suppliers, such as call-centers or even colleagues in the office next door. But this can render your results erroneous. Therefore, exclude your own IP-address as well as that of internal and external employees from the measurement. Note: not every tool provides the opportunity to filter-out IP addresses from the results afterwards.#9 Browsers get old too
Older browsers, such as IE 7 and 8, die off. By looking at traffic numbers, older browsers can often be neglected. Based on experience, browsers such as IE 7 or IE 8 require enormous additional time in development and quality assurance.
Source: http://www.w3schools.com/browsers/browsers_stats.asp
According to statistics from December 2014, of 1,000 visitors we had an allotment of 1.5% of visitors using IE 7 and 8. This corresponds to 15 visitors.
#10 Are iFrames available?
External service providers will gladly expand your site via functions. It can happen that these functions are implemented via an iFrame. iFrames can be manipulated if they are located in the same domain. If this is not the case, your testing tool must be implemented in the supplier domain. If you are lucky this is possible. In most cases, however, it is not. Check the concept together with your developer in advance for feasibility.#11 Take note of quality assurance expense
The time required for quality assurance is often underestimated. Typical, problematic cases involve tests on the product detail page or checkout. The expense can climb exponentially. Talk to your quality assurance in advance, in order not to be surprised. Here is an example of a simplified calculation for the time required for checkout tests: variations x Browser x Pages x Types of customers x Types of delivery x Types of payment x Special cases = Pages to be considered variationen x Browser x Seiten x Kundentypen x Lieferarten x Zahlungsarten x Sonderfälle = Zu betrachtende Seiten Pages to be considered x Time per page = Time needed by quality assurance A test with two variations for only four browsers, with five pages, three types of delivery and five types of payment corresponds to 600 pages to be investigated. At only two minutes per page the time used is 20 hours.#12 Is the product diversity known?
The product detail page frequently offers great optimization potential. From small adjustments to complete restructuring. Often, however, the quantity of products and their varying depictions (special cases) do no get attention. The result is necessary adjustments for development and additional expense for quality assurance. The consequence is a delay in the start of the test.
You will find a multitude of differently structured product detail pages on Amazon.
In case you are not familiar with all possible special cases, seek out colleagues who can further help you. Combine the corresponding special cases before the test is conceived and provide the appropriate information to development and quality assurance.
Should it be too late for this or if unexpected problems arise, you should take note of the following: Can individual categories or brands be tested for the time being? This will make the start of the test possible and provides a buffer for adjusting the test to your entire assortment.
#13 Take note of Ajax or the adjustment of product images
Not infrequently, changes are undertaken through dynamically reloaded content, e.g. information on inventory. But this is not always immediately obvious. In the best case, there is merely added expense for development, in the worst case implementation is not possible. When product images are being reduced or enlarged, the quality of the images must be noted. A pixelated image prevents optimal product illustration.
When enlarging thumbnails, it quickly becomes clear that image quality is no longer adequate.
Reloaded content is not immediately available for manipulations. Whether this can lead to problems in the implementation of tests in this case must be checked in detail.
Take note of reloaded content and take development into account early on. The enlargement of product images as well as the addition of larger content (text or image) should be tested in advance in order to assure feasibility.
#14 Conceal complex modifications through loading animations
A visitor may become unsettled as a result of visible modifications. In the worst case trust is lost and the visit is terminated. This effect can be reduced by using loading animation. The affected area is first faded out and provided with loading animation. This area becomes visible after all necessary manipulations are done. Modifications are thus at best rendered invisible and possibly increase the joy of use.#15 100% play-outs as an intermediate solution
Following a successful test, the adjustment that is made is often intended to be implemented on the site permanently as quickly as possible. Depending on the deployment cycle of an enterprise, however, this can easily take a couple of weeks to a month. 100% play-out represents a temporary bridge. As a rule, the costs of the testing tool are less than the increased turnover from play-out adaptation. In this way the increased turnover can be picked-up directly.Conclusion
As soon as the desire arises to test more than the red vs. the blue button, the entire process, from conceptualization right through to a finished test, becomes more complex. Unfortunately, this often goes unnoticed in practice. Many problems can be prevented by good preparation and communication with colleagues as early as before development. Be aware of the expense due to implementation and quality assurance, and hold consultations with your developer and quality assurance in due time. In this way, expenses and the possible occurrence of problems can be better assessed. Reference is often made to negative test results in order to generate improvements and learning. A test was carried out successfully? Many people will not obtain information on the test results. Relay the experience of success to all participants and thereby increase the motivation of the whole team. Celebrate the success together! What is your procedure in the development of tests? I would love to hear your feedback in the comments!Start structuring your Experiments today!
Try iridion for free as long as you want by signing up now!
Create your free Optimization Project
The four-fields of A/B-Testing
- All optimizers dream of “low hanging fruit”(little effort – high impact)
- Big things have small beginnings: “High Frequency Testing” (reach your goal in many small steps)
- Far too popular, unfortunately: “The dead zone” (great effort, but not well thought-out – hardly any impact)
- High-contrast path to the goal: “High Impact Testing” (great effort, but cleverly thought-out – high impact)
Which test field are you testing in?
Even though many optimizers dream of “low hanging fruit,”it usually remains just that – a dream. Regrettably, a great deal of testing is done in blind confusion, incorporating technical errors, wasting money and resources and failing to acquire knowledge. In this case, the “high frequency testers” are already ahead of the game, as they enjoy support from many, small micro- conversions. In this way, they aren’t pushing conversions to breath-taking heights from one minute to the next, but they learn from every single action and thus approach the goal step by step.“High impact testers,”are particularly clever, however, as they generate high contrast with well thought-out concepts for testing. High contrast leads to a change in user behavior and thus to a high conversion rate impact.
If your intent is a respectable impact for your conversion rate, then draw up clever, well thought-out test concepts! Simple, isn’t it?
(All previous Conversion Whiteboards videos are available on our YouTube channel)Start structuring your Experiments today!
Try iridion for free as long as you want by signing up now!
Create your free Optimization Project
Growth Hacking vs. Optimization – What’s the difference, anyway?
Do Growth Hackers dig deeper?
There seems to a consensus that a growth hacking approach is more holistic and starts at a higher point on the value chain. Like conversion optimizers, hackers optimize onsite, but they also look closely at traffic and the funnel. They ask the hard questions to get beneath the surface at the deeper mechanics of user acquisition and monetization.Optimizers are in it for the long run
To some extent, it’s just a matter of naming. Successful conversion optimizers will no doubt dig deeper as well, examining where users come from. Optimizers have been successfully optimizing for ten years or more, and the term growth hacking is newer, having ridden in on the Silicon Valley start-up wave. Also, some people have an inborn resistance to the phrase ‘growth hacking’: it can sound too shiny or self-promotional. Optimization is more conservative, slower perhaps, but substantial and sustainable.Hackers are more product people
It might be clear already, but the difference between the two approaches also have something to do with their origin. Growth hacking springs from the startup culture where everyone ‘does product’. More traditional organizations (such as big corporates)might be wary of the approach. Actually, there is a certain risk that a jump in conversions from hacking will just be a blip on the chart, a surge in metrics that falls off just as quickly. More than images of foosball-playing millenials, this is what the corporate world fears most about a growth hacking approach. Conversion optimization is tried and tested, if perhaps slow at times, and is likely to be more popular in larger or more conservative organizations. But at the end of the day, it doesn’t matter whether it’s called optimization or conversion, just that it works for you and your company. This post is an excerpt from our Masters of Growth Show. Check it out for actionable advice on how to tackle your most important challenges as an optimizers. Also leave us your questions on twitter using #mog to have your questions answered as well. The show is also available on YouTube and Apple Podcast.Start structuring your Experiments today!
Try iridion for free as long as you want by signing up now!
Design your site now
These 3 growth hacking and conversion secrets are missed by 90% of companies
1. Growth hacking is only for startups and tech companies
This is a common misconception on both sides of the start-up/corporate fence. We’ve addressed this question in depth and basically came to the conclusion that everyone can benefit from growth hacking, and optimization too for that matter. If your company gets stuck here – if there is a voice inside your organization that seems to be calling out, “But wait, we can’t be hacking, that’s for start-ups!” – this is the first step.2. Simply doing A/B tests *is* optimizing
This is another area where a certain way of thinking could be limiting your company from being all it could be. Optimization (and growth hacking) are not possible as a side task from one individual. Neither is an ‘A/B test pilot project‘ going to bring the results your company is hoping for. You’ll need an department to work on these tasks, or even a dedicated resource, for continuous A/B testing and mid- to long-term success in optimization.3. Your company is already (almost) growth optimized
Every organization is different – and not every element of corporate culture supports growth and optimization. If you’re looking to improve your company’s growth, start with the mindset. An overall approach that does not enable or even permit constant optimization is not conducive to growth. So start by asking the hard questions and looking closely at your company’s culture. This post is an excerpt from our Masters of Growth Show. Check it out for actionable advice on how to tackle your most important challenges as an optimizers. Also leave us your questions on twitter using #mog to have your questions answered as well. The show is also available on YouTube and Apple Podcast.Start structuring your Experiments today!
Try iridion for free as long as you want by signing up now!
Create your free Optimization Project
Will new checkout technology give us a boost or give us a headache?
Optimize for the customer!
When you’re looking to grow, it all depends on your relationship with the customer. Obviously, you want to maintain and deepen this relationship. So if new technology can help you improve the quality of your customer relationship, what are you waiting for? You should be welcoming it.Make it easy to buy but don’t lock yourself in
Some checkout methods such as Apple Pay are very strict about the details, such as the design and placement of the ‘Buy’ button. In this case, a dedicated optimizer has to balance the needs of the customer (who might want Apple Pay) with the requirements of optimizing (which might mean testing different designs or button formats). At the end of the day, it comes down to the question: Is this solution making it easier for a customer to do what she already wants to do? Our goal as optimizers is to leverage customer needs with technology, and if a certain solution does that, we should at least look into it.What about AI and chatbots – aren’t they going to take away my job?
Again, selling better means better understanding your customers and their needs. Do you really think chatbots or machines are going to be able to do that any time soon? It seems unlikely that any AI technology will seriously replace human contact in a customer-facing context. On the other hand, if you manage to operationalize the customer contact and sales process into a chatbot, you may be able to scale the process to your advantage. In the next few years, this area could become very interesting for optimizers. Ask yourself how you can reproduce the effect of building great relationships with customers. This post is an excerpt from our Masters of Growth Show. Check it out for actionable advice on how to tackle your most important challenges as an optimizers. Also leave us your questions on twitter using #mog to have your questions answered as well. The show is also available on YouTube and Apple Podcast.Start structuring your Experiments today!
Try iridion for free as long as you want by signing up now!
Create your free Optimization Project