2023 Experimentation Elite Winter Conference

Jim Sterne – Experimenting with Generative AI

1. Don’t trust Chat GPT with numbers and statistics, as it often makes up numbers because the answer to your question does not exist in the information it has been fed and it cannot 100% predict the answer.

2. Rather, teach Chat GPT like it is a person, using prompt engineering: tell it who it is, what you want that person to do, and why they need to do it.

3. Use it for qualitative research by feeding it data through gptforwork.com and using it to pick out themes and trends.

Bhav Patel – Past, Present and Future of CRO

1. Experimentation has come a long way – we can now scale at low cost due to technological advancement, most companies are adopting experimentation, and we no longer buy into false promises of huge returns based on small changes such as button colour

2. However, statistical significance in experimentation is still prioritised over meaningful significance, where we chase a particular number that only really has minimal output

3. Proper research is still not at the heart of CRO, we instead tend to try anything until we find something that works rather than having structured and data-driven experimentation

4. You should base your confidence level on your appetite for risk, not necessarily at the 95% standard. For example, you can afford to take more risk with an experiment on a homepage than on a checkout page

Jonny Longden – How Can Experimentation Become More Strategic?

1. “Conversion rate optimization” is a limiting framework as it often focuses too much on UX optimisations rather than tackling bigger problems and can cause us to ignore learnings as long as we have our desired metric movement. Instead we should use a “strategic decision support” framework

2. A strategic format for testing is to begin with a theoretical question, break it down into strategic hypotheses, and break each of these down into functional hypotheses. From each of these functional hypotheses we design tests to draw observations that inform the overarching theoretical question

3. Most companies doing experimentation fail in breaking down ideas into smaller and more manageable things to test, making their experimentation less strategic

Hannah Tickle – Avoid Signalling Problems: How Trainline Decides What Track to Take

1. Implement AA testing alongside AB testing to validate your experiment set up and gather baseline data to put your decisions into context

2. Compelling results can still be misleading, as small uplifts can be construed as statistically significant when working with big data. To mitigate this we need to rely on baseline data information to inform our testing decisions

Lucas Vos – How to Bust Myths About Experimentation That Block Your Experimentation Programme

1. Experimentation might sometimes test the obvious, but it might also show you unexpected results. Even if it is validating something likely to be true, it can prevent loss in the future and by not focusing too much on winning experiments, you can learn from every experiment

2. Experimentation might seem like too much work to others, but if you integrate it into big projects rather than just adding it to the top of the pile of tasks to do, it’s not too much work. You can achieve this by showing its added value in decision making, as a service to the rest of your organisation

3. Experimentation might be expensive, but the cost of assuming you’re right without first testing can be greater. Plus, not experimenting can also waste time as well as money

4. Experimentation seems risky, but through effective planning you can mitigate certain risks before and stick to the four-eyes principle

Mirjam de Klepper – Finding the Heart in CRO: A Guide to Ethical Practices and Office High-Fives

1. Ethics is subjective; some people might put the emphasis on intent and the morality of the action itself, whereas others might believe that morality is determined by the outcome of the action

2. When testing, we need to consider the complete context in which people are operating when they use an interface, as experiments impact lives and influence decisions. Slight changes can have dangerous effects for example when working with an insulin injection software

3. Integrate ethics into your design model by adding an evaluation stage to consider what potential negative consequences your action might have, and if you are solving the right problem

4. Add a forecasting stage to consider alternate personas to your main target audience and how changes might affect them. For example, what if your product falls into the hands of an anti-persona with bad motivations? What about the impact of your changes on non-human personas such as the surrounding environment?

5. Prioritise taking consistent action by conducting user research, social listening and customer care to understand the impact of your experiments.

Ben Labay – Experimentation ROI ‘system’ for Ecom vs SaaS

1. There is currently a lack of metrics to measure experimentation and therefore a lack of accountability in the field, with 62% of respondents to one survey stating that they didn’t have measurements for their experiments

2. Consider goal tree mapping as a structure for centralised experimentation programmes, to allow your team to build experiments that are relevant to your company growth strategy. Particularly helpful if you lack clarity on which teams are focused on which KPIs

3. Velocity is his 1 magic bullet to growing experimentation within an organisation, as the more testing you do, the more you can clean up

Annamarie Klaassen –  Treat Your CRO Programme as a Product

1. Just as you have to convince customers of your product, you need to convince your stakeholders of your CRO programme, particularly with more established companies where CRO will often have not been part of the programme from the start

2. Consider product marketing and messaging – determine the value proposition of your CRO programme by asking your stakeholders what they are working on and what they are struggling with, then find an angle of how your CRO programme can help these pains

3. Keep stakeholders informed about results and updates through readable and structured progress reports, using simple imagery such as infographics

4. Have cross-team communication, don’t work in silos – have a CRO channel for different teams

5. Normalise failure to encourage discussion and learning – have an annual failure party!

Daphne Tideman – What to do When Experiments Fail

1. Before your experiment, define a clear optimisation strategy including overarching goal, area you are testing, duration, measure of success, budget / resources and how you plan to test it. Conduct a pre-mortem to troubleshoot what might go wrong or be unexpected, and what you will do in those scenarios

2. Be wary of peeking as this might get you overexcited or panicked, and can bias the experiment. However, peeking can add value if you are checking everything is working correctly and making sure the test isn’t a waste of time. Just be careful to not share the results if you do peak

3. If the experiment does fail, create theories of why this was the case, rather than just moving on to the next one

4. Don’t just manage your stakeholders, align with them by speaking their language. Try to understand their pain points, what they are trying to achieve and how testing and failing fits into this, and use this framework to communicate about experimentation

Michael Aagaard – The Psychology of Disappointment

1. Focus on your customer expectation gap – whether the reality matches up to what they expected. This is what is often described in bad reviews, you can scrape them and upload to Chat GPT to draw out categories and themes

2. Conduct expectation gap analysis to troubleshoot what might go wrong – comparing pre-test expectation measurement and post-test experience measurement

3. Focus on making customers feel secure in their choices when they are purchasing, from the angle of disappointment – they might be thinking, how do I know I won’t be disappointed? What if I am disappointed?

4. In summary, disappointment is a conversion killer, mind the expectation gap