Running business experiments requires two main things: a control group and a way to get feedback.
Control Group: This is a group that stays the same to compare against the group you’re testing. It’s important to choose this group randomly and make sure they don’t mix with the test group.
Feedback Mechanism: You need a way to see how customers react to your changes. There are two types of feedback:
Objective: This tracks actual actions, like purchases, showing real customer behavior.
Perceived: This measures how customers think they will react, often through surveys or focus groups.
While both types of feedback are useful, focusing on objective feedback can give you a more direct understanding of customer behavior, which is often the main goal for most businesses.
Rules for Running Experiments
As with every bet in life, the best ones start with the low-hanging fruit—experiments that are easy to implement and give quick, clear insights. Such as raising/ lowering a price or sending out an offer—and observing users’ reactions.
However this can’t be the only principle around which experimentation should happen. Let’s dive deep into a few ones that can enhance this process
Act Quickly and Make Decisions Quickly
- When testing things, it’s best to look at what people actually do, like buying stuff, instead of what they say they’ll do. This way, you can see right away if your changes are working.
- Focus on tests where you can see how people react on the spot.
For example, if you’re trying to get more people to invest through your service, start by looking at how you’re bringing in new customers rather than trying to make existing customers stay longer or spend more. You can see the results of new customer tests right away, but seeing changes in how long customers stay or how much they spend takes more time.
Use What You Have
- Look for simple tests you can do with the resources you already have.
- For instance, if users like learning when they invest in your platform, try sending them an article instead of making a whole new section for it.
Big tests can give us a lot of information, but small tests can teach us just as much without using too many resources at once. This saves time and money, allowing you to do more tests in the future.
3. Start with a proof-of-concept first
Theoretically, researchers change one variable at a time so that they know what caused the outcome. In a business setting, it’s important to first establish proof of concept. Change as many variables in whatever combination you believe is most likely to get the result you want.
Moving on from Fintech examples, If Big Bazaar wants to test the best way to shift demand from national brands to its private-label brands, it should increase the prices of the national brands and decrease the private-label brand prices. Then establish that demand can be shifted; they can refine their strategy by varying each of the prices individually.
4. Break Down the Results
When you have a lot of customers split into test and control groups, it’s like running several tests at once.
For instance, if you have both men and women in your test, check how each group reacts separately. If you’re looking at investors, consider how long they’ve been investing to gauge their experience.
Different things work for different people. So, when you get your test results, look into smaller groups within your big test. This way, you won’t miss out on noticing important details. Looking at the big picture alone might make you think your test didn’t change anything, which might not be true.
It’s best to always test and focus on specific small groups to really understand what’s effective.
5. Think Creatively/Out of the Box
If you don’t consider “what-if” scenarios, you might miss out on big improvements from your tests.
Take Tickertape, a finance data analysis platform, as an example. They wanted more people to use their Stock Market Simulation game. So, they increased the number of daily winners from 2 to 200. This meant more users had a chance to win, even though the prize for each was smaller. By doing this, 90% more current users started using the game.
6. Track What’s Important
It’s important to measure things correctly to see the full effect of your tests.
For example when testing an onboarding flow for activation, you can split the test to see the amount of communication/knowledge you will have to provide the user to activate them.
Some users receive 5 PNs/emails catalogs over 1 week (intended to educate them about the platform), whereas another randomly selected group received 3 emails/PNs over the same time period (intended to push them to take just one action). Then see how each user set behaves over a period of time. Because an educated user might stay with you longer.
Always consider how one change might affect other areas or how quick fixes might have long-term effects.
7. Look for what comes naturally rather than force fitting
The goal is not to conduct perfect experiments; it is to make better decisions.
- The trick is to spot and use natural tests where groups are formed by chance due to external reasons, rather than being handpicked by you
- For instance, when offering discounts, it’s smart to check if those discounts actually affect the people you’re offering them to.
- Another example is using location to split groups, like comparing how big city (Tier 1) and smaller city (Tier 2) customers react to the same prices or ads in their local languages. But remember, location isn’t always the key factor that makes groups different.
These are a few rules that you follow to run better experiments. There are some good resources out there to become better at this. Martin’s newsletter is highly recommended by me.
