How we check that a change worked

After this guide, you know how InfraStudio checks a change after it is on your product, and what each result means.

This is a measurement, not a proof. A number can move for other reasons, such as a holiday, an advertisement or a bad week.

What it checks

Every suggestion names one number that the change should move. For example: the share of visitors who finish checkout. You see it under The number to watch.

When the change is on your product, InfraStudio notes the day. Then it reads the same number again, the same way, two times:

• after 7 days;
• after 14 days.

Each time, it compares two periods of the same length. For the check after 7 days, it compares the 7 days before the change with the 7 days after it. The day the change went live is in neither period. For the check after 14 days, both periods are 14 days long.

The data needs two more days to settle in your analytics. So each check happens 2 days after its period ends.

Who reads the number

The Product analytics review, or Error triage for its own suggestions, reads the number. It does this in the run that happens anyway. It does not start a run of its own.

The AI only reads the two numbers, and how many visits, sessions or events each one is based on. The AI does not decide the result. InfraStudio decides it with the rule below.

The four results

• Better. The number moved in the direction that is good for your product. For example: "Checkout conversion went from 2.1% to 2.6% in the 7 days after the change."
• No clear change. The number moved a little, or not at all, or by an amount that could be chance.
• Worse. The number moved in the wrong direction. The suggestion shows it. Your Telegram and Slack channels get a message too.
• Can't tell yet. There is not enough to tell. InfraStudio does not guess.

You see the results on the suggestion, under the 7-day and 14-day lines.

The rule

A result of Better, No clear change or Worse needs all of these:

1. Each period has at least 100 visits, sessions or events behind its number.
2. The number was not 0 before the change.
3. For a percentage, such as a conversion rate, the number moved by at least a tenth of its old value. It also moved by more than two standard errors. A standard error is how much a percentage moves by chance, given how many visits are behind it. A small shop needs many visits to show a small change.
4. For any other number, such as errors a day, the number moved by at least a fifth of its old value.

If the number moved enough, the result is Better when it moved the way the suggestion wanted. It is Worse when it moved the other way. If not, the result is No clear change.

When it says Can't tell yet

It says this, with the reason, when:

• there are too few visits, sessions or events in one of the two periods;
• the number could not be read for both periods;
• the suggestion has no number to watch, for example a change that only cleans up the code;
• the source of the number is not connected;
• another change went live in the same weeks and is about the same thing, so the number cannot show which change moved it;
• the review found that tracking changed or that the two periods cannot be compared;
• no review read the number within three weeks.

A small product has little traffic, so this result is common. It is the honest one.

What it does not see

InfraStudio knows only the changes that it made live. A change that you made by hand, an advertisement or a holiday is not in its rule. Read the result as a hint, and look at your analytics if the change matters.

To set up the reading, connect your analytics. See Connect analytics and Sentry.