How to measure the effect of an AI implementation so it is more than an impression

A few weeks after an AI tool is introduced, the same question usually comes up: was it worth it? And usually the same answer: “it seems faster”. One person is delighted, another says they correct more than they gain, and a third stopped using the tool in the second week. Nobody is right, because nobody has a number.

Good measurement does not need a reporting system or an analyst. It needs a few things written down before anything changes.

Choose one thing to measure

Measuring everything at once ends with measuring nothing. Choose one task the change is about and one measure that tells you whether that task is going better. For example:

• time to prepare a reply to a typical enquiry,
• time from a client’s request to the first substantive reply,
• the number of corrections a draft needs before it is sent,
• the number of repeat contacts about the same issue.

The measure should be directly tied to the task. “Team satisfaction with AI” is interesting, but it does not tell you whether the work is faster or better.

Measure the starting point without a reporting system

A plain spreadsheet and a week or two of normal work are enough. For each example you record:

• the date and type of task,
• the time from start to finish,
• the number of corrections or returns,
• a short note if the example was unusual.

Two rules matter most here. First, the person who does the work measures it, not someone watching from the side. Second, you measure normal work, not especially careful work. When a team knows it is being measured, the first two days are often faster than usual - which is why a week is better than a single day.

A dozen or so examples are enough to see the order of magnitude. This is not a statistic for publication, it is a reference point for a decision.

Measure time together with quality

“Faster” on its own is easy to achieve at the expense of quality. That is why every time measure should be paired with a quality measure:

• time to prepare a reply together with the number of corrections before sending,
• time to first reply together with the number of follow-up questions from the client,
• the number of requests handled together with the number of repeat contacts about the same issue.

If time went down but corrections went up, the change may only have moved work from writing to correcting.

Agree success criteria before the start

Criteria written down after a pilot tend to adjust themselves to the result. So write them down beforehand, ideally in the pilot card:

• what has to change for the change to count as a success - for example a shorter time to prepare a reply with no more corrections than today,
• what must not get worse - for example the number of complaints,
• when you make the decision and what the possible decisions are: go further, change the scope or stop.

I describe the whole pilot card and how a pilot runs in the article AI pilot step by step.

Compare under the same conditions

A result is credible when you compare like with like:

• the same kinds of tasks - not easy examples after the change against hard ones from before,
• the same people or a similar team,
• the period after learning, not the first week with a new tool - that week is learning, not a result,
• a similar time of year, if the work is seasonal.

Count the cost honestly

Time saved is half the calculation. On the other side are:

• licences for the people who actually use the tool,
• time spent preparing the implementation and instructions,
• time the team spends learning,
• time spent checking answers - without it the result is inflated.

If you are working out whether it pays off, write the assumptions next to the numbers: how many times a month the task is done, how much time is realistically saved each time, how long checking takes. An estimate with its assumptions written down can be checked and corrected. A number without assumptions is just a claim.

What the spreadsheet does not show also counts

Next to the numbers, it is worth noting a few observations: whether the team uses the tool without being reminded, which tasks went back to the old way of working and why, where the tool made mistakes most often. These tell you what to fix before deciding to roll it out further.

Five mistakes that spoil measurement

1. Measuring on selected, easy examples. The result looks great until the first difficult case.
2. Comparing with memory instead of a record. Memory always suggests things used to be slower.
3. Leaving out checking time. A draft produced in two minutes that takes ten minutes to read is not a two-minute draft.
4. Changing several things at once. A new process, a new template and a new tool at the same time - you cannot tell what worked.
5. No decision date. A pilot without a deadline turns into a state that continues because nobody decided otherwise.

Knowing what to measure is also where the order of introducing AI in a company begins. In AI implementation, measuring the starting point and comparing after the pilot are a standard part of the work, not an extra at the end.