AI pilot step by step: how to test a solution before rolling it out company-wide

Between “this might work” and “we are rolling it out company-wide” there is a step many companies skip: the pilot. Without it, the decision to implement rests on a presentation and the enthusiasm of the first few days. With it, it rests on what happened on your team’s real tasks.

A pilot, a demo and an implementation are three different things

• A demo shows that something is possible - on prepared examples, under good conditions.
• A pilot checks whether it works for you - on normal tasks, with people who also have other work to do.
• An implementation extends what worked in the pilot to more people or more tasks.

Calling a demo a pilot is an easy way to make a decision based on the best possible scenario.

The pilot card

Before the pilot starts, write down on one page:

• the use case - one, described specifically: who does what, with which materials, how often,
• what the pilot does not cover - so that by week three it does not grow into a project,
• an owner on the company side - a person who is responsible for how it runs and has time for it,
• the participants - usually two to five people,
• the starting point - measured before the start,
• success criteria - related to the starting point,
• a decision date - most often after four to six weeks,
• stop conditions - what would have to happen to end it early.

I describe how to measure the starting point and set the criteria in the article on measuring the effect of an AI implementation.

Week zero: preparation

• prepare the materials the tool will work on and check that they are up to date,
• write short instructions for participants: how to use the tool for this task, what not to paste in and how to check answers,
• measure the starting point on normal work,
• train the participants on their own examples, not with a generic demo.

Week one: learning

In the first week you do not judge the result. People are learning the tool, the instructions turn out to be incomplete, questions come up that nobody foresaw.

What matters most at this stage are short, regular conversations with participants and writing problems down. Every problem is a fix to the instructions or the materials - better now than after rolling out to the whole team.

Weeks two to four: work and measurement

• participants use the tool in their normal tasks,
• measurement is done the same way as for the starting point - the same measures, the same way of recording,
• the tool’s mistakes are recorded together with the stage at which checking caught them or failed to,
• the pilot owner reviews the record once a week and steps in if something deviates from the plan.

Decision week

At the end you compare the result with the starting point and with the criteria written in the card. Three decisions are possible:

1. Go further - the criteria are met, we extend it to more people or tasks.
2. Change the scope - something works, but not the way we assumed. A new card is written, with a revised scope and a new decision date.
3. Stop - the criteria are not met and there is no simple way to change that.

The third decision is not a failure. A pilot that showed after five weeks that a use case does not work has saved the company months of an implementation that would have delivered nothing.

Who to invite to the pilot

• people who do this task every day - not only those most interested in AI,
• at least one sceptic - if they are convinced, the team will accept the result more easily,
• an owner with authority to decide - without one, pilots usually fade out because nobody has time to close them.

Why pilots settle nothing

• the scope was too big, so after six weeks there is progress but no result,
• there was no starting point, so there is nothing to compare with,
• nobody owned it, so the pilot goes on until everyone forgets about it,
• the verdict was given after the first week, in the middle of learning,
• the task, the tool and the instructions were all changed at the same time.

A pilot is usually the third step after talking to the team and choosing a use case - I describe the whole order in the article on where to start with AI in your company. I prepare the pilot card as part of an AI audit, and we run the pilot itself as part of AI implementation.