A trial of a new spool usually goes like this: a small quantity arrives, somebody uses them, everyone agrees they seem fine, and a larger order follows. Then, six months in, something emerges that the trial never had a chance to reveal.
Why small trials mislead
Three specific reasons:
- Quantity too small to see variation. Five spools tell you a supplier can make five good spools. Variation shows up in the fiftieth.
- Duration too short for cumulative effects. Creep, bore wear, joint loosening and UV all need time. Nothing that depends on cycles can be assessed in a week.
- No comparison baseline. “Seems fine” is not a measurement. Against what?
What to decide before starting
Write these down first, because they determine everything else:
- What question is the pilot answering? Fit? Service life? Whether it reduces a specific problem? A pilot that tries to answer everything answers nothing.
- What would make it a pass? A measurable criterion, agreed in advance.
- What would make it a fail? Also worth stating, so the decision is not relitigated afterwards.
- How long does it run? In cycles or months, not “until we know”.
How many
Enough to see variation and enough to survive the losses the pilot will incur. Practically, that usually means dozens rather than five, and enough that some will get lost or damaged without invalidating the trial.
If the quantity is limited by cost, say so and design the pilot around it — a short pilot with a narrow question is better than a long one that gets abandoned.
Running them alongside the incumbent
The single biggest improvement over the usual approach. Do not swap the fleet over; run both at once and compare:
- Use them on the same machines and the same product where possible
- Mark them so they can be identified later — the methods in our markings article
- Record which is which at every inspection point
- Resist telling operators which is which if you are assessing anything subjective
That last one is optional and slightly awkward, but it removes a real source of bias.
What to measure
Depends on the question, but the useful measurements are:
- Failures by type and by cycle count — the tally from our records article
- Dimensional drift on identified spools — the method in our drift article
- Machine interventions — how often somebody has to do something
- Product quality metrics if the spool is implicated
- Handling time if changeover speed matters
Machine interventions and handling time are usually the most informative and the least often measured.
What not to do
- Run it during an unusual period. A pilot over a shutdown or a seasonal peak tells you about that period.
- Change something else at the same time. If you are also changing tension settings, you cannot attribute anything.
- Let the supplier run it. Their incentives are not yours, and operators behave differently when watched.
- Stop early because it looks good. The cumulative failures are exactly the ones that have not happened yet.
Deciding
At the end, go back to the criteria written at the start. If they were met, adopt. If not, do not. If the result is marginal, the honest answer is often to keep the incumbent, because switching costs are real and a marginal improvement rarely justifies them.
What we suggest
Tell us what you are trying to establish and we will suggest a quantity and a duration, and we will usually suggest fewer spools for longer rather than more for a short period. For our hollow board spools we would rather you ran fifty for six months than ten for two weeks, because the six-month version produces a decision you can stand behind.