An odm development pilot run is easy to request and hard to specify, and the gap between those two states is where most pilot programmes spend money without buying information. A buyer asks for twenty units, receives twenty units, and then discovers that those twenty units cannot answer the question that justified them. The cause is rarely a factory refusing to help. It is that the pilot was specified as a quantity when it needed to be specified as a decision.
A pilot run answers one question. Which question depends on what has not yet been established, and in this category the things not yet established are largely the things the published data does not contain: the frame weight for anything other than one model, the sensor model, the field of view, the video specification, the microphone figures, the lens performance, the ingress report numbers and the seal material are all Not published in our catalogue. The catalogue publishes no certification row for any model and no certificate number, so the scope that applies to a given model in a given market is written material to request and file before the order rather than a mark a buyer can read off a page. So an odm development pilot run here is not a formality before production. It is the mechanism by which a specification stops being a description and becomes something a purchase order, a warranty clause and a compliance file can all rest on.
ODM development pilot run: three pilots that get confused with each other
The single most common specification error in this category is treating a pilot as one activity. It is three, with different quantities, different acceptance bases and different outputs, and a programme that runs one when it needed the other loses both the time and the money.
The process pilot is a short run whose purpose is to find out whether the line can hold the tolerances and whether the first-article standard is achievable. Its output is a yield figure and a defect list. Its acceptance basis is consistency between units, not performance of the product. A process pilot that comes back clean tells you the line is capable; it tells you nothing about whether the design meets the requirement, because the requirement was never tested against it. This is the pilot that catches a manufacturing problem, and it is the one most factories are set up to run quickly.
The evaluation pilot is a short run whose purpose is to produce units against which the buyer's own acceptance criteria can be measured. Its output is evidence: a weight distribution across a sample, a camera evaluation at stated conditions, a sizing sweep, an acoustic measurement, an ingress retest. Its acceptance basis is criteria the buyer wrote, because the published data does not contain them. This is the pilot that closes the specification gaps, and in this category it is the pilot that actually matters, because the gaps are where the risk sits.
The field pilot is a small run placed in the customer's actual environment with actual users. Its output is behavioural data: a return rate over a period, an interaction pattern, a week of real use in a classroom or a warehouse or a retail floor. Its acceptance basis is what happens in use, not what a bench reports. This is the pilot most programmes skip, and it is the only one that can tell you whether the product solves the problem, as distinct from conforming to a specification that the buyer wrote themselves.
The three are sequential in a well-run programme and independent in a badly-run one. A common failure is running the process pilot, being pleased with the yield, and treating that result as if it were the evaluation pilot's. The two answers are not interchangeable, and a buyer who accepts the first as the second has closed no gap at all.
The cost of that confusion is not abstract. A process pilot is cheap, quick and comfortable, because its criteria are internal to the factory and its result is usually favourable. An evaluation pilot is slower and more likely to produce an uncomfortable answer, because its criteria belong to the buyer and its instruments may not be the ones the factory normally uses. Programmes drift towards the first kind for exactly that reason, and then discover at the pre-production review that the specification questions are still open with the tooling already committed. Deciding which pilot a question requires, before anyone builds anything, is the cheapest hour in the entire programme.

Choosing the pilot your open question actually requires
The selection is easier than programmes make it, because the question is almost always already written somewhere in the buyer's own documentation. Start from the acceptance criteria, not from the unit count.
Step one: write the open question in one sentence. If it cannot be written in one sentence, the pilot is carrying more than one question and will answer neither. "Is the battery capacity adequate?" is a question. "Are the battery, the camera and the acoustics right?" is three questions with three different sample designs and three different failure consequences.
Step two: mark each criterion by evidence status. Published data is already established and does not need a pilot. Data that is claimed but unevidenced needs a pilot. Data that nobody has stated needs a written criterion first, because there is nothing to test against. In our own catalogue that middle column is long: the acoustic performance, the lens performance, the ingress evidence and the camera specification are all claimed or implied rather than published.
Step three: ask whether the question is answerable on a bench. Weight, dimensions, charge time, interface behaviour and ingress are bench questions. Sizing across a range of head shapes is a laboratory-plus-panel question. Reminder fatigue, an audio environment with real ambient noise, a long working session and a return rate are field questions and cannot be compressed into a laboratory schedule, whatever the supplier is willing to attempt.
Step four: derive the unit count from the claim, not from a round number. A weight spread across units from more than one batch needs enough units to see the spread. A camera comparison between two configurations needs a small number but needs the same configuration on both sides. A class or a site needs a class or a site. Round numbers such as ten or twenty are almost always inherited from a budget rather than derived from a claim, and they are the most common reason a pilot result gets explained away later.
Step five: decide in advance what happens if it fails. A pilot with no defined failure consequence produces a pilot that is interpreted. That is not a criticism of the factory; it is what happens to any result that arrives without a threshold attached. The consequence should be written before the units are built, because afterwards the buyer is commercially attached to one side of the answer.
There is a sixth step that experienced buyers add and that most suppliers do not ask for, which is to state where the pilot units will physically come from in the schedule. A pilot drawn from a pre-production build competes with tooling, firmware and documentation work for the same days. A pilot drawn from the first production batch competes with nothing, but arrives later and cannot influence the design. That trade-off is a programme decision, not a factory decision, and it belongs in the quotation rather than in the first email after the purchase order is signed. Where a pilot is scheduled between a design freeze and a compliance assessment, expect the documentation to be on the critical path rather than the build, and ask which of the two is actually driving the date.
Writing the pilot protocol before the units are built
The protocol is the artefact that turns a run into evidence. Six elements belong in it, and a programme that writes them before the build gets a result it can quote; a programme that writes them afterwards gets a story.
The criteria, numeric and thresholded. "Battery performs acceptably" cannot fail. "Rated runtime within the published figure under the stated workload, measured on units from the assessed configuration" can. Every unevidenced parameter in the specification should appear as a line in this table, with a method, a threshold and a decision attached.
The unit selection rule. Which batches, which positions in the batch, which configuration. Units drawn from a single run at the end of a long build are the weakest possible sample, because they share everything that varies batch to batch. Where the pilot is intended to demonstrate process capability, that is exactly the sample you want; where it is intended to characterise the product, it is exactly the sample you do not.
The measurement method and instrument. Two teams can produce different numbers from the same units and both be right, because one measured after a warm-up cycle and one measured from cold. The method belongs in the protocol, including the conditioning, the workload, the ambient condition and the number of repetitions.
Conditioning deserves its own line because it is where measurement disputes are usually won. A cell-related result depends on how long the unit has been running, what it has been doing and how long it rested before the measurement started. A surface temperature result depends on the ambient condition and on how long the workload has been running. An acoustic result depends on the environment it is measured in, and an open bench is not the same environment as a street. None of these are exotic requirements; they are simply details that get decided by whoever measures first, and they should be decided by the protocol instead. Where a figure will end up in a technical file, record the instrument identification as well as the reading, because an untraceable measurement is not evidence in a due diligence review even when it is correct.
The sample size, with the reason written. Not "20 units" but "20 units, because the claim is a spread across a batch and a spread needs more than one unit per position". Writing the reason makes the number defensible when the result is challenged, and it makes it obvious when the number was inherited rather than derived.
The raw data, not the summary. A pilot report that gives a single figure without the distribution is a summary, and summaries are where favourable readings survive and unfavourable ones do not. Ask for the per-unit readings.
The reporting commitment. What the supplier commits to do with a result that misses the threshold: stop, adjust and re-run, report and continue, or report and hold. In our own position we will run a pilot against written criteria and report the outcome unchanged, including the outcome where the honest conclusion is that the published data remains the best description of the product after testing.

What a pilot run cannot establish in this category
A pilot is a bounded exercise, and the most expensive misunderstanding in this category is treating it as a complete one. Four things it will not do, however many units it produces.
It will not demonstrate production consistency at volume. A pilot establishes that the process can do something. Volume capability comes from run-at-rate evidence, capability data across a sustained window and a control plan, none of which a short run produces. A pilot of twenty units is an indication and nothing more, and a supplier who presents a pilot yield as a production yield is making a claim the data does not support.
It will not show behaviour over the life of the product. Cell degradation, seal ageing, hinge wear, battery cycle drift and frame fatigue are all time-dependent. A pilot measured in week two says nothing about week forty. Where longevity matters, the evidence is a separate programme with its own schedule, and it should be started early precisely because it is slow.
It will not fix the documentation position. A technical file, a declaration of conformity and a certificate position are a function of documentation, not of testing. A pilot can produce measurements that support a file; it cannot produce the file. The catalogue publishes no certification row for any model, so a pilot has nothing to work from on that point and no pilot run changes what is missing. Only a document filed against a named model string and a named market does.
It will not tell you whether the market wants the product. A pilot can tell you the unit conforms, works and survives the run. Demand is established by a commercial test with a different structure, a different cost and a different risk profile, and mixing the two in one programme usually means the commercial question gets answered by the engineering answer.
There is a fifth limit specific to this category, and it is worth naming directly. A pilot built from pre-production units against pre-production criteria will reproduce the gaps it was designed to close, because the units themselves are built from the same unevidenged assumptions as the specification. That is not a reason to skip the pilot. It is a reason to fix the model code and the assessed configuration before the run, so that the result describes the configuration that will actually ship.
And there is a sixth, which is the one most often discovered late. A pilot demonstrates the units in front of it. If a component is substituted after the pilot — because the original becomes unavailable, because a supplier changes its specification, because a second source is qualified for cost reasons — then the pilot's result describes a product that no longer exists, and the gap reopens silently. This is why a change-control rule belongs in the pilot agreement rather than in the general quality manual: a substitution that was reasonable at the time can still invalidate the evidence, the assessment and the technical file simultaneously. The rule that protects a programme here is simple to state and easy to skip. Any change to a component in an assessed configuration triggers a documented review of what evidence the change invalidates. The review is short. Skipping it is what creates the argument six months later about whether the certificate still applies to the unit in the carton.

Questions to ask a supplier before you commit to a pilot run
The list below is the one we would want answered before our own purchase order was drafted. It is written so that a supplier can answer it directly, and a supplier who answers it directly is telling you something about how the programme will run.
- What does this pilot establish, and what is it explicitly not establishing? A clear answer names both sides. An answer that only names the first is a sales answer.
- Which criteria are published, which are unevidenced, and which are unknown to both sides? This forces the specification gaps into the open at the point where they are still cheap.
- How many units, from which batches, in which configuration, and why that number? The reason matters more than the number.
- Is the configuration the one that will be assessed for compliance, and is the model code fixed? If the answer is no, the result will not transfer to the shipped product.
- What is the measurement method, the instrument, and the conditioning? If two suppliers would measure the same property differently, the results are not comparable and the criterion needs rewriting.
- What is the acceptance threshold for each criterion, in numbers? Adjectives are not thresholds.
- Will you provide per-unit raw readings rather than a summary? A distribution supports decisions; an average hides the unit that fails.
- What happens if the pilot misses the threshold? Stop, adjust and re-run; report and continue; or report and hold. Each is legitimate. Not knowing which one applies is the problem.
- What is on the critical path for this run? Usually a tooling iteration, a component availability issue or a documentation dependency, and each of those behaves differently on the schedule.
- How many tool cycles do you expect, and what triggers another? A first tool on a new shape is rarely right first time, and the number affects both cost and date.
- Which decisions are frozen at this point, and who maintains the decision log? In an ODM the lost decisions are the ones each side assumed the other had recorded.
- What is the unit cost at the planned volume, broken down? A prototype is the first point at which a real number can be produced rather than an estimate, and that number is often worth more than the pilot itself.
Twelve questions is the working minimum, and the fourth and eighth are the two that most often reveal the real state of the programme. A supplier who can answer both precisely is a supplier whose pilot result can be relied on in a contract discussion. A supplier who cannot is telling you that the pilot will be a demonstration rather than a test, which is a legitimate outcome as long as it is a decision rather than a surprise.
There are four answer patterns worth treating as red flags, and they are worth naming because each one looks reasonable in isolation. The first is a unit count offered before any criteria exist, which means the count came from a price list rather than from a claim. The second is a threshold described in qualitative terms — excellent, as expected, in line with the target — which means the result will be characterised rather than measured. The third is a pilot offered on units that will be built after the compliance assessment, which inverts the useful order: the pilot should inform the specification that gets assessed, not confirm a specification that has already been locked. The fourth is a refusal to state the failure consequence, usually framed as flexibility. Flexibility applied to a passing result is a commercial virtue; flexibility applied to a failing result is an unresolved dispute waiting for a date.
The reverse pattern is worth noticing too. Suppliers who answer question three by explaining the sampling logic, and question six by offering to agree the method before the run rather than after, are usually working from a real process rather than a script. In a due diligence conversation that behaviour is itself evidence, and it is more informative than a fast answer to question one.
Where our published data ends, and what we will commit to in writing
Our own published position sets the frame for this article, and it is thinner in some places and firmer in others than a marketing document would suggest. We publish the rated runtimes, the colour options, the structural features, the charging interfaces, the charge times, the battery capacities, the camera tiers and the case capacities for our range. We publish a frame weight for one model in the catalogue — 158 g for the MT6 — and Not published for the rest. The sensor model, the field of view, the video specification, the microphone figures, the lens performance, the ingress report numbers and the seal material are Not published. On compliance, nothing in the catalogue text names CE, FCC or RoHS, and no row ties a conformity statement to a model, so the applicable scope for any model in any market has to arrive as written material attached to the order rather than as a line the buyer can read from this document.
We do not publish our pilot lead times, our development costs, our tooling costs, our minimum order quantities or our production capacity, and a request for those numbers will get an honest answer rather than a figure. What we will commit to in writing is narrower and more useful: we will work through the twelve questions above against a specific design, tell you which of them we can answer today, which require a build, and which are the ones most likely to be the reason the programme slips, and we will report a pilot result whichever way it falls.
The practical position is that a pilot in this category is worth most where the published data is thinnest — the frame weight, the camera specification, the acoustic performance, the lens performance and the ingress evidence. Those are the gaps a pilot can close, and they are the gaps a pilot should be aimed at rather than spent on re-confirming figures that are already published.
Related reading
- ODM development
- The prototype phase in an ODM programme
- Development process stage by stage
- Can I start with a small test order
Tell us what the pilot is meant to establish and we will say which kind it is, and what it can realistically answer.



