Smart eyewear OEM/ODM manufacturer · Shenzhen, China +86 19835872450 sales@cynrin.com
OEM & ODM

Development Process: How Sample Validation Should Work

Sample validation is the gate most often treated as a formality and the gate that most often becomes expensive later. A programme arrives at the sample…

By CYNRIN Engineering & Sourcing Team/ Jul 24, 2023/ 16 min read
Development Process: How Sample Validation Should Work

Sample validation is the gate most often treated as a formality and the gate that most often becomes expensive later. A programme arrives at the sample stage with a quotation, a drawing, a firmware build and a unit on a desk. Everyone wants to move forward. The question that should decide what happens next is narrow and technical: does this unit, as configured right now, do what the frozen requirement document says it must do, and does it do it the same way the reference unit did?

This article is written for the brand-side buyer and the channel procurement manager who has to sign at that gate, and for the quality lead who has to defend the signature later. We treat sample validation as a document-producing gate rather than a moment of inspection. A stage is finished when it produces a document, not when a date arrives. If your programme leaves the sample stage without a signed verification record and a signed golden sample, you have not passed the gate — you have only visited it.

The catalogue behind this article covers twenty-eight models, and the way those models differ in published detail is itself instructive. Eight audio models are listed with model numbers such as LC-105-1 and K01, D11 and G07, YK12 and D12, and K02 and LC-105-2. Three sport models appear as YKC02, D17 and ZH12. Eleven camera models are published without the same naming consistency, three kids models are published without a model number at all, and three more entries carry no model number. Bluetooth version is published for fourteen of the models and not published for the eleven camera models or for A05, A06 and the bone conduction reference. Remote control shutter appears in the published detail on exactly two models, D17 and ZH12. IP54 appears on exactly two models, V06 and S7.

That asymmetry is the whole argument of this article. A catalogue that publishes a shutter feature on two models and does not publish it on the rest is a catalogue that can support a verification record, because it tells you which claims attach to which model string. A catalogue where every row claims everything is a catalogue that cannot support one. The difference between those two catalogues is not marketing effort; it is whether a written requirement can be turned into a written verification record at all.

Development process sample validation: what samples are for

A sample exists to answer one question: does the unit in front of me satisfy the frozen requirement, and is it representative of what a production unit will do? That is all. Everything else a sample is asked to do is a misuse of the artefact, and each misuse creates a specific downstream problem.

A sample is not a showroom unit. A showroom unit is a display asset, often configured to look better than a shipping unit, frequently running different firmware, sometimes running a different enclosure finish. If a buyer takes a show unit into a customer meeting, or if a sales team photographs one, none of that tells you whether your production units will pass your verification record. The showroom unit is a communication tool. It is not evidence.

A sample is not a substitute for the golden sample. This is the most damaging confusion in the whole process, because the golden sample has a different job. The golden sample is the reference definition of the boundary between acceptable and not acceptable. It is kept, marked, signed over, and used as the comparison target whenever a later unit — a second sample, a pilot-run unit, a line unit — needs to be judged. If you accept a sample as the reference and then the reference changes, you have no anchor. Every subsequent judgement becomes an opinion, and opinions cannot be audited.

A sample is not a warranty of what the first production batch will do. The purpose of a pre-production sample is to expose the difference between an engineering sample and a production sample. If your sample passes and your pilot run still fails, that is not a contradiction — that is the gate doing its job. The sample tells you the design works; the pilot run tells you the process works. Those are different questions and the sample should never have been asked to answer the second one.

A sample is also not the place where requirement changes belong. If someone wants to change the language pack, the button mapping, the charging behaviour, or the device name, that is a requirement change and it needs a change control route with a new version of the written requirement document. Verifying a changed unit against the old requirement and then declaring success is a way of losing the audit trail without noticing.

What a sample is for, stated as a working discipline, is this: the sample is the first physical evidence that the engineering work produced something that matches the written requirement. Everything downstream — the compliance file, the pilot run, the purchase order for mass production — assumes that this evidence exists and is recorded.

In a development process that runs through six gates, the sample gate sits fourth: requirements frozen, feasibility reviewed, engineering and tooling, samples built and verified, compliance file closed, pilot run before mass production. Its position matters. By the time units arrive, the requirement is already written and signed, feasibility has already produced a boundary, and tooling has already been paid for and verified. The sample gate cannot be used to renegotiate any of those three. It verifies.

We will hold the line on this: the sample stage produces a verification record and a golden sample, signed. If it produces an email saying the samples look good, it has not produced its output, and we will not allow the programme to advance to the compliance gate on the strength of an email.

AI-generated photograph illustrating development process: how sample validation should work — image 1 of 3

What the verification record has to contain

The verification record is the document that makes the sample stage auditable. It has to be specific enough that a different person, one who was not in the room when the sample was tested, could repeat the test and reach the same conclusion. That is the standard we apply to our own records, and it is the standard we expect a supplier to meet.

The record has a fixed structure. For each verification item there is a written criterion, the observed result, the verdict, the tester, the date, and the identity of the unit under test. A record with a column of pass marks and no criteria is not a verification record; it is an attendance sheet.

For a smart-glasses device, the following items have to appear. This is the numbered verification list that a brand-side buyer should expect to see, and that a supplier should be able to produce on request:

  1. Unit identity and configuration. The model string exactly as it will ship, the firmware build identifier, the hardware revision, the batch or build number of the sample, and the accessories supplied with it. A verification record that cannot name the unit it describes cannot be tied to a declaration later.
  2. Device name and identity behaviour. The name the unit reports over pairing, and whether it is stable across a power cycle. The name a device reports is the first thing a channel partner sees and the first thing an end customer complains about, so it is verified rather than assumed.
  3. Pairing behaviour. Pairing to a host device, the reconnection path after the host disconnects, the behaviour when a second host is present, and the observed range the buyer considers acceptable for the target use case. Each of these is a written criterion with a recorded result, not a general remark.
  4. Audio path. Audio output through each available path the model actually has, the audio route on connection, the behaviour with an incoming call while audio is playing, and the volume ceiling the buyer will accept. If the model has bone conduction, that path is verified separately rather than assumed from the audio output path.
  5. Charging behaviour and indication. The charging case, the interface, the charge time the unit exhibits, the indicator behaviour in each state, and what the unit does when the case is open or the cable is absent. The buyer records the case behaviour against a stated criterion.
  6. Camera function, where the model has one. For the eleven camera models in the catalogue, the camera behaviour is verified only where the published product detail actually describes it. Remote control shutter is published on D17 and ZH12, and it is on those two model strings that a shutter test belongs. If your model does not publish a shutter, the record does not contain a shutter line.
  7. Control behaviour. The physical control surface, the input method, the response latency the buyer will tolerate, and the behaviour of the controls during an active call or an active recording.
  8. Environmental and ingress claims, only where published. IP54 is published on exactly two models, V06 and S7. On those two, the verification record references the supporting evidence for the published claim. On every other model, no such line appears, because there is no published claim to verify. Inventing a line here would put a claim on the unit that the supplier has not actually made.
  9. Accessory and packaging completeness. What ships in the box, and whether the packaging is the packaging that will be used for the shipment.
  10. Defect log. Every defect observed, including the ones judged acceptable, with a written reason for acceptance. This prevents the same observation from being re-raised at the pilot run as though it were new.

Two deliberate omissions are worth naming, because they are the ones that get filled in by accident. Bluetooth version is published for fourteen models and not published for the eleven camera models or for A05, A06 and the bone conduction reference; a verification record therefore cannot state a Bluetooth version for those entries, and should say "not published" instead of recording a value. And the catalogue contains no capacitive-touch row and no translation row anywhere, so neither appears as a verification item. The same discipline applies to certification status: CE, FCC, RoHS, RED and EN 18031 appear zero times in the catalogue body, so this record verifies nothing about certification, and certification evidence belongs to the compliance gate instead.

None of this requires figures that are not published. The record contains criteria and observations, not price, not lead time, not MOQ, not capacity. When something is unknown, the record says so and the buyer turns that gap into a line under what to ask the supplier. We will not write a value into a record that later has to be defended, so an unknown stays written as unknown.

Making the record auditable later

A verification record earns its keep only if it can be reopened. The practical way to guarantee that is to ask for the right things in writing, at the gate, rather than reconstructing them a year later when a channel partner raises a complaint.

Ask for the record itself, as a document, at the gate — not a summary afterwards, and not a set of photographs. Ask for the golden sample to be marked on the outside with its unit identity and firmware build, and for its storage location to be written down, because a golden sample that cannot be located cannot be used as the comparison target and the gate effectively never happened. Ask for the retained failed units and their defect records. Ask for the build identifiers that were tested. Ask for the component and supplier list behind each verified configuration, since component substitution is the most common cause of a sample that passes and a pilot order that does not match it.

Ask the supplier which specifications their own catalogue does not publish for your configuration, and put the answers into the requirement document, not into the verification record. The published detail and the programme detail are different sources of truth, and keeping them apart is what stops a later stage from inheriting an ambiguity. The same request should cover the commercial terms the record cannot settle: the treatment of sample charges against the purchase order, the incoterms the programme will run on, spares quantities, and how the warranty applies to units supplied for evaluation rather than to units supplied for sale. Those are things to negotiate as part of the quotation and the purchase order, not to settle in a comment under a test result.

Ask, finally, what is not yet verified at this gate. The sample gate does not close the compliance file, does not establish that a pilot run can hold the golden sample, and does not settle certification evidence. Each of those belongs to a later gate with its own document. If a buyer treats the signed sample record as permission to ship, the programme has skipped two gates, and it will find out at the point where certification evidence, or a batch of production units that do not match the sample, has to be explained to someone else.

AI-generated photograph illustrating development process: how sample validation should work — image 2 of 3

How a tester decides pass or fail

A tester decides against two references and never against a third. The first is the written criterion in the verification record, which descends from the frozen requirement. The second is the golden sample. Personal preference, aesthetic taste, and a general feeling that the unit "feels right" are not inputs to a pass or fail decision. They are inputs to a requirement change request, which is a separate process with a separate document.

The decision rule itself is simple and should be stated in the record's header so that nobody has to remember it: a unit passes an item when the observed result meets the written criterion for that item. Where an item has no written criterion, it is not a verification item and it cannot produce a pass; it is either written as a criterion before testing, or it is removed from the record. Nothing is verified retrospectively against a criterion that was invented after the observation.

The golden sample is the second reference, and it plays a different role than the criterion. The criterion says what is required. The golden sample says what the requirement looks like when it is met in physical form — the finish, the fit, the acoustics, the indicator behaviour, the way the controls feel. Where a criterion is numeric or binary, the golden sample settles appearance and feel questions. Where the criterion is qualitative, the golden sample is what makes the qualitative judgement repeatable rather than personal.

There is a failure mode here that is common and expensive. A tester who is handed a unit and a checklist, without a golden sample, will produce a record full of subjective judgements dressed in objective language. Months later, when a pilot-run unit is judged against that record, the disagreement is unresolvable because there is no shared physical reference. The golden sample is not paperwork for its own sake; it is the only thing that makes the gate's output survive a change of personnel.

There is a second discipline, and it is about who decides. The buyer decides the acceptance criteria, because the buyer knows the market, the channel partner, the end customer and the standard the product must meet. The supplier verifies against those criteria, because the supplier owns the build. When a buyer asks a supplier to "assess whether this sample meets your quality standards", the criterion has been handed away and the gate is now testing nothing except whether the unit passes someone else's internal preference. Write your own criteria into the requirement document. We will use them for the golden sample comparison, and we will sign the verification record against them.

A note on granularity, which matters more than volume. One criterion covering "audio works" is not a criterion. "Audio plays through the open-ear speaker at the maximum volume the buyer has set, and the level at one metre is at least the recorded reference measurement" is a criterion. The first form can only produce an opinion; the second can produce a pass or a fail that two people would agree on.

The record also needs a verdict vocabulary, and it needs to be small. We will keep it to pass, fail, and pass-with-recorded-limitation, where a limitation is a defect accepted in writing by the buyer against a stated reason. A fourth common verdict, pass informally, is what produces the downstream compliance problem — because it is a verdict that exists in someone's memory and in no document.

AI-generated photograph illustrating development process: how sample validation should work — image 3 of 3

What happens when a sample fails

A failure is the sample stage working. A programme that has never had a failed sample has usually failed to test hard enough. The correct response is a defined loop, not an argument, and we will run the loop in the same order every time regardless of which unit failed.

The loop has five steps, and each step produces a record.

  1. Record the failure. The observation, the unit identity, the item, the observed result against the written criterion, and the date. No interpretation at this stage. A failure record is a factual statement about one unit and one criterion.
  2. Classify the failure. This is where the cost decision gets made. A failure that is a design defect will not be fixed by re-running the build. A failure that is a process defect will recur in production unless the process is changed. A failure that is a supply or component substitution will recur unless the component is locked. A failure that is a documentation defect means the requirement was never precise enough, and the requirement needs correcting before anything else happens.
  3. Fix. The fix is owned by the supplier, against a named criterion, and it is agreed whether the fix changes the requirement document. If it does, the requirement document goes to a new version and the verification record is reissued against it.
  4. Re-verify. A new unit is built or a corrected unit is re-tested, against the same written criteria, with a new record entry. Re-verification is not a verbal confirmation that the fix worked.
  5. Keep the failed unit. This is the step that gets skipped and the one that matters most for auditability. The failed unit is labelled, retained, and stored where it can be referenced later. A failure that cannot be reproduced cannot be proven to have been fixed. Retaining the unit is what allows a pilot order with the same symptom, months later, to be compared against the known-bad reference instead of being investigated from zero.

Classification also determines commercial handling, and this is where a buyer can be caught out. A design defect discovered at the sample gate is a design cost. A component substitution found at the pilot run, where the same substitution was the cause of a sample-stage failure that was closed verbally, is a different matter entirely: it is a programme cost that was accepted rather than escalated. We will agree in writing, at the tooling stage, what happens to tooling cost if the project stops. That agreement, made early, is what makes a sample-stage failure a normal event instead of a crisis.

The number of samples a programme involves is a decision the buyer makes, and the reason it is the buyer's decision is that the sample set has to represent the buyer's risk, not the supplier's convenience. The trade-off is coverage against cost, and the buyer owns that trade-off rather than inheriting it. There is no single correct number. There is a correct basis for choosing one: the number of distinct configurations in scope, the number of distinct failure hypotheses the buyer cares about, and the number of distinct suppliers or component sources the programme depends on. One unit per configuration tests whether the design works. Multiple units per configuration test whether the build is repeatable. Both are legitimate; what is not legitimate is deciding the number by convenience and discovering the gap at the pilot run.

Two related commercial points sit here and belong in the purchase order language rather than in an email. The first is sample cost and the treatment of sample charges against the eventual purchase order — that is a negotiation, and it should be negotiated as one, along with incoterms, tooling cost and spares. The second is the warranty and after-sales treatment of the sample units themselves, because a sample that fails under test is not a warranty claim against a shipping product and the two should not be conflated in the paperwork. We will raise both in the quotation stage rather than at the gate.

The final piece is the signature. At the sample gate, the buyer signs a record that states: the model string verified, the firmware build verified, the criteria applied, the units tested, the verdict per item, the golden sample identified and its storage location, the accepted limitations, and the open items carried forward. The signature transfers the definition of acceptable from a conversation into a document, which is the only durable form it has.

Related process reading: how the development process works sets the six gates in context, and what OEM quality expectations should cover is the buyer's-side view of the same question. For the brand angle, see smart glasses for consumer electronics brands, and for what happens once units reach the channel, see after-sales support channels.

The sample validation gate as a checklist of verification items

Three kinds of sample and the document each one produces

The failure loop from record to retained failed unit

Development process sample validation gate and its framework

Chat on WhatsApp
Your experience on this site will be improved by allowing cookies.