Smart eyewear OEM/ODM manufacturer · Shenzhen, China +86 19835872450 sales@cynrin.com
Smart Glasses Technology

How to Test Before Ordering Smart Glasses AI Features

The reason an AI feature cannot be evaluated by watching a demonstration is not that demonstrations mislead. It is that a demonstration is the one…

By CYNRIN Engineering & Sourcing Team/ Dec 15, 2021/ 18 min read
How to Test Before Ordering Smart Glasses AI Features

Smart glasses ai features: relationship, not a component

The reason an AI feature cannot be evaluated by watching a demonstration is not that demonstrations mislead. It is that a demonstration is the one configuration in which an assistant looks best: a prepared phrase, a quiet room, a known-good network, and a unit whose accounts were linked in advance by the person who built it. Everything a buyer needs to know is a property of the relationship between the device, the model provider, the network and the region — and none of those four is the product in the box.

The decision, in three sentences. If a voice assistant is load-bearing in your listing, test it as a relationship and not as a feature: write the phrase list and the pass conditions before the sample arrives, run the plan on a unit that is not the demonstration unit, and record the firmware build with every result. The metric that decides whether the product is shippable is usually the false activation rate rather than the wake rate, because a device that interrupts a customer conversation is a product that gets taken off. If you cannot get a build identifier, a data path description and a named offline behaviour in writing, do not put the assistant in the copy — the feature will exist in the product page and nowhere else.

Who is running this plan, and what the plan has to produce. The reader is usually deciding whether an AI assistant belongs in the product at all, or comparing two suppliers' implementations of the same claimed feature. What they hold is samples, a feature list, a channel, a target price and a marketing plan that already assumes the feature exists. What they do not hold is performance data, and the reason is structural: the data belongs to the model provider rather than to the factory, and it changes with the build rather than with the hardware. The plan therefore has to produce three things — a number rather than an impression, a build identifier attached to that number, and a written list of the conditions the number was measured under.

What our own catalogue says, and what it does not. We name an AI voice assistant among the key technologies for the AI camera glasses family, and our independent site describes AI translation on the K01, which is an audio model. We publish no wake rate, no false activation rate, no command success rate, no latency figure, no recognition accuracy, no translation accuracy and no offline behaviour for any model in the range, and we will not estimate any of them. That thinness is not a data problem we failed to collect; it is what a category with no published performance baseline looks like, and it is the reason the plan below exists.

Where the cost of an AI feature sits. The hardware cost of an assistant is close to zero — the microphone and the radio are already in the product. What an assistant actually costs is the four things around it: the work to specify a behaviour that can be tested, the localisation of the phrase list and the documentation into each launch market, the support script for the failure cases, and the contractual machinery that tells you when the provider has changed something underneath you. Two suppliers quoting the same assistant will differ far more on these four lines than on any unit price, and the difference is invisible until the listing is live.

Five numbers, and the denominator each one has to carry

AI-generated photograph illustrating how to test before ordering smart glasses ai features — image 1 of 3

The five numbers to demand, and the three nobody reports

Send these five questions in one message, in identical words, to every candidate, and ask for the answers in a table rather than in prose. The denominator column is the one that matters, because a percentage without a denominator is a marketing line.

Metric What it measures Denominator required Pass line
Wake rate The assistant starting when it should Attempts, number of speakers, environment A percentage with all three stated
False activation rate The assistant starting when it should not Attempts, and hours of ordinary use A percentage, or a written statement that it is not measured
Command success rate Your phrase list being understood The phrase list you supplied, and the repetitions per phrase A percentage against your own list
Latency The delay before the answer arrives Network state and firmware build A fixed threshold, not a judgement
Recognition accuracy Whether the right thing was heard The scene, and whether the speaker is facing forward A figure for the quiet room and for the real environment

The thresholds in plain language, and the three rows that are usually missing

A rate without an attempt count is not a rate. Ninety-eight per cent measured over ten attempts in a quiet office predicts less than eighty-five per cent measured over three hundred attempts on a street, and the second number is the one that describes the product. The pass line to hold is a percentage with a denominator, a speaker count and an environment attached, and a supplier who cannot supply all three has not measured the product.

The false activation rate is a usability gate, not a technical footnote. It decides where the product can be sold rather than whether the feature works, because a meeting room and a street are different markets for the same device. A device that wakes on a cough in a quiet office is a meeting-room product; the same device on a street is a nuisance. The threshold a buyer sets here is a product decision about the channel, and it should be written down before the product page is drafted.

The command success rate has to be measured against your list, not the supplier's. A supplier's list contains greetings and a timer, because those are easy. Your list contains the phrase that a warehouse supervisor actually says eleven times a day. The pass line is a percentage against the phrases you supplied, with a stated number of repetitions per phrase.

Latency needs a fixed number rather than an adjective. Set the threshold — under two seconds is a reasonable starting point — and measure it. It varies with the network, which is why the network state has to be part of the measurement, and a latency figure without a network state is a figure from the supplier's office.

The build number is the column that makes the results worth keeping. An assistant changes between builds, and a measurement without a build identifier cannot be reproduced, cannot be defended and cannot be compared with the next supplier's result. It is the single most important column in the sheet, and it is the one most often left blank.

Three rows that almost never come back, and what to do about each. The offline behaviour — what the device does with the network physically removed — is the row that settles the most common dispute in this category, whether the product does what its documentation says when the network is not there; ask for it as a written statement, and if the answer is silence, treat the offline behaviour as undefined in the copy. The data path — what leaves the device, where it goes and how consent is handled — is the row a compliance reviewer asks for; verify it by observation rather than by reading it, and make the privacy statement in your listing describe what you observed. And the update mechanism — how a shipped unit receives a new build, and who decides — is the row that tells you whether the feature can change without contractual notice to you.

The order the assistant plan runs in, and why the plan comes first

Running the plan: build identifiers, environments, and a second day

Step one, before the sample arrives. Write the phrase list. It should contain the commands the customer will actually issue, in the language of the launch market, including the two or three that are awkward to say. Set the pass conditions as numbers. Decide which tests need which environment. Fix the handset, the network and the audio conditions. And identify the firmware build you expect the sample to carry, so that the first thing you do on arrival is confirm it.

Step two, on the day, before anything is measured. Confirm the build identifier on the unit and record it. Confirm the language, the region and the account state. Check that the unit is a plain out-of-box configuration rather than the demonstration unit, which usually means a fresh account, no linked services and default processing settings. A result from a linked demonstration unit is not a result about the product the customer receives, and it is the single most common way an assistant evaluation flatters a supplier.

Step three, the wake rate, over a large number of attempts. Use several different speakers, because voice assistants vary more by speaker than by hardware. Record the attempt count, the number of speakers and the environment, and compute the rate yourself from the log rather than trusting a rounded figure.

Step four, the false activation rate, in the environment that matters. This is the longest test in the plan and the one most often skipped. It means leaving the assistant enabled through ordinary activity — a desk, a meeting, a street — and logging every activation that was not requested. The number you get is the number that decides the channel.

Step five, the command success rate, on your phrase list. Each request is run a fixed number of times, and the failures are recorded with a reason, because "did not understand" and "understood but did the wrong thing" are two different failures with two different fixes.

Step six, latency, against the fixed threshold. Measure the interval from the end of the utterance to the start of the response, with the network state recorded, and repeat it enough times to see the tail rather than the average.

Step seven, the offline behaviour, with the network physically removed. Not airplane mode with a cached result available — the network genuinely unavailable, with the result recorded. A supplier who has never tested this has a product whose behaviour in a lift is unknown.

Step eight, the data path, verified by observation. Watch what the device does on the network, compare it with what the documentation claims, and keep the comparison in the file. A privacy statement written from a marketing sentence is a liability, and a privacy statement written from an observation is defensible.

Every rate has two values, and only one of them predicts the customer

Test two or three units of the same build wherever the feature is load-bearing. A feature that works on one sample and fails on another is a yield problem, and a yield problem is invisible in a single-unit test. Where the feature is the reason the customer bought the product, a single unit is not evidence of anything.

Run the whole plan twice, on different days, and treat the spread as a finding. An assistant's behaviour varies with the day, the network and the ambient noise, and a large difference between two runs tells you the feature is not stable enough to be the basis of a claim. Where the spread is small, the result is worth keeping; where it is large, the finding is more valuable than the number.

Record the build identifier, the environment and the time with every result. Keep the recordings where the platform allows it. A measurement that cannot be reproduced is not evidence, and in this category the build is the thing that changes.

Smart glasses ai features: a plan that can be repeated after the build changes

What to record with every result. The model designation and configuration, the firmware build identifier, the app version, the language and region, the account state, the handset and network, the environment and its noise level, the phrase or test attempted, the result, the time and the tester. The sheet is long, and that is the point: the row count is what makes the result defensible six months later.

What this plan cannot establish. It does not establish how the feature behaves for your whole customer population, it does not establish how it behaves in a market or a language you did not test in, it does not establish how it behaves after the provider's next change, and it does not establish the compliance position, which comes from documents. The last one matters: the Radio Equipment Directive carries cybersecurity requirements that form part of CE marking, described by the EN 18031 series of harmonised standards and covering secure communication, encryption, secure storage, user consent and the process for handling identified vulnerabilities. A voice assistant sits squarely inside that scope, and a measured offline behaviour is the evidence that the product does what its documentation says when the network is not there.

AI-generated photograph illustrating how to test before ordering smart glasses ai features — image 2 of 3

Where an AI programme goes wrong, and the early warning for each

The feature is announced in a listing and fails in a customer's meeting. The symptom is a complaint with a description of a conversation that was interrupted. The root cause is that only the wake rate was measured. The prevention is to measure both rates and to set the false activation threshold before the product page is written.

The customer reports that the assistant stopped working after an update. The symptom is a cluster of complaints that begins on a date nobody can account for. The root cause is that the measurement was tied to a build and the build changed without a re-test. The prevention is to record the build with every result, to re-run the plan against the build that will actually ship, and to ask for notice of a change that affects the assistant's behaviour.

A translation claim is made and the target-language reviewer rejects it. The symptom is a listing that promises interpretation and a reviewer who cannot reproduce it. The root cause is that the accuracy was assessed by someone who is not a native speaker, on greetings rather than on working phrases. The prevention is to have a native speaker of the target language score the phrases the customer needs, with the phrase list attached to the score.

A privacy question is answered with a marketing sentence. The symptom is a corporate customer's security review asking for the data path, and the brand having nothing but a policy URL. The root cause is that the data path was never verified. The prevention is to verify it by observation and to make the privacy statement describe what was observed.

The product works in the demo country and fails in the launch market. The symptom is a cluster of returns from one market only. The root cause is that the plan was run in one region on one network. The prevention is to run the plan in the launch market, on the networks the customer will use, and to report both sets of results side by side.

Two evaluators reach opposite conclusions on the same day. The symptom is a shortlist that changes between a Tuesday meeting and a Friday one. The root cause is that the pass conditions were adjectives. The prevention is to fix the pass conditions as numbers before the samples are ordered, so that a disagreement points at a vague criterion rather than at the product.

AI-generated photograph illustrating how to test before ordering smart glasses ai features — image 3 of 3

What you can contract for when the provider changes something

The build identifier and the update mechanism are the two lines to put on the request for quotation. A supplier who cannot name both is telling you that the assistant can change without contractual notice to you, and that is a larger commercial exposure than the unit price. The lines cost nothing to ask for and they are the only ones that give you a hook.

The phrase list and the pass conditions belong in the purchase order, referenced as acceptance criteria. The assistant is the one part of the product that a customer can blame on somebody else, so the criteria have to exist in writing before the sample rather than after the first complaint. A sample that meets the criteria on the build that ships is a sample you can accept.

A notice clause covering behaviour changes is the most valuable clause in this category. Any change of microphone, DSP tuning, default processing state or assistant model can alter the observable behaviour without altering the hardware, so the clause has to cover behaviour rather than parts.

The localisation work is negotiable and worth negotiating. The phrase list, the documentation language and the support script per market are each a round of work, and each round is a week. The negotiation worth having is the number of rounds included and whether a second market is priced as a copy of the first or as a new translation exercise.

The minimum order quantity and the packaging work are ordinary commercial lines. A retention held against the documentation package protects the buyer on the failure that costs most, which is a unit arriving with paperwork that does not cover it, and a deposit with a balance on inspection before shipment is the standard structure.

The schedule lines in an assistant programme are the build freeze and the certification calendar, and neither of them moves because a buyer pushes. The build freeze has to sit before the order rather than after it, because a build that changes after the order invalidates every measurement the acceptance criteria were based on. Certification cost and certification lead time follow the process and the testing body, and pressing on either produces a longer schedule rather than a shorter one. The tooling cost is a one-off item usually amortised across the first order, and the questions worth asking are over what volume it is amortised and what happens to it if the programme is discontinued.

The documents to request before the deposit. The firmware build identifier and the update mechanism. The certificate numbers with the issuing body and the validity dates. The declaration of conformity for the exact model designation. A written description of the data path and the consent mechanism. A written statement of the offline behaviour. The model designation written on the quotation.

The clauses to write into the purchase order. The model designation as a line item with the certificate numbers attached. The acceptance criteria from the eight tests, with the phrase list attached. A clause requiring notice before a firmware or model change that affects the assistant behaviour, with a re-test obligation. A clause on who bears the cost when the sample does not meet the acceptance criteria.

A worked plan for a brand launching into one market

This is a worked case rather than a customer case. That order is the point of it.

Assume a consumer electronics brand with an eleven-model camera line, an existing camera platform, and a plan to add a voice assistant to two configurations as the headline feature of a launch into a single market. The brand's own product manager has drafted listing copy that says the assistant "understands naturally" and "always there when you need it", and the brand's compliance reviewer has already asked for the data path.

The first decision is to drop the assistant from the copy until it is measured. The product manager removes both adjectives and replaces them with nothing, because a feature that cannot be described accurately should not be described at all yet. That single edit removes the brand's largest exposure at no cost.

The second decision is the launch market, and it is made before the plan is written. The brand picks one market, one language and the two networks its customers actually use, and it accepts that the assistant will not launch in the second market until it has been measured there. The reason is cost: a phrase list in one language is a translation exercise, while a phrase list in a language the team cannot judge is an unusable asset.

The third decision is the build. The brand asks the supplier to freeze a build identifier for the evaluation and to confirm that the same build will ship. The supplier confirms a build, and the brand records it. This is the decision that makes every later number meaningful, and it costs one email.

The fourth step is the run, on a unit that is not the demonstration unit. The brand orders a plain unit, sets up a fresh account, and runs the eight tests across two days with two evaluators. The wake rate is high in the quiet room and lower in the target environment. The false activation rate, measured through an ordinary working day, is the finding that matters: it is high enough that the brand's own marketing claims would not survive a week of use in that environment.

The fifth decision follows from the false activation rate. The brand does not launch the assistant in that market, and it keeps the hardware line as it is. The alternative would have been to launch and to absorb a complaint cluster that the sample programme had already predicted. The evaluation cost a sample round and a week; the alternative cost a launch.

The sixth decision is what to do with the result. The brand puts the false activation threshold in the purchase order as a condition for a later firmware release, and negotiates a notice clause on behaviour changes. It keeps the recordings and the sheet, because a future firmware release will need a re-test and the sheet is what makes that re-test comparable.

The outcome is a product line that launched with less than the brand wanted, and a supplier relationship that now has a written standard for the feature the brand eventually wants to add. The point of the walk-through is that the decisive number came from a test on a sample rather than from a demonstration, and it cost one unit and two days.

What our assistant records can and cannot settle

Concretely, we would rather a buyer test an AI feature than believe a description of one, and a test plan is the only thing that turns a capability into a specification. We name an AI voice assistant among the key technologies for the AI camera glasses family, and our independent site describes AI translation on the K01, which is an audio model — we treat that as an observation from our own marketing material rather than as a specification. We publish no wake rate, no false activation rate, no command success rate, no latency figure, no recognition accuracy, no translation accuracy and no offline behaviour for any model, and we will not estimate any of them.

We will also not put a number in a listing that was measured on a demonstration unit, because a number that cannot be reproduced is a claim that cannot be defended when the customer is standing in front of you. And we will supply units with a known firmware build, and work through a plan with you rather than send a demonstration.

Three things to settle before an assistant is promised in a listing

Ask for the build identifier and the update mechanism in the first message. They are the only two answers that tell you whether the feature can change underneath your programme, and a supplier who answers both has a process.

Write the phrase list before the sample arrives. It is the only part of the plan the supplier cannot supply for you, and it is the part that makes the success rate yours rather than theirs.

Send us the features you intend to publish, the launch market and the language, and we will write the test plan with you. The background is under AI features, the sample route is sample evaluation, the measurement method is in how to measure AI features, and the supplier questions are in AI supplier verification.


Related reading

Tell us the features you intend to publish and the environment, and we will write the test plan with you.

Chat on WhatsApp
Your experience on this site will be improved by allowing cookies.