We had to disappoint our backers recently — we let them know that Misty II robots would not be shipping on our crowdfunding target date of December 4th, as planned. This acknowledgment was a clear disappointment for our backers and for us, as it’s a promise broken.
Companies lose the opportunity to build trust and relationships every time a promise isn’t kept. And backers bear the painful burden of delays. Nobody feels good about that — not the disappointed backers, not the Misty team.
Do I think we’re an exception among startups in missing our target release date? No. The examples of other startups who missed their targets by months to years is much too long and boring to present here.
Yes, hardware cycles are long by nature, development costs go beyond labor, and schedules are only as good as the estimates of the teams doing the estimating. But… why? And what can any hardware company do to mitigate risks and improve estimates?
I’d like to suggest that the journey of hardware development is a lot like a plane trip. I’m the pilot on this particular flight, and I’ll be sharing with you the view from the cockpit.
At Misty Robotics we believe there’s a different way for hardware startups to communicate with their customers, and it begins with a much greater level of transparency. Not just about the product, but also about the process, the stages/milestones of the process, and the risks that are (one hopes) eliminated at each milestone.
Buckle up, but please keep your phones and laptops on for the duration of this journey.
Risk is a function of distance, and schedule is a function of risk
The day before a startup hardware company ships its product to its first customer, the risk of the product missing its schedule is insignificant — approaching zero. The startup should have better than 99.9% odds that the product ships “tomorrow”. A week before, there may be a little more risk, but the chances of missing the scheduled date should still be relatively low. A month before, we should expect rather more risk. And so on.
You can picture this as an airplane landing on an air strip. At 50 miles out, the pilot isn’t 100% confident everything is in place to be landed and that they’ll be pulling in to the announced gate at the specified hour. At 1 mile out, they’re a whole lot more confident on being on the ground at a given time. And at 1 foot off the ground, the odds are extremely close to 100% that they’ll land the plane where and when expected.
In other words, there are a lot more unknowns — or risks to the pilot’s confidence — at 50 miles out than at a foot off the ground. And this translation between flight distance for a plane and schedule distance for a product is pretty direct. The further out your product is on its schedule, the more risks.
The chart below depicts this relationship between time and risk for a product’s journey along its production “flight” plan. It also notes some “typical” time ranges from a given production milestone to the first customer ship. For example, if the first engineering “pilot” (EP1) production run took place on July 1, then one could reasonably expect first customer ship (FCS) to be somewhere between January 1st to July 1st of the next year.
Truthfully, if you wanted to get really geeky about risk probabilities, each hardware milestone would itself have its own modified bell curve of risk, where the standard deviations at the risk/delay end of the curve could go well past the “normal” range associated with that milestone. A true bell curve wouldn’t fit, because one can’t readily accelerate / remove risk and ship hardware earlier by multiple standard deviations. Some things, like building tools for a manufacturing line, simply take time, and you can’t accelerate it by many multiples. Whereas on other end of the curve, with technical risk, some problems become so entrenched that they can elongate the risk to the ship window by several standard deviations.
For both travelers and startups, milestones make navigating easier
So, if the risks to our scheduled arrival directly correspond to our distance from a destination, how do we reduce those risks? One way is to shorten the distances being measured. We do this by breaking down our journey with some key milestones. Just as for travelers, milestones are designed to eliminate risk and increase confidence with hardware engineering and manufacturing.
Perhaps you’ve already done all the hard work of product design, as well as hand-building some working prototypes of your hardware. You might rightly feel that you are a good distance along in your product journey. Absolutely. But you’ve not landed that plane yet. You’re just entering the manufacturing part of your journey. Moving from hand-built prototypes to actual manufactured products is terrifically exciting, but this part of the trip is more complex than it appears, so there are multiple milestones needed to stay on course.
Milestone One: Engineering Pilot 1
The first milestone is for the initial manufacturing pilot run, where one actually builds manufactured (not hand-built) prototypes on an assembly line at the manufacturing site. Some teams call this milestone the “Engineering Validation Test” (EVT); our team calls it “Engineering Pilot 1” (EP1).
The intent of the EP1 Stage is to validate that the product is technically achievable (and manufacture-able. This stage typically involves one EP1 “build”, with actual output ranging from 20–500 manufactured units. At this milestone, the amount of risk is still quite high because significant issues can emerge in the translation from engineering design to first units of something manufacturable.
You can’t exit the EP1 milestone until you pass these waypoints on your journey:
- All technical risks are resolved. This is a very short way of covering a LOT of risk-filled territory — more on this below.
- Product design is frozen (minor cosmetic tweaks and software changes post EP1 are okay, if not desirable).
- All critical functional, aesthetic, and user interface features are validated.
- The final evaluation of components/sub-assemblies from alternative/competing suppliers is completed.
- Buys of long lead-time components for Mass Production volumes are initiated.
- Manufacturing assembly and test processes are defined and documented.
- Rework and repair processes are identified and documented.
- Second-pass Reliability/Environmental tests are conducted.
- Key supply chain partners are determined to be qualified.
- Most mechanical parts are received from hard tooling. Some temporary tools are still acceptable, but action plans must be in place to resolve any outstanding technical issues with the build.
- Cost of Good and Services (COGS) estimates are verified or updated.
- And, going forward, Engineering Change Orders (ECOs) need to be approved by Engineering and Operations.
The trip from the beginning (the initial work to create the first prototype on the test assembly line) to the end of this milestone (meeting the criteria above), is highly variable. There are many aspects still in flux, from the product’s parts to the test apparatus to the assembly line tooling itself. And, most importantly, there’s the “all technical risks are resolved” point.
The actual manufactured product coming off the line needs to be verified to work as specified. You can test your hand-built prototypes extensively, but until those same tests succeed on a manufactured prototype, you can’t get past this milestone. All together, that means that exiting the EP1 milestone is a long journey with significant technical risk.
And, as a result, the “predictability” of the schedule at the time of EP1 is highly variable. Our estimate, shown in the chart above, is a product can be shipped anywhere from 6 months to 1 year from the commencement of EP1.
At the end of this phase, fully functioning prototypes, a final risk assessment and an updated COGS estimate are presented for approval to proceed to the EP2 phase.
Milestone Two: Engineering Pilot 2
At the next milestone the risk to an on-time arrival is still quite high. Some teams call it the “Design Verification Test” (DVT), others “Engineering Pilot 2” (EP2), but the bottom line is that you are still some distance from your desired landing strip.
- The bugs one caught during EP1 may still be present in the EP2 build (hopefully fewer exist because one knocked them out successfully).
- New issues might surprisingly appear (you really hope not, but this does have a small chance of happening).
- The final manufacturing assembly process is still not proven.
The intent of the EP2 milestone is to verify that the product can be built in a production environment (using plan of record assembly plus test processes and equipment), in the proper form factor, with all functional, aesthetic, and user interface features being met.
This phase should have a limited duration, with a single EP2 build, in a greater quantity than EP1. In general, the number of EP2 units will vary depending on the number of changes introduced after the initial EP1 build, the complexity of the new product relative to any previous products already introduced, and a whole bunch of other factors.
In addition to proving out all functional, aesthetic, and user interface features, all environmental, regulatory, and reliability tests must be successfully completed during this phase. The only changes allowed in this milestone are cosmetic changes which aren’t high risk or high priority. At the end of this milestone it’s “pencils down” for all Engineering changes (EP1 it was “pencils down” for almost all except the minor / cosmetic). Any design changes after exiting this milestone must be phased in after the product launch.
As with EP1, you cannot exit the EP2 milestone until you pass these waypoints on your journey:
- All specifications are frozen and released to production.
- Final functional, environmental, and regulatory testing is completed.
- All product testing is complete per final mass production plan.
- Mechanical parts are built from fully-textured hard tools.
- Qualification of duplicate hard tools (where required) is underway.
- All materials are from approved suppliers on the Authorized Vendor List.
- Changes are going through a formal Engineering Change Order approval process. This is similar to what any software team does during the “code freeze” part of a software release schedule.
- Ongoing reliability test processes have begun. These tests are designed to help a hardware company determine how long a warranty can be achieved, what the “mean time between failure” for various components might be, etc.
- The manufacturing line is fully qualified, including operator training, work instructions, equipment calibration, etc.
- The rework and repair process is fully qualified.
- A second sourcing of all critical materials is underway.
- Final COGS is validated.
At the end of this milestone, fully-functioning EP2 units with final test results are presented for approval, with the goal that you are now much closer to your destination of sellable customer units. However, because there’s still some variability in terms of test apparatus, manufacturer training and preparation of operators, assembly “bugs” that need to be fixed, etc., the time range from start of EP2 to shipping your product can still be wide — perhaps 90 to 180 days.
Milestone Three: Final Engineering Pilot & Pre-Production
It’s not until you get to the milestone for Pre-Production (some teams call it “Production Validation Test”, others a two-step process called “Final Engineering Pilot” and “Pre-Production”) that the risk in your journey drops significantly. This is the point at which a customer might consider the uncertainty low enough to provide a specific date estimate of an arrival time, i.e., shipment date. From here, you are definitely closing in on a landing.
The intent of the FEP/PP milestone is a final verification that the product can be built in high volume on a production line using:
- qualified parts
- production personnel
- production tooling/equipment
- production quality/test systems
This milestone consists of one FEP/PP build of a sufficient number of units, depending on the projected cost and sales volume of the new product. The FEP/PP build is monitored on-site by appropriate company Engineering and Operations staff, but their participation is completely hands-off. Assuming there are no unexpected quality problems during the FEP/PP build, these units can be sold.
Key deliverables of FEP/PP include:
- High-volume manufacturing process is validated, including all production processes, test systems, work instructions, certified personnel, quality systems, etc.
- Order administration, technical support, and reverse logistics systems/personnel are fully qualified and operational.
- Continuous Improvement Programs (e.g. cost reduction, yield improvement, cycle time reduction, etc.) are launched and in full gear.
At the end of FEP/PP, final test results (including throughput capacity, yield, and product mortality) and a final COGS summary are presented for approval. If you’re good, you’ve passed the milestone and can look at shipping products to customers for revenue.
However, even after the FEP/PP milestone is complete, there are still risks to a timely landing at FCS (First Customer Ship). These might range from kinks in final high-volume assembly to shipping issues such as labor strikes, holiday constraints, or even Customs and Border Patrol issues or unexpected tariffs. So, given the risks inherent in the FEP/PP milestone, there may still be a 60- to 120-day window from the time you start FEP to FCS.
It’s really only once you’ve shipped your first shipment to your warehouse that your risk is truly low. That’s when you should be able to safely give customers a shipping date that is accurate to within 1 week.Yes, as pilot you have to give an ETA for your journey when the flight departs, but it’s really only when you have the landing strip in sight that you can say with surety when you’ll be on the ground. And, even then, we’ve all been on flights that have landed but sat on the runway for 20–30 minutes to accommodate last-minute gate changes.
Where we went off course (and where we are now)
We were scheduled to enter EP1 with Misty II the first of June (hence the Dec. 4th ship date), but EP1 was delayed by 45 days (to mid-July). Unfortunately, we experienced a few significant technical risk issues. Most significantly, one related to the amount of heat generated in the two quad-core processor head — when we finally had the two processors near each other in a very dense circuit board we learned there was far more heat generated by them and related electronics than anticipated. This necessitated a different heat management solution (with fans) which then necessitated research into the effect on microphone reception. The microphone investigation, in turn, injected a pretty unique and challenging-to-estimate unknown into the equation (given most voice assistants like Alexa and Siri don’t have fans blowing near their microphones).
In thinking about our situation, as it relates to most other startup companies’, we think the most transparent thing to do is:
- Provide customers with what milestone has been entered (EP1, EP2, FEP, etc…)
- Provide a “shipping window” associated with that milestone. For example, if we’ve entered the EP1 stage (as Misty Robotics has), the shipping window might be “anywhere between 180 to 365 days from entry”. After the company has entered Pre-Production, that might become “anywhere from 30 to 60 days from entry of the stage”.
- If one wanted to be super transparent one could even provide a few of the core “blockers” from achieving the next milestone — the things the company is currently working on that it knows it must resolve before the next milestone is achieved.
That’s what we at Misty Robotics are working hard to provide. For our crowdfunding backers we’ll maintain this information in our Community Forum, and for visitors to our website we’ll provide a shipping window estimate.
Stage / Milestone Entered: EP1
Date of Entry: July 15, 2018
Estimated Date of First Customer Ship: April 15th to July 15th, 2019
Core current blockers of EP2: validated audio solution for fan noise, test calibration for 3D sensor, heat management solution for the wireless charger and validated PCBA tests