Last week, we covered how Misty is able to move and this week, we’re covering how she’s able to stop moving. More specifically, how she stops moving in order to keep from running into things or plunging off of them. We’ve got Misty Robotics’ Senior Software Engineer Adam Citarella and Firmware Engineer Steven Kibler on deck to answer all of our questions.
In a nutshell, what are the systems in place that help to keep Misty from running into things and off of things like tabletops and stairs?
Adam: There are two ways that Misty can avoid obstacles and falling off ledges. First, Misty has a built-in obstacle avoidance system; we call this her hazards system. The second way Misty can avoid obstacles or taking a plunge is for developers to use sensor data to code their own obstacle avoidance skills.
Tell us more about Misty’s hazards system – What is it comprised of?
Adam: Misty’s hazards system is composed of her time-of-flight (ToF) sensors and bump sensors. There are eight time-of-flight (ToF) sensors on Misty: Three in the front and one in the back, which serve primarily for obstacle avoidance, plus one on each corner facing down which detect edges. Misty’s four bump sensors mainly function for obstacle avoidance; there are two on her front corners and two on her back corners, each of which can be triggered by pushing in from the side. They’re useful when the ToF sensors don’t have time to react because something – a human, a cat, a ball – has suddenly entered her path. Misty’s bump sensors can also be programmed to serve as “buttons” that can initiate a new sequence of actions when they’re touched.
There are threshold settings for all eight of the ToF sensors (ie consider something less than 20 cm away from a hazard), and they can be disabled entirely. Bump sensors can also be turned on or off for use in the hazards system so users can adjust and totally disable the whole system through the API.
How do the ToF and bump sensors that make up Misty’s hazards system work?
Adam: While Misty drives around, she constantly checks information from her ToF sensors and bump sensors at a firmware level as long as her hazards system is enabled.
If there is something too close to one of her front or rear ToF sensors, or if one of her downward facing ToF sensors detects that she’s about to drive over an edge, or if one of her bump sensors is triggered, then Misty goes into a “hazard” state. This means her drive motors will stop running any commands that would cause her to continue moving in the direction where she sensed a hazard. For example, if Misty drove forward towards the edge of a desk she would stop and then ignore subsequent “drive forward” commands. It would then be up to the user to invoke commands, or have a skill that invokes commands, that backs Misty away from the edge.
Is there anything that can affect Misty’s hazards system?
Steven: There is about one inch of robot movement between when the edge sensors detect an edge and when the robot has to stop, In that distance, we need to process the sensor data, detect the hazardous situation, and issue a stop command to the motors. If Misty is moving fast, it can affect her hazards system. Technically, her ToF and bump sensors work fine regardless of her speed. The issue is that Misty’s momentum prevents her from stopping in time when the hazards detect an edge while traveling higher than 45% of her max speed which is 1000 mm/second; so, at 450 mm/second or faster, you might run into problems (no pun intended).
At higher speeds, we are also limited by the relatively high center of mass of Misty. If we stop too quickly from too high a speed, she can faceplant. Some work has been started to use the obstacle (horizontal) ToF sensors to detect edges from further away which would allow the edge detection system to work at higher speeds by slowing down from further away.
Lastly, there are two more things that can affect Misty’s hazards system. The ToF sensors can be affected by dirt and dust but this one has a pretty straight-forward solution: We’ll provide some guidance on cleaning them periodically with a blast of compressed air when robots are delivered. And again, we mentioned earlier that the user can disable her hazards system via API so of course, that will affect it.
Ok, let’s say Misty does tip over. Will her motor continue to run and her tracks continue to turn?
Steven: If Misty tips over, each of her edge sensors (ie the ones on the bottom corners) read this as a hazard and her motor and treads stop running. And, as long as the edge sensors are enabled, picking her up will automatically cause a hazard and stop her drive as well. Eventually, we’ll have the cap touch do the same thing.
Currently, there is an Inertial Measurement Unit (IMU) related behavior that acts as a sort of hazard-handling feature to help prevent Misty from tipping over. If Misty has been commanded to turn and the IMU does not detect the expected rotation, then the movement will be cancelled. This would come into play if Misty is up against some kind of side obstacle preventing the rotation. Users can read the IMU values into their skill and react with some appropriate behavior of their choosing. Additional reactions to specific IMU readings are planned for the future and we will likely also use the drive encoders to detect the same “stuck” hazard for movements that are forward and back movements.
We also plan to incorporate Misty’s IMU in order to stop the motors. The reason we want to do this is because the user can disable the edge hazard sensors in which case the motors won’t stop if she is tipped over. It’s also possible that Misty could fall into a blanket or something else in a way that keeps her hazard sensors seeing a surface, meaning her hazards won’t detect a reason to stop her motors. The IMU will always detect if she’s tipped and can’t be disabled, so we’ll use it (in addition to the edge sensors) to stop the motors in that case.
This is all a great segue to my next question: how can developers use sensor data to code their own obstacle avoidance skill?
Adam: Users can register for data from Misty’s ToF, Bump, IMU, and depth sensors to use in their own hazard/obstacle avoidance skill.
When they do this, their skill will constantly check the information from the sensors that they want to use. When that information indicates Misty is too close to an obstacle, their code could have Misty stop and find another path. For example, if they’ve decided they want the minimum distance of an object to be .2 meters when Misty responds, they could write code to have her back up and find another path whenever an object comes within that distance.
But wait, isn’t that what the hazards system is supposed to do already?
Steven: Yes, but some users will simply want to code it themselves. Or, they may need a behavior other than that in the firmware or in addition to it. For example, you could also program her to turn and have her LED lights glow when her ToF sensors detect an edge from a specific distance you determine, or you could program her to say “Ow!” if her bump sensor is triggered. It just depends on what people are trying to do. If they do want to code it themselves, they’ll need to disable the hazards system in the firmware so the two don’t compete.
We have some sample code on how to program your own hazards system, and there is a tutorial in the docs. In that tutorial, Misty registers for ToF data and drives straight forward. If the ToF data indicates an object is closer than 0.2 meters, it triggers a callback function that executes a misty.Stop() command, so that Misty doesn’t keep driving.
Are there sensors people can add to Misty that can help with obstacle avoidance in some way?
Steven: There are! We’ve made Misty hardware extensible, so adding more sensors to her is entirely possible. There are plenty of object proximity sensors out there, most of which fall into three basic categories: touch, time-of-flight, or diffuse sensors. Depending on the category and medium, the range and resolution of your sensor will vary, and selection of the sensor is often done with those two things particularly in mind. Sites like Sparkfun have a lot of helpful information and technology comparisons to help you decide what to add. As the Sparkfun site alludes to, it’s important to also consider the type of communication the sensor uses to talk to its host microcontroller or computer.
In the case of Misty, a UART channel is used to allow your skill to talk to the sensor through her backpack. If the sensor you choose does not have direct UART capability, you’ll then also need to add a microcontroller to be able to exchange data with the head and the UART backpack. You can use your own microcontroller, or you can purchase Misty’s Arduino-compatible backpack we’ve already made that will let you quickly and easily attach sensors to Misty. We have produced another blog post to cover the topic of how to interface to Misty’s backpack UART port that people should check out if they’re interested in adding sensors.
Whether you use Misty’s built-in hazards systems, code your own obstacle avoidance skills, or choose to use both, Misty’s sensors help to keep her safely moving so you can keep building skills. Thank you to both Adam and Steven for lending their knowledge, insight, and tips this week!
Have more questions about Misty’s moving (and stopping) abilities? Have any questions about Misty? The Community Forum is the best place to discuss all things Misty but we also want to hear your questions on Twitter @MistyRobotics.