To celebrate National Camera Day, we spotlighted the four cameras Misty has packed away neatly in her visor. These cameras work simultaneously to make it possible for her to explore and interact with her environment, returning important data you can use to build skills.
While we touched briefly on cameras in our last post, today we’re taking a deeper dive into the cameras we chose to include in Misty. We’ll cover what they are, how they work, examples of skills you can build that use them, and how to use the data they can collect in your code. We’ll also share how you can add your own cameras to Misty.
The RGB camera
In a process called interpolation, an RGB camera picks up the colors red, green, and blue as they are combined in various proportions to obtain any color in the visible spectrum. This mimics what the light receptors in our own eyes do to detect color and shades.
Currently, Misty’s RGB camera is used in skills that leverage face detection, training, and recognition. Soon, it will also be able to detect and be trained on object recognition. It will also be able to stream video giving you the ability to perform image processing on the raw feed; in other words, it will be able to serve as a remote viewer in the telepresence skills you build.
Misty’s RGB camera factors heavily into skills related to Security and Human Interaction. For example, in the Misty Security Guard Skill, Misty can “guard” your desk by taking a picture of any unfamiliar face that comes into her view. Before she’s able to do this, she must first be face “trained” and you must create a callback for when she doesn’t recognize a face (ie “there’s an intruder!”); we share the entire process for face training and creating a callback in this post. And of course, if Misty does recognize the face, there’s no need for her to sound the alarm. Instead, you can set up a callback for her to issue a warm welcome to the visitor, even greeting them by name.
Now, let’s take a look at a use case which uses Misty’s RGB camera to ensure business flows smoothly, even when you’re away. In the Misty at the distillery example, Misty uses the RGB camera to snap photos of equipment at regular intervals (predetermined by the distillery owner) and then sends each picture to Microsoft Azure for analysis. Using Cognitive Services, the relevant temperature data is extracted from each image and written to the Azure database for later access.
The owner is now able to keep a close eye on their temperature in real-time no matter where he is. By setting up a callback to send him a text if the temperatures drop below or rise above a predetermined threshold, he can carry on with other business knowing everything’s under control under Misty’s watchful “eye”.
The stereo infrared camera and wide-angle fish-eye lens
Misty’s stereo infrared camera is the mapping component of her Occipital Structure Core depth sensor. Physically speaking, the stereo infrared camera is a set of two, software-correlated cameras situated inside Misty’s visor.
Just as each of our eyes sees a slightly different scene at a given time and our brain can make sense of it, each camera that makes up Misty’s stereo vision system take pictures of the scene Misty “sees” and an algorithm is used to create the geometry of the environment, resulting in a detailed 3-D map.
This allows Misty to quickly start navigating her space autonomously which is helpful in any skill you build that requires her to move, including the distillery assistant example we shared above.
The second component to Misty’s Occipital Structure Core depth sensor is equally important to her autonomy: the wide-angle fish-eye lens. While the stereo infrared camera allows Misty to map out the environment she’s in, without the wide-angle fish-eye lens, Misty wouldn’t know where she is situated within that environment. Whether she’s static or moving, the wide-angle fish-eye lens camera allows Misty to know where her body is in space, how she’s angled, and whether there is something in her path (a flight of stairs, a cat, or something else).
Many of the Misty Skills that already exist as well as the “Future Misty Skills” incorporate Misty bringing something to or taking something from someone. This could mean delivering a perfectly poured beer to someone at the bar using a 3-D printed tin holder arm or it could mean helping a waiter to bus tables. In either case, Misty will need to navigate to and from people while avoiding obstacles in her path.
While these two types of camera are more focused on space versus visuals (as opposed to the RGB camera), both are also capable of generating 2-D visuals. Using Misty’s stereo infrared camera, you can generate an “occupancy grid” — a two-dimensional matrix of values indicating areas of open space, occupied space, and “covered” space. It’s easy to imagine how a picture like this could be useful in any business setting where efficient storage is necessary (such as a public storage facility or a warehouse) or where unfamiliar spaces need to be measured and maximized for efficiency (such as by an event planner or interior design planning). You can capture black-and-white photos using her fish-eye lens which is especially useful in a very large space.
Adding your own cameras
One of Misty’s key features is her extensibility meaning you can add new sensors (like cameras). In the Misty World Camera Day video, we used a Pixy camera with a Raspberry Pi which snaps onto the UART serial port on her backpack. Note that you must add another processor if you are adding an additional camera.
Here are some recommendations from some of the Misty Robotics team for anyone looking to add additional cameras to Misty:
Misty’s cameras enable essential capabilities such as facial recognition, 3D mapping, locomotion, photo capture, and video capture that will factor into the skills you build; adding additional camera sensors on top of those capabilities opens the door to even more use cases.
If you have ideas and questions, join the Misty Community Forum!