In Part 1 of Misty as a security guard, we learned how Misty uses facial recognition to detect unknown people in your space. Part 2 focused on Misty’s use of web services to communicate with IoT devices. So what’s left?
In Part 3, we learn how security guard Misty photographs intruders and transforms their portraits with a Raspberry Pi.
The “camera” category of robot sensors includes a diverse collection of hardware, and these sensors enable an assortment of robot capabilities. Learning to recognize objects? Mapping a new environment? Analyzing the chemical makeup of Martian soil? Whatever your task, there’s a camera for you.
Next to all the innovative use cases for cameras in robotics, taking pictures might seem tame. But photography can be a shared language between us humans and our high-tech companions. Our human ability to interpret visual images comes standard issue — no additional computation required. Pair this notion with a robot’s ability to go where you can’t and you unlock a bit of magic.
Misty’s cameras include a wide-angle grayscale camera, a depth camera, and a 13-megapixel color camera. She can’t use this hardware to vaporize rocks on Mars (yet), but she’s up for printing pictures of intruders she catches in your space. Ready to see how she does it?
Taking the photo
Misty’s API provides raw access to data from each of Misty’s cameras. The security guard job uses the 13-megapixel color camera. In Misty’s skills, you take a picture with this camera by calling
• whether to return the image data as a Base64 string
• what name to save the file with
• whether to resize the captured photo
• whether to display the photo on her screen
• whether to overwrite an existing image with the same filename as the new photo
In the security guard skill,
misty.TakePicture() is called in the block that executes when Misty doesn’t recognize a face. Misty saves the photo with the name
"Intruder" and sets the size of the image to 1200 x 1600 pixels. The photo doesn’t appear on Misty’s display, and if an image called
Intruder already exists, Misty overwrites it.
With these arguments in place, the command looks like this:
When this command executes, Misty takes a portrait of your intruder, who probably looks a little bewildered by your home robot security system. The next step is for Misty to do something interesting with the photo. For us, that means sending the image to a Raspberry Pi and printing a mugshot.
Printing the photo
Before the Raspberry Pi can print the photo, Misty needs to communicate that she’s captured a new portrait. She does this by calling the
misty.SendExternalRequest() command to send an update to Dweet.io. Dweet is a web service that internet-connected devices can use to post data objects (called “dweets”) online. Misty sends this request to Dweet after executing
When the security guard skill runs, you can see Misty’s latest dweet at https://dweet.io/get/latest/dweet/for/misty. It looks something like this:
Meanwhile, a Raspberry Pi on the same WiFi network as Misty runs a Python script with the logic for handling the mugshots Misty captures. This script uses libraries like Dweepy and OpenCV to check for dweets from Misty and manipulate intruder portraits in interesting ways.
When this script runs, it
The bulk of this action happens in the
prepare_image() function (defined on line 35 here). This function downloads the photo of the captured intruder via Misty’s REST endpoint for the
GetImage command. It uses the scikit-image library to save the image to a variable, like this:
After the image downloads,
With the image prepared, the Raspberry Pi connects to a printer and calls a function to print the new mugshot. It then reconnects to the robot’s network and jumps back into the loop to check for new
Misty’s security guard job builds on her core capabilities of face recognition, external HTTP calls, and image capture. In this series, we’ve learned about the technology behind each of these capabilities and explored how you can use Misty’s API to build a security guard skill yourself.
Be sure to check out the full repository for the security guard skill on GitHub, and browse the Misty Sample Skills organization for even more Misty skills. Have an idea for another job for Misty? Don’t keep it to yourself — share it with the Misty Community!