Misty knows the importance of playing as hard as you work. That’s why she’s willing to risk a few grass stains in her Follow Ball skill.
In this skill, Misty employs the object recognition capabilities of a Pixy2 vision sensor to chase a soccer ball as it moves around the room. Here’s a high-level overview of how it works:
• A Pixy2 – “trained” to recognize and detect a soccer ball – sends image data to a microcontroller connected to Misty.
• The microcontroller calculates the distance and heading from Misty to the ball, then sends these values to the code running on Misty by way of the UART serial port built-in to Misty’s back.
• When Misty receives this data, she transforms it into linear and angular velocity values for her drive command and moves toward or away from the ball.
• The Pixy2 sends updated information to the microcontroller, and the process repeats with the new image data.
This post breaks down each point in a little more detail, so you can build your own Follow Ball skill for Misty. We’ll start by looking at the Pixy2.
Priming the Pixy2
The Pixy2 is a lightning-fast vision sensor for robots. Training the Pixy2 to recognize an object is as simple as holding the object in front of the sensor and pressing a button. It’s built to easily connect with microcontrollers like an Arduino or Raspberry Pi, and Pixy provides libraries to simplify using image data in your code.
In the Follow Ball skill, we connect the Pixy to the Misty (Arduino-Compatible) Backpack using the ISCP connectors on the microcontroller. We use a 3D-printed attachment to position the sensor front-and-center on Misty’s base. Once this setup is done, we “teach” the sensor to recognize a soccer ball by pointing it in the right direction and holding down the button on the Pixy2.
Now that the Pixy2 knows the object, we’re ready to write the code for the microcontroller.
Preparing the microcontroller
For this skill, Misty uses the microcontroller in her own Arduino-compatible backpack. This microcontroller is a clone of the Arduino Uno that’s been changed in a few important ways. One of these changes is the addition of magnets that attach the board to Misty’s back for a direct connection to her UART serial port.
We call code that runs on Misty’s Arduino-compatible backpack a “sketch”. You write a sketch with the same tools used to program an Arduino Uno. To process data from the vision sensor, we include the Pixy2 Arduino library in the sketch.
We initialize the pixy object in the
setup() function of the sketch. The
Serial.begin() function from the Serial library opens a serial port and sets the data transfer rate to 9600 baud (this is the rate for communicating with Misty’s UART serial port). We also set the values of a few variables used in calculating the ball’s movement.
Before she can execute her drive command, Misty needs to know her position relative to the ball. Specifically, she needs to know her heading, or the angle between the direction she’s facing and the ball’s position. Additionally, she needs to know the ball’s distance from her and its current direction of movement.
loop() function in this sketch transmits all of this information to Misty once every 200 milliseconds. In the
loop() function, we call
pixy.ccc.getBlocks() to return data about objects the vision sensor detects. This data includes the width and height of objects in the frame (in pixels), which lets us approximate a value for how close the ball is to Misty. A larger size is closer, and a smaller size is more distant.
We get Misty’s heading by calculating the difference between the center of the image and the X coordinate of the center of the ball within the frame. To determine which direction the ball is moving, we compare this value to the heading angle from the last loop.
We then package all this data –
pan (heading), and
direction – into a stringified JSON object and send it to Misty using
Serial.println(). Check out the sketch on GitHub, or see the
loop() function below.
The code running on Misty processes each message the microcontroller sends. It converts this data into arguments that, when passed into Misty’s drive command, move her closer to (or further away from) the ball.
To manage this in the skill, we use
misty.RegisterEvent() to register for
StringMessage events. These events occur each time the microcontroller sends a message through the receiver (RX) pin on Misty’s UART serial port.
By default, when
StringMessage events occur, the skill looks for a
_StringMessage() callback function to pass the event data. This data includes any messages the microcontroller sends to Misty. You define how to process this data in the function definition for the
In the Follow Ball skill, the
_StringMessage() callback parses out the data from the microcontroller and converts it into
angularVelocity values for the
misty.Drive() method. Each of these values is an integer between -100 and 100. For
linearVelocity, a value of -100 translates to “drive backward at max speed”, while 100 means “drive forward at max speed”. For
angularVelocity, -100 means “rotate clockwise at max speed”, and 100 means “rotate counterclockwise at max speed”.
We use the size of the ball in the image from the Pixy2 to calculate an appropriate value for the
linearVelocity argument. If the ball appears smaller than it should, Misty drives toward it, and if it appears larger, she backs away. The value for
angularVelocity correlates with Misty’s heading – we calculate a value that has Misty rotate or drive in a curve toward the ball. See lines 35 – 72 of followBall.js to find out how these calculations are made in the code.
Misty recalculates these values each time the microcontroller sends new image data, so she’s constantly adjusting her speed and position relative to the ball. All this adds up to a dynamically playful robot, optimized for cuteness. Watch the video to see what we mean.
Extensibility means limitless potential
You’d be hard-pressed to find a robot whose hardware enables every job you dream up. That’s what makes extensible platforms so compelling. A robot with robust native capabilities that you can augment with third-party hardware has a lot of opportunities for employment, not to mention creative recreation.
Don’t miss the full repository for the Follow Ball skill on GitHub,and if you’re interested in more talk about robots, be sure to join the conversation in the Community Forums.