The brand new Face Detection model has recently been released in alpha and does exactly what you should expect it to - you provide an image with some lovely faces (or not so lovely faces) and it returns the position of any faces it has found. I immediately thought of Snapchat’s photo/video filters as an application and wondered what would happen if I combined Clarifai’s Face Detection model with the 💩 emoji, because I am a serious and mature adult adulting all over the place.
This project isn’t too complex at all, and if you’re happy to just read (commented) code, here is a link to the GitHub repository for this project.
How is this going to work?
Get set up with this project
Firstly, we’ll need a Clarifai account. You can get one here. Create a new application and take note of your Client ID and Client Secret - don’t share these with anyone else.
Next, rename keys.example.js to just keys.js and paste in your Client ID and Client Secret.
Let’s build this thing!
Right, let’s do some housekeeping and create some initial variables which we will populate later.
The overlay variable contains the emoji which we wish to paste onto the faces we find in the image. The canvas and ctx variables will contain our HTML canvas context - which is how we’re going to draw the emojis on top of the image. Finally, the imageDetails object and properties will contain information about our image and face positions later in the file.
Now let’s create the code which will be run when the file input has a file selected. I’ve split this application up into three functions, which run one after another. You should hopefully see the flow from one function to the next.
This is an event listener, and is waiting for the file input to change. If a file has been selected, we do the following:
- Convert the image to a base-64 encoded string and insert the string into the <img> src. This displays it on the screen. This string is stored in imageDetails.
- Creates a version of the base-64 encoded string without metadata, which is what Clarifai needs. This is also stored in imageDetails.
- Stores the images width and height in imageDetails too, which we’ll use later to determine where the faces are in our image.
This function finishes by firing off our faceDetection() function. Let’s take a look at how it’s put together.
This function takes the Clarifai-ready base-64 encoded string and calls the face detection alpha model. It waits for a response and pushes each of the bounding boxes (the location information of each face) into the imageDetails.clarifaiFaces array.
The important thing to note about these values is that they provide the top-left and bottom-right positions of each box with values ranging from 0 to 1. We will later have to relate these to the actual position on our image using the width and height propertieswe stored earlier.
But I’m getting ahead of myself. Let’s look at the final function which is called at the end of faceDetection(), and that’s a function called drawBoxes().
The first line of this function stores the canvas element in the canvas element we set up at the start of this file. The second line makes the canvas the same size as the image, in both its width and its height. Next, we store the canvas context to the ctx variable - this is how we will interact with the canvas when drawing to it. Finally for the canvas setup, we set the baseline to the top, which is required to place the emoji properly.
Next, we iterate over the bounding boxes which we received from Clarifai and do some arithmetic which does the following for each box:
- Takes the top-left values from being between 0 and 1 to the correct pixel values on the image (for example, a box top-left position may be (0.124, 0.52), but these ais assuming the bottom-right of the image is (1, 1). We make them into pixel-correct values).
- Takes the bottom-right values and change them to show the width and height of the image (if the image width starts at 1.2 and ends at 2.5, then the width is 1.3, for example).
- We store both the top-left, width and height values for each box in imageDetails.realFaces.
- We draw the emoji set at the top of the image to the position and size of the box.
So wait, how does this work again?
There you have it - a poopifying app using face detection. Of course, you can be more generous with your friends and use hearts, unicorns, or other emojis to show you care - we're not like that. Share your particular use case with @Clarifai for a chance to be featured in the blog!