Create digital art with p5.js and Clarifai's color recognition model

p5.js is a great way to get started when making digital art for the browser with JavaScript. In just a few lines of code, you can start drawing interactive artwork which works on computers, phones and more. Here’s a video tutorial to help you explore p5.js and Clarifai’s Color recognition model!

On a personal level, I feel that making digital art is really exciting, and p5 took away lots of the scary parts and replaced it with a toolkit which was fun and easy to work with.

In this video tutorial, I expect that you have a very basic understanding of JavaScript (what a function is) and not anything else. Together, we’ll learn to use Clarifai’s Color Model to make an interactive representation of the colors in our image.

I’ve built it on a platform called Glitch, If you’d like to look at my complete project and ‘remix’ it in your own account, you can find it here. And if you have any questions at all or are wondering what else you can do with this project, feel free to find me on Twitter as @_phzn.


Use Clarifai's Face Detection model to find faces in images

Clarifai’s new Face Detection model finds faces in images and returns bounding box location coordinates. This model is useful for security camera footage, photo filter apps, dating apps, digital photography, and more. Here’s a copy + paste tutorial on how to use our Face Detection model and Javascript to build a fun face photo filter!

kevinlewis

The brand new Face Detection model has recently been released in alpha and does exactly what you should expect it to – you provide an image with some lovely faces (or not so lovely faces) and it returns the position of any faces it has found. I immediately thought of Snapchat’s photo/video filters as an application and wondered what would happen if I combined Clarifai’s Face Detection model with the 💩 emoji, because I am a serious and mature adult adulting all over the place.

This project isn’t too complex at all, and if you’re happy to just read (commented) code, here is a link to the GitHub repository for this project.

How is this going to work?

We’re going to wait for an image to be selected, and then use the Clarifai JavaScript client to get the position of the faces it’s found. Finally, we’ll take those values, relate them directly to our image and overlay poo emojis on their unsuspecting faces.

Get set up with this project

Firstly, we’ll need a Clarifai account. You can get one here. Create a new application and take note of your Client ID and Client Secret – don’t share these with anyone else.

Next, rename keys.example.js to just keys.js and paste in your Client ID and Client Secret.

Next, let’s set up our markup for this project. If you’re starting from scratch, create a file called index.html and make it look the same as mine. If you’re building this into an existing project, just make sure you have jQuery and the Clarifai JavaScript client included, and it has the following markup in the body:

<div class="input-group">
      <label for="image">Choose an image</label>
      <input type="file" id="image">
</div>
<img src="">
<canvas id="canvas"></canvas>

Let’s build this thing!

If you want to follow along with the finished JavaScript file, here it is.

Right, let’s do some housekeeping and create some initial variables which we will populate later.

var overlay = "💩";
var canvas, ctx;
var imageDetails = {
  clarifaiFaces: [],
  realFaces: []
};

The overlay variable contains the emoji which we wish to paste onto the faces we find in the image. The canvas and ctx variables will contain our HTML canvas context – which is how we’re going to draw the emojis on top of the image. Finally, the imageDetails object and properties will contain information about our image and face positions later in the file.

Next up, we’re going to initialize a new application using the Clarifai JavaScript client.

var app = new Clarifai.App(cId, cSec);

Now let’s create the code which will be run when the file input has a file selected. I’ve split this application up into three functions, which run one after another. You should hopefully see the flow from one function to the next.

$("input#image").on("change", function() {
  if(this.files[0]) {
    var reader = new FileReader();
    reader.onload = function(e) {
      imageDetails.b64 = e.target.result;
      $("img").attr("src", imageDetails.b64);
      imageDetails.b64Clarifai = imageDetails.b64.replace(/^data:image\/(.*);base64,/, '');
      imageDetails.width = $("img").width();
      imageDetails.height = $("img").height();
      faceDetection(imageDetails.b64Clarifai);
    }
    reader.readAsDataURL(this.files[0]);
  }
});

This is an event listener, and is waiting for the file input to change. If a file has been selected, we do the following:

  1. Convert the image to a base-64 encoded string and insert the string into the <img> src. This displays it on the screen. This string is stored in imageDetails.
  2. Creates a version of the base-64 encoded string without metadata, which is what Clarifai needs. This is also stored in imageDetails.
  3. Stores the images width and height in imageDetails too, which we’ll use later to determine where the faces are in our image.

This function finishes by firing off our faceDetection() function. Let’s take a look at how it’s put together.

function faceDetection(b64Img) {
  app.models.predict("a403429f2ddf4b49b307e318f00e528b", {
    base64: b64Img
  }).then(
    function(res) {
      var data = res.outputs[0].data.regions;
      if (data !== null) {
        for (var i = 0; i < data.length; i++) {
          imageDetails.clarifaiFaces.push(data[i].region_info.bounding_box);
        }
      }
      drawBoxes();
    },
    function(err) {
      console.log(err);
    }
  )
}

This function takes the Clarifai-ready base-64 encoded string and calls the face detection alpha model. It waits for a response and pushes each of the bounding boxes (the location information of each face) into the imageDetails.clarifaiFaces array.

The important thing to note about these values is that they provide the top-left and bottom-right positions of each box with values ranging from 0 to 1. We will later have to relate these to the actual position on our image using the width and height propertieswe stored earlier.

But I’m getting ahead of myself. Let’s look at the final function which is called at the end of faceDetection(), and that’s a function called drawBoxes().

function drawBoxes() {
  canvas = document.getElementById("canvas");
  $(canvas).attr("width", imageDetails.width).attr("height", imageDetails.height);
  ctx = canvas.getContext("2d");
  ctx.textBaseline = "top";

  for(var i=0; i<imageDetails.clarifaiFaces.length; i++) {
    box = {
      x: imageDetails.clarifaiFaces[i].left_col * imageDetails.width,
      y: imageDetails.clarifaiFaces[i].top_row * imageDetails.height,
      w: (imageDetails.clarifaiFaces[i].right_col * imageDetails.width) - (imageDetails.clarifaiFaces[i].left_col * imageDetails.width),
      h: (imageDetails.clarifaiFaces[i].bottom_row * imageDetails.height) - (imageDetails.clarifaiFaces[i].top_row * imageDetails.height)
    }
    imageDetails.realFaces.push(box);
    ctx.font = (box.w * 1.4) + "px monospace";
    ctx.fillText(overlay, box.x - (box.w / 5), box.y - (box.h/4));
  }
}

The first line of this function stores the canvas element in the canvas element we set up at the start of this file. The second line makes the canvas the same size as the image, in both its width and its height. Next, we store the canvas context to the ctx variable – this is how we will interact with the canvas when drawing to it. Finally for the canvas setup, we set the baseline to the top, which is required to place the emoji properly.

Next, we iterate over the bounding boxes which we received from Clarifai and do some arithmetic which does the following for each box:

  1. Takes the top-left values from being between 0 and 1 to the correct pixel values on the image (for example, a box top-left position may be (0.124, 0.52), but these ais assuming the bottom-right of the image is (1, 1). We make them into pixel-correct values).
  2. Takes the bottom-right values and change them to show the width and height of the image (if the image width starts at 1.2 and ends at 2.5, then the width is 1.3, for example).
  3. We store both the top-left, width and height values for each box in imageDetails.realFaces.
  4. We draw the emoji set at the top of the image to the position and size of the box.

So wait, how does this work again?

There you have it – a poopifying app using face detection. Of course, you can be more generous with your friends and use hearts, unicorns, or other emojis to show you care – we’re not like that. Share your particular use case with @Clarifai for a chance to be featured in the blog!

I learned how to code #💩 over all your faces with #JavaScript + #Clarifai Face Detection model … GET READY 💩💩💩


How to check user-generated content and images for unwanted nudity

User-generated content is like a box of chocolates filled with potentially unwanted nudity – you never know what you’re going to get. Whether you’re a developer building an app that relies on user-uploaded listings (like AirBnb or eBay) or you’re helping a brand run an online contest with user-generated content (like GoPro), you can use this simple and effective method for checking user-generated content for Not Safe for Work subject matter and preventing it from being uploaded and shared.

kevinlewisThe Not Safe For Work (NSFW) model is super useful for moderation and filtering user-generated content, especially stopping questionable content from being seen by users on platforms where they are not expecting it. Recently, I built a really small JavaScript-based project which will automatically check that an image selected for upload in a form is Safe For Work (SFW) before it can be submitted.

Before we start though, it may be worth noting that this is a completely client-side based project. If you require something more robust, I’d recommend that you look into implementing a similar solution as part of your data validation server-side. It also required JavaScript, so it won’t work if this has been disabled.

Right, the disclaimers out of the way, let’s crack on. If you just want to jump ahead and look a the code, the GitHub repository is here, and the readme should be enough to get you started. The code is also heavily commented, so it should be good if you’re comfortable enough to jump straight in.

How is this going to work?

We’re going to disable the form submit button with JavaScript (this means that the form will still submit if JS is turned off), and then check if the image is SFW. If it is, we will enable the form submit. If it does not pass the test, the button remains disabled.

Get set up with this project

Firstly, we’ll need a Clarifai account. You can get one here. Create a new application and take note of your Client ID and Client Secret – don’t share these with anyone else.

Next, let’s set up our markup for this project. If you’re starting from scratch, create a file called index.html and make it look the same as mine. If you’re building this into an existing project, just make sure you have jQuery, and the Clarifai JavaScript client.

Now to create an keys.js file. It’s important that this is a separate file as it will house our Client ID and Secret. If you’re using git, please make sure to add this file to your .gitignore so you do not share this information.

Finally for project setup, I’d like you to create an options.js file. This is where our configuration will happen. Make it look like mine. Here’s a rundown of the options:

  • The SFW_LOWER_LIMIT variable sets the lowest acceptable SFW value (on a scale from 0 to 1) which will pass the test.
  • The FORM_ID variable is the ID given to the form we’re conducting a test on.
  • The FILE_CLASS variable is the class which the specific file input is given.
  • The function clarifaiCheckPass() will run if the user’s image passes the test, and clarifaiCheckFail() will run if it does not.

Let’s build this thing!

If you want to follow along with the finished JavaScript file, here it is.

First of all, let’s initialize a new application using the Clarifai JavaScript client.

var app = new Clarifai.App(CLIENT_ID, CLIENT_SECRET);

Next, a small bit of boilerplate code which will take an image file from an input and convert it to a Base64-encoded string. It’s this string which Clarifai needs to accept in order to run it through the NSFW model.

File.prototype.convertToBase64 = function(callback) {
  var reader = new FileReader();
  reader.onload = function(response) {
    base64 = response.target.result.replace(/^data:image\/(.*);base64,/, '');
    callback(base64);
  };
  reader.onerror = function(err) { callback(err); };
  reader.readAsDataURL(this);
};

Let’s introduce a way to visually see what the state of a file input is. We’re going to do this by adding the class ‘working’ to the input when it has been submitted to Clarifai, and then changing it to either ‘approved’ or ‘rejected’ once we get a result. I suggest styling them with CSS for quick visual feedback of state.

function stateClass(classToAdd) {
  $("#" + FORM_ID + " ." + FILE_CLASS).removeClass("working approved rejected");
  $("#" + FORM_ID + " ." + FILE_CLASS).addClass(classToAdd);
}

Now for the heavy lifting, although luckily Clarifai will make it a lot easier than you might expect. In this next function, we’re going to make a call to Clarifai. Below is the code, and we’ll run through a detailed explanation afterwards.

function validateFileInputs(image) {
  stateClass("working");
  app.models.predict(Clarifai.NSFW_MODEL, {base64: image}).then(
    function(response) {
      var pass; 
      data = JSON.parse(response.request.response);
      concepts = data.outputs[0].data.concepts;
      $.each(concepts, function(k, v) {
        if(v.name == "sfw") {
          pass = v.value > SFW_LOWER_LIMIT;
        }
      });
      parseResponse(pass);
    },
    function(err) { console.log(err); }
  )
};

We feed this function an image (by this point it would have been converted to a Base64-encoded string). We then use app.models.predict() and tell it which model we’re querying against (NSFW) and give it the image string.

It will then go through the results looking for the SFW concept and return true if the value is higher than the lower limit, and false if it does not. Finally, it calls the parseResponse() function which we’ll go through below with the true or false result.

parseResponse() is an incredibly simple function. It sets the correct state to the form input, and then calls clarifaiCheckPass() or clarifaiCheckFail(), which we define in the options file.

function parseResponse(pass) {
  if(pass === true) {
    stateClass("approved");
    clarifaiCheckPass();
  } else {
    stateClass("rejected");
    clarifaiCheckFail();
  }
}

Pulling it all together

So far we’ve written all of these useful function together, but we’re not ever calling them. There are two pieces of functionality left to get this application working – the first is to actually trigger the NSFW checker when the form file input is changed, and the second is to stop the form being submitted until the check has passed.

$(document).ready(function() {

  $("#" + FORM_ID + " ." + FILE_CLASS).on('change',function(){
    var selectedFile = this.files[0];
    selectedFile.convertToBase64(function(base64){
      validateFileInputs(base64);
    });
  });

  $("#" + FORM_ID + " input[type=submit]").click(function(e) {
    e.preventDefault();
    if($("#" + FORM_ID + " ." + FILE_CLASS).hasClass("approved")) {
      console.log("clicked");
      $(this).unbind('click').submit();
    }
  });
});

So wait, how does this work again?

There you have it – your very own nudity checker for your online forms. You can use this as a lightweight way to solve many problems, like making sure users on your dating app can only submit tasteful, non-nude pics, or making sure the photo contest you’re running doesn’t sear your eyeballs unexpectedly. Share your particular use case with @Clarifai for a chance to be featured in the blog!

♥️ this Copy+Paste guide – taught me how to make apps check if images are NSFW using #Javascript + #Clarifai API