How Clarifai builds accurate and unbiased AI technology

At Clarifai, we have a team called the Data Strategy Team, a group of conscientious, diverse people who train our visual recognition models. They ensure we’re building accurate, unbiased AI – learn about what they do and how you can apply their best practices to your own model building!

Back in the early ages of computing, companies used to rely on teams of people doing complex calculations by hand. These operations now take a fraction of a second for a computer to do. Today, it seems like every company wants to build or incorporate artificial intelligence that can make tasks faster and more accurate than when they were done by humans. However, there are things about the human mind that we are not able to fully replicate with a computer alone (yet!).

We have a fantastic team of minds at Clarifai we call the Data Strategy Team that helps us curate and assess the quality of data for creating robust AI models. Along with research, engineering, and our client success teams, the team distills all the feedback they get from every side on custom models, and work to constantly improve the API in a way that best reflects the big, beautiful world. The team’s diverse backgrounds allow for them to see something that others may not. When building an AI model, the team has to ask, “What are we supposed to see? Is what we are asking to find visible and distinguishable? If we aren’t able to answer these questions ourselves, does it make sense for us to ask a computer to do this?” Here are some of our Data Strategy Team’s tips to consider when building out a model!

Break down the visual components

AI models receive inputs so we need to make sure that our inputs have the correct elements for the model to understand. What if we were to make a model and we wanted it to recognize a leaf on a plant? When we give it several images of various species, we are educating the model on different shapes, colors and textures leaves can take on. We have these visually tangible aspects for it to recognize. What if we wanted to train our model to identify an emotion like anger? Anger is expressed differently by different cultures and people.

When trying to teach a more metaphysical concept, you need to be sure that your input represents those variations as well. Determine the things that represent your concept and make sure that examples of them get incorporated to the training set. This will achieve higher accuracy for what you want your model to focus on. You’ll be able to refine the accuracy after you evaluate your input.

Incorporate relevant training data

One of the biggest misconceptions about AI models is that they recognize everything correctly each time. A model is only as good as the data that is used to train it. A model could fail to make accurate predictions due to training data that doesn’t look like what it will be tested on. Imagine if you wanted to build a model that could detect different items for recycling. Yet, all of your training data is of stock photography of objects that are on tables and being held by people either drinking or eating. When you want someone to use this model, is this the way that we intend to use it? Would the model detect photos of people’s trash bins out in the world? Probably not.

Not only should you make sure the data you incorporate is relevant but also you should make sure that your training data has the same visual aspects as the intended test data. Will your test data be inverted or blurry? Will it be on grayscale versus colorized? These can impact a model’s precision and accuracy too. A model is simply a block of clay and it is your job to make its shape as effective as possible.

Remove biases at all costs

Just like human beings, the artificial intelligence model is susceptible to what it is taught. To the model, the inputs are a source of truth that describe its world and it can only understand the world from its teachings. We have seen this to be true in the extreme cases of Google’s misprediction of tags for photos or Microsoft’s chatbot Tay. When we are shaping the models, we want to make sure that we aren’t introducing any of our human biases.

When you are giving concepts that describe a profession, you may want to be representative of all the demographics involved rather than merely the most prominent. Even well established datasets can be biased to the culture that they were found in. Look at FaceScrub, a popular dataset for celebrity face detection.

This dataset contains white-majority celebrities. We could increase its effectiveness by incorporating more celebrities from other parts of the world. If we don’t acknowledge our biases when we  gather a set of data, we only build for what we know rather than looking beyond that.

Where to go from here?

Machine Learning models are often trained on data blindly scraped from the Internet. After all, it’s easy to use search terms on thousands of images and then upload them as training data.

However, this doesn’t reflect on how diverse our world is. With these tips, you are equipped with realizing these nuances and building models that give meaningful results. Know that at Clarifai, we are aware of these possible influencers and we use our Data Strategy Team to carefully improve our neural net models. The team collaborates with our enterprise customers to make sure to address their needs and iterate on building models that can enhance a platform’s experience. If you have any questions or want to learn more about building effective models, reach out to us at hackers@clarifai.com!


How Visual Similarity and Custom Metadata can Enhance Your Search

Data can be thought of simply as a thing we want to remember for future use. Metadata helps describe the data we’re trying to remember. So when we are putting it in context of visual similarity, we are adding metadata to describe inputs (data) that may not be visually distinguishable. Here’s what that means for searching images!

When we say metadata at Clarifai, this describes some attribute relating to a particular image that may or may not be visibly tangible. Think about it as where we could store all of the information we want the image to hold onto that is valuable to our project or business. Here are a few examples:

  • We could have users searching through a catalog and we want to record which product they viewed. We would be able to reference metadata of the product’s ID without having to make a callback to our database.
  • We could add metadata to our inputs so that when a user comes along and wants to find an item nearby, we filter based on a zip code or region.

We can craft Custom Metadata to suit any need.

Let’s go through an example in-depth. We’re going to look at the case of a shoe store and how we can search using an image and our metadata. We will be able to find items that are visually similar to what we want and also filter items based upon them being on sale. If you want to see all of the code already written up, you can check out this GitHub repo and its README.

Requirements

If you haven’t already, make sure to sign up for a free account on Clarifai and create an application. We will also need to be sure to have NodeJS installed.

We have a rather short list of data for our little footwear shop. It is a CSV with several columns for each of the data points we want to represent: Product ID, Type, Color, Brand, Price, On Sale, In Stock, Image Source. Let’s use the prebuilt data here for our example and save it to our project folder as shoe-data.csv.

Convert CSV and upload to Clarifai

The team over at Adaltas decided to share their code to help people parse data from spreadsheets in Node. It allows for flexibility of our data, so give them some kudos. We will install it with npm install --save csv-parse in Terminal. Before this data is useable we need to convert each row over to a JSON Object. Open up a new file upload.js and we will write all of our actions here:

/* upload.js */
const parse = require('csv-parse')
const fs = require('fs')

// ~Hidden magic we will come back to~

fs.readFile(__dirname+'/shoe-data.csv', 'utf8', (err, data) => {
  if(err) { return console.log(err) }
  parse(data, { columns: true }, (err, output) => { 
    if(err) { return err; }
    shoeData = output.map((shoe) => { return convertData(shoe) })
    uploadInputs(shoeData);
  });
})

We still have the hidden magic to fill in that explains our convertData() and uploadInputs()! We have to start with adding in the code that will let us use Clarifai. To install the Clarifai JavaScript client, go to Terminal and write npm install clarifai --save. Include the client with const Clarifai = require('clarifai') right below the other modules. Be sure to place our API Key for the application:

/* upload.js */
const parse = require('csv-parse')
const fs = require('fs')
const clarifai = require('clarifai')

const app = new Clarifai.App({ apiKey: 'YOUR_API_KEY' })

// ...

Now we want to be sure to write out the convertData() and uploadInputs() functions. convertData() will take our results from reading the CSV and then convert the data into inputs with metadata. uploadInputs() then takes that data and sends it to Clarifai for it to store.

/* upload.js */
const parse = require('csv-parse')
const fs = require('fs')
const Clarifai = require('clarifai')

const app = new Clarifai.App({ apiKey: 'YOUR_API_KEY' })

const uploadInputs = (inputs) => {
    app.inputs.create(inputs).then(
      // Success
      (response) => { console.log('Successful Upload!') },
      // Error
      (error) => { console.error(error) }
    )
}

const convertData = (data) => {
  return {
    url: data.imageSrc,
    metadata: data
  }
}
// ...

With us running node upload.js we have everything handled for taking a CSV and uploading it onto Clarifai.

Searching on Clarifai

Let’s write another small script that will perform our search which we will cleverly name search.js. We will use this image and then apply metadata for items on sale as TRUE. Our image would filter what item a user would want to have. This is how we incorporate visual similarity into our search. However, there is no visual cue in the image alone to find out which items are on sale. That’s a great reason why we need to apply the metadata.

/* search.js */
const Clarifai = require('clarifai')

let app = new Clarifai.App({ apiKey: 'YOUR_API_KEY' })

// Searching by visual similarity
app.inputs.search([
    {
      input: {
        url: 'https://farm4.staticflickr.com/3370/3344620504_b547190891_o_d.jpg'
      }
    },
    {
      input: {
        metadata: {
          sale: 'TRUE'
        }
      }
    }])
.then((response) => {
  response.hits.map(
    (hit) => {
      console.log(`Price: $${hit.input.data.metadata.price}USD; URL: ${hit.input.data.image.url}`)
  })
})

We can run it with node search.js in Terminal. We get a list to all of our prices and the URLs associated with the items. An important note is that our metadata field is sensitive to how the data is searched. If we have capitals in any of our keys or values, we need to be sure that they match exactly when we go to search. Otherwise, they will consider the object you put in and the object you wanted to find as different things.

Conclusion

We’ve just played around with one way to use metadata. Remember, we added a bunch of different labels onto our inputs. Some metadata may be more valuable than others but it is entirely freeform. If you are curious about some more search features such as by geo location, public concepts or anything else, read more about it all in our guide. Let us know if you need any help or how you make use of custom metadata at hackers@clarifai.com!


Search images by visual similarity with the Clarifai API

When you’re searching for images, words are often not enough to find exactly what you need. Wouldn’t it be amazing if you could just show your computer a picture and say, “Find me images that look like this?” With the Clarifai API, you can search for any image by visual similarity – here’s how!

The Clarifai Search API has a variety of different ways for you to query your inputs. In one of our previous posts, we talked about searching your content by geo location. In this post, we will have an image do all of the talking. You won’t need to do any training on your dataset; simply upload images and then you can search over those images seamlessly.

Getting Started

For the first step, you need to have Python (version 3.6.2) installed for your appropriate operating system. You can head over to a terminal and write in python --version. If you happen to have a version of Python 2 installed you may need to try python3 --version. You should see a prompt come up reading: Python 3.6.2. If neither of these work, you may need to check if Python is included in your PATH environment.

Many people who develop in Python suggest to have a “virtual environment”. These help to manage any application-specific dependencies. Having one won’t be necessary for this tutorial but if you are curious you can read more about them in the Python documentation.

Now, install the official Python Clarifai client using pip install clarifai. The last thing you’ll need to do is make sure to sign up for a free Clarifai account and create an application.

Adding your inputs

You will need to have a dataset of inputs to search against. These inputs get indexed on your application and Clarifai will be able to “see” the image. To a machine, an image is nothing but vectors that describe what each pixel looks like. When you perform a search using an image, Clarifai will look to see how close these vectors are to one another to determine if the images are visually similar. The visual similarity can be more effective if your dataset includes objects that are relevant to what you’d like to search against. If you were to add a dataset full of only food and try to search using an image of a dog your search results wouldn’t be as strong.

Here’s a dataset of images from ImageNet on food. You should see a file containing nothing but lines of URLs of images of food. Save this file as food-data.txt in the same directory as your code then we will take this file and upload the images in batches.

# upload.py
import os
from clarifai.rest import ClarifaiApp
from clarifai.rest import Image as ClImage

app = ClarifaiApp(api_key='YOUR_API_KEY')

FILE_NAME = 'food-data.txt'
FILE_PATH = os.path.join(os.path.curdir, FILE_NAME)

# Counter variables
current_batch = 0
counter = 0
batch_size = 32

with open(FILE_PATH) as data_file:
    images = [url.strip() for url in data_file]
    row_count = len(images)
    print("Total number of images:", row_count)

while(counter < row_count):
    print("Processing batch: #", (current_batch+1))
    imageList = []

    for current_index in range(counter, counter+batch_size - 1):
        try:
            imageList.append(ClImage(url=images[current_index]))
        except IndexError:
            break

    app.inputs.bulk_create_images(imageList)

    counter = counter + batch_size
    current_batch = current_batch + 1

Wowza! Over one thousand images being uploaded effortlessly.

Searching using an Image

Now the amazing part you’ve been waiting for is being able to search by visual similarity using an image. Let’s say you wanted to find out if there is anything similar to this picture of cookies. All you would need from here is:

# search.py
from clarifai.rest import ClarifaiApp

app = ClarifaiApp(api_key='YOUR_API_KEY')

# Search using a URL
search = app.inputs.search_by_image(url='https://images-gmi-pmc.edge-generalmills.com/cbc3bd78-8797-4ac9-ae98-feafbd36aab7.jpg')

for search_result in search:
    print("Score:", search_result.score, "| URL:", search_result.url)

The search_by_image() function returns a list of Image objects that wrap the response but we will have it print out the score and the url associated with it. You can also use image bytes or a filename to query against.

# Response
Score: 0.8486366 | URL: http://farm4.static.flickr.com/3502/4000853007_0f1e33cdc0.jpg
Score: 0.79205513 | URL: http://farm4.static.flickr.com/3213/3080197227_e4b28c76ae.jpg
Score: 0.7901007 | URL: http://farm1.static.flickr.com/222/470272746_1674448c07.jpg
Score: 0.741455 | URL: http://farm4.static.flickr.com/3317/3289848643_bf1f2e7b5b.jpg
Score: 0.7173992 | URL: http://farm4.static.flickr.com/3620/3473362088_c90b72c819.jpg
Score: 0.68365324 | URL: http://farm1.static.flickr.com/150/365771958_06e87421d1.jpg
Score: 0.6734046 | URL: http://farm4.static.flickr.com/3077/3160541712_b879bf7a22.jpg
Score: 0.6723133 | URL: http://farm4.static.flickr.com/3399/3185443954_26bf37dc8a.jpg
Score: 0.66024935 | URL: http://farm3.static.flickr.com/2481/3943873688_f094d211a3.jpg
Score: 0.6529919 | URL: http://farm3.static.flickr.com/2124/2239822705_419fffe609.jpg
Score: 0.6473093 | URL: http://farm2.static.flickr.com/1319/540399285_142ae1822e.jpg
Score: 0.63564813 | URL: http://farm3.static.flickr.com/2030/2310386017_8741472785.jpg
Score: 0.6230167 | URL: http://farm4.static.flickr.com/3338/3501581193_be17c2d04e.jpg
Score: 0.61543244 | URL: http://farm1.static.flickr.com/228/499181350_b01a280789.jpg
Score: 0.61172754 | URL: http://farm4.static.flickr.com/3169/2802528597_0483e7aa39.jpg
Score: 0.6057524 | URL: http://farm3.static.flickr.com/2111/2413962121_41b412c39c.jpg
Score: 0.60092676 | URL: http://farm1.static.flickr.com/118/296736486_e721b93e82.jpg
Score: 0.60023034 | URL: http://farm3.static.flickr.com/2264/2410275528_d7a69df963.jpg
Score: 0.5992471 | URL: http://farm3.static.flickr.com/2317/2435454915_2947203717.jpg
Score: 0.5968622 | URL: http://farm3.static.flickr.com/2664/3979835380_7748ddf164.jpg

The search response will show a value for score from 0 to 1. A score closer to 1 means the image is more visually similar; a score closer to 0 means the image is less visually similar. The response from Clarifai also defaults from the top 20 results. If you want to change, that you can add a different value for the per_page parameter in the search_by_image() function.

Conclusion

You’re probably thinking to yourself, “That’s all it took?” and the answer is yes. The part that took the longest was getting the dataset. Otherwise, performing our search was only five lines of code. The Search API doesn’t stop here:

If you have any questions, concerns, or even friendly notes feel free to reach out to us over at hackers@clarifai.com or comment below!


Getting Started with Search by Geo Location

Fact: Clarifai’s Search API allows you to search your images and video by visual similarity. Lesser known fact: it also lets you search your media by geo location data! Learn more about this feature and how to put it into action for your own developer application.

The Search API allows you to send images to Clarifai and have them indexed using concepts and their visual representation. After they are indexed, you can search over your inputs using concepts (e.g. dog) or images (e.g. visually similar dogs). Clarifai can extend this search by adding extra data points onto our inputs such as geo location data. A search by geo location acts as a filter of inputs so you get only results within a specified range.

We’ll look at attaching geo location data to your inputs and then querying that data with different measurements of distance to see the change in results. For this tutorial, we are going to use Node.js. Let’s get started by installing the official Clarifai JavaScript client with

npm install clarifai

Adding Inputs

You need to sign up for Clarifai and create an application before you can get started. Inputs are added using either a URL or bytes. Along with your input source we’ll add a geo object that will contain keys for GPS coordinates (longitude and latitude). Remember that the coordinate system is using the cardinal directions: North, South, East, and West. North and East will be positive numbers and South and West will be negative numbers.

Below we will add an image of the Statue of Liberty using a URL from Wikipedia:

const Clarifai = require('clarifai')
const app = new Clarifai.App({ apiKey: 'YOUR_API_KEY_HERE' })

app.inputs.create({
  url: "https://upload.wikimedia.org/wikipedia/commons/thumb/a/a1/Statue_of_Liberty_7.jpg/1200px-Statue_of_Liberty_7.jpg",
  geo: {
    latitude: 40.689247,
    longitude: -74.044502
  }
})

and use a local image of the Golden Gate Bridge from HISTORY and a function for file-to-base64 conversion:

const Clarifai = require('clarifai')
const app = new Clarifai.App({ apiKey: 'YOUR_API_KEY_HERE' })

const convert_bytes = (img) => {
  const img_file = fs.readFileSync(img)
  return new Buffer(img_file).toString('base64')
}

app.inputs.create({
  base64: convert_bytes('./golden_gate.jpg')
  geo: {
    latitude: 37.807812,
    longitude: -122.475164
  }
})

Searching for Images

Once your images are uploaded, you will be able use them with search. When searching by geo location, you can refine your results using a single point and some radius given ‘withinMiles’, ‘withinKilometers’, ‘withinDegrees’, or ‘withinRadians’.

Let’s say we only want results of images within a mile of the Empire State Building in New York City (because it’s the best city in the world, naturally). Our search would look like this:

app.inputs.search({
  input: {
    geo: {
      latitude: 40.748817,
      longitude: -73.985428,
      type: 'withinMiles',
      value: 1.0
    }
  }
}).then((response) => { console.log(response.hits)})

// Response
[]

The above example returned no results due to how small the range is. If we were to increase this value we will get our hit.

app.inputs.search({
  input: {
    geo: {
      latitude: 40.748817,
      longitude: -73.985428,
      type: 'withinMiles',
      value: 7.0
    }
  }
}).then((response) => { console.log(response.hits)})

// Response
[{ 
  score: 1,
  input: { 
    id: 'd7b80aac52f14399b98a9472fc201e64',
    data: [ 
      { 
        score: 1,
        input: { 
          id: 'd7b80aac52f14399b98a9472fc201e64',
          data: { 
            image: { 
              url: 'https://upload.wikimedia.org/wikipedia/commons/thumb/a/a1/Statue_of_Liberty_7.jpg/1200px-Statue_of_Liberty_7.jpg' },
              concepts: [
                { 
                id: 'statue',
                  name: 'statue',
                  value: 1,
                  app_id: 'fb70b904750c4891aecddf82082181c2' 
                },
                { 
                  id: 'bridge',
                  name: 'bridge',
                  value: 0,
                  app_id: 'fb70b904750c4891aecddf82082181c2' 
                }
              ],
              metadata: {},
              geo: { 
                geo_point: { 
                  longitude: -74.0445, 
                  latitude: 40.689247 
                }
              }
          },
          created_at: '2017-09-05T17:52:38.616686Z',
          modified_at: '2017-09-05T17:52:39.029363Z',
          status: [Object] 
        } 
      } 
    ],
    created_at: '2017-09-05T17:52:38.616686Z',
    modified_at: '2017-09-05T17:52:39.029363Z',
    status: [Object] 
  } 
}]

The search result will give us an array of matches. Each object within the array is one of our inputs that matches the search criteria. This also includes a score that will indicate how much of a match this the result is to our query. Whenever you want to access any of the data related to a specific object you would need to access its input.data key. You can see what the custom concepts, geo location data, or custom metadata it has attached to it. You can also find the input’s original url or bytes in this object.

Conclusion

The Search API is a powerful tool and searching using geo location is just one of many useful features for developers. You can do much more with the Search API like:

If you have any questions, concerns, or even friendly notes feel free to reach out to us over at hackers@clarifai.com or comment below!


How to demo your localhost app using ngrok

Ngrok is a reverse proxy that allows you to expose your locally running web service to other developers. It’s a timesaver for developers when it comes down to showcasing a project. Follow along and learn how you can put this into your toolbelt of applications!

Howdy everyone! Have you ever wanted to showcase an amazing project that you built at a hackathon; or show a client the progress of the app you’re building without deploying it to some server? A reverse proxy tool like ngrok can solve this. Let’s see how it works.

The Sample App

We’ll need to start with a server running locally. If you don’t have a web app to test you can use a quick Sinatra “Hello, World”:

# app.rb
require 'sinatra'

get '/' do
  'Hello World!'
end

We need to make sure to have the Sinatra gem. Install the gem through Terminal using:

$ gem install sinatra

To run our app, go into Terminal and write:

$ ruby app.rb

And BOOOM!! 3-line Hello World app.

Ngrok

Ngrok is a tool made by Alan Shreve, who built it to be as accessible as possible by breaking it down into 3 painless steps:

  1. Download ngrok for your OS
  2. Unzip it
  3. Launch it!

Ta-dah! You’re a natural!

Here we will launch ngrok through both macOS/Linux and Windows. Let’s move the executable from my Downloads to our Home folder (marked as the tilde or ~) for our macOS/Linux friends. Open up a new terminal and write:

$ mv /your/path/to/ngrok ~

For Windows we are moving the ngrok.exe to my particular User directory. Mine is called Administrator but yours could be your name (i.e. “Prince Wilson”).

$ move \your\path\to\ngrok \Users\<PC_Name>

We are going to execute ngrok straight from our current directory. In macOS you prepend a script with a ‘./’ to make it execute and in Windows you would use ‘.\’. Otherwise your Terminal will look for ngrok in your PATH. ngrok requires at least two fields: 1) the protocol (i.e. HTTP) and 2) the port we want to expose. If you are running your own app pop in the port it’s using otherwise Sinatra defaults to port 4567.

$ ./ngrok http 4567

$ .\ngrok http 4567

It fires up with a bunch of outputs but we only care about the forwarding url right now. Mine is  “http://da0dcb04.ngrok.io” or “http://f4fcf6f0.ngrok.io”. Notice that it points to where our sample server is, meaning anyone who hits our link will be forwarded to our sample app!

Conclusion

And with that, you have learned the basics of ngrok. But, there is so much more to learn! Try these to improve your experience:

  • To inspect the traffic coming through and replay requests check out http://localhost:4040. Learn more about that in the ngrok documentation.
  • Sign up for an ngrok account if you want to use your own Basic Authentication. What’s there to lose?
  • There are a ton of neato burrito things you can also get out of supporting the project so  pony up if you have the funds to help the software grow.

If you have any questions or want to share your experience with ngrok feel free to send me a note over at hackers@clarifai.com; I’d love to hear from you!


Train a custom visual recognition model using Clarifai's Python client

Last week, our v2 API graduated from developer preview to general availability. Don’t they grow up so fast?! As such, here is a useful tutorial on how to use one of Clarifai v2 API’s most popular features – Custom Training. Custom Training gives you the ability to teach new concepts for our technology to recognize using only a handful of examples and a couple lines of code!

If you’ve seen our new v2 API documentation, you’ve probably noticed that things are a bit different than they were in v1 – new SDKs, a cool new visual tool, and new projects waiting for you to build them using Custom Training! In one of our previous blog posts, we talk about how to use the Preview UI for Custom Training. While that was awesome (like really awesome), we’re gonna crank it up a gear and show you how to do some of the same things using _PROGRAMMING_.

It isn’t as flashy as pulling up our sleek, chic UI, but it is just as rad. You can go to your friends and show them your mad teaching-a-computer-how-to-see skills!


# An API Client

First thing’s first, we are gonna need to find a way to interface with the API with our code. We have written a few different clients for you to use in several different languages. For this tutorial, we are gonna stick with the Python client. Who doesn’t like programming languages that came out of Monty Python?

NOTE: Be sure to have installed a Python that is at least 2.7 or higher. If you aren’t sure what which one you might have, go into your terminal and check out what `python –version` gets out for you. For instance this is what happens on my system:

$ python --version
Python 2.7.10
# Or if you are using Python3
# Python 3.5.1

Also to install the Clarifai API Client, we are gonna make sure we have pip, the Python Package manager:

$ pip --version
pip 9.0.1 from /path/to/Python/2.7/site-packages (python 2.7)
# or if you are using Python3
# pip 9.0.1 from /path/to/python3.5/site-packages (python 3.5)

With those two out of the way, we can now finally get to installing the Python Client!

$ pip install clarifai
# NOTE: If this doesn’t work, consider having a sudo in front of the command to tell your computer who is boss.

If that went off without a hitch, we need to do one more thing before we can write our Custom Training. We are going to add our Client ID and Client Secret into our app. Every time you want to hit Clarifai, you want to be sure to have these two at least! No matter what language you are using, this will be the only way you can get access. So head over to the Developer website, and create yourself a new application:

image03

Once that is done, head back to terminal and type in:

$ clarifai config

When you hit enter, it will prompt you to put in your Client ID and Client Secret, respectively. That way, when we make calls, this doesn’t have to be saved in your project but on your file system so long as you are using that particular Application:

CLARIFAI_APP_ID: []: ************************************YQEd
CLARIFAI_APP_SECRET: []: ************************************gCqT

Congratulations! If you made it this far, you have successfully added the Clarifai Python client to your system. Go you! Now, with that out of the way, we are gonna write some code!


# Add Images with Concepts

With Custom Training, you can teach Clarifai to recognize whatever you want in an image! So for me, I am gonna teach Clarifai about corgis (because corgis are the best). For this part, you can find 10-20 images that you want to train your model with. This will be known as the “training set”, a set of images you use to train with that will have positive and negative examples of whatever concept you are trying to get Clarifai to recognize.

So, I gathered 15 images of corgis shot from different angles (including the all-important “corgi butt” pose) with various background objects to make sure Clarifai can recognize what I want it to. Though all my images are JPGs, Clarifai supports JPG, PNG, BMP, and TIFF files! Depending on what you want to train for, it can be better to have a more realistic photo of what you want Clarifai to be looking for and sometimes it is okay to want just the object itself in front of a solid background.

In my project folder, I made a folder to house all of the images I collect for a concept. So all my images in my `corgi/` directory will house images that will have ‘corgi’ as a concept.

image01

image04

 

To add an image object with the concept of corgi, we are gonna write:

from clarifai.rest import ClarifaiApp
from clarifai.rest import Image as ClImage

# Our app
app = ClarifaiApp()

# Adding our image onto our app
app.inputs.create_image_from_filename('./corgi/corgi-1.jpg', concepts=['corgi'])

When we run this (`python .py`), we can see our input is added onto our app on the Preview UI:

image02

To utilize the power of programming, we are gonna automate the process. Let’s make a function that will import all of our images inside a directory with the same concept. In this case, we will have all of the photos under `corgi/` with our ‘corgi’ concept!

# Packages for reading local files
import os
from glob import glob

def create_image_set(img_path, concepts):
    images = []
    for file_path in glob(os.path.join(img_path, '*.jpg')):
        print(file_path)
        img = ClImage(filename=file_path, concepts=concepts)
        images.append(img)

    return images

This will allow us to easily take in any sort of images, no matter where, and also allow for us to set some images to have concepts! This is just some of my code and you are more than welcome to improve it to fit your needs.

NOTE: For this function, we are making the assumption that our number of images per batch is relatively small. If you are sending large sets of images at one time, with numbers around 100,000+ images per function call, you might want to consider abstracting this function to asynchronously handle sending out the request.

So now our code will look more like this:

from clarifai.rest import ClarifaiApp
from clarifai.rest import Image as ClImage
from glob import glob
import os

def main():
    app = ClarifaiApp()
    image_set = create_image_set('corgi/', ['corgi'])
    app.inputs.bulk_create_images(image_set)

def create_image_set(img_path, concepts):
    images = []
    for file_path in glob(os.path.join(img_path, '*.jpg')):
        img = ClImage(filename=file_path, concepts=concepts)
        images.append(img)

    return images

if __name__ == '__main__':
    main()

Notice we call our create_image_set in the main method and set our image_set equal to the returned results. Then we send this into `app.inputs.bulk_create_images`. Now you should be seeing all of the images you have added when you run the script!

image00

Now we have added some images with our corgi concept! Let’s try to add this onto a model and train!


# Create a Model and Train!

Since we have already added our inputs, all we need to do now is create a model and train it! To avoid adding duplicate inputs, we are going to comment out the image stuff and change our main function to only:

def main():
    app = ClarifaiApp()
    # image_set = create_image_set('corgi/', ['corgi'])
    # app.inputs.bulk_create_images(image_set)

    model = app.models.create(model_id="puppy", concepts=['corgi'])
    model.train()

Here we created our model, with a model_id of “puppy” and concepts that are on the model align with concepts that we have on our model. From there, we make sure to call train. Voila! We have trained our model to recognize the concept of corgis!

Now let’s say we wanted to use our model and see whether or not it works, we can change our main to be:

def main():
    app = ClarifaiApp()
    # image_set = create_image_set('corgi/', ['corgi'])
    # app.inputs.bulk_create_images(image_set)
    # model = app.models.create(model_id="puppy", concepts=['corgi'])
    # model.train()
    model = app.models.get('puppy')

    url = raw_input("URL of image: ")
    # if on Python3, change the above line to
    # url = input("URL of image: ")

    print model.predict_by_url(url)['outputs'][0]['data']['concepts']
    # if on Python3, change the above line to
    # print(model.predict_by_url(url)['outputs'][0]['data']['concepts'])

We will ask our user to give us a URL and then we will print against our puppy model! This will give us the output of all of our concepts on our model:

$ python app.py
URL of image: https://s-media-cache-ak0.pinimg.com/236x/37/5d/48/375d488c51f51c72a01e0e88420becca.jpg
[{u'app_id': u'b066d570bcae40b3b0d5d50176d09cef', u'id': u'corgi', u'value': 0.9865137, u'name': u'corgi'}]

That’s it for this tutorial! There are a number more functions and ways of doing things in Custom Training. If you ever need any help, be sure to reach out to us at support@clarifai.com. We are more than happy to help!

Before I send you on your way, here are some simple tips and tricks to remember about Custom Training:

  1. Be sure to have your inputs (images) on your application, and tell them what concepts (or not-concepts) they are associated with! 
  2. Make sure you have a model (a place to put a group of concepts) and concepts! Just having images on your application isn’t enough. 
  3. If you are unsure how to access the API using any of the clients, be sure to check out the repo’s documentation. For Python, you can see all the available functions here.

Go out my compatriots of the world. Teach computers how to see the world from your perspective!