A common struggle: You have searched way too many combinations of words to find the image or video you need and it is still not popping up. Keeping your content organized and searchable, especially as the number of assets grows over time, can require a lot of manual work. What if you could automatically classify any image or video you upload to OneDrive for Business and move it to the right folder with zero manual effort?
This article puts together a quick guide to build an Azure Logic App flow that integrates OneDrive for Business with the image recognition capability of Cognitive Services. Logic App makes it easy to integrate multiple APIs, and Cognitive Services quickly analyze images and return tags to be used for file classification.
Cognitive Services
Microsoft Cognitive Services are a set of APIs, SDKs and services available to developers to make their applications more intelligent, engaging and discoverable. Microsoft Cognitive Services expands on Microsoft’s evolving portfolio of machine learning APIs and enables developers to easily add intelligent features, such as emotion and video detection, facial, speech and vision recognition, and speech and language understanding, into their applications.
In this article, I’ll focus on the Vision API, a collection of image-processing algorithms to smartly identify, caption and moderate your pictures. The Vision API includes the following capabilities:
- Image classification
- Scene and activity recognition in images
- Object recognition in images
- Celebrity and landmark recognition in images
- Optical character recognition (OCR) in images
- Handwriting recognition
Image classification is what we need for analyzing images. This feature returns information about visual content found in an image, which can then be used to automatically classify images. You can try the Vision API with a live demo in the browser that requires no installation of software and no coding. From there, you can upload an image and have the image analyzer identify multiple features of the image.
Feature Description
The full set of features detected by the image analyze API is listed below. The “Categories” feature, containing properties “name” and “score” is the one you may want to look at particularly, because the Logic App flow described later in the article will use this feature for automatically moving images into folders named after the identified category. The score value is of importance as well, as it determines the accuracy of image detection in a range between 0 and 1, with 1 being the most accurate value.
| FEATURE NAME: | VALUE |
| Description | { “tags”: [ “yellow”, “car”, “road”, “transport”, “black”, “driving”, “street”, “parked”, “city”, “riding” ], “captions”: [ { “text”: “a yellow and black car”, “confidence”: 0.90520674 } ] } |
| Tags | [ { “name”: “yellow”, “confidence”: 0.996363 }, { “name”: “car”, “confidence”: 0.9835354 }, { “name”: “transport”, “confidence”: 0.8202089 } ] |
| Image format | “Jpeg” |
| Image dimensions | 430 x 700 |
| Clip art type | 2 |
| Line drawing type | 0 |
| Black and white | false |
| Adult content | false |
| Adult score | 0.008509474 |
| Racy | false |
| Racy score | 0.009332338 |
| Categories | [ { “name”: “trans_car”, “score”: 0.99609375 } ] |
| Faces | [] |
| Dominant color background | “White” |
| Dominant color foreground | “Black” |
| Accent Color | #B29019 |
OneDrive for Business
Let me describe with an example how the process of image classification looks like from an end-user perspective. Assume that we have a few images representing cars, flowers and people each.
Classification
When uploading any of these pictures in a folder in OneDrive for Business, a background flow runs that invokes the Vision API, analyzes the image and returns its category and confidence score. We can set a threshold here stating that if the score is higher of, say, 0.8, that is the categorization of the image is a level of at least 80% certainty, that image can be automatically moved to a folder named after the identified category itself. For our sample pictures, we end up having a folder structure that looks like the following:
File > Pictures
Intentionally, I’ll challenge the image analysis API with an image that is confusing: Is that a car or a flower decoration?
Car or flower decoration?
A human person would easily spot the car. The AI cannot (yet), returning a very low confidence score in the wrong category.
plant_=0.2421875
The Logic App flow won’t move the image to any folder because of the low level of certainty on the identified category. It will rather output a log file containing category name and score.
Output a log file
Azure Logic App
Logic Apps helps you build, schedule, and automate processes as workflows so you can integrate apps, data, systems, and services across enterprises or organizations. Logic Apps simplifies how you design and create scalable solutions for app integration, data integration, system integration, enterprise application integration (EAI), and business-to-business (B2B) communication, whether in the cloud, on premises, or both. To build integration solutions with logic apps, you can choose from a growing gallery of more than 200 built-in connectors, such as SQL Database, Azure services, Office 365, Salesforce, Google, and others. These connectors provide triggers and actions for creating logic apps that securely access and process data in real time.
To start building a workflow in Logic App, sign into your Azure portal and search for the Logic App resource in the Azure Marketplace.
Azure Marketplace
Then create a new Logic App flow by specifying its name, assigning it to an Azure subscription, resource group and location (for billing purposes).
Create logic app
Now we can design the flow as follows:
- When a file is created in a specific folder in OneDrive for Business…
- Analyze the image with the Computer Vision API and obtain a category name and confidence score…
- If the score is greater than 0.8, and here you can decide your own threshold level…
- Move the image in a folder named after the category…
- Otherwise, create a log file containing category name and score.
Step by step, the design of the workflow requires to identify the event that triggers the beginning of the flow itself. After choosing the “OneDrive for Business” connector, select the “When a file is created” trigger and set the folder where images are being uploaded, and the frequency of check for new files in that folder (Logic App flows work as a poll mechanism that looks for changes at regular intervals).
When a file is created
After a file is created, as a second step we want to send its content to the Computer Vision API for analysis of the image. The Vision API will return the identified category and confidence score.
Computer Vision API
There are two ways of analyzing images: by providing its URL, or its content. In our case, as we are uploading a file to OneDrive for Business, we can analyze images by content.
Analyze Image
Once obtained the result of the analysis, we’d like to set a threshold on the confidence score. As some images may not fit perfectly into a category, it wouldn’t be possible to move them into a single folder. Let’s assume that we trust the correct identification of a category only if the score is greater than 0.8, representing an 80% of confidence.
Dynamic Content
If this condition is true, the flow will move the image into a folder named after the category. If the folder doesn’t exist, it will be created the first time.
Condition
Otherwise, if the identification of the image is not accurate enough to provide a score higher than 0.8, the flow won’t move the file, but rather it will create a log file and write the category name and score in it.
If false
That’s all. Run this flow to enable it on your OneDrive for Business folder and start enjoy automatic classification of images based on their content, as identified by the Cognitive Services Vision API.
About the Author:
I am a Microsoft Regional Director, Microsoft MVP and technology ambassador of beautiful software.
Reference
Tempesta, S. (2018). AI-powered Folders in OneDrive for Business. Available at: https://code.msdn.microsoft.com/AI-powered-Folders-in-2eb1666f [accessed 17th January 2019]
