I recently experienced the excitement of using data captured from AI models from AI Builder in a Power Automate cloud flow. To get this kind of functionality in the past I used Power Automate Desktop, which worked, but it felt a little clunky to wire together. Though it’s probably been available for quite a while, I just discovered that by using the capability baked into AI models that are available in AI Builder, you can bypass Power Automate Desktop with a very smooth tool to harvest data from a document formatted as a PDF, PNG, or JPEG. Very cool indeed!
Finding AI Builder
AI Builder is available from the Power Apps maker portal (make.powerapps.com). Once you get to the Power Apps home page, look down on the left-hand navigation panel, select the More option, and you’ll see an option to go to the AI hub. AI hub is the landing page for AI builder. There’s so much you can tackle here, but I want to narrow it down to using the AI models portion, since I want to demonstrate how you can use an existing model to pull data into Power Automate.
AI models provide a jumpstart
Once you’ve selected AI models, AI Builder presents you with options to choose a model. You can filter the results, but you can see all of them by simply selecting All. As of this writing, there are less than twenty, but I bet we’ll see more of these in the future. Notice how some have the label “PREBUILT” where others have “CUSTOM” categories. Additionally, some have the warning “PREVIEW” next to them. At this point I think it’s safe to try all of them, but I’d shy away from using anything with “preview” in a business solution.
Side note: If you’re doing this for production, you always want to be careful about automatically picking up data, but one of the cool features of these models is that they let you know a “confidence” factor. This factor lets you know the AI’s confidence that it is extracting correct information from a document.
In this example, I’m using Extract information from invoices, which is one of the most popular models.
When you select the model, AI Builder presents a preview of it, allowing you to view several documents that are already loaded into the model, along with a scrollable panel showing the Extracted information. There are links below to either Use prebuilt model or Create custom model, along with the possibility to View documentation (which opens another browser tab directly to the documentation for the selected model, very helpful). I was even able to use the Upload new icon to evaluate a random test invoice, and the prebuilt model picked up the data:
The arrows to the left and right of the popup panel give you a preview of the other models, which is an excellent way to make sure you grab the one that best suits your purpose. Switching back to the screen above still had my uploaded document in the preview window, too.
By selecting the Use prebuilt model you’ll notice that you have options to use the model in an app or flow. In this case I’m letting AI Builder know that I want to use it in a flow.
After a moment, a new browser tab opens in Power Automate and asks to verify the connections. After doing that and selecting Continue, Power Automate spits out the following flow:
The Manual trigger in the flow is simply asking for an invoice file in PDF, PNG or JPEG format. Notice how the steps of this flow have descriptive notes below the title of the selected steps letting you know about the flow and what it’s doing. Very nice! Right away, you get the idea (even suggested by Power Automate) that you can change the flow to suit your needs.
The key step in this flow with respect to AI Builder is the Extract information from invoices step. This step attempts to pull the information from your document, according to the selected model.
The Create HTML table step puts basic information into an HTML table. This step is modifiable so you can put whatever data you want into your table.
The next step gets the user profile for the person activating the flow, and finally, the flow sends an email to the person activating the flow. The email also includes the original invoice.
One cool thing about the Send an email step is worth a look. In the body of the email, Power Automate feeds you the Confidence score for each of the values! What?!? That’s right, the AI Builder step provides this information as step metadata! With this confidence score, you could use some conditions in your flow to route that item past an actual person for a second look.
Another cool thing about the Send an email step is that it attaches the document to the email. Power Automate uses the updated Power Platform format for including the document “Name” and the “ContentBytes”. The weird piece is that Power Automate writes a formula for the “Name” string, while the “ContentBytes” come directly from the Dynamic content. That’s a little clunky, but it gets the job done.
Wow, that’s a really great start to my flow! Now I can modify my flow to update data.
Modifying the flow
With Copilot’s help, I create a Dataverse table with most of the information that I’m looking for. I compare it to the model (open in another browser tab) and make some manual adjustments to it, so that I can match the columns from the model. Again, you can build a custom model to perfectly match your invoices, I just wanted to see the capabilities of the already developed model. The nice thing about using Copilot is that you get some sample data, too, so I absolutely recommend having Copilot lend you a hand.
With my table called “Invoice Information” I can now go in and change my flow to write the data to my new table. I use the Dataverse Add a new row step in my flow, and because I’ve named my fields like the model, I can easily plug in the dynamic content into the fields.
Running the flow has positive results! I can feed it the demo invoice and it picks up data that the model looks for. My “fake” invoice didn’t provide all of the information that my model is able to extract, but it did pick up most of the relevant data and add it to my table.
Going back to look at my new Dataverse table, I find that the data is there, and I didn’t have to look at the invoice to input any of it!
Summary
AI Builder + Power Automate = Amazing! I’m sure you can grasp the potential of this type of AI integration. Being able to have AI read numbers/text from a document for data entry is not a pipe dream – it is available to you right now in the Microsoft Power Platform. You must give it a try!
About the Author
References
Cooley, P., (2024), ‘Using AI Builder with Power Automate’, available at: https://www.powerapps911.com/post/using-ai-builder-with-power-automate, [accessed 27th March 2024].
