In this blog I’m going to give a high-level overview on how we created a proof of concept that used the Azure Form Recognizer service to automate the processing of purchase orders into a resource planning system.

Forms Recognizer. What is it?

Different from other document cracking services, Forms Recognizer instead targets key value pairs and table data typically found in forms such as invoices, receipts, statements etc. The service uses machine learning technology to identify the structured data, which is then output in a format that matches the original document. Microsoft shows a good example of a typical form on their website here and below you can see how Form Recognizer has extracted the key-value pairs and kept the relationships intact.

How can it be used?

A lot of forms are still manually processed by business every day. These forms are often entered into a structured system with attached metadata or associations to allow the business to identify its purpose and the content within. Using AI to gain insights into these locked documents can eliminate the headache of this manual process as well as push new automation possibilities.

Automate form processing

Forms Recognizer scenario – Purchase Order Processing

In this proof of concept our fictional company receives a large amount of purchase orders daily from a major client. Currently data from these orders must be entered manually into a resource planning system to allocate the work to be completed. Let’s see how we can leverage AI to automate this process.

Train

To get training we only need five filled in forms or an empty form plus two filled in forms. Including an empty form can improve accuracy so is recommended. These are submitted to the Forms Recognizer service so it can learn the structure and extract the key data. Here’s the purchase order layout we need the service to learn.

Purchase Order Training

Submitting five of these forms with an additional empty form should be enough. I’ll go ahead and train the model using python as explained in this Microsoft guide. After training it’s worth noting that subtotal, vat and gross total were all be identified as key value pairs with the remaining classed as table data. This seems dependent on the table layout and how clearly it’s defined.

Process

Now we have the model trained we need a way to process the incoming email and automate the extraction. In this example I’ll use a Logic App, but this could easily be achieved using Power Automate or Azure Functions. You can see this process at a high level below:

Logic App Flow

Testing

When sending a purchase order to my fictional company I can see that the Logic App processes the attachment and the Forms Recognizer service responds with the key value pairs and table data. Here’s the snippets of the Gross total being returned.

My tests showed there was no problem repeatedly identifying simple table and key value pair data, however understanding how the service identifies tables was essential. For example, having multiple lines of text without line separation seemed to cause confusion and may require your forms to be amended to provide clearer separation. At the time of testing complex tables (nested tables, merged headers or cells, and so on) are not supported so I imagine this will improve in the future.

Finally, for a more user-friendly experience, we can visualise the purchase order database using Power-BI. This gives us a way to see the extracted information as it comes in, which in this case is the purchase order highlighted with the gross total of 600.00.

Power BI visualisation

Conclusion

Using only a few resources and almost no code I was able to create an automated system to process purchase orders received by email. Once cracked, the information from these documents can then feed other systems, removing the labour-intensive data entry previously required.

References:

What next?

Please do use the form below to get in touch with any questions or queries.

We host regular events so check our schedule of current seminars, webinars and events.  We also post regular blogs on the latest updates and expert advice on Microsoft 365, Cloud and Hybrid IT, User Adoption and the Power Platform, so please do follow us.

Other Resources