As I used to be writing these different publish and coding the brokers, I used to be having some doubts about whether or not or not they brokers have been truly working or not. I did add some Terminal prompting however there was nonetheless some doubt. I wanted to find some further views on easy methods to monitor and measure my AI Brokers inference and output.
The answer is to configure Utility Insights. I created this quick YouTube video: Add Utility Insights To Azure App Service, about Utility Insights, however this isn’t required for fundamental logging, solely if you’d like extra custom-made logging.
In Determine 1 you’ll be able to see the potential to attach Utility Insights to an AI Agent is straightforward and in addition intuitive.
Determine 1, monitoring and measuring your AI Brokers, configuration
It’s essential first manually provision an Utility Insights workspace earlier than you may make the configuration. One nice development I seen is that in the course of the connection between the AI Agent and the AI workspace the platform configured the RBAC permission. This was a typical stumbling block, however now it not solely configures the permission for you, it chooses the permission to grant, there are plenty of them, so this protects numerous time.
So after you make the configuration, execute your hub-and-spoke, fan-in/fan-out, or some other multi-agent workflow answer and you’ll see the small print of their execution. Determine 2 illustrates such.
Determine 2, monitoring and measuring your AI Brokers, dashboard
This reveals some good info, price is a vital one, good to see it isn’t too costly, I’m simply testing right here so, anticipated. Enter tokens, output tokens, and period are additionally useful. If you choose the Hint ID you’ll be able to see the small print which could be very good as properly.
Determine 3, monitoring and measuring your AI Brokers, particulars (enter immediate and inferred AI output)
There are totally different views, however one thing I actually like is that I can take a look at the Enter and Output which was despatched and inferred. That is very useful to get some insights into what customers are experiencing. Additionally, word that though it is a 1-by-1 view, Utility Insights is backed by Log Analytics which is a full information retailer the place bigger queries may be executed, analytics run to realize nice insights into giant scale manufacturing multi-agent AI workflows.
Determine 4, monitoring and measuring your AI Brokers, particulars (suggestions annotations)
Lastly, there’s an Annotation in case you occur to be doing a little guide, small scale evaluation. Which once more may be retrieved from the Log Analytics information retailer later. I might anticipate that there’s an interface which might enable for this to be introduced to shoppers of this multi-agent workflow to present suggestions. Utilizing Log Analytics once more would give nice insights.

