Azure OpenAI Emit Token Metric Policy

This policy addresses the need for detailed monitoring and analysis of token usage in applications using Azure OpenAI models.

By providing comprehensive metrics, it helps organizations:

• Optimize Resource Allocation: Understand and manage token consumption effectively.
• Improve Decision-Making: Gain insights into usage patterns to make informed decisions about scaling and resource management.
• Enhance Performance Monitoring: Track and analyze token usage to identify and address potential issues proactively

How to use Emit Token Metric Policy

To use this policy, you need to add it to the inbound processing pipeline of the API operation. Here’s how you encode it in XML:

XMLCopy

<azure-openai-emit-token-metric
        namespace="metric namespace" >      
        <dimension name="dimension name" value="dimension value" />
        ...additional dimensions...
</azure-openai-emit-token-metric>

Here’s an example using several dimensions:

XMLCopy

<policies>
  <inbound>
      <azure-openai-emit-token-metric
            namespace="AzureOpenAI">   
            <dimension name="User ID" />
            <dimension name="Client IP" value="@(context.Request.IpAddress)" />
            <dimension name="API ID" />
        </azure-openai-emit-token-metric> 
  </inbound>
  <outbound>
  </outbound>
</policies>

In the preceding example:

• the policy is configured to emit token metrics to the AzureOpenAI namespace with dimensions for User ID, Client IP, and API ID.
• The value of the Client IP dimension is set to the IP address of the client making the request.

Imagine now that you can show these metrics in a dashboard and you can monitor the usage of your API in real time. For example, you can see how many tokens are being used by a specific user or how many tokens are being used by a specific API. This powerful feature that can help you optimize your resources and make informed decisions about scaling and resource management.