Track retries and model invocations in AI workloads

1. Confirm the execution surface and review window.
   - Choose one explicitly scoped account or subscription, service, region or location, and time window.
   - Read existing application, SDK, gateway, provider-metric, evaluation, and billing data for that scope.
   - Keep this review read-only. Do not change retry settings, deploy code, or start an evaluation.

2. Map fields that already exist, then record gaps for a later implementation plan.
   - Look for request and trace identifiers, attempt number, retry reason or mode, maximum attempts, loop iteration, and stop reason.
   - Look for model or deployment ID, API operation, status, error code, latency, and available input, output, cached-input, or total-token fields. Record each field's documented unit or definition when available.
   - Look for tool name or type and tool-call count, plus evaluation and job or run identifiers, dataset version, generator or judge model, metric name, and sample count.
   - Look for application, feature, environment, team, tenant or customer pseudonym, and business transaction identifiers. Mark absent fields as gaps. Do not treat this list as provider-guaranteed columns.

3. Read documented provider metrics and evaluation metadata separately.
   - For Amazon Bedrock agents, review CloudWatch namespace AWS/Bedrock/Agents: InvocationCount, ModelInvocationCount, InputTokenCount, OutputTokenCount, latency, throttle, and client- and server-error metrics. Review the documented Operation, ModelId, and AgentAliasArn dimensions.
   - For Bedrock LLM-as-a-judge evaluations, read the job scope, generator and evaluator models, prompt dataset bucket, and output bucket from the job configuration.
   - For Azure Foundry or Azure OpenAI workloads, review existing gateway telemetry when present. Microsoft describes gateway capture of client identity or custom business-unit, tenant, or application identifiers with model and token data.

4. Compare and classify the results.
   - Compare consistently scoped service telemetry with the billing data available for the same account or subscription, service, region or location, and time window. Verify the identifiers, time basis, units, and granularity in your enabled data before comparing.
   - Mark unmatched records for investigation rather than treating the comparison as guaranteed reconciliation.
   - Label retries, loops, tool calls, and evaluations as telemetry event types. Count another model-cost event only when the available telemetry shows another model invocation or another metered operation.