orchestrate-cost-and-ops-metricslisted

# Orchestrate: Cost and Operational Metrics **Direct evidence**: the multi-modal-review (June) challenge's evaluation criteria explicitly list *"operational metrics: model calls, token usage, image usage, cost estimates, runtime, and TPM/RPM considerations"* as mandatory analysis. This reflects a broader signal from HackerRank's stated philosophy — evaluating whether a candidate/agent-builder thinks like someone who has to actually operate the system, not just get it to work once. ## What to instrument, concretely - **Model call count**: total calls made across the full run, broken down by purpose (classification calls vs. retrieval calls vs. validation-retry calls) if your architecture has distinct stages. - **Token usage**: input and output tokens, ideally per-call-type, summed for the full dataset run. - **Image usage** (multi-modal challenge specifically): how many images were sent to a vision model, at what resolution/size, since this is often the dominant cost driver in multi-modal pipelines. - **Cost estimate**: token/image counts × the provider's published per-unit pricing, presented as a real dollar figure for the full run — not just "it uses tokens." - **Runtime**: wall-clock time for the full dataset, and whether that's dominated by model latency, retrieval, or something else. - **TPM/RPM considerations**: whether your call pattern would hit a provider's tokens-per-minute or requests-per-minute limit at production scale, and what you'd do about it (batching, bac