How to Measure Quality, Response Time, and Utilization of an LLM Server?

A Dashboard Does Not Replace a Technical Test

A productive LLM service requires both technical and professional metrics. For quality, a fixed, versioned test set is created from real tasks, unanswerable questions, security cases, and expected sources. Depending on the task, Exact Match, F1, or classification accuracy are suitable; open responses additionally require a documented human evaluation framework, such as for factual accuracy, completeness, and citation fidelity. The lm-evaluation-harness supports reproducible tasks and various metrics but does not replace company-specific test cases.

For user experience, Time to First Token and end-to-end latency are recorded separately. The first metric describes when a visible response begins; the second, when it is complete. p50 shows the typical case, while p95 and p99 represent the slower edge cases. Averages alone can obscure queues and load spikes.

In operation, output and input tokens, requests per second, ongoing and waiting requests, error rate, GPU utilization, GPU memory, and KV cache usage are all included. vLLM provides a Prometheus endpoint at /metrics and documents, among other things, histograms for end-to-end latency and Time to First Token. Alarms are tied to agreed SLOs, not arbitrary standard values.

Each measurement run stores model ID and revision, quantization, prompt version, RAG data state, hardware, context and output lengths, and parallelism. This allows for associating a change. Before a rollout, thresholds are defined, followed by a limited Canary operation. A faster model will not be released if professional quality or authorization tests worsen. Similarly, a qualitatively good model is not production-ready if p95 latency and error rate fail to meet operational goals.