How energy per output token is measured

Metrics

Benchmark 1.5 keeps the five core metrics of Benchmark 1.0 and adds three lifecycle metrics (Section 2.2).

Steady-state energy per output token (J/token)

GPU energy within the steady-state window divided by the output tokens in the same window. Numerator and denominator come from the same window. Suited to services that run continuously.

Full-lifecycle energy per output token (J/token)

GPU energy of the startup, formal request and resource release stages, divided by the output tokens of the formal request stage. Suited to cold starts, short tasks and elastic services.

Startup energy

GPU energy from the start of the inference service to the start of formal requests: model loading, service initialization and warm-up.

Resource release energy

GPU energy from the end of formal requests until the inference process exits, GPU memory is released and power returns to baseline.

Output throughput (tok/s)

Total output tokens of the formal request stage divided by the duration of that stage.

Average GPU power (W)

GPU energy in the statistical window divided by the window duration, reported as the combined power of all GPUs.

Time to first token (TTFT) and time per output token (TPOT)

TTFT is the time from sending a request to receiving the first output token, reported as P50. TPOT is generation time after the first token divided by the output tokens of that stage.

Steady-state and full-lifecycle J/token share a unit but have different measurement boundaries. The paper states they must not be mixed or subtracted from each other.

Measurement boundary

All energy values are GPU energy, summed across the 8 GPUs. CPU and memory readings that Zeus can provide on some hardware are not used in any metric. CPU, memory, storage, network, cooling and power supply losses are excluded, so the values cannot be equated with wall-plug or data center energy.

Benchmark 1.0 measured only the formal request window. Benchmark 1.5 extends the boundary to the full service lifecycle (Figure 3-1).

Power measurement and sampling

• Before each test, 20 seconds of idle power are collected to confirm the initial GPU state.
• Startup and release: each GPU's instantaneous board power is sampled at roughly 1-second intervals. Power at stage boundaries is linearly interpolated, each GPU's energy is integrated with the composite trapezoidal rule, and the GPUs are summed.
• Formal requests and steady state: GPU energy is measured by Zeus 0.13.1 through NVML.
• The idle baseline is not subtracted. Pre- and post-test baselines are used only to confirm idle state and resource recovery.
• Release is complete when each GPU's power falls back within ±5% of its pre-test baseline and holds for at least 30 seconds. If that does not happen within the maximum wait time, the run is recorded as timed out and the release interval is flagged as not closed. Runs that do not recover resources within the specified time are excluded from the formal results.
• Prometheus collects queue, KV cache and scheduling state from the engine's /metrics endpoint on a side path. It is used only to explain run state, not to set steady-state boundaries or compute energy.

Steady-state determination

Candidate windows of 100 requests are formed in request completion order, sliding forward one request at a time. For each window, the mean of per-request J/token is the energy level and the standard deviation divided by the mean is the coefficient of variation (CV). When consecutive non-overlapping windows all satisfy CV < 0.03, the system is deemed to be in steady state, and the steady-state window runs from the start of the first qualifying window to the end of the formal requests.

If no window meets the threshold by the end of the formal requests, the run is recorded as not reaching steady state. Its steady-state J/token is excluded from cross-model comparison and flagged with the reason.

Energy for concurrent requests is attributed by each request's generation interval. GPU energy in overlapping intervals is split in proportion to each request's output tokens in that interval.

What is counted and excluded

Counted in startup energy

Service startup, weight loading from a local snapshot, JIT and kernel compilation, service readiness and the 8 warm-up requests.

Excluded

Network downloads of weights. Warm-up requests in formal performance metrics. Failed requests in the token count. Idle baseline subtraction. All non-GPU energy.

Token denominator

Completion tokens returned by the API for requests that completed successfully in the formal request stage. Models and engines return reasoning content differently, mixed into ordinary output in some runs and in a separate reasoning field in others, so the count cannot uniformly be read as visible output tokens only. The paper does not split the two. The thinking setting of each run is listed in Table B-1.

Cache conditions

Each test is a cold start of the inference service. Weights are a fixed local snapshot prepared in advance; each test gets its own engine run directory and compile cache directory; the operating system file page cache is left in its existing state.

Test point, hardware and software

All tests run on one local server at a unified test point. The test point is used to compare deployments under the same resources and does not represent each model's best configuration (Tables 3-1 and 3-3).

SGLang runs use the configuration functionally equivalent to vLLM. Model-specific differences in engine version, image and quantized artifact are kept and recorded with the results (Table 3-4).

Scenarios and workloads

Three public datasets of real interactions or code, rather than synthetic templates (Table 3-2).

Each model runs only the tasks it supports. Unsupported tasks are marked not applicable and are not counted as failures.

Test procedure

1. Preparation. Check model ID, revision, precision, snapshot, engine, image, parallel configuration and request set; confirm GPUs, storage, leftover processes and containers are in a normal state.
2. Start lifecycle monitoring. Collect 20 seconds of idle power, then sample board power through startup.
3. Start and verify the service. Start the engine with the frozen configuration; once ready, re-check model identity, revision, engine, context length and parallel configuration.
4. Warm-up. Run 8 informal requests to trigger first-request handling, compilation and tokenizer processing. Their energy is counted in startup, not in formal metrics.
5. Formal requests. Submit all requests without rate or concurrency limits. Record success, input and output tokens, TTFT, TPOT, inter-token latency and end-to-end latency per request.
6. Shutdown and release. Stop the service and keep sampling board power and GPU memory until release completes or times out.
7. Verification and archiving. Save run configuration, lifecycle data, GPU telemetry, request records and logs. Only runs that pass all checks enter the formal results.

Quality and comparability thresholds

• The paper does not evaluate answer accuracy or task quality, and sets no output quality threshold. Capability scores in Table 2-1 are taken from model cards, technical reports and LMArena, not re-tested; for some quantized artifacts they are the base model's published scores.
• Steady state requires CV < 0.03 over windows of 100 consecutive requests.
• vLLM and SGLang results for the same model and task are compared only when their total output tokens differ by no more than 20%, computed as the absolute difference divided by the larger total. 11 of 14 pairs meet this.
• Models are compared within size groups by total parameters: no more than 50B, above 50B and below 200B, and at least 200B. Grouping does not imply equal computation.

Repetition and statistics

Each configuration completed one formal run. Means, standard deviations and statistical significance from three repetitions are not yet available. The paper summarises results with medians per scenario, the ratio of median full-lifecycle to median steady-state J/token, and Spearman rank correlation between the model orders under the two bases.

For a service that handles N requests after one startup, the paper gives this estimate of lifecycle GPU energy, where e_ss is the average steady-state GPU energy per request:

The fixed overhead per request is (E_start + E_release) / N. The estimate applies only when load and configuration match the measurement conditions and excludes idle gaps between requests; measured values take precedence. Converting GPU J/token to electricity cost needs a whole-server to GPU energy ratio and PUE, neither of which was measured, so the paper draws no electricity cost conclusions (Section 2.3).

Verification and reproducibility

• Each formal run records model, task, engine and steady-state result, including the steady-state window's start and end times and request numbers, GPU energy and output tokens in the window, and the computed J/token.
• Total formal-stage output tokens and whether GPU power returned to the idle baseline range during release are archived.
• Runs that fail verification keep their original records. Re-tests create separate records without overwriting existing data.
• Model repository IDs, pinned revisions and commit dates are listed in Appendix A; per-run configuration and stage energy in Appendix B; engine versions in Table B-3.
• Container image digests are recorded with the formal results but are not listed in the paper.

Limitations stated in the paper