Benchmarking a quantum computer is not like benchmarking a classical processor. There is no single clock speed or core count that summarizes capability, and the number that vendors emphasize most—qubit count—tells you almost nothing about whether a device can run a useful circuit without errors . A 100-qubit processor with poor gate fidelity and limited connectivity may be less capable than a 50-qubit device with better coherence and all-to-all coupling.
This guide covers the practical work of benchmarking quantum devices: which metrics matter, how to run tests that survive vendor compilation, and the failure modes that make published comparisons misleading. It assumes you have access to one or more cloud-accessible quantum backends and need to make an informed judgment about which device suits a particular workload.
What Benchmarking Quantum Devices Actually Measures
Quantum benchmarking spans three layers, and confusing them is the root of most bad comparisons. Device-level metrics describe the physical qubits: coherence times (T1, T2), readout fidelity, and single- and two-qubit gate fidelities . These are necessary but not sufficient. Operation-level metrics describe how well the system executes gates and reads results under realistic conditions. Application-level metrics attempt to predict performance on a specific algorithm or workload class.
The Chinese national standard GB/T 47468-2026, which took effect in November 2026, organizes performance indicators into exactly this structure: device indicators (qubit count, connectivity, survival rate, coherence), basic control indicators (fidelity, operation duration, gate capacity, parallelism, calibration capability), and comprehensive performance indicators (maximum entangled qubits, algorithmic qubits, quantum volume, circuit execution efficiency, mirror benchmark fidelity, random circuit sampling fidelity, and energy consumption) . The standard’s inclusion of energy consumption is notable—it is often omitted from academic benchmarking but matters for operational planning.
The practical implication is that a single number cannot characterize a quantum device. A benchmark suite should report at least one metric from each layer, with the test conditions (qubit selection, circuit depth, number of shots) stated explicitly.
Why Qubit Count Is a Poor Proxy for Capability
A processor may contain 100 physical qubits, but only a subset may be controllable, and a smaller subset still may be readable with acceptable fidelity. The Chinese standard distinguishes total qubit count from controllable qubit count and readable qubit count . A device that advertises 100 qubits but only reliably reads 60 of them has an effective capacity closer to 60.
Connectivity compounds this. A superconducting processor with square-lattice connectivity requires the compiler to insert swap gates to route operations between distant qubits, and each swap adds error . A trapped-ion device with all-to-all connectivity does not require routing, so the submitted circuit structure is preserved under compilation. For a circuit with many long-range interactions, the connectivity difference can outweigh a modest fidelity advantage.
The benchmark that captures this interaction is quantum volume, which measures the largest square circuit of arbitrary two-qubit gates that a device can execute with acceptable fidelity. But quantum volume has limits: it saturates on small devices and does not scale to utility-scale processors . It is a useful floor, not a ceiling.
The Compiler Problem: Why Your Benchmark May Not Measure What You Think
The most consequential failure mode in cross-platform quantum benchmarking is compiler-induced artifact. A recent study of three platforms accessed via Amazon Braket documented a case where a depth-based parity benchmark produced a null result on IQM Garnet—no measurable dependence on circuit depth—which would naively suggest the device had no multi-qubit coherence . A separate parity oscillation measurement on the same hardware showed clear sinusoidal oscillations with 85.8% visibility at 5 qubits, the second-best in the study.
The discrepancy was resolved by retrieving the compiled circuits from the platform metadata. The submitted circuits contained barrier instructions designed to create depth variation. The compiler removed the barriers and absorbed the phase information into native gate parameters, producing operationally equivalent circuits that differed only in angle values . The experimental variable was silently eliminated.
This is not a hardware flaw. It is a benchmark design flaw. The lesson is that any benchmark relying on structural circuit features—barriers, idle gates, specific gate decompositions—must verify that those features survive compilation. The audit workflow is straightforward: retrieve the compiled circuit from the platform (if available), compare gate counts and critical path length to the submitted circuit, and confirm that the variable you intended to manipulate was not optimized away . If the platform does not expose compiled circuit metadata, the benchmark cannot be audited, and cross-platform comparisons using that platform should be treated with caution.
A more robust design encodes the experimental variable in gate parameters rather than circuit structure. A phase-sweep parity witness, for example, varies a rotation angle and measures the resulting oscillation visibility. Compilation cannot remove a parameter; it can only transform how that parameter is implemented.
A Practical Benchmarking Sequence
For a team evaluating devices for a specific workload, a staged approach avoids wasted effort.
Step 1: Establish the hardware baseline. Run single-qubit randomized benchmarking and two-qubit gate characterization on the specific qubits you intend to use. Vendor-reported fidelities are often median values across the device; your workload may use qubits above or below that median. The Chinese standard’s inclusion of calibration cycle and duration as performance indicators reflects the reality that calibration state drifts and matters for reproducibility .
Step 2: Run a connectivity-aware circuit. Choose a circuit that exercises the interaction patterns your workload requires. If your algorithm needs long-range entanglement, run a GHZ state preparation across non-adjacent qubits and measure fidelity decay with qubit count. IBM’s full-stack benchmarking work describes this approach: benchmarks defined at different widths and depths, including utility-scale algorithm fragments alongside standard tests like GHZ generation .
Step 3: Audit the compiled circuit. For every benchmark result you intend to report or act on, retrieve the compiled circuit and verify that the intended structure survived. The Zenodo study provides a reproducible template: compare submitted versus compiled gate set, qubit labels, entangling gate counts, single-qubit gate counts, barrier instructions, and phase representation . If barriers are absent from the compiled circuit when they were present in the submitted one, the depth-proxy benchmark is invalid.
Step 4: Report success probability and runtime together. A benchmark that reports only solution quality without runtime, or only runtime without quality, is not a benchmark. The optimization benchmarking literature has documented the “semantic discord” problem: papers claim one solver “outperforms” another based on success probability while omitting that the winning solver ran orders of magnitude longer . For quantum devices, this means reporting both the probability of obtaining a correct or high-quality result and the wall-clock time required, including compilation and queuing.
Decision Guidance: Which Metric Matters for Which Workload
No single benchmark predicts performance across all applications. The metric that matters depends on what the device will be used for.
| Workload type | Primary metric | Why it matters |
|---|---|---|
| Variational algorithms (VQE, QAOA) | Two-qubit gate fidelity, connectivity | Circuit depth is limited by error accumulation; routing overhead reduces effective fidelity |
| Quantum simulation | Coherence time, gate fidelity, maximum entangled qubits | Requires sustained coherent evolution over many qubits |
| Algorithmic benchmarking | Quantum volume or algorithmic qubits | Measures largest executable circuit of arbitrary structure |
| Error-corrected workloads | Logical error rate, distance scaling | Only relevant once fault tolerance is demonstrated |
For teams without a specific algorithm yet, quantum volume provides a reasonable general-purpose floor, but it should not be the only metric. A device with high quantum volume but poor connectivity for your intended circuit topology may underperform a device with lower quantum volume and better topology match.
Common Mistakes and How to Avoid Them
Mistake 1: Trusting vendor-reported median fidelities. Vendor benchmarks often report median or best-case values across the device. Your circuit uses specific qubits, and those qubits may be below median. Run your own characterization on the qubits you will actually use.
Mistake 2: Comparing results across platforms without auditing compilation. A benchmark that produces different results on two platforms may be measuring the compilers, not the hardware . Audit compiled circuits before drawing conclusions.
Mistake 3: Reporting success probability without runtime. A high success probability achieved with heroic tuning or extended runtime is not comparable to a lower success probability achieved quickly . Report both dimensions.
Mistake 4: Ignoring calibration state. A device that was calibrated hours before your benchmark may have drifted by the time you run. Check calibration timestamps and, if possible, request recalibration before critical benchmarks.
Mistake 5: Using benchmarks that cannot scale. Quantum volume does not scale beyond small devices; random circuit sampling can scale but has limited practical relevance. Choose a benchmark suite with a path to utility-scale circuits .
How to Verify That Your Benchmarking Is Sound
Before reporting or acting on benchmark results, confirm four things. First, the submitted and compiled circuits have been compared, and the intended experimental variable survived. Second, the qubits used are identified and their individual fidelities are reported, not just device medians. Third, both success probability (or fidelity) and runtime are reported. Fourth, the benchmark conditions—shots per configuration, calibration timestamp, compilation settings—are documented so that another team can reproduce the result.
If any of these are missing, the benchmark is a data point, not a decision input.
Frequently Asked Questions
Q1. What is the most important metric for benchmarking a quantum computer?
There is no single most important metric. Two-qubit gate fidelity, connectivity, and coherence time together determine what circuits a device can execute. Quantum volume provides a composite floor, but it does not predict performance on specific workloads.
Q2. Why is qubit count a misleading benchmark?
A device may have 100 physical qubits but only 60 that are reliably controllable and readable. The Chinese national standard explicitly distinguishes total, controllable, and readable qubit counts . Effective capability depends on the usable subset.
Q3. What is compiler-induced artifact in quantum benchmarking?
It is when a compiler optimizes away the feature a benchmark intended to test. For example, a compiler may remove barriers designed to create circuit-depth variation, producing operationally equivalent circuits that cannot measure depth-dependent decoherence .
Q4. How can I check if my benchmark was affected by compilation?
Retrieve the compiled circuit from the platform metadata and compare gate counts, barrier presence, and phase representation to the submitted circuit. If the platform does not expose compiled circuit metadata, the benchmark cannot be audited .
Q5. What is quantum volume and what are its limitations?
Quantum volume measures the largest square circuit of arbitrary two-qubit gates a device can execute with acceptable fidelity. It is useful as a general capability floor but does not scale beyond small devices and does not predict performance on specific algorithm structures .
Q6. How do I benchmark a device for a specific algorithm?
Run circuit fragments from that algorithm at multiple widths and depths, audit the compiled circuits, and measure both fidelity and runtime. IBM’s full-stack approach uses utility-scale algorithm fragments alongside standard benchmarks like GHZ state preparation .
Q7. Why do some benchmark papers report contradictory results?
Contradictions often arise from different test conditions: different qubits, different compilation settings, or different definitions of “outperforms.” The optimization literature has documented cases where success probability is reported without runtime, making comparisons meaningless .
Q8. Should I trust vendor-published benchmark results?
Vendor benchmarks are useful starting points but should be independently verified on the specific qubits and circuit structures you intend to use. Independent access to hardware, as IBM has provided for protocol-based benchmarking, enables verification that vendor-published numbers do not .
Q9. What is a reasonable benchmark suite for a team new to quantum computing?
Start with single-qubit and two-qubit randomized benchmarking on the specific qubits you will use, then a GHZ state preparation across non-adjacent qubits to test connectivity and entanglement fidelity. Report results with calibration timestamps and compiled circuit audits.
Q10. How do standards like GB/T 47468-2026 affect benchmarking practice?
The Chinese national standard formalizes a three-layer indicator structure—device, basic control, and comprehensive performance—and includes often-omitted metrics like energy consumption and calibration capability . It provides a useful checklist for what a complete benchmark report should include.
Conclusion
Sound quantum benchmarking requires auditing what the compiler actually executed, reporting fidelity and runtime together, and characterizing the specific qubits your workload will use. The most common failure is not hardware limitation—it is a benchmark design that does not survive compilation. Before your next device comparison, retrieve the compiled circuits, confirm the experimental variable survived, and document the conditions. That audit is the difference between a data point and a decision.