ByteOrbital
Benchmark Methodology
Last updated
Benchmark numbers are only useful if you know where they came from. Every result on ByteOrbital carries a provenance label, and we do not blur the lines between them.
The three categories
1. ByteOrbital Tested
Results from tests we ran ourselves. For each one we publish what we can: the hardware and software versions, drivers, settings, power and thermal conditions, the test procedure, number of runs and how the figure was derived. We keep the raw evidence. Where a test cannot be repeated or the setup limits what it shows, we say so.
2. Independent Third-Party Results
Results measured by someone else, such as a testing lab, a standards body or a published open dataset, shown with attribution and a link to the source. We only include them if we have the right to: through an authorised API, a licence, permissible public data or manual entry with permission. We do not scrape or reproduce proprietary benchmark databases. We do not independently verify these figures unless we say we have.
3. Manufacturer Claims
Figures published by the company that makes the product, for example "up to 2x faster". These are always labelled as claims and shown with the conditions the manufacturer gave, where available. They are marketing until someone independent confirms them.
Never mixed without explanation
A single chart or table will not combine categories without clear visual separation and a note explaining why. If a manufacturer figure appears next to our own result, they are visibly distinct, and we say what the difference is.
Comparability rules
We only put results side by side when they are genuinely comparable. That means the same test and version, similar settings, resolution, precision or quantisation, and comparable software and drivers. When they are not, we either leave out the comparison or warn you. For AI models, results can depend heavily on prompts, context length, quantisation and runtime, so we record these. Averages and rankings state exactly which results they include.
No invented results
We do not estimate, extrapolate or fill gaps with made-up numbers. If we have no data for a product, the table says so. AI tools help us organise and present data, but they do not generate benchmark results. Every figure has a recorded source.
Corrections and updates
Software changes performance. If new drivers or firmware change a result, we retest or mark the old figure with its date. Spotted a mistake? Use our corrections process.
Related
See our editorial policy and sources and copyright.
Test methods in use
Each method has a version. When its harness or settings change, the version moves and results measured under different versions are never charted together.
AIME 2026 (provider-reported) current version 1
The 2026 American Invitational Mathematics Examination problems; score is the share answered correctly. Provider-reported: the score is copied from the model provider's own launch post, model card or system card, with the test name and evaluation setting as printed there. ByteOrbital has not run or reproduced it, so it is shown with a "Provider-reported" badge and is never ranked together with our own lab-measured results.
HarnessRun by the model provider; settings as stated in the source
Version history (1)
- v1 · 16 November 2026 · Created by catalog refresh
ByteOrbital lab runs (bo-bench) current version 1
Results measured by ByteOrbital on its own hardware with the in-house bo-bench harness. Each suite runs on one machine at a time; most tests run three times and the median is kept, tests that take hours run once. Every raw run value, the exact command and the tool version are stored with the result. Where we change one setting on the same machine (for example forcing the GPU), the result carries that configuration name, so only like-for-like results are compared. These are measurements, not manufacturer claims.
Harnessbo-bench 0.1.0
Version history (1)
- v1 · 14 November 2026 · Initial version, first-party results of 14 November 2026
GPQA Diamond (provider-reported) current version 1
Graduate-level multiple-choice science questions (biology, physics, chemistry) written to resist web search; the Diamond subset is the hardest, expert-validated slice. Provider-reported: the score is copied from the model provider's own launch post, model card or system card, with the test name and evaluation setting as printed there. ByteOrbital has not run or reproduced it, so it is shown with a "Provider-reported" badge and is never ranked together with our own lab-measured results.
HarnessRun by the model provider; settings as stated in the source
Version history (1)
- v1 · 16 November 2026 · Created by catalog refresh
Humanity's Last Exam (provider-reported) current version 1
Expert-written, closed-ended questions across many academic subjects, built to be hard for current frontier models. Provider-reported: the score is copied from the model provider's own launch post, model card or system card, with the test name and evaluation setting as printed there. ByteOrbital has not run or reproduced it, so it is shown with a "Provider-reported" badge and is never ranked together with our own lab-measured results.
HarnessRun by the model provider; settings as stated in the source
Version history (1)
- v1 · 16 November 2026 · Created by catalog refresh
MMLU-Pro (provider-reported) current version 1
Harder, ten-option version of the MMLU multi-subject knowledge and reasoning exam. Provider-reported: the score is copied from the model provider's own launch post, model card or system card, with the test name and evaluation setting as printed there. ByteOrbital has not run or reproduced it, so it is shown with a "Provider-reported" badge and is never ranked together with our own lab-measured results.
HarnessRun by the model provider; settings as stated in the source
Version history (1)
- v1 · 16 November 2026 · Created by catalog refresh
Terminal-Bench (provider-reported) current version 1
Agentic command-line tasks (build, debug, administer) in a sandboxed terminal; score is the share of tasks completed. Provider-reported: the score is copied from the model provider's own launch post, model card or system card, with the test name and evaluation setting as printed there. ByteOrbital has not run or reproduced it, so it is shown with a "Provider-reported" badge and is never ranked together with our own lab-measured results.
HarnessRun by the model provider; settings as stated in the source
Version history (1)
- v1 · 16 November 2026 · Created by catalog refresh