Skip to content
ElaraHive

Mathematical infrastructure for complex systems · Institute for Decision Systems and Number Theory

Benchmarks · 19 June 2026

Elara on GPQA Diamond and HumanEval

These results show how Elara performed on science questions and Python programming tasks in June 2026. Each score includes the sample size, settings and source needed to interpret it.

Built by the ELARA·CORTEX Institute for Decision Systems and Number TheoryMeasured in public: drift proof · benchmarks · research

Research archiveCurrent features

We tested Elara's reasoning engine through the production API available at the time. The saved results contain a row for every item, including unanswered questions.

  • 89.1%

    GPQA Diamond

    139 of 156 answered items, production API, 19 June 2026

  • 90.9%

    HumanEval pass@1

    149 of 164 problems, first attempt against the hidden tests

  • 2,000,692

    Tokens of context

    on the recall test, every fact recovered in 72 of 72 trials

Method

GPQA answers were graded against the official key. HumanEval code was executed against the hidden tests, using the first attempt only. Provider errors count as unanswered; the GPQA table gives accuracy both for answered questions and for the full set.

Comparator figures come from lab announcements and system cards read on 6 September 2026. Epoch AI's independent results appear alongside them where available. Settings differ between runs, so the columns should be read separately. Elara's results have not been independently replicated; the saved item-level answers and keys are available for review.

GPQA Diamond

198 questions in physics, chemistry and biology, written and checked by domain experts (Rein et al., 2023). The paper reports 65% for experts with or pursuing PhDs in the domain, 74% after discounting the mistakes those experts identified in retrospect, and 34% for skilled non-experts.

ModelLab-publishedEpoch AI, independentBasis
Gemini 3.1 Pro94.3%92.6%lab: self computed; Epoch: high
Claude Opus 4.794.2%86.9%lab: system card; Epoch: max
GPT-5.593.6%92.0%lab: announcement; Epoch: none
Claude Opus 4.893.6%88.0%lab: 25 trials; Epoch: none
GPT-5.492.8%91.1%lab: xhigh; Epoch: medium
Gemini 3 Pro91.9%90.2%lab: model page
Claude Opus 4.691.3%87.4%lab: system card; Epoch: max
Claude Sonnet 4.689.9%83.2%lab: 3-run average; Epoch: max
Elara, maximum reasoning89.1%139 of 156 answered; 42 provider errors; 70.2% of all 198
Grok 487.5%82.7%lab: without tools; Epoch: a second run reads 87% (±2)
OpenAI o383.3%75.8%lab: without tools; Epoch: medium
DeepSeek R1-052881.0%68.4%lab: model card
DeepSeek R171.5%62.3%lab: pass@1, README

Rows are ordered by the lab-published score. The Basis column records the available reasoning settings; results from the two sources are not averaged.

Elara by domain

These percentages use all questions in each domain as the denominator, including provider errors. They therefore differ from accuracy on answered questions alone.

DomainCorrectItemsShare
Physics768688%
Chemistry529356%
Biology111958%
All items13919870.2%

The 19 June 2026 run used the .LEKOLA-MAX tier at maximum reasoning, then priced at $5 for input and $25 for output per million tokens. Results: docs/proof/gpqa_diamond_receipt.json.

HumanEval

164 Python problems (Chen et al., 2021), each answer executed against the hidden tests, with a problem counted as passed when the first attempt passes every test.

Modelpass@1PassedBasis
Elara V5 Pro90.9%149 of 164production API, 19 June 2026; the fifteen failing problems are listed in the receipt

The comparator sources checked on 6 September 2026 contained no HumanEval results: the GPT-5.5, GPT-5.4 and o3 announcements; Claude Opus 4.7, Opus 4.8, Opus 5 and Sonnet 4.6 system cards; Gemini 3.1 Pro table; DeepSeek-R1 README; and Grok 4 page. Elara's results are in docs/proof/humaneval_receipt.json.

Working memory

Elara recovered every planted fact in a separate 72-trial recall test across 2,000,692 tokens. That experiment used external memory mapped to the model. The other rows list published context-window specifications, so they measure a different property. See the recall method for details.

ModelContext, tokensBasis
Elara2,000,692measured: 72-trial exact recall across the whole context
GPT-5.51,050,000published; 128,000 output
GPT-5.41,000,000published; 272,000 standard
Gemini 3.1 Pro1,000,000published; 64,000 output
Claude Opus 4.81,000,000published; 128,000 output
Claude Sonnet 4.61,000,000published; 128,000 output
Grok 4256,000published
OpenAI o3200,000published; 100,000 output
DeepSeek R1128,000published

Context-window specifications were read from the labs' model documentation on 6 September 2026.

Sources

  1. The Elara receipts are docs/proof/gpqa_diamond_receipt.json (198 rows) and docs/proof/humaneval_receipt.json (164 rows) from the runs of 19 June 2026, and docs/proof/zero_drift_receipt.json, whose receipt is dated 30 August 2026.
  2. Anthropic publishes its GPQA Diamond figures in the system cards for Claude Opus 4.7, Claude Opus 4.8 and Claude Sonnet 4.6, and its context windows in the model documentation for Opus 4.8 and Sonnet 4.6.
  3. OpenAI publishes its figures in Introducing GPT-5.5, Introducing GPT-5.4 and Introducing OpenAI o3 and o4-mini, and its context windows in the model documentation for GPT-5.5 and o3.
  4. Google DeepMind publishes the Gemini 3.1 Pro figure on the model page and in the model card, with a separate evaluation methodology note.
  5. xAI publishes the Grok 4 figures in its Grok 4 announcement.
  6. DeepSeek publishes the R1 figures in the DeepSeek-R1 README and the R1-0528 figures in that model card.
  7. Epoch AI publishes its independent runs as benchmark data under a CC-BY licence, updated 5 September 2026, with a GPQA Diamond page.
  8. The benchmarks are defined in Rein et al., GPQA: A Graduate-Level Google-Proof Q&A Benchmark (2023), and Chen et al., Evaluating Large Language Models Trained on Code (2021).

Try Elara

Open ElaraHive or compare the current plans. The results above describe the dated runs and settings shown on this page.