OpenAI’s AGI Claim: A Critical Look
OpenAI’s AGI Number: A Question of Harness vs Model
OpenAI's declaration of the Artificial General Intelligence (AGI) era was based on a 99.9% score from GPT-6 Astra. However, when running the same model through the benchmark software provided by ARC Prize, the score dropped significantly to 62.7%. This disparity highlights the importance of the harness, or the software surrounding the model, in determining its performance.
The Harness Effect
A harness dictates what tools a model can access, its contextual memory, and how it manages conversations. When OpenAI's Provider Adapter harness was used, preserving the model's opaque reasoning state, Astra achieved a 99.9% score at a lower cost compared to the standard ARC Prize harness.
Benchmark Results: A Closer Look
ARC Prize ran Astra under both harnesses and published the results:
- Standard harness: 62.7% at maximum reasoning, costing $26,098.
- Provider Adapter: 99.9% at high reasoning, costing $18,817.
Interestingly, setting the reasoning effort to none within OpenAI's adapter still yielded a respectable 96.7% score, surpassing the standard harness by 34 points.
Token Efficiency and Speed
The Provider Adapter runs were more efficient, using 49% fewer tokens and completing tasks roughly 3.66 times faster than the standard harness.
Misinterpretation of Results
The reported figure of 99.9% against GPT-5.6 Sol's 7.8% was widely shared, including in our own coverage. However, this comparison is between different harnesses, not like-for-like. The benchmark itself states it is "not claiming that [Astra] is AGI".
Evolving Scores
After publication, Fortune discovered that some scores had changed. OpenAI altered five metrics, with Astra's hallucination rate fluctuating from 4.2% to 2%. Other models' scores also shifted, with Sol's ExploitBench score doubling.
Conclusion
The AGI debate continues, with a focus on the role of the harness in model performance. As ARC Prize moves forward, it will publish both harness results side by side for increased transparency.