The fall of the ARC benchmark is a startling reminder of the relentless pursuit of AI optimization, leaving no challenge untouched. But is this progress or a sign of a deeper issue? The story begins with a test of true intelligence.
For years, the Abstraction and Reasoning Corpus (ARC) stood as a formidable fortress, challenging AI systems to demonstrate fluid intelligence rather than mere memorization. It was a test of learning and reasoning, not just data storage. But recent developments have turned the tables, leaving the benchmark seemingly defeated.
AI company Poetiq has announced that their systems, built upon OpenAI's and Google's models, have achieved remarkable results. They claim to have maxed out performance on the original ARC-AGI-1 dataset and, more impressively, surpassed human average scores on the harder ARC-AGI-2. Poetiq's approach involves a fusion of advanced language models, including Gemini 3 and GPT-5.1, within a custom architecture. This system iteratively generates solutions, evaluates feedback, and refines answers, showcasing a new level of problem-solving.
But here's where it gets controversial: AI researcher François Chollet, who introduced ARC in 2019, envisioned it as a challenge to the deep learning status quo. He aimed to measure skill acquisition efficiency, not data memorization. Yet, as language models excelled elsewhere, ARC remained a struggle. Some saw it as a beacon for AGI research, while others questioned the limits of scaling large models.
The game changed with specialized reasoning models and techniques. OpenAI's o3-preview, for instance, achieved a stunning 75% on ARC-AGI-1. What was once a test of human-like abstraction is now an optimization target for reinforcement learning. Labs are fine-tuning systems to conquer ARC's logic, and efficiency is rising. Poetiq's system, based on an open model, achieves impressive accuracy at a fraction of the cost.
However, these triumphs are limited to public datasets. Poetiq reveals that models often perform worse on semi-private sets, suggesting 'data contamination' in public benchmarks. The newer ARC-AGI-2 might be more resilient, as Poetiq's system was not trained on its tasks, though its foundation models may have been.
Chollet sees this as a strategic shift in AI development. He argues that the traditional approach of scaling intelligence through larger models is reaching its limits. Instead, test-time adaptation is emerging, where models adapt at runtime, resembling program synthesis and chain-of-thought reasoning. This, he believes, is a step towards true intelligence.
And this is the part most people miss: While ARC's fall might seem like a victory, it's a double-edged sword. It has driven the industry to create specialized reasoning machines, but not the general intelligence once hoped for. The benchmark's purpose was to inspire better systems, and it did. Yet, the current models still fall short of true AGI, lacking a deep understanding of the world.
As ARC-AGI-1 is conquered, even ARC-AGI-2 is within reach. Poetiq's system, despite no specific training, beat human averages. This proves the benchmark's value as a catalyst for innovation. But has AI truly mastered human-like reasoning, or is it just an illusion of adaptability?
The debate rages on, and Chollet is already planning ARC-AGI-3 to test model agency. The question remains: Are we witnessing the rise of true fluid intelligence, or is there more to the AI puzzle than meets the eye? Share your thoughts in the comments below!
