AI models flub these intelligence tests. Can you fare any better?

AI performance on puzzles has improved quickly: a Columbia University team found leading models solved only 18% of New York Times Connections puzzles in late 2024, while some models approached near-perfect results by early 2025. Yet modified riddles, visual tasks, and problems that grow in complexity still expose sharp weaknesses.

The article surveys mental rotation, Knights and Knaves variants, SimpleBench, ARC-AGI, intuition traps, river crossings, and logic grids. Language models struggle particularly with manipulating three-dimensional objects and often perform better when an ARC grid is encoded as numbers rather than shown as an image. Studies also find performance dropping as Tower of Hanoi, river-crossing, and logic problems become larger.

The value of this collection is diagnostic, not competitive. Asking whether a person can beat a model at seven puzzles makes an engaging frame, but each task probes a different failure mode. A model that recalls a familiar riddle too quickly has a different problem from one that cannot mentally rotate an object or sustain a constraint set through six steps.

Contamination and memorization remain persistent benchmark hazards. Slight variations of familiar Knights and Knaves questions can reveal whether a system learned the rule or merely recognizes a template. The article's SimpleBench examples make the point cleanly: ordinary physical expectations can matter more than the arithmetic or narrative detail that attracts a language model.

Signal Desk's read: aggregate reasoning scores conceal the product risk that matters most—brittleness under small changes in representation or complexity. Buyers should prefer evaluations that transform the same underlying task across text, image, scale, and wording, then measure consistency rather than celebrating one benchmark total.

Human cognition is not the universal baseline either. The lightning-round examples exploit intuitive mistakes that people make while models may deliberate correctly. A useful evaluation program should map complementary strengths, not assume that every human advantage reflects general intelligence or every model advantage reflects genuine understanding.

Games have served as AI test beds since early machine learning, including Arthur Samuel's checkers work, chess, and Go. Newer puzzle suites target abstraction, memory, visual reasoning, intuition, and planning rather than only mastery of a fixed game.

ARC-AGI requires inferring transformations from example grids. Research cited in the article suggests that models can reach correct answers through complex rules that fail to generalize, while people tend to use simpler visual concepts. This gap between answer accuracy and solution quality is central to interpreting results.