A knockout blow for LLMs
Created: 2025-06-24
Apple has a new paper; it’s pretty devastating to LLMs, a powerful followup to one from many of the same authors last year.
See in context at A knockout blow for LLMs?
Created: 2025-06-24
On the one hand, it echoes and amplifies the training distribution argument that I have been making since 1998: neural networks of various kinds can generalize within a training distribution of data they are exposed to, but their generalizations tend to break down outside that distribution.
See in context at A knockout blow for LLMs?
Created: 2025-06-24
The new Apple paper adds to the force of Rao’s critique (and my own) by showing that even the latest of these new-fangled “reasoning models” still —even having scaled beyond o1 — fail to reason beyond the distribution reliably, on a whole bunch of classic problems, like the Tower of Hanoi
See in context at A knockout blow for LLMs?
Created: 2025-06-24
It is truly embarrassing that LLMs cannot reliably solve Hanoi. (Even with many libraries of source code to do it freely available on the web!)