LLMs are solving MCAT, the bar test, SAT etc like they’re nothing. At this point their performance is super human. However they’ll often trip on super simple common sense questions, they’ll struggle with creative thinking.
Is this literally proof that standard tests are not a good measure of intelligence?
Kinda the same thing though. You spent time finding the right auto-complete in your head. You weighed the words that fit the sentence you’d constructed in order to find the point must frequently encountered in conversations or documents that include specific related words. We’re much more sophisticated at this process, but our whole linguistic paradigm isn’t fundamentally very different from good auto-complete.