No, it’s not that these models are unoptimized. Infact the opposite. These models are optimised and are optimised well.
Gpt 6 Astra is a looped transformer. While idea sounds simple, loop the output through the transformer again and again, these transformers punch way above their weight in tasks that are iterative, like coding or higher level planning. Now again, by higher level planning I don’t mean your everyday tasks, it’s mostly the tasks that benefit from being revisited, actually more like refining. I’ve read a 7b looped transformer beating qwen 72b coder model, and I’ll link it in the edit.
It’s naive to think these models are unoptimized, but it’s still even more naive to think these models are AGI. There are some genuinely great architectural advancements being made by both USA and China, yet I think it’s for the better this bubble pops as soon as possible. No technology in this world is worth destroying natural resources, and despite these advancements, LLMs are way too far from generating the economic output they project.
Edit: Multilingual-Multimodal-NLP/LoopCoder-V2 · Hugging Face huggingface.co/…/LoopCoder-V2
The version I was talking about was the 2 loop one.