@panos
Programming logic almost always requires some kind of recursion, and the way LLMs work limits how recursive it can technically be; also, programming logic often requires thinking in reverse/backwards, and LLMs are essentially forward-oriented (i.e. "which token comes next" instead of "which token comes at a given moment in the past"). In fact, when we read the CoT of a LLM (when it's not hidden; Gemini and Claude, for example, seem to be hiding their CoT), we see this phenomenon in action, with the LLM talking over itself ("The user requested me to analyze a ritualistic photograph taken on September 11th 2026... wait, this date is future, as the current year is 2024.”) as the previous tokens change the course of inference.
They can surely produce code, and the code can be surprisingly good (e.g. Catodon), but unless it's something like a Diffusion model (e.g. Mercury from Inception Labs) which can generate independent tokens in parallel for an unified output, they're constrained to this "arrow of time" (always the next token, can't rewrite an earlier token).
So how could they produce code if their mechanism is unable to go backwards? Because it's deriving from an already-recognized pattern, in turn derived from its training data.
Similar thing with their ability to find security vulnerabilities.
Most of the time, they don't seem to be doing what an Assembly programmer would do, which is keeping a blank notepad to keep track of every CPU register and allocated RAM segments as the dev manually follow the program flow, JMPing at every branching, simulating INTerrupts and SYSCALLs, as well as doing the changes/assignments to variables, including testing bogus input to variables, doing deliberate attempts to provoke buffer overflow and other known software gore. Or perhaps using gdb to make this without the need of a manual code execution.
They're doing heuristics, akin to how antivirus software detect a malware just by inspecting the binary payload. By the way, this is how EICAR test triggers AVs, it's matched against AV datasets (curiously, Anthropic's Claude has a magic string that triggers its safety guardrail mechanisms and stops the inference, and it's a string very similar in shape to the EICAR test string).
Because vulz can be modular, the tendency is for a conjunction of vulnerability detections to be trained upon and add new combinations that will then be matched against through heuristics.
This is important to be said because we may end up thinking the LLM is doing some kind of gdb-like bare metal inspection of the program, instruction by instruction, due to the MoE-oriented structure the LLM is using to answer back, when its actually "simply" (which, to be fair, is far from simple) maximizing a given set of floating points (context window) based on an optimized matrix multiplication involving up to hundreds of billions of floating points (model and its weights).
One main problem, the thing I end up simultaneously converging with AND diverging from the anti-AI people, is how it's slowly making us highly dependent on specific capitalist corps and conglomerates. That's part of the reasons why I use Qwen, DeepSeek, and to a lesser extent GLM (AI uses of which is part of the "diverging from the anti-AI people" part) more than I use, say, Mercury, Gemini, Claude or other more known options (including OpenAI's Chatgpt which I personally don't even use anymore)... because, even though Qwen belongs to Alibaba, they released open-weighted models... so if Alibaba ever walled their garden someday (unlikely, given how Zhongguó/China seems serious with the academic and scientific goals for AI; strikingly differently from the West AI corps, whose goals are profit to increasingly-billionaire CEOs and shareholders), we'd still have good models to run locally.
As Western corps continue improving their closed LLMs, sure they become more able and more accurate to detect software vulz, which leads to people using said LLMs to check for vulz, an usage of which gets fed back to train and make even more powerful LLMs (e.g. Mythos and Fable), but then people would be at the hands of OpenAI, Anthropic, Google and Meta, and it'll be way more difficult for other possibly open solutions (esp. DeepSeek and Qwen) to git gud at it. AKA, the well-known Network Effect.
Whenever I use Qwen and DeepSeek (mostly for learning new words and languages, cross-referencing concepts across disparate fields of knowledge, and sometimes as a digital Ouija board; ergo, more esoteric usage than objective usage, but I also tested LLMs with analysis of code and deciphering of cipher/steganography I created for fun), it's deliberate so to try and make them better than the Western/USian ones and pave the way to truly open and scientific AI options.
tldr LLMs can be powerful and useful, but IMHO we should be very careful with the Western ones lest we'd end up overly reliant on capitalist corps and conglomerates.
👾
