One other comment pointed me at one issue that might be a major difference. Is the code you generate in one of those ultra-verbose languages like Java where we had basically IDEs generating code from much shorter descriptions already 20 years ago? I could see LLMs doing well with those.
I tend to try to generate code mostly in Rust or sometimes shell or config files or DSL for various programs and 99% of the time the code does not even come close to what I wanted it to do, mainly because it just hallucinates itself some library interfaces that do not exist.
One other comment pointed me at one issue that might be a major difference. Is the code you generate in one of those ultra-verbose languages like Java where we had basically IDEs generating code from much shorter descriptions already 20 years ago? I could see LLMs doing well with those.
I tend to try to generate code mostly in Rust or sometimes shell or config files or DSL for various programs and 99% of the time the code does not even come close to what I wanted it to do, mainly because it just hallucinates itself some library interfaces that do not exist.