If you already know what you want but want to save time writing it

IME, going to ChatGPT for code usually meant losing time, cause I’d go back and forth trying to get a usable snippet and it would just keep refactoring the same slop that didn’t work in its first attempt

They’re pretty reasonable for consensus-based programming prompts as well like “Compare and contrast popular libraries for {use case} in {language}” or “I want to achieve {goal/feature} in {summary of project technologies}, what are some ways I could structure this?”

Of course you still shouldn’t treat any of the output as factual without verifying it. But at least in the former case, I’ve found it more useful than traditional search engines to generate leads to look into, even if I discard some or all of the specific information it asserts

Edit: Which is largely due to traditional search engines getting worse and worse in recent years, sadly

I enjoyed reading this, thank you.

So, they’ve basically accidentally (or intentionally) made Eliza with extra steps (and many orders of magnitude more energy consumption).

This is the best article I’ve seen yet on the topic. It does mention the “how” in brief, but this analogy really explains the “why” Gonna bookmark this in case I ever need to try to save another friend or family member from drinking the Flavor-Aid

Except they also aren’t reliable at parsing and summarizing links, so it’s irresponsible to use their summary of a link without actually going to the link and seeing for yourself.

It’s a search engine with confabulation and extra steps.

Perplexity has been great for my ADHD brain and researching for my master’s.

It all depends on the training data and preprompt. With the right combination of those, it will admit when it doesn’t know an answer most of the time.

they have to provide an answer

Indeed. That’s the G in chatGPT. It stands for generative. It looks at all the previous words and “predicts” the most likely next word. You could see this very clearly with chatGPT-2. It just generated good looking nonsense based on a few words.

Then you have the P in chatGPT, pre-trained. If it happens to have received training data on what you’re asking, that data is shown. It it’s not trained on that data, it just uses what is more likely to appear and generates something that looks good enough for the prompt. It appears to hallucinate, lie, make stuff up.

It’s just how the thing works. There is serious research to fix this and a recent paper claimed to have a solution so the LLM knows it doesn’t know.

LLMs don’t “feel”, “know”, or “understand” anything. They spit out statistically most significant answer from it’s data-set, that is all they do.

It’s trained on internet discussions and people on the internet rarely say, “I don’t know”.

The issue is: What is right and what is wrong?

"mondegreen"s are so ubiquitous that there are multiple websites dedicated to it. Is it “wrong” to tell someone that the song where Jimi Hendrix talked about kissing a guy is Purple Haze? And even pointing out where in the song that happens has value.

In general, I would prefer it if all AI Search Engines provided references. Even a top two or three pages. But that gets messy when said reference is telling someone they misunderstood a movie plot or whatever. “The movie where Anthony Hopkins pays Brad Pitt for eternal life using his daughter is Meet Joe Black. Also you completely missed the point of that movie” is a surefired way to make customers incredibly angry because we live in bubbles where everything we do or say (or what influencers do or say and we pretend we agree with…) is reinforced, truth or not.

And while it deeply annoys me when I am trying to figure out how to do something in Gitlab CI or whatever and get complete nonsense based on a single feature proposal from five years ago? That… isn’t much better than asking for help in a message board where people are going to just ignore the prompt and say whatever they Believe.

In a lot of ways, the backlash against the LLMs reminds me a lot of when people get angry at self checkout lines. People have this memory of a time that never was where cashiers were amazingly quick baggers and NEVER had to ask for help to figure out if something was an Anaheim or Poblano pepper or have trouble scanning something or so forth. Same with this idea of when search (for anything non-trivial) was super duper easy and perfect and how everyone always got exactly the answer they wanted when they posted on a message board rather than complete nonsense (if they weren’t outright berated for not searching for a post from ten years ago that is irrelevant).

I’ve had a similar experience. Except in my case I used lyrics for a really obscure song where I knew the writer. I asked Chat GPT, and it gave me completely the wrong artist. When I corrected it, it apologized profusely and agreed with exactly what I had said. Of course, it didn’t remember that correct answer, because it can’t add to it update its data source.

Yes!!! It doesn’t know Trump has been convicted and told me that even when I give it sources, it won’t upload to a central database for privacy reasons. 🤷‍♀️

It does remind me of that recent Joe Scott video about the split brain. One part of the brain would do something and the other part of the brain that didn’t get the info because of the split just makes up some semi-plausible answer. It’s like one part of the brain does work at least partially like an LLM.

It’s more like our brain is like a corporation, with a spokesperson, a president and vice president and a number of departments that with semi-independently. Having an LLM is like having only the spokesperson and not the rest of the work force in that building that makes up an AGI.

It’s literally just Markov chains with extra steps

based on weighted averages of ‘what people are saying’ with a little randomization to spice things up

That is massively oversimplified and not really how neural networks work. Training a neural network is not just calculating averages. It adjusts a very complex network of nodes in such a way that certain input generates certain output. It is entirely possible that during that training process, abstract mechanisms like logic get trained into the system as well, because a good NN can produce meaningful output even on input that is unlike anything it has ever seen before. Arguably that is the case with ChatGPT as well. It has been proven to be able to solve maths/calculating tasks it has never seen before in its training data. Give it a poem that you wrote yourself and have it write an analysis and interpretation - it will do it and it will probably be very good. I really don’t subscribe to this “statistical parrot” narrative that many people seem to believe. Just because it’s not good at the same tasks that humans are good at doesn’t mean it’s not intelligent. Of course it is different from a human brain, so differences in capabilities are to be expected. It has no idea of the physical world, it is not trained to tell truth from lies. Of course it’s not good at these things. That doesn’t mean it’s crap or “not intelligent”. You don’t call a person “not intelligent” just because they’re bad at specific tasks or don’t know some facts. There’s certainly room for improvement with these LLMs, but they’ve only been around in a really usable state for like 2 years or so. Have some patience and in the meantime use it for all the wonderful stuff it’s capable of.

Not all AIs, since many AIs (maybe even most) are not LLMs. But for LLMs, you’re right. Minor nitpick.

That’s exactly the issue here. ChatGPT’s current training set ends right around the time the Meta Quest 3 came out. It’s not going to have any discussions in there of No Man’s Sky with tech that wasn’t out yet.

LLMs are basically just really fancy search engines. The reason the initial code is garbage is that it’s cut and pasted together from random crap the LLM found on the net under various keywords. It gets more performant when you ask because then the LLM is running a different search. The first search was “assemble some pieces of code to accomplish X”, while the second search was “given this sample of code find parts of it that could be optimized”, two completely different queries.

As noted in another comment the true fatal flaw of LLMs is that they don’t really have a threshold for just saying " I don’t know that" as they are inherently probabilistic in nature. When asked something they can’t find an answer for they assemble a lexically probable response from similar search results even in cases where it’s wildly wrong. The more uncommon and niche your search is the more likely this is to happen. In other words they work well for finding very common information, and increasingly worse the less common that information is.

Well I asked some questions yesterday about classes of DAoC game to help me choose a starter class. It totally failed there attributing skills to wrong class. When poking it with this error it said : you are right, class x don’t do Mezz, it’s the speciality of class Z.

But class Z don’t do Mezz either… I wanted to gain some time. Finally I had to do the job myself because I could not trust anything it said.

They seem to give the average answer, not the correct answer. If you can bound your prompt to the range of the correct answer, great

If you can’t bind the prompt it’s worse than useless, it’s misleading.