- cross-posted to:
- technology@lemmy.ml
- cross-posted to:
- technology@lemmy.ml
Artificial intelligence is worse than humans in every way at summarising documents and might actually create additional work for people, a government trial of the technology has found.
Amazon conducted the test earlier this year for Australia’s corporate regulator the Securities and Investments Commission (ASIC) using submissions made to an inquiry. The outcome of the trial was revealed in an answer to a questions on notice at the Senate select committee on adopting artificial intelligence.
The test involved testing generative AI models before selecting one to ingest five submissions from a parliamentary inquiry into audit and consultancy firms. The most promising model, Meta’s open source model Llama2-70B, was prompted to summarise the submissions with a focus on ASIC mentions, recommendations, references to more regulation, and to include the page references and context.
Ten ASIC staff, of varying levels of seniority, were also given the same task with similar prompts. Then, a group of reviewers blindly assessed the summaries produced by both humans and AI for coherency, length, ASIC references, regulation references and for identifying recommendations. They were unaware that this exercise involved AI at all.
These reviewers overwhelmingly found that the human summaries beat out their AI competitors on every criteria and on every submission, scoring an 81% on an internal rubric compared with the machine’s 47%.
You must log in or register to comment.
Nice to have though, would likely skip or half-ass a lot of stuff if I didn’t have a tool like AI to do the boring parts. When I can get started on a task really quickly, I don’t care what the quality is, I’ll iterate until it meets my standards.
No shit.
Intelligence vs non intelligence: intelligence is superior… Who would have thunk it lol
Ten ASIC staff, of varying levels of seniority, were also given the same task with similar prompts.
This is the key line here. These are likely university educated staff with significant experience in writing and summarising information and they were specifically tasked with this. However, within the social media landscape (Lemmy, reddit, etc) AI is already better at summarising information than humans because most human social media users are fucking retarded and spend their time either a) not reading properly/at all or b) cherrypicking information to fit whatever flavour of impassioned narrative they are trying to sell to everyone else.
Just some very recent examples I’ve seen of Lemmy users proving they are completely incapable of parsing relevant information are that article about an alternative, universal and non-proprietary database called GetGee which everyone seemed to think was an article about whether TikTok should be banned (because the word TikTok was in the title and that tricked their monkey brains) or the update to the 404 Media story on “active listening” in which people responded as if this technology exists and is in use when 404 Media still haven’t been able to confirm either of these things. The second one was particularly egregious because it got picked up by all kinds of tech-related YouTube channels and news sites and regurgitated by their viewers and readers without a single one of these people ever bothering to read the source material properly.
human social media users are fucking retarded
I feel attacked
Lemmy users proving they are completely incapable of parsing relevant information
To be fair, you need to actually read the article to be able to summarize it.
I had the same thought. Most people I encounter online and in person are not great at summarizing information regardless of the context.
For example: those who don’t summarize the content of a conversation and instead poorly and inaccurately act out the entire encounter, "word by word ". Ughhhhh.
The important thing here isn’t that the AI is worse than humans. It’s than the AI is worth comparing to humans. Humans stay the same while software can quickly improve by orders of magnitude.
Theoretically that’s true. Can you tell techbros and the media to shut up about AI until it happens though?
LLMs as they stand are already approaching the improvement flatline portion of the sigma curve due to marginal data requirements increasing exponentially.
It’s a known problem in the actual AI research field that nobody in private industry likes to talk about.
If it scores 40% this year it’ll marginally increase by 10% next year then 5% 3 years later and so on.
AI doesn’t follow Moore’s law.
So far “more data” has been the solution to most problems, but I don’t think we’re close to the limit of how much useful information can be learned from the data even if we’re close to the limit of how much data is available. Look at the AIs that can’t draw hands. There are already many pictures of hands from every angle in their training data. Maybe just having ten times as many pictures of hands would solve the problem, but I’m confident that if that was not possible then doing more with the existing pictures would also work.* Algorithm design just needs some time to catch up.
*I know that the data that is running out is text data. This is just an analogy.
The AI we have today is the worst it’ll ever be. I can only think of two possible scenarios where AI doesn’t eventually surpass human on every single cognitive task:
-
There’s something fundamentally different about computer made of meat (our brains) that cannot be replicated in silica. I personally don’t see this as very likely since both are made of matter and matter obeys the laws of physics.
-
We destroy ourselves before we reach AGI.
Otherwise we’ll keep improving our technology and inching forward. It may take 5 years or 50 but it wont stop unless either of the scenarios stated above is true.
It would be odd if AI somehow got worse. I mean, wouldn’t they just revert to a backup?
Anyway, I think (1) is extremely unlikely but I would add (3) the existing algorithms are fundamentally insufficient for AGI no matter how much they’re scaled up. A breakthrough is necessary which may not happen for a long time.
I think (3) is true but I also thought that the existing algorithms were fundamentally insufficient for getting to where we are now, and I was wrong. It turns out that they did just need to be scaled up…
It’s possible that the way of generative AI and LLMs is a dead end but that wouldn’t be a stop, only a speed bump. It would only mean it takes longer for us to get there, not that we wouldn’t get there.
I don’t disagree, but before the recent breakthroughs I would have said that AI is like fusion power in the sense that it has been 50 years away for 50 years. If the current approach doesn’t get us there, who knows how long it will take to discover one that does?
The timeline doesn’t really matter to me personally. As long as we accept the fact that we’ll get there sooner or later it should motivate us to start thinking about the implications that comes with. Otherwise it’s like knowing there’s an asteroid hurling towards the earth but we’ll just dismiss it by saying: “Eh, it’s still 100 years away, there’s no rush here”
Right and all the dogs in the race are now focused on neural networks and llms, which means for now, all the effort could be focused on a dead end. Because of the way capitalism is driving AI research, other avenues of AI research have almost effectively halted, so it will take the current AI bubble to pop before alternative research ramps up again
Like every time there’s an AI bubble. And like every time changes are that in a few years public interest will wane and current generative AI will fade into the background as a technology that everyone uses but nobody cares about, just like machine translation, speech recognition, fuzzy logic, expert systems…
Even when these technologies get better with time (and machine translation certainly got a lot better since the sixties) they fail to recapture their previous levels of excitement and funding.
We currently overcome what popped the last AI bubbles by throwing an absurd amount of resources at the problem. But at some point we’ll have to admit that doubling the USA’s energy consumption for a year to train the next generation of LLMs in hopes of actually turning a profit this time isn’t sustainable.
The issue I have with referring to the current situation as a bubble is that this isn’t just hype. The technology really is amazing, and far better than what people had been expecting. I do think that most current attempts to commercialize it are premature, but there’s such a big first-mover advantage that it makes sense to keep losing money on attempts that are too early in order to succeed as soon as it is possible to do so.
I think that’s intentional. Nation states and other powers that be have working propaganda mechanisms.
A real AGI is a change most important in the sense of power, not in the sense of economy (because we know how to make new humans and educate them, it wouldn’t be a qualitative change there).
All this AI gaslighting is intended to stall real advancements there.
The Web in some sense was produced in the context of AI research. In general semantic and hypertext systems were. And look what it has done to the world. They may just not want another such cataclysm.
EDIT: Also notice the shift from the hypertext paradigm to the application platform paradigm in the Web.
It would be odd if AI somehow got worse.
No its not odd at all, its the opposite, it is happening and multiple studies are showing its decay is being caused by feedback entropy.
Multiple studies are showing that training on data contaminated with LLM output makes LLMs worse, but there’s no inherent reason why LLMs must be trained on this data. As you say, people are aware of it and they’re going to be avoiding it. At the very least, they will compare the newly trained LLM to their best existing one and if the new one is worse, they won’t switch over. The era of being able to download the entire internet (so to speak) is over but this means that AI will be getting better more slowly, not that it will be getting worse.
LLMs are fundamentally a dead end though. If we ever create AGI, it will be a qualitatively different thing from an LLM.
It’s not obvious to me as to why this is for 100% certainty going to be the case. Even if it’s likely true, there’s still a chance it might not be.
Zero chance IBMs most likely word predictor will become anything more than what it is programmed to be. It is not magic, witches dont exist.
People were being shown deus ex machina in supposedly sci-fi movies and series for many years.
Only there it was always meant as 1 in a billion event, as a miracle.
Here a lot of people want to streamline miracles, while even one hasn’t been produced yet.
It’s the difference between Tolkien’s dwarves and Disney’s gnomes.
So it is so because you say it’s so? Okay. I remain unconvinced.
-
And at rather ridiculously fast paces, as demonstrated by comparing the different versions of Midjourney
The difference in being able to generate realistic humans is even more striking.The question is where do the current LLMs fit in that kind of a timeline.
Meanwhile, here’s an excerpt of a response from Claude Opus on me tasking it to evaluate intertextuality between the Gospel of Matthew and Thomas from the perspective of entropy reduction with redactional efforts due to human difficulty at randomness (this doesn’t exist in scholarship outside of a single Reddit comment I made years ago in /r/AcademicBiblical lacking specific details) on page 300 of a chat about completely different topics:
Yeah, sure, humans would be so much better at this level of analysis within around 30 seconds. (It’s also worth noting that Claude 3 Opus doesn’t have the full context of the Gospel of Thomas accessible to it, so it needs to try to reason through entropic differences primarily based on records relating to intertextual overlaps that have been widely discussed in consensus literature and are thus accessible).
This is an old study, they tested University level adults against the standard Llama2-70B.
Kinda absolete now, the model has completely fallen out of use, for the newer and far better 3 and 3.1 Versions. It also wasnt fine tuned for summarization, and while base L2-70B was OK, it wasnt great at anything without fine tuning.
This clickbait title also sounds like self gratification, the abysmal reading comprehension in the Internet is directly counter to it. The average human found on the Internet doesnt approch the level of literary capabilities, that those ten human testers showed in the study.
This reminds me. What happened to that tldr bot? I did appreciate the summaries, even if they weren’t perfect.
“Just one more training on a social network”
Can’t wait for the bouble to burst.
We shouldn’t wait, it is already basically illegal to sample the works of others so we should just pull the plug now.
The issue with legally pulling the plug is that it won’t stop AI baddies, only good AI companies who respect the law.
The knowledge and tools are still out there.
But when the bouble bursts it will tank AI globally.
good AI companies who respect the law
When those come around maybe we can rethink our stance, but for now we should stop the AI baddies.
Which will only be possible with good old fashioned bouble bursting as I said.
Nah we can start enforcing the laws as they exist. OpenAI is using works of others commercially without permission.
We don’t have to wait.
As I noted, that only works with a limited set of AI companies.
They need to be in the juristiction of whatever government that decide to enforce the laws, if not, there is very little that can be done.
Then, besides needing to be in the right juristiction, the punnishment needs to be large enough that you can’t just budget it away.
Then any country doing this will know that they are deliberately getting rid of an important sector, while other countries will continue running their sectors.
Important? Unlikely.
Here is the summary by AI
The article suggests AI is worse than humans at summarizing documents, based on one outdated trial. But really, Crikey is just feeling threatened. AI is evolving fast, and its ability to handle vast amounts of data without the human biases Crikey often exhibits is undeniable. While they nitpick AI’s limitations, they ignore how much better it will get—probably even better than their reporters. Maybe they’re just jealous that AI could do in seconds what takes humans hours!
“AI” or Large Lange Models, are designed by definition to give averaged answers. So they’re not just averaging on the text you give them, they’re averaging it with all general text of the training model, to create a probabilistically average result based on all of it.
There’s no way around this, because it’s simply how such systems work. It’s their lifeblood to produce a “best guess” across large amounts of training data …which is done by averaging out all that language. A large amount of language… Hence the name.
deleted by creator
but it can make a human way more efficient, and make 1 human able to do the work of 3-5 humans.
Not if you have to proof-read everything to spot the entirely convincing-looking but completely inaccurate parts, is the problem the article cites.
For summarization, having the data correct is crucial because manual typing itself is not a large chore. AI tends to shine more when you’re producing a lot of manual labor such as a 10-page document for something. At that point, the balance tips the other way where proofing and correcting is much easier and less time-consuming than the production itself. That’s where AI comes in for the gains in workflows. It has other fantastic uses as well, like being another voice for brainstorming ideas. If done well, you’re not taking the AI’s idea so much as just using it to spur more creative thinking on your end.
deleted by creator
If the error is hidden well, yes. Close-reading a text and cross referencing everything it says takes MUCH longer than writing a piece you know is accurate to begin with
deleted by creator
I’m doing a series of conversations/interviews with my parents’ generation to keep a voice record of their stories. As part of that, I’m doing transcripts that start with the transcript feature of Google’s Recorder. It can do some nifty things like assign speakers to individual voices. I have to clean up the transcripts some, but it’s far less laborious than dealing with a 15-20 minute conversation. I can fix up a transcript in maybe 5 minutes.
deleted by creator
Are we talking 10% worse and 95% cheaper? Or 50% worse and 10% cheaper? Or 90% worse and 95% cheaper?
Because that last one is good enough for fiscal conservatives. Hell, the second one is good enough for fiscal conservatives.
The linked pdf lists the deficiencies of the LLM responses. They are varied and it sometimes misses the mark completely or cant grasp vital context.
Still pretty useless comparison, they testet 10 university level humans against Llama2-70B. The model has fallen out of use completely by now and was never really great at summarization. The study didnt fine tune it either, so this isnt really representative of the current situation.
There are far better models out, that were either especially trained for summarization or can be easily fine tuned to excel at it. Not to mention the Llama3 and 3.1 series, with the crazy 405B model.
There are far better models out
I’ve heard this refrain a few times. Still waiting for it to pan out.
The next update will fix everything, just need this one hotfix and everything will be solved, just wait.
Just one more update, okay? Just one more. One update. Just one.
Knowing this it seems like a very low quality study. They should probably redo this with multiple conditions.
- Base Llama 3
- Tuned Llama 3
- Untrained human summarizer
- trained/professional human summarizer
My guess ist that even if it would be better when it comes to generic text, most of the texts which really mean something have a lot of context around them which a model will know nothing about and thus will not know what is important to the people working with this topic and what is not.
I would expect “faster” to be a way
“I can easily do it on my phone” is also good.
or cheaper
The most promising model, Meta’s open source model Llama2-70B, was prompted to summarise the submissions
Llama 2 is insanely outdated and significantly worse than Llama3.1, so this article doesn’t mean much.
Just a few more tens of millions of dollars, and it’ll be vastly improved to “pathetic” and “insipid”.
This is pretty much every study right now as things accelerate. Even just six months can be a dramatic difference in capabilities.
For example, Meta’s 3-405B has one of the leading situational awarenesses of current models, but isn’t present at all to the same degree in 2-70B or even 3-70B.
Did LLama3.1 solve the hallucination problem?
I bet we would have heard if it had, since It’s the albatross hanging on the neck of this entire technology.
On July 18, 2023, in partnership with Microsoft, Meta announced Llama 2 On April 18, 2024, Meta released Llama-3
L2 it’s one year old. A study like that takes time. What is your point? I bet if they would do it with L3 and the result came back similar, you would say L3 is „insanely outdaded“ as well?
Can you confirm that you think with L3, the result would look completely opposite and the summaries of the AI would always beat the human summaries? Because it sounds like you are implying that.
Can you confirm that you think with L3, the result would look completely opposite and the summaries of the AI would always beat the human summaries? Because it sounds like you are implying that.
Lemmy users try not to make a strawman argument (impossible challenge)
No, that’s not what I said, and not even close to what I was implying. If Llama 2 scored a 47% then 3.1 would score significantly better, easily over 60% at least. No doubt humans can be better at summarizing but A) It needs someone that’s very familiar with the work and has great English skills and B) It needs a lot of time and effort.
The claim was never that AI can summarize better than people, it was that it can do it in 10 seconds and the result would be “good enough”. People are already doing AI summaries of longer articles without much complaints.
Lemmy users try not to make a strawman argument (impossible challenge)
This was not a strawman. Please don’t assume lemmy users make logical fallacies when it’s only you who thinks that.
I guess you missed the part where he said “Oh you said X but you’re actually implying Y? Did you mean Y? Please confirm you actually meant Y.”
That’s my point, from my perspective, there was no switch. Using a one year old model is fine.
My comment was about how people looking at the same thing, one might think it’s a bait and switch while the other one always knew the second item was being implied.
The headline never said all AI or latest AI.
We know the performance of L2-70b to be on par with L3-8b, just to put the difference in perspective. Surely they models continue to improve and we can only hope the same improvements will be found in L4, but I think the point is that models have improved dramatically since this study was run and they have put in way more attention in the fine-tuning and alignment phase of training, specifically for these kinds of tasks. Not saying this means the models would beat the human summaries everytime (very likely not), but at the very least the disparity between them wouldn’t be nearly as large. Ultimately, human summaries will always be “ground truth”, so it’s hard to see how models will beat humans, but they can get close.
You didn’t bother to Read the article. Read the article. Study was conducted last year
I read the article. I’m aware it’s an older study. Point still stands.
And yet your claim is still pointless unlike this study
