Hi all!
As many of you have noticed, many Lemmy.World communities introduced a bot: @MediaBiasFactChecker@lemmy.world. This bot was introduced because modding can be pretty tough work at times and we are all just volunteers with regular lives. It has been helpful and we would like to keep it around in one form or another.
The !news@lemmy.world mods want to give the community a chance to voice their thoughts on some potential changes to the MBFC bot. We have heard concerns that tend to fall into a few buckets. The most common concern we’ve heard is that the bot’s comment is too long. To address this, we’ve implemented a spoiler tag so that users need to click to see more information. We’ve also cut wording about donations that people argued made the bot feel like an ad.
Another common concern people have is with MBFC’s definition of “left” and “right,” which tend to be influenced by the American Overton window. Similarly, some have expressed that they feel MBFC’s process of rating reliability and credibility is opaque and/or subjective. To address this, we have discussed creating our own open source system of scoring news sources. We would essentially start with third-party ratings, including MBFC, and create an aggregate rating. We could also open a path for users to vote, so that any rating would reflect our instance’s opinions of a source. We would love to hear your thoughts on this, as well as suggestions for sources that rate news outlets’ bias, reliability, and/or credibility. Feel free to use this thread to share other constructive criticism about the bot too.
Here’s the comment reply from when I first asked what was wrong with MBFC. Gotta say. I agree with that comment. I’m surprised more people haven’t posted similar examples here.
https://lemmy.dbzer0.com/comment/12328918
Edit: here is the text from the linked comment.
I’m just gonna drop this here as an example:
https://mediabiasfactcheck.com/the-jerusalem-report/
https://mediabiasfactcheck.com/the-jerusalem-post/
The Jerusalem Report (Owned by Jerusalem Post) and the Jerusalem Post
This biased as shit publication is declared by MBFC as VEEEERY slightly center-right. They make almost no mention of the fact that they cherry pick aspects of the Israel war to highlight, provide only the most favorable context imaginable, yadda yadda. By no stretch of the imagination would these publications be considered unbiased as sources, yet according to MBFC they’re near perfect.
This biased as shit publication is declared by MBFC as VEEEERY slightly center-right. They make almost no mention of the fact that they cherry pick aspects of the Israel war to highlight
You keep repeating this lie.
From their report on the Jerusalem Post:
Overall, we rate The Jerusalem Post Right-Center biased based on editorial positions that favor the right-leaning government. We also rate them Mostly Factual for reporting rather than High due to two failed fact checks.
Until 1989, the Jerusalem Post’s political leaning was left-leaning as it supported the ruling Labor Party. After Conrad Black acquired the paper, its political position changed to right-leaning, when Black began hiring conservative journalists and editors. Eli Azur is the current owner of Jerusalem Post. According to Ynetnews, and a Haaretz article, “Benjamin Netanyahu, the Editor in Chief,” in 2017, Azur gave testimony regarding Prime Minister Benjamin Netanyahu’s pressure. Current Editor Yaakov Katz was the former senior policy advisor to Naftali Bennett, the former Prime Minister and head of the far-right political party, “New Right.”
In review, The Jerusalem Post covers Israeli and regional news with strongly emotionally loaded language with right-leaning bias with articles such as this “Country’s founding Labor party survives near extinction” and “Netanyahu slams settler leader for insulting Trump.” . . . During the 2023 Israel-Hamas conflict, the majority of stories favored the Israeli government, such as this Netanyahu to Hezbollah: If you attack, we’ll turn Beirut into Gaza. In general, the Jerusalem Post holds right-leaning editorial biases and is usually factual in reporting.
They literally mention their bias over and over. Center-right is consistent with how they’re rated everywhere. Allsides rates them center with the note that the community thinks they lean right. Wikipedia rates them as centre-right/conservative. Your “VEEEERY slightly” bit is pure fabrication. They specifically note that they’re highly biased source on the conflict in Gaza.
The news source of this post could not be identified. Please check the source yourself. Media Bias Fact Check | bot support
Okay, so maybe we don’t need a comment if it’s a meta post or a mod announcement. Thanks for your inadvertent feedback, bot!
It’s this uninvited commenting on the bots part that has me downvoting it. It’s presenting itself at an authority here. If a user in the comments called the bot to fact check something and the bot did a bad job, i’d just block the bot. I’d even be able to look over that users history to get an idea of the bot’s purpose. But this bot comes in and says “here’s the truth”, then spits out something i’d expect to see on twitters current itteration.
If the problem you’re trying to solve is the reliability of the media being posted here. Take the left/right bias call out and find a decent databse on new source quality. Start the bots post out with resources for people to develop their own skill at spotting bad news content.
If the problem you’re trying to solve is the visibility of political bias in content posted here. So the down vote button isnt acting as a proxy for that. Adding a function for the community to rate left/right lean like rotten tomatoes sounds interesting, so long as you take the reliability rating out of the bot. You can’t address both media reliability and political bias in one automated post. nyt and npr being too pearl clutchy for my taste. and some outlet that only exists only on facebook having the same assumed credibility as the associated press. are wildly different issues.
it also does this with a bunch of weird little local newspapers or etc which I’ve never heard of, which is like the one time I actually want it to be providing me with some kind of frame of reference for the source. MSNBC and the NYT, I feel like I already know what I think about them.
Yeah, it’s tricky because who reviews those small guys? Granted, most of them are probably owned by a giant like Gannett, but that doesn’t mean we can just apply a rating from 1 small Gannett-owned paper to another. We’d like there to be some way for users to share their feedback/ratings on those small guys. But then it’s also true that some people will create a news site and try to share links on here to promote their new website and that’s typically just spam bots.
One problem I’ve noticed is that the bot doesn’t differentiate between news articles and opinion pieces. One of the most egregious examples is the NYT. Opinion pieces aren’t held to the same journalistic standards as news articles and shouldn’t be judged for bias and accuracy in the same way as news content.
I believe most major news organizations include the word “Opinion” in titles and URLs, so perhaps that could be something keyed off of to have the bot label these appropriately. I don’t expect you to judge the bias and accuracy of each opinion writer, but simply labeling them as “Opinion pieces are not required to meet accepted journalistic standards and bias is expected.” would go a long way.
This contributes significantly to the noise issue most people complain about
Interesting that people say that opinion pieces should not be held to the same standard. I personally see such pieces contribute to fake news going around. Shouldn’t a platform with reach, held accountable for wrong information, they hide behind an opinion piece?
This. Otherwise op-eds get a free pass to launder opinions the paper wants to publish, but can’t.
It’s not a question of “should” - an opinion piece is rhetoric, not reporting. You can fact check some of it sometimes but functionally can’t hold it to the same standards as a regular news article. I agree that this can sometimes lead to “alternative facts” and disingenuous arguments, but the only other option is to forbid the publication of them which is obviously an infringement of first amendment rights. It’s messy, and it can lead to people being misinformed, but it’s what we’re stuck with.
Can you explain how a piece with a title like “Helldivers is awesome and fun” can be judged at all for factual accuracy?
The NYT ran an opinion recently where the author pretty clearly was using the NYT along with other outlets as part of a voter demobilization tactic in which the author lied about not voting. The NYT was skewered on twitter, and had to alter the opinion after the fact. It seems like some basic fact checking would have been useful in that situation. Or really, just any amount of critical thought on the part of the NYT in general.
Thanks for this. As a mod of /c/news, I hadn’t really thought about that. We don’t allow opinion pieces, but this is very relevant if we roll out a new bot for all the communities that currently use the MBFC bot.
Hi. I have a suggestion:
Try to make it more clear that this is not a flawless rating (as that is impossible).
Ways to implement:
- Make sure the bot says something along the lines of “MBFC rates X news as Y” and not “X news is Y”.
- Make a caveat (collapsable) at the bottom, that says something along the lines of “MBFC is not flawless. It has an american-centric bias, is not particularly clear on methodology, to the point where wikipedia deems it unreliable; however, we think it is better to have this bot in place as a rough estimate, to discourage posting from bad sources”
- If possible, add other sources, Like: “MBFC rates the Daily Beast as mostly reliable, Ad Fontes Media rates it as unreliable, and Wikipedia says it is of mixed reliability”
- Remove the left right ratings. We already have a reliability and quality rating, which is much more useful. The left-right rating is frankly poorly done and all over the place, and honestly doesn’t serve much purpose.
No problem. Specifically came to my attention about a week ago on this post where the bot reported on an opinion piece as if it was straight news.
BTW, I actually do appreciate the bot and think it’s doing about as well as it can given the technical limitations of the platform.
This bot was introduced because modding can be pretty tough work at times and we are all just volunteers with regular lives.
Then maybe it can be an internal thing only. Let people do their own critical thinking. I believe that if you’re on Lemmy, you can make informed decision.
It has been helpful and we would like to keep it around in one form or another.
Bull fucking shit. The majority of feedback has been negative. I can’t recall a single person arguing in its favor, but I can think of many, myself included, arguing against it. I hope you can find my report of one particular egregious example, because Lemmy doesn’t let me see a history of things I reported. I recall that MBFC rated a particular source poorly because they dared to use the word “genocide” to describe what’s going on in Gaza. Trusting one person, who clearly starts from an American point of view, and has a clearly biased view of world events, to be the arbiter of what is liberal or conservative, or factual or fictional, is actively harmful.
No community, neither reddit nor Lemmy nor any other, has suffered for lack of such a bot. I strongly recommend removing it. Non-credible sources, misinformation, and propaganda are already prohibited under rule 8. If a particular source is so objectionable, it should be blacklisted entirely. And what is and is not acceptable should be determined in concert with the community, not unilaterally.
Just as a point of clarification, there is certainly not a community consensus among the feedback.
While you are absolutely correct in stating that there are vocal members of the community opposed to it in any form, there is also a significant portion of the community that would prefer to keep or modify how it works. The most team will be taking all of these perspectives into account. We hope that you will be respectful of community members with whom you disagree.
I haven’t seen any strong arguments for keeping it up.
I will start by saying that I feel like we are trying to address the criticism in your first paragraph with these changes. That being said, thanks for your feedback. I particularly like the comment you shared under the “edit,” because I hadn’t seen that sentiment shared before (not saying nobody else had that issue, just appreciating you for contributing that and challenging me to think more about how we execute things).
I also would like it not add to the comment count. I am now getting inured to comment counts of “1”.
I generally like the bot and its intentions, but feel it inaccurate with my perception too often.
Yes! The mods starting out the discussion with their preferred outcome is so incredibly telling. This is a tool to reinforce the mods bias, deliberately or not
I’m not sure what to do here. On my mobile device the compacted media bias fact check post still takes up 50% of my phone screen.
How a post tag if we have a tagging system in Lemmy, instead of a whole long comment?
Maybe the bot could just post a one line summary with a link to more information?
Thanks for the feedback. Can you elaborate a bit about the 50% of your screen thing? Is it the text itself, or is the issue that the app provides links at the bottom of the comment? I’m thinking of my experience on Voyager, where the links are summarized at the bottom of each comment, which does lead to a decent amount of screen being taken up. Would it be better if there weren’t any links?
yep I’m using Voyager on my iPhone. Maybe a super short summary without links. People could open the bot’s profile and look at the bot’s posts (not comments) if they want to dig deeper to understand a source.
Interesting, so you think the bot should make posts too? Like, a post for each source with a summary of relevant info? Just making sure I understand what you mean
Yeah. It’s an idea for a way to create a user repository within Lemmy that could be edited by the bot as needed. I’m sure there are better ways.
I’m frankly rather concerned about the idea of crowdsourcing or voting on “reliability”, because - let’s be honest here - Lemmy’s population can have highly skewed perspectives on what constitutes “accurate”, “unbiased”, or “reliable” reporting of events. I’m concerned that opening this to influence by users’ preconceived notions would result in a reinforced echo chamber, where only sources which already agree with their perspectives are listed as “accurate”. It’d effectively turning this into a bias bot rather than a bias fact checking bot.
Aggregating from a number of rigorous, widely-accepted, and outside sources would seem to be a more suitable solution, although I can’t comment on how much programming it would take to produce an aggregate result. Perhaps just briefly listing results from a number of fact checkers?
I second this. This community is better than most social media, but it’s still that, and social media popularity is pretty bottom of the barrel as a means of determining accuracy. Additionally, that’d just open it up to abuse from people trying to weight the votes with fake accounts, scripts, whatever.
That’s fair. One idea could be a separate “community rating” and one that is more professional. Think Metacritic, RottenTomatoes, etc
I appreciate the bot. I like to play a game of “guess what the bot will say” before checking. I usually win, but it’s cool to have.
- Please, move the bias and reliability outside of the first accordion/spoiler.
- While I personally don’t see the point of the controversy, it wouldn’t be too hard to manually enter Wikipedia’s Perennial Sources list into the database that the bot references, especially with MediaWiki’s watchlist RSS feed.
- Open source the database and the bot. Combined with #2, this could also offer an API to query Wikipedia’s RSP for everyone to use in the spirit of fedi and decentralization.
For example, if we retain MBFC, the layout could look something like this:
Rolling Stone Bias: Left, Credibility: High, Factual Reporting: High - United States of America
MBFC report | bot support | Search topics on Ground.News
in which “Rolling Stone” is linked to the Wikipedia article.
With RSP, it could look something like this:
Rolling Stone is generally reliable on culture
There is consensus that Rolling Stone has generally reliable coverage on culture matters (i.e., films, music, entertainment, etc.). Rolling Stone’s opinion pieces and reviews, as well as any contentious statements regarding living persons, should only be used with attribution. The publication’s capsule reviews deserve less weight than their full-length reviews, as they are subject to a lower standard of fact-checking. See also Rolling Stone (politics and society), 2011–present, Rolling Stone (Culture Council).
Rolling stone is generally unreliable on politics and society, 2011–present
According to a 2021 RfC discussion, there is unanimous consensus among editors that Rolling Stone is generally unreliable for politically and societally sensitive issues reported since 2011 (inclusive), though it must be borne in mind that this date is an estimate and not a definitive cutoff, as the deterioration of journalistic practices happened gradually. Some editors have said that low-quality reporting also appeared in some preceding years, but a specific date after which the articles are considered generally unreliable has not been proposed. Previous consensus was that Rolling Stone was generally reliable for political and societal topics before 2011. Most editors say that Rolling Stone is a partisan source in the field of politics, and that their statements in this field should be attributed. Moreover, medical or scientific claims should not be sourced to the publication.
RSP listing | bot support | Search topics on Ground.News
Both examples with everything necessary linked, of course
We are looking at a composite rating, of many sources.
Would you open source that on launch? What about the nuances of reporting from the same source on different topics?
The distance will be open source day one. As to the other question: still in the planning phase
Seconded. #1 is why I currently have it blocked.
For 3 they said they’d release the code when it was announced, but have been completely silent since. Maybe it’ll be public when sublinks goes live lol
- Open source the database and the bot.
Yes. A certain amount of my complaint about MBFC bot is not that it’s a bad idea per se, it’s just that the database and categorizations are laughably bad. It puts Al Jazeera in the same factual classification as TASS. It lists MSNBC as factually questionable and then when you look at the actual list, a lot of them are MSNBC getting it right and MBFC getting it wrong. It might as well be retitled “The New York Times’s Awful Neoliberal Idea of Reality Check Bot”. (And not talking about the biases ranking – if that one is skewed it is fine, but they claim things are not factual if they don’t match the appropriate bias, and the bias is unapologetic center-right.)
You can’t set yourself up to sit in judgement of sources that write dozens of articles every single day about unfolding world events where the “objectively right” perspective isn’t always even obvious in hindsight, and then totally half-ass the job of getting your basic facts straight about the sources you’re ranking, and expect people to take you seriously. I feel like mostly the Lemmy hivemind is leaps and bounds ahead of MBFC bot at determining which sources are worth listening to.
- it wouldn’t be too hard to manually enter Wikipedia’s Perennial Sources list into the database that the bot references
FUCK FUCK FUCK YES
This is an actual up-to-date and very extensive list that people who care bother to keep up to date in detail (even making distinctions like "hey this source is ok for most topics but they are biased when talking about X, Y, Z). This would immediately do away with like 50% of my complaint about MBFC bot.
I wish the comment count on a post didn’t include MBFC (or maybe bots in general).
I’ll be honest, that’s probably outside of the scope of what we can do for now. It’s definitely valuable feedback in general and I wish I could offer some kind of solution but that’s probably even outside the control of the instance admins.
Someone can feel free to correct me if I’m wrong!
On this topic the information would probably be ideally delivered by flairs/post tags which lemmy doesn’t support yet (AFAICT).
Simply having (bias:left) (factuality: high) would be much better than a whole comment.
Not directly related to MBFC bot, but what’s your opinion on other moderation ideas to improve the nature of the discussion? Something Awful forums have strawmanning as a bannable offense. If someone says X, and you say they said Y which is clearly different from X, you can get a temp ban. It works well enough that they charge a not-tiny amount of money to participate and they’ve had a thriving community for longer than more existing social media has been alive. They’re absolutely ruthless about someone who’s being tricksy or pointlessly hostile with their argumentation style simply isn’t allowed to participate.
I’m not trying to make more work for the moderators. I recognize that side of it… the whole:
This bot was introduced because modding can be pretty tough work at times and we are all just volunteers with regular lives. It has been helpful and we would like to keep it around in one form or another.
… makes perfect sense to me. I get the idea of mass-banning sources to get rid of a certain type of bad faith post, and doing it with automation so that it doesn’t create more work for the moderators. But to me, things like:
- Blatant strawmanning
- Saying something very specific and factual (e.g. food inflation is 200%; someone told me that today) and then making no effort to back it up, just, that’s some shit that came into my head and so I felt like saying it and now that I’ve cluttered up the discussion with it byeeeeee
… do a lot more damage than just simple rudeness, or posting something from rt.com or whatever so-blatant-that-MBFC-is-useful type propaganda.
Some of what you describe is likely against our community rules. We do not allow trolling, and we do not allow misinformation. We tend to err on the side of allowing speech when it is unclear, but repeat offenders are banned.
When you see these behaviors, please make a report that that we can review it. We cannot possibly see everything.
It’s tricky because we could probably make 100 rules if we wanted to define every specific type of violation. But a lot of what you’re talking about could fall under Rules 1 and 8, which deal with civility and misinformation. If people are engaging in bad faith, feel free to report them and we’ll investigate.
Hm
I can try it – I generally don’t do reports; I actually don’t even know if reports from mbin will go over properly to Lemmy.
For me it’s more of a vibe than a set of 100 specific rules. The moderation on political Lemmy feels to me like “you have to be nice to people, but you can argue maliciously or be dishonest if you want, that’s all good.” Maybe I am wrong in that though. I would definitely prefer that the vibe be “you can be kind of a jerk, but you need to be honest about where you’re coming from and argue in good faith, and we’ll be vigorous about keeping you out if you’re not.” But maybe it’s fair to ask that I try to file some reports under that philosophy before I assume that they wouldn’t be acted on.
Just kill it with fire.
Indeed. Thermite is usually the recommended method to erase hard drives you don’t want to be recoverable.
Tell the bot to never be the first comment. I find it very frustrating when I see “a comment on this post” and it’s just the bot. I’m here to read what people have to say so it is very annoying when I think someone said something and it’s just the bot.
There was even a front page meme about this last year, but with another noisy bot. Lemmy doesn’t bury new comments like Reddit does, so there’s no real penalty to making the bot wait.
Just to clarify, are you saying the bot should be triggered after someone else comments on a post?
Yeah. Maybe after a few tbh. Save on API calls
I hate that I have to expand the section to see the rating. If that could be fixed, it’d be better.
Please God no
We could also open a path for users to vote, so that any rating would reflect our instance’s opinions of a source.
If you’d program something, perhaps this should be the start.
I noticed you got a couple downvotes, so this comment is more for the voters: if you have thoughts on this, please comment them so we can understand why you feel the way you do.
BTW, I really like the suggestion to just get rid of the bias rating from the bot’s comments. That should be a lot less work than implementing a bias crowdsourcing system. Given limited volunteer time and all that.
