Wikipedia:Village pump (WMF)

Slow Editing Towards Equity

I will start with one that caught my eye:

• Project page: meta:Research:Slow Editing Towards Equity
• Amount Spent: $42,356.25 USD
• Result: meta:Research:Slow Editing Towards Equity#Results
• Promised result: meta:Grants:Programs/Wikimedia Research Fund/Slow Editing towards Equity#Impact
• Related: (that last one makes some interesting claims about Wikipedia)

Was this $42K well spent? Who benefited? What was accomplished?

Was this the sort of thing that the donation banners describe your contributions as funding?

Why are some of the links at dead or useless? Are we not capable of posting these publications on WMF servers? --Guy Macon (talk) 18:38, 28 September 2026 (UTC)

Takeaways from the linked paper at [14]:

Theory Section

- The power to shape consensus on Wikipedia is not evenly distributed. Even aside from groups like ArbCom, things like technical knowledge, access to bots, and understanding of the site's jargon gives the small slice of the community with mastery over them ("experienced users") an overwhelming presence in policy discussions.

- The rules pages act as entrenched technical authority that can be used by experienced users against outsiders, not merely as consensus records.

- As proof of the above, most policy pages are very old. If a page survives its first year of official status, it gains enough momentum to be effectively permanent. 11 of the 15 sampled policy pages were fully stabilized by 2011.

- The rules pages are very stable, and that's good. However, they are resistant to change in a way that discourages any new user that isn't already aligned with Wikipedia's way of thinking. Such users don't become experienced users with the ability to shape discussion, and so can't reform the very rules preventing their participation. This is a barrier to gaining new editors from outside the spheres Wikipedia originated in (American male tech nerds).

- Wikipedia's policies, both content and social, can be used as a cudgel against users who threaten local consensus and/or individual editors' worldviews. Examples cited.

- All of the above to say, Wikipedia's current policies fail to reflect actual community consensus due to how they empower users who already agree with them.

There's more stuff about the actual form said empowerment takes, lot of sociology jargon and such. It records the various levels of authority a rules page can have, including unofficial levels like "essay endorsed by popular community members."

Data Section

- The number of individual users involved in creating each policy is very low. It's mostly the same handful of people talking to each other. There were more "critical discursive moments" identified in the sample than unique editors.

- enwiki prefers discussion among a few people with strong opinons and arguments over the flatter voting system of eswiki. The other three languages didn't have much of either, and just kind of trusted whoever was writing the rules.

- Recommendation made that the processes used to define and change policy be adjusted to better reflect actual editor consensus rather than consensus among the technocratically empowered few. No hard suggestion included.

Overall I think it's a useful study, if not for Wikipedia itself then certainly for the history and sociology of internet communities. I'm not mad about it having been funded. ~2026-52337-89 (talk) 23:56, 28 September 2026 (UTC)

The promise (when they were asking for $42K) was "identifying gaps in policy that are necessary for the needs of all English Wikipedians." Please name a gap in policy that was identified, and how we can fix that gap.

And again I ask, Was this the sort of thing that the donation banners describe your contributions as funding? See The Huge Fight Behind Those Pop-Up Fundraising Banners on Wikipedia in Slate --Guy Macon (talk) 02:14, 29 September 2026 (UTC)

Thinking on how to resolve this, perhaps that's the solution - we establish a consensus that donations banners run on enwiki must accurately detail where the money is going. If something is beyond the scope of what donation banners said the WMF is spending money on, then either the WMF needs to cut the program, or not run banners on enwiki. BilledMammal (talk) 02:19, 29 September 2026 (UTC)

@Guy Macon, I don't think expecting research to "succeed" in doing something 100% of the time is a reasonable expectation to have. Research can and does "fail" even with the best effort. Doing research on policies on Wikipedia is within Wikimedia Foundation's overarching programmatic goals to better the community to which donators donate to. There was a certain amount of funding allocated to the researchers, they tried to gain some useful information, they failed in the overall goal but found some interesting findings about internet communities which got published in a (what I assume) is a well respected journal. Sohom (talk) 02:55, 29 September 2026 (UTC)

No need to ping me. I have a watchlist and know how to use it.

Useful? Sure. $42,356 USD of the donor's contributions useful? Not so much. A lot of those donors are living in poverty and sacrifice to give because they were told that this was needed to keep Wikipedia free.

There are many, many researchers who have published papers on Wikipedia in well respected journals. I question why only a handful of insiders get funding for doing that from the WMF. --Guy Macon (talk) 03:06, 29 September 2026 (UTC)

seful? Sure. $42,356 USD of the donor's contributions useful?, that's how research works??? You provide a chunk of funding so that a group of researchers can try different methodologies and techniques and build prototypes to see if any work. If anything, 42K is fairly cheap in the research world for the length of time the project took (2022-2025) actually since if you think about it, a single grad student's yearly tuition and stipend in CS (being paid barely enough to sustain themselves, while also being one of the most funding rich areas) amounts to roughly that number.

I question why only a handful of insiders get funding for doing that from the WMF.. Guy Macon, I would suggest looking at the page for the dissemination of research funds which provides full transparency into who is allowed to put in proposals, how much money WMF is willing to give for research projects, the proposals submitted and rejected, the folks who decide who to fund (who are well respected researchers within the Wikimedia Research community). I don't think characterizing this as "insiders get funding" is a correct characterization. Sohom (talk) 03:22, 29 September 2026 (UTC)

As I have said many times, I think that everyone at the WMF -- from engineers to the people who decide who and what to to fund, are doing a great job at what they are trying to do. I have zero doubt that you are really looking hard at who you decide to pay to do research, and by "insiders" I don't mean "your buddies" or anything implying corruption or malfeasance. I believe you are like the fine people back in the 1960s who were managing the war in Vietnam. They did seriously great work deciding whether to fund more jeeps or more tanks, whether the jungle boots could be made better, etc. Thousands of good decisions were being made by dedicated individuals doing a great job, but never once considering or even being allowed to consider whether we should be fighting a war in Vietnam at all.

Now consider the entire world population of people who have published research on Wikipedia. All of them. They all get paid by someone, usually their university. What percentage get WMF grants? Do an honest estimate. One in a thousand? What differentiates the two groups? One group knows how to craft a proposal that will get them funding in the usual ways academics get funding. The other has learned how to work the system to get WMF funding. How many of each group get free travel and lodging to go to Paris and spend time with Wikimedia staff and Wikipedia editors? How many in each group know the names of multiple WMF employees? How can you call the tiny minority that get grants anything other than insiders? --Guy Macon (talk) 06:47, 29 September 2026 (UTC)

And again I ask, was this the sort of thing that the donation banners describe your contributions as funding? --Guy Macon (talk) 06:47, 29 September 2026 (UTC)

There's lots of things the WMF is doing badly, but I don't think this particular funding decision is one of them. The article seems to be of good quality to me, and seems in line with the research questions in the application. TietoTeekkari (talk) 07:56, 29 September 2026 (UTC)

And I fully agree, just as I agree that the military made some great decisions about funding jungle boots during the Vietnam war. They were great boots. I still have a pair and they look like new. And, just as I think that same military made those great boot decisions without even considering whether we should be there we should be fighting a war is Vietnam, I think the WMF did a fine job of picking which researcher to fund while never even considering whether we should be funding academic papers at all. And again I ask, was this the sort of thing that the donation banners describe your contributions as funding? Is there some special feature in Wikipedia's software that makes the preceding words invisible when I write them? --Guy Macon (talk) 13:50, 29 September 2026 (UTC)

Not Tieto, but the last time I donated to Wikipedia I recall the banner saying something about "supporting and defending open knowledge projects." I'm probably not getting the phrasing exactly right. This research seems very relevant to Wikipedia and how our internal community and governance systems operate, along with how community-based open knowledge projects operate more broadly. ThadeusOfNazereth(he/him)^{Talk to Me!} 19:17, 29 September 2026 (UTC)

You mean this banner? The actual promise was "Please join the 2% of readers who give what they can to to help keep this valuable resource ad-free, up-to-date, and available for all". Nothing about supporting and defending open knowledge projects. The banner also talks about what you are doing when you visit Wikipedia, which I thought was a nice addition. Please note that something can be "very relevant to Wikipedia" without being what the donors were told their donations were funding.

You can see a bunch of fundraising banners here. I couldn't find any documentation as to which were shown when and to who. Perhaps someone else can find that information. --Guy Macon (talk) 23:45, 29 September 2026 (UTC)

Guy, it sounds to me that you want the WMF to support Wikimedia and open knowledge projects but to do no research into how to support those projects, no research into how effective support for those projects is, whether support would be better directed towards maintaining existing (governance) structures or implementing different (governance) structures? If so, why do you think that? If not, please try explaining what I'm getting wrong. Thryduulf (talk) 19:47, 29 September 2026 (UTC)

Not at all. The choice is not between "funding no research" and believing without evidence that this particular research actually ended up "maintaining/implementing governance structures". Can you name a single governance structure that was maintained or implemented by this research? Look at the results of this research again: The first two results are trivial and the third is just plain wrong.

In the banner at the donors -- many of whom live in poverty -- were not told that they were funding research, just as they were never told that they were funding trips to talk about Wikipedia in exotic vacation destinations. They were told that they were keeping Wikipedia ad-free, up-to-date, and available for all. I am fine with donors funding any research that has a reasonable chance of advancing those goals, but they should be told that some of their donations will be given to other individuals and organizations. --Guy Macon (talk) 23:45, 29 September 2026 (UTC)

Research does not have to be successful to have been valuable, and if you only fund research that produces the results that you want it to then that's not research. Please explain how funding people and projects that write and maintain the projects, write and maintain the sources we use, research how the project can be improved, research how best to support the projects, etc. are not "keeping Wikipedia ad-free, up-to-date and available for all" either directly or indirectly? Thryduulf (talk) 00:04, 30 September 2026 (UTC)

By those criteria I can't think of a single thing that the WMF spends money on that doesn't keep Wikipedia ad-free, up-to-date and available for all either directly or indirectly. I can't see any plausible way that not funding this sort of thing could possibly result in Wikipedia being ad supported, stop the volunteers from updating it, or make it unavailable, but you clearly do so I won't waste your time with further disagreement. --Guy Macon (talk) 00:18, 30 September 2026 (UTC)

Can you name a single governance structure that was maintained or implemented by this research? To this among other research appears to be part of a body of research into policies that eventually led to the establishment of the NPOV Working Group and the subsequent implementation of the global baseline NPOV policy. Sohom (talk) 00:27, 30 September 2026 (UTC)

Just to be clear: I think that Wikiresearch is in general a good thing, and funding some of this by WMF seems reasonable. The specifics of funding, methods, and transparency and ethics of the coordination of the research should, of course, be subject to transparent debate (apart from privacy issues). I'm not judging the validity of funding for this particular project. Boud (talk) 11:39, 30 September 2026 (UTC)

I found meta:Baseline NPOV policy (a baseline NPOV policy for the Wikipedias which do not yet have an NPOV policy) to be a fine document. The question is whether it is worth the $42,356.25 USD ($88.43 per word) it cost the donors.

I would be interested in how many Wikipedias did not yet have an NPOV policy when it was created and how many have adopted it. If the donor is paying for "governance structures implemented by this research" it would seem reasonable to ask how many actual governance structures were implemented, not just a web page that seems like it would be useful to someone doing that.

I also noticed that it is only in English and French. One would think that some of that $42,356.25 USD would have been spent on translating it to the languages of the Wikipedias which do not yet have an NPOV policy. I'm just saying. --Guy Macon (talk) 16:14, 30 September 2026 (UTC)

Couldn't small wikis just copy the NPOV policy from English, French or any other well-scrutinised Wikipedia, gratis, and alter any bits they deem inappropriate for their project? Certes (talk) 16:23, 30 September 2026 (UTC)

@Certes, I would suggest looking at Serbian, Bosnian Wikipedia situations where despite copying the policy text over over they've had multiple significant issues surrounding the enforcement of said policies. Similar issues exists to smaller extent on other projects as well. Even on the rather bare-bones baseline policy, smaller communities have pushed back on the baseline policy due to concerns over the fact that it often constrains smaller language wikis in multiple ways in terms of prioritizing spoken history vs colonial written sources or preferring accounts in local language sources over the broader international consensus on a topic. Sohom (talk) 16:38, 30 September 2026 (UTC)

Certes, the real problem in searching for worldwide solutions is that in my experience different language Wikis are effectively different planets with totally different climates. I sometimes look at the Italian, French, German and Spanish wikis, and their culture and customs are strikingly different, and range from those with a highly controlled system (eg German) to others with freewheeling on non-crucial topics. Their wiki cultures adre as different as the foods they eat. You can not give them a universal cookbook. Yesterday, all my dreams... (talk) 18:02, 30 September 2026 (UTC)

@Guy Macon The policy hasn't yet gone into effect. It is still in proposal phase. Also re cost, 42K (and more actually) spent on building policy to avoiding incidents where Wikimedia project's credibility is called into question is imo money fairly well spent. Sohom (talk) 16:40, 30 September 2026 (UTC)

That would indeed be money well spent if that's what the $42K bought. A bargain, I would say. The problem is that you have offered no examples where having One More Web Page On Meta actually avoided any incident where any Wikimedia project's credibility was called into question. Not even a plausible path where that might happen in the future. You seem to be saying that money spent on goals without any hint of a way to actually accomplish those goals is money well spent. I say that $88.43 per word is way above the going price for good intentions, which are currently so cheap that they pave roads with them. --Guy Macon (talk) 22:34, 30 September 2026 (UTC)

The problem is that you have offered no examples where having One More Web Page On Meta actually avoided any incident where any Wikimedia project's credibility was called into question - The m:CheckUser policy and the m:UCoC would be prominent examples of the a "page on meta fixes problems" effect. Sohom (talk) 00:21, 1 October 2026 (UTC)

Please accept my apologies in advance for saying this, but with phrasing like "Wikimedia Foundation's overarching programmatic goals to better the community" you could run for senate. The problem with their research results was similar. Too many words, too little substance. Sorry, but bluntness was needed here. Yesterday, all my dreams... (talk) 15:07, 30 September 2026 (UTC)

Foundation's overall goal of making the wiki community better is what I meant. Ignore the word programmatic, it's a reference to the way WMF and other non-profits calculates it's budget, (as programmatic expenses and non programmatic expenses). Sohom (talk) 16:28, 30 September 2026 (UTC)

Guy, I think $42,355.25 was wasted on the word salad they produced. The other $1 was useful, given that it made me laugh. Yesterday, all my dreams... (talk) 14:56, 30 September 2026 (UTC)

In this edit, Guy Macon wrote, The first two results are trivial and the third is just plain wrong. I think it would be good if Textaural, who is an experienced enwiki editor, could respond here. The peer-reviewed research paper is technically a WP:RS, so usable in enwiki articles, but peer review does not guarantee correctness.

I agree that the third key result The status of rules [on these 5 Wikipedias is] established largely by individual decisions and rarely through collective decision making is extremely dubious for at least enwiki and frwiki. The strings !vote and not vote seem to be completely absent from the published paper. In a nutshell: If I'm the individual to first write the enwiki policy that 1+1=2 is a fact and nobody contests that, it's not due to my dictatorial authority; it's due to the WP:!VOTE decision-making algorithm and the nature of 1+1=2 being widely considered as reasonable, without anybody needing to argue about it (leaving aside modern foundations of mathematics). The whole point of not holding a vote on my hypothetical new policy is that getting 10,000 votes (not !votes) with a likely result of 99.9% in favour would be a huge waste of time. @Textaural: How do you justify your extraordinary conclusion that Within the English-language rules, the authority of rules was based on the editorial authority of one user, who may have found support through limited deliberation without having studied the role of WP:!VOTEs in the establishment of enwiki rules? !VOTEs are a form consensus decision-making, which is a form of collective decision-making. The authority of rules is not based on the editorial authority of one user; it's based (often) on the ability of one user to successfully describe and summarise the current consensus or predict the likely consensus.

Textaural: do you have any evidence for this claim that you and your authors assert in the paper? The Methods section of your paper says nothing about how you quantify the unexpressed non-objections to enwiki rules (a possible method would be to obtain permission to do a survey of Wikipedians to see how much they agree with existing rules and whether or not they have tried objecting to rules they disagree with; but you don't seem to have done that). Boud (talk) 11:07, 30 September 2026 (UTC)

"Rules" !== policies to my understanding. Sohom (talk) 11:23, 30 September 2026 (UTC)

The author gave examples of what they consider to be rules. Two of the examples are in English; Wikipedia:Proposed deletion (a policy) and Wikipedia:Disruptive editing (a behavioral guideline.). --Guy Macon (talk) 12:37, 30 September 2026 (UTC)

Fair enough, my understanding is that the paper discusses the individual rules inside the policies instead of the policy as a whole but I can see it being taken in both ways Sohom (talk) 14:04, 30 September 2026 (UTC)

I also noticed that the author uses https://artandfeminism.org/resources/research/unreliable-guidelines/ as a source. Artandfeminism.org appears to be funded by the WMF through WikiCed ( https://www.wikicred.org/). I can't find any record of how much the WMF gave artandfeminism.org to say "This research project identifies the ways that organizational values and processes around reliability and the reliable source guidelines are implicated in maintaining hierarchies and excluding marginalized knowledges and communities" and "Ultimately, the Reliable Source guidelines are an unreliable and incomplete guide to provide editors with a meaningful understanding of how to assess source reliability".

So what are we supposed to replace our reliable source guidelines with? "Triangulation". If anyone can understand what (cited by artandfeminism.org) is talking about please explain it to me. --Guy Macon (talk) 13:16, 30 September 2026 (UTC)

I don't disagree that Art+Feminism is funded to a certain extent by the Foundation, but last I checked, it wasn't through WikiCred? WikiCred organizes meetups in the West Coast and builds tools for media reliability on Wikipedia, Art+Feminism advocates for better coverage of topics related to gender, feminism and art (among other things). They were one of the advocates for a global UCoC and and have long advocated for changing our guidelines to be more inclusive towards marginalized communities and better enforcement of civility policies. I don't think there is a correlation between WMF funding Art+Feminism, Art+Feminism being cited as community criticism of the reliability guidelines and WikiCred organizing conferences. Sohom (talk) 16:25, 30 September 2026 (UTC)

Re: "last I checked, it wasn't through WikiCred?" you need to do a better job of checking. Look at Read the funding section. What we have here is an informal community of organizations united in the goal of writing proposals that will result in the WMF giving them grant money. If only the WMF would check to see if the goals in the proposal were actually met and, if not, find some other applicant to award the next round of grants to. --Guy Macon (talk) 23:00, 30 September 2026 (UTC)

To quote the paper, Unreliable Guidelines, a project of Reading Together: Reliability and Multilingual Global Communities, is an inaugural Art+Feminism research report partially funded by WikiCred, which supports research, software projects and Wikimedia events about information reliability and credibility. WikiCred and Art+Feminism are together funding the paper with At+Feminism being the primary sponsor and WikiCred being a partial sponsor. The Funding section in particular say The expertise of the main researchers was partially funded by WikiCred and Art+Feminism. i.e. that this project was a collaboration between AF and WikiCred. Nowhere does it say that the Art+Feminism is funded by WikiCred. Sohom (talk) 00:27, 1 October 2026 (UTC)

When I skimmed that paper yesterday, I also had several qualms with the methodology leading to those conclusions: off the top of my head, they ignored pages with a creation date before 2005; they only looked at edits which change the label (between essay, guideline, policy, etc.); and they only looked talk pages when explicitly mentioned in an edit summary.

To expound a little: if I'm reading it correctly, it would ignore (to list a few non-exhaustive examples), talk discussions not mentioned in the edit summary; discussions which affirm the status quo; changes which align content to community consensus without changing the "label" of a page; and implicit consensus where thousands of people have read a page and no one has found reason to change it. LittlePuppers (talk) 21:56, 30 September 2026 (UTC)

Well hello there! I've been meaning to respond for a while and just got the chance to do so and thanks for notifying me that this conversation was happening. I'm replying to @Boud here, but it is more of a general response to the discussion.

First, the goal as a researcher is to have your research read and discussed and if for anything else, I am grateful for that happening here. And I appreciate those of you who have defended my work (and "research" in general), but also Guy who is asking some hard questions that should be answered. I will break down my response to answer three questions: Why did it cost 42K to produce one paper? Is the research worth it to those who donate to Wikipedia? Are the results of this research interesting, valid, and valuable to the community? Textaural (talk) 13:13, 19 October 2026 (UTC)

Why did it cost 42K to produce one paper?

The most direct answer for this is that the university that manages the money also sets the pay scale for research time, and so that cost is actually somewhat outside of the individual grantee's hands. For some background, The WMF was interested in ensuring that the project I conducted covered as many possible language editions as possible because there is so little research done on policy itself, let alone Arabic, Spanish, and Dutch policies. To do this, I also needed to hire more researchers to do this work, and two of the researchers were multilingual (which did helped to limit the number of people involved). But to meet those requests by the WMF and to draw from the university's labour pool I did have to hire two post-docs.

In the end, a significant proportion of money went to paying all the researchers for their labour in accordance with university's own pay scale for research time. The secondary cost, but far far less than the cost of labour, was travel to one conference. We attended other conferences that we attended, but these were online conferences which incurred no extra cost for the grant. We also published with New Media & Society which is a top ranking journal in the field, but importantly, has an agreement with the university where the costs of publishing open access is covered by the university, and not individual researchers. This meant that the open access publishing cost (which can be about $4000-$5000 depending on the journal) was paid for by the university and itself, and not the WMF. As surprising as this may seem, in comparison to other research projects, my project was quite frugal.

That being said, the project was supposed to last only one year, I extended it for a second year because the research required it (but with no increase in grant money) and I ended up working many hours on it in my free time into a third year. So in fact, this project would have cost more if the entire research time was paid for. So while we are accustomed to thinking of knowledge as free here on Wikipedia, research itself in general is quite expensive.

So that is some of the details on the cost. Textaural (talk) 13:13, 19 October 2026 (UTC)

Is the research worth it to those who donate to Wikipedia?

I understand the reasoning of this kind of question, and I have two considerations of it. The first is that the WMF funds do not just come from the community. From the 2024–2025 Annual Report, 22 corporations (Google, Microsoft, Rockafeller, etc) and private citizens donated _at least_ over 50,000 each. A conservative estimate puts this group of major donors donating a minimum of 1.1 million. 47+ donated 15K-49K; another 108+ donated 5K-14K. This is missing sustaining donations and one-time donations. So a very conservative calculation is that at least 2.3 million of the donations in 2024-2025 came from organizations or wealthy individuals. What I'm trying to say is that Wikimedia's funding is diverse. So perhaps, if you are less keen on research like mine, the consolation prize is that we can consider that Google paid for it, not Wikipedians. But I obviously think this research does has value (see final question).

One thing I'd like to add is that the WMF continually highlights in the grant process that they expect researchers to connect with the community during the research phase. On this front I'll admit that I should have done better. But I was also a bit unsure at the time exactly how to go about it and where is the right place to notify Wikipedians that I was historically analyzing policy and what kind of feedback the research required. Despite my uncertainty, I should have tired. I am now more aware of how research can be communicated, and I at least know who to ask now when I start my next project. But any further advice would be welcome. Textaural (talk) 13:13, 19 October 2026 (UTC)

Are the results of this research interesting, valid, and valuable to the community?

First, this research is historical in nature, looking at how policies first emerged and gained their authoritative status in the form of policy templates, and so the immediate outcome for the community is to gain an additional view on how the encyclopedia developed. Second, one critique was that some of the results were trivial, and one was wrong. The two "trivial" conclusions are (at least from my perspective) descriptive conclusions, they are facts. Making facts that are rooted in empirical evidence is difficult, time-consuming and painstaking work, something that all Wikipedians are aware of. So now there is a reliable source for these facts.

It was also pointed out that this research was framed on its meta page as wanting to find gaps in the policies. On this front, what we identified is that a the explicit authority of policies (as it is advertised in the policy template at the top of the page: "This is a policy") was itself often not created out of a consensus process, but was created through the decisions of a very few users. In the language of gaps, I see our research as indicating the ways that some things that we think are made through consensus on Wikipedia, are assumed to be, when in fact a close inspection shows this not to be the case. As such, our article makes a small but substantial argument: that here, in the specific cases of how early policies were tagged as policy, guidelines, etc, was not itself derived through a process of consensus. This does not mean other parts of a rule were not created through consensus, just this specific part that tells everyone on Wikipedia which wikipages are policies and which ones are not.

The evidence was based on looking at 15 rule pages, and examined their development (usually the late 2000s; early 2010s), tracking who said and did what around the times when the status of a policy changed. We have a clear We found that some fundamental policies were given the authority of a "policy" adding the policy template, rather than a coherent and explicit deliberation of many users. As such, I argue that this does not qualify as consensus as broadly imagined. Indeed, trying to understand what consensus is and how it manifests on Wikipedia is an interesting question that is a focus of my research (here and here and here) and this paper extends these ideas.

So why does it matter? There are cases where the process where a rule gains its authority as a policy is not in keeping with the ethos of consensus. The application of a template of "This is a policy" is self-reinforcing. In other words, the article asks that we actual examine how the authority of these policies were designated in the first place, rather than taking it for granted: something that I think is important to consider for all policies. And I hope leads to making Wikipedia get even closer to its ideals.

And if you are more curious about the argument, I encourage you to read it, and if you have questions about the empirical side of the argument, the dataset is also available. Textaural (talk) 13:14, 19 October 2026 (UTC)

Hi Textaural, thanks for the response. I'm curious how you would respond to my initial conclusions from skimming that paper: basically, that (as far as I could tell from reading your methodology) only looking at changes specifically to the page and those directly referenced in edit summaries ignores many forms of consensus-building. For example, a talk page discussion that was not mentioned in an edit summary; community discussion at some other venue (e.g. WP:VPP); a discussion which affirms the current status (thus strengthening consensus but not leaving anything in the page history); etc.

And regardless, I think it's also worth considering how changes to a policy over time bring it more in line with consensus; in short, perhaps back in the early 2000s (well, late 2000s, as policies from the early 2000s were excluded from your study) there were sometimes these nigh-unilateral status changes, but (and I'm not trying to put words in your mouth) I don't see a compelling argument that, after hundreds of discussions about and tweaks to these policies and guidelines, and thousands of people referring to and using them, they don't reflect something near to consensus now.

(Sorry, I know that's not particularly eloquent prose, but hopefully it's comprehensible. I'm just trying to understand your research better and what it does and does not cover.) LittlePuppers (talk) 21:11, 19 October 2026 (UTC)

On the point: The two "trivial" conclusions are (at least from [Textaural's] perspective) descriptive conclusions, they are [factual claims]. [Rooting factual claims] in empirical evidence [for or against them] is difficult, time-consuming and painstaking work, something that all Wikipedians are aware of. So now there is a reliable source for these facts, I fully agree (though I've reworded a bit more carefully from a WP:WIKILAWYERING POV). Boud (talk) 12:32, 22 October 2026 (UTC)

@Textaural: Glad that you've responded. However, you have not responded in plain language to the basic concern about your third claimed result: that you have not measured the full participation in the rule making used in the standard Wikipedia consensus decision-making algorithm and you have nevertheless made strong claims that the rule making was highly non-participatory, i.e. your published claim is not supported by your evidence, and you don't yet seem to have acknowledged that. Per Rhododendrites' suggestion below, I've started a section on metawiki on the talk page describing your article. I propose that people interested continue the discussion over there. I already pinged two people over at meta, I'm not sure if Sohom Datta wants to follow that discussion too. And, of course, there are the non-commenting implicit participants in this discussion who are leaving no trace in the wikirecord: those who feel that the things already stated are sufficient and feel that their overt participation would make the thread more messy rather than clearer. I cannot ping those implicit participants. Boud (talk) 13:34, 22 October 2026 (UTC)

Can I suggest separating, to the extent possible, the funding question from an evaluation of the paper? The WMF does not read the outcome of a project before funding the project, of course, so really it's a question of "should the WMF invest in research about Wikipedia governance" not "should the WMF invest in this paper". Yes, "was this worth it" is a fine question to ask, and one that the WMF answers internally, too, but any evaluation of the funding question should be based not on "I found one I'm skeptical of" but on a full accounting of related projects. I haven't engaged deeply with this paper yet (open in a tab in my "to read" window for some time, sadly), but the project sounds worth doing to me. We have precious few studies that examine Wikipedia governance anywhere but the English Wikipedia, and a direct comparison between several language editions is very potentially valuable. And NM&S is easily one of the top journals for non-computational Wikipedia research these days. That doesn't mean everything it publishes is perfect, but it has a high rejection rate, intense peer review process, and deep bench of Wiki-knowledgeable reviewers. That said, as with any Wikipedia research, it's also valuable when volunteers evaluate the content and correct the record, if appropriate. A quick search through the signpost archives doesn't turn up a match, so maybe it's a good fit for the next Research Report. — Rhododendrites ^talk \\ 16:16, 30 September 2026 (UTC)

I couldn't find an easy way to view multiple grants to the same researcher, but The WMF does read the outcome of a project before funding the next project by the same researcher. --Guy Macon (talk) 10:28, 6 October 2026 (UTC)

On a couple of occasions Guy Macon has said A lot of those donors are living in poverty and sacrifice to give because they were told that this was needed to keep Wikipedia free. I wonder what evidence supports this claim. I find it far fetched. Johnjbarton (talk) 15:33, 22 October 2026 (UTC)

I will be happy to support my claims with citations to reliable sources, but I need to know exactly what you are objecting to.

Are you claiming that the WMF doesn't send banner adds that get read by people living in poverty in Appalachia, India, and western Africa?

Are you claiming that the WMF doesn't clam in those banner ads that donations are needed keep Wikipedia free?

Are you claiming that the banner ads are totally ineffective and do not result in donations?

Are you claiming that for some reason only people above the poverty line respond to banner ads by donating?

Something else?

Please let me know exactly what you are disputing so I can post evidence to support my claims. --Guy Macon (talk) 16:15, 22 October 2026 (UTC)

I am challenging you to support your claim that a significant number of donors suffer hardship because of WMF actions. I do not think you polled donors living below the poverty line and determined that they were deceived. I do not think you represent a harmed class. Your critique of 0.0002% of the budget is fine, because an open source project should be open to criticism. But you should speak for others. Johnjbarton (talk) 16:43, 22 October 2026 (UTC)

Got it. You agree that the WMF sends banner adds that get read by people living in poverty, that those banner ads say that donations are needed keep Wikipedia free, that people respond by donating, and that some of those people are living in poverty. Your only objection is that the WMF (the only source that has access to that information) hasn't published data on donations by people living in poverty, and you refuse to believe that they exist if you can't count them. Thanks for clarifying your objection. --Guy Macon (talk) 17:15, 22 October 2026 (UTC)

Guy, it is not in dispute what the banners say, or that people living in poverty see those banners. What is being asked for is evidence for your claims that

• People living in poverty donate in response to those banners
• People suffer hardship because they donate in response to those banners

You say in your most recent comment that evidence for those claims does not exist, if so then you need to withdraw those claims and stop making them. If that evidence does exist, then you need to present that evidence. You cannot continue to cast aspersions on the WMF without evidence. Thryduulf (talk) 17:33, 22 October 2026 (UTC)

The burden of proof is on anyone who makes the extraordinary claims that you can take money from people living in poverty without affecting them or that poor people don't respond to WMF banners. --Guy Macon (talk) 17:48, 22 October 2026 (UTC)

You are the one making the extraordinary claims that the WMF are taking money from people living in poverty and that people are donating so much to the WMF that they are suffering hardship as a result. You have still presented no evidence for either of those claims. Thryduulf (talk) 18:39, 22 October 2026 (UTC)

That's not how it works. None of my claims are extraordinary. Yours are so extraordinary and outlandish that I frankly refuse to believe that you actually think that poor people don't respond to WMF banner adds or that poor people are not harmed by making them poorer. Who made the claim is irrelevant. Whether I make a claim that ghosts exist or that plants do calculus or whether you make the claim that they don't, the burden is on the person making the extraordinary claim, not on whoever made the claim first. -Guy Macon (talk) 23:01, 22 October 2026 (UTC)

Guy Macon, you said I will be happy to support my claims with citations to reliable sources... Please let me know exactly what you are disputing so I can post evidence to support my claims. We would like sources showing that A lot of those donors are living in poverty. LittlePuppers (talk) 23:17, 22 October 2026 (UTC)

• 5.46% or Wikimedia foundation ads get clicked on. 3.77% result in donations.

Source 2023-2024 Wikimedia Foundation Fndraising Report

• The Wikimedia Foundation does fundraising campaigns in India, Latin America, and South Africa.

Source: Wikimedia Foundation kicks off fundraising campaigns in India, Latin America, and South Africa

• 5.3% of India's population falls below the extreme poverty line threshold of $3 per day.

Source: The World Bank's estimates of poverty in India

• 9.8% of Latin America lives in extreme poverty.

Source: Latin America Reached Its Lowest Ever Poverty Rate in 2024

• 27.4% of South Africans live in extreme poverty.

Source South Africa sees significant poverty reduction over 17 years

• Families living in deep poverty face profound material, social, and emotional hardships. Households in deep poverty suffer from food shortages, unemployment, unstable housing, inadequate medical care, electrical shutoffs, and isolation. All US states have at least 5% of their children living deep poverty, including nearly one in five Black children under the age of five (Deep poverty is defined as living in a household with total cash income that is below 50% of the poverty threshold. It is a higher income level that the extreme poverty levels discussed above.)

Source: A World of Hardship: Deep Poverty and the Struggle for Educational Equity

--Guy Macon (talk) 00:26, 23 October 2026 (UTC)

I would also like to see your reliable sources that show:

• People living in poverty donate to the WMF because of the WMF's fundraising banners
• People suffer hardship because they donate to the WMF because of the fundraising banners

Those are claims that you made. You offered to support claims you make with reliable sources. Please do so. The burden is not on me to support claims that I am not making. Thryduulf (talk) 23:52, 22 October 2026 (UTC)

That's not how the burden of proof works. The burden in on the person making extraordinary claims, not on whoever spoke first. If I say that the Loch Ness monster doesn't exist and you reply that it does, the burden of proof is still on you to prove it exists because your is the extraordinary claim. --Guy Macon (talk) 00:26, 23 October 2026 (UTC)

Those are claims you made. Those are extraordinary claims. Therefore the burden of proof is on you. I am not making any claims, extraordinary or otherwise, I'm simply asking you to provide the evidence that backs up the claims that you are making, claims which you have claimed are supported by reliable sources. Thryduulf (talk) 00:48, 23 October 2026 (UTC)

None of those sources support any of the claims that you are making. They do support claims that nobody is disputing, specifically:

• The WMF run ads in those territories
• Some people who see ads click on them, and some of those people donate
• Some people who live in those territories live in (extreme) poverty
• Deep poverty causes harm to people who live it

Remember that correlation does not imply causation, and you haven't even shown correlation here. Thryduulf (talk) 00:56, 23 October 2026 (UTC)

This is not something that will be resolved by citations. It's a rhetorical move -- invoke a hypothetical sympathetic donor to delegitimize whatever the criticized-expense-of-the-day happens to be. In my opinion, the "people donate their hard-earned money because they think Wikipedia will die without it" argument makes some sense when criticizing the intensity of banner appeals, but just sounds condescending when trying to litigate specific expenses. At the end of the day you can accept it or ignore it, but it's not a citation issue. — Rhododendrites ^talk \\ 17:34, 22 October 2026 (UTC)

Good point. If the banner ads said something like "your donations go towards all the good things Wikipedia does" my entire argument would collapse and I would withdraw it. It is the combination of claiming that without your donations Wikipedia will no longer be free and the spending of those donations on things that have nothing to do with Wikipedia remaining free that I object to. Remove either leg and the argument falls over. --Guy Macon (talk) 17:48, 22 October 2026 (UTC)

Ok so you just represent Guy Macon. I think your argument makes sense in these terms, even if I don't agree with your conclusion.

In my opinion, the research effort Slow Editing Towards Equity was a high leverage investment in keeping Wikipedia "free". High-leverage because this amount of money is very small; investment because it was risky. It resulted in a published paper, above average results for that small investment. It engaged with scholars which may help further scholarly work on the structure of Wikipedia and its effectiveness. This kind of work might even be more important than paying electric bills for servers: Wikipedia has been challenged as unfair by both progressive and regressive ideologs so some exploration in to the effectiveness of policy structure makes sense. Being taking over by ideologs seems to me one o greatest risks to Wikipedia. Since "some" is in this case "almost trivial" this investment seems ok to me. Johnjbarton (talk) 18:29, 22 October 2026 (UTC)

Conclusions

We reviewed 134 source verification suggestions from the dataset and discussed them together as a team. What follows are the conclusions we're reaching:

1. Article claim and source URL extraction worked well. Generally, the model succeeded at extracting claims and source URLs from Wikipedia articles. This enabled us to focus on the quality of the verdicts the model reached and the rationale it provided for each.
2. The model outputs sufficiently reliable verdicts. In 71% of the suggestions we reviewed, internal reviewers agreed the cited source did not fully support the claim, including 49% where it did not support the claim at all. Several of these were concrete discrepancies between article and source (see the Lucille Ball and Alamgir examples below).

Taken together, we think the source verification suggestion is reliable enough to warrant a wider volunteer-led review. Please see "Proposed next steps" for how we see this playing out.

Proposed next steps

With the "Conclusions" above in mind, we see the next steps as the following:

1. On Tuesday, 3 November 2026, we (staff) make the initial dataset of source verification suggestions available to you all in two forms:
   1. As an experimental edit suggestion within Suggestion Mode.
      1. Note: These suggestions would only be available to people who have both 1) published ≥100 cumulative edits and 2) enabled experimental edit suggestions within Preferences
   2. As a spreadsheet so that you can more easily see patterns across suggestions and locate articles where you can experience the suggestions in-context via Suggestion Mode
      1. Note: we do not think reviewing suggestions in the spreadsheet alone is sufficient for forming an opinion about the usefulness of the suggestion.
2. You all (volunteers) review the suggestions, fix inaccuracies, and report back on what patterns you're noticing within each on User talk:Alaexis/AI Source Verification/Suggestions.
   1. Note: we see the purpose of y'alls review as making sure we wouldn't be wasting unsuspecting experienced volunteers' time with a large number of erroneous suggestions by enabling source verification suggestions as default-on within Suggestion Mode.
3. We (staff + volunteers) will then look at data from both reviews to decide: Do we make the source verification suggestion available by default within Suggestion Mode to experienced volunteers so that we can evaluate how helpful they find it in real world use?
   1. Note: to evaluate "how helpful [volunteers] find it in real world use" we would be analyzing engagement with the suggestion, the quality of edits made with it, the edit completion rates of edits where it was activated, and qualitative feedback from experienced volunteers who tried it.

Open questions

1. What additional filters might be useful to place on the model and/or what articles it runs on to create a more useful user experience? E.g. Setting a cap on how many suggestions are shown per article, preventing suggestions from appearing on list pages, etc.
2. What information, shown alongside a suggestion, might help you decide more quickly whether it’s right? E.g. deep-linking to the relevant passage within the source, linked to the diff where the claim was added, etc.
3. Currently if the claim is not supported, we expect that the editor will fix it or add a {{failed verification}}. Is that reasonable or are there other actions we should try to directly enable?
4. How should the suggestion behave when it can't read a source (e.g. paywalled, offline, etc.)?

Detailed findings

Internal review findings Process We generated a total of 16,585 suggestions for a sample of articles that met the criteria listed here We compiled those 16,585 suggestions into a spreadsheet, split up by category (e.g. blp, history, ctop, {{dubious}}, {{Failed_verification}}, {{ai generated}}, etc.) 6 staff members reviewed a total of 134 of the 16,585 suggestion For each of the 134 suggestions, staff: Reviewed the claim text the model extracted from the article and evaluated whether they thought the claim was complete Reviewed the verdict the model reached (Not supported or Partially supported), the rationale the model gave for why it reached said verdict, determined to what (if any) extent they thought the model prediction was correct, and offered an explanation as to why they reached the conclusion that they did We then looked over the suggestions our colleagues reviewed and documented general reflections about them Lastly, we calculated the rates at which staff verdicts did and did not align with the verdicts the model reached (see "Quantitative findings" below) Quantitative Findings In 49% of cases, reviewers agreed with model in concluding source does not support claim at all Flagged claim Notes AdventHealth opened a 200,000 square feet (19,000 m2) emergency room and primary care facility on Walt Disney World's campus on May 17, 2023. "AdventHealth facility is only 19 thousand sqft, not 200 thousand" (page: AdventHealth, check id 3c1bac7e1c82eaae) [Lucille Ball’s] acting forays were stilled at an early stage when she became ill with rheumatic fever and was unable to work for two years. "Claim states she had rheumatic fever but source states it was rheumatoid arthritis." (page: Lucille Ball, check id: 3f99f519c8f527fc) [Mary of Teck] is buried beside her husband in the nave of St George's Chapel, Windsor Castle. "Queen Mary was buried in the North Nave Aisle, not the nave itself." (page: Mary of Teck, check id 012418710cd2e07e) "Though Alamgir was unhurt, three vehicles (including the one in which he was riding) were vandalized, and ten people were injured." Link to claim The claim included within the Wikipedia article states that the car the subject was riding in was damaged. Although, the source cited says the car he was riding in was NOT damaged. These two statements contradict one another. (page: Mirza Fakhrul Islam Alamgir, check id: 01836695cfc49863) In 22% of cases, reviewers agreed with model in concluding source does not support claim fully Flagged claim Notes On 17 June 2026, Keir Starmer announced that the Children's Wellbeing and Schools Act 2026 has passed, which puts a ban on children under 16 having accounts and accessing social media apps The act was passed on a different date and doesn’t ban anything by itself (page: Online age verification laws by country, check id: cbeeec4e88959d7a) [Amis Harvest Festival] takes place every July to September. Article claims that the Amis Harvest Festival takes place every July to September. The source only mentions summer time, not fixed dates, and that the dates can vary across villages. (page: Amis people, check id: 740542a585970b96) A statue of Earhart by Ernest Shelton was erected circa 1971 in Los Angeles, California. Source talks about the statue and when it was erected, but doesn’t mention the sculptor’s name as listed in the claim. (page: Amelia Earhart, check id: 0366ad517930495e) In 28% of cases, reviewers disagreed with model in concluding source does not fully support claim Flagged claim Notes “Marie Tussaud was employed to make a death mask of her head.” - The model thought that “her” refers to Marie Tussaud herself (page: Marie Antoinette, check id: 770d5c8b71c195f2) It would also be played by Weir's RatDog after the dissolution of the Grateful Dead. Source does say that Ratdog played "Victim or the Crime", and the dates mentioned in the article are after the dissolution of the Grateful Dead. (page: Victim or the Crime, check id: 2d912ad52fc7efe4) Following the release of Citizen Vigilante in late June 2026 and its ban in Germany, the film was promoted by Elon Musk on X, greatly boosting its reach. A second source verifies the part of the claim that the model says the source doesn’t verify (page: Uwe Boll, check id: 0e4b8066c9135210) Qualitative findings Perceived significance of discrepancies: There are some claims that the model flagged as not being supported by their source(s) that, upon closer inspection, involved relatively small discrepancies between the information included within Wikipedia and the source. Editor opinions may vary on how consequential they consider these discrepancies to be. E.g. a film reaching some sort of revenue milestone within 3 vs. 4 days after launch. Selection bias: The quantitative findings should be taken with a grain of salt considering we did not review suggestions randomly. As such, there may be some selection bias at play. URL Extraction: as part of the initial internal review process, each person was asked to verify whether the model had accurately extracted the source URL from Wikipedia. In 100% of cases, the model was successful in this operation. As a result, we removed this URL checking from the review process. Claim Density: the number of claims the model flagged as needing review varied article-by-article. Some of this variance can be explained by genuine differences in content quality, others, as we came to learn, can be explained by the model handling claims that start with pronouns (e.g. "he/she") less well. There are a series of improvements we think could improve the model that we'd value y'all input on. See "Open questions" below. For context: A typical article in our sample has over 150 citations, and the model flagged about 17 of them as not fully supporting their claims. There are outliers: 2% of articles have >100. Lists and some biographies (e.g. 2026 deaths in the United States - see screenshot). This is due to a combination of the higher background level of sloppiness, citation density and tool deficiencies, for instance the pronoun problem. Suggestion invalidation Suggestions appear if and only if the claim and source have not changed since the article was initially analyzed and the source is not followed by a qualifying template[i]. That is, if the suggestion has been addressed in any capacity - a user has updated the source or the claim, whether prompted by suggestion mode or not, or has added a qualifying template - then the suggestion will no longer show to any user. If a user dismisses a suggestion by clicking the ‘dismiss button’, they won’t see that suggestion for the remainder of the edit session. However, they - and other users - will continue to come across it in future edit sessions until it is actually addressed. The Editing Team recognizes the importance of preventing what experienced volunteers agree are invalid suggestions from appearing for themselves or other people in the future. We are tracking this work in T427236 and T433147. While we have not yet defined a timeline for when this functionality will be in place, you can see the technical thought we've put into the architecture that would help enable it. i. Wikis have the option to locally configure a ‘templateToInsert’ for Source Verification in the on-wiki config. This is the template that (1) will be inserted after the claim when the user dismisses the suggestion with the ‘suggestion is valid’ response, and (2) will be checked for to determine if the suggestion is still valid and should be surfaced to other users. For example, the recommended ‘templateToInsert’ for enwiki would likely be {{Failed-verification}}.

PPelberg (WMF) (talk) 16:45, 29 October 2026 (UTC)

Pinging a few folks who expressed interest in this work in previous discussions (on-wiki and off-): Levivich, Asilvering, Thryduulf, Ahecht. RoySmith, CMD, ClaudineChionh, The Other Karma, Kepler-1229b, Qcne, ActivelyDisinterested, Gnomningstuff, Toadspike, and Dreamyshade. PPelberg (WMF) (talk) 19:31, 29 October 2026 (UTC)

Hi, does the WMF Tool already have access to the wikipedia library? I testet the script from Alaexis however in all the testet articles, 90% of the sources were not available, even open access sources. If its like that, the tool is useless. While i can certainly copy the text into Alexis's tool and let it be checked, but then I might as well just ask the AI in my browser. Or myself. In my opinion, the tool is only useful if it saves me the trouble of looking up the claims in the sources. Then all I have to do is verify the AI's output. The Other Karma (talk) 07:27, 31 October 2026 (UTC)

I am not sure 71% is sufficiently reliable - that's calling someone out to look specifically at a source to double-check something for no reason more than 1/4 of the time. And since we can't actually dismiss the suggestions, that... will be a lot of incorrect suggestions constantly coming back up.

Re: whether a source agrees fully or not, I'm not really seeing a difference between "agree not fully" #3 (the Earhart one) and "disagree not fully" #3 (Uwe Boll). In both cases, as I read these, there is some information that was not in the source, and the model is equally correct in both instances. It's just that in the second there is also another citation. This will surely affect quite a lot of articles - can the model not be trained to look at all of the footnotes given, not just the first?

Re: "north nave aisle", it's quite normal to describe that part of the cathedral as "the nave". So the claim is actually fully correct, if not as specific as it perhaps ought to be. In solidarity, asilvering (talk) 19:51, 29 October 2026 (UTC)

I am not sure 71% is sufficiently reliable…

You may end up being right in thinking this suggestion is not reliable enough, in its current form, to warrant a deployment as a default-on suggestion within Suggestion Mode.

Although to clarify, we were not seeking, nor equipped to, make a decision about a wider deployment with this initial internal review. This internal review was narrowly scoped to answering, "Do we think these suggestions are reliable enough to warrant experienced volunteers allocating valuable time to reviewing them?"

And based on the review we conducted, we do think the suggestion is reliable enough to invite interested experienced volunteers to review them.

It's only after we (staff and volunteers) have reviewed these suggestions would we together decide "Do we make the source verification suggestion available by default within Suggestion Mode to experienced volunteers so that we can evaluate how helpful they find it in real world use?" (See Step 3 above)

Does this get at the point you're making? Might there be something I'm missing/misunderstanding?

...that's calling someone out to look specifically at a source to double-check something for no reason more than 1/4 of the time.

On the time spent piece, you may well be right and I think it's only through real world use will y'all be able to make a judgement call on this.

As I see it, what matters is whether the time spent checking false positives outweighs the time saved by catching real errors. To what extent (if any) does this thinking align with how you're viewing this tradeoff?

And since we can't actually dismiss the suggestions, that... will be a lot of incorrect suggestions constantly coming back up…

We agree with you here: without the ability to dismiss suggestions, multiple volunteers could see the same unhelpful/incorrect suggestion which would in turn increase how precise the model would need to be before we’d (staff + volunteers) be comfortable putting it in front of people. Potentially, to an unachievable point even.

So it's not ambiguous: we see work on dismissals as a question of when not if and we're open to the possibility that this functionality would need to be in place before source verification suggestions are enabled by default.

Thank you for thinking through this critically. Please let me know if any facet of the above leaves questions/concerns open in your mind or causes new ones to emerge. PPelberg (WMF) (talk) 22:33, 29 October 2026 (UTC)

@PPelberg (WMF), we must be misunderstanding each other, because I read "Do we think these suggestions are reliable enough to warrant experienced volunteers allocating valuable time to reviewing them?" as basically "is this reliable enough to be deployed?" with extra words. What's the difference, for you? In solidarity, asilvering (talk) 02:05, 30 October 2026 (UTC)

@Asilvering, the distinction is between enabling them as experimental suggestions, visible only to editors who have opted in to experimental suggestions, and making them regular suggestions, visible to all editors who meet the edit-count criteria. Alaexis_¿question? 07:34, 30 October 2026 (UTC)

Thank you for adding this context, @Alaexis. @Asilvering: does what Alaexis shared above address the confusion you spotted? PPelberg (WMF) (talk) 22:30, 30 October 2026 (UTC)

No, I think we may have a fundamental disagreement. But I think that disagreement could be resolved by reframing how the edit suggestion is presented to editors. Right now, the wording of the suggestion begins from the presumption that the model is intended to highlight problematic sentences; the purpose of this edit suggestion is to find and fix incorrect sentences on Wikipedia. It is understandable that the suggestion is worded in this way, because that was indeed the intent of this project. However, with a 71% hit rate, we cannot truthfully say that the model is highlighting problematic sentences. So in fact what we have is an edit suggestion that fails 29% of the time even if the editor correctly applies WP:V when responding to the check, which we can't be certain about either.

If you cannot raise the hit rate over 71%, the suggestion should be reframed as a spot check. We do sourcing spot checks all the time (for example, at WP:GAN), and they do not start from the presumption that the sentence is incorrect. (At GAN, you're instead starting from the presumption that the sentence is correct, if anything.) So instead of "a model has identified that this claim might not be supported by its source", you end up with something neutral like "a model has selected this sentence for source-checking". Under those terms, you have an edit suggestion that works 100% of the time, and ideally results in improvements to the article 71% of the time or more often. Does you follow?

In any case, we definitely need the ability to dismiss suggestions as incorrect (ie, to mark it as "fact checked, it's fine") before this leaves "experimental mode". In solidarity, asilvering (talk) 00:03, 31 October 2026 (UTC)

I like the idea of carefully framing the suggestion as a prompt to spot-check a citation. As a relatively neutral framing, it could reduce automation bias and increase the chance that a person actually goes and looks at the source. It also opens up the opportunity to prompt spot-checking of sensitive/high-risk statements where the automated verifier can't access or parse the citation. The research for the reference need model (paper) might suggest ways to help find those statements. Dreamyshade (talk) 00:57, 31 October 2026 (UTC)

+1, I assumed this was already going to be the case Kowal2701 (talk, contribs) 01:03, 31 October 2026 (UTC)

@Kowal2701, the current UI is in the initial post. In solidarity, asilvering (talk) 01:08, 31 October 2026 (UTC)

Agreed re the dismissal and I think that it should be recorded on-wiki. This way the community will be able to measure the effectiveness of suggestions (edits vs dismissals). Having it recorded will also reduce abuse as the editor will know that the dismissal is recorded and attributed.

Not sure about the framing as "spot checks". Spot checks ("a quick examination of a few members of a group instead of the whole group" ) make it look like a random sample, while in reality it's more like a screening (medicine), where a positive result means that a more thorough check is needed.

I'm sure the language can be adjusted, I think one widely used tool that has the same probabilistic nature is Earwig's Copyvio Detector. It describes likely copyvios as "Violation suspected" (example - here the top one is actually not a copyvio but a wiki mirror). Alaexis_¿question? 17:20, 31 October 2026 (UTC)

Re: whether a source agrees fully or not, I'm not really seeing a difference between "agree not fully" #3 (the Earhart one) and "disagree not fully" #3 (Uwe Boll). In both cases, as I read these, there is some information that was not in the source, and the model is equally correct in both instances. It's just that in the second there is also another citation. This will surely affect quite a lot of articles - can the model not be trained to look at all of the footnotes given, not just the first?

Good catch. The model is trained to evaluate all footnotes following a claim, not just a subset of them. If it happens to return a verdict that doesn't cover the complete set, then we do a second check on the suggestion mode side to ensure this suggestion isn't shown to users. So this example is one that was not caught by the model but would still not have been shown in suggestion mode; including it as an example here was our mistake, as it doesn't reflect the actual user experience. CMedelius-WMF (talk) 22:47, 29 October 2026 (UTC)

Aha. Thanks @CMedelius-WMF, that's very reassuring. In solidarity, asilvering (talk) 01:59, 30 October 2026 (UTC)

With regards to #1, I think it'd make sense to avoid checking articles being rapidly edited, new articles, and claims that cite paywalled or offline sources as referred to in #4. On #2, making the model's reasoning available could help the community figure out where strengths and weaknesses are. ^MetalBreaks_AndBends (One for all) 19:57, 29 October 2026 (UTC)

On #2, making the model's reasoning available could help the community figure out where strengths and weaknesses are.

@MBAB, could you please say a bit more here about the sort of reasoning you'd find helpful?

I ask this because currently, the model outputs a rationale for each suggestion it outputs. E.g. “Claim states she had rheumatic fever but source states it was rheumatoid arthritis”. This leads me to think you might be thinking of something else/in addition...

I think it'd make sense to avoid checking articles being rapidly edited, new articles...

You and @CMD each raised this. Could you please share a bit more about where this recommendation is coming from?

For what it's worth, we'd been assuming (perhaps inaccurately!) that it'd be useful to check articles that are new and being rapidly edited because they'd be more likely to contain inaccuracies. Then again, maybe you're thinking adding another actor into these situations (in this case a machine) could complicate things? PPelberg (WMF) (talk) 23:46, 29 October 2026 (UTC)

I was talking about reasoning tokens, as reading them can be helpful for understanding a model's thinking in further depth than a 50 word summary can provide. Obviously, this shouldn't be exposed to editors by default (especially if your model of choice has a GPT-6-astra-esque affinity for not using the spacebar).

I suggested avoiding rapidly edited and new articles for 2 slightly different reasons. New articles because they're going to be in the NPP queue, which means they're going to be checked soon (although ai verification built for the NPP workflow could be promising. e.g. not doing the check until the page is being reviewed and factoring in other steps like a source's value for notability). For rapidly edited articles, it's because there's a higher chance that any given checked text will be modified requiring rechecking. My concerns on both of these is less that the system might get something wrong and more that compute might be misallocated. ^MetalBreaks_AndBends (One for all) 00:26, 30 October 2026 (UTC)

I was talking about reasoning tokens, as reading them can be helpful for understanding a model's thinking in further depth than a 50 word summary can provide. Obviously, this shouldn't be exposed to editors by default...

Ah, got it. I agree with you in thinking that exposing this level of detail is best suited for somewhere deeper in the experience so folks who are especially curious can find it. Thank you for adding this context.

In case it's interesting to know: I think the checks we ran used an un-reasoning version of Qwen 3.8. Although, @SSalgaonkar-WMF will be able to say for certain. PPelberg (WMF) (talk) 22:33, 30 October 2026 (UTC)

I suggested avoiding rapidly edited and new articles for 2 slightly different reasons. New articles because they're going to be in the NPP queue, which means they're going to be checked soon (although ai verification built for the NPP workflow could be promising. e.g. not doing the check until the page is being reviewed and factoring in other steps like a source's value for notability).

Understood. To confirm the NPP use case, you're thinking that it would be worthwhile to wait to run the suggestion on new articles until after an experienced volunteer has validated the article meets notability requirements? Specifics aside, we're with you in thinking there is/are NPP-specific use cases here that feel promising.

For rapidly edited articles, it's because there's a higher chance that any given checked text will be modified requiring rechecking. My concerns on both of these is less that the system might get something wrong and more that compute might be misallocated.

Oh, okay. I see. Understood. Thank you for explaining that. PPelberg (WMF) (talk) 22:37, 30 October 2026 (UTC)

...and claims that cite paywalled or offline sources as referred to in #4.

Right now, the suggestion ignores offline and paywalled sources. Although, this leads me to wonder: can you see it being useful for the UI to indicate when the model tried to evaluate a claim and was unable to do so because a source was offline or paywalled? Do you think it ought to instead show nothing silently? Something else? PPelberg (WMF) (talk) 23:49, 29 October 2026 (UTC)

This is also mostly based in compute allocation, although there is also a volunteer allocation angle too. If a paragraph has multiple citations at the end with one or more being inaccessible, it can't be known if any given claim not supported by the accessible sources is actually unsupported or just supported in the inaccessible ones. So, for those paragraphs any given suggestion is less likely to be helpful, due to it only being trustworthy where the content explicitly contradicts a source. A flag for inaccessible sources would help avoid errors where correct content is removed because inaccessible sources were skipped. But unless a system were devised to direct verifiers with access to inaccessible content to suggestions, I think bailing at the first sight of problematic sources would offer the best rate of "tokens per abstract unit of improvement". ^MetalBreaks_AndBends (One for all) 00:57, 30 October 2026 (UTC)

I've been working with a Master students evaluating the script, and they found a couple of categories of mistakes it often makes. I will ask them to share. Perhaps that can raise the 71% to a higher level. I agree that 71% is not quite the accuracy we would like to see. Source verification is incredibly time-consuming as an activity, and newish (100-1000 edits) editors are also not great at it. Will this result in properly cited text being removed?

In my two GAs I've run it on today, I disagreed with all 5 suggestion that things were wrong (2 were subjective), but maybe the script uses a different model than the one you have access to? In solidarity, —Femke (talk) 🐦 21:19, 29 October 2026 (UTC)

...I disagreed with all 5 suggestion that things were wrong (2 were subjective), but maybe the script uses a different model than the one you have access to?

@Femke: might you be able to share the version of the script and/or model you were using so that we can check if they're the same? Also: might you be able to share the following so that we can see the suggestions you encountered? It'd be helpful to know:

• The permanent link to the version of the article you were editing
• The text of the claims the model generated suggestions for
• The verdicts and rationale the model gave

I've been working with a Master students evaluating the script, and they found a couple of categories of mistakes it often makes. I will ask them to share.

Seeing what they've found would be a big help. Thank you for offering!

I agree that 71% is not quite the accuracy we would like to see. Source verification is incredibly time-consuming as an activity, and newish (100-1000 edits) editors are also not great at it. Will this result in properly cited text being removed?

I'm going to hold off on responding to the "71% accuracy" part for a moment because I'm going to come back to that in a response to the point Asilvering raised about this above. Now, with regard to who the proper audience might be for this suggestion, we think y'all are best equipped to decide and implement what you think is best via Special:EditChecks. PPelberg (WMF) (talk) 21:41, 29 October 2026 (UTC)

I would love 100+ edits to be the audience, as that's the type of editors we want to teach to fact check everything. But.. I think putting the weapon in their hands would be a bit too dangerous as is. I suspect many people will trust the tool, and where the tool is wrong, it's usually in more complicated facts I also think editors will get wrong.

I'm off to bed soon, but I ran it on post-exertional malaise (4 claims flagged as incorrect) and Ursula Bethell (1 claim flagged). Two of the errors seems to be, if I understand it correctly, from sentences supported by two sources, where 1 of the sources didn't completely cover all the content. One said that 'irreversible decline' was not the same as 'permanent decrease', which is probably a bit more subjective as permanent may be a slightly stronger word, but I'd say that's the same. Don't remember the other two, would have to rerun. Will rerun tomorrow with the requested info.

I used Aleaxis' script with the default settings, so Lift Wing. In solidarity, —Femke (talk) 🐦 21:50, 29 October 2026 (UTC)

For reporting issues that you notice from particular suggestions, it may be easier to use the tool's built-in "comment" button, which sends reports to User talk:Alaexis/AI Source Verification with metadata about the suggestion itself. Sdkb‑WMF ^talk 18:39, 30 October 2026 (UTC)

The edit suggestions have a different UI which doesn't have a comment button (see the screenshot above). @CMedelius-WMF, maybe you can explain what options the editor has and how they work? Alaexis_¿question? 19:17, 30 October 2026 (UTC)

Yes, the edit suggestion appears with two options: "Revise", which brings them into a focused edit state, and "Dismiss", which opens up a survey to provide an answer on why.

It also has the normal dropdown that is shared by all edit suggestions, which contains an option to provide feedback that is then posted to mw:Talk:VisualEditor/Suggestion_Mode/Feedback. We've internally been discussing a change to make the feedback destination configurable; and in this case, it may be helpful to direct feedback to User talk:Alaexis/AI Source Verification. CMedelius-WMF (talk) 19:48, 30 October 2026 (UTC)

I would love 100+ edits to be the audience, as that's the type of editors we want to teach to fact check everything. But.. I think putting the weapon in their hands would be a bit too dangerous as is. I suspect many people will trust the tool, and where the tool is wrong, it's usually in more complicated facts I also think editors will get wrong.

1) It's important newer editors be equipped with the know-how and tools to help out with verifying claims are corroborated by their accompanying sources

2) I see potential in this suggestion helping newer editors onboard into and complete this work

3) Before "2." can happen, we need to be confident that the user experience is – let's call it – resilient to people assuming the suggestion is always right.

Assuming the three points above accurate reflect the points you were making @Femke, I think we're aligned here. PPelberg (WMF) (talk) 22:45, 30 October 2026 (UTC)

Great to see this work continuing and progress being made.

For my part, idk if ~70% is too low or not. Obviously, higher is ideally better, but I think most people aren't right 70% of the time, so seems to me the model is doing pretty good :-)

The concern I have is that ~150 out of 16k is a tiny percentage, and as they were apparently not randomly selected, I'm not sure if the results are representative at all of anything. Of course, the solution to that is what you're already planning to do: have more people do more testing, which seems like the right next step regardless. Levivich (talk) 22:01, 29 October 2026 (UTC)

yeah, ~70% is well worth it, way better than any other way we have of finding WP:V failures (which is nothing). Issue is whether people will put absolute trust in it or be lazy/go with the balance of probability and not actually read the source before removing stuff Kowal2701 (talk, contribs) 00:50, 30 October 2026 (UTC)

@Kowal2701, that's true for, say, Alaexis's userscript. I'd be quite happy to use a userscript with a 71% hit rate, because I'd be using that mostly informationally - ie, my use case is that I'm already trying to fact-check the article, and the script would be helping me focus on the parts that seem most likely to be wrong. That's great. But in Suggestion Mode, what it's doing is surfacing an error to an editor, possibly a relative newbie, and directly asking them: hey, does this source verify, because I don't think it does? And that's a very different kind of mode that ought to have a higher hit rate than 71%, in my opinion. In solidarity, asilvering (talk) 02:03, 30 October 2026 (UTC)

@Asilvering, agreed that default-on suggestions is a different mode from a userscript. However right now these suggestions will be experimental, visible only to opted-in users, and the whole point of the experimental stage is to check whether these suggestions work before anyone decides to make them regular ones. It's quite possible that based on the findings of the experimental stage we'd fine-tune the pipeline and/or enwiki could raise the edit count threshold. Alaexis_¿question? 07:43, 30 October 2026 (UTC)

I think people should easily be able to opt-out of it. Tbh what I don't like about Alaexis' script is reading the LLM's reasoning, I'd rather it just listed the claims with their corresponding sources, possibly on the talk page, and people can just {{tick}} or {{cross}} and sign or whatever. I expect its false positive rate is much higher on good quality articles, we could exclude FAs and GAs but idk what else Kowal2701 (talk, contribs) 15:20, 30 October 2026 (UTC)

...hat I don't like about Alaexis' script is reading the LLM's reasoning, I'd rather it just listed the claims with their corresponding sources, possibly on the talk page, and people can just Y or N and sign or whatever...

@Kowal2701: the simplicity of the workflow you're describing resonates with me and I'm curious: what don't you like about seeing the LLM's reasoning? E.g. might you not trust it and therefore find its presence non-additive?

...I expect its false positive rate is much higher on good quality articles, we could exclude FAs and GAs but idk what else...

The thinking there being those articles have already been closely reviewed by knowledgable volunteers? PPelberg (WMF) (talk) 17:14, 30 October 2026 (UTC)

what don't you like about seeing the LLM's reasoning? Yeah, partly because I wouldn't trust it, partly because I'm sick to death of reading AI-generated text lol. I don't think there's really any benefit to it since in all cases the human will have to check the source for themselves anyway, though ofc having the reasoning is useful for testing the feature re: how accurate it is.

The thinking there being those articles have already been closely reviewed by knowledgable volunteers? Yes, and content written by experienced editors is usually of a similar calibre. Maybe we could use the mw:WikiWho API to exclude content written by editors who have GAs/FAs, or if we wanted it to be more restrictive exclude content authored by users w autopatrolled? Kowal2701 (talk, contribs) 18:24, 30 October 2026 (UTC)

One of the findings from my student this summer was that the script detected loads of issues at a FA that they believed were true errors. I'm curious to see whether I agree. In solidarity, —Femke (talk) 🐦 18:29, 30 October 2026 (UTC)

yeah, I'm sure it would identify true WP:V issues in FAs and content by users w autopatrolled, but given such issues generally occur less often in such content I assume there'd be a higher rate of false positives? Kowal2701 (talk, contribs) 18:32, 30 October 2026 (UTC)

Yeah, you're right that there will be more false positives in such articles (the false discovery rate would be higher). On the other hand, it's also more embarrassing to have such errors in GA/FA, so fixing them may be worth extra time spent on checking duds, and Femke’s student found real errors at an FA. I think it's worth tracking this while the suggestions are in the experimental phase: errors fixed vs discarded suggestions in GA/FA. Alaexis_¿question? 18:58, 30 October 2026 (UTC)

it's probably worth incorporating it into GAR/FAC instead of random spotchecks Kowal2701 (talk, contribs) 22:37, 30 October 2026 (UTC)

@Kowal2701, I don't think that's a good idea, because the script only works on a subset of sources, and that subset of sources is already the set that are easier to check. Because reviewers who make random spotchecks are often quite determined to maintain that randomness, they will request access to harder-to-find sources from the nominator and go digging where otherwise they might not, and this is an important part of that process. Alaexis's script can be perfectly useful at GAN/FAC, especially if you're pretty sure the whole article is dubious and you're looking for some low-hanging fruit so you can fail the review and send it back for more editing. But that random spot-check has real benefits and it would be a mistake to incorporate this script instead. In solidarity, asilvering (talk) 00:13, 31 October 2026 (UTC)

Another thing, we might want to exclude articles with {{AI-generated}} tags, since it's not really fair to have someone hone/polish such content when it'll have to be removed or rewritten from scratch Kowal2701 (talk, contribs) 18:30, 30 October 2026 (UTC)

Oh, the notes are the AI's reasoning. Here I thought it was the reasoning of the staff who had agreed or disagreed with the model. That makes me much less grumbly. Though I still think 71% hit rate is far too low for a useful edit check. In solidarity, asilvering (talk) 20:26, 30 October 2026 (UTC)

The concern I have is that ~150 out of 16k is a tiny percentage, and as they were apparently not randomly selected, I'm not sure if the results are representative at all of anything. Of course, the solution to that is what you're already planning to do: have more people do more testing, which seems like the right next step regardless.

Very fair - the results of our internal review shouldn't be taken as a measure of overall accuracy, for the reasons you've stated. However, the purpose of the internal review was more so to perform a sanity check that the suggestions were good enough to be worth additional reviewers’ time. We see volunteer review as the next step because we think you all, as people experienced with verifiability work, bring the nuance needed to differentiate between suggestions that identify genuine issues and those that you'd consider noise. CMedelius-WMF (talk) 08:23, 30 October 2026 (UTC)

For question 1, I agree with MBAB, an automatic tool should avoid frequently edited articles (if someone wants to run it manually, they know what they might be getting into). I don't see a problem with running these on all lists, if the question is whether the tool can understand table sourcing then have it avoid tables on all articles. For question 2, the linked diff is very helpful, especially as it will pick up WP:TSI issues where sources and text get separated by later edits. For 3, please include a nudge for an editor to use the |reason= parameter in fv. For 4, shouldn't the tool simply avoid those situations? More broadly, there seem to be different categories of errors that could be picked up, and it might be useful for the model to distinguish. Of the four "does not support claim at all", the first seems a case of somehow putting in the wrong units. I've found these sorts of numeral mismatches something that models are good at picking up with relatively few errors, if the model can figure out if something counts as a numeral mismatch that might provide a useful subset. The rheumatic fever is another sort of mismatch, worth flagging and seems correct in this case but it seems vulnerable to an llm not understanding wider context. The nave example asilvering has mentioned above, our nave article describes "nave" as having sensu strico and sensu lato meanings, this is perhaps an llm not understanding context and being overly prescriptive. The fourth example is a mix of numbers and prose, harder to classify. However, I don't see how it is distinguished from "does not support claim fully", as the source does support other parts of the text, such as the individual being unhurt. CMD (talk) 23:11, 29 October 2026 (UTC)

In general, I like these next steps. I think that they're reasonable and would allow a much better analysis of reliability. I would guesstimate that in my own experimentation with Claude, about 75% of its suggestions on source-text matches are accurate, so your findings so far roughly track my own experience. In terms of answers to the questions, I think a cap on suggested edits per-page, per-editing-session, is reasonable and that a link to a relevant passage would be helpful. I think if the model/suggestion cannot access a source, it should not take any action to avoid a situation where a new editor also cannot access the source and improperly removes a valid claim. ThadeusOfNazereth(he/him)^{Talk to Me!} 23:39, 29 October 2026 (UTC)