Wikipedia:Edit check

The Editing Team has been working on a set of improvements for the visual editor to help new volunteers understand and follow some of the policies and guidelines necessary to make constructive changes to Wikipedia projects.

Below, you can find information about the goals of this project, the history that has informed it, and why the Wikimedia Foundation's Product Department is prioritizing this work.

Watch mw:Editing team/Community Conversations for scheduled meetings about this project.

Features

Several features are available within Edit Check:

The Editing team works on new checks:

The Editing team also works on the related features:

How to test a check?

See also:

Objectives

  1. Newcomers and Junior Contributors from Sub-Saharan Africa will feel safe and confident enough while editing to publish changes they are proud of and that experienced volunteers consider useful.
  2. Moderators at the English and French Wikipedias will notice improvements in the quality of edits newcomers are making and be motivated to configure how Edit Check presents policies to them.

Status

The current status of Edit check is regularly updated at mediawiki.org.

Strategy and approach

To equip newcomers and Junior Contributors from Sub-Saharan Africa with the know-how and tools to publish changes they are proud of, and that experienced volunteers consider useful, the Editing Team will be introducing new functionality within the visual editor (desktop and mobile) that will check the changes people are attempting to make, and present them with actions they can take to improve these changes in accordance with established Wikipedia policies and guidelines.

The first "check" the Editing Team will introduce is to detect attempts to add new content to an existing article without a corresponding reference, and to prompt them to add the reference. The functionality will be accompanied by features that will enable moderators to configure the user experience of newcomers and Junior Contributors, to ensure the software is guiding them to take actions that align with project policies and conventions.

Challenges

The trouble is that the visual editor and other editing interfaces do not make people aware of the Wikipedia policies and guidelines they are expected to follow.

As a result, the changes inexperienced volunteers publish often break established best practices and lead to undesirable outcomes for inexperienced volunteers, experienced volunteers, and Wikipedia projects as a whole:

  1. Inexperienced volunteers – become disappointed and frustrated when the good-faith change(s) they arrived to the wiki seeking to make are undone (read: reverted), deleted, and/or scrutinized in inequitable ways. These poor interactions are demotivating and drive these could-be volunteers and community members, and the knowledge that are uniquely positioned to offer, away.
  2. Experienced volunteers/moderators – need to do more work reverting low-quality edits and posting messages on inexperienced volunteers' talk pages to make them aware of the policies and/or guidelines they are likely to have unknowingly broken. Continually needing to educate inexperienced volunteers and undo their changes can lead to experienced volunteers becoming skeptical of inexperienced volunteers and impatient with them.
  3. Wikipedia projects – struggle to grow and diversify their volunteer populations and shrink the knowledge gaps present within Wikimedia wikis.

This project seeks to address the challenges above by:

  1. Offering inexperienced volunteers relevant and actionable feedback about Wikipedia policies in the precious moments when they are in the midst of making a change using the visual editor.
  2. Equipping moderators with a new ability to specify the feedback inexperienced volunteers are presented with while they are editing

Theory of change

This project is built on the belief that by surfacing relevant guidance in the precious moments when inexperienced volunteers are in the midst of making a change to Wikipedia and equipping them with the know-how and tools necessary to apply this guidance, they will make changes they are proud of and that experienced volunteers value.

In the longer term, the Editing Team thinks that people who are new, particularly people who have historically been excluded from and harmed by established power structures, will feel safe and motivated making changes to Wikipedia if they can accurately predict whether the changes they are attempting to make are aligned with existing Wikipedia policies, guidelines, and/or cultural conventions.

More broadly, the Editing Team thinks that to evolve towards a future where wikis' policies and cultural norms – and ultimately, content – reflect the diverse experiences of the people these projects are intended to serve, we first need to make the norms and standards that are currently in place legible and actionable to people while they are editing. This way, volunteers can develop shared awareness of cases where these norms and standards are not having the impacts they were intended to have and decide what – if any – changes they think are worth making to them in response.

Primary audiences

The Editing Team is centering the needs of people in this work who are:

  1. Experience: Learning the basics of contributing to Wikipedia
    • In the context of this project, we are considering people who are still "learning the basics" to be people who have published <100 cumulative edits to a single, or multiple, Wikipedias. This includes people who are editing Wikipedia for the first time.
  2. Location: Living in Sub-Saharan Africa
  3. Projects: Contributing to the English and French Wikipedias
  4. Motivation: Seeking to fill gaps they notice within Wikipedia

The four focus criteria listed above are outgrowths of:

Design

Reference detection

To start, the Editing Team is pursuing an approach with Edit Check that minimizes the likelihood of false positives and is implemented in ways that empower volunteers, on a per-project basis, to evolve the heuristic to become more robust over time.

This strategy amounts to the initial reference Edit Check becoming activated if/when all of the following conditions are met:

  1. A minimum of one new paragraph of text is added to the article someone is editing
  2. The "new paragraph(s) of text" someone has added does NOT include a reference
  3. The changes described in "1." and "2." are happening on a page within the main namespace (NS:0)

The conditions above are implemented and maintained in code here: editcheck/init.js.

The Editing Team arrived at the decision to start with a relatively limited and straightforward set of rules in order to:

  1. Increase the likelihood that newcomers and Junior Contributors find the guidance Edit Check is presenting them with, and the editing experience more broadly, to be intuitive and straightforward so that they feel encourage to return to edit again
  2. Decrease the likelihood that Edit Check is creating more work for experienced volunteers by prompting newcomers and Junior Contributors to add sources when they are not needed

You can learn more about the assumptions that informed the thinking above in phab:T329988#8654867.

Other applications

See also Edit check/ideas.

Configurability

The Editing Team thinks it is crucial that moderators be empowered to configure when, and for whom, Edit Check becomes activated. This way, they can be confident the software is promoting behavior they deem to be productive and modify the software when it is not.

In line with the above, and drawing inspiration from how the Edit filter and Growth Team Community configuration systems afford volunteers the ability to audit and configure how they function on-wiki, Edit Check will enable volunteers, on a per project basis to:

Work to implement the above is ongoing in phab:T327959.

User experience

Mobile

The first version of Edit Check will add a new step to the publishing workflow of the mobile visual editor. Editors will see this step when they add new content without a citation.

Desktop

Mock-up of the old paste check. The paste check now also talks about LLM

Edit check shows up on the right on Desktop. In a previous version, the checks appeared in the mid-text, but that caused more friction (lower completion rates).

Experiments

Multi-Check (References) A/B Test

Leading Indicators (T388731)

  • New(er) volunteers are encountering Multi-Check
    • In the test group, multiple reference checks were shown within a single editing session at 19% of all published new content VE edits (549 edits) by unregistered users and users with 100 or fewer edits.
    • For edits shown multiple checks, the majority of edits (73%) were shown between 2 to 5 Reference Checks.
  • People shown multiple Reference Checks w/in an edit go on to publish at a relatively high rate
    • The edit completion rate for sessions that were shown multiple checks within a session was 76.1% compared to 75% for sessions shown only one check, indicating that multiple checks are not causing significant disruption or confusion to the editors.
  • Likelihood to include a reference
    • Sessions shown multiple checks are more likely to include at least one new reference in the final published edit compared to sessions shown just a single check.
    • In the test group, 52.5% of all published edits shown multiple Reference checks included at least one new reference compared to 39.7% of edits that were shown a single check.
  • Disruption (revert and block rates)
    • In the test group, the revert rate of new content edits shown multiple Reference Checks (17%) is currently lower compared to sessions shown a single Reference Check (26%).
    • No significant changes in the proportion of users blocked after being shown multiple Reference Checks compared to a single Reference Check.


Multi-Check Phase 1 Impact Analysis

Multi-Check (Phase 1)

In December 2024, we released a new design for the Edit Check desktop experience. This change shifted Edit Checks from appearing within articles to appearing alongside them, in a new "siderail."

To decide how – if at all – this change impacted volunteer disruption and edit quality, we compared analyzed several key metrics shifted before and after this change.

Findings

Conclusion(s)

The findings above suggest this design change has been net positive. As a result, we will continue depending on this new design paradigm as we introduce new types of Edit Checks and moments within the editing workflow when they are presented.

Reference Check A/B Test

To learn whether the Reference Edit Check is effective at causing newcomers to make edits they intended and experienced volunteers value, we conducted an A/B test with 15 Wikipedias.

Below you can read more about what this experiment demonstrated, what the Editing Team is planning in response, and more details about the test's design.

Conclusion and next step(s)

Reference Check caused an increase in the quality of edits newcomers publish and did not cause any significant disruption.

This combination is leading the mw:Editing team to be confident that offering Reference Check as a default-on feature would have a net positive impact on all wikis and the people who contribute to them.

You can read the full A/B test report here.

Findings

There was 2x increase in the proportion of new content edits by newcomers, Junior contributors, and unregistered users that included a reference when Reference Check was shown to eligible edits.

New content edits *with* a reference

People shown the Reference Check are 2.2 times more likely to publish a new content edit that includes a reference and is constructive (not reverted within 48 hours).

  • Increases were observed across all reviewed user types, wikis, and platforms.
  • The highest observed increase was on mobile where contributors are 4.2 times more likely to publish a constructive new content edit with a reference when Reference Check was shown to eligible edits.

Revert rate

  • New content edit revert rate decreased by 8.6% if Reference Check was available.
    • New content edits by contributors from Sub-Saharan Africa are 53% less likely to be reverted when Reference Check is shown to eligible edits.
We observed increases on both desktop and mobile. On mobile, users are 4.2 times more likely to include a reference with their new content when the Reference Check is shown to eligible edits.

While some non-constructive new content edits with a reference were introduced by this feature (5 percentage point increase), there was a higher proportion of constructive new content edits with a reference added (23.4 percentage point increase). As a result, we observed an overall increase in the quality of new content edits.

There was a -8.6% decrease in the revert rate of all new content edits comparing edits where Reference Check was shown in the test group to edits that were eligible but not shown Reference Check in the control group.

Constructive Retention Rate

  • Contributors that are shown Reference Check and successfully save a non-reverted edit are 16 percent more likely to return to make a non-reverted edit in their second month (31-60 days after).
    • This increase was primarily observed for desktop edits. There was a non-statistically significant difference observed on mobile.

Guardrails

Edit Completion Rate

  • We observed no drastic decreases in edit completion rate from intent to save (where Reference Check is shown) to save success overall or by wiki.
  • Overall, there was a 10% decrease in edit completion rate for edits where Reference Check was shown.
    • There was a higher observed decrease in edit completion rate on mobile compared to desktop. On mobile, edit completion rate decreased by -24.3% while on desktop it decreased by -3.1%.

Block Rate

  • There were decreases or no changes in the rate of users blocked after after being shown Reference Check and publishing an edit compared to users in the control group.

False Negative Rate

  • There was a low false negative rate. Only 1.8% of all published new content edits in the test group did not include a new reference and were not shown Reference Check.

False Positive Rate

  • 6.6% of contributors dismissed adding a citation because they indicated the new content being added does not need a reference. This was the least selected decline option overall.

Test design

11 Wikipedias participated in the test. At each wiki, 50% of users were randomly assigned to a test group and 50% were assigned to a control group.

Users in the test group were shown the Reference Check notice prompting them to decide whether the new content they were adding need a reference (if they had not already added one themselves).

User in the control group were shown the default editing experience, even if they did not accompany the new content they were adding with a reference.

Timing

This analysis was completed on 16 April 2024 and analyzed engagement data at the 11 participating wikis from 18 February 2024 through 4 April 2024.

Evaluating impact

The viability of the features introduced as part of the Edit Check project depends on the impacts it causes and averts.

This section describes the:

  1. Impacts the features introduced as part of the Edit Check are intended to cause and avert
  2. {{[[Template:|]]}}
  3. Evaluation methods we will use to gather the data necessary to determine the impact of a given feature
Desirable Outcomes
ID Outcome Data Evaluation Method(s)
1. Increase the quality of edits newcomers and Junior Contributors editing from within Sub-Saharan Africa publish in the main namespace Decrease in the proportion of published edits that add new content and are reverted within 48 hours or have a high revision risk score

Comments/reports from experienced volunteers about the quality of edits Edit Check is activated within

 A/B test, qualitative feedback (e.g. talk page discussions, false positive reporting)
2. Increase the likelihood that newcomers and Junior Contributors editing from within Sub-Saharan Africa will accompany the new content they are adding with a reference Increase in the percentage of published edits that add new content and include a reference

Increase in the percent of newcomers or Junior Contributors from SSA that publish at least one new content edit that includes a reference

Increase in the likelihood that someone includes a reference the next time they contribute new content.

 A/B test
3. Newcomers and Junior Contributors editing from within Sub-Saharan Africa will report feeling safe and confident making changes to Wikipedia Newcomers and Junior Contributors find the feedback and calls to action Edit Check presents them with to be:
  1. Helpful
  2. Supportive
  3. Motivating
 Qualitative feedback via channels like:Community Calls, talk pages, event organizers, etc.
4. Experienced volunteers will independently audit and iterate upon Edit Check's default configurations to ensure Edit Check is causing newcomers and Junior Contributors to make productive edits.
5. Newcomers and Junior Contributors will be more aware of the need to add a reference when contributing new content because the visual editor will prompt them to do so in cases where they have not done so themselves. Increase in the percent of newcomers or Junior Contributors from SSA that publish at least one new content edit that includes a reference. A/B test
Risks (Undesirable Outcomes)
ID Outcome Data Evaluation Method(s)
1. Edit quality decreases Increase in the proportion of published edits that add new content and are reverted within 48 hours or have a high revision risk score

Comments/reports from experienced volunteers about the quality of edits Edit Check is activated within

 A/B test, qualitative review and feedback
2. Edits become more difficult to patrol because unreliable citations are difficult to detect Significant increase in the percentage of new content edits new and developing volunteers make that include a reference

Comments/reports from experienced volunteers about the quality of edits Edit Check is activated within

 A/B test, qualitative review and feedback
3. Edit completion rate drastically decreases Proportion of edits that are started (event.action = init) that are successfully published (event.action = saveSuccess). A/B test
4. Edit abandonment rate drastically increases Proportion of contributors that are presented Edit Check feedback and abandon their edits (indicated by event.action = abort and event.abort_type = abandon) A/B test
5. Blocks increase Proportion of contributors blocked after publishing an edit where Edit Check was shown is significantly higher than edits in which Edit Check was not shown A/B test
6. High false positive or false negative rates Proportion of new content edits published without a reference and without being shown Edit Check (indicator of false negative)

Proportion of contributors that dismiss adding a citation and select "I didn't add new information" or other indicator that the change they are making doesn't require a citation

 A/B test, qualitative feedback received from volunteers about the accuracy and usefulness of Edit Check's current configuration
7. Edit Check is too resource intensive to scale Efficiencies do not emerge over time making each new Edit Check as "expensive" to implement as the first one Qualitative assessment by the Editing team

Deployment process

Please see Deployment status#Deployment process.

Background

Volunteers throughout the movement have a long history of working to:

The Editing Team and this project have been inspired by these efforts, some of which are listed below. If there is a project or resource you think we should be aware of, please add it here!

Initiative Description Initiator(s)
Wikimedia-l: Mopping with the tap open A discussion wherein Romaine describes a pattern of people who are new adding information to Wikipedia without citing source(s). nl:Gebruiker:Romaine
Model: meta:Eno-Prompt A project to train a fine-tuned open-source LLM (large language model) that will detect disinformation based on linguistic analysis.
Paper: Counter-Misinformation Dynamics: The Case of Wikipedia Editing Communities during the 2024 US Presidential Elections Recommendations to make Wikipedia more resilient to misinformation. Includes recommendations relevant to Edit Check.
Wish: Make editnotices display as pop-ups Make people aware when they are at risk of changing an article volunteers consider to be about a contentious topic User:Theleekycauldron
Corrector ortográfico, Helferlein/Rechtschreibprüfung, ויקיפדיה:סקריפטים/בודק איות, Revisor_ortográfico A tool that checks pages loaded in the browser against a list of common spelling errors. Poco_a_poco, Benutzer:APPER, :משתמש:ערן, Elisardojm
Error Finder A tool to find common errors in Persian texts Reza1615
Wish: Warn when large amount of content has been copy-pasted A wish to warn people who are pasting text into Wikipedia and to annotate edits in which this occurs so that patroller can consider this as they are reviewing edits/looking for edits to review. Matěj_Suchánek
fr:Projet:Articles sans sources (along with 3 other wikiprojects) A WikiProject intended to add sources to articles that need them
CopyPatrol Tool that allows you to see recent Wikipedia edits that are flagged as possible copyright violations Community Tech Team
paper: Automatically Neutralizing Subjective Bias in Text Method for automatically bringing inappropriately subjective text into a neutral point of view ("neutralizing" biased text). Reid Pryzant, Richard Diehl Martinez, Nathan Dass, Sadao Kurohashi, Dan Jurafsky, Diyi Yang
Wikipedia:Citation watchlist User script that adds visual indicators to watchlist and recent changes entries when unreliable sources are added to articles. Harej, Ocaasi
Internet Archive Reference Explorer Explore references included in Wikipedia articles via a range of criteria
WikiScore A tool created to validate edits and count scores of participants in wikicontests.
Earwig's Copyvio Detector This tool attempts to detect copyright violations in articles. The Earwig
CiteUnseen A user script that adds categorical icons to Wikipedia citations, providing readers and editors a quick initial evaluation of citations at a glance. SuperHamster
Credibility bot Monitors and collects data on source usage within Wikipedia articles Harej
Salebot (French Wikipedia) A counter-vandalism bot that uses regex to identify issues.
Edit intros (English Wikipedia) A message is shown automatically when editing a page categorized as either Category:Living people or Category:Possibly living people.
Make edit notices more visible in Visual Editor How might we make it so people who are in the midst of an edit are likely to see and "internalize" the information that is currently presented within Edit Notices? User:Stjn
Internet Archive Reference Explorer Automatically detect source quality Ocaasi
Wish: Reference requirement for new article creation Require new article to include references User:Mega809
2=Edit Notices Enables individual volunteers and projects to display a custom notice above the edit form, depending on the page, namespace, or other circumstances.
Page notices
Maintenance templates
mw:Extension:AbuseFilter Enables privileged users to set specific actions to be taken when actions by users, such as edits, match certain criteria.
mw:Extension:Disambiguator Displays a notification in the 2006/2010 wikitext editor whenever one adds a link to a disambiguation page. Community Tech
mw:ORES Halfak (WMF)
Suggested Edits
CiteHighlighter Highlights 1800 sources green, yellow, or red depending on their reliability. Novem Linguae
Checkwiki Helps clean up syntax and other errors in the source code of Wikipedia Stefan Kühn, Bgwhite
Edit Diff Tagging Showcases all the different tags that can be automatically determined (generally via basic heuristics) for a given Wikipedia edit diff. Isaac (WMF)
CivilityCheck A project to evaluate the civility in the comments of Wikipedia discussions in order to address the problem of abuse that leads to declining editorship within the Wiki community. Deus Nsenga, Baelul Haile, David Ihim, and Elan Houticolo-Retzler
BOTutor A bot that sends a message to people who attempt to publish an edit that triggers an existing set of rules ValeJappo
Gadget-autocomplete.js ערן
Text reactions A proposal that would make it possible for the editing interface to react to what the people enter in the editing area SD0001
Editwizard A step-by-step process for guiding newcomers to source the content they are attempting to add to Wikipedia articles Ankit18gupta, Enterprisey, Firefly, and SD0001
Headbomb/unreliable "The script breaks down links to various sources in different 'severities' of unreliability. In general, the script is kept in sync with WP:RSPSOURCES, {{Predatory open access source list}}, WP:NPPSG, WP:SPSLIST (not fully implemented yet) and WP:CITEWATCH, with some minor differences." Headbomb, SD0001
The Wikipedia Adventure Game based on the tech of mw:Extension:GuidedTour that teaches basic wikitext markup and the rules about reliable sources and neutral point of view. Research into its effectiveness is described at m:Research:Impact of The Wikipedia Adventure on new editor retention. Ocaasi
w:Help:Introduction The primary tutorial for new editors at English Wikipedia, covering both policies and technical how-to for VisualEditor and wiki markup. Most recently overhauled in late 2020 and more actively maintained than TWA. Sdkb, Evolution and evolvability, and others
User:Phlsph7/
HighlightUnreferencedPassages		 A user script to highlight passages that lack references with a red background. Its main purpose is to help users quickly identify unreferenced passages, paragraphs, and sections in mainspace articles and drafts Phlsph7
Wish: Add notice to the visual editor that unsourced edits may be reverted A notice in the "Publish changes" dialogue of the visual editor that states that unsourced edits will be reverted User:Lectrician1
Wish: Warn when adding a url reference that matches the SpamBlacklist Warn when the url added as reference is registered in the SpamBlacklist, and thus prevent the warning from appearing when saving the page. User:DSan
Edit FIler #686 Edit Filter that is triggered when a new user possibly adding unreferenced material to BLP User:Rich Farmbrough
WikiLearn Platform for training
DannyS712/copyvio-check.js Automatically checks the copyvio percentage of new pages in the background and displays this info with a link to the report in the 'info' panel of the Page curation toolbar. DannyS712
XLinkBot A bot that warns people who have added an external link that is inappropriate in some way. Versageek, Beetstra

See also

References

  1. ↑ Growth Team: IP editing Research Report
  2. ↑ The Tyranny of Structurelessness
  3. 1 2 Community Insights 2021 Report
  4. ↑ Regional Quarterly Learning Sessions (June 2022, google document)
  5. ↑ Superset
  6. ↑ T327959
  7. ↑ T324730
  8. ↑ Where "viability" in this context refers to a feature being fit for being scaled to all projects as determined by the extent to which it has been proven to have a net positive impact on wikis and the volunteers who build and maintain them.
  9. ↑ Emphasis on "help" seeing as how all decisions will depend on a variety of data, all of which need to be weighted and considered to make informed decisions.
  10. ↑ Emphasis on "help" seeing as how all decisions will depend on a variety of data, all of which need to be weighted and considered to make informed decisions.
  11. ↑ T325838 - Finish Edit Check measurement plan proposal
  12. 1 2 3 At every project where Edit Check is available, volunteers will be able to use the editcheck-reference-activated tag to review edits where the reference check is shown to people in the process of publishing an edit. Learn more about Edit Check tags.
  13. 1 2 3 4 5 6 7 8 9 [Analysis] Run an A/B test to evaluate Edit Check (references) impact
  14. ↑ T325851 - Conduct pre-mortem for Edit Check project
  15. ↑ In addition to existing feedback channels (Phabricator, talk pages, etc.) there will be a minimum of two additional ways for people to share feedback about Edit Check: A) reporting edits that you think Edit Check should not have been shown within and B) declining to add a reference mid-edit by indicating you think Edit Check was shown when it shouldn't have been.