
Teahouse:pilot report
» pilot planning, learnings and next steps from the Teahouse project
The Teahouse Project was piloted on English Wikipedia in the first half of 2012. The project was planned in December 2011 and January 2012, launched in February 2012, and results were measured for 13 weeks between February 27th and May 27th 2012.
This pilot project report will cover the following areas:
A complete metrics report containing all data from surveys and queries used to assess impact and results of the Teahouse is reported separately.
"I have mentioned before and I will state again that I think the Teahouse is the best thing I have seen happen to the encyclopedia since I started editing." —Ryan Vesey, volunteer Teahouse host and Wikipedian with 10,400 edits
| Goal/Target | Impact |
|---|---|
| Improve editor retention among promising new editors, especially women | 28% of Teahouse participants are women, up from 9% of editors on Wikipedia in general. 2 weeks later, 33% of Teahouse guests are still active on Wikipedia, as opposed to 11% from a similar control group. |
| Give new users a place where they can easily, safely and comfortably ask questions and receive explanations | "I liked how it was a person who answered my question just not a machine. I also liked how quick an answer was put up. I also like how they wrote my answer so I could understand what it meant."
"low-key, friendly smart people; I'm learning from the Teahouse." "(Teahouse is) making something like Wikipedia so simple." |
| Model a social approach to new user support, help and socialization into the community distinct from existing 1 on 1 and self-support options. | "I received a number of really helpful responses from many editors about my question about translated sources."
"It's good to see there are so many people willing to help each other and the Wikipedia project" |
| Provide a space that is specifically designed for new users which is visually appealing, interactive, and communicates a clear sense of purpose. | "I like the design and friendliness"
"I like the emphasis on simplicity so new users don't get frustrated" "I love the atmosphere and how the Teahouse works." |
| 20 hosts active throughout the pilot | 24 hosts participated at any given point in the pilot, and an average of 14 hosts were active in the Q&A forum each week |
| 75 new editors utilizing the Teahouse weekly | 409 new editors participated during the entire pilot period, approximately 39 new editors participated in the Teahouse per week |
| 50 to 100 new editors invited to participate in the Teahouse daily | 7339 invites were sent total, an average of 80 per day |
| A measurable increase in engagement and retention of new editors should be apparent | New editors who participate in the Teahouse edit 10x the number of articles, make 7x more global edits, and 2x as much of their content survives on Wikipedia compared to the control group. 33% of these participants are still making edits 2 weeks later, and they create 2x the content of their control-group counterparts |
In a series of conversations and brainstorming sessions held at WMF with volunteers and staff about solutions to closing the gender gap on Wikipedia, Gender Gap Fellow Sarah Stierch and Head of Fellowships Siko Bouterse began to form an idea for a social cafe-like space that might help support more new editors on-wiki, in hopes that offering a different support style could encourage more women to stick with editing. Based on feedback in these sessions, we put together a project to test the idea. Research Fellow Jonathan Morgan and Designer Heather Walls were recruited to the project to lend research, analytics, usability, and design expertise, and the Teahouse project was born.
Next, the research phase began. This work included:
All of this research was developed into an online resource on meta to provide not only clarification, but justification as to why the Teahouse would be developed into the unique space it is today.
We built the project plan on meta wiki so that we could gather feedback from interested volunteers at every point in the process. Once we'd chosen English Wikipedia as the logical place to pilot the approach, the plan was also copied to English Wikipedia for easy access by that community.
Planning started with defining clear goals, targets, and metrics to measure the pilot's success.
We then defined the kind of users we were aiming to have impact on, and the features that would best serve their needs. To help determine what were the most critical components to include in the pilot, we prioritized our set of user scenarios, and then worked back from these to a set of required features for the Teahouse.
We built metrics into the project early and pulled them as often as possible to track activity and outcomes of the pilot and help us iterate on the concept throughout the experiment. Metrics were collected about every 2 weeks and results were published at m:Wikipedia:Teahouse/Host lounge/Metrics as well as in the host lounge.
We tracked measurements in the following key areas:
Throughout the pilot, the Teahouse project team aimed to be as accessible to the community as possible. We also worked with WMF Global Communications Manager Matthew Roth to develop a communications strategy aimed at audiences outside the movement. We used a wide array of communication channels in order to reach a wide array of audiences.
Communication channels include:
Members of the project team all wore many hats, but each took on primary roles most suited to our skillsets. By assembling a team where each member contributed in the areas he or she knew best, we were able to cover a lot of ground. For example, having an accomplished designer work on the design aspects of the project meant the Teahouse had great design and freed up other team members to focus on what they were best at. Some of our specific focuses were:
Many others volunteered their time and skills to the Teahouse pilot as well:
The Teahouse design is something that makes the space unique. To create a visually welcoming and engaging space, design was incorporated into our process from the very beginning.

The first step in design is often to create a structuring concept distilled from the intended audience, setting, and message of the project.
At the Teahouse we want to soften a user's first contact with Wikipedia, to allow new editors to connect and find community before being put-off by warning templates and technical difficulties. We set out to "reduce the learning curve for new editors," a concept from Wikimedia's strategic plan to improve interface and tools for community health.
The Teahouse is designed to give people a way to emotionally connect with Wikipedia. Decisions were made based on the following considerations:
To combine these priorities with the concept of a "Teahouse", we began collecting a set of colors and images. Traditional teahouse imagery was selected and a palette was constructed. The internal pattern of a traditional Japanese tea room influenced the introduction of guests and hosts to the Teahouse; guests from one side and hosts from another. The garden path is represented in the graphics derived from this image; as well as the door and roof, creating a visual parallel to a welcoming approach.
Elements, interactions, and placement on each page were designed with the following priorities in mind:
User responses to our design choices have been very positive. In survey responses and other discussions, participants often mentioned the Teahouse design:
"I like the emphasis on simplicity so new users don't get frustrated"
"The interface is pretty."
"I like the design and friendliness"
"The interface for asking a question is very slick"
"I love the atmosphere and how the Teahouse works."
"I liked how you could look at other questions that you might have not known the answers to."
"It was warm and friendly."
"I like the name, and I really like that the language is inviting. It is unlike many of the "Help" pages that read more like a technical document with the occasional "we're here to help each other" message. Teahouse overall is very inviting and proves to be quite useful."
"it was very easy to use and beautifully designed"
"How easy it is to use"
"The interface being so simple compared to normal talk page editing"
While the Teahouse was designed to look and feel distinct from other pages on Wikipedia, it was implemented almost entirely within the standard parameters of Mediawiki as configured on English Wikipedia. The emphasis on simplicity was partially pragmatic: we were on a short development timeline and had few technical resources at our disposal. Keeping the implementation of Teahouse simple helped us to:
In order to make it easier for other wikis to create Teahouses of their own, we're creating a Teahouse menu that includes the documents, scripts, interface elements and external resources we used to create WP:Teahouse.
| More research findings are available in the metrics report. |
The only change to the core Mediawiki code, and the only non-standard functionality intended for use by Teahouse guests, was a Javascript widget on the Q&A page called the "Teahouse gadget", which was built by Werdna and supported by Kaldari. The gadget pops-up an inline edit form when the guest clicks the "Ask a question" button on the Q&A page, and posts questions as a new section on the top of the board, rather than the bottom of the page. The gadget is enabled by default for all Wikipedia users, but can be de-activated on a per-user basis on their 'preferences' page. A fallback option was provided for users who did not have Javascript enabled on their browsers, which used the default "new section" functionality.

| More research findings are available in the metrics report. |
We developed templates to give new editors step-by-step instructions for creating a simple introduction about themselves on the Teahouse guest page. We also asked all active hosts to create an introduction on the host page.
Teahouse was divided into a customer-facing space and a "back of the house" space in order to keep things simple for new editors while giving hosts everything they needed to do their jobs well.

We created an automated daily report of new editors who had made at least 10 edits within their first 24 hour period or who had made at least 20 edits over at least three editing sessions within their first four days. This report included links to the contribution histories and user talk pages of these new editors, as well as a link to send the editor an email if that editor had Wikipedia's "Email this user" feature enabled. Teahouse hosts used this report to select which new editors to invite. See the invitations section of this report for a more detailed description of the invite process.
We developed a bot, HostBot, that uses scripts and database tables on the InternProxy database to perform the following routine tasks on Teahouse pages:
Teahouse hosts also developed a variety of useful, opt-in userscripts to streamline ‘back end’ processes such as inviting and notifying guests, and facilitate easier navigation of Teahouse pages for these power-users. These scripts included
A host lounge was built as a coordination area for hosts. See Hosts section below for more info on the lounge and host resource pages).
We included a 'wishlist' for project members and hosts to suggest additional functionality or features for Teahouse, or to volunteer to contribute these features. Although this list experienced relatively little traffic, we believe that it effectively communicated the project team's receptivity to community input and encouraged several technically savvy hosts to contribute to the development of useful tools and features.
To be effective, Q&A forums and online peer-support spaces must be populated. We knew we needed to staff the space before bringing new editors into it, and make sure there would be consistent participation from experienced editors throughout the pilot. Participation of Wikipedia volunteers and Teahouse project fellows has been critical for making the Teahouse a welcoming and helpful space. While Teahouse Hosts formed the basis for welcoming and inviting new editors to participate in the space, Wikipedia editors of all experience levels have contributed to the Teahouse's growth.
| Total Teahouse participants | 586 |
| New editors who participated | 398 |
| Q&A board participants | 442 |
| Questions posted | 514 |
| Guest intros created | 215 |
| Invitations sent | 7339 |
Teahouse Hosts serve a critical function of making the Teahouse welcoming, active, and helpful. These volunteers are a first point of contact for new editors who visit the Teahouse. Hosts have not only provided answers and a friendly "face" to new editors, but, they have also helped improve the pilot with their opinions, input, templates and scripts.
| Hosts at launch | 25 |
| New hosts joining during the pilot | 17 |
| Hosts who retired | 16 |
| Hosts at end of pilot | 24 |
| Average hosts active in Q&A forum per week | 14 |

To ensure we'd have enough friendly and experienced editors staffing the space at launch, we developed an application process similar to the Online Ambassador process: potential hosts answered a series of questions about their experience in helping new editors. Stierch actively recruited editors with a history of positive Wikipedia interactions with new editors.
We had 25 applicants, and all were chosen to be the inaugural Teahouse pilot hosts.
"I really like the fact that Teahouse hosts were searched out, and the board was not just created with the attitude of 'If we build it, someone will help'." —Experienced Wikipedian, wrapup survey
While the application process proved useful for launch purposes, it was a bit cumbersome and not transparent or scalable enough for recruiting future hosts. As the Teahouse garnered more attention on-wiki, a process for those interested in being Teahouse Hosts had to be developed. An on-wiki space (formally called "Your hosts" which is now defunct) for interested participants was created which explained the roles and responsibilities of Teahouse hosts during the pilot, listed current hosts, and asked interested parties to sign up to participate. This space and the process by which hosts are recruited and added to the project still has obvious room to grow, so that we ensure Teahouse Hosts can rely on each other to represent the project well, without creating unnecessary walls between hosts and the wider community.
Throughout the pilot a number of hosts participated in the project. The following are a list of hosts who contributed to the project (not including Teahouse project staff):
* is a Host that has retired or has taken a break from the project.
A number of Hosts came and went throughout the pilot. To ensure that the space stayed active and guests didn't find too many hosts that weren't actively available to them, we contacted Hosts after three weeks to a month of inactivity, and if they indicated being unable to continue we removed them from the hosts page. Hosts are welcome to return when they become available again.
During the pilot, the main hosts tasks were to:
Participation of individual hosts in individual activities and in the project as a whole fluctuated over the course of the pilot, and as some hosts dropped off others took up the role.
Before the launch of the Teahouse, we started to build out the Host lounge, which would be further developed by the hosts themselves. This space provides tools and resources for hosts and its talk page serves as the main brainstorming and discussion hub for hosts. This space also served as a pre-launch training space.
The space is broken up into different sections:
In preparation for the launch, we started discussions with hosts about their early experiences as editors, in order to help them connect with the early experiences that inspired them to continue editing and inspire them in their host duties.
Ultimately, some areas of the space proved to be more used than others during the pilot period. The invite guide and host tips were used more frequently than the other resources. The talk page of the lounge was a very useful discussion hub.
In keeping with the spirit of wiki, anyone can participate in the Teahouse. Editors of all experience levels have asked and answered questions and added themselves to the guest page. When the Teahouse launched, we received more interest than expected from other experienced editors who also wanted to answer questions in the Q&A forum. This was a challenge for the project, because Teahouse Hosts had been through training about their unique roles and had agreed to do things a bit differently than on other Wikipedia pages. For example, hosts were asked to first greet and welcome guests before addressing their question and to avoid wiki-speak and linking to archives or lengthy documentation. While we assumed good faith with all contributors, many of the experienced editors, who were excited to help, were unaware of the unique style of the Teahouse.
The team wondered: How do we preserve a very newbie-friendly tone in the Teahouse without becoming a club from which some long-time Wikipedians feel excluded?
Community input continues to be key in developing the Teahouse. Throughout the crafting of the Teahouse and the launch of the pilot on Wikipedia the community participated on the meta talk page and the Teahouse talk page. Community fellows and Wikimedia staff engaged with the community to explain the reasoning behind certain aspects of the project and understand and respond to concerns about the approach on topics including:
As this pilot period wraps up, hosts and other experienced Wikipedians will continue to be critical stakeholders in the future of the Teahouse.
| More research findings are available in the metrics report. |
Personalized outreach tools are a major aspect of the Teahouse experience. These tools were used to by hosts and other Wikipedians to invite new editors to the Teahouse. Invite templates were created to have a personalization aspect and a unique color scheme that fit in with the look and feel of the Teahouse. Invitations would be sent to anyone that the inviter thought of as a promising new editor (no vandals, please!).
An invite guide was created for host use, which showcased the best ways to find and invite new users, including:
These were the most valuable places to find new editors. Hosts and editors also invited new editors via their own watchlists and other means. The overall response rate to these invitations was about 4.5%.
Teahouse project Fellows worked with the Wikipedia Education Program to explore outreach opportunities with English language classes which participated in the Spring 2012 semester. We hoped to provide some backup to online and campus ambassadors who often were just one person supporting many students, and we also hoped to encourage and retain these often prolific and promising new editors. WMF's Global Ed team provided a list of classes that would also have a larger number of women students, allowing us to aim towards experimenting with the Teahouse as a space to support female editors. Nine classes were chosen by Global Ed staff to participate. We reached out individually to each professor to let them know about the Teahouse and offer the space to their students.
Only two classrooms appear to have sent students to the Teahouse, though, and these classes were led by professors connected with Teahouse project leaders, not affiliated with Global Ed.
| More research findings are available in the metrics report. |
In order to determine whether participating in Teahouse has a positive impact on an editor's likelihood of continuing to edit Wikipedia or on their other editing behaviors, we analyzed the outcomes for three groups of editors in three different 'conditions'--two different control groups and an experimental group--drawn from editors who joined between February 25th and May 19th.
We measured the level and type of activities these editors performed after the date they were invited to Teahouse. We highlight particularly notable findings from this study below. A detailed report is available in the Pilot metrics report.
| More research findings are available in the metrics report. |
Overall, new editors really enjoyed their Teahouse experience. 71% of new editors surveyed reported being satisfied or very satisfied with the Teahouse.

New editors also had many positive things to say about Teahouse:
"I'm very happy to be a Guest here and to have the Teahouse's support in my editing efforts." —User:AsadUK200
"I like the name, and I really like that the language is inviting. It is unlike many of the "Help" pages that read more like a technical document...Teahouse overall is very inviting and proves to be quite useful." —Teahouse guest
"Overall the execution of this idea has been brilliant...the teahouse is what changed me from being a gawker into being a (very newbie) editor." —Teahouse guest
Two new editors' personal experiences with Teahouse help illustrate the potential of this approach:

Wikipedian Doctree created his account in 2011. He didn't edit much at first, but in March 2012 he started to become more active. As he started to make his first edits, a welcome template appeared on his talk page. "Gad, the thing has 65 links to WP articles!" he exclaimed when reflecting on his early editing experiences in April 2012. "I felt overwhelmed rather than welcomed."
Doctree didn't let the overwhelming template deter him, instead he decided to seek help improving his editing skills so he could contribute articles as part of WikiProject Birds. He decided to explore the adoption program. Despite efforts to reach out to mentors with an interest in science, he discovered that most were inactive. While not able to find a mentor, he came across Pluma's adoptee homework assignments. He found a lot of use in this, and learned "more in one night than in two weeks of muddling through the links in the official Welcome template."
Doctree used what he'd learned to write his first article Living Bird. He then started revising the article Chimney Swift. He wrote the revision in his sandbox, and then didn't quite know how to move it out to the mainspace. Doctree found the Teahouse and asked for help. "I had one simple question: How to get the revision into mainspace? Sarah provided a simple, straightforward answer, not a referral to WP pages. Thanks. The experience you offer, as Hosts, makes a huge difference. Thanks again."
But Doctree's participation in the Teahouse didn't stop with asking questions. Today, he gives back to his peers by answering other questions at the Teahouse - he's answered 13 questions so far. Doctree is a good example of the Teahouse's potential to build community for new editors and how peer-to-peer support systems like Teahouse can be self-sustaining, because as new editors get help and become more confident, they too can help others. Doctree plans to retire from his day job soon, he notes a goal on his userpage: "Retire in July 2013 and become a real Wikipedian."
Gmhayes4 joined Wikipedia approximately four months ago. You can call her Gmhayes4, or you can call her G, as her friends prefer. As a freelance writer and marketing consultant, G wanted write an article about a client, April Masini. Masini's article, like many articles written new editors with a conflict of interest in their chosen topic, had seen it's fair share of bad days - it was created, deleted, created again, nominated for deletion, and so forth.
Gmhayes4 appeared on the Teahouse's daily database report of promising new editors. Sarah saw how motivated she was to contribute and posted a Teahouse invitation on her talk page, as well as an email invite. Moments later, G had written back to Sarah, stating her conflict of interest and how she needed help. The article was being drafted in G's sandbox, and Sarah started working with G to learn about conflict of interest issues and teach G how to edit ethically within Wikipedia's policies of neutrality, COI and reliable sourcing.
As Sarah didn't have enough time to help G one-on-one as she would have liked, she soon directed G towards the Teahouse for further support. G was one of the first visitors at the Teahouse, asking a question about how to reference a PDF. Host Bilby jumped in to lend a hand and G was beyond impressed with the fast assistance: "Thank you so much, Bilby! That is exactly what I needed to move forward."
G continued to participate at the Teahouse. She followed other conversations at the Teahouse and chimed in when she had similar concerns or comments. With guidance and perseverance, G not only learned how to edit collaboratively on Wikipedia, but she also created a neutral, well sourced article about April Masini which has held its own (without nomination for deletion) since late March. G reflected on her experience at the Teahouse and how it helped her write a neutral article: "Hello Teahouse! Wow! I've learned so much just by reading the articles listed in this talk...[I] am having a blast creating the article! It's been a challenge to reign in my excitement about [April Masini], and stay objective in my article."
While G is busy with real life: "working, not working, reading, writing, and graphic art," she's also making edits here and there about subjects like social media. G is a great example of an editor that is often turned away by Wikipedians because of her conflict of interest with subjects. However, with patience, careful guidance, and the support of the Teahouse, which led to broader support in the community, G has become an active Wikipedian with over 220 edits in four months.
"Seeing editors who have got into problems turn themselves around and become productive is good too." —Experienced editor
| More research findings are available in the metrics report. |
Teahouse feedback from experienced editors (Hosts and non-Hosts) has been highly favorable. 70% of experienced Wikipedians said they were "satisfied" or "very satisfied" with their Teahouse experience. Only 5% of experienced Wikipedians said they were "very dissatisfied" or "dissatisfied".
Experienced editors were impressed with the friendly, interactive and helpful landscape that the Teahouse provided:
"The friendly atmosphere - we could do with more of that on wiki."
"I liked that the Teahouse is a ground for new users so they have lots of support, and that hosts and new users interact a lot with each other."
"I believe editors are much more friendly with newcomers, its the opposite of don't bite the newcomers, its more like give the newcomers a sucker and a high five."
"I like the emphasis on simplicity so new users don't get frustrated."
"I think Teahouse is a wonderful way for new and inexperienced editors to get involved in Wikipedia while making friends with both hosts and other editors alike. Questions are answered quickly and thoroughly and most of the time the editors find the information extremely helpful."—LuK3
"The data has proven that the Teahouse is retaining editors and I think it is the friendly nature of the Teahouse that does that." —Ryan Vesey
Experienced editors also liked the design, which is a refreshing change to the standard Wikipedia fare:
"The interface is pretty."
"The interface for asking a question is very slick."
While some experienced editors liked the design, others were uncomfortable with the top-posting mechanism to the Q&A board:
"Also, as an aside, I know this is a losing battle, but the "new threads on top" thing is still a bad idea, but I know that people who are more important than me have decided that it will go that way, so I guess we are stuck with it. Still feel the need to lodge my standing complaint on that one."—Jayron32
It's also a nice change to the standard help documents:
"A very specific answer to a specific question. Sometimes the regular help and policy pages are so broad that even after hours of reading, I still didn't find an answer to a specific question."
Carolmooredc appreciated the atmosphere of the Teahouse, as well, believing it provided respite from the at times contentious culture of Wikipedia:
"A little haven of tranquility away from the battles outside. :)"
Experienced editors also found the Teahouse an important tool in improving their own skills in communication and mentorship. Gareth Griffith-Jones started using the Teahouse to ask questions and answer them, "I consider Teahouse a very valuable tool." He was happy to see his "protege" start improving their own participation skills, "my protogé/e uses the Discussion page [now], whereas the experienced user relies solely on lengthy edit summaries."
"The other big plus for me has been how much I've learned either from others answers or in researching answers myself."—NtheP
Host Rosiestep noticed the successes and hopes they would be long term:
"I feel proud to be associated with the Teahouse pilot project; talk page comments from new editors abound with notes about how they appreciate the service provided. If just one of those folks becomes prolific as a result of the Teahouse encounter, then job well done here."

The unique look, feel and supportive environment of the Teahouse aimed to not only retain new editors but specifically support new female editors. Women who participated in the project have responded positively to it. Although exact gender numbers are never simple to benchmark on Wikipedia, we can use survey responses as an approximation. 28% of guests surveyed indicated their gender was female. This is higher than the 9% figure currently believe to represent female editors on Wikipedia based on wider editor surveys, but its still not enough to close the gap.
Specific tactics were used to reach out to new women editors:
Women who viewed or participated in the project as either new or experienced editors responded positively, indicating that the Teahouse does have potential to impact the gender gap.
Five women participated as hosts throughout the pilot. While female hosts were highly supportive of the project, many of them did not participate as often as men who were hosts.
In discussions with hosts, the following themes emerge that may factor into female host participation in the Q&A forum:
Women's contributions as hosts is imperative to making the Teahouse a welcoming and inclusive place for all editors, so there is clearly more work to be done in order to encourage and retain more female hosts at the Teahouse.
The Teahouse has demonstrated potential to positively impact the new editor experience, retain and incorporate more editors into the community, and compliment other approaches to editor support in the Wikipedia ecosystem. The Teahouse project was and continues to be worth doing.
A successfully completed pilot is not the end of the Teahouse story, though. From feedback of those involved in the project, it is apparent that more work is needed in order to scale and sustain the project, solve remaining issues, add additional features, and put new systems in place to ensure that Teahouse will continue to be of value to Wikipedians for years to come. Some of this work can and should be completed by volunteers, but some of this work will benefit from continued WMF investment.
The WMF project team would like to continue to partner with the community in order to help take Teahouse to the next level. The WMF Fellowship Program intends to continue to invest in this project, support the volunteers involved, and measure outcomes over time so that we can all keep learning from the Teahouse.
Opportunities for continued project development under consideration include:
During June 2012, the Teahouse team will be working on a new project plan for Phase 2 of the Teahouse project, which is expected to run July-September. This plan will be based on the lessons learned that are documented in this report, feedback from Teahouse participants, and ongoing community conversations. We hope that everyone will get involved in contributing to the post-pilot phase in order to continue to make Teahouse and Wikipedia a more welcoming and productive place for all editors to contribute to the sum of all human knowledge.