Usenet (/ˈjuːznɛt/), short for User's Network, is a worldwide distributed discussion system available on computers. It was developed from the general-purpose Unix-to-Unix Copy (UUCP) dial-upnetwork architecture. Tom Truscott and Jim Ellis conceived the idea in 1979, and it was established in 1980. Users read and post messages (called articles or posts) to one or more topic categories, known as newsgroups.
Usenet resembles a bulletin board system (BBS) in many respects and is the precursor to the Internet forums that were developed after the introduction of the World Wide Web. Usenet is distributed among a large set of interlinked news servers that store and forward messages to one another via "newsfeeds". Individual users may read messages from and post to a preferred news server, and those posts will automatically be forwarded to any other news servers peered with the preferred one, while the preferred server will receive any news its peers have that it currently lacks. This results in the automatic proliferation of content posted by any user on any server to any other user subscribed to the same newsgroups on other servers.
Usenet is culturally and historically significant in internet culture, having given rise to, or popularized, many widely recognized concepts and terms such as "FAQ", "flame", "sockpuppet",[citation needed] and "spam". In the early 1990s, shortly before access to the Internet became commonly affordable, Usenet connections via FidoNet's dial-up BBS networks made long-distance or worldwide discussions and other communication widespread. Use of Usenet as a text communication medium started to decline in the late 1990s, but it is still used to share files between users.
Introduction
Usenet was conceived in 1979 and publicly established in 1980, at the University of North Carolina at Chapel Hill and Duke University, over a decade before the World Wide Web went online (and thus before the general public received access to the Internet), making it one of the oldest computer network communications systems still in use.[citation needed] It was originally built on the "poor man's ARPANET",[clarification needed] employing UUCP as its transport protocol to offer mail and file transfers, as well as announcements through newly developed news software. The name "Usenet" emphasizes its creators' hope that the USENIX organization would take an active role in its operation.
The articles that users post to Usenet are organized into topical categories known as newsgroups, which are themselves logically organized into hierarchies of subjects. For instance, "sci.math" and "sci.physics" are within the "sci.*" hierarchy; "talk.origins" and "talk.atheism" are in the "talk.*" hierarchy. When a user subscribes to a newsgroup, the news client software keeps track of which articles that user has read.
In most newsgroups, the majority of articles are responses to some other article.[citation needed] The set of articles that can be traced to one single non-reply article is called a thread. Most modern newsreaders display the articles arranged into threads and subthreads. Over time, a discussion might branch into several sub-threads in a tree-like form.
A diagram of Usenet servers and clients. The coloured dots on the servers represent the newsgroups they carry. Coloured arrows between servers indicate newsgroup content exchanges (news feeds). Arrows between clients and servers indicate that a user is subscribed to a certain newsgroup and reads or submits articles there. Notably, clients never connect with each other, but still have access to each other's posts even when they also never connect to the same server.
When a user posts an article, it is initially only available on that user's news server. Each news server talks to one or more other servers (its "newsfeeds") and exchanges articles with them. In this fashion, the article is copied from server to server and should eventually reach every server in the network. Usenet was designed under conditions when networks were slow and not always available. Many sites on the original Usenet network would connect only once or twice a day to batch-transfer messages in and out. This is largely because the POTS network was typically used for transfers, and phone charges were lower at night.[citation needed]
Usenet has diminished in importance with the creation and rise in popularity of Internet forums, blogs, mailing lists and social media. Usenet differs from such media in several ways: it requires no personal registration with the group concerned; information need not be stored on a remote server; and reading the messages does not require a mail or web client, but a news client. However, it is now possible to read and participate in Usenet newsgroups to a large degree using ordinary web browsers since most newsgroups are now copied to several web sites. The groups in alt.binaries are still widely used for data transfer.[citation needed]
Functionality
Usenet provider map
Usenet provider companies operate news servers for their users to access. Newsgroups provided by the news servers can be accessed with a newsreader—an application that allows a user to read and reply to postings in newsgroups. These applications act as clients to one or more news servers.
Propagation
Usenet uses a set of protocols for generating, storing and retrieving articles, and for exchanging them among a distributed readership. Whenever an article reaches a server, that server forwards it to all its network neighbors. Only one copy of an article is stored per server, and each server makes it available on demand to the readers able to access that server. Articles are generally propagated between all news servers globally in the order of minutes.
In cases where users want to retract articles, Usenet has support for automated removal of a posting from the whole network by creating a cancel message, although due to a lack of authentication and resultant abuse, this capability is frequently disabled.
Moderation
A minority of newsgroups are moderated, meaning that messages submitted by readers are not distributed directly to Usenet, but instead are emailed to the moderators of the newsgroup for approval. The moderator receives submitted articles, reviews them, and injects approved articles so that they can be properly propagated worldwide. Moderators ensure that the messages that readers see in the newsgroup conform to the charter of the newsgroup, though they are not required to follow any such rules or guidelines. Typically, moderators are appointed in the proposal for the newsgroup, and changes of moderators follow a succession plan.
In unmoderated newsgroups, messages submitted by readers are immediately propagated for everyone to see. Unmoderated newsgroups tend to receive large amounts of spam.
Retention
Each news server allocates a certain amount of storage space for content in each newsgroup. When this storage has been filled, each time a new post arrives, old posts are deleted to make room for the new content. Modern Usenet news servers have enough capacity to archive years of content.
The major set of worldwide newsgroups is contained within nine hierarchies, eight of which are operated under consensual guidelines that govern their administration and naming. The current Big Eight are:
news.* – discussions and announcements about news (meaning Usenet, not current events) (news.groups, news.admin)
rec.* – recreation and entertainment (rec.music, rec.arts.movies)
sci.* – science related discussions (sci.psychology, sci.research)
soc.* – social discussions (soc.college.org, soc.culture.african)
talk.* – talk about various controversial topics (talk.religion, talk.politics, talk.origins)
The alt.* hierarchy is not subject to the procedures controlling groups in the Big Eight, and it is as a result less organized. Groups in the alt.* hierarchy tend to be more specialized or specific—for example, there might be a newsgroup under the Big Eight which contains discussions about children's books, but a group in the alt hierarchy may be dedicated to one specific author of children's books. Binaries are posted in alt.binaries.*, making it the largest of all the hierarchies.
Many other hierarchies of newsgroups are distributed alongside these. Regional and language-specific hierarchies such as japan.*, malta.* and ne.* serve specific countries and regions such as Japan, Malta and New England. Companies and projects administer their own hierarchies to discuss their products and offer community technical support, such as the historical gnu.* hierarchy from the Free Software Foundation.
Binary content
A visual example of the many complex steps required to prepare data to be uploaded to Usenet newsgroups. These steps must be done again in reverse to download data from Usenet.
Usenet was originally created to distribute text content encoded in the 7-bitASCII character set. With the help of programs that encode 8-bit values into ASCII, it became practical to distribute binary files as content. Binary posts, due to their size and often-dubious copyright status, were in time restricted to specific newsgroups, making it easier for administrators to allow or disallow the traffic.
The oldest widely used encoding method for binary content is uuencode, from the Unix UUCP package. In the late 1980s, Usenet articles were often limited to 60,000 characters, and larger hard limits exist today. Files are therefore commonly split into sections that require reassembly by the reader.
With the header extensions and the Base64 and Quoted-Printable MIME encodings, there was a new generation of binary transport.[when?] In practice, MIME has seen increased adoption in text messages, but it is avoided for most binary attachments. Some operating systems with metadata attached to files use specialized encoding formats. For Mac OS, both BinHex and special MIME types are used. Other lesser known encoding systems that may have been used at one time were BTOA, XX encoding, BOO, and USR encoding.
In an attempt to reduce file transfer times, an informal file encoding known as yEnc was introduced in 2001. It achieves about a 30% reduction in data transferred by assuming that most 8-bit characters can safely be transferred across the network without first encoding into the 7-bit ASCII space. The most common method of uploading large binary posts to Usenet is to convert the files into RAR archives and create Parchive files for them.[citation needed] Parity files are used to recreate missing data when not every part of the files reaches a server.
Binary newsgroups can be used to distribute files and, as of 2022[update], some remain a popular way to share and download files.
Legal issues
While binary newsgroups can be used to distribute completely legal user-created works, free software, and public domain material, some binary groups are used to illegally distribute proprietary software and copyrighted media. Commercial Usenet service providers claim to operate as a telecommunications service, and assert that they are not responsible for the user-posted binary content transferred via their equipment.[citation needed] In the United States, Usenet providers can qualify for protection under the DMCASafe Harbor regulations, provided that they establish a mechanism to comply with and respond to takedown notices from copyright holders.
Removal of copyrighted content from the entire Usenet network is a nearly impossible task, due to the rapid propagation between servers and the retention done by each server. Petitioning a Usenet provider for removal only removes it from that one server's retention cache, but not any others. It is possible for a special post cancellation message to be distributed to remove it from all servers, but many providers ignore cancel messages messages because they can be easily falsified and submitted by anyone. For a takedown petition to be most effective across the whole network, it would have to be issued to the origin server to which the content has been posted, before the content is propagated to other servers. With modern high speed links, however, content can be propagated as fast as it arrives, allowing no time for content review and takedown issuance by copyright holders.
Establishing the identity of the person posting illegal content is equally difficult due to the trust-based design of the network. Like SMTP email, servers generally assume the header and origin information in a post is true and accurate. However, as in SMTP email, Usenet post headers are easily falsified so as to obscure the true identity and location of the message source. In this manner, Usenet is significantly different from modern P2P services; most P2P users distributing content are typically immediately identifiable to all other users by their network address, but the origin information for a Usenet posting can be completely obscured and unobtainable once it has propagated past the original server.
Also unlike modern P2P services, the identity of the downloaders is hidden from view. On P2P services a downloader is identifiable to all others by their network address. On Usenet, the downloader connects directly to a server, and only the server knows the address of who is connecting to it. Some Usenet providers do keep usage logs, but not all make this logged information casually available to outside parties such as the Recording Industry Association of America. The existence of anonymising gateways to USENET also complicates the tracing of a posting's true origin.[clarification needed]
History
Newsgroup experiments first occurred in 1979. Tom Truscott and Jim Ellis of Duke University came up with the idea as a replacement for a local announcement program, and established a link with nearby University of North Carolina using Bourne shell scripts written by Steve Bellovin. The public release of news was in the form of conventional compiled software, written by Steve Daniel and Truscott. In 1980, Usenet was connected to ARPANET through UC Berkeley, which had connections to both Usenet and ARPANET. Mary Ann Horton, the graduate student who set up the connection, began "feeding mailing lists from the ARPANET into Usenet" with the "fa" ("From ARPANET") identifier. Usenet gained 50 member sites in its first year, including Reed College, University of Oklahoma, and Bell Labs, and the number of people using the network increased dramatically; however, it was still a while longer before Usenet users could contribute to ARPANET.
Network
UUCP networks spread quickly due to the lower costs involved, and the ability to use existing leased lines, X.25 links or even ARPANET connections. By 1983, thousands of people participated from more than 500 hosts, mostly universities and Bell Labs sites but also a growing number of Unix-related companies; the number of hosts nearly doubled to 940 in 1984. More than 100 newsgroups existed, more than 20 devoted to Unix and other computer-related topics, and at least a third to recreation. As the mesh of UUCP hosts rapidly expanded, it became desirable to distinguish the Usenet subset from the overall network. A vote was taken at the 1982 USENIX conference to choose a new name. The name Usenet was retained, but it was established that it only applied to news. The name UUCPNET became the common name for the overall network.
In addition to UUCP, early Usenet traffic was also exchanged with FidoNet and other dial-up BBS networks. By the mid-1990s there were almost 40,000 FidoNet systems in operation, and it was possible to communicate with millions of users around the world, with only local telephone service. Widespread use of Usenet by the BBS community was facilitated by the introduction of UUCP feeds made possible by MS-DOS implementations of UUCP, such as UFGATE (UUCP to FidoNet Gateway), FSUUCP and UUPC. In 1986, RFC 977 provided the Network News Transfer Protocol (NNTP) specification for distribution of Usenet articles over TCP/IP as a more flexible alternative to informal Internet transfers of UUCP traffic. Since the Internet boom of the 1990s, almost all Usenet distribution is over NNTP.
Software
Early versions of Usenet used Duke's A News software, designed for one or two articles a day. Matt Glickman and Horton at Berkeley produced an improved version called B News that could handle the rising traffic (about 50 articles a day as of late 1983). With a message format that offered compatibility with Internet mail and improved performance, it became the dominant server software. C News, developed by Geoff Collyer and Henry Spencer at the University of Toronto, was comparable to B News in features but offered considerably faster processing. In the early 1990s, InterNetNews by Rich Salz was developed to take advantage of the continuous message flow made possible by NNTP versus the batched store-and-forward design of UUCP. Since that time INN development has continued, and other news server software has also been developed.
Newsreaders
Historically, Usenet was associated with the Unix operating system developed at AT&T, but newsreaders were soon available for all major operating systems. Email client programs and Internet suites of the late 1990s and 2000s often included an integrated newsreader. Newsgroup enthusiasts often criticized these as inferior to standalone newsreaders that made correct use of Usenet protocols, standards and conventions.
With the rise of the World Wide Web (WWW), web front-ends (web2news) have become more common. Web front ends have lowered the technical entry barrier requirements to that of one application and no Usenet NNTP server account. There are numerous websites now offering web based gateways to Usenet groups, although some people have begun filtering messages made by some of the web interfaces for one reason or another. Google Groups is one such web based front end and some web browsers can access Google Groups via news: protocol links directly.
Public venue
Usenet was the first Internet community and the place for many of the most important public developments in the pre-commercial Internet. It was the place where Tim Berners-Lee announced the launch of the World Wide Web, where Linus Torvalds announced the Linux project, and where Marc Andreessen announced the creation of the Mosaic browser and the introduction of the image tag, which revolutionized the World Wide Web by turning it into a graphical medium.
Internet jargon and history
Many jargon terms subsequently used on the web originated from Usenet or were popularized by its users.[unreliable source?] Likewise, many conflicts which later spread to the rest of the Internet, such as the ongoing difficulties over spamming, began on Usenet.
Decline
Sascha Segan of PC Magazine said in 2008 that "Usenet has been dying for years". He argued that it was dying by the late 1990s, when large binary files became a significant proportion of Usenet traffic, and Internet service providers "sensibly started to wonder why they should be reserving big chunks of their own disk space for pirated movies and repetitive porn".
AOL discontinued Usenet access in 2005. In May 2010, Duke University, whose implementation had started Usenet more than 30 years earlier, decommissioned its Usenet server, citing low usage and rising costs. On February4, 2011, the Usenet news service link at the University of North Carolina at Chapel Hill (news.unc.edu) was retired after 32 years.[citation needed]
In response, John Biggs of TechCrunch said: "As long as there are folks who think a command line is better than a mouse, the original text-only social network will live on". While there are still some active text newsgroups on Usenet, the system is now primarily used to share large files between users, and the underlying technology of Usenet remains unchanged.
Over time, the amount of Usenet traffic has steadily increased. As of 2010[update] the number of all text posts made in all Big-8 newsgroups averaged 1,800 new messages every hour, with an average of 25,000 messages per day. However, these averages are minuscule in comparison to the traffic in the binary groups. Much of this traffic increase reflects not an increase in discrete users or newsgroup discussions, but instead the combination of massive automated spamming and an increase in the use of .binaries newsgroups in which large files are often posted publicly.
In 2008, Verizon Communications, Time Warner Cable and Sprint Nextel signed an agreement with Attorney General of New YorkAndrew Cuomo to shut down access to sources of child pornography. Time Warner Cable stopped offering access to Usenet. Verizon reduced its access to the "Big 8" hierarchies. Sprint stopped access to the alt.* hierarchies. AT&T stopped access to the alt.binaries.* hierarchies. Cuomo never specifically named Usenet in his anti-child-pornography campaign. David DeJean of PC World said that some worry that the ISPs used Cuomo's campaign as an excuse to end portions of Usenet access, as it is costly for the Internet service providers and not in high demand by customers. In 2008 AOL, which no longer offered Usenet access, and the four providers that responded to the Cuomo campaign were the five largest Internet service providers in the United States; they had more than 50% of the U.S. ISP market share. On June 8, 2009, AT&T announced that it would no longer provide access to the Usenet service as of July 15, 2009.
AOL announced that it would discontinue its integrated Usenet service in early 2005, citing the growing popularity of weblogs, chat forums and on-line conferencing. The AOL community had a tremendous role in popularizing Usenet some 11 years earlier.
In August 2009, Verizon announced that it would discontinue access to Usenet on September 30, 2009. JANET announced it would discontinue Usenet service, effective July 31, 2010, citing Google Groups as an alternative. Microsoft announced that it would discontinue support for its public newsgroups (msnews.microsoft.com) from June 1, 2010, offering web forums as an alternative.
Primary reasons cited for the discontinuance of Usenet service by general ISPs include the decline in volume of actual readers due to competition from blogs, along with cost and liability concerns of increasing proportion of traffic devoted to file-sharing and spam on unused or discontinued groups.
Some ISPs did not include pressure from Cuomo's campaign against child pornography as one of their reasons for dropping Usenet feeds as part of their services. ISPs Cox and Atlantic Communications resisted the 2008 trend but both did eventually drop their respective Usenet feeds in 2010.
Public archives of Usenet articles have existed since the early days of Usenet, such as the system created by Kenneth Almquist in late 1982. Distributed archiving of Usenet posts was suggested in November 1982 by Scott Orshan, who proposed that "Every site should keep all the articles it posted, forever." Also in November of that year, Rick Adams responded to a post asking: "Has anyone archived netnews, or does anyone plan to?" by stating that he was "afraid to admit it, but I started archiving most 'useful' newsgroups as of September18". In June 1982, Gregory G.Woodbury proposed an "automatic access to archives" system that consisted of "automatic answering of fixed-format messages to a special mail recipient on specified machines".
In 1985, two news archiving systems and one RFC were posted to the Internet. The first system, called keepnews, by Mark M.Swenson of the University of Arizona, was described as "a program that attempts to provide a sane way of extracting and keeping information that comes over Usenet". The main advantage of this system was to allow users to mark articles as worthwhile to retain. The second system, YA News Archiver by Chuq Von Rospach, was similar to keepnews, but was "designed to work with much larger archives where the wonderful quadratic search time feature of the Unix... becomes a real problem". Von Rospach in early 1985 posted a detailed RFC for "archiving and accessing usenet articles with keyword lookup". This RFC described a program that could "generate and maintain an archive of Usenet articles and allow looking up articles based on the article-id, subject lines, or keywords pulled out of the article itself". Also included was C code for the internal data structure of the system.
The desire to have a full text search index of archived news articles is not new either, one such request having been made in April 1991 by Alex Martelli, who sought to "build some sort of keyword index for [the news archive]". In early May, Martelli posted a summary of his responses to Usenet, noting that the "most popular suggestion award must definitely go to 'lq-text' package, by Liam Quin, recently posted in alt.sources".
The Alt Sex Stories Text Repository (ASSTR) site archived and indexed erotic and pornographic stories posted to the Usenet group alt.sex.stories.
The archiving of Usenet has led to fears of loss of privacy. An archive simplifies ways to profile people. This has partly been countered with the introduction of the X-No-Archive:Yes header, which is itself controversial.
Web-based archiving of Usenet posts began in March 1995 at Deja News with a very large, searchable database. In February 2001, this database was acquired by Google; Google had begun archiving Usenet posts for itself starting in the second week of August 2000.
Google Groups hosts an archive of Usenet posts dating back to May 1981. The earliest posts, which date from May 1981 to June 1991, were donated to Google by the University of Western Ontario with the help of David Wiseman and others, and were originally archived by Henry Spencer at the University of Toronto's Zoology department. The archives for late 1991 through early 1995 were provided by Kent Landfield from the NetNews CD series and Jürgen Christoffel from GMD.
Google has been criticized by Vice and Wired contributors, as well as former employees, for its stewardship of the archive and for breaking its search functionality.
As of January 2024, Google Groups carries a header notice, saying:
Effective from 22 February 2024, Google Groups will no longer support new Usenet content. Posting and subscribing will be disallowed, and new content from Usenet peers will not appear. Viewing and searching of historical data will still be supported as it is done today.
An explanatory page adds:
In addition, Google's Network News Transfer Protocol (NNTP) server and associated peering will no longer be available, meaning Google will not support serving new Usenet content or exchanging content with other NNTP servers. This change will not impact any non-Usenet content on Google Groups, including all user and organization-created groups.
↑Pre-Internet; Usenet needing "just local telephone service" in most larger towns, depends on the number of local dial-up FidoNet "nodes" operated free of charge by hobbyist "SysOps" (as FidoNet echomail variations or via gateways with the Usenet news hierarchy. This is virtual Usenet or newsgroups access, not true Usenet.) The participating SysOps typically carry 6–30 Usenet newsgroups each, and will often add another on request. If a desired newsgroup was not available locally, a user would need to dial to another city to download the desired news and upload one's own posts. In all cases it is desirable to hang up as soon as possible and read/write offline, making "newsreader" software commonly used to automate the process. Fidonet, bbscorner.com Archived February 7, 2022, at the Wayback Machine fidonet.org, Randy_Bush.txtArchived December 3, 2003, at the Wayback Machine
↑From Usenet to CoWebs: interacting with social information spaces, Christopher Lueg, Danyel Fisher, Springer (2003), ISBN1-85233-532-7, ISBN978-1-85233-532-8
↑"Unraveling the Internet's oldest and weirdest mystery". The Kernel. March 22, 2015. Archived from the original on May 18, 2015. Retrieved May 7, 2015. Groups filled with spam, massive fights took place against spammers and over what to do about the spam. People stopped using their email addresses in messages to avoid harvesting. People left the net.
↑"The American Way of Spam". Archived from the original on May 18, 2015. Retrieved May 7, 2015. ...many of the newsgroups have since been overrun with junk messages.
↑"Cancel Messages FAQ". Archived from the original on February 15, 2008. Retrieved June 29, 2009. ...Until authenticated cancels catch on, there are no options to avoid forged cancels and allow unforged ones...
↑Microsoft knowledgebase article stating that many servers ignore cancel messages "Support.microsoft.com".{{cite web}}: CS1 maint: deprecated archival service (link)
↑...every part of a Usenet post may be forged apart from the left most portion of the "Path:" header..."By-users.co.uk".{{cite web}}: CS1 maint: deprecated archival service (link)
↑Kaltenbach, Susan (December 2000). "The Evolution of the Online Discourse Community"(PDF). Archived(PDF) from the original on July 14, 2011. Retrieved May 26, 2010. Verb Doubling: Doubling a verb may change its semantics, Soundalike Slang: Punning jargon, The -P convention: A LISPy way to form questions, Overgeneralization: Standard abuses of grammar, Spoken Inarticulations: Sighing and <*sigh*>ing, Anthropomorphization: online components were named "Homunculi," daemons," etc., and there were also "confused" programs. Comparatives: Standard comparatives for design quality
↑DeJean, David. "Usenet: Not Dead Yet." PC World. Tuesday October 7, 2008. "2". October 7, 2008. Retrieved September 17, 2017.{{cite web}}: CS1 maint: deprecated archival service (link). Retrieved on April 30, 2009.
Bryan Pfaffenberger (December 31, 1994). The USENET Book: Finding, Using, and Surviving Newsgroups on the Internet. Addison Wesley. ISBN978-0-201-40978-9.
Don Rittner (June 1997). Rittner's Field Guide to Usenet. MNS Publishing. ISBN978-0-937666-50-0.
Konstan, J.; Miller, B.; Maltz, D.; Herlocker, J.; Gordon, L.; Riedl, J. (March 1997). "GroupLens: applying collaborative filtering to Usenet news". Communications of the ACM. 40 (3): 77–87. doi:10.1145/245108.245126. S2CID15008577.
Miller, B.; Riedl, J.; Konstan, J. (January 1997). Experiences with GroupLens: Making Usenet useful again(PDF). Proceedings of the 1997 Usenix Winter Technical Conference. Archived(PDF) from the original on March 6, 2006. Retrieved December 13, 2005.
