Wayback Machine

From Wikipedia for FEVERv2
Jump to navigation Jump to search

For the time machine from Peabody's Improbable History, see Wayback Machine (Peabody's Improbable History). Wayback Machine_sentence_0

Wayback Machine_table_infobox_0

Wayback MachineWayback Machine_table_caption_0
Type of siteWayback Machine_header_cell_0_0_0 ArchiveWayback Machine_cell_0_0_1
Area servedWayback Machine_header_cell_0_1_0 Worldwide (except China and Russia)Wayback Machine_cell_0_1_1
OwnerWayback Machine_header_cell_0_2_0 Internet ArchiveWayback Machine_cell_0_2_1
URLWayback Machine_header_cell_0_3_0 Q648266#P856Wayback Machine_cell_0_3_1
RegistrationWayback Machine_header_cell_0_4_0 OptionalWayback Machine_cell_0_4_1
LaunchedWayback Machine_header_cell_0_5_0 October 24, 2001; 19 years ago (2001-10-24)Wayback Machine_cell_0_5_1
Current statusWayback Machine_header_cell_0_6_0 ActiveWayback Machine_cell_0_6_1
Written inWayback Machine_header_cell_0_7_0 Java, PythonWayback Machine_cell_0_7_1

The Wayback Machine is a digital archive of the World Wide Web, founded by the Internet Archive, a nonprofit library based in San Francisco. Wayback Machine_sentence_1

It allows the user to go “back in time” and see what websites looked like in the past. Wayback Machine_sentence_2

Its founders, Brewster Kahle and Bruce Gilliat, developed the Wayback Machine with the intention of providing "universal access to all knowledge" by preserving archived copies of defunct webpages. Wayback Machine_sentence_3

Since its launch in 2001, over 492 billion pages have been added to the archive. Wayback Machine_sentence_4

The service has also sparked controversy over whether creating archived pages without the owner's permission constitutes copyright infringement in certain jurisdictions. Wayback Machine_sentence_5

History Wayback Machine_section_0

Internet Archive founders Brewster Kahle and Bruce Gilliat launched the Wayback Machine in 2001 to address the problem of website content vanishing whenever it gets changed or shut down. Wayback Machine_sentence_6

The service enables users to see archived versions of web pages across time, which the archive calls a "three dimensional index". Wayback Machine_sentence_7

Kahle and Gilliat created the machine hoping to archive the entire Internet and provide "universal access to all knowledge." Wayback Machine_sentence_8

The name Wayback Machine was chosen as a reference to a fictional time-traveling device, the "Wayback Machine" (pronounced way-back), used by the characters Mister Peabody and Sherman in the animated cartoon The Rocky and Bullwinkle Show from the 1960s. Wayback Machine_sentence_9

In one of the animated cartoon's component segments, Peabody's Improbable History, the characters routinely used the machine to witness, participate in, and often alter famous events in history. Wayback Machine_sentence_10

The Wayback Machine began archiving cached web pages in May 1996, with the goal of making the service public five years later. Wayback Machine_sentence_11

From 1996 to 2001, the information was kept on digital tape, with Kahle occasionally allowing researchers and scientists to tap into the clunky database. Wayback Machine_sentence_12

When the archive reached its fifth anniversary in 2001, it was unveiled and opened to the public in a ceremony at the University of California, Berkeley. Wayback Machine_sentence_13

By the time the Wayback Machine launched, it already contained over 10 billion archived pages. Wayback Machine_sentence_14

Today, the data is stored on the Internet Archive's large cluster of Linux nodes. Wayback Machine_sentence_15

It revisits and archives new versions of websites on occasion (see technical details below). Wayback Machine_sentence_16

Sites can also be captured manually by entering a website's URL into the search box, provided that the website allows the Wayback Machine to "crawl" it and save the data. Wayback Machine_sentence_17

On October 30, 2020, the Wayback Machine began fact checking content. Wayback Machine_sentence_18

Technical details Wayback Machine_section_1

Software has been developed to "crawl" the web and download all publicly accessible World Wide Web pages, the Gopher hierarchy, the Netnews (Usenet) bulletin board system, and downloadable software. Wayback Machine_sentence_19

The information collected by these "crawlers" does not include all the information available on the Internet, since much of the data is restricted by the publisher or stored in databases that are not accessible. Wayback Machine_sentence_20

To overcome inconsistencies in partially cached websites, Archive-It.org was developed in 2005 by the Internet Archive as a means of allowing institutions and content creators to voluntarily harvest and preserve collections of digital content, and create digital archives. Wayback Machine_sentence_21

Crawls are contributed from various sources, some imported from third parties and others generated internally by the Archive. Wayback Machine_sentence_22

For example, crawls are contributed by the Sloan Foundation and Alexa, crawls run by IA on behalf of NARA and the Internet Memory Foundation, mirrors of Common Crawl. Wayback Machine_sentence_23

The "Worldwide Web Crawls" have been running since 2010 and capture the global Web. Wayback Machine_sentence_24

The frequency of snapshot captures varies per website. Wayback Machine_sentence_25

Websites in the "Worldwide Web Crawls" are included in a "crawl list", with the site archived once per crawl. Wayback Machine_sentence_26

A crawl can take months or even years to complete depending on size. Wayback Machine_sentence_27

For example, "Wide Crawl Number 13" started on January 9, 2015, and completed on July 11, 2016. Wayback Machine_sentence_28

However, there may be multiple crawls ongoing at any one time, and a site might be included in more than one crawl list, so how often a site is crawled varies widely. Wayback Machine_sentence_29

As of October 2019, users are limited to 5 archival requests and retrievals per minute. Wayback Machine_sentence_30

Storage capacity and growth Wayback Machine_section_2

As technology has developed over the years, the storage capacity of the Wayback Machine has grown. Wayback Machine_sentence_31

In 2003, after only two years of public access, the Wayback Machine was growing at a rate of 12 terabytes/month. Wayback Machine_sentence_32

The data is stored on PetaBox rack systems custom designed by Internet Archive staff. Wayback Machine_sentence_33

The first 100TB rack became fully operational in June 2004, although it soon became clear that they would need much more storage than that. Wayback Machine_sentence_34

The Internet Archive migrated its customized storage architecture to Sun Open Storage in 2009, and hosts a new data center in a Sun Modular Datacenter on Sun Microsystems' California campus. Wayback Machine_sentence_35

As of 2009, the Wayback Machine contained approximately three petabytes of data and was growing at a rate of 100 terabytes each month. Wayback Machine_sentence_36

A new, improved version of the Wayback Machine, with an updated interface and a fresher index of archived content, was made available for public testing in 2011. Wayback Machine_sentence_37

In March that year, it was said on the Wayback Machine forum that "the Beta of the new Wayback Machine has a more complete and up-to-date index of all crawled materials into 2010, and will continue to be updated regularly. Wayback Machine_sentence_38

The index driving the classic Wayback Machine only has a little bit of material past 2008, and no further index updates are planned, as it will be phased out this year." Wayback Machine_sentence_39

Also in 2011, the Internet Archive installed their sixth pair of PetaBox racks which increased the Wayback Machine's storage capacity by 700 terabytes. Wayback Machine_sentence_40

In January 2013, the company announced a ground-breaking milestone of 240 billion URLs. Wayback Machine_sentence_41

In October 2013, the company introduced the "Save a Page" feature which allows any Internet user to archive the contents of a URL, and quickly generates a permanent link unlike the preceding liveweb feature. Wayback Machine_sentence_42

It became a threat of abuse by the service for hosting malicious binaries. Wayback Machine_sentence_43

As of December 2014, the Wayback Machine contained 435 billion web pages—almost nine petabytes of data, and was growing at about 20 terabytes a week. Wayback Machine_sentence_44

As of July 2016, the Wayback Machine reportedly contained around 15 petabytes of data. Wayback Machine_sentence_45

As of September 2018, the Wayback Machine contained over 25 petabytes of data. Wayback Machine_sentence_46

As of December 2020, the Wayback Machine contained over 70 petabytes of data. Wayback Machine_sentence_47

Between October 2013 and March 2015, the website's global Alexa rank changed from 163 to 208. Wayback Machine_sentence_48

In March 2019 the rank was at 244. Wayback Machine_sentence_49

Website exclusion policy Wayback Machine_section_3

Historically, Wayback Machine has respected the robots exclusion standard (robots.txt) in determining if a website would be crawled; or if already crawled, if its archives would be publicly viewable. Wayback Machine_sentence_50

Website owners had the option to opt-out of Wayback Machine through the use of robots.txt. Wayback Machine_sentence_51

It applied robots.txt rules retroactively; if a site blocked the Internet Archive, any previously archived pages from the domain were immediately rendered unavailable as well. Wayback Machine_sentence_52

In addition, the Internet Archive stated that "Sometimes a website owner will contact us directly and ask us to stop crawling or archiving a site. Wayback Machine_sentence_53

We comply with these requests." Wayback Machine_sentence_54

In addition, the website says: "The Internet Archive is not interested in preserving or offering access to Web sites or other Internet documents of persons who do not want their materials in the collection." Wayback Machine_sentence_55

On April 17, 2017, reports surfaced of sites that had gone defunct and became parked domains that were using robots.txt to exclude themselves from search engines, resulting in them being inadvertently excluded from the Wayback Machine. Wayback Machine_sentence_56

The Internet archive changed the policy to now require an explicit exclusion request to remove it from the Wayback Machine. Wayback Machine_sentence_57

Oakland Archive Policy Wayback Machine_section_4

Wayback's retroactive exclusion policy is based in part upon Recommendations for Managing Removal Requests and Preserving Archival Integrity published by the School of Information Management and Systems at University of California, Berkeley in 2002, which gives a website owner the right to block access to the site's archives. Wayback Machine_sentence_58

Wayback has complied with this policy to help avoid expensive litigation. Wayback Machine_sentence_59

The Wayback retroactive exclusion policy began to relax in 2017, when it stopped honoring robots.txt on U.S. government and military web sites for both crawling and displaying web pages. Wayback Machine_sentence_60

As of April 2017, Wayback is ignoring robots.txt more broadly, not just for U.S. government websites. Wayback Machine_sentence_61

Uses Wayback Machine_section_5

From its public launch in 2001, the Wayback Machine has been studied by scholars both for the ways it stores and collects data as well as for the actual pages contained in its archive. Wayback Machine_sentence_62

As of 2013, scholars had written about 350 articles on the Wayback Machine, mostly from the information technology, library science, and social science fields. Wayback Machine_sentence_63

Social science scholars have used the Wayback Machine to analyze how the development of websites from the mid-1990s to the present has affected the company's growth. Wayback Machine_sentence_64

When the Wayback Machine archives a page, it usually includes most of the hyperlinks, keeping those links active when they just as easily could have been broken by the Internet's instability. Wayback Machine_sentence_65

Researchers in India studied the effectiveness of the Wayback Machine's ability to save hyperlinks in online scholarly publications and found that it saved slightly more than half of them. Wayback Machine_sentence_66

"Journalists use the Wayback Machine to view dead websites, dated news reports, and changes to website contents. Wayback Machine_sentence_67

Its content has been used to hold politicians accountable and expose battlefield lies." Wayback Machine_sentence_68

In 2014, an archived social media page of Igor Girkin, a separatist rebel leader in Ukraine, showed him boasting about his troops having shot down a suspected Ukrainian military airplane before it became known that the plane actually was a civilian Malaysian Airlines jet (Malaysia Airlines Flight 17), after which he deleted the post and blamed Ukraine's military for downing the plane. Wayback Machine_sentence_69

In 2017, the March for Science originated from a discussion on reddit that indicated someone had visited Archive.org and discovered that all references to climate change had been deleted from the White House website. Wayback Machine_sentence_70

In response, a user commented, "There needs to be a Scientists' March on Washington". Wayback Machine_sentence_71

Furthermore, the site is used heavily for verification, providing access to references and content creation by Wikipedia editors. Wayback Machine_sentence_72

In September 2020, a partnership was announced with Cloudflare to automatically archive websites served via its "Always Online" service, which will also allow it to direct users to its copy of the site if it cannot reach the original host. Wayback Machine_sentence_73

Limitations Wayback Machine_section_6

In 2014 there was a six-month lag time between when a website was crawled and when it became available for viewing in the Wayback Machine. Wayback Machine_sentence_74

Currently, the lag time is 3 to 10 hours. Wayback Machine_sentence_75

The Wayback Machine offers only limited search facilities. Wayback Machine_sentence_76

Its "Site Search" feature allows users to find a site based on words describing the site, rather than words found on the web pages themselves. Wayback Machine_sentence_77

The Wayback Machine does not include every web page ever made due to the limitations of its web crawler. Wayback Machine_sentence_78

The Wayback Machine cannot completely archive web pages that contain interactive features such as Flash platforms and forms written in JavaScript and progressive web applications, because those functions require interaction with the host website. Wayback Machine_sentence_79

This means that, since June 2013, the Wayback Machine has been unable to display YouTube comments when saving YouTube pages, as, according to the Archive Team, comments are no longer "loaded within the page itself." Wayback Machine_sentence_80

The Wayback Machine's web crawler has difficulty extracting anything not coded in HTML or one of its variants, which can often result in broken hyperlinks and missing images. Wayback Machine_sentence_81

Due to this, the web crawler cannot archive "orphan pages" that contain no links to other pages. Wayback Machine_sentence_82

The Wayback Machine's crawler only follows a predetermined number of hyperlinks based on a preset depth limit, so it cannot archive every hyperlink on every page. Wayback Machine_sentence_83

Starting in April 2018, administrative staff members of the Wayback Machine's archive team have enforced the Quarter month rule, by occasionally deleting time intervals of 23 days or 39 days (3/4 and 5/4 of a month, respectively), in order to reduce the queue size. Wayback Machine_sentence_84

In legal evidence Wayback Machine_section_7

Civil litigation Wayback Machine_section_8

Netbula LLC v. Chordiant Software Inc. Wayback Machine_section_9

In a 2009 case, Netbula, LLC v. Chordiant Software Inc., defendant Chordiant filed a motion to compel Netbula to disable the robots.txt file on its website that was causing the Wayback Machine to retroactively remove access to previous versions of pages it had archived from Netbula's site, pages that Chordiant believed would support its case. Wayback Machine_sentence_85

Netbula objected to the motion on the ground that defendants were asking to alter Netbula's website and that they should have subpoenaed Internet Archive for the pages directly. Wayback Machine_sentence_86

An employee of Internet Archive filed a sworn statement supporting Chordiant's motion, however, stating that it could not produce the web pages by any other means "without considerable burden, expense and disruption to its operations." Wayback Machine_sentence_87

Magistrate Judge Howard Lloyd in the Northern District of California, San Jose Division, rejected Netbula's arguments and ordered them to disable the robots.txt blockage temporarily in order to allow Chordiant to retrieve the archived pages that they sought. Wayback Machine_sentence_88

Telewizja Polska Wayback Machine_section_10

In an October 2004 case, Telewizja Polska USA, Inc. v. Echostar Satellite, No. Wayback Machine_sentence_89

02 C 3293, 65 Fed. Wayback Machine_sentence_90

R. Evid. Wayback Machine_sentence_91

Serv. Wayback Machine_sentence_92

673 (N.D. Ill. October 15, 2004), a litigant attempted to use the Wayback Machine archives as a source of admissible evidence, perhaps for the first time. Wayback Machine_sentence_93

Telewizja Polska is the provider of TVP Polonia and EchoStar operates the Dish Network. Wayback Machine_sentence_94

Prior to the trial proceedings, EchoStar indicated that it intended to offer Wayback Machine snapshots as proof of the past content of Telewizja Polska's website. Wayback Machine_sentence_95

Telewizja Polska brought a motion in limine to suppress the snapshots on the grounds of hearsay and unauthenticated source, but Magistrate Judge Arlander Keys rejected Telewizja Polska's assertion of hearsay and denied TVP's motion in limine to exclude the evidence at trial. Wayback Machine_sentence_96

At the trial, however, District Court Judge Ronald Guzman, the trial judge, overruled Magistrate Keys' findings, and held that neither the affidavit of the Internet Archive employee nor the underlying pages (i.e., the Telewizja Polska website) were admissible as evidence. Wayback Machine_sentence_97

Judge Guzman reasoned that the employee's affidavit contained both hearsay and inconclusive supporting statements, and the purported web page, printouts were not self-authenticating. Wayback Machine_sentence_98

Patent law Wayback Machine_section_11

Main article: Internet as a source of prior art Wayback Machine_sentence_99

Provided some additional requirements are met (e.g., providing an authoritative statement of the archivist), the United States patent office and the European Patent Office will accept date stamps from the Internet Archive as evidence of when a given Web page was accessible to the public. Wayback Machine_sentence_100

These dates are used to determine if a Web page is available as prior art for instance in examining a patent application. Wayback Machine_sentence_101

Limitations of utility Wayback Machine_section_12

There are technical limitations to archiving a website, and as a consequence, it is possible for opposing parties in litigation to misuse the results provided by website archives. Wayback Machine_sentence_102

This problem can be exacerbated by the practice of submitting screenshots of web pages in complaints, answers, or expert witness reports when the underlying links are not exposed and therefore, can contain errors. Wayback Machine_sentence_103

For example, archives such as the Wayback Machine do not fill out forms and therefore, do not include the contents of non-RESTful e-commerce databases in their archives. Wayback Machine_sentence_104

Legal status Wayback Machine_section_13

In Europe, the Wayback Machine could be interpreted as violating copyright laws. Wayback Machine_sentence_105

Only the content creator can decide where their content is published or duplicated, so the Archive would have to delete pages from its system upon request of the creator. Wayback Machine_sentence_106

The exclusion policies for the Wayback Machine may be found in the FAQ section of the site. Wayback Machine_sentence_107

Archived content legal issues Wayback Machine_section_14

A number of cases have been brought against the Internet Archive specifically for its Wayback Machine archiving efforts. Wayback Machine_sentence_108

Scientology Wayback Machine_section_15

See also: Scientology and the Internet Wayback Machine_sentence_109

In late 2002, the Internet Archive removed various sites that were critical of Scientology from the Wayback Machine. Wayback Machine_sentence_110

An error message stated that this was in response to a "request by the site owner". Wayback Machine_sentence_111

Later, it was clarified that lawyers from the Church of Scientology had demanded the removal and that the site owners did not want their material removed. Wayback Machine_sentence_112

Healthcare Advocates, Inc. Wayback Machine_section_16

In 2003, Harding Earley Follmer & Frailey defended a client from a trademark dispute using the Archive's Wayback Machine. Wayback Machine_sentence_113

The attorneys were able to demonstrate that the claims made by the plaintiff were invalid, based on the content of their website from several years prior. Wayback Machine_sentence_114

The plaintiff, Healthcare Advocates, then amended their complaint to include the Internet Archive, accusing the organization of copyright infringement as well as violations of the DMCA and the Computer Fraud and Abuse Act. Wayback Machine_sentence_115

Healthcare Advocates claimed that, since they had installed a robots.txt file on their website, even if after the initial lawsuit was filed, the Archive should have removed all previous copies of the plaintiff website from the Wayback Machine, however, some material continued to be publicly visible on Wayback. Wayback Machine_sentence_116

The lawsuit was settled out of court, after Wayback fixed the problem. Wayback Machine_sentence_117

Suzanne Shell Wayback Machine_section_17

Activist Suzanne Shell filed suit in December 2005, demanding Internet Archive pay her US$100,000 for archiving her website profane-justice.org between 1999 and 2004. Wayback Machine_sentence_118

Internet Archive filed a declaratory judgment action in the United States District Court for the Northern District of California on January 20, 2006, seeking a judicial determination that Internet Archive did not violate Shell's copyright. Wayback Machine_sentence_119

Shell responded and brought a countersuit against Internet Archive for archiving her site, which she alleges is in violation of her terms of service. Wayback Machine_sentence_120

On February 13, 2007, a judge for the United States District Court for the District of Colorado dismissed all counterclaims except breach of contract. Wayback Machine_sentence_121

The Internet Archive did not move to dismiss copyright infringement claims Shell asserted arising out of its copying activities, which would also go forward. Wayback Machine_sentence_122

On April 25, 2007, Internet Archive and Suzanne Shell jointly announced the settlement of their lawsuit. Wayback Machine_sentence_123

The Internet Archive said it "...has no interest in including materials in the Wayback Machine of persons who do not wish to have their Web content archived. Wayback Machine_sentence_124

We recognize that Ms Shell has a valid and enforceable copyright in her Web site and we regret that the inclusion of her Web site in the Wayback Machine resulted in this litigation." Wayback Machine_sentence_125

Shell said, "I respect the historical value of Internet Archive's goal. Wayback Machine_sentence_126

I never intended to interfere with that goal nor cause it any harm." Wayback Machine_sentence_127

Daniel Davydiuk Wayback Machine_section_18

Between 2013 and 2016, a pornographic actor named Daniel Davydiuk tried to remove archived images of himself from the Wayback Machine's archive, first by sending multiple DMCA requests to the archive, and then by appealing to the Federal Court of Canada. Wayback Machine_sentence_128

Censorship and other threats Wayback Machine_section_19

Archive.org is currently blocked in China. Wayback Machine_sentence_129

After the Islamic State terrorist organization was banned, the Internet Archive had been blocked in its entirety in Russia as a host of an outreach video from that organization, for a short time in 2015–16. Wayback Machine_sentence_130

Since 2016 the website has been back, available in its entirety, although local commercial lobbyists are suing the Internet Archive in a local court to ban it on copyright grounds. Wayback Machine_sentence_131

Alison Macrina, director of the Library Freedom Project, notes that "while librarians deeply value individual privacy, we also strongly oppose censorship". Wayback Machine_sentence_132

There are known rare cases where online access to content which "for nothing" has put people in danger was disabled by the website. Wayback Machine_sentence_133

Other threats include natural disasters, destruction (remote or physical), manipulation of the archive's contents (see also: cyberattack, backup), problematic copyright laws and surveillance of the site's users. Wayback Machine_sentence_134

Kevin Vaughan suspects that in the long-term of multiple generations "next to nothing" will survive in a useful way, stating, "If we have continuity in our technological civilization" by which "a lot of the bare data will remain findable and searchable". Wayback Machine_sentence_135

In an article reflecting on the preservation of human knowledge, The Atlantic has commented that the Internet Archive, which describes itself to be built for the long-term, "is working furiously to capture data before it disappears without any long-term infrastructure to speak of." Wayback Machine_sentence_136

See also Wayback Machine_section_20

Credits to the contents of this page go to the authors of the corresponding Wikipedia page: en.wikipedia.org/wiki/Wayback Machine.