Research on the Free Web Tutorial: G. Beyond Google: the Deep Web

The deep (or "invisible") web are the parts of the web that are freely available, but that search engines cannot see. This information is usually:

• Database content: password protected, intentionally hidden by the database producers, or for technical reasons hard for spiders to crawl.
• Webpages intentionally hidden by the producers (illegal, illicit, unethical, or just plain private)
• Non-text files (such as many PDFs, images, videos, multimedia, and software)
• Some dynamic social media content (comments, Twitter...)

The line between the surface web and the deep web is constantly moving, and regular search engines continue to improve their ability to find some of this content, but it still takes a bit of savvy to locate the vast quantity of valuable goods in the deep web.

The deep web is so large and complex that there is no single good tool or way to find exactly what you want. Here are several strategies:

1.   Use a general tool designed to search the deep web:

• Complete Planet
• InfoMine

2.   Use a regular search engine to find an entry point to a piece of the deep web, usually a database

• Search with your topic and the word "database" or "record" or "data" or "statistics" or a similar variation

3.   Search for a tool that will lead you to the type of information resource you want, for example:

• The Library of Congress Jukebox
• USGS National Geologic Map Database

Well-informed people are essential to a democracy, and information is essential to becoming well-informed. This is enabled by the system of public libraries in our communities, the many debates about the quality and content of public education AND...information produced and distributed by government agencies about all sorts of things.

According to the Federal Library Depository Act of 1962, a government document is defined as “informational matter which is published as an individual document at government expense, or as required by law”. In the pre-web world, certain libraries were part of the depository program and government publications were sent to them to be put on the shelves. Now we can find almost all of it through the web. This information is an underutilized goldmine for academia because it is authoritative, generally unbiased, thorough, and well-researched. 

In addition to single documents, there are many internal databases, such as census information, that are on the deep web.

Strategies for locating Government Documents and databases:

• Use a search tool designed for it, such as: 
  • FDSys
  • USA.gov
  • Catalog of U.S. Government Publications
• If you need a particular type of information, then:
  • Use Google to find a search tool or database designed for the type you need (e.g. searching for Government Statistics gets you both Data.gov and the Data and Statistics section of USA.gov)
• If you need a particular source of information, then:
  • Determine which government agencies (such as the Center for Disease Control or the Bureau of Labor Statistics) might have what you need and go directly to their website.
• Ask a Librarian

Some government documents can be enormously significant, impact society in profound ways, or are treasure troves of data. Some represent legislation, supreme court decisions, or public policy. But others are just, well, goofy. Check these out:

• Preparedness 101: Zombie Pandemic (large PDF file of graphic novella)
• Best Titles Ever (photos of actual government documents)
• "Regulations Governing Mountain Climbing Expeditions in Nepal - Relating to Yeti"
• Unexplained phenomenon from the FBI