Skip to main content

Research on the Free Web Tutorial: G. Beyond Google: the Deep Web

What is the Deep Web?

The deep (or "invisible") web are the parts of the web that are freely available, but that search engines cannot see.  This information is usually:

  • database content: password protected, intentionally hidden by the database producers, or for technical reasons hard for spiders to crawl.
  • Webpages intentionally hidden by the producers (illegal, illicit, unethical, or just plain private)
  • non-text files (such as many PDFs, images, videos, multimedia, software)
  • some dynamic social media content (comments, twitter...)

The line between the surface web and the deep web is constantly moving, and regular search engines continue to improve their ability to find some of this content, but it still takes a bit of savvy to locate the vast quantity of valuable goods in the deep web.

Searching the Deep Web

The deep web is so large and complex that it there is no single good tool or way to find exactly what you want.  Here are several strategies:

1.   Use a general tool designed to search the deep web:

2.   Use a regular search engine to find an entry point to a piece of the deep web, usually a database

  • Search with your topic and the word "database" or "record" or "data" or "statistics" or a similar variation

3.   Search for a tool that will lead you to the type of information resource you want, for example:

Government Information

The United States, democracy and free country that it is, considers an educated citizenry to be essential.  Thus the system of public libraries in our communities, the many debates about the quality and content of public education AND...information produced and distributed by government agencies about all sorts of things.

According to the Federal Library Depository Act of 1962, a government document is defined as “informational matter which is published as an individual document at government expense, or as required by law”.  In the pre-web world, certain libraries were part of the depository program and the government publications were sent to them to be put on the shelves.  Now we can find almost all of it through the web.  This information is an underutilized goldmine for academia because it is authoritative, generally unbiased, thorough, and well-researched. 

In addition to single documents, there are many internal databases, such as census information, that are in the deep web.

Strategies for locating Government Documents and databases:

  • Use a search tool designed for it, such as: 
  • If you need a particular type information, then:
  • If you need a particular source of information, then:
    • Determine which government agency (such as the Center for Disease Control or the Bureau of Labor Statistics) might have what you need and go directly to their web site.
  • Ask a Librarian

Cool Gov Docs - Yes!

Some government documents can be enormously significant, impact society in profound ways, or have treasure troves of data.  Some are legislation, supreme court decisions or policy.  But others are just, well, goofy.  Check these out:

 

End of Module 2

 

Previous  /  Begin Module 3