So the concept of the Invisible Web has been around since 1994…when the term was first coined by Jill Ellsworth (as cited by M.K. Bergman in “The Deep Web: Surfacing Hidden Value” in 2001). But as with much of cyberspace and life, one can be a late-comer to terms. The changing sophistication of search engines is making the invisible Web much more findable…but first, a little more about this term.
So the Invisible Web (or Deep Web) refers to the parts of the Web that are not as easily findable by the spiders of search engines. These are unconnected pages and contents. These are dynamic pages which are formed in response to certain queries and searches. These are databases of contents hidden behind authentication layers, which require registration. These are also repositories of private information.
Various sources suggest that this Invisible Web is 500 times the size of the so-called Surface Web (aka Searchable Web). This Invisible Web is said to consist of some 91,000 terabytes of information as compared to the 167 terabytes of the Surface or Visible Web. One implication is that there is a much larger cloud of information than one imagines. It’s a guarantee that one’s personal information is much more widespread than one might want.
Another implication is that very dedicated information seekers may find a lot more than they may have thought was available. Anyone who has engaged in deep searching and research will run up against the limits of known information, but the surprise now is that, yikes, there’s more, much much more—potentially.
There’s more than the superficial. In other words, there may be deep-sea profundity if one were to probe much more deeply.
While the term is full of mystique, as one begins to probe further, it turns out that one has been accessing the Deep Web on a daily basis anyway…in terms of regular accessing of library-enabled databases (to which the university subscribes).
Google Scholar (http://scholar.google.com/) also access federated data, from various databases. So does OAISTER (http://www.oclc.org/oaister/default.htm), which connects the world’s libraries…through metadata.
I did find plenty of database search engines…and other tools that enable federated searches. Many of these seem to have been created by librarians, in their archival and search work. Many of these tools are listed here. http://www.searchtools.com/info/database-search.html
The Invisible Web: http://www.insibileweb.com
Internet Public Library Reference Center: http://www.ipl.org/ref
Librarians’ Index to the Internet: http://lii.org
Lycos Invisible Web Catalog: http://www.lycos.com/
The Scout Report: http://www.scout.cs.wisc.edu/report/sr/current/index/html
The WWW Virtual Library: http://vlib.org/
Quite a few resources that were referred to were not live. Quite a few others were dead links. In poking around these search tools, I found that many turn up fewer links but often more relevant ones.
“Invisible or Deep Web: What it is, How to find it, and Its inherent ambiguity”: http://www.lib.berkeley.edu/TeachingLib/Guides/Internet/InvisibleWeb.html
“Deep Web” (Wikipedia): http://en.wikipedia.org/wiki/Deep_Web
“The Invisible Web: How to Find and Search the Invisible Web”: http://websearch.about.com/od/invisibleweb/a/invisible_web.htm
Webliminal.com’s “The Invisible Web”: http://www.webliminal.com/essentialweb/invisible.html