Blog Entry

A Tour of the Invisible (Deep) Web

12 comments

So the concept of the Invisible Web has been around since 1994…when the term was first coined by Jill Ellsworth (as cited by M.K. Bergman in “The Deep Web: Surfacing Hidden Value” in 2001). But as with much of cyberspace and life, one can be a late-comer to terms. The changing sophistication of search engines is making the invisible Web much more findable…but first, a little more about this term.

So the Invisible Web (or Deep Web) refers to the parts of the Web that are not as easily findable by the spiders of search engines. These are unconnected pages and contents. These are dynamic pages which are formed in response to certain queries and searches. These are databases of contents hidden behind authentication layers, which require registration. These are also repositories of private information.

The Implications of the Invisible Web

Various sources suggest that this Invisible Web is 500 times the size of the so-called Surface Web (aka Searchable Web). This Invisible Web is said to consist of some 91,000 terabytes of information as compared to the 167 terabytes of the Surface or Visible Web. One implication is that there is a much larger cloud of information than one imagines. It’s a guarantee that one’s personal information is much more widespread than one might want.

Another implication is that very dedicated information seekers may find a lot more than they may have thought was available. Anyone who has engaged in deep searching and research will run up against the limits of known information, but the surprise now is that, yikes, there’s more, much much more—potentially.

There’s more than the superficial. In other words, there may be deep-sea profundity if one were to probe much more deeply.

Less Mystique

While the term is full of mystique, as one begins to probe further, it turns out that one has been accessing the Deep Web on a daily basis anyway…in terms of regular accessing of library-enabled databases (to which the university subscribes).

Google Scholar (http://scholar.google.com/) also access federated data, from various databases. So does OAISTER (http://www.oclc.org/oaister/default.htm), which connects the world’s libraries…through metadata.

I did find plenty of database search engines…and other tools that enable federated searches. Many of these seem to have been created by librarians, in their archival and search work. Many of these tools are listed here. http://www.searchtools.com/info/database-search.html

Some Tools to Access the Invisible Web

Beaucoup: http://www.beaucoup.com

Complete Planet: http://aip.completeplanet.com/aip-engines/browse?thisPage=%2Fbrowse%2Fbrowse.jsp&succes...

DeepPeep: http://www.deeppeep.org/

DeepWebTech: http://www.deepwebtech.com/

Fossick.com: http://www.fossick.com

IncyWincy: http://www.incywincy.com/

InfoMine: http://infomine.ucr.edu/

Infoplease: http://www.infoplease.com/index.html

The Invisible Web: http://www.insibileweb.com

Internet Public Library Reference Center: http://www.ipl.org/ref

Intute: http://www.intute.ac.uk/

Librarians’ Index to the Internet: http://lii.org

LibrarySpot: http://www.libraryspot.com

Lycos Invisible Web Catalog: http://www.lycos.com/

Scirus: http://www.scirus.com/srsapp/

The Scout Report: http://www.scout.cs.wisc.edu/report/sr/current/index/html

TechXtra: http://www.techxtra.ac.uk/index.html

The WWW Virtual Library: http://vlib.org/

Quite a few resources that were referred to were not live. Quite a few others were dead links. In poking around these search tools, I found that many turn up fewer links but often more relevant ones.

General Information about the Invisible Web

“Invisible or Deep Web: What it is, How to find it, and Its inherent ambiguity”: http://www.lib.berkeley.edu/TeachingLib/Guides/Internet/InvisibleWeb.html

“Deep Web” (Wikipedia): http://en.wikipedia.org/wiki/Deep_Web

“The Invisible Web: How to Find and Search the Invisible Web”: http://websearch.about.com/od/invisibleweb/a/invisible_web.htm

Webliminal.com’s “The Invisible Web”: http://www.webliminal.com/essentialweb/invisible.html

Comments

GFD 3 years, 11 months ago

pipl.com claims to be searching the deep web, is that related to this in any way? Thx for a nice post, learned some new interesting things.

Eruditio Loginquitas 3 years, 11 months ago

Hello, GFD: Even from a cursory look at the pipl site, it look like this site surfaces information from the "deep web" and can pull up some pretty impressive results.

Thanks for your comment. Without it, I would not have dug further about this site.

jack thompson 3 years, 11 months ago

Google is big brother for invisible web. i am also developing my own simple web crawler :)

Cheap Air Tickets 3 years, 11 months ago

These facts are really surprising, why don't search engines try to explore this side of the coin. They can come up with lots of informative content for web users.

Cheap Air Tickets India 3 years, 11 months ago

These facts are really surprising, why don't search engines try to explore this side of the coin. They can come up with lots of informative content for web users.

Rapid Free Traffic 3 years, 11 months ago

:| I tried pipl.com This is very extensive search.

I tried my name and I tried my name on google with my city and country.

Google came up with some thousand results only a few of them relevant. Pipl came up with 30 results all pointing to me! That makes me uncomfortable!

sears carpet cleaning 3 years ago

It is interesting what you can learn if you look hard enough in the search engines these days.

computer shop perth 2 years, 7 months ago

The Invisible web is also known as The Deep web. Though I have not heard much about it but I certainly know that search engines index less than 10% of the entire web and the remaining 90% is called the Invisible web.

Michael Oakwood 2 years, 4 months ago

Search engines is pandora's box. It is hard to learn their language

Jacob Nolan 2 years, 3 months ago

This just shows how important it is to check different search engines and don't just stick to one. Variety is the spice of life!

Lauren Williams 2 years, 2 months ago

There is a difference between the Deep Web and the hidden networks within it like Tor. The best guide I've found so far is "Deep Web Secrecy and Security". It's an inter-active ebook on Amazon.

Joe Fletcher 2 years ago

The invisible web? At least it doesn't sound black hat or could harm you in anyway!!!

Post a comment Please do not post URLs in the text of your response. Thanks.

Comments are closed and no additional comments will be allowed.