Deep Web Search Engines

Where to start a deep web search is easy. You hit Google.com and when you brick wall it, you go to scholar.google.com which is the academic database of Google.
After you brick wall there, your true deep web search begins. You need to know something about your topic in order to choose the next tool. To be fair, some of these sites have improved their index-ability with Google and are now technically no longer Deep Web, rather kind-of-deep-web. However, there are only a few that have done so.  I recommend you use your browsers ‘search’ option to locate on this page your topic of interest, as the page has gotten long.
To all the 35F and 35G’s out there at Fort Huachuca and elsewhere, you will find some useful links here to hone in on your AO.

If you find a bad link, Comment the link below. I try to keep this up to date, but only with your help!

Last updated Jan 1, 2018 – updated 2 economic assets


Multi Search engines

Deeperweb.com – (broken as of Sept 2016, hopefully not dead) This is my favorite search engine. It breaks your results down into categories – general web, blogs, news, academic, cloud, metrics, research, etc. This allows you to quickly focus on the type of answer you were looking for.  Makes my top 10 websites!
Surfwax – They have a 2011 interface for rss and a 2009 interface I think is better. Takes 60 seconds to understand how to use it.
Dogpile – another multi engine aggregator
Scout Project- scout.wisc.edu — Since 1994, the Scout Project has focused on developing better tools and services for finding, filtering, and presenting online information and metadata.
www.findsmarter.com – You can filter the search by domain extension, or by topic which is quite neat. Sources Yahoo, Bing, Wiki, Blekko, and Alltheweb

Cluster Analysis Engine

TouchGraph – A brilliant clustering tool that shows you relationships in your search results using a damn spiffy visualization. The smart way to use it, is to let it help you find new sources to your search topic. I have to add, the wiggly effect on the visualization is damn cool, just grab the center item and move it to understand what I’m talking about. (sometimes it doesn’t wiggle, however. Java issue?)
Yippy.com – A useful, non-graphical clustering of results. Give it 2 minutes of your time to understand how it works and it will give back hours of saved research time. They are actively funded and are acquiring other search tech assets, so worth using.
www.navagent.com/ – Not a web based search engine, requires you download software. Highly rated, very interesting especially to the 35F intel types.

Speciality Deep Web Engines

Archive.org – Huge behemoth of media now public domain – rare books, sound recordings, video, 20 year archived images of all old websites, and free audio books! Makes my top 10 list. (and my top 3)
WWW virtual library – a listing of indexes to industries. Need to know about Architecture? Biochemical war? Zoology? This may get you there.
FindArticles.com – FindArticles has articles from about 500 periodicals with coverage back to 1998, and is completely free of charge. Appears funded by CBS.
Library of Congress – loc.gov  – Phenomenal digitized archives, “American Memory” especially interesting. Includes a good newspaper archive.
National Security Archive – Declassified papers and such. In their words – “National Security Archive Electronic Briefing Books provide online access to critical declassified records on issues including U.S. national security, foreign policy, diplomatic and military history, intelligence policy, and more.  Updated frequently, the Electronic Briefing Books represent just a small sample of the documents in our published and unpublished collections.”
www.osti.gov – Government research archives, if your tax dollars paid for it, the results are here. Also a huge collection of science presentation videos.
US Geologic Survey – Imagery and Maps galore. 3 portals to fetch data, EarthExplore, Glovis and Seamless. Extremely complex.  There are tutorials on how to get free aerial photos over at learninggis.com.
US National Map by USGS – The source for current geospatial data from the USGS.
http://adswww.harvard.edu/ – Physics and Astronomy data engine for academic papers
http://databank.worldbank.org/data/home.aspx  – Specialty statistical data on all kinds of subjects, from countries GDP to levels of blindness.
http://www.quandl.com/ – For acquiring raw data for studies, useful for data scientists.  An awesome collection of 9,000,000 of financial, economic, and social datasets.

General

GPO’s Catalog of US Government Publications — Federal publications database.
Smithsonian Institution Libraries — 20 libraries from museum complexes around the world.
The National Archives — National Archives’ research tools and online databases.
HighWire Press —  Online catalog of the largest repository of free full-text and non-free text, peer-reviewed content, from over 1000 different journals.
Education Resources Information Center (ERIC) — A catalog with more than 1.2 million bibliographic records, many with full text. Sponsored by the US Department of Education.
Encyclopædia Britannica — The old and authoritative encyclopedia searchable with full text online. No longer printing paper copies. The internet killed the library star.
Topix.net — A news search engine.
Internet Public Library — Internet’s public library. More for kids than adults, IMHO.
San Francisco Public Library – A great online library. This is just one example of many such local public libraries that offer similar services. Sorry, you can’t use their access to commercial archives unless you are a member.
Xrefer (commercial) — Fee based database of 236 titles and over 2.9 million entries.
LexisNexis (commercial) — Billed as the world’s largest collection of public records, unpublished opinions, legal, news, and business information. Over 35,000 individual sources are claimed as searchable. I’ve not been able to justify subscribing, so I don’t know.
Forrester Research (Commercial) — An independent technology and market research company, publishing in-depth research reports on a variety of subjects.
Factiva (Commercial) — Online collection of about 10,000 individual sources. It’s a fact.
Online Archive of California — Has the phenomenal archive of all the Keystone Stereoscopic images from 1860-1968

File and Metadata Discovery

www.findthatfile.com – locates files even if embedded or hidden in a .zip format. Also checks podcasts, FTP and other venues.  (Appears to be dead.)

Find People & Background Checks

Pipl.com– for finding people
zabasearch.com – finding people
Intellus (commercial) – Finding people plus background checks on people and other features.
US Search (commercial) – Finding people plus background checks on people.
123 People (commercial) – A multi search engine built around finding people.
Integrascan – Finding people plus background checks on people.
State of Texas DOT Criminal Background Check – The central background check for felonies provided by the state. Most misdemeanours don’t show up.
socialcatfish.com – Has background check capability plus a photo reverse lookup. Deeper level of search for subscription.

Books Online

Archive.org – Has books online in epub, txt, and pdf formats. The collection encompasses others such as Gutenberg Press, etc. So this is the best site to start with. Again, this makes my top 10 websites. Share the love.
Hathi Trust – http://www.hathitrust.org/  — a partnership of major research institutions and libraries working to ensure that the cultural record is preserved and accessible long into the future. There are more than sixty partners in HathiTrust, and membership is open to institutions worldwide.
Books.google.com – They are putting the squeeze on all the book scanning businesses. They want to scan the world to add it to the Google Borg. You will be assimilated.
The Online Books Page — A searchable database of more than 28,000 English works with full text available for free online.
Bibliomania — A database of free literature from more than 2,000 classic texts. Archive.org crushes this.
Project Gutenberg — The granddaddy of online books with a catalog of more than 20,000 free books with full text available online. Included in Archive.org.
The National Academies Press — Only about 3,000 free books online and ~900 for-sale PDFs.
UNZ – An odd collection of periodicals and book scanned.
Get Abstracts (commercial) — Large online library of more than 8,000 business book summaries. It is the most efficient way to get the best business titles.
Getty Research Institute – http://www.getty.edu/research/library/ – The Getty Research Institute library collections include over one million books, periodicals, study photographs, and auction catalogs as well as extensive special collections of rare and unique materials. Focusing on art history, architecture, and related fields, they begin with the archaeology of prehistory and extend to the contemporary moment.

Newspaper Archives Online

US newspaper coverage – ResearchGuides lists links to many newspapers
Library of Congress Newspaper Resource List – LOC does a great job getting the list together of wonderful newspaper archives.
NewspaperArchive.com – (commercial) – Known for a large collection. I haven’t used it, so I can’t confirm this.
xooxleanswers.com – Great list of newspaper archives from Xooxle. Good list and a funky name. Two thumbs up.
University of Penn Newspaper Archive – List of US newspaper archives and dates. Looked like a deeper list of Texas newspapers, so this effort may be a deep comprehensive list.
Australia Newspaper archives – http://trove.nla.gov.au/ndp/del/about – A phenomenal newspaper archive of most Aussie newspapers from 1820’s going to 1980’s. Including Taz as well.

Audio Books Online

LibreVox.org – Huge selection of audio recordings and AudioBooks read by volunteer voice artists

Videos

www.liveleak.com – A Video news aggregator of citizen supplied videos. Great for OSInt in foreign countries. The site appears to have some good aggregation functions to turn randomly submitted videos into a logical collection around a topic.  Proven useful for aggregating the Feb 15, 2013 meteorite strike in Russia.
VideoLectures.net – High quality videos of science based lectures.

Business Deep Web Engines

AAAAgencySearch.com — Advertising agencies via the American Association of Advertising Agencies.
Agency ComPile — Advertising and marketing agencies.
Alibaba – An international marketplace of businesses looking for businesses.
Kompass — Business to business search engine.
GPO Access Economic Indicators — Gateway for economic indicators from January 1998 to present.
Government Printing Office — Big catalog of stuff published by the Government Printing office. Has business stuff but much much more. Environmental reports, legal docs, nature stuff. Hell, I typed in ‘mushroom’ and pulled up 34 entries.
Hoover’s — The Big Boy of info on businesses.
ThomasNet — Just an industrial product search directory.
SBA Loan Data — Loan program approval activity from the Small Business Administration.
GuideStar.org — A searchable database of non-profit organizations including 501c.
Slideshare.net – A searchable database for slides uploaded by crowdsourcing.

Consumer Engines

US Consumer Products Safety Commission Recalled Products — Listing of products, sortable by company name.
Melissa Data — Comprehensive directory of demographic data, sortable by ZIP code.
Kelley Blue Book — A  guide to pricing new and used vehicles.
Edmunds — A recognized and established guide to pricing new and used vehicles.
Consumer Reports — (commercial)  A trusted guide to product reviews, including autos, appliances, electronics, computers, personal finance, etc. I use it.

Economic and Job Search Engines

EUROPA Press Release Database — Database of press releases by the European Union.
FreeLunch.com —Directory of free economic data.
Bureau of Labor Statistics — Job-based and consumer economic info from the US Department of Labor.
Salary Wizard Calculator — Tools that shows national average salaries adjusted by location for different jobs.
Economagic — A data directory containing over 200,000 econ files.
Penn World Tables — National income data for all countries for the years 1950-2007.
America’s Job Bank — Database of jobs and resumes.
USAJOBS — Portal of data on federal government jobs.
Regional Economic Conditions (RECON) — Economic data available by state, county, and MSA.
Federal Reserve –  FRED contains a plethora of economic indicator datasets, something over 500,000
Smartscholar – Link of Economic based links some with Deep Web assets.

Finance

Bankrate.com — Database of interest rates for different loan types, mortgages, and savings accounts.
InvestIQ — Market data from around the world in regions.
BigCharts — Quotes and performance charts on different stocks and mutual funds.
SmartMoney.com Tools — A portal of for stock analysis tools.
NASDAQ Trader — A database of stock data from the NASDAQ stock exchange.
SEC Info — EDGAR and SEC filings searchable by name, industry, SIC code, etc.
EDGAR Online — SEC filings searchable by ticker or company name.

Government Search Engines

Copyright Records (LOCIS) — Online copyright records, documents, serials, and multimedia.
American FactFinder — Aggregate census bureau data to be searched by city, county, or ZIP code.
FedStats — Gateway for statistics on 100 US federal agencies.
United States Patent and Trademark Office — Database of patents,  full-text and full-page images.
Historical Census Browser — Repository of historical US census data. Going back to 1790 compiled by the University of Virginia.
Geospatial One Stop — Awesome GIS data warehouse of geographic data, shapefiles, imagery, and displayable on maps.
Grants.gov — Grant opportunities, from everything under the sun.
Technology Opportunities Program Grants Database — Listing of technology grants, peruse by keyword, state, and year.
United States Government Printing Office (GPO) — I mentioned this earlier, they seem to have everything. A search engine for mutliple government databases: US budgets, campaign reform hearings, code of federal regulations, congressional bills, etc
CIA Electronic Reading Room — The usual uninteresting declassified CIA documents.
POW/MIA Databases and Documents — Info on POWs and MIAs.
ZIP+4 Lookup — US ZIP codes and ZIP+4 codes

International Search Engines

International Data Base (IDB) — Statistical tables and demographic information for countries and areas.
FIRST — http://first.sipri.org/ Military aggression database and weapons holdings.
Economics of Tobacco Control — Information regarding tobacco usage and policy for180 countries.
Country Indicators for Foreign Policy —Statistical tables on countries’ foreign policies.
World Bank Data — Key development data and statistics for countries and worldwide groups.
CIA Factbook — Reference materials containing information on every country in the world.
US International Trade Statistics — International trade statistics,by country or type of good.
US Foreign Trade Highlights — Information of US international trade.
Energy Information Administration International Energy Data and Analysis — Energy balances sorted by country. Explains why the US gives huge amounts of cash to oil producing countries.

Law and Politics

THOMAS (Library of Congress) — Legislative information from the Library of Congress.
Law Library of Congress — Allegedly, the largest collection of legal materials in the world, over 2 million volumes.
Global Legal Information Network — Laws, regulations, judicial decisions, and other legal sources.
FindLaw — Free legal database, with collections of cases and codes, legal news.
Office of Postsecondary Education Security Statistics — Contains college campus crime statistics, sortable.
Bureau of Justice Statistics — http://bjs.ojp.usdoj.gov/ Legal and judicial statistics, everything from crime to law enforcement.
The Avalon Project at Yale Law School — Documents in law, history, and diplomacy.
US Congressional Documents and Debates, 1774–1873 (Library of Congress) — The historical Congressional records, bills, documents, statutes, journals, and debates from LOC.
Lobbying Database — Who spent what on firms who have spent lobbying money from 1998. The US, with the finest Congress money can buy!
Legislative Activities — Synopsis of House of Representatives’ bill summary, status, public laws, and votes.
heinonline.org  – (commercial) – Claims to be the ‘worlds largest image based database of legal documents’ . I was able to find an obtuse document on using Bayes Theorem for fact finding in a criminal case.
Project Vote Smart — Government officials and election candidates database, order by last name or ZIP code.

Medical and Health

PubMed  — The U.S. National Library of Medicine contains over 16 million citations from MEDLINE and other life science journals going back to the 1950s. Contains links to full-text articles and external resources. Supposed to be the best damn resource for medical out there.
National Institute for Health Research Archive — http://www.nihr.ac.uk/Pages/NIHRArchive.aspx Database of ongoing or completed projects funded by the British  NHS.
National Institutes of Health — Encyclopedia of health topics. More of a kindof-deep-web resource, as Google has this indexed.
American Hospital Directory — Index of US hospital information.
Globalhealthfacts.org — Searchable world health information, by country, disease, condition, program, or demographic. Quickly lay out the conditions in a country.
Centers for Disease Control and Prevention Data and Statistics — Statistical health information according to the CDC. Not true deep web, but useful.
New England Journal of Medicine  (commercial) — A Leading medical journal with full text past issues available online. Commericial, but you can access most for free.

Science and Academic

ScienceResearch.com — Searchable access to scientific journals and databases.
Academic Index – Main search is a filtered Google search aimed at high authority rank sites, mainly .edu and .gov which filters a great deal out. Second search ties into deep web academic and non-academic databases skewed to librarians and educators.
Science.gov — Gateway to  science info provided by US government agencies.
VideoLectures.net –  Phenomenal video lecture coverage from high authority rank sources.  A great go-to place to find peer-reviewed, conference presented, in depth coverage of a topic at a conference. A nice bonus, is the presentation slides are shown separately, and you can jump to slides of interest to you. Heavily technology based, and 66% is in English. Most lectures 45 minutes or longer.
WebCASPAR — A horrible interface to an alleged wealth of statistical info on science and engineering. I found the site slow, cludgy and designed around 1965 run off of candle power.  From their website:”The WebCASPAR database provides easy access to a large body of statistical data resources for science and engineering (S&E) at U.S. academic institutions. WebCASPAR emphasizes S&E, but its data resources also provide information on non-S&E fields and higher education in general. ”
The Complete Work of Charles Darwin — Charles Darwin’s published works,  search-able and available online. He’s still old and his works still ramble. Scanning didn’t help him much.
USGS Real-Time Water Data — Real-time map of streamflow and water quality data of the USA’s rivers and reservoirs.
USGS Earthquake Hazards Program — Showing real-time earthquake data. Focus on US but covers world as well.
IEEE Publications (Commercial) — Contains over 1.4 million documents from the Institute of Electronics and Electrical Engineers.
Society of Petroleum Engineers Archive (Commercial) — Petroleum engineers technical papers.
Arxiv – arxiv.org/ — Cornell University repository. Access to 700,000+  technical papers on everything from quantitative biology to computer science. Appears to offer full text in several formats.
VADLO – www.vadlo.com/ — Life Science Search Engine. Very hit and miss. Don’t have high expectations.
Deep Dyve  (Commercial) — www.deepdyve.com   DeepDyve has aggregated millions of articles across thousands of journals from the world’s leading publishers, including Springer, Nature Publishing Group, Wiley-Blackwell and more. Haven’t paid the premium to give it a test ride, if someone has, please write a review below.
Data Mining Data sources – http://www.kdnuggets.com/datasets/index.html  – Links to gobs of free and commercial datasets used for data mining.

Transportation

FAA Flight Delay Information — A map of the United States with flight delay information from the nation’s largest airports.
NTSB Accident Database and Synopses — The National Transportation Safety Board’s database of aviation accidents, ranging from 1962 to present.
NTSB Aviation Accident Database— Aviation accident data from the National Transportation Safety Board
National Highway Traffic Safety Administration — A database of car and car part defects, searchable by item number or car make and model.
SaferCar.gov — Crash test safety ratings for automobiles since 1990.
flightwise.com — Real-time flight tracking, with support for Google Earth.
FlightAware — Fee flight tracking with history, graphs, and maps.

History

Ohio State – Compiles lots of Civil War info on troop movements, camps, battles, etc.

Cyber War / InfoSec

Iron Geek irongeek.com — An excellent library of videos explaining many facets of InfoSec and hacking & security
Security Tube – securitytube.net — A large library of videos covering many topics in InfoSec, cyberwar, and most of the hacking conferences.
DefCon — The main hackers Con, so well known that now the Feds send their folks here and it has become a wild west training ground for coming trends. Archives go back to Defcon 1. They are now on Defcon 20, I think.
semanticommunity.info– General military hardware overview. Some deeper links to deep web
Shodan – The deep web search for what things are connected to the internet.  Controversial, but a good tool.

Hey people, we are cool folks here! If you know of a useful deep web resource, put a comment below and share the love!