How Big Is the Searchable Web?

We spend a lot of time thinking about web research. How do we find the right information? How do we exclude irrelevant information? And how do we do this regardless of how names are written or presented in source documents?

Search engines have discovered trillions of web addresses, yet only a portion of those pages make it into searchable indexes. Google alone indexes hundreds of billions of pages, and a growing number of independent search engines maintain their own indexes.

For anyone conducting due diligence or other online research, this raises an interesting question: How much of the web can we actually search, and how much of that information is useful for what we're trying to find?

The first question, how big is the web?

Spoiler: Nobody knows exactly how many web pages exist. Websites are constantly being created, updated, and removed. Some pages are public, others require special access, and many are generated automatically by websites and databases.

Leaving aside things locked in databases (which is the majority of the deep web), web crawlers help us understand the scale. These automated programs visit websites, follow links, and record the addresses of pages they discover.

In 2017, Google reported that its crawling systems processed more than 130 trillion web addresses. That figure is nearly a decade old, but it remains one of Google's largest publicly disclosed measurements of web discovery.

Other web crawlers provide more recent evidence. Majestic, a company that maps links between websites, reports that its historical database contains approximately 21.7 trillion discovered URLs, including 4.5 trillion URLs it actually crawled. That collection covers the period from 2006 through March 2024.

A URL is a web address, and finding an address does not mean the page still exists or that its contents have been collected. A single document can also appear at several addresses, while websites can generate large numbers of pages containing very similar information.

The number of unique web pages containing useful content is therefore considerably smaller than the number of URLs crawlers have discovered.

How much of the web is indexed?

When a search engine discovers a page, it decides whether to retrieve and store information about its contents. That information becomes part of a search index, which the engine uses to find pages matching a search.

Maintaining an index takes significant computing resources. Search providers must decide which pages to include, which to exclude, and how often to check them for changes.

Google currently describes its index as containing hundreds of billions of web pages. It does not publish an exact current total.

One of the clearest comparisons comes from the UK's Competition and Markets Authority (CMA). In its 2020 investigation of search competition, the CMA obtained figures directly from Google and Microsoft. It reported that Google's index contained approximately 500 to 600 billion pages, while Microsoft's Bing index held 100 to 200 billion pages.

Since that investigation, several independent search providers have grown significantly. Some now publish their own index sizes.

There is also a distinction between the number of URLs recorded inside an index and the number of pages whose content can actually be searched.

In October 2025, a U.S. federal court order described Bing's index as containing upwards of one trillion URLs. Microsoft testimony cited in the order explained that a URL may exist in the index without the page's actual content being available.

This example helps explain the gap between the trillions of URLs found by crawlers and the much smaller collections available for content-based search.

We can reasonably say that major search engines make hundreds of billions of web pages searchable. The number of distinct pages available across all independent indexes remains unknown. The other challenge is there is no visibility into the overlap between these indexes.

How much of that information actually matters?

The size of a web index tells us something about its potential coverage. For professional research, the next question is whether that coverage includes the information needed for a particular investigation.

The answer is difficult to quantify. Most of the public web has little relevance to any particular due diligence investigation, and there is no reliable way to determine what percentage of indexed pages contain useful risk information. That percentage also changes depending on the person, company, or type of risk being investigated.

Consider two hypothetical search indexes. One contains a billion pages about cats, while another contains 10,000 regulatory enforcement orders. For financial crime research, the much smaller collection could be far more valuable. The same principle applies to general web indexes, where the number of pages says little about how much relevant information they contain.

A search engine with hundreds of billions of pages could have all those documents indexed and still fail to return them for a particular query. The choice of search terms, names, languages, and sources affects what researchers can find. Some records may be published under a former company name, in another language, or in a document that makes only a brief reference to the company.

There are also important collections of information that deserve direct searches. The U.S. Securities and Exchange Commission's EDGAR system provides access to millions of corporate filings, with full-text search across more than 20 years of records. The ICIJ Offshore Leaks Database contains information on more than 810,000 offshore entities and the people and companies connected to them.

Both are available through the web, but their dedicated search tools and structured data offer ways to investigate relationships and records that ordinary web searches may miss.

This matters particularly for adverse media (or "negative news") screening, a term often associated with searching news articles for negative information. The Wolfsberg Group's 2022 guidance defines negative news much more broadly, covering public information relevant to financial crime risk. Its examples include online news, court records, regulatory disclosures, and penalty notices.

A regulatory enforcement order may never become a newspaper article. A securities filing may disclose an investigation or legal proceeding that would be difficult to find without searching the filing directly.

Making web research useful for due diligence

The growth of independent search engines creates more ways to discover information. Combining several search providers can improve coverage because each has different crawling methods, indexing decisions, and search results.

A more complete research process also draws directly from specialized sources, including government databases, corporate filings, and investigative datasets. Documents can then be processed to extract names, events, dates, and relationships, while preserving links to the original evidence.

This approach is part of how Threat.Digital develops its DiligenAI research technology. DiligenAI searches public-source information across languages and organizes relevant findings about people and companies into structured, source-linked intelligence for due diligence, adverse media, and financial crime research.

Ultimately, there is no meaningful way to determine how much of the indexed web matters for due diligence as a whole. The more useful measure is how much relevant information a research system can find about a particular subject. That includes information found across multiple search engines, information unique to a particular index, and records discovered through targeted sources.

Measuring that coverage requires testing actual investigations and comparing which relevant records each research method finds. A larger index may improve the chances of finding something, but its size alone tells us very little about the quality or completeness of the results.

Or you could just come up with a search string and look at the top 10 results ;-)