Archives & Historical Data

Learning Objectives

• Explain why “deleted” web content usually still exists somewhere
• Use the Wayback Machine, including its CDX API, to find removed pages and past versions
• Compare archive services (Internet Archive, archive.today, Common Crawl) and their trade-offs
• Find historical satellite and street-level imagery for geolocation and change-over-time analysis
• Judge the reliability of archived evidence and cite it correctly

Why History Is a Source

The web is not permanent, but it is also not truly erasable. When an organization cleans up its website (removes a staff page, takes down a leaked document, edits an embarrassing statement), the old version frequently survives in an archive. For an investigator, archives turn “they deleted it” into “let’s go read the old version.”

Historical data has two big advantages:
1. It recovers removed information — pages, documents, and endpoints the target no longer publishes.
2. It stays passive. Reading an archived copy of a target’s site never touches the target’s servers, so nothing lands in their logs (unlike browsing the live site, which is semi-passive — see Module 5 file 03 and Module 8).

The Wayback Machine (Internet Archive)

The Internet Archive’s Wayback Machine (web.archive.org) is the largest web archive, holding snapshots of pages captured over decades. In October 2025 the Internet Archive announced it had archived one trillion web pages.

Core uses:
- View a page as it looked on a specific date; use the calendar to compare versions over time.
- Recover deleted pages, removed PDFs, old staff/leadership listings, and prior wording of statements.
- Read an archived robots.txt or sitemap.xml to learn about paths without touching the live site.
- Use Save Page Now to capture a live page as evidence right now (remember: your saved snapshot becomes public).

The CDX API (the power-user feature):
The Wayback CDX Server API lists every capture the archive holds for a URL or domain, which surfaces forgotten pages and endpoints in bulk.
- Base endpoint: http://web.archive.org/cdx/search/cdx
- Useful parameters:
- url= the target, with matchType=domain to include all subdomains (or prefix/host)
- a trailing * implies prefix mode; a leading *. implies domain mode
- output=json
- collapse=urlkey to remove duplicate URLs
- from=/to= for a date range (yyyyMMddhhmmss)
- filter= to match fields with a regex; fl= to choose returned fields
- Example idea: list all unique URLs the archive has ever seen under a domain, then look for interesting old paths (admin pages, uploads, backups) that may hint at current structure.
- Tools built on it: waybackurls and gau (GetAllUrls) automate pulling a domain’s historical URLs for reconnaissance.

Reliability and caveats:
- Not every page or date is captured; gaps are common.
- Pages may render imperfectly (missing images, broken scripts) because dependencies weren’t archived.
- Availability: in October 2024 the Internet Archive suffered DDoS attacks and a data breach (about 31 million account records exposed), and the Wayback Machine was read-only or offline for parts of that month. Treat it as a critical but occasionally unavailable resource, and don’t depend on a single archive.

archive.today (archive.ph / archive.is)

A separate, independent on-demand archiving service (mirrors include archive.today, archive.ph, archive.is, archive.li, archive.md).

• How it differs from the Wayback Machine:
• It captures on demand and stores two versions: a working copy and a screenshot image.
• It handles JavaScript-heavy sites (maps, some social posts) better than the Wayback Machine often does.
• It ignores robots.txt, so it sometimes has pages the Wayback Machine won’t archive.
• It frequently has snapshots the Internet Archive lacks, and vice versa — check both.
• Caveats (be careful here): archive.today has drawn recent controversy. In October 2025 the FBI subpoenaed its registrar seeking the operator’s identity, and in February 2026 Wikipedia blacklisted its links after evidence of tampered archived content and abuse tied to the service’s infrastructure. The lesson for students: an archive is not automatically trustworthy. Prefer the Internet Archive for evidence when you can, corroborate important snapshots across archives, and be cautious treating a single archive.today capture as ground truth.

Common Crawl

• A massive, free, open dataset of crawled web pages, updated regularly and hosted on AWS.
• It’s built for large-scale research (it’s a common training source for language models), not point-and-click lookups, but it has a searchable URL index (index.commoncrawl.org) you can query by URL pattern via a CDX-style API.
• Use case: finding historical pages or bulk-analyzing a domain’s footprint when you can work with data at scale.

Memento (Aggregated Archive Search)

• The Memento protocol aggregates many web archives at once, so a single lookup can find the closest snapshot of a URL across multiple archives (Internet Archive, national libraries, and others).
• Useful when the Internet Archive has a gap that another archive filled. (Availability of specific Memento front-ends varies — verify before class.)

Historical Imagery (Maps and Satellites)

History isn’t only text. Old imagery is central to geolocation and to proving what a place looked like on a given date (see Module 7 verification and the imagery source list in the intro script).

• Google Earth Pro (desktop): the historical-imagery time slider lets you step back through past satellite/aerial images of a location — invaluable for seeing what stood somewhere before a building, or dating a change.
• Google Maps Street View “See more dates”: on desktop, drop Pegman and click “See more dates” to view earlier street-level captures. Coverage varies by location, and this feature is desktop-focused.
• Why it matters for OSINT: comparing a photo or claim against historical imagery can confirm or debunk when and where something happened. Bellingcat-style investigations lean heavily on this.
• Other providers (Bing Maps, Yandex Maps, and archived satellite datasets) can fill gaps in a specific region.

Caches and “Deleted” Content

• Search-engine caches are mostly gone. Google retired its cache: operator and cached-page links in 2024 and now points users to the Wayback Machine instead (see file 01). Don’t rely on live search caches anymore.
• Other places deleted content lingers:
• RSS feeds and email newsletters that captured the original text
• Quote-tweets, screenshots, and reposts by other users
• Third-party scrapers and aggregators
• Google Docs/Sheets revision history (for documents you have access to)

Ethics and Reliability

• Archives are evidence, so treat them like evidence. Record the archive URL, the capture timestamp, and the original URL (see Module 7). An archived page’s timestamp is often the single most useful fact.
• Archives can be wrong or manipulated. Rendering can be incomplete, timestamps can be misread (a page captured today may display old content), and at least one archive has had tampering allegations. Corroborate important findings across more than one archive.
• Your own captures are public. Using Save Page Now or archive.today publishes the page (and the fact that you archived it). Don’t archive something sensitive you didn’t mean to expose.
• Deleted-but-archived is still sensitive. Recovering someone’s deleted personal post because an archive kept it doesn’t erase the ethics of Module 2 — apply minimization and purpose.

Key Terms

• Snapshot / capture: a saved copy of a web page as it existed at a specific time
• Wayback Machine: the Internet Archive’s web-archiving service at web.archive.org
• CDX API: the Wayback Machine’s query interface for listing all captures of a URL or domain
• On-demand archiving: capturing a page when a user requests it (archive.today’s model)
• Common Crawl: a large, free, open dataset of crawled web pages for bulk research
• Memento: a protocol/aggregator for searching many web archives at once
• Historical imagery: past satellite or street-level images used for geolocation and change analysis

Key Takeaways

• “Deleted” rarely means gone: archives keep old pages, docs, and endpoints
• Reading an archive is passive; browsing the live site is not
• The Wayback Machine’s CDX API lists every capture of a domain, surfacing forgotten pages
• Check multiple archives: Internet Archive and archive.today often hold different snapshots
• Google’s live cache is gone (2024); use web archives instead
• Historical satellite and Street View imagery are key geolocation tools
• Archives are evidence and can be incomplete or manipulated: record timestamps and corroborate

References

• Wayback CDX Server API (Internet Archive, GitHub)
• Internet Archive (Wikipedia) — 2024 breach and archive milestones
• Archive.today (Wikipedia)
• Common Crawl Index Server
• Find street-level imagery from other dates (Google Maps Help)
• Google Removes Cache: Search Operator Documentation (Search Engine Journal)