Analysis & Cognitive Bias

Learning Objectives

• Separate facts, assumptions, and judgments in their own analysis
• Express likelihood and confidence using standard estimative language instead of vague words
• Grade sources and information using the Admiralty Code (reliability A–F, credibility 1–6)
• Run an Analysis of Competing Hypotheses (ACH) matrix on an OSINT question
• Recognize common cognitive biases in OSINT work and use techniques to challenge their own conclusions

From Information to Judgment

Analysis is the step in the OSINT cycle where collected, verified information becomes an answer to the original question. It is also where most analytic errors happen, because it depends on human reasoning, and human reasoning has predictable blind spots.

A good analytic product clearly separates three kinds of statements:

Type	Definition	Example (fictional)
Fact / information	Something observed or collected, with a source	“Company X’s MX records point to Microsoft 365 (DNS query, 2026-09-10 14:02 UTC).”
Assumption	Something taken as true to fill a gap, not directly confirmed	“We assume employees use the same email format for all staff.”
Judgment	A conclusion drawn from facts and assumptions	“Company X very likely uses Microsoft 365 for employee email and login.”

Why this matters: if an assumption turns out to be wrong, the reader needs to know which judgments depend on it. The US Intelligence Community’s analytic standards (below) require exactly this separation.

Estimative Language and Confidence

Words like “maybe,” “possibly,” and “probably” mean very different things to different readers. One person’s “probably” is 55%; another’s is 90%. Intelligence analysts solve this with standardized terms.

ICD 203 (US Intelligence Community Analytic Standards)

Intelligence Community Directive 203, Analytic Standards, signed January 2, 2015 (with later technical amendments), sets the standards US intelligence analysis is evaluated against. OSINT professionals in the private sector often borrow its approach.

It lists five analytic standards. Analysis should be:
1. Objective
2. Independent of political consideration
3. Timely
4. Based on all available sources of intelligence information
5. Implementing the analytic tradecraft standards, which are:
1. Properly describes quality and credibility of underlying sources, data, and methodologies
2. Properly expresses and explains uncertainties associated with major analytic judgments
3. Properly distinguishes between underlying intelligence information and analysts’ assumptions and judgments
4. Incorporates analysis of alternatives
5. Demonstrates customer relevance and addresses implications
6. Uses clear and logical argumentation
7. Explains change to or consistency of analytic judgments
8. Makes accurate judgments and assessments
9. Incorporates effective visual information where appropriate

Likelihood terms

ICD 203 requires one of these sets of terms when expressing probability:

Probability	01–05%	05–20%	20–45%	45–55%	55–80%	80–95%	95–99%
Set 1	almost no chance	very unlikely	unlikely	roughly even chance	likely	very likely	almost certain(ly)
Set 2	remote	highly improbable	improbable (improbably)	roughly even odds	probable (probably)	highly probable	nearly certain

Notes:
- The scale stops at 1% and 99%. Estimates are never 0% or 100%; if something is certain, it’s a fact, not an estimate.
- ICD 203 strongly encourages analysts not to mix terms from the two rows.

Confidence is a separate thing

• Likelihood is about the event: how probable is it that this is true or will happen?
• Confidence is about the analysis: how solid is the evidence and reasoning behind the judgment?

Confidence depends on the quantity and quality of sources, how well they agree, how many assumptions are involved, and how well the analyst understands the topic. Analysts commonly use three levels:
- High confidence: good-quality information from multiple independent sources; few assumptions
- Moderate confidence: credible information, but not enough corroboration, or some important assumptions
- Low confidence: fragmentary, poorly corroborated, or questionable information; significant assumptions

(These descriptions reflect common usage in US intelligence products; ICD 203 requires explaining the basis for confidence but does not define the three levels word-for-word.)

ICD 203 also says not to combine a confidence level and a likelihood term in the same sentence, because readers mix them up.

Writing judgments: weak vs. strong

Weak	Better
“The dump is probably old.”	“We assess the credential dump very likely comes from a 2019 third-party breach rather than a new compromise of Company X. We have moderate confidence in this judgment because it relies partly on client-provided HR records we could not independently verify.”
“This might be the CEO’s personal account.”	“The Instagram account @example_handle likely belongs to the CEO: it uses the same profile photo as the CEO’s corporate bio and mostly follows Company X accounts. No account email or phone number was available to confirm the link.”
“It’s definitely a fake.”	“The image appeared online at least two years before the event it claims to show (TinEye, earliest match 2024-03-02). The claim that it shows the event is false.”

Grading Sources and Information: The Admiralty Code

The Admiralty Code (also called the NATO System) is a grading system used by NATO members and the Five Eyes intelligence community. Its key idea: rate the source and the information separately. A usually reliable source can pass on false information, and an unreliable source can occasionally be right.

Source reliability (letters)

Grade	Meaning
A	Completely reliable: no doubt about authenticity, trustworthiness, or competency; history of complete reliability
B	Usually reliable: minor doubt; history of valid information most of the time
C	Fairly reliable: some doubt, but has provided valid information in the past
D	Not usually reliable: significant doubt, but has provided valid information in the past
E	Unreliable: lacking authenticity, trustworthiness, and competency; history of invalid information
F	Reliability cannot be judged: no basis for evaluating the source

Information credibility (numbers)

Grade	Meaning
1	Confirmed by other sources: independently confirmed, logical, consistent with other information
2	Probably true: not confirmed, but logical and consistent with other information
3	Possibly true: not confirmed, reasonably logical, agrees with some other information
4	Doubtful: not confirmed, possible but not logical, no other information on the subject
5	Improbable: not confirmed, not logical, contradicted by other information
6	Truth cannot be judged: no basis for evaluating the information

Grades are written together, e.g., B2 = a usually reliable source providing probably true information.

Applying it to OSINT (fictional examples)

Item	Grade	Reasoning
Company X’s own careers page lists “Palo Alto firewall experience required”	B2	The company is a reliable source about its own job needs; it suggests but doesn’t confirm current use
A brand-new anonymous forum account claims to be selling Company X’s database	F3 or F4	No history to judge the source; plausible but unconfirmed claim
crt.sh shows a certificate for vpn.companyx.com, and Shodan shows a VPN login page on the matching IP	A1 / B1	Technical records from established services, confirmed by two independent sources
A people-search site lists a home address for an employee	D3	These sites frequently mix up records; treat as a lead only

Tip: many OSINT teams adapt the Admiralty Code rather than use it word-for-word. What matters most is being consistent and explaining each grade.

Analysis of Competing Hypotheses (ACH)

ACH is a structured method described by Richards J. Heuer Jr., a longtime CIA analyst, in Psychology of Intelligence Analysis (published by the CIA’s Center for the Study of Intelligence in 1999). It is designed to counter confirmation bias by forcing the analyst to consider several explanations at once and to focus on evidence that disproves hypotheses rather than evidence that seems to support a favorite one.

Heuer’s eight steps

1. Identify the possible hypotheses. Brainstorm, ideally with people who have different perspectives.
2. List the significant evidence and arguments for and against each hypothesis, including assumptions and the absence of evidence you’d expect to see.
3. Build a matrix with hypotheses across the top and evidence down the side. For each cell, ask whether the evidence is consistent (C), inconsistent (I), or not applicable (N/A) for that hypothesis. Look for diagnostic evidence: evidence that fits some hypotheses but not others.
4. Refine the matrix. Reword or merge hypotheses as needed; drop evidence that is consistent with everything, because it doesn’t help you choose.
5. Draw tentative conclusions by trying to disprove hypotheses. The hypothesis with the fewest inconsistencies is usually the strongest, not the one with the most “consistent” marks.
6. Analyze sensitivity. Which few pieces of evidence drive the conclusion? What if they’re wrong, deceptive, or misinterpreted?
7. Report conclusions, discussing the relative likelihood of all hypotheses, not just the winner.
8. Identify milestones for future observation: signs that would show events are going differently than expected.

Worked example (fictional)

Question: A paste site post contains 1,200 email/password pairs ending in @companyx.com, labeled “fresh Company X hack.” What best explains it?

Hypotheses:
- H1: A new breach of Company X’s systems
- H2: Old data from a previous third-party breach, repackaged (a “combolist”)
- H3: Infostealer malware on employees’ devices
- H4: Fabricated data (fake dump)

Evidence (all fictional; E2, E3, E6, and E7 come from Company X’s own security team as part of an authorized engagement):
- E1: 1,200 email/password pairs using Company X’s email format
- E2: About 90% of the emails belong to accounts deactivated before 2021
- E3: Many password entries match records from a 2019 breach of a third-party site where employees used their work email
- E4: The post has no URLs, browser cookies, or computer information (infostealer logs normally include these)
- E5: The poster claims it’s a “fresh hack”
- E6: Company X’s logs show no unusual access to its user databases in the past six months
- E7: Company X’s IT team confirmed 3 entries still match current employees’ passwords

Evidence	H1 New breach	H2 Old combolist	H3 Infostealer	H4 Fabricated
E1 Emails match company format	C	C	C	C
E2 90% deactivated before 2021	I	C	I	C
E3 Matches 2019 third-party breach	I	C	I	C
E4 No URLs/cookies/system info	N/A	C	I	C
E5 Poster says “fresh hack”	C	C	C	C
E6 No unusual database access	I	C	N/A	C
E7 3 passwords still valid	C	C	C	I
Inconsistencies	3	0	3	1

Walking through it:
- E1 and E5 are consistent with everything, so they have no diagnostic value (step 4). E5 also comes from an unreliable source: sellers routinely call old data “fresh.”
- H2 has no inconsistencies, so it is the strongest hypothesis.
- H4 is weakened by E7: fabricated data wouldn’t contain real, current passwords.
- E7 is explained under H2 by password reuse: three current employees still use the same password they used on the breached third-party site years ago.

Sensitivity (step 6): The conclusion depends heavily on E2 and E3, which come from the client. If the HR records are wrong, H1 becomes more plausible.

Report (step 7): “We assess the posted data very likely comes from a 2019 third-party breach that has been repackaged, rather than a new compromise of Company X. A new breach and infostealer infections are unlikely given the account ages and missing infostealer data. We have moderate confidence in this assessment because two key pieces of evidence rely on client records. Regardless of the source, three current employees are exposed through password reuse and should reset their passwords immediately.”

Milestones (step 8): New posts containing current employee accounts; posts that include URLs or cookies (suggesting infostealers); unusual login attempts against Company X.

Common Cognitive Biases in OSINT

Cognitive biases are predictable shortcuts in human thinking. Knowing their names doesn’t make you immune, but it helps you spot them in your own work and build in checks.

Bias	What it looks like in OSINT	Countermeasure
Confirmation bias: favoring evidence that supports what you already believe	You decide early that a username belongs to your target, then only search for evidence that it does	ACH; actively search for disconfirming evidence; ask “what would prove me wrong?”
Anchoring: relying too heavily on the first piece of information	The first address you find becomes “the” address, even after newer records appear	Revisit early findings; record dates for everything
Availability bias: judging likelihood by what comes to mind easily	Assuming a leak is from ransomware because ransomware is in the news	Consider base rates; list several explanations
Satisficing: accepting the first answer that seems good enough	Stopping at the first matching LinkedIn profile	Keep checking until identifiers are confirmed
Mirror imaging: assuming others think and act like you	Assuming a threat actor would never reuse a username because you wouldn’t	Consider the other person’s context, skills, and incentives
Automation bias: trusting tool output without checking	A username tool says “found” on 40 sites, and all 40 go into the report	Manually verify tool results; understand how the tool works
Groupthink: a team converging on one view to avoid conflict	Nobody questions the lead analyst’s theory	Assign a devil’s advocate; collect individual judgments before discussing
Overconfidence from more data: more information increasing confidence faster than accuracy	Collecting hundreds of weak data points and treating the volume as proof	Weigh quality over quantity; grade sources
Vividness: giving too much weight to dramatic or emotional evidence	A striking photo outweighs dull but reliable records	Grade all evidence with the same system

Heuer’s book makes the point that simply knowing about biases does little to prevent them. Structured techniques are what help.

Red-Teaming Your Own Conclusions

Before a finding goes into a report, challenge it:

• Key assumptions check: List every assumption your judgment depends on. For each, ask: How confident am I that this is true? What would happen to my conclusion if it were false?
• Devil’s advocate: Have someone (or yourself, deliberately) argue the strongest case against your conclusion.
• Premortem: Imagine it’s six months later and your conclusion turned out to be wrong. Write down the most likely reasons why. Then check those reasons now.
• Indicators: Write down what you’d expect to see if your conclusion is right, and what would show it’s wrong. Go look for the second list.
• Reproduce it: Can a teammate follow your investigation log (see 7.3-documentation-and-evidence.md) and reach the same finding? If not, the finding isn’t ready.
• Consider deception: Could someone have planted this information to mislead investigators? Threat actors create fake personas and false trails on purpose.
• Check for harm: If this judgment is wrong, who gets hurt? The higher the potential harm, the higher the bar for confidence.

Key Terms

• Judgment: a conclusion drawn from information and assumptions
• Assumption: something treated as true to fill a gap, without direct confirmation
• Estimative language: standardized words that express likelihood (e.g., “likely” = 55–80% under ICD 203)
• Confidence level: how solid the evidence and reasoning behind a judgment are (high, moderate, low)
• ICD 203: US Intelligence Community Directive setting analytic standards
• Admiralty Code / NATO System: grades source reliability (A–F) and information credibility (1–6) separately
• ACH (Analysis of Competing Hypotheses): a matrix method for testing several explanations against the evidence
• Diagnostic evidence: evidence that fits some hypotheses but not others
• Combolist: a list of username/password pairs compiled from previous breaches
• Cognitive bias: a predictable error in human judgment
• Premortem: imagining a conclusion has failed and working out why, before it’s final

Key Takeaways

• Separate facts, assumptions, and judgments; label each
• Use standard likelihood words (“likely” = 55–80%), not vague ones
• Likelihood is about the event; confidence is about your evidence
• Grade the source and the information separately (Admiralty Code: B2, F3, etc.)
• ACH: the best hypothesis has the fewest inconsistencies, not the most support
• Evidence that fits every hypothesis doesn’t help you decide
• Confirmation bias is the #1 OSINT trap; look for what would prove you wrong
• Before reporting: check assumptions, argue the opposite, and consider who’s harmed if you’re wrong

References

• ICD 203: Analytic Standards — Office of the Director of National Intelligence (PDF)
• Intelligence Community Directive 203 — HLS 476 Homeland Security Intelligence (Penn State)
• Admiralty code — Wikipedia
• The Admiralty Code / NATO 6x6 System — Jessica Stutzman
• Psychology of Intelligence Analysis — Richards J. Heuer Jr. (Internet Archive)
• Richards Heuer — Wikipedia
• Analysis of Competing Hypotheses (PDF) — Futuribles