Source Reliability and Evidence Grading

Summary

Source reliability describes how dependable a source is for a specific claim. Information credibility describes how well the available information supports that claim. They are related but not identical: a reliable source can publish incomplete information, and an unfamiliar source can offer verifiable direct evidence. Evidence grading should preserve what was observed, what was reported, what was inferred, and what remains unknown.

Why it matters

Threat intelligence, vulnerability records, advisories, telemetry, research, and reporting can conflict or describe different scopes. Popularity is not reliability, and recency is not accuracy. A defensible record identifies the original source, preserves provenance, distinguishes direct evidence from analysis, and makes the next validation step visible.

When to use this guidance

Use this when comparing vendor advisories, CVE Program records, CISA KEV, NVD, research, proof-of-concept claims, news, social media, internal logs, EDR or SIEM telemetry, asset inventory, and change records. It helps analysts write bounded conclusions before escalating, scheduling remediation, or claiming observed exploitation.

Prerequisites

Define the exact claim first. Preserve the source name, record ID, URL, publication date, modification date, retrieval context, and relevant scope. Do not paste secrets, customer data, credentials, raw incident evidence, or private reports into public tools.

Core concepts

Authority is claim-specific. A vendor may be authoritative for its fixed version; the CVE Program for CVE record state; CISA for KEV inclusion; and reliable local telemetry for observed local activity.
Provenance records where a value came from and how it reached the assessment. An aggregator or news article can be useful context but should not replace an original source.
Direct evidence is an observation or source record close to the claim. Indirect evidence may support an inference but should not be written as observation.
Corroboration is independent support, not repetition of the same unverified assertion. Completeness and freshness remain separate from confidence.
Confidence does not equal severity, exploit likelihood, business impact, local exposure, local compromise, or risk.

Source types and their limitations

Vendor advisories, release notes, and product security response teams can clarify product scope and fixed versions. CVE Program records define CVE record state; NVD can add enrichment; CISA KEV establishes catalog inclusion; national CERTs and government advisories can provide public guidance. Security research and reproducible exploit repositories may provide strong technical evidence. Commercial intelligence, news, forums, social media, and anonymous claims can offer leads but require provenance and corroboration. Internal logs, EDR or SIEM telemetry, incident evidence, asset inventory, and change records can be authoritative for reliable local observation, but may have access, collection, or coverage limits.

Claim-specific authority

Claim	Strongest expected source	Bounded note
CVE record status	CVE Program record	Use the record state and revisions, not an unsourced summary.
Vendor fixed version	Vendor advisory or release note	Confirm product branch, edition, and platform scope.
KEV inclusion	CISA KEV catalog	Inclusion does not prove local compromise.
Public exploit code exists	Reproducible repository or technical research	Claimed code without setup details is limited evidence.
Active exploitation observed globally	Credible incident, government, or corroborated research reporting	Separate reports from verified observation.
Local exploitation or compromise	Reliable internal telemetry and validated investigation evidence	Coverage gaps limit a negative conclusion.
Asset deployment or exposure	Validated inventory plus network, service, identity, or application evidence	Public reporting cannot replace local evidence.

This is guidance, not an exhaustive hierarchy. A vendor is not automatically authoritative for every independent exploitation claim, and a researcher may be closer to a technical claim than an aggregator.

Evidence-grading dimensions

Assess proximity to the claim, authority for that claim, directness, provenance, corroboration, recency, reproducibility, completeness, correction or retraction history, conflict state, and local relevance. Different claims in the same record can receive different grades. A preferred display value must not erase conflicting source values.

Evidence-grade states

Use strong for direct, authoritative, well-provenanced support; supported for credible corroborated information with meaningful limits; limited for incomplete, indirect, or weakly reproducible evidence; unverified when a claim lacks sufficient support; conflicting when material sources disagree; stale when the information may no longer represent the current state; and unknown when evidence has not been established. These labels describe evidence support, not an opaque risk score or severity ranking.

Practical workflow

Define the exact claim being evaluated.
Identify and preserve the original source.
Record source name, record ID, URL, publication date, modification date, and retrieval context.
Mark whether the information is direct observation, a source statement, or analyst inference.
Evaluate authority for the specific claim.
Check missing context, completeness, and scope.
Seek independent corroboration.
Review recency, corrections, retractions, and stale references.
Preserve conflicting values rather than silently overwriting them.
Assign a bounded evidence grade and confidence statement.
State limitations and what additional evidence would change the assessment.
Reassess when source or local evidence changes.

Handling conflicting sources

Preserve each material source value and its provenance. Check whether an apparent conflict is actually caused by different product branches, stale versus updated advisories, publication timing, global versus local evidence, exploit code versus observed exploitation, or vendor severity versus NVD severity. Do not average incompatible severity values, merge incompatible fixed-version claims, or treat source absence as contradiction. Label unresolved conflicts, identify resolving evidence, and update the conclusion when authoritative information changes.

Evidence to collect

Collect the exact claim; original source and type; record ID and URL; publication, modification, and retrieval dates where supported; a bounded quotation or summary; observed versus inferred state; provenance; corroborating and conflicting sources; local telemetry and coverage limitations; freshness; completeness; evidence grade; confidence statement; unresolved questions; and reassessment trigger.

Common mistakes

Do not repeat an aggregator without finding the original source, treat popularity or recency as reliability, treat one source as universally authoritative, hide conflicts, average incompatible evidence, present inference as observation, treat no evidence as evidence of absence, treat absence from KEV as proof no exploitation exists, treat KEV inclusion as proof of local compromise, call a proof-of-concept claim weaponized exploitation, treat scanning as compromise, ignore modification dates or retractions, use a confidence label without its basis, or equate confidence with business risk.

Worked example

A fictional vulnerability affects a fictional product family. A vendor advisory explicitly confirms affected products and one fixed release, so affected-product evidence is strong where that scope is explicit and fixed-version evidence is strong only for the stated branch. A public repository claims proof-of-concept code but lacks reproducible setup details, making exploit-code availability limited rather than confirmed. Several social-media posts claim active exploitation, and a news article repeats them. A research organization reports scanning activity but not confirmed exploitation. The issue is not in a government exploitation catalog at review time; that absence does not prove exploitation has not occurred. No matching local telemetry is identified, but collection coverage is incomplete, so it does not prove no compromise. Global active exploitation remains unverified or conflicting; further validation is required.

Writing a bounded conclusion

Use a concise structure: Claim: state the precise question. Available evidence: identify primary records and direct observations. Evidence grade: name the bounded state and basis. Conflicts: retain material differences. Local relevance: state known asset, exposure, or telemetry context. Confidence: describe support without implying severity or risk. Limitations: name gaps. Next validation step: identify the owner or evidence needed. Reassessment trigger: name the change that requires review.

Reassessment triggers

Reassess after a vendor revision, CVE Program update, KEV change, reproducible technical evidence, retraction, new corroboration, asset inventory update, telemetry coverage improvement, observed local event, or change in product scope. Evidence grades should change when material information changes.

Limitations

Public reporting may be incomplete and source availability can change. Some authoritative sources require authentication; internal telemetry can have coverage gaps; and absence of evidence does not prove absence. Reliability varies by claim, legal or contractual handling restrictions can apply, and this guidance does not replace incident-response, legal, regulatory, or vendor-specific requirements.