Hallucination Guard

Verify AI outputs against source data. Flag unsupported claims. Score factual confidence.

Quick Start

# Verify a text against source documents
python3 scripts/hallucination-guard.py verify --text "claim to check" --sources file1.md file2.md

# Verify from a file
python3 scripts/hallucination-guard.py verify --file output.md --sources research.md data.csv

# Verify with web cross-reference
python3 scripts/hallucination-guard.py verify --text "Tesla has 500 Supercharger stations" --web

# Score factual confidence of a document
python3 scripts/hallucination-guard.py score --file report.md

# Extract and verify all claims in a document
python3 scripts/hallucination-guard.py extract --file article.md

# Generate citation report
python3 scripts/hallucination-guard.py cite --file output.md --sources research.md

# Batch verify multiple outputs
python3 scripts/hallucination-guard.py batch --dir outputs/ --sources data/

How It Works

Claim Extraction

The guard extracts individual claims from text:

• Numerical assertions ("revenue grew 45%")
• Named entity claims ("founded in 2015")
• Causal claims ("X causes Y")
• Comparative claims ("faster than competitor Z")
• Temporal claims ("launched in Q3 2025")

Verification Levels

| Level | Method | Confidence |

|-------|--------|------------|

| L1 | Source document match | High (direct citation) |

| L2 | Semantic similarity to source | Medium (paraphrased) |

| L3 | Web cross-reference | Medium (external corroboration) |

| L4 | Internal consistency check | Low (no external support) |

Trust Score

Each claim gets a 0-100 trust score:

• 90-100: Directly supported by source with citation
• 70-89: Semantically supported (paraphrased from source)
• 50-69: Partially supported (some elements confirmed)
• 30-49: Weakly supported (tangentially related)
• 0-29: Unsupported (potential hallucination) ⚠️

Document-level score = weighted average of all claim scores.

Output Format

📊 Hallucination Guard Report
═══════════════════════════════════════
Document: output.md
Claims found: 12
Trust score: 78/100 🟡

✅ VERIFIED (8/12):
  • "Revenue grew 45% YoY" → source: earnings.md L42
  • "Founded in San Francisco in 2019" → source: about.md L3

⚠️ UNVERIFIED (3/12):
  • "Market share exceeds 60%" → no source match (score: 25)
  • "Patent filed in 2024" → no source match (score: 15)

❌ CONTRADICTED (1/12):
  • "Team size is 500+" → source says "200 employees" (earnings.md L88)

Integration

Add to agent workflow for automatic verification:

### Output Verification
Before delivering any research, analysis, or factual content:
1. Run hallucination-guard verify against source documents
2. If trust score < 70, flag for human review
3. Remove or caveat any claim scoring < 30