Step 1: Create a curated Question/Answer test set (ground truth)
            ↓
Step 2: Feed the questions to your RAG system → get RAG responses
            ↓
Step 3: Compare RAG responses with ground truth answers
            ↓
Step 4: Calculate evaluation metrics
            ↓
Step 5: Analyze results and identify areas for improvement

test_set = [
  { "question": "When was the Eiffel Tower built?",
    "ground_truth": "The Eiffel Tower was built in 1889 for the Paris Exposition",
    "rag_response": "The Eiffel Tower was constructed in 1889 for the Paris World Fair",
    "context": "The Eiffel Tower was built in 1889 by Gustave Eiffel..." },
  { "question": "Who invented the telephone?",
    "ground_truth": "The telephone was invented in 1876 by Alexander Graham Bell",
    "rag_response": "Alexander Graham Bell invented the telephone in 1876 and his
     assistant Margaret Thomson conducted crucial experiments",
    "context": "Alexander Graham Bell patented the telephone in 1876." },
  # ... more examples
]

def exact_match(reference, candidate):
    """Binary: 1 if exact match after normalization, 0 otherwise."""
    return 1.0 if reference.lower().strip() == candidate.lower().strip() else 0.0

def token_f1(reference, candidate):
    """Compute precision, recall, and F1 at the token (set) level."""
    ref_set  = set(reference.lower().split())
    cand_set = set(candidate.lower().split())
    common   = ref_set & cand_set

    precision = len(common) / len(cand_set) if cand_set else 0.0
    recall    = len(common) / len(ref_set)  if ref_set  else 0.0
    f1 = 2 * (precision * recall) / (precision + recall) \
         if (precision + recall) > 0 else 0.0
    return f1, precision, recall

Reference: "the cat is on the mat"
Candidate: "the cat on the mat"  (missing "is")

1-gram precision: 5/5 = 1.0000
2-gram precision: 3/4 = 0.7500
3-gram precision: 1/3 = 0.3333
4-gram precision: 0/2 = 0.0000

BLEU-4 = 0.0000

Reference: "Alice gave a book to Bob"
Candidate: "Bob gave a book to Alice"

Meaning is COMPLETELY DIFFERENT!

ROUGE-1 F1 = 1.0000   (same words → perfect match!)
ROUGE-L F1 = 0.8333   (LCS = "gave a book to" → lower score)

Example 1 — Eiffel Tower (good paraphrase):
  EM=0.0  F1=0.73  BLEU=0.23  ROUGE-1=0.73  ROUGE-L=0.58

Example 2 — Telephone (hallucinated detail: "Margaret Thomson"):
  EM=0.0  F1=0.64  BLEU=0.14  ROUGE-1=0.73  ROUGE-L=0.64

Example 3 — Ottawa (hallucinated founding claim):
  EM=0.0  F1=0.33  BLEU=0.04  ROUGE-1=0.44  ROUGE-L=0.38

Example 4 — Washington ("Emperor" instead of "President"):
  EM=0.0  F1=0.93  BLEU=0.74  ROUGE-1=0.93  ROUGE-L=0.93

Reference: "...first President of the United States..."
Generated: "...first Emperor of the United States..."

ROUGE-1 F1 = 0.93   Token F1 = 0.93   BLEU = 0.74

Reference: "Photosynthesis is the process by which plants
            convert sunlight into chemical energy"
Generated: "Sunlight into chemical energy plants convert
            process is photosynthesis"

ROUGE-1 F1 = 1.0   (same words, completely unreadable!)

Reference: "Ottawa is the capital of Canada. It is located in Ontario."
Generated: "Ottawa is the capital of Canada. It was founded in 1826
            by French explorer Jean-Baptiste Rousseau."

ROUGE-1 F1 = 0.44   (decent — partially overlapping!)

CAN Measure:                  CANNOT Measure:
✓ Surface similarity           ✗ Question relevancy
✓ N-gram overlap               ✗ Factual correctness
✓ Word order (ROUGE-L)         ✗ Fluency / Grammar
                               ✗ Hallucinations

Use them for:                  Don't rely on them for:
✓ Reproducible baseline        ✗ Judging answer quality alone
✓ Automated screening          ✗ Detecting hallucinations
✓ Quick comparison             ✗ Evaluating relevancy

relevancy_prompt = """
You are a QA evaluator.

QUESTION: {question}
ANSWER: {answer}

Score relevancy (0-5):
5 = Perfectly answers the question
4 = Addresses question with minor gaps
3 = Addresses question with some gaps
2 = Partially addresses the question
1 = Barely relevant
0 = Off-topic

RESPOND: SCORE: [0-5] REASON: [one sentence]
"""

comprehensive_prompt = """
You are an expert QA evaluator.

QUESTION: {question}
REFERENCE ANSWER: {reference}
SOURCE DOCUMENTS: {sources}
GENERATED ANSWER: {answer}

Evaluate on 4 dimensions (0-5 each):
1. RELEVANCY: Does it address the question?
2. ACCURACY: Are facts correct?
3. FLUENCY: Is it well-written and clear?
4. HALLUCINATIONS: Any made-up facts? (5=none, 0=mostly fabricated)

Respond in JSON:
{
  "relevancy": <0-5>, "accuracy": <0-5>,
  "fluency": <0-5>, "hallucinations": <0-5>,
  "explanation": "<brief explanation>"
}
"""

hallucination_prompt = """
You are checking for hallucinations (made-up facts).

REFERENCE: {reference}
ANSWER: {answer}

For each claim in the answer, mark as:
  SUPPORTED    — In the reference
  UNCLEAR      — Not explicitly stated but reasonable
  UNSUPPORTED  — Not in reference, not contradicted
  HALLUCINATION — Made up, contradicted, or incorrect

Respond in JSON:
{
  "hallucination_score": <0-5 where 5=none>,
  "claims": [
    {"claim": "...", "status": "...", "evidence": "..."}
  ]
}
"""

< 10,000 evaluations     → General-purpose LLM judges
10,000 – 100,000 evals   → Either approach works
> 100,000 evaluations    → Specialized / fine-tuned models
Real-time scoring needed → Only specialized models (10-100ms)

        Question
            │
            ▼
     ┌────────────-──┐
     │   Retriever   │ ← Context Precision (are retrieved docs relevant?)
     │               │ ← Context Recall (are all needed facts retrieved?)
     └──────┬────────┘
            │
       Retrieved Docs
            │
            ▼
     ┌───────────-───┐
     │  Generator    │ ← Faithfulness (is answer grounded in context?)
     │   (LLM)       │ ← Answer Relevancy (does it answer the question?)
     └──────┬────────┘
            │
        Generated Answer

Context: "Python was created in 1989 by Guido van Rossum"
Answer:  "Python was created in 1989 by Guido van Rossum at MIT"
Score:   0.67 — "at MIT" is NOT in the context (hallucination)

Original Q:    "When was Python created?"
Answer:        "Python was created in 1989"
Generated Qs:  "What year was Python released?" ✓
               "When did Python development start?" ✓
Score: 1.0 — all generated questions align with original

from ragas import evaluate
from ragas.metrics import (
    faithfulness, answer_relevancy,
    context_precision, context_recall
)

test_data = {
    "question": ["When was Python created?"],
    "answer": ["Python was created in 1989"],
    "contexts": [["Python was created in 1989 by Guido van Rossum"]],
    "ground_truth": ["1989"]
}

results = evaluate(
    test_data,
    metrics=[faithfulness, answer_relevancy,
             context_precision, context_recall]
)
# {'faithfulness': 0.95, 'answer_relevancy': 0.92,
#  'context_precision': 1.0, 'context_recall': 1.0}

┌────────────────────────────────────────────────────┐
│ STAGE 1: SYNTHETIC DATA GENERATION (One-Time)      │
│  Your Documents → LLM generates Q&A pairs          │
│  Output: 1000-5000 labeled examples                │
│  Cost: ~$10-100 (one-time)                         │
└────────────────────┬───────────────────────────────┘
                     ↓
┌────────────────────────────────────────────────────┐
│ STAGE 2: TRAIN EVALUATOR MODELS (One-Time)         │
│  Fine-tune small models (Mistral-7B, DistilBERT)   │
│  Train 3 specialized evaluators:                   │
│    • Retrieval Evaluator                           │
│    • Answer Relevance Evaluator                    │
│    • Factuality Evaluator                          │
│  Cost: ~$500-1000 (compute)                        │
└────────────────────┬───────────────────────────────┘
                     ↓
┌────────────────────────────────────────────────────┐
│ STAGE 3: EVALUATE (Fast & Cheap — Repeated!)       │
│  Speed: 10-100ms per evaluation                    │
│  Cost: ~$0 per evaluation                          │
│  Can run offline                                   │
└────────────────────────────────────────────────────┘

Document: "Python was created in 1989 by Guido van Rossum"

Generated Questions:
  Q1: "When was Python created?"
  Q2: "Who created Python?"
  Q3: "What language did Guido van Rossum create?"

Training Examples:
  ✓ Positive: (Q1, Python_history_doc, "1989")   → RELEVANT
  ✗ Negative: (Q1, Java_programming_doc, "1989") → IRRELEVANT

from transformers import AutoModelForSequenceClassification, Trainer

model = AutoModelForSequenceClassification.from_pretrained(
    "mistralai/Mistral-7B",
    num_labels=2  # Binary: relevant or not
)
trainer = Trainer(
    model=model,
    train_dataset=synthetic_data,
    args=TrainingArguments(learning_rate=2e-5, num_train_epochs=3)
)
trainer.train()

Method              | Cost (10k evals) | Time (10k evals) | Accuracy
────────────────────┼──────────────────┼──────────────────┼─────────
Token Metrics       | $0               | < 1 sec          | 50-60%
RAGAS               | $10-100          | 1-10 hrs         | 70-75%
LLM Judge           | $10-100          | 10-50 hrs        | 75-85%
ARES (post-setup)   | $0.10            | 2-5 min          | 75-85%
Human evaluation    | $5000+           | 50+ hrs          | 90-95%

START: "I need to evaluate my RAG system"
│
├─ Is this a prototype/POC?
│  YES → Token Metrics + RAGAS
│  Cost: $0, Time: hours
│
├─ < 10,000 evaluations?
│  YES → LLM Judges + RAGAS
│  Cost: $10-100, Time: days
│
├─ 10,000-100,000 evaluations?
│  YES → LLM Judges + human sampling
│  Cost: $100-500, Time: days
│
└─ > 100,000 evaluations?
   YES → ARES + LLM Judges for edge cases
   Cost: $2000-5000 setup, then ~$0

Tier 1: Fast Screening (ARES or RAGAS)
  → Score ALL answers
  → Cost: ~$0-10, Time: minutes
          │
    ┌─────┴────────┐
    ▼              ▼
  Flagged       Good Items
  (Low scores)   → Accept
    │
    ├─ Tier 2: LLM Judge (20% of flagged)
    │  → Deeper analysis, Cost: $100-500
    │
    └─ Tier 3: Human Expert (5% of flagged)
       → Final validation, Cost: $500+

Result: 95-99% issue detection at 20% of LLM-only cost

Example: Simulated score distributions (n=100)

Human scores:   well-distributed across 0-5
LLM Judge:      well-distributed (good discrimination)
ROUGE:          clustered around 2-3 (poor discrimination)
RAGAS:          moderate spread (acceptable)

Typical Cross-Metric Correlation Matrix:

              Human    ROUGE    LLM Judge   RAGAS
Human          1.00     0.52      0.85      0.72
ROUGE          0.52     1.00      0.48      0.55
LLM Judge      0.85     0.48      1.00      0.70
RAGAS          0.72     0.55      0.70      1.00

Step 1: Get human labels for ~100 examples
Step 2: Run all automated metrics on those 100 examples
Step 3: Compute Spearman correlation for each metric
Step 4: Decision:
        • ρ > 0.7  → Metric is reliable, use it
        • ρ < 0.7  → Revise prompt/approach, retry
Step 5: In production, periodically re-validate
        • Check for score drift over time
        • Flag outliers for manual review

2022: Token metrics (ROUGE/BLEU)    → Fast but inaccurate
2023: LLM judges (Claude/GPT-4)     → Accurate but expensive
2024: Frameworks (RAGAS/ARES)        → Structured, scalable
2025: Hybrid systems                 → Best of all worlds