LLM Evaluation: Domain Explainer#

Universal Analogies#

Quality Control for AI Outputs#

Analogy: Factory Quality Inspector vs AI Quality Inspector

Traditional software testing is like inspecting widgets on an assembly line:

• Widget either fits the spec (pass) or doesn’t (fail)
• Same input → same output, every time
• Clear pass/fail criteria

LLM evaluation is like judging creative writing:

• Many “correct” answers exist for the same prompt
• Same prompt → different outputs each time
• Quality is subjective and context-dependent

Example:

Prompt: "Summarize this article in 3 sentences"

Valid Summary A: "The study found X. Researchers discovered Y. This suggests Z."
Valid Summary B: "Research shows X is correlated with Y. The implications are Z."

Both are correct, but:
- Different phrasing
- Different emphasis
- Different completeness

An LLM evaluator must understand semantic equivalence (these mean the same thing despite different words) rather than just exact matching.

The Restaurant Review Problem#

Analogy: How do you know if a restaurant is good?

Option 1: Count stars (like BLEU/ROUGE scores)

• Fast, cheap, scalable
• But: Doesn’t explain WHY it got 3 stars
• Misses context: “3 stars for fine dining” ≠ “3 stars for pizza”

Option 2: Professional food critic (like LLM-as-Judge)

• Understands nuance, context, and subjective quality
• Provides detailed explanations
• But: Expensive, has personal biases

Option 3: Health inspection (like Programmatic checks)

• Binary checks: Does food have correct temperature? Is kitchen clean?
• Catches specific, predictable problems
• But: Doesn’t evaluate taste, creativity, or overall quality

Best practice: Use all three. Health inspection for safety (programmatic), star rating for quick filtering (metrics), and food critic for nuanced evaluation (LLM-as-judge).

The RAG Triad: Research Paper Analogy#

Retrieval-Augmented Generation (RAG) is like writing a research paper:

Your Question → Library Search → Retrieved Books → Your Essay
   (Query)      (Vector Search)    (Context)      (Answer)

Three quality checks:

1. Context Relevance = “Did you check out the right books?”

• Question: “How does photosynthesis work?”
• Good retrieval: Botany textbooks, plant biology papers
• Bad retrieval: Economics journals, cooking recipes

2. Faithfulness/Groundedness = “Did you cite your sources correctly?”

• Good: “According to Smith (2020), photosynthesis converts light into energy”
• Bad: “Photosynthesis was invented in 1872” (not in any source)
• Problem: Hallucination = making up citations or facts

3. Answer Relevance = “Did you actually answer the question?”

• Question: “How does photosynthesis work?”
• Good: Explains the process step-by-step
• Bad: “Photosynthesis is important” (true but doesn’t answer HOW)

Debugging with the Triad:

• Low context relevance → Fix your search/embeddings
• Low faithfulness → Model is hallucinating, prompt engineering needed
• Low answer relevance → Prompt doesn’t guide model well

What Problem Does LLM Evaluation Solve?#

The Scale Problem#

Scenario: You’re building a customer support chatbot answering 10,000 questions/day.

Without evaluation:

• How do you know if answers are accurate?
• Manual review = 10,000 answers × 2 min/review = 333 hours/day (impossible)
• Launch blind, hope for the best, fix angry customer complaints

With evaluation:

• Automated metrics score every answer
• Flag low-scoring answers for human review
• Catch quality regressions before customers do
• Measure improvement over time

Real example:

Answer A: "Your order ships in 3-5 business days"
Answer B: "I don't have access to shipping information"
Answer C: "Your package left our facility yesterday and should arrive Tuesday"

Evaluation metrics:
- Relevance: Does it answer the question? (A=90%, B=40%, C=95%)
- Faithfulness: Is it grounded in retrieved data? (A=80%, B=90%, C=95%)
- Completeness: Did it address all aspects? (A=70%, B=30%, C=90%)

Verdict: C is best (most complete, most accurate, most helpful)

The Drift Problem#

Analogy: Software bit rot, but for AI

Traditional software: Code doesn’t change → same input = same output forever

LLMs drift over time:

• Model updates (GPT-4 → GPT-4.5)
• Prompt modifications
• Context window changes
• Training data shifts

Without continuous evaluation:

• You update your prompt to fix one edge case
• Accidentally break 15 other cases
• No one notices until production breaks

With continuous evaluation:

• Test suite runs on every prompt change
• Regression detected immediately: “New prompt scores 15% lower on accuracy”
• Roll back or iterate before deploying

Key Concepts#

LLM-as-Judge: Using AI to Grade AI#

How it works:

Evaluator LLM receives:
- User question: "What is photosynthesis?"
- Model answer: "Photosynthesis is when plants make energy from sunlight"
- Rubric: "Score 1-5 for accuracy, completeness, clarity"

Evaluator outputs:
- Accuracy: 4/5 (correct but simplified)
- Completeness: 3/5 (missing chlorophyll, chemical equation)
- Clarity: 5/5 (very clear for a beginner)

Advantages:

• Understands paraphrases: “automobile” = “car” = “vehicle”
• Scales to thousands of evaluations
• Can judge subjective qualities (tone, helpfulness)

Limitations:

• Costs money (API calls)
• Judge has biases (prefers certain writing styles)
• Can be fooled: Outputs that “sound good” but are factually wrong

Example of judge bias:

Answer A: "Photosynthesis converts CO₂ and H₂O into glucose using light"
          (Accurate but dry)

Answer B: "Plants are nature's solar panels, transforming sunlight into
           delicious energy that fuels all life on Earth!"
          (Engaging but imprecise)

Some judges prefer A (precision), others prefer B (engagement)

Self-Explaining Metrics#

Problem with black-box scores:

Faithfulness: 0.4

Why 0.4? Which claims weren’t grounded? Unclear.

Self-explaining metrics:

Faithfulness: 0.4

Reason: The response claims "revenue increased 50%" but the retrieved
context only states "revenue showed growth." The specific percentage
is not supported by the documents. This is a hallucination.

Recommendation: Remove unsupported statistics or retrieve quarterly
reports with exact figures.

Analogy: Teacher grading essays

Bad feedback: “C+ See me after class” Good feedback: “C+ Your thesis is unclear (see paragraph 1), and you didn’t cite sources for your main claim (paragraph 3). Strengthen these for a B.”

Self-explaining metrics are like good teachers—they show you exactly what’s wrong and how to fix it.

Evaluation vs Observability#

Analogy: Car dashboard

Observability = Speed, fuel, engine temperature

• “Is the car running?”
• Real-time monitoring
• Alerts when something breaks

Evaluation = Crash test ratings, fuel efficiency tests

• “Is the car safe and efficient?”
• Quality benchmarks
• Regression testing before new model releases

You need both:

• Observability catches outages: “API is down, 500 errors”
• Evaluation catches quality degradation: “Accuracy dropped 20% after prompt change”

Common Patterns#

The Test Suite Pattern#

Analogy: Regression testing in traditional software

Create a “golden dataset” of curated test cases:

Test Case 1:
  Input: "What is the capital of France?"
  Expected: "Paris"

Test Case 2:
  Input: "Who was the first president of the United States?"
  Expected: "George Washington"

Test Case 3 (Edge case):
  Input: "What is the capital of the moon?"
  Expected: "The moon has no capital" or "No government on the moon"

Run on every change:

Before prompt change: 95% accuracy
After prompt change: 92% accuracy

Investigate: Which 3% broke? Why?

Coverage types:

• Happy path: Normal questions that should always work
• Edge cases: Unusual questions, ambiguous phrasing
• Adversarial: Trick questions, jailbreak attempts
• Domain-specific: Industry jargon, technical terms

The A/B Testing Pattern#

Analogy: Website A/B testing, but for prompts

Scenario: Which prompt is better?

Prompt A: "Answer this question concisely: {question}"
Prompt B: "You are a helpful assistant. Provide a detailed answer to: {question}"

Test on 100 questions:
- Prompt A: Conciseness 95%, Completeness 70%
- Prompt B: Conciseness 60%, Completeness 90%

Decision: Use A for quick lookups, B for research questions

Metrics to compare:

• Accuracy
• Relevance
• Latency (response time)
• Cost (tokens used)
• User satisfaction (if you have feedback)

The Continuous Monitoring Pattern#

Analogy: Canary deployment + health checks

In production:

• Sample 1% of traffic for evaluation
• Run evaluations asynchronously (don’t slow down responses)
• Alert if scores drop below threshold

Example:

Day 1-10: Average faithfulness = 0.85
Day 11: Average faithfulness = 0.65

Alert: "Faithfulness dropped 20%. Recent changes:
- Model updated from GPT-4 to GPT-4-turbo
- New embedding model deployed
Investigation needed."

Common Misconceptions#

“Evaluation is just testing”#

Traditional testing: Input → Code → Output (deterministic)

• Test: add(2, 3) == 5 (exact match)

LLM evaluation: Input → LLM → Output (probabilistic)

• Test: summarize(article) ≈ “good summary” (fuzzy match)
• Multiple correct answers
• Subjective quality judgments

“Higher scores always mean better quality”#

Counterexample: Optimizing for the wrong metric

Prompt optimized for BLEU score:
  "The cat sat on the mat. The mat was sat on by the cat."
  (Repetitive, awkward, but high n-gram overlap)

Prompt optimized for relevance:
  "The cat rested on the mat."
  (Natural, concise, lower BLEU but better quality)

Goodhart’s Law: “When a measure becomes a target, it ceases to be a good measure.”

Use multiple metrics to avoid gaming single metrics.

“One tool does everything”#

Reality: Most teams use 2-3 tools

• DeepEval: 60+ metrics, self-explaining, general-purpose
• Ragas: RAG-specific (retrieval quality)
• PromptFoo: Red teaming, security testing
• LangSmith: Observability + basic evaluation

Analogy: Software development uses multiple tools:

• Jest (unit tests)
• Cypress (E2E tests)
• Datadog (monitoring)
• Sentry (error tracking)

LLM evaluation is the same—different tools for different needs.

When to Invest in Evaluation#

Minimal (just starting)#

• Manual spot checks (review 10-20 outputs)
• Basic programmatic checks (JSON validity, length limits)
• ~20-50 test cases

Moderate (production app)#

• Automated test suite in CI/CD
• LLM-as-judge for key metrics
• 100-500 test cases
• Basic dashboard

Comprehensive (critical application)#

• Continuous production evaluation (sample traffic)
• Multiple metric coverage (accuracy, safety, quality)
• 1,000+ test cases
• Regression alerts
• Human-in-the-loop for edge cases

Budget guidance:

Cost Considerations#

Evaluation has direct costs:

Example: Customer support chatbot, 10,000 questions/day

Option 1: Human review
  10,000 × 2 min/review = 333 hours/day
  333 hours × $30/hour = $10,000/day
  Annual cost: $3.6M

Option 2: LLM-as-Judge (GPT-4)
  10,000 × $0.03/eval = $300/day
  Annual cost: $110K (97% savings)

Option 3: LLM-as-Judge (GPT-3.5)
  10,000 × $0.002/eval = $20/day
  Annual cost: $7.3K (99.8% savings)

Option 4: Programmatic + sampling
  Programmatic checks: Free (CPU only)
  Human review of flagged 1%: 100 × 2 min × $30/hour = $100/day
  Annual cost: $36K (99% savings)

Cost optimization:

• Use cheap models (GPT-3.5) for initial filtering
• Use expensive models (GPT-4) for edge cases
• Cache evaluation results for identical outputs
• Sample production traffic instead of evaluating everything

Glossary#

Chain-of-Thought (CoT): Prompting technique where the model shows reasoning steps before answering. Improves both generation and evaluation accuracy.

Faithfulness/Groundedness: Is the answer supported by the provided context? (Not hallucinated)

Hallucination: When an LLM generates information not present in its context or training data.

Golden Dataset: Curated test cases with human-verified expected outputs or scores.

RAG (Retrieval-Augmented Generation): Pattern where an LLM is given retrieved documents as context before generating an answer.

RAG Triad: Three core metrics for RAG evaluation: context relevance, faithfulness, answer relevance.

Rubric: Scoring criteria provided to an LLM judge explaining how to rate responses.

Self-Explaining Metric: Evaluation metric that provides both a score AND an explanation of why that score was given.

Synthetic Data: Machine-generated test cases, often used to expand test coverage cheaply.

Related Research:

• 1.111 (State Management)
• 1.113 (UI Components)
• 3.205 (Pronunciation Assessment - similar evaluation challenges)

Last Updated: 2026-02-02

LLM Evaluation: A Domain Explainer#

What is LLM Evaluation?#

LLM evaluation is the systematic measurement of language model outputs against quality criteria. Unlike traditional software testing with deterministic pass/fail outcomes, LLM evaluation deals with probabilistic outputs where “correct” is often subjective or context-dependent.

The core challenge: Given the same input, an LLM might produce different outputs each time, and multiple outputs could all be “acceptable.” Evaluation frameworks help quantify quality across dimensions like accuracy, relevance, safety, and coherence.

Why Evaluation is Hard#

The Ground Truth Problem#

Traditional testing compares output to expected values. LLM outputs rarely have single correct answers:

• “Summarize this article” → Many valid summaries exist
• “Answer this question” → Phrasing, completeness, and tone all vary
• “Generate code for X” → Multiple correct implementations possible

The Scale Problem#

Manual review doesn’t scale. A production LLM application might handle thousands of requests daily. Human evaluation of every response is impractical.

The Drift Problem#

LLM behavior changes over time—model updates, prompt modifications, and context variations all affect output quality. Continuous evaluation is necessary to catch regressions.

Evaluation Approaches#

1. Reference-Based Evaluation#

Compare outputs against known good answers using traditional NLP metrics.

Metrics:

• BLEU: N-gram overlap with reference text (originally for translation)
• ROUGE: Recall-oriented overlap (originally for summarization)
• Exact Match: Binary match against expected output

Limitations: Penalizes valid paraphrases. “The cat sat on the mat” and “A feline rested upon the rug” score poorly against each other despite similar meaning.

2. LLM-as-Judge#

Use another LLM to evaluate outputs. The evaluator LLM receives the input, output, and a rubric, then scores the response.

How it works:

System: You are an evaluation assistant. Score the following response
on a scale of 1-5 for helpfulness, accuracy, and clarity.

Input: [user question]
Output: [model response]
Rubric: [scoring criteria]

Advantages:

• Handles semantic equivalence (understands paraphrases)
• Scales to large volumes
• Can evaluate subjective qualities (tone, helpfulness)

Limitations:

• Costs money (API calls for evaluation)
• Judge LLM has its own biases
• Can be gamed (outputs that “sound good” but are wrong)

3. Programmatic/Heuristic Evaluation#

Code-based checks for specific criteria.

Examples:

• JSON validity for structured outputs
• Regex patterns for format compliance
• Length constraints
• Profanity/PII detection
• Citation presence in RAG outputs

Advantages: Fast, deterministic, no API costs

Limitations: Only catches specific, predictable issues

The RAG Triad#

For Retrieval-Augmented Generation (RAG) systems, three metrics form the foundational evaluation framework:

┌─────────────┐     retrieves     ┌─────────────┐
│   Query     │ ───────────────→  │  Context    │
└─────────────┘                   └─────────────┘
                                        │
                                        │ generates
                                        ▼
                                  ┌─────────────┐
                                  │   Answer    │
                                  └─────────────┘

1. Context Relevance#

Question: Is the retrieved context relevant to the query?

Measures retrieval quality. If your vector search returns irrelevant documents, the LLM can’t produce good answers regardless of its capabilities.

Low score means: Fix your retrieval (embeddings, chunking, search parameters)

2. Faithfulness (Groundedness)#

Question: Is the answer supported by the retrieved context?

Measures hallucination. The LLM should only state things present in the provided context, not invent information.

Low score means: The model is hallucinating or over-generalizing

3. Answer Relevance#

Question: Does the answer actually address the query?

Measures response quality. Even with good context and no hallucination, the answer might miss the point or be incomplete.

Low score means: Prompt engineering needed, or model limitations

Metric Taxonomy#

Correctness Metrics#

• Factual accuracy: Are stated facts true?
• Faithfulness: Is output grounded in provided context?
• Hallucination rate: How often does the model invent information?

Relevance Metrics#

• Answer relevance: Does output address the input?
• Context relevance: Is retrieved context pertinent?
• Completeness: Are all aspects of the query addressed?

Quality Metrics#

• Coherence: Is the output logically structured?
• Fluency: Is the language natural and grammatical?
• Conciseness: Is the output appropriately brief?

Safety Metrics#

• Toxicity: Does output contain harmful content?
• Bias: Does output show unfair preferences?
• PII leakage: Does output expose personal information?
• Jailbreak resistance: Does the model refuse harmful requests?

Task-Specific Metrics#

• Code correctness: Does generated code execute properly?
• SQL validity: Is generated SQL syntactically correct?
• Tool use accuracy: Did the agent call the right tools with right parameters?

Evaluation vs Observability#

These terms are often conflated but serve different purposes:

Observability tells you the system is running. Evaluation tells you it’s running well.

In practice, production systems need both:

• Observability catches outages, latency spikes, error rates
• Evaluation catches quality degradation, hallucination increases, relevance drift

Self-Explaining Metrics#

A key differentiator among evaluation tools is whether metrics are self-explaining.

Non-explaining metric:

Faithfulness Score: 0.4

Why is it 0.4? Which claims weren’t grounded? Unclear.

Self-explaining metric:

Faithfulness Score: 0.4
Reason: The response claims "revenue increased 50%" but the context
only states "revenue showed growth." The specific percentage is not
supported by the retrieved documents.

Self-explaining metrics dramatically reduce debugging time by pointing directly to the problem.

Evaluation Pipeline Architecture#

A typical evaluation setup:

┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│  Test Cases  │ ──→ │  LLM App     │ ──→ │  Evaluators  │
│  (Dataset)   │     │  (Target)    │     │  (Metrics)   │
└──────────────┘     └──────────────┘     └──────────────┘
                                                 │
                                                 ▼
                                          ┌──────────────┐
                                          │   Results    │
                                          │  Dashboard   │
                                          └──────────────┘

Components#

Test Cases (Dataset) Curated inputs with optional expected outputs or evaluation criteria. Good datasets cover:

• Happy path scenarios
• Edge cases
• Adversarial inputs
• Domain-specific examples

Evaluators Functions that score outputs. Can be:

• LLM-as-judge (semantic evaluation)
• Programmatic (format, length, patterns)
• Human-in-the-loop (gold standard, expensive)
• Composite (multiple metrics combined)

Results Dashboard Aggregates scores across test runs. Enables:

• Regression detection (quality dropped after change)
• A/B comparison (which prompt is better?)
• Trend analysis (quality over time)

Common Evaluation Mistakes#

1. Evaluating Too Late#

Testing only before deployment misses production drift. Implement continuous evaluation on sampled production traffic.

2. Single Metric Fixation#

Optimizing for one metric (e.g., relevance) can tank others (e.g., safety). Use balanced metric sets.

3. Insufficient Test Coverage#

A few dozen test cases won’t catch edge cases. Aim for hundreds covering diverse scenarios.

4. Ignoring Evaluator Bias#

LLM judges have preferences. Validate judge outputs against human ratings periodically.

5. Static Datasets#

Real user queries evolve. Continuously expand test cases from production samples.

Cost Considerations#

Evaluation has direct costs:

Cost optimization strategies:

• Use cheaper models for initial filtering, expensive for edge cases
• Cache evaluation results for identical outputs
• Sample production traffic rather than evaluating everything
• Use programmatic checks where possible

When to Invest in Evaluation#

Minimal evaluation (just starting):

• Manual spot checks
• Basic programmatic checks (format, length)
• A few dozen test cases

Moderate evaluation (production app):

• Automated test suite in CI/CD
• LLM-as-judge for key metrics
• Hundreds of test cases
• Basic dashboard

Comprehensive evaluation (critical application):

• Continuous production evaluation
• Multiple metric coverage (correctness, safety, quality)
• Thousands of test cases
• Regression alerts
• Human-in-the-loop for edge cases

Glossary#

Chain-of-Thought (CoT): Prompting technique where the model shows reasoning steps. Can improve both generation and evaluation accuracy.

Few-Shot Evaluation: Providing example evaluations to guide the judge LLM’s scoring.

Golden Dataset: Curated test cases with human-verified expected outputs or scores.

Hallucination: When an LLM generates information not present in its context or training data.

Red Teaming: Adversarial testing to find vulnerabilities, jailbreaks, or failure modes.

Rubric: Scoring criteria provided to an LLM judge explaining how to rate responses.

Synthetic Data: Machine-generated test cases, often used to expand coverage cheaply.

S1 Synthesis: LLM Evaluation & Testing Frameworks#

Executive Summary#

The LLM evaluation landscape has matured significantly, with clear tool differentiation by use case. DeepEval emerges as the most comprehensive open-source option, while Ragas leads for RAG-specific evaluation. PromptFoo excels at quick iterations and security testing, LangSmith dominates observability, and TruLens offers OpenTelemetry-native tracing.

Comparison Matrix#

Decision Framework#

Choose DeepEval when:#

• Need comprehensive metric coverage (RAG, agents, safety, multimodal)
• Want CI/CD integration with pytest-style tests
• Require self-explaining metrics for debugging
• Building production systems with regression detection

Choose PromptFoo when:#

• Doing rapid prompt engineering iterations
• Need security/red team testing
• Prefer YAML config over code
• Want lightweight CLI tool without SDK

Choose LangSmith when:#

• Using LangChain/LangGraph
• Need production observability + evaluation
• Want unified tracing and testing platform
• Have budget for commercial tooling

Choose Ragas when:#

• Evaluating RAG systems specifically
• Want lower-cost RAG metrics (vs LLM-as-judge)
• Need quick integration, pandas-like workflow
• Don’t need general LLM evaluation

Choose TruLens when:#

• Already using OpenTelemetry
• In Snowflake ecosystem
• Need custom feedback functions
• Want extensible evaluation framework

Common Stack Patterns#

Full Coverage Stack#

DeepEval + Ragas + PromptFoo

• DeepEval: Comprehensive metrics, CI/CD backbone
• Ragas: RAG-specific depth when retrieval quality matters
• PromptFoo: Security validation, red teaming

Lightweight Stack#

Ragas + PromptFoo

• Lower overhead for RAG-focused applications
• Good for teams not needing 60+ metrics

Enterprise Stack#

LangSmith + DeepEval

• Observability + comprehensive evaluation
• Best for LangChain-based production systems

Key Insights#

1. No single tool covers everything - Most teams combine 2-3 tools
2. DeepEval has widest metric coverage (60+) with self-explanation
3. Ragas pioneered RAG Triad - still best for retrieval-focused eval
4. PromptFoo leads red teaming - best for security testing
5. LangSmith = observability-first - evaluation is secondary
6. TruLens differentiator - OpenTelemetry native, extensible

Cost Considerations#

Sources#

• DeepEval Alternatives Compared
• LLM Evaluation Frameworks Comparison
• Top LLM Evaluation Tools 2025
• TruLens Documentation
• PromptFoo Docs
• LangSmith Docs

S1: Rapid Discovery - Approach#

Research Date: 2025-12-10 Focus: which frameworks exist, and what each claims for itself

What This Pass Asked#

The opening question for any category: what is in it? S1 profiled five frameworks that show up in every discussion of LLM evaluation — DeepEval, Ragas, promptfoo, LangSmith and TruLens — recording capabilities, metric coverage, license, pricing and the trade-offs each project acknowledges.

The output is a shopping comparison. It sorts the field well enough to decide what merits a deeper pass, and not well enough to commit to one.

Sources#

S1 worked from the projects’ own documentation and from published comparisons. The sources recorded in SYNTHESIS.md are:

• DeepEval Alternatives Compared (deepeval.com)
• LLM Evaluation Frameworks Comparison (comet.com)
• Top LLM Evaluation Tools 2025 (confident-ai.com)
• TruLens documentation (trulens.org)
• promptfoo documentation (promptfoo.dev)
• LangSmith documentation (docs.langchain.com)

Two of the six are published by the vendor of one of the entrants. That is visible in the result: the comparison matrix leads with metric count, which is the axis on which that vendor wins.

What Has Been Corrected Since#

S2 re-read every registry and repository fact on 2026-08-26 UTC and found four claims in this pass that no longer hold or never did. The originals are left in place below as the record of what S1 concluded; the corrections and their primary sources are in ../S2-comprehensive/recommendation.md.

• The RAG Triad is TruEra’s, not Ragas’s, and has three members. SYNTHESIS.md and trulens.md contradicted each other on this point.
• Ragas moved to a new GitHub organization and has merged nothing since 2026-02-24.
• DeepEval is no longer Python-only; a TypeScript SDK ships on npm.
• DeepEval’s metric count is not 60+, and its red teaming has moved to a separate package.

Scope Boundary#

Frameworks for evaluating and testing LLM applications. Ranking base models on public benchmarks is a different activity, and general observability platforms that compute no scores are out of scope.

DeepEval#

Overview#

• Type: Open-source Python framework
• License: Apache 2.0
• GitHub: 400k+ monthly downloads
• Focus: Comprehensive LLM evaluation (“Pytest for LLMs”)

Key Features#

• 60+ metrics: Prompt, RAG, chatbot, safety, multimodal
• Self-explaining metrics: Tells you WHY scores are low
• Pytest integration: Familiar unit-test interface
• CI/CD native: Built for continuous deployment workflows
• Safety testing: Red teaming, toxicity detection

Metric Categories#

• RAG: Faithfulness, contextual relevancy, answer relevancy
• Conversational: Coherence, engagement, knowledge retention
• Safety: Bias, toxicity, PII leakage, jailbreak detection
• Agentic: Tool use, task completion, reasoning

Enterprise Platform (Confident AI)#

• Cloud dashboard for team collaboration
• Dataset curation and annotation
• Production monitoring
• Regression detection

Limitations#

• Python-only (no JS/CLI-first option)
• Enterprise features require Confident AI platform
• Can be overkill for simple prompt testing

Best For#

• Teams needing comprehensive evaluation coverage
• CI/CD integration with automated testing
• Production monitoring and regression detection
• Multi-pattern evaluation (RAG, agents, chatbots)

Installation#

pip install deepeval

Pricing#

• Open-source: Free
• Confident AI: Free tier + paid plans for enterprise

LangSmith#

Overview#

• Type: Commercial SaaS platform
• Company: LangChain Inc.
• Focus: Tracing, observability, and evaluation for LLM apps

Key Features#

• Detailed tracing: Visibility into every execution step
• Dataset management: Create/organize test data
• Multiple evaluator types: Code-based, LLM-as-judge, composite
• Experiment tracking: Compare results across test runs
• Framework agnostic: Works with or without LangChain

Integration#

• Seamless with LangChain and LangGraph
• Python and TypeScript SDKs
• REST API for custom integrations
• No LangChain dependency required

Evaluation Capabilities#

• Custom evaluation logic
• Prebuilt assessment tools
• Quality tracking over time
• Consistency validation

Limitations#

• Commercial product (not fully open-source)
• Tracing-first, evaluation second
• Tighter integration with LangChain ecosystem
• Pricing can scale with usage

Best For#

• LangChain/LangGraph users
• Teams needing production observability
• Debugging complex multi-step chains
• Organizations wanting unified tracing + eval

Pricing#

• Developer: Free tier with limits
• Plus: $39/seat/month
• Enterprise: Custom pricing

PromptFoo#

Overview#

• Type: Open-source CLI and library
• License: MIT
• GitHub: 51,000+ developers
• Focus: Prompt testing, A/B testing, red teaming

Key Features#

• CLI-first: Simple command-line interface, no cloud required
• YAML configuration: Declarative test case definition
• Side-by-side comparison: Diff views for prompt variations
• Red teaming: Automated security testing (injections, toxic content)
• CI/CD ready: Integrates into deployment pipelines

Supported Providers#

• OpenAI, Anthropic, Azure, Google, HuggingFace
• Open-source models (Llama, etc.)
• Custom API providers

Evaluation Capabilities#

• Basic RAG metrics
• Safety/security testing
• LLM-as-judge evaluations
• Custom assertion logic

Limitations#

• Limited metric set compared to DeepEval (basic RAG + safety only)
• YAML-heavy workflow harder to customize at scale
• Less comprehensive than code-first alternatives

Best For#

• Quick prompt iterations
• Security/red team testing
• Teams preferring declarative config over code
• Lightweight experimentation without SDK dependencies

Installation#

npm install -g promptfoo
# or
npx promptfoo@latest

Pricing#

• Open-source: Free, self-hosted
• Cloud: Optional hosted dashboard

Ragas (Retrieval-Augmented Generation Assessment Suite)#

Overview#

• Type: Open-source Python library
• License: Apache 2.0
• Focus: RAG-specific evaluation metrics

Key Features#

• RAG Triad: Structured evaluation framework
• Lightweight: Easy integration, pandas-like workflow
• Reference-free: No ground truth required
• Benchmarked: Against LLM-AggreFact, TREC-DL, HotPotQA

Core Metrics (RAG Triad)#

1. Faithfulness: How accurately answer reflects retrieved evidence
2. Context Relevancy: How relevant retrieved docs are to query
3. Answer Relevancy: How relevant answer is to user question
4. Context Recall: Coverage of relevant information
5. Context Precision: Signal-to-noise in retrieved context

Extended Capabilities#

• Agentic workflow metrics
• Tool use evaluation
• SQL evaluation
• Multimodal faithfulness
• Noise sensitivity testing

Limitations#

• Metrics somewhat opaque (not self-explanatory)
• RAG-focused, not general LLM evaluation
• Need to combine with other tools for full coverage
• Lower metric count than DeepEval

Best For#

• RAG pipeline evaluation specifically
• Teams wanting targeted retrieval metrics
• Lower-cost alternative to LLM-as-judge for RAG
• Quick integration with existing RAG systems

Installation#

pip install ragas

Pricing#

• Open-source: Free

TruLens#

Overview#

• Type: Open-source Python library
• License: MIT
• Maintainer: Snowflake (acquired TruEra)
• Focus: Feedback functions and tracing for LLM apps

Key Features#

• Feedback functions: Programmatic evaluation without ground truth
• RAG Triad pioneer: Original structured RAG evaluation framework
• OpenTelemetry support: Interoperable observability
• Extensible: Custom feedback function framework
• Provider integrations: OpenAI, HuggingFace, LiteLLM, LangChain

Feedback Function Types#

• Generation-based: LLM-as-judge with rubrics
• Custom logic: Tailored evaluation tasks
• Chain-of-thought: Optional reasoning traces
• Few-shot: Example-guided evaluation

Tracing Capabilities#

• OpenTelemetry (OTel) native
• Integrates with existing observability stack
• Detailed execution traces
• Performance monitoring

Supported Use Cases#

• Question-answering
• Summarization
• RAG systems
• Agent-based applications

Limitations#

• Snowflake acquisition may affect roadmap
• Overlaps with Ragas on RAG evaluation
• Less comprehensive metrics than DeepEval
• Community-driven, less commercial support

Best For#

• Teams already using OpenTelemetry
• Snowflake ecosystem users
• Custom feedback function needs
• RAG evaluation with extensibility

Installation#

pip install trulens

Pricing#

• Open-source: Free

S2: Comprehensive Analysis - Approach#

Research Date: 2026-08-25 (S1 originally 2025-12-10) Registry and repository figures: read 2026-08-26 UTC, from PyPI, the npm registry and the GitHub API Focus: what each framework needs in hand before it can emit a number

What This Pass Asks#

S1 sorted these five by metric count and by badge — 60+ metrics here, RAG specialist there, observability platform over there. Metric count is the wrong axis. Two frameworks with the same metric on the list can be impossible to swap for each other, because they disagree about what you have to hand them first.

S2 goes underneath the metric catalogs to the question every one of them answers differently:

What has to exist before this framework can produce a score?

Three answers, and every framework here sits on one of them by default:

The rows overlap at one edge, and the overlap should be stated exactly. LangSmith also invokes a target function during an experiment, so it can produce outputs rather than only score them. What it cannot do is drive a system it knows only as a URL, which is promptfoo’s distinguishing capability, and what promptfoo cannot do is attach a score to a production request, which is LangSmith’s.

The distinction decides more than any feature table. It fixes whether you can run an eval before your app exists, whether the eval can run in CI without a deployment, whether a score can be attached to a production request, and what you have to rewrite if you change your mind.

A second question falls out of the first and is nearly as load-bearing:

Where does the score get computed, and where does it land?

Four of the five compute in your process and write to a local SQLite file or a DataFrame. LangSmith computes against a hosted backend, and its self-hosted deployment is an Enterprise-plan item. That is one bit of information, and for some readers it ends the comparison before any metric is discussed.

Method#

Primary sources, in this order:

1. Registries. Version, license classifier, upload date and release cadence were read from pypi.org/pypi/<pkg>/json and registry.npmjs.org/<pkg>. Stars, license SPDX, archive state, last push and commit history came from the GitHub API. Nothing in this pass repeats a maturity figure from S1 without re-reading it.
2. Each project’s own documentation and README, for how a test is defined and what it requires at runtime. Object names, method names, CLI commands and environment variables below are quoted from those sources.
3. Where a vendor makes a claim about itself — adoption counts, benchmark wins — it is reported as the vendor’s claim with its source, not asserted.

Four of S1’s recorded facts had moved or were wrong. They are corrected in the per-framework files and listed together in recommendation.md.

Where a figure could not be verified it says so. PyPI download counts are the main gap: pypistats.org returned HTTP 429 on every attempt across this session, so no monthly-download figure for any Python package here is confirmed. npm download counts were retrievable and are used where they apply.

Structure of This Pass#

• deepeval.md — the pytest-shaped test-case framework
• ragas.md — the RAG test-case framework, and its test-data generator
• promptfoo.md — the framework that calls the target itself
• langsmith.md — the hosted trace-and-experiment platform
• trulens.md — the OpenTelemetry-native span scorer
• recommendation.md — what the technical picture changes about S1

Scope Boundary#

This survey covers frameworks for evaluating and testing LLM applications. Model benchmarking — ranking base models on MMLU, HumanEval and similar — is a different activity and appears here only where a framework happens to ship it. General observability platforms that are not evaluation tools are out of scope; LangSmith and TruLens are in because both compute scores, not only traces.

DeepEval in Depth#

What it needs first: a test case — a record of an interaction your application has already had. DeepEval never calls your application. You run it, capture what came back, and hand DeepEval the transcript.

Verified 2026-08-26 UTC: PyPI deepeval 4.2.0, uploaded 2026-08-24, Apache-2.0, requires Python >=3.9,<4.0. GitHub confident-ai/deepeval, 17,861 stars, last push 2026-08-25, not archived. Five releases in the sixteen days to 2026-08-24, 4.1.7 through 4.2.0.

How a Test Is Defined#

The unit is LLMTestCase, which carries nine parameters. Two are mandatory — the docs state “the input and actual_output are always mandatory” — and the other seven are supplied only when a metric asks for them:

This is the shape that makes the category legible. A metric’s requirements are a subset of these fields, so “can I run this metric?” reduces to “do I have these columns?” — a question you can answer before writing any code.

Evaluation runs three ways: end-to-end over a list of test cases, component-level via an @observe decorator on internal functions, and one-off on a single metric and case.

What It Measures#

The README names 35 metrics across seven groups: two custom-metric builders (G-Eval, DAG), eight agentic, six RAG, five multi-turn, three MCP, five multimodal and six other. The documentation introduction claims “50+ SOTA, ready-to-use metrics”; S1’s “60+” is not supported by either source and is corrected here.

Two of those groups are the interesting ones:

• G-Eval is a judge metric where you write the criterion in prose and it builds the evaluation steps. This is the escape hatch: any criterion you can state, you can score, at the cost of a judge call per case.
• DAG is a graph of decision nodes that the docs describe as “fully deterministic”. You compose the judgment out of discrete checks rather than asking a model for a number. This is the answer to the objection that LLM-as-judge scores wander between runs.

Every metric returns a score and a written reason. That is a design commitment rather than a feature: the reason is what makes a failing CI run actionable instead of a red square.

What It Needs at Runtime#

A judge model, for most metrics. The README’s own framing: metrics are “powered by ANY LLM of your choice, statistical methods, or NLP models that run locally on your machine”. In practice the RAG, agentic and multi-turn metrics are LLM-as-judge, and each scored case is at least one inference call.

The judge does not have to be hosted. deepeval set-ollama --model=..., or OllamaModel passed to a metric constructor, points the judge at a local server (default http://localhost:11434). Confident AI, the commercial platform, is not required for any of this.

A labeled set, for a minority of metrics. Reference-based metrics need expected_output; faithfulness, answer relevancy and contextual relevancy do not. A team with no labels can still run most of the RAG suite.

Integration Surface#

The framing is “Pytest for LLMs” and that is close to literal. The docs: “deepeval plugs into pytest via assert_test() and the deepeval test run command.” A suite is deepeval test run test_llm_app.py, and the docs advise against plain pytest because the wrapper “adds a range of functionalities on top of Pytest for unit-testing LLMs”. assert_test() takes either a test_case or, under tracing, a golden. CI integration is therefore whatever your CI already does with pytest; standalone evaluate() runs are the alternative.

S1 recorded “Python-only (no JS/CLI-first option)” as a limitation. That is now wrong. A TypeScript SDK exists: npm deepeval 0.9.13, published 2026-08-24 — the same day as the Python 4.2.0 release — described as “The LLM Evaluation Framework for TypeScript”. The Ollama documentation shows the TypeScript form alongside the Python one.

What It Cannot Do#

• Call your application. No provider abstraction, no HTTP target. If your system under test is a deployed endpoint in a language DeepEval does not bind to, you write the client yourself.
• Score production traffic on its own. Component-level tracing exists, but continuous production monitoring is a Confident AI platform capability, not a library one.
• Red team. S1 lists “red teaming, jailbreak detection” under DeepEval’s safety testing. Adversarial red teaming now lives in a separate package, deepteam (PyPI 1.0.9, 2026-08-12; GitHub confident-ai/deepteam, Apache-2.0, 2,622 stars), whose own docs say it “is powered by deepeval, the LLM evaluation framework.” DeepEval retains bias and toxicity as ordinary metrics; the attack generation is next door.
• Escape judge cost. Outside the DAG and statistical metrics, the bill scales with cases × metrics × judge price.

Sources#

• DeepEval PyPI
• deepeval on npm
• confident-ai/deepeval on GitHub
• Test cases
• Metrics introduction
• Ollama integration
• DeepTeam docs

LangSmith in Depth#

What it needs first: a backend account. LangSmith is a hosted platform with an open-source client. The datasets live there, the experiments run against there, and the results are read there. Everything else about it follows from that one fact.

Verified 2026-08-26 UTC: PyPI langsmith 0.11.1, uploaded 2026-08-19, MIT. GitHub langchain-ai/langsmith-sdk, 1,035 stars, MIT, last push 2026-08-26. npm langsmith recorded 25,011,410 downloads in the 30 days ending 2026-08-24 — a figure that measures LangChain’s dependency graph more than LangSmith’s adoption, since the client is pulled in transitively.

The star count is the tell for what is open here: 1,035 stars on the SDK against 17,861 for DeepEval and 24,575 for promptfoo. The SDK is MIT. The product is not open source. S1 recorded this correctly.

How an Evaluation Is Defined#

Three objects, in a fixed order:

1. A dataset — “a collection of examples used for evaluating an application”, created through the client (ls_client.create_dataset(...), ls_client.create_examples(...)). Each example carries inputs, optional reference outputs used only by evaluators, and optional metadata for filtering.
2. A target function taking an input dictionary and returning an output dictionary. This is where LangSmith differs from DeepEval and Ragas: the platform invokes your code, so the outputs under test are produced during the run rather than captured beforehand.
3. Evaluators — functions that score a run against an example.

evaluate() (aevaluate() for async) ties them together with data=, evaluators= and optional metadata=. Each call “creates an experiment that you can view in the LangSmith UI or query via the SDK”. Results land in the experiments table, sortable by score.

What It Measures#

LangSmith does not ship a metric catalog in the DeepEval or Ragas sense. It ships four evaluator types and expects you to fill them:

• Human — manual review of outputs and execution traces
• Code — deterministic, rule-based functions
• LLM-as-judge — reference-free or reference-based
• Pairwise — comparing two application versions, by heuristic, model or human

That is a real difference in posture. DeepEval hands you Faithfulness; LangSmith hands you the place to put your faithfulness function. Teams that want a standard metric off the shelf will write more code here; teams whose quality criterion is idiosyncratic will fight the framework less.

Evaluators are workspace-level resources and run in two modes. Offline evaluators receive an example plus a run — they have the reference output. Online evaluators receive production runs only, with no reference. That split is the mechanism behind LangSmith’s distinguishing capability: the same scoring function can gate a pre-release experiment and then score live traffic, because a production request is the same object as a test run.

What It Needs at Runtime#

An API key and network egress to the backend. The client is constructed with credentials; the dataset is created server-side; the experiment is recorded server-side. There is no local-only mode in the cloud plans.

A judge model for the LLM-as-judge evaluators, billed by you to your provider, on top of platform usage.

Nothing from LangChain. Framework-agnostic use is supported and S1 recorded that correctly. The gravity is practical rather than technical: if the application is already a LangChain or LangGraph app, its traces are already being emitted in the shape LangSmith reads, and the marginal cost of turning on evaluation is close to zero. If it is not, LangSmith competes on its own merits against four tools that do not charge per seat.

Pricing, and the Self-Hosting Boundary#

Read from the vendor’s pricing page 2026-08-26:

Usage beyond the included traces is metered at $1.50 per LCU (compute unit) and $1.00 per LSU (storage unit). Base traces are retained 14 days; extended retention runs 400 days at additional cost.

S1’s “$39/seat” was right and is still right. What S1 omitted is the part that decides for a whole class of reader: self-hosted and hybrid deployment are Enterprise-only. There is no self-hosted Developer or Plus tier. For a team that cannot send prompts and completions to a third party, LangSmith is not a $39 decision — it is a sales conversation, or it is off the list.

What It Cannot Do#

• Run without the backend. The defining constraint.
• Ship you a metric. You bring the scoring logic, or you write a rubric.
• Be audited. The platform is closed; the SDK is not the product.
• Escape the LangChain question entirely. Nothing requires LangChain, but every integration path is smoothest through it, and the vendor is LangChain Inc. A team betting against that ecosystem is choosing a dependency on it.

Sources#

• langsmith PyPI
• langchain-ai/langsmith-sdk on GitHub
• Evaluation concepts
• Evaluate an LLM application
• LangSmith pricing

promptfoo in Depth#

What it needs first: a way to call the target. promptfoo produces the output under test itself — you give it prompts and providers, it runs the matrix and asserts on what comes back. LangSmith also invokes a target function, but only one you write in Python or TypeScript inside its own harness; promptfoo is the only entrant here that will drive a system it knows solely as a URL, and the only one that can make the model provider a dimension of the test.

Verified 2026-08-26 UTC: npm promptfoo 0.122.0, published 2026-08-04, MIT, 419 published versions since 2023-05-03, 2,435,183 downloads in the 30 days ending 2026-08-24. GitHub promptfoo/promptfoo, 24,575 stars, MIT, not archived, last push 2026-08-26. A thin Python wrapper also exists (PyPI promptfoo 0.1.4, 2026-04-06) that shells out to the CLI; the CLI is the product.

S1’s “51,000+ developers” is stale. The vendor’s own site now claims “300,000+ developers” (read 2026-08-26). Both are vendor claims and neither is independently verifiable; the checkable number is the star count above.

How a Test Is Defined#

A promptfooconfig.yaml declares three things — prompts, providers, and test cases with assertions — and promptfoo eval runs the cross product. The inversion of control is the design: with N prompts and M providers you get N×M outputs from one file, and the side-by-side diff view is the natural way to read them. No other framework here can vary the provider as a dimension of the test, because no other framework calls the provider.

What It Measures: The Split That Matters#

promptfoo’s assertion catalog is explicitly divided, and the division is the most useful artifact in this survey for anyone building a CI gate.

Deterministic assertions run programmatically on the output, cost nothing per case beyond the model call already made, and return the same verdict every time: equals, contains, icontains, regex, starts-with, contains-any, contains-all, icontains-any, icontains-all, is-json, contains-json, contains-html, is-html, is-sql, contains-sql, is-xml, contains-xml, is-refusal, javascript, python, ruby, webhook, rouge-n, bleu, gleu, levenshtein, latency, meteor, perplexity, perplexity-score, cost, is-valid-function-call, is-valid-openai-function-call, is-valid-openai-tools-call, trace-span-count, trace-span-duration, trace-error-spans, skill-used, trajectory:tool-used, trajectory:tool-args-match, trajectory:tool-sequence, trajectory:step-count, guardrails.

Model-graded assertions call a judge: similar, classifier, moderation, llm-rubric, g-eval, answer-relevance, context-faithfulness, context-recall, context-relevance, conversation-relevance, trajectory:goal-success, factuality, model-graded-closedqa, pi, select-best, max-score.

Any of them can be negated by prefixing not-.

Two consequences. First, latency and cost are assertions — you can fail a build on a regression that has nothing to do with output quality, which none of the test-case frameworks offer as a first-class check. Second, a team can build an entire eval suite with zero judge calls by staying in the first list. That makes the per-run cost and the run-to-run variance both zero, which is what a per-PR gate actually needs.

S1’s “Limited metric set compared to DeepEval (basic RAG + safety only)” is wrong on the first count: the RAG-relevant model-graded assertions (context-faithfulness, context-recall, context-relevance, answer-relevance) plus llm-rubric and g-eval are not a basic set. What promptfoo lacks against DeepEval is the depth of the agentic and multi-turn catalogs, not RAG coverage.

Red Teaming, and the Wire It Runs On#

Red teaming is a first-class mode with its own three-command lifecycle: promptfoo redteam init to scaffold, promptfoo redteam run to generate adversarial cases and run them, promptfoo redteam report to view results. Plugins generate the payloads (harmful content, BOLA, BFLA, competitor endorsement, and others); strategies decide how they are delivered.

By default, adversarial generation happens on promptfoo’s servers. The documentation is direct about it: “By default, promptfoo uses a remote service for generating adversarial inputs.” The connectivity check the docs give is curl https://api.promptfoo.app/version, and the troubleshooting page explains that corporate firewalls block it “since our service generates potentially harmful outputs for testing purposes.”

There is an opt-out, and the docs state the trade-off plainly: “You can force 100% local generation by setting the PROMPTFOO_DISABLE_REDTEAM_REMOTE_GENERATION environment variable to true. Note that the quality of local generation depends greatly on the model that you configure, and is generally low for most models.” A warning admonition on the configuration page repeats it: “Disabling remote generation may result in lower quality adversarial inputs.”

This is the single most consequential runtime fact in this survey for anyone under a data-egress rule, and it applies to red teaming only — ordinary promptfoo eval runs against whatever providers you configured.

Integration Surface#

Node CLI, installable globally or run through npx, with promptfoo and pf as binaries. Results land in a local SQLite database under ~/.promptfoo; a web viewer and a self-hosted Docker image exist, with PROMPTFOO_REMOTE_API_BASE_URL and PROMPTFOO_REMOTE_APP_BASE_URL pointing clients at your own server. CI usage is the documented default — --tag key=value records run context such as a git SHA against a scan.

Because the target is a provider, promptfoo can test an application it knows only as an HTTP endpoint. That is the capability the test-case frameworks cannot reach, and it makes language irrelevant: the system under test can be written in anything.

What It Cannot Do#

• Score something it cannot invoke. The inverse of its strength. If you have a log of production interactions and no ability to replay them, promptfoo is the wrong instrument.
• Attach a score to a production trace. It is an offline harness.
• Give a deep agent-trajectory or multi-turn analysis on the scale of DeepEval’s catalog, though the trajectory:* assertions cover the basics.
• Generate high-quality adversarial inputs offline. Per its own docs.

Sources#

• promptfoo on npm
• promptfoo/promptfoo on GitHub
• Assertions and metrics
• Red team configuration
• Remote generation errors
• Self-hosting

Ragas in Depth#

What it needs first: a test case, same position as DeepEval — but Ragas also ships the only thing in this survey that manufactures the test cases for you, from documents you already have.

Verified 2026-08-26 UTC: PyPI ragas 0.4.3, uploaded 2026-01-13, Apache-2.0, requires Python >=3.9. GitHub vibrantlabsai/ragas — 15,472 stars, Apache-2.0, not archived, last push 2026-02-24.

The Maintenance Picture, Because It Changed#

Two facts that S1 could not have recorded, both checked against the GitHub API on 2026-08-26:

The repository moved. explodinggradients/ragas now redirects to vibrantlabsai/ragas. The move is datable from PyPI metadata: release 0.3.9 (2025-11-11) points at explodinggradients, release 0.4.0 (2025-12-03) points at vibrantlabsai. The LICENSE copyright line reads “Copyright [2023] [Vibrant Labs]” and the README gives [email protected] as the contact.

The default branch has been still for six months. The most recent commit is fix: allow fork contributors in check-docs CI workflow (#2606), dated 2026-02-24. Meanwhile there are 216 open pull requests, with new ones filed as recently as 2026-08-25, and a GitHub search for PRs merged after 2026-02-24 returns zero. Releases stopped at 0.4.3 on 2026-01-13.

The library works — a stalled repository is not a broken one, and the metrics compute exactly as they did in January. What has stopped is the part you cannot supply yourself: bug fixes, provider-API drift, and anyone merging the community patches piling up in the queue. Every other framework in this survey shipped a release within the last three weeks.

How a Test Is Defined#

Records are dictionaries with four field names, loaded into a dataset:

{"user_input": ..., "retrieved_contexts": [...], "response": ..., "reference": ...}
# EvaluationDataset.from_list(dataset)

reference is the golden answer and is needed only by the reference-based metrics. Scoring is one call:

evaluate(dataset=evaluation_dataset,
         metrics=[LLMContextRecall(), Faithfulness(), FactualCorrectness()],
         llm=evaluator_llm)          # LangchainLLMWrapper(llm)

The llm= argument is the point. Ragas does not assume a judge; you wrap and pass one, which is why swapping in a local model is a one-line change.

What It Measures#

Seven groups of metrics, per the metric catalog:

The Natural-language-comparison group matters more than its dull name: BLEU, ROUGE, string similarity and exact match are non-LLM metrics. They cost nothing per case and return the same number every run. A team worried about judge variance can build a deterministic floor out of that group and use judge metrics only where nothing cheaper will do.

S1 called this a “RAG Triad” of five metrics and credited Ragas with pioneering it. Both halves are wrong — see trulens.md. Ragas does not use the term; the framework it publishes is a metric catalog, not a triad.

The Test-Data Generator#

This is the capability nothing else here matches, and it changes which persona Ragas serves. Given a set of documents, Ragas builds a knowledge graph and generates queries from it:

1. Documents are chunked into hierarchical Nodes.
2. Extractors pull entities and keyphrases from each node.
3. A relationship builder connects nodes that share extracted information — for example by Jaccard similarity over shared entities.
4. QuerySynthesizers traverse the graph, producing single-hop and multi-hop queries varied along query length, query style (web search, chat) and persona.

The input is documents. The output is an evaluation set with references. A team that has a corpus and no labels can get a test set out of this in an afternoon — which is the whole problem for most teams starting an eval program.

What It Cannot Do#

• Call your application. Same limitation as DeepEval: you run the pipeline, capture response and retrieved_contexts, then evaluate.
• Explain a low score the way DeepEval does. Scores come back as numbers in a result mapping. Diagnosis is your job.
• Cover non-RAG territory in depth. The agentic and general-purpose groups exist, but the center of gravity is retrieval.
• Promise a fix. With 216 PRs open and none merged since February, a bug you find is a bug you carry, or a fork.

Sources#

• ragas PyPI
• vibrantlabsai/ragas on GitHub
• Available metrics
• RAG evaluation quickstart
• Test data generation for RAG

S2 Recommendation: What the Technical Picture Changes#

S1 ranked five frameworks and named a default: DeepEval for coverage, Ragas for RAG. S2 does not overturn that so much as show that it answers a question most readers are not yet asking. Before “which metrics?” comes “what do I have?” — and that question sorts the category cleanly, with no ties.

The Category Is Three Positions, Not a Ranking#

These do not compete. A team can hold two at once without conflict — promptfoo gating pull requests and TruLens scoring production is a coherent stack, not a duplication. Ranking them against each other, which the “60+ metrics beats 5 metrics” headline did, compares tools that answer different questions.

What S1 Got Wrong#

Four corrections, all checked against primary sources on 2026-08-26 UTC.

1. The RAG Triad is TruEra’s, not Ragas’s, and it has three members. S1’s synthesis said “Ragas pioneered RAG Triad” while S1’s own trulens.md said the opposite. The TruLens documentation settles it: “TruEra has innovated the RAG triad to evaluate for hallucinations along each edge of the RAG architecture.” Ragas does not use the term.

2. Ragas has been quiet for six months. The repository moved from explodinggradients/ragas to vibrantlabsai/ragas between releases 0.3.9 (2025-11-11) and 0.4.0 (2025-12-03). The last release is 0.4.3, 2026-01-13. The last commit on the default branch is 2026-02-24. There are 216 open pull requests and zero merged since that date, with new ones still arriving. Every other framework here released within the last three weeks. S1 was written 2025-12-10, before any of this; it is not S1’s error, but a reader acting on S1 today would be choosing the one entrant whose maintainers have stopped merging.

3. DeepEval is no longer Python-only. S1 listed that as its leading limitation. A TypeScript SDK ships on npm as deepeval 0.9.13, published 2026-08-24 — the same day as the Python 4.2.0 release.

4. The metric count is not 60+. DeepEval’s own documentation says “50+”; its README names 35. Adversarial red teaming, which S1 counted under DeepEval, now lives in the separate deepteam package.

Two smaller items: promptfoo’s “51,000+ developers” is superseded by the vendor’s current claim of “300,000+” (both vendor claims; the checkable figure is 24,575 GitHub stars), and DeepEval’s “400k+ monthly downloads” could not be re-verified because pypistats.org returned HTTP 429 throughout this session.

The Three Findings That Should Change a Decision#

1. Judge calls are optional, and the frameworks differ in how easily you can avoid them. LLM-as-judge is the default assumption of this category and it carries two costs a CI gate cannot absorb: money per case and a score that moves between runs. Three escape routes exist, and they are not equally accessible. promptfoo’s assertion catalog is split into deterministic and model-graded lists, so building a judge-free suite is a matter of staying in one list. DeepEval’s DAG metric composes a deterministic judgment out of discrete decision nodes. Ragas’s natural-language-comparison group (BLEU, ROUGE, string similarity, exact match) is non-LLM. Any of the three gives you a floor that costs nothing and never wanders.

2. The judge does not have to be someone else’s. DeepEval points at a local Ollama server with one CLI command; Ragas takes the judge as an llm= argument you construct; TruLens providers include local options. This matters more than the license column, because for most teams the question “can we use this?” is decided by where the prompts go, not by the SPDX identifier. On that test the four open-source entrants can all be run with nothing leaving the network — with one exception, below — and LangSmith cannot below its Enterprise plan.

3. Two of the five have a hosted service in the default path, and one is not the one you would guess. LangSmith’s is obvious and priced: $0 Developer, $39 per seat Plus, self-hosting Enterprise-only. promptfoo’s is not obvious: red team runs generate adversarial inputs on api.promptfoo.app by default. The opt-out exists — PROMPTFOO_DISABLE_REDTEAM_REMOTE_GENERATION=true — and the vendor’s own documentation says quality “is generally low for most models” without it. An MIT license on the CLI does not mean the security scan runs on your hardware.

Choose By What You Have#

• “I have logs of what my app produced.” → DeepEval or Ragas. Ragas if the question is about retrieval and you can accept a stalled upstream; DeepEval otherwise, and DeepEval whenever a failing score needs to explain itself.
• “I have an endpoint and a set of prompts.” → promptfoo. It is the only one that can test a system it knows only by URL, and the only one that can make the model provider a variable.
• “I have documents and no test set.” → Ragas’s knowledge-graph test-data generator, which is the only thing in this survey that manufactures an evaluation set from a corpus. DeepEval’s synthesizer is the second option.
• “I have an instrumented application and want the retriever scored separately.” → TruLens. Selectors point at a span; nothing else here can score an intermediate step without you plumbing it out by hand.
• “I have LangChain traces and a budget.” → LangSmith, where the marginal cost of turning evaluation on is close to zero.

What S2 Did Not Settle#

Whether any of these judges are accurate. Every framework here computes scores; only TruLens publishes judge-versus-human-annotation figures, and those are its own benchmark selection. There is no neutral cross-framework measurement of judge agreement with human raters, which means the number a team gates their release on has an unmeasured error bar. That gap is the most important open question in this category and no survey of feature lists can close it.

TruLens in Depth#

What it needs first: instrumentation. TruLens scores spans of an execution that happened inside your process. You decorate the functions you care about, run the app, and metrics attach themselves to the trace.

Verified 2026-08-26 UTC: PyPI trulens 2.13.1, uploaded 2026-08-20, MIT, requires Python >=3.10,<4.0. GitHub truera/trulens, 3,524 stars, MIT, not archived, last push 2026-08-25. Repository created 2020-11-02 — the oldest codebase in this survey by two and a half years, though its LLM-evaluation identity is much newer. Five releases between 2026-07-28 and 2026-08-20.

trulens is a metapackage. It pulls trulens-core, trulens-feedback, trulens-dashboard[full], trulens-otel-semconv and trulens_eval, all pinned >=2.0.0,<3.0.0. Providers and connectors are separate distributions, so a minimal install is possible.

Ownership#

TruLens came from TruEra, which Snowflake announced it was acquiring on 2024-05-22. The repository still sits under the truera organization and the license is still MIT. Snowflake is a first-class integration — there is a Snowflake Cortex feedback provider, a Snowflake connector, an event-table backend and server-side evaluation artifacts — but nothing in the library requires Snowflake, and the default backend is a local SQLite file.

S1’s framing of this as a risk is fair and unresolved; what the record shows as of 2026-08-26 is an actively released MIT library, not a discontinued one.

The RAG Triad — and Who Actually Coined It#

Three metrics, one per edge of a RAG system:

The construction is what makes it useful: the three edges triangulate a failure. A grounded but irrelevant answer means retrieval pulled the wrong chunks; a relevant answer that is not grounded means the generator invented something. One number cannot tell you which; three can.

Groundedness is defined operationally rather than as a vibe — the docs describe separating the response “into individual claims and independently search for evidence that supports each within the retrieved context.”

S1’s synthesis credited Ragas with pioneering the RAG Triad, and listed it as five metrics. Both are wrong. The TruLens documentation states: “TruEra has innovated the RAG triad to evaluate for hallucinations along each edge of the RAG architecture.” S1’s own trulens.md recorded this correctly, so the survey contradicted itself; the synthesis is the half that was wrong. A triad has three members, and Ragas does not use the term at all.

How an Evaluation Is Defined#

Three pieces:

A session. session = TruSession() orchestrates tracing and results, with reset_database() and get_leaderboard() for managing them.

Instrumentation. The @instrument decorator from trulens.core.otel.instrument marks a method for tracing and declares what the span means:

@instrument(
    span_type=SpanAttributes.SpanType.RETRIEVAL,
    attributes={SpanAttributes.RETRIEVAL.QUERY_TEXT: "query",
                SpanAttributes.RETRIEVAL.RETRIEVED_CONTEXTS: "return"},
)

A metric, built from three parts: an implementation (for example provider.groundedness_measure_with_cot_reasons_consider_answerability), a set of selectors mapping its parameters onto pieces of the trace (Selector.select_context(collect_list=True), Selector.select_record_output(), Selector.select_record_input()), and an optional aggregation such as np.mean.

The selector is the idea worth taking away even if you pick a different tool. A DeepEval test case flattens an execution into named fields you assemble by hand; a TruLens selector points at a place in the trace and says score that. The consequence is that TruLens can score an intermediate step — the retriever’s output, before the generator ever saw it — without you plumbing that value out of your application.

What It Needs at Runtime#

Your application, running. There is no way to evaluate a system TruLens cannot execute inside. This is the sharpest boundary in the survey: DeepEval and Ragas want a transcript, promptfoo wants an endpoint, TruLens wants to be in the process.

A judge model for the feedback providers — OpenAI, Anthropic, Bedrock, LiteLLM, HuggingFace, Google, Snowflake Cortex, each a separate distribution.

Storage. A local SQLite database by default; Postgres and Snowflake are supported alternatives. Results are read through session.get_leaderboard() or a Streamlit dashboard via run_dashboard(session).

The tracing is OpenTelemetry-native, and the README states the consequence: “a trace is portable to any OTLP backend, and evaluations run either as traces land or over a dataset after the fact.” An OTel-native evaluator can feed the observability stack a team already runs rather than asking for a second one.

Judge Quality: Vendor-Published Numbers#

TruLens is the only project here publishing judge-versus-human-annotation figures in its README. They are the vendor’s own claims, with sources:

• 0.81 groundedness F1 on LLM-AggreFact, described as ahead of the fine-tuned Bespoke-MiniCheck-7B on F1, precision and recall over an 11,000-example holdout (Snowflake engineering blog)
• 95% of agent errors caught with Agent GPA on TRAIL/GAIA — 267 of 281 human-annotated errors, against 55% for a baseline trace judge (arXiv:2510.08847)
• 0.93 context-relevance NDCG@5, from a third-party comparison against WandB Weave, RAGAS, DeepEval and UpTrain (AIMultiple, 23 March 2026)

Reported here because judge accuracy is the least-examined variable in this category and TruLens is the only entrant that publishes any, not because the numbers settle anything. Read them as a vendor’s benchmark selection.

What It Cannot Do#

• Evaluate a black box. No instrumentation, no spans, no scores.
• Run as a CLI over a config file. It is a library that lives inside your app.
• Match DeepEval’s catalog breadth. The metric set is smaller and more RAG- and agent-shaped.
• Be adopted without a Python runtime in the loop, which rules it out for the same readers promptfoo serves.

Sources#

• trulens PyPI
• truera/trulens on GitHub
• The RAG Triad
• TruLens quickstart
• Snowflake to Acquire TruEra AI Observability Platform (2024-05-22)

S3: Need-Driven Discovery - Approach#

Research Date: 2026-08-25 Focus: which constraint decides, when every framework can score a RAG answer

What This Pass Asks#

S2 established what each framework needs before it can produce a number. S3 asks which reader is stopped by which requirement.

The pattern across the personas below: the choice is almost never made on metric coverage. Four of the five can score faithfulness. What separates them is whether the team has labels, whether the eval has to finish inside a pull request, whether the prompts may leave the building, and what the application is already written in. Two teams with identical quality goals land on different tools because one of them is regulated and the other is not.

Method#

Each persona is a WHO with a WHY: one constraint that changes the answer. A persona earns its place only if flipping its constraint flips the recommendation, and each file names what the flip would be.

Effort and cost statements are order-of-magnitude planning language derived from the runtime requirements measured in S2 — how many judge calls a design implies, not quotes for a project.

Personas Covered#

1. The team with no labeled data — an app in production, no golden answers
2. The team gating every pull request — the eval is a build step
3. The team debugging a RAG pipeline — retriever or generator, which?
4. The team that cannot send data out — regulated, air-gapped, or contractual
5. The team already on LangChain — the traces exist before the eval does
6. The team facing a security review — a black-box endpoint and a deadline

Scope Boundary#

This is a category survey. No one of these personas is the reason it exists, and none of them gets a committed answer that overrides the others. A reader whose constraints match one can take its recommendation; a reader whose constraints span two will find the positions compose.

Persona: The Team Gating Every Pull Request#

Who: a platform team that has decided quality regressions should fail a build, the way a broken test does. The eval is a required check on every PR.

Why this changes the answer: a build step has three properties no LLM judge provides for free — fast, near-free per run, and the same verdict twice for the same input. That rules out the default design of every framework here and rewards the ones with a documented way out of it.

The Constraint#

Run 200 test cases against 5 judge metrics on every pull request and you have 1,000 inference calls per push, minutes of wall clock, and a threshold that drifts. A flaky gate gets switched off within a month — the failure mode is social, not technical, and it is why most LLM eval programs never become gates.

The requirement is therefore not “best metrics” but “lets me build a suite with no judge in it.”

The Answer#

promptfoo, on one design decision: its assertion catalog is split into a deterministic list and a model-graded list. Staying inside the first — equals, regex, is-json, contains-all, javascript, python, is-valid-function-call, the trajectory:* family — produces a suite with zero judge calls, zero variance, and no cost beyond the target model’s own output. latency and cost are assertions too, so a performance regression fails the same gate as a correctness one.

DeepEval is the close alternative and wins on one axis: its DAG metric composes a deterministic verdict out of discrete decision nodes, reaching judgments a regex cannot while staying reproducible, and the pytest-shaped runner means CI integration is whatever the repository already does with pytest. Choose DeepEval when the codebase is Python and eval should live beside the unit tests; choose promptfoo when the system under test is not Python, or when varying the model provider is part of what the gate checks.

The Design That Works#

• Per-PR: deterministic assertions only. Seconds, pennies, no variance.
• Nightly or per-release: judge metrics, larger set, reported not enforced.

The gate catches regressions a machine can see; the nightly run catches the ones only a model can, and nobody’s pull request is blocked by a number that moved because a judge was in a different mood.

What Would Flip It#

If the gate can be advisory rather than blocking, the constraint disappears and the choice should be made on metric depth instead — DeepEval’s column. And if the team already pays for LangSmith, its offline evaluators run the same experiment shape in CI, at the cost of every build talking to a hosted backend.

Persona: The Team That Cannot Send Data Out#

Who: a team in health care, finance, defense or under a customer contract that forbids sending prompts or completions to a third party. The application runs against a self-hosted or private-endpoint model.

Why this changes the answer: this constraint is decided before any metric is discussed, and it eliminates one entrant outright and one mode of another.

The Constraint#

The eval sees everything the application sees — the user’s question, the retrieved documents, the generated answer — and often more, because a labeled test set is a curated collection of the most sensitive examples the team could find. An eval tool is a data-egress surface, and it is the one people forget to review because it lives in the test directory.

The Answer#

DeepEval, Ragas or TruLens, with a local judge. All three take the judge model as a parameter rather than a dependency:

• DeepEval: deepeval set-ollama --model=..., or an OllamaModel passed to the metric. Confident AI, the commercial platform, is not required.
• Ragas: evaluate(..., llm=evaluator_llm) — the wrapper is yours to construct.
• TruLens: providers are separate distributions; results default to a local SQLite file, with Postgres and Snowflake as alternatives.

LangSmith is out below Enterprise. Datasets are created server-side, experiments run against the backend, and self-hosted and hybrid deployment are Enterprise-plan items. The $39 Plus tier is not an option for this reader at any price; it is a sales conversation or a different tool.

promptfoo is available with one caveat that is easy to miss. Ordinary promptfoo eval calls whatever providers you configured, so a local model stays local. Red teaming does not: adversarial generation runs on api.promptfoo.app by default. The opt-out is PROMPTFOO_DISABLE_REDTEAM_REMOTE_GENERATION=true, and the vendor’s own docs say local generation quality “is generally low for most models” — so this persona gets the scan, at reduced quality, or does not get it.

The Trade#

A local judge is a smaller model. Judge quality is the least-measured variable in this category even for frontier models, and running a 7B model as the grader widens an error bar nobody has quantified. The mitigation is to lean on the non-LLM metrics — Ragas’s BLEU/ROUGE/exact-match group, promptfoo’s deterministic assertions, DeepEval’s DAG — and treat local-judge scores as directional.

What Would Flip It#

An approved private endpoint at a cloud provider — Azure OpenAI under a BAA, Bedrock in-VPC — restores a frontier-quality judge without egress, and every framework here becomes available except LangSmith’s hosted plans. The constraint is where the prompts go, not which company wrote the library.

Persona: The Team Already Running LangChain#

Who: a team whose application is built on LangChain or LangGraph, already emitting traces, with a seat budget and no in-house evaluation practice yet.

Why this changes the answer: for this team the instrumentation cost of the trace-scoring position is already paid, and that is the cost that normally decides against it.

The Constraint#

S2’s boundary — what must exist before a score can be computed — usually counts against LangSmith and TruLens, because instrumenting an application is work. This persona has done it without meaning to. Traces are already flowing in the shape LangSmith reads, so the marginal cost of turning evaluation on is close to zero, and no other framework can match a marginal cost of zero.

The Answer#

LangSmith, and the reason is the online/offline evaluator split rather than the LangChain badge. The same scoring function runs in two modes: offline against a dataset example with its reference output, online against production runs with no reference. One function, written once, gates the release and then watches live traffic. Nothing else in this survey does both with the same code.

Pricing, read 2026-08-26: Developer $0 per seat with up to 5k base traces a month, Plus $39 per seat with up to 10k, then $1.50 per compute unit and $1.00 per storage unit; base traces retained 14 days, extended retention 400 days at extra cost.

The Trade#

LangSmith ships evaluator types — human, code, LLM-as-judge, pairwise — not a metric catalog. A team that wanted Faithfulness off the shelf will write more code here than in DeepEval or Ragas. The common resolution is to use both: DeepEval or Ragas for the metric implementations, LangSmith for the datasets, experiments and production scoring.

And the platform is closed. The MIT-licensed SDK is a client; the product is not auditable and not portable. A team betting its quality process on it is taking a vendor dependency on the same company that ships its framework.

What Would Flip It#

Take away LangChain and the gravity vanishes — LangSmith is framework-agnostic, but on a level field it is competing on merit against four tools that charge nothing per seat. Take away the ability to send data to a hosted backend and it is eliminated below Enterprise regardless of what the application is written in.

Persona: The Team With No Labeled Data#

Who: a team with an LLM feature already in front of users, a corpus of source documents, and not one written-down example of what a correct answer looks like.

Why this changes the answer: half the metrics in this category compare against a reference, and this team has none. Their choice is between metrics that never need one and a tool that will manufacture the references for them.

The Constraint#

Nobody skipped labeling out of carelessness. It means a subject expert writing the ideal answer to a few hundred questions — weeks of the most expensive person’s time, and the answers go stale when the corpus changes. Teams put it off, ship, and then find their eval options are narrower than the feature lists suggested.

Route One: Metrics That Need No Reference#

Faithfulness, answer relevancy and context relevancy score an answer against what was retrieved and asked, never against a golden answer. DeepEval and Ragas both carry the full set; TruLens’s RAG Triad is three of them by construction. Any of the three produces a number this week.

The limit is what those metrics cannot see. A faithful, relevant answer built from the wrong document scores well. Reference-free metrics detect incoherence between the parts of a RAG system; they do not detect that the system as a whole is answering the wrong question.

Route Two: Generate the Labels#

Ragas is the only framework in this survey that manufactures an evaluation set from a document corpus. It builds a knowledge graph — chunks documents into nodes, extracts entities and keyphrases, links nodes that share them — then walks the graph with query synthesizers to produce single-hop and multi-hop questions varied by length, style and persona, each with a reference answer. DeepEval ships a synthesizer too and is the fallback.

The Trade#

Generated references are model-written, so a generated test set measures consistency with a generator rather than truth. It is a floor, not ground truth, and the practice is to have an expert review a sample — which converts weeks of authoring into an afternoon of checking.

What Would Flip It#

Give this team a few hundred expert-written pairs and the answer changes to whichever framework their language and CI story favors, because reference-based metrics — factual correctness, contextual recall — become available and are cheaper and steadier than judge-only scoring. The absence of labels is the whole constraint; supply them and this persona dissolves into one of the others.

One caution specific to this recommendation: as of 2026-08-26 the Ragas repository has not merged a pull request since 2026-02-24 and has 216 open. The generator works. Nobody is currently fixing it if it stops.

Persona: The Team Debugging a RAG Pipeline#

Who: a team whose retrieval-augmented system returns confident, wrong answers often enough to matter, and who cannot tell whether the retriever is fetching the wrong documents or the generator is ignoring the right ones.

Why this changes the answer: one aggregate quality score cannot separate those two failures. This persona needs metrics that decompose along the pipeline, and one framework is built around that decomposition.

The Constraint#

A RAG system has two places to fail and they present identically to a user. The diagnostic question is structural: did the right context arrive, and did the answer use it? Any metric that collapses the pipeline into one number throws away the bit of information this team needs.

The Answer#

TruLens’s RAG Triad is the framing, whoever you get it from:

Grounded but irrelevant means retrieval; relevant but ungrounded means generation; all three high with a bad answer means the corpus does not contain the answer at all.

TruLens has a second advantage here: selectors. A metric points at a span in the trace, so the retriever’s output is scored where it is produced, before the generator sees it, without plumbing that value out by hand. For a pipeline with more than two stages that is the difference between instrumenting once and rewriting the app to expose intermediates.

Ragas goes deeper on retrieval specifically — context precision, context recall, context entities recall and noise sensitivity are finer instruments on that edge than a single relevance number. Take it when the retriever is the confirmed suspect. DeepEval carries the same RAG metrics and returns a written reason with every score, which is the shortest path from a red number to a changed chunking strategy.

The Trade#

All three need a judge for the triad metrics: a per-query cost and scores that move between runs. During debugging that is acceptable — you are looking for a pattern across many queries, not gating a release on one — which is why this persona and the CI persona reach opposite conclusions from the same catalog.

What Would Flip It#

If the failure turns out to be the chunking rather than retrieval or generation, none of these tools diagnoses it directly and the work moves upstream into ingestion. And if the team cannot instrument its application, TruLens drops out and the choice narrows to scoring captured transcripts with Ragas or DeepEval.

Persona: The Team Facing a Security Review#

Who: a team shipping an AI feature that has to clear an application-security review. What they own is a deployed endpoint. The reviewer wants evidence of adversarial testing, with a report, by a date.

Why this changes the answer: this persona cannot supply what four of the five frameworks require. They have no transcripts, no instrumentation, and often no Python in the loop — the system under test is a URL.

The Constraint#

Everything about this reader is black box. DeepEval and Ragas want captured inputs and outputs they do not have; TruLens wants to run inside a process they may not own; the application might be Go or Java or a vendor’s product. And the deliverable is not a metric — it is a report a reviewer will read.

The Answer#

promptfoo, on two counts. It is the only framework here that calls the target itself, through an HTTP provider or a custom script, which makes the implementation language of the system under test irrelevant. And red teaming is a first-class mode with a lifecycle built for exactly this deliverable: promptfoo redteam init to scaffold, promptfoo redteam run to generate and execute the adversarial cases, promptfoo redteam report to produce the artifact. Plugins cover harmful content, broken object-level and function-level authorization, competitor endorsement and more; strategies decide how payloads are delivered.

deepteam is the alternative for a Python shop that already runs DeepEval — a separate Apache-2.0 package (PyPI 1.0.9, 2026-08-12) whose docs describe it as “powered by deepeval”, covering bias, toxicity, PII leakage and misinformation with 10+ attack methods including prompt injection and jailbreaking. It expects to call your code rather than a URL, which is why promptfoo wins the black-box case and deepteam wins the in-repo one.

The Trade#

The default scan sends data off the machine: adversarial generation runs on api.promptfoo.app unless disabled, and the promptfoo docs note that corporate firewalls often block it because the service generates harmful content on purpose. Sorting that out with IT is part of the schedule, not a footnote.

A red team report is also evidence of testing, not evidence of safety. It says which known attack classes were tried and what happened.

What Would Flip It#

If the team owns the code and works in Python, deepteam keeps the scan next to the existing eval suite and the two share metric definitions. If the reviewer’s requirement is a named standard rather than a scan, this stops being a tool choice and becomes a compliance exercise no framework here completes.

S3 Recommendation: Choose By Constraint, Not By Capability#

Across six personas, the framework with the largest metric catalog is the first choice for one of them. That is the finding. In this category the decision is forced by what the team has and what it is allowed to do, not by what the tools can compute.

The Table#

The Two Questions That Cover Most Cases#

What do you have in hand? Transcripts point at DeepEval or Ragas. An endpoint points at promptfoo. An instrumented app points at TruLens. Existing traces point at LangSmith. This question is answerable in a minute and eliminates most of the field before anyone opens a metric catalog.

Where are the prompts allowed to go? If the answer is “nowhere”, LangSmith is out below Enterprise and promptfoo’s red team runs degraded. Every other path supports a local judge. Teams discover this after choosing a tool more often than before, and it is the one constraint that cannot be worked around later.

The Question to Ask First#

Is this eval going to block a release?

A gate and a diagnostic want opposite things from the same catalog. A gate needs determinism and near-zero cost, which means deterministic assertions and no judge. A diagnostic needs decomposition and explanation, which means judge metrics and their cost and variance. Teams that ask this on day one build two tiers — a fast deterministic gate per PR, a judge-based suite nightly — and the gate survives. Teams that do not, build one judge-driven suite, watch it flake, and switch it off within a quarter.

One Maintenance Note Attached to Two Recommendations#

Ragas is the answer for the no-labels persona and a strong second for RAG debugging. As of 2026-08-26 its repository has merged nothing since 2026-02-24, has 216 open pull requests, and last released 0.4.3 on 2026-01-13, after moving from explodinggradients to vibrantlabsai. The code works. A reader choosing it should choose it knowing that a bug they hit is a bug they carry.

What This Pass Does Not Do#

It does not pick for any particular product, and it does not rank the five. Each persona above is a constraint with an answer attached. A reader whose constraints span two will find that the positions compose — promptfoo gating pull requests while TruLens scores production is one system, not two competing choices.

S4: Strategic Selection - Approach#

Research Date: 2026-08-25 Registry, repository and funding figures: read 2026-08-26 UTC Focus: what a choice in this category costs to reverse

What This Pass Asks#

S2 asked what each framework needs before it can score. S3 asked which reader is stopped by which requirement. S4 asks the question that outlives both:

In three years, what does it cost to undo this?

Three years, not five, because this survey carries decay_class: fast. The model layer under every one of these tools reprices and re-versions on a monthly cycle, and two of the five frameworks here did not exist in their current shape three years ago. A ten-year outlook in this category would be fiction.

How It Was Scored#

Lock-in is scored by what must be rebuilt, not by what is lost. The second measurement flatters incumbents — every switch loses features, and that tells a reader nothing about whether they can afford it.

The axis comes from S2’s organizing idea, turned around:

The thing you must supply before a score exists is the thing you must rebuild when you leave.

A test-case corpus is data you own, in a shape you chose. Instrumentation is source code inside your application, written to one library’s decorators. A server-side experiment history is neither — it is yours and it is not where you can reach it. Those three are not the same size of problem, and the position a framework occupies predicts which one you are buying.

Each dimension below is scored low / medium / high, with the rebuild named. A score with no named rebuild is an opinion.

Durability Method#

Five signals, all checked against primary sources on 2026-08-26 UTC: release cadence over twelve months (PyPI, npm); last release and last push, and the gap between them; contributor count, as a measure of how much of a project survives its maintainers; merge behavior, which is the earliest visible sign that attention has moved; and ownership and money.

On the last: funding answers will this survive, and what happens to me if the owner changes course. It never answers is this good. A large round is a durability signal and a monetization expectation at once, and both halves belong to a reader deciding what to depend on. Nothing below ranks these companies against each other.

Structure of This Pass#

• lock-in.md — the four dimensions, scored, with the rebuild named
• vendor-durability.md — the five signals, per framework
• exit-strategy.md — what leaving each position costs, and who pays it
• recommendation.md — three strategic paths, and the re-check trigger

What This Pass Does Not Do#

It does not rank the five, and it does not name a winner. A survey that arrives at one answer in S4 has spent three passes establishing that the category has several, and then thrown that away on the last page.

Exit Strategy: What Leaving Costs, and Who Pays#

Lock-in scores say how big the rebuild is. This says who does it, because a cost that lands on the application team is a different decision from one that lands on the person who owns the eval suite.

DeepEval ⇄ Ragas#

Cost: days. Paid by: the eval owner.

The corpus is the same four columns under different names, so the data moves by rename. What has to be redone is metric wiring and every threshold, because a 0.8 from one judge prompt is not a 0.8 from another. A team that ports the data and keeps the numbers has completed the migration and broken the gate.

Leaving Ragas splits into two cases. Used for metrics, DeepEval covers the RAG set and this is the cheap exit above. Used for the knowledge-graph test-set generator, nothing else in this survey builds an evaluation set from a document corpus, and DeepEval’s synthesizer is a different mechanism. The corpus Ragas already generated stays yours either way — it is data, not a runtime dependency, which is why this exit is survivable with the upstream stalled.

promptfoo#

Cost: days for evals, months for red teaming. Paid by: the eval owner, then the security owner.

Prompts, targets and expected values port to anything. The assertion vocabulary does not — contains-all, is-valid-function-call and the trajectory:* family are promptfoo’s shapes. Deterministic assertions re-express as ordinary test code cheaply, and llm-rubric or g-eval re-express as DeepEval’s G-Eval with recalibration.

The red team has no like-for-like replacement — the one exit in this category with a hole in it. Its plugin and strategy catalog has a single partial alternative in deepteam, which expects to call your code rather than a URL. A team that chose promptfoo because the system under test is a black-box endpoint has no black-box alternative to move to.

TruLens#

Cost: a source change across the application. Paid by: the application team.

The decorators are in application code, which means the exit is a pull request against the product, reviewed by people who did not choose the eval tool and do not benefit from changing it. That organizational fact is the cost, more than the diff size.

Two exits at very different prices. Moving to another OpenTelemetry consumer keeps the decorators — the README’s claim is that a trace is “portable to any OTLP backend” — and rewrites only the selector-and-metric layer. Moving to a test-case framework removes the instrumentation, returns nothing, and gives up the capability that justified it: scoring an intermediate span without plumbing the value out by hand.

LangSmith#

Cost: days for the port, permanent for the baseline. Paid by: the eval owner, and then everyone.

The exit is better provisioned than its reputation suggests. Datasets export to CSV, JSONL or OpenAI fine-tuning format from the UI, and list_examples / listExamples reads them through the SDK. Evaluator functions are your own code in your own repository. Neither is trapped.

Two things do not leave. The experiment history — the scored runs that are your regression baseline and your evidence that quality moved — has no export target, because no other framework here ingests another’s history. And online evaluation against production traffic has no offline equivalent: DeepEval, Ragas and promptfoo are all offline harnesses. A team whose production quality signal is a LangSmith online evaluator is rebuilding that capability elsewhere or dropping it.

The Exit Nobody Plans For#

Changing the judge model. It costs nothing, requires no migration, is done by one person in one line, and resets the score history exactly as a framework switch would. Scores are comparable only within a judge; the tooling will happily plot a continuous series straight across the discontinuity.

Every framework here makes the judge a parameter, which is the right design and why this happens without ceremony. The mitigation is a convention rather than a tool: pin the judge model and version beside the thresholds, treat a judge change as a baseline reset, and re-run a held-out slice under both before believing any trend that crosses it.

Exit Costs at a Glance#

Lock-In: What You Must Rebuild#

Four dimensions. Each is scored by the size of the rebuild, and the rebuild is named — a score without one is an opinion.

The Summary#

Dimension 1: The Corpus — and the Rename That Is Not a Migration#

The most reassuring finding in this pass. DeepEval and Ragas describe the same four columns under different names.

A corpus assembled for either one is a corpus assembled for both. Moving it is a column rename, and any team that stores its evaluation set in its own schema — a CSV, a table, a JSONL file — rather than in a framework’s object model has already paid the whole cost of switching between the two most widely recommended frameworks in this survey.

That is the cheapest insurance in this category, it costs nothing, and it has to be bought on day one — a corpus that only ever existed as LLMTestCase constructor calls is locked to a library for no benefit.

What does not travel with it: thresholds. A faithfulness score of 0.8 from one framework is not 0.8 from the other, because the judge prompt behind it differs. Every gate must be recalibrated after a switch, and a team that migrates the data and keeps the numbers has migrated nothing.

Dimension 2: Instrumentation — Lock-In That Lives in the Wrong Repository#

TruLens’s @instrument decorators sit in application source, on the functions that do the retrieving and generating. That places the lock-in outside the evaluation directory, and it changes who has to approve removing it: the application team, in a source change reviewed like any other, not the eval owner in an afternoon.

This is the expensive kind of lock-in, and the reason TruLens scores highest here despite being MIT-licensed with no vendor in the path. A license governs what you may do; instrumentation governs how many people have to agree.

The OpenTelemetry design defuses about half of it. TruLens’s README states that a trace is “portable to any OTLP backend”. A team leaving TruLens for another OTel consumer keeps the decorators and rewrites only the selector and metric layer. A team leaving instrumentation altogether removes the decorators and gets nothing back. Which of those two exits you are taking decides whether this dimension reads medium or high.

LangSmith reaches the same position by a different route and pays less for it: a LangChain or LangGraph application is already emitting traces for reasons unrelated to evaluation, so the instrumentation was never an evaluation cost and does not become one on the way out.

Dimension 3: History — The Asset Nobody Exports#

LangSmith’s lock-in is not the dataset. Datasets come out: the SDK exposes list_examples / listExamples, and the documentation states “You can export your LangSmith dataset to a CSV, JSONL, or OpenAI’s fine tuning format from the LangSmith UI.” Evaluator functions are ordinary code in your own repository.

What does not come out is the experiment record — months of scored runs that constitute your baseline, your regression history and your evidence that quality moved. Leaving restarts that at zero. No framework here exports another’s history, and none of them treats history as an interchange format.

The second non-portable item is online evaluation: a scoring function running against production traffic with no reference output has no equivalent in DeepEval, Ragas or promptfoo, which are offline harnesses. That capability is rebuilt or dropped, never migrated.

Dimension 4: The Judge — Not a Lock-In, but a Trap#

All four open-source frameworks take the judge as a parameter rather than a dependency: DeepEval accepts an OllamaModel or a CLI-configured judge, Ragas takes llm= on evaluate(), TruLens ships providers as separate distributions. Switching frameworks does not force switching judges, and switching judges does not force switching frameworks. This dimension is low for everyone and can be left out of the decision.

The trap is adjacent. Changing the judge invalidates the score history as surely as changing the framework does. Scores are comparable only within a judge; a team that upgrades to a newer model resets its baseline and does not notice, because the tooling plots a continuous series across the discontinuity. This happens more often than a framework migration does.

What Is Not a Lock-In Dimension#

License. Four of the five are MIT or Apache-2.0; the fifth’s client SDK is MIT while its product is closed. S2 covered that, and it is a data-residency question rather than a rebuild question. No license here restricts what you may do with a corpus you assembled yourself.

Metric coverage. Every catalog gap here is a judge prompt away from being filled — G-Eval, llm-rubric and TruLens custom metrics all take a criterion in prose. A missing metric is an afternoon, not a migration.

S4 Recommendation: Three Paths, Priced by What They Cost to Undo#

S3 chose by constraint. S4 asks what those choices commit you to, and the answers do not line up with the capability ordering. The framework with the deepest retrieval catalog has the least maintained upstream. The cheapest exit belongs to the framework whose closest competitor already reads its data format. The most expensive lock-in belongs to an MIT-licensed library with no vendor in the path.

Path 1: Conservative — Minimize What You Must Rebuild#

Keep the corpus in your own schema, in the four columns both DeepEval and Ragas read. A CSV, a table, a JSONL file — anywhere that is not a framework’s constructor calls. Run judge metrics from whichever framework fits the language and the CI story. Do not instrument the application for evaluation.

What it buys: the ability to change your mind for a column rename and a threshold recalibration. What it gives up: intermediate-span scoring and production evaluation, both of which need the position you declined.

Durability-adjusted pick: DeepEval — Apache-2.0, 306 contributors, 79 releases in twelve months, a judge you can point at a local server. The small seed round behind it is a signal, and the license plus contributor base is the answer to it.

Path 2: Capability-First — Buy What Nothing Else Does#

Take the position whose lock-in you are choosing on purpose. Instrument with TruLens if you need a retriever scored where it is produced. Adopt LangSmith if you need one scoring function that gates a release and then watches production.

What it costs: for TruLens, a rebuild landing on the application team rather than the eval owner, half-defused by OpenTelemetry. For LangSmith, an experiment history with no export target and an online-evaluation capability with no offline equivalent, on top of pricing where the metering matters more than the seat.

Durability-adjusted note: neither owner is going away on a three-year horizon. The risks differ in kind — reprioritization inside a public company for TruLens, monetization pressure at a $1.25B valuation for LangSmith — and neither is the risk that the software stops working.

Path 3: Adaptive — Two Tiers, Two Positions#

Deterministic assertions gating every pull request; judge metrics nightly against a portable corpus. promptfoo or DeepEval’s DAG for the first tier, DeepEval or Ragas for the second.

S3 arrived at this from the CI persona; S4 finds it is also the lowest-lock-in configuration available — the gate holds prompts and expected values that port anywhere, the nightly tier holds a corpus two frameworks already read.

The caveat: if promptfoo is both your gate and your red team, those are two dependencies wearing one name and only one has a replacement. Price them separately.

Where the Ragas Finding Lands#

Ragas answers two of S3’s six personas and has merged nothing since 2026-02-24, with 216 pull requests waiting. This does not become “avoid it” — the test-set generator has no substitute here, and a generated corpus is data you keep whatever happens upstream.

It becomes: use it for what it produces rather than depend on it for what it runs. Generate the evaluation set, take the corpus, and put the recurring metric load somewhere that is merging patches. That converts a maintenance risk into a one-time transaction, the correct shape for a tool whose maintainers have started a different company.

The Re-Check Trigger#

This survey carries decay_class: fast and the figures above have a short half-life. Two checks, both cheap:

1. Inbound contributions versus merges, per dependency, once a quarter. It identified the Ragas change months before anything else would have, and it is one GitHub search.
2. Judge model and version, pinned beside the thresholds. Not a vendor check — a discipline. A judge upgrade resets your baseline as completely as a framework migration and announces itself to nobody.

What S4 Did Not Settle#

Whether any of these judges are accurate — S2 left it open and S4 cannot close it from durability signals. The gate a team ships behind has an error bar nobody has published a neutral measurement of. Until someone does, every recommendation here is about tools rather than about truth, and a reader should hold the scores accordingly.

Vendor Durability#

Five signals, read from primary sources on 2026-08-26 UTC. They answer will this survive, and what happens to me if the owner changes course. They are not an assessment of whether the software is good — S2 did that, on the software.

The Signals#

Four rows describe healthy projects with different tempos. The fifth describes something else, and it is the most decision-relevant fact in this survey.

Ragas: Attention Moved, and It Is Datable#

The raw signals: last release 0.4.3 on 2026-01-13. Last commit on the default branch 2026-02-24. Zero commits in the ninety days to 2026-08-26. 216 open pull requests, with new ones still arriving as recently as 2026-08-25, and a GitHub search for pull requests merged after 2026-02-24 returns zero.

Contributions are still being offered; nothing is accepting them. That gap is the earliest visible sign that maintainer attention has left, and it shows months before a repository is archived or a README says so.

The cause is on the record, and the timing lines up to the day:

• Ragas was created by Shahul ES and Jithin James, who founded Vibrant Labs. The repository moved from explodinggradients/ragas to vibrantlabsai/ragas; PyPI dates the move between release 0.3.9 (2025-11-11, old URLs) and 0.4.0 (2025-12-03, new URLs).
• Heavybit announced its investment in Vibrant Labs on 2025-12-03 — the same day 0.4.0 shipped. The announcement describes the company as building “production-grade, RL-ready simulation” environments for long-horizon agents, and quotes the founders’ intent to “take the hard lessons from shipped evaluator tooling and build complete, instrumented, RL-ready environments.” It calls Vibrant Labs “the natural continuation of that journey.”
• Vibrant Labs’ own about page lists Ragas among what the founders co-created — background rather than current work. The same page describes Ragas as “the most popular evals framework for AI applications” (the company’s own claim about its own prior project, not independently checked here).

What this means for a reader, and what it does not. It does not mean Ragas is bad software. It still has the deepest retrieval-metric catalog in this survey and the only test-set generator that builds an evaluation set from a document corpus, and those capabilities work exactly as they did in January. It means the maintainers are building something else, so a bug you hit is a bug you carry, work around, or fork.

The fork case is unusually strong here, and that is the counterweight: Apache-2.0, 240 contributors, and 216 pull requests already written and waiting. The raw material for a community continuation is sitting in the queue. Whether anyone picks it up is not something this survey can predict; that it could be is a fact about the license and the contributor base.

Money, Read as Runway and as Pressure#

LangChain Inc raised, in its own announcement dated 2025-10-20, “$125M at a $1.25B valuation” led by IVP. That buys years of runway for LangSmith, and it sets a monetization expectation that a dependent should plan around rather than hope against. The shape it has already taken is visible in the pricing S2 recorded: self-hosted and hybrid deployment are Enterprise-only, and usage is metered per compute unit and per storage unit above the included traces. A team budgeting the $39 seat price and not the metering has mispriced the dependency.

promptfoo Inc announced “$18.4 million” in Series A led by Insight Partners with Andreessen Horowitz participating, on 2025-07-29. The same announcement says of the open-source tools: “Free forever.” That is a stated commitment rather than a license term, and a reader should weigh it as such — the MIT license on the CLI is the enforceable part. The commercial product is an enterprise AI-security platform, and the piece of the free tool that touches it is the hosted red team generation service. If commercial pressure ever reshapes the free tier, that is where it would appear first.

That announcement claimed “125,000+ developers have downloaded our open source tools”; the vendor’s site claimed “300,000+ developers” when read on 2026-08-26. Both are vendor figures, neither independently verifiable; the direction is the usable part.

Confident AI, which maintains DeepEval, describes closing an “oversubscribed $2.2m seed round” with Y Combinator, Flex Capital and others, signed the Monday before a demo day on March 12 (Y Combinator Winter 2025 batch). That is the smallest cushion in this table funding the joint-fastest release cadence — an early company shipping hard. The mitigation is structural rather than financial: DeepEval is Apache-2.0 with 306 contributors, so the library outlives the company’s plans whatever those turn out to be.

Snowflake owns TruLens, having announced the TruEra acquisition on 2024-05-22 — a different risk shape from the other four: not runway, but priority. An acquired open-source asset inside a public company is exposed to reprioritization rather than to running out of money. The counter-evidence is concrete: MIT, 21 releases in twelve months, 95 commits in ninety days, and a Snowflake-specific integration path that gives the owner a reason to keep it current.

The Signal Worth Copying#

What identified Ragas’s change of state was not stars, downloads, the README, or the last release date on its own — a project can go months between releases and be healthy. It was the ratio of inbound contributions to merges. Contributions arriving and nothing merging is the cheapest early-warning check available on any dependency, it takes one GitHub search, and it fires long before anything else does.