> ## Content Index
> Fetch the complete content index at: https://stalin-pereira.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Implementing Unit Tests for LLM Outputs using DeepEval
- URL: https://stalin-pereira.com/implementing-unit-tests-for-llm-outputs-using-deepeval/
- Published: 2026-01-02T10:37:00.000Z
- Updated: 2026-10-16T19:08:16.000Z
- Description: Systematic testing of LLM outputs with DeepEval/Ragas
- Author: Stalin Pereira
- Tags: AI Engineering

**You Can't 'Vibe Check' Your Way to Production**

Why manual testing is killing your AI product's reliability.

## The 'Essay Grading' Analogy

- **The Vibe Check** — Glancing at a student's essay and saying 'Looks long enough, A+'.
- **The Reality** — The grammar is fine, but the facts are completely made up.
- **The Business Risk** — Your chatbot sounds confident while lying to your customers.

## Enter: Unit Tests for AI

Moving from 'gut feelings' to quantifiable metrics.

- **The Challenge** — Traditional tests verify exact matches (2+2=4). LLMs vary every time.
- **The Solution** — Frameworks like DeepEval treat 'meaning' as the testable unit.

## How It Works: LLM-as-a-Judge

Using a smarter model to grade the junior model.

- **The Student** — Your production chatbot generates an answer.
- **The Teacher** — A powerful model (like GPT-4) evaluates that answer against a rubric.
- **The Score** — It returns a score (0.0 to 1.0) and a reason for failure.

## Metric 1: Faithfulness (Hallucinations)

Is your bot making things up?

- **The Check** — Does the generated answer contradict the retrieved context?
- **Example Failure** — Context: 'We open at 9AM.' Bot: 'Come in at 8AM!'

## Metric 2: Answer Relevance

Is your bot actually listening?

- **The Check** — Does the response directly address the user's specific question?
- **Example Failure** — User: 'Reset password.' Bot: 'Here is our pricing page.'

## The Old Way vs. The New Way

- **Manual Testing (Old)** — Spreadsheets, 3 PMs reading logs, subjective 'thumbs up'.
- **DeepEval (New)** — Automated pipelines that run every time you push code.
- **The Speed** — Feedback in minutes, not days.

## Testing RAG Systems (Two Layers)

Retrieval Augmented Generation requires double validation.

- **Retrieval Score** — Did we find the right documents in the database?
- **Generation Score** — Did the LLM write a good answer based on those documents?

## The 'Code' Concept

It looks just like a normal Python unit test.

- **Define Metric** — metric = FaithfulnessMetric(threshold=0.7)
- **Run Assertion** — assert\_test(test\_case, \[metric\])
- **CI Integration** — Fails the build pipeline if the LLM becomes toxic or inaccurate.

## Why Business Stakeholders Care

It's not just tech debt; it's brand safety.

- **Risk Mitigation** — Prevent PR disasters from rogue bot responses.
- **Cost Control** — Catch bad prompts before they waste tokens in production.
- **Confidence** — Deploy on Fridays because you have the data.

## The Takeaway

Evals are the new Unit Tests.

> Treat prompts like code. Test them before you ship them.

---

How are you validating your AI models today?