Skip to main content

Answer Relevancy

The answer relevancy metric measures the quality of your RAG pipeline's generator by evaluating how relevant the actual_output of your LLM application is compared to the provided input. deepeval's answer relevancy metric is a self-explaining LLM-Eval, meaning it outputs a reason for its metric score.

tip

Here is a detailed guide on RAG evaluation, which we highly recommend as it explains everything about deepeval's RAG metrics.

Required Arguments​

To use the AnswerRelevancyMetric, you'll have to provide the following arguments when creating an LLMTestCase:

  • input
  • actual_output

Example​

from deepeval import evaluate
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

# Replace this with the actual output from your LLM application
actual_output = "We offer a 30-day full refund at no extra cost."

metric = AnswerRelevancyMetric(
    threshold=0.7,
    model="gpt-4",
    include_reason=True
)
test_case = LLMTestCase(
    input="What if these shoes don't fit?",
    actual_output=actual_output
)

metric.measure(test_case)
print(metric.score)
print(metric.reason)

There are SEVEN optional parameters when creating an AnswerRelevancyMetric:

  • [Optional] threshold: a float representing the minimum passing threshold, defaulted to 0.5.
  • [Optional] model: a string specifying which of OpenAI's GPT models to use, OR any custom LLM model of type DeepEvalBaseLLM. Defaulted to 'gpt-4o'.
  • [Optional] include_reason: a boolean which when set to True, will include a reason for its evaluation score. Defaulted to True.
  • [Optional] strict_mode: a boolean which when set to True, enforces a binary metric score: 1 for perfection, 0 otherwise. It also overrides the current threshold and sets it to 1. Defaulted to False.
  • [Optional] async_mode: a boolean which when set to True, enables concurrent execution within the measure() method. Defaulted to True.
  • [Optional] verbose_mode: a boolean which when set to True, prints the intermediate steps used to calculate said metric to the console, as outlined in the How Is It Calculated section. Defaulted to False.
  • [Optional] evaluation_template: a class of type AnswerRelevancyTemplate, which allows you to override the default prompt templates used to compute the AnswerRelevancyMetric score. You can learn what the default prompts looks like here, and should read the How Is It Calculated section below to understand how you can tailor it to your needs. Defaulted to deepeval's AnswerRelevancyTemplate.
info

If you are looking to generate a comprehensive evaluation report for your test_case or run multiple metrics on a single test case, use the evaluate function.

...

evaluate([test_case], [metric])

How Is It Calculated?​

The AnswerRelevancyMetric score is calculated according to the following equation:

Answer Relevancy=Number of Relevant StatementsTotal Number of Statements\text{Answer Relevancy} = \frac{\text{Number of Relevant Statements}}{\text{Total Number of Statements}}

The AnswerRelevancyMetric first uses an LLM to extract all statements made in the actual_output, before using the same LLM to classify whether each statement is relevant to the input.

tip

You can set the verbose_mode of ANY deepeval metric to True to debug the measure() method:

...

metric = AnswerRelevancyMetric(verbose_mode=True)
metric.measure(test_case)