Evaluation

We have modules for both LLM-based evaluation and retrieval-based evaluation.

Evaluation modules.

class llama_index.evaluation.BaseEvaluator

Base Evaluator class.

abstract async aevaluate(query: Optional[str] = None, response: Optional[str] = None, contexts: Optional[Sequence[str]] = None, **kwargs: Any) → EvaluationResult

Run evaluation with query string, retrieved contexts, and generated response string.

Subclasses can override this method to provide custom evaluation logic and take in additional arguments.

async aevaluate_response(query: Optional[str] = None, response: Optional[Response] = None, **kwargs: Any) → EvaluationResult

Run evaluation with query string and generated Response object.

Subclasses can override this method to provide custom evaluation logic and take in additional arguments.

evaluate(query: Optional[str] = None, response: Optional[str] = None, contexts: Optional[Sequence[str]] = None, **kwargs: Any) → EvaluationResult

Run evaluation with query string, retrieved contexts, and generated response string.

Subclasses can override this method to provide custom evaluation logic and take in additional arguments.

evaluate_response(query: Optional[str] = None, response: Optional[Response] = None, **kwargs: Any) → EvaluationResult

Run evaluation with query string and generated Response object.

Subclasses can override this method to provide custom evaluation logic and take in additional arguments.

get_prompts() → Dict[str, BasePromptTemplate]: Get a prompt.

update_prompts(prompts_dict: Dict[str, BasePromptTemplate]) → None

Update prompts.

Other prompts will remain in place.

pydantic model llama_index.evaluation.BaseRetrievalEvaluator

Base Retrieval Evaluator class.

Show JSON schema

{
   "title": "BaseRetrievalEvaluator",
   "description": "Base Retrieval Evaluator class.",
   "type": "object",
   "properties": {
      "metrics": {
         "title": "Metrics"
      }
   }
}

Config

arbitrary_types_allowed: bool = True

Fields

metrics (List[llama_index.evaluation.retrieval.metrics_base.BaseRetrievalMetric])

field metrics: List[BaseRetrievalMetric] [Required]: List of metrics to evaluate

async aevaluate(query: str, expected_ids: List[str], mode: RetrievalEvalMode = RetrievalEvalMode.TEXT, **kwargs: Any) → RetrievalEvalResult

Run evaluation with query string, retrieved contexts, and generated response string.

Subclasses can override this method to provide custom evaluation logic and take in additional arguments.

async aevaluate_dataset(dataset: EmbeddingQAFinetuneDataset, workers: int = 2, show_progress: bool = False, **kwargs: Any) → List[RetrievalEvalResult]: Run evaluation with dataset.

classmethod construct(_fields_set: Optional[SetStr] = None, **values: Any) → Model: Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data. Default values are respected, but no other validation is performed. Behaves as if Config.extra = ‘allow’ was set since it adds all passed values

copy(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, update: Optional[DictStrAny] = None, deep: bool = False) → Model

Duplicate a model, optionally choose which fields to include, exclude and change.

Parameters

include – fields to include in new model
exclude – fields to exclude from new model, as with values this takes precedence over include
update – values to change/add in the new model. Note: the data is not validated before creating the new model: you should trust this data
deep – set to True to make a deep copy of the model

Returns

new model instance

dict(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, by_alias: bool = False, skip_defaults: Optional[bool] = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False) → DictStrAny: Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.

evaluate(query: str, expected_ids: List[str], mode: RetrievalEvalMode = RetrievalEvalMode.TEXT, **kwargs: Any) → RetrievalEvalResult

Run evaluation results with query string and expected ids.

Parameters

query (str) – Query string
expected_ids (List[str]) – Expected ids

Returns

Evaluation result

Return type

RetrievalEvalResult

classmethod from_metric_names(metric_names: List[str], **kwargs: Any) → BaseRetrievalEvaluator

Create evaluator from metric names.

Parameters

metric_names (List[str]) – List of metric names
**kwargs – Additional arguments for the evaluator

classmethod from_orm(obj: Any) → Model

json(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, by_alias: bool = False, skip_defaults: Optional[bool] = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False, encoder: Optional[Callable[[Any], Any]] = None, models_as_dict: bool = True, **dumps_kwargs: Any) → unicode

Generate a JSON representation of the model, include and exclude arguments as per dict().

encoder is an optional function to supply as default to json.dumps(), other arguments as per json.dumps().

classmethod parse_file(path: Union[str, Path], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model

classmethod parse_obj(obj: Any) → Model

classmethod parse_raw(b: Union[str, bytes], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model

classmethod schema(by_alias: bool = True, ref_template: unicode = '#/definitions/{model}') → DictStrAny

classmethod schema_json(*, by_alias: bool = True, ref_template: unicode = '#/definitions/{model}', **dumps_kwargs: Any) → unicode

classmethod update_forward_refs(**localns: Any) → None: Try to update ForwardRefs on fields based on this Model, globalns and localns.

classmethod validate(value: Any) → Model

class llama_index.evaluation.BatchEvalRunner(evaluators: Dict[str, BaseEvaluator], workers: int = 2, show_progress: bool = False)

Batch evaluation runner.

Parameters

evaluators (Dict[str, BaseEvaluator]) – Dictionary of evaluators.
workers (int) – Number of workers to use for parallelization. Defaults to 2.
show_progress (bool) – Whether to show progress bars. Defaults to False.

async aevaluate_queries(query_engine: BaseQueryEngine, queries: Optional[List[str]] = None, **eval_kwargs_lists: Dict[str, Any]) → Dict[str, List[EvaluationResult]]

Evaluate queries.

Parameters

query_engine (BaseQueryEngine) – Query engine.
queries (Optional[List[str]]) – List of query strings. Defaults to None.
**eval_kwargs_lists (Dict[str, Any]) – Dict of lists of kwargs to pass to evaluator. Defaults to None.

async aevaluate_response_strs(queries: Optional[List[str]] = None, response_strs: Optional[List[str]] = None, contexts_list: Optional[List[List[str]]] = None, **eval_kwargs_lists: List) → Dict[str, List[EvaluationResult]]

Evaluate query, response pairs.

This evaluates queries, responses, contexts as string inputs. Can supply additional kwargs to the evaluator in eval_kwargs_lists.

Parameters

queries (Optional[List[str]]) – List of query strings. Defaults to None.
response_strs (Optional[List[str]]) – List of response strings. Defaults to None.
contexts_list (Optional[List[List[str]]]) – List of context lists. Defaults to None.
**eval_kwargs_lists (Dict[str, Any]) – Dict of lists of kwargs to pass to evaluator. Defaults to None.

async aevaluate_responses(queries: Optional[List[str]] = None, responses: Optional[List[Response]] = None, **eval_kwargs_lists: Dict[str, Any]) → Dict[str, List[EvaluationResult]]

Evaluate query, response pairs.

This evaluates queries and response objects.

Parameters

queries (Optional[List[str]]) – List of query strings. Defaults to None.
responses (Optional[List[Response]]) – List of response objects. Defaults to None.
**eval_kwargs_lists (Dict[str, Any]) – Dict of lists of kwargs to pass to evaluator. Defaults to None.

evaluate_queries(query_engine: BaseQueryEngine, queries: Optional[List[str]] = None, **eval_kwargs_lists: Dict[str, Any]) → Dict[str, List[EvaluationResult]]

Evaluate queries.

Sync version of aevaluate_queries.

evaluate_response_strs(queries: Optional[List[str]] = None, response_strs: Optional[List[str]] = None, contexts_list: Optional[List[List[str]]] = None, **eval_kwargs_lists: List) → Dict[str, List[EvaluationResult]]

Evaluate query, response pairs.

Sync version of aevaluate_response_strs.

evaluate_responses(queries: Optional[List[str]] = None, responses: Optional[List[Response]] = None, **eval_kwargs_lists: Dict[str, Any]) → Dict[str, List[EvaluationResult]]

Evaluate query, response objs.

Sync version of aevaluate_responses.

class llama_index.evaluation.CorrectnessEvaluator(service_context: Optional[ServiceContext] = None, eval_template: Optional[Union[BasePromptTemplate, str]] = None, score_threshold: float = 4.0)

Correctness evaluator.

Evaluates the correctness of a question answering system. This evaluator depends on reference answer to be provided, in addition to the query string and response string.

It outputs a score between 1 and 5, where 1 is the worst and 5 is the best, along with a reasoning for the score. Passing is defined as a score greater than or equal to the given threshold.

Parameters

service_context (Optional[ServiceContext]) – Service context.
eval_template (Optional[Union[BasePromptTemplate, str]]) – Template for the evaluation prompt.
score_threshold (float) – Numerical threshold for passing the evaluation, defaults to 4.0.

async aevaluate(query: Optional[str] = None, response: Optional[str] = None, contexts: Optional[Sequence[str]] = None, reference: Optional[str] = None, sleep_time_in_seconds: int = 0, **kwargs: Any) → EvaluationResult