Skip to content
LlamaIndex
Evaluation using Prometheus model
Home
Learn
Use Cases
Examples
Component Guides
Advanced Topics
API Reference
Open-Source Community
LlamaCloud
LlamaIndex
Home
Home
High-Level Concepts
Installation and Setup
How to read these docs
Starter Examples
Starter Examples
Starter Tutorial (OpenAI)
Starter Tutorial (Local Models)
Discover LlamaIndex Video Series
Frequently Asked Questions (FAQ)
Starter Tools
Starter Tools
RAG CLI
Learn
Learn
Using LLMs
Building a RAG pipeline
Building a RAG pipeline
Loading & Ingestion
Loading & Ingestion
Loading Data (Ingestion)
LlamaHub
Indexing & Embedding
Storing
Querying
Building an agent
Building an agent
Building a basic agent
Agents with local models
Adding RAG to an agent
Enhancing with LlamaParse
Memory
Adding other tools
Building Workflows
Building Workflows
A basic workflow
Branches and loops
Maintaining state
Streaming events
Concurrent execution
Subclassing workflows
Nested workflows
Observability
Unbound syntax
Tracing and Debugging
Evaluating
Evaluating
Evaluating
Cost Analysis
Cost Analysis
Usage Pattern
Putting it all Together
Putting it all Together
Full-stack web application
Full-stack web application
A Guide to Building a Full-Stack Web App with LLamaIndex
A Guide to Building a Full-Stack LlamaIndex Web App with Delphic
Q&A Patterns
Q&A Patterns
A Guide to Extracting Terms and Definitions
Chatbots
Chatbots
How to Build a Chatbot
Structured data
Structured data
Agents
Use Cases
Use Cases
Prompting
Question-Answering (RAG)
Chatbots
Structured Data Extraction
Agents
Multi-Modal Applications
Fine-Tuning
Examples
Examples
Agents
Agents
💬🤖 How to Build a Chatbot
GPT Builder Demo
Building a Multi-PDF Agent using Query Pipelines and HyDE
Step-wise, Controllable Agents
Controllable Agents for RAG
Building an Agent around a Query Pipeline
Agentic rag using vertex ai
Agentic rag with llamaindex and vertexai managed index
Function Calling Anthropic Agent
Function Calling AWS Bedrock Converse Agent
Chain-of-Abstraction LlamaPack
Building a Custom Agent
DashScope Agent Tutorial
Introspective Agents: Performing Tasks With Reflection
Language Agent Tree Search
LLM Compiler Agent Cookbook
Simple Composable Memory
Vector Memory
Function Calling Mistral Agent
Multi-Document Agents (V1)
Multi-Document Agents
Function Calling NVIDIA Agent
Build your own OpenAI Agent
Context-Augmented OpenAI Agent
OpenAI Agent Workarounds for Lengthy Tool Descriptions
Single-Turn Multi-Function Calling OpenAI Agents
OpenAI Agent + Query Engine Experimental Cookbook
OpenAI Agent Query Planning
Retrieval-Augmented OpenAI Agent
OpenAI Agent with Tool Call Parser
OpenAI Agent with Query Engine Tools
OpenAI Assistant Agent
OpenAI Assistant Advanced Retrieval Cookbook
OpenAI agent: specifying a forced function call
Benchmarking OpenAI Retrieval API (through Assistant Agent)
ReAct Agent - A Simple Intro with Calculator Tools
ReAct Agent with Query Engine (RAG) Tools
Controlling Agent Reasoning Loop with Return Direct Tools
Structured Planning Agent
Chat Engines
Chat Engines
Chat Engine - Best Mode
Chat Engine - Condense Plus Context Mode
Chat Engine - Condense Question Mode
Chat Engine - Context Mode
Chat Engine - OpenAI Agent Mode
Chat Engine with a Personality ✨
Chat Engine - ReAct Agent Mode
Chat Engine - Simple Mode REPL
Cookbooks
Cookbooks
GraphRAG Implementation with LlamaIndex
GraphRAG Implementation with LlamaIndex - V2
AirtrainAI Cookbook
Anthropic Haiku Cookbook
Trustworthy RAG with the Trustworthy Language Model
Codestral from MistralAI Cookbook
Cohere init8 and binary Embeddings Retrieval Evaluation
Contextual Retrieval
CrewAI + LlamaIndex Cookbook
Llama3 Cookbook
Llama3 Cookbook with Groq
Llama3 Cookbook with Ollama and Replicate
MistralAI Cookbook
mixedbread Rerank Cookbook
Optimizing for relevance using MongoDB and LlamaIndex
Oracle AI Vector Search with Document Processing
Components Of LlamaIndex
Evaluating RAG Systems
Ingestion Pipeline
Metadata Extraction
Observability
Agents
Router QueryEngine and SubQuestion QueryEngine
Multi-Modal RAG System
Advanced RAG with LlamaParse
Prometheus-2 Cookbook
Customization
Customization
Azure OpenAI
ChatGPT
HuggingFace LLM - Camel-5b
HuggingFace LLM - StableLM
Chat Prompts Customization
Completion Prompts Customization
Streaming
Streaming for Chat Engine - Condense Question Mode
Data Connectors
Data Connectors
Chroma Reader
DashVector Reader
Database Reader
DeepLake Reader
Discord Reader
Docling Reader
Faiss Reader
Github Repo Reader
Google Chat Reader Test
Google Docs Reader
Google Drive Reader
Google Maps Text Search Reader
Google Sheets Reader
Make Reader
Mbox Reader
MilvusReader
MongoDB Reader
MyScale Reader
Notion Reader
Obsidian Reader
Pathway Reader
Pinecone Reader
Preprocess
Psychic Reader
Qdrant Reader
Slack Reader
Twitter Reader
Weaviate Reader
Web Page Reader
Deplot Reader Demo
HTML Tag Reader
Oracle AI Vector Search: Document Processing
Simple Directory Reader
Parallel Processing SimpleDirectoryReader
Simple Directory Reader over a Remote FileSystem
Discover LlamaIndex
Discover LlamaIndex
Discord Thread Management
Docstores
Docstores
Demo: Azure Table Storage as a Docstore
Docstore Demo
Dynamo DB Docstore Demo
Firestore Demo
MongoDB Demo
Redis Docstore+Index Store Demo
Embeddings
Embeddings
Anyscale Embeddings
LangChain Embeddings
OpenAI Embeddings
Aleph Alpha Embeddings
Bedrock Embeddings
Embeddings with Clarifai
Cloudflare Workers AI Embeddings
CohereAI Embeddings
Custom Embeddings
Dashscope embeddings
Databricks Embeddings
Deepinfra
Elasticsearch Embeddings
Qdrant FastEmbed Embeddings
Fireworks Embeddings
Google Gemini Embeddings
Gigachat
Google PaLM Embeddings
Local Embeddings with HuggingFace
IBM watsonx.ai
Local Embeddings with IPEX-LLM on Intel CPU
Local Embeddings with IPEX-LLM on Intel GPU
Optimized BGE Embedding Model using Intel® Extension for Transformers
Jina 8K Context Window Embeddings
Jina Embeddings
Llamafile Embeddings
LLMRails Embeddings
MistralAI Embeddings
Mixedbread AI Embeddings
Nomic Embedding
NVIDIA NIMs
Oracle Cloud Infrastructure Generative AI
OctoAI Embeddings
Ollama Embeddings
Local Embeddings with OpenVINO
Optimized Embedding Model using Optimum-Intel
Oracle AI Vector Search: Generate Embeddings
PremAI Embeddings
Interacting with Embeddings deployed in Amazon SageMaker Endpoint with LlamaIndex
Text Embedding Inference
TextEmbed - Embedding Inference Server
Together AI Embeddings
Upstage Embeddings
Interacting with Embeddings deployed in Vertex AI Endpoint with LlamaIndex
Voyage Embeddings
Yandexgpt
Evaluation
Evaluation
🏔️ Step-back prompting with workflows for RAG with Argilla
BEIR Out of Domain Benchmark
🚀 RAG/LLM Evaluators - DeepEval
HotpotQADistractor Demo
QuestionGeneration
RAGChecker: A Fine-grained Evaluation Framework For Diagnosing RAG
Self Correcting Query Engines - Evaluation & Retry
Tonic Validate Evaluators
How to use UpTrain with LlamaIndex
Answer Relevancy and Context Relevancy Evaluations
BatchEvalRunner - Running Multiple Evaluations
Correctness Evaluator
Faithfulness Evaluator
Guideline Evaluator
Benchmarking LLM Evaluators On The MT-Bench Human Judgement
Benchmarking LLM Evaluators On A Mini MT-Bench (Single Grading)
Evaluating Multi-Modal RAG
Pairwise Evaluator
Evaluation using Prometheus model
Evaluation using Prometheus model
Table of contents
Note: We are showcasing original Prometheus model for the analysis here. You can re-run the analysis with quantized version of the model.
Download Datasets
Define Prometheus LLM hosted on HuggingFace.
Prompt templates.
Correctness Evaluation Prompt
Faithfulness Evaluation Prompt
Relevancy Evaluation Prompt
Define parser function
Define Correctness, FaithFulness, Relevancy Evaluators
Let's create a function to create and for different datasets.
Function to run batch evaluations on defined evaluators
Function to check the distribution of scores
Function to check correctness, faithfulness and relevancy evaluation score
Function to compute .
Evaluation on PaulGraham Essay text
Compute Correctness, Faithfulness and Relevancy Evaluation
Correctness Evaluation score distribution with Prometheus Evaluator.
Correctness Evaluation score distribution with GPT-4 Evaluator.
Feedback comparision between prometheus and gpt-4.
Observation:
Prometheus Faithfulness and Relevancy Evaluation scores.
GPT-4 Faithfulness and Relevancy Evaluation scores.
Hamming Distance comparison between Prometheus and GPT-4
Observation:
GPT-4 Cost analysis
Evaluation with Llama2 paper
Compute Correctness, Faithfulness and Relevancy Evaluation
Correctness Evaluation score distribution with Prometheus Evaluator.
Correctness Evaluation score distribution with GPT-4 Evaluator.
Feedback comparison between prometheus and gpt-4 for correctness.
Observation:
Prometheus Faithfulness and Relevancy Evaluation scores.
GPT-4 Faithfulness and Relevancy Evaluation scores.
Hamming Distance comparison between Prometheus and GPT-4
Observation:
Feedback comparison between prometheus and gpt-4 for faithfulness and relevancy
If you compare the feedback and contexts, there is mention of range of parameters in the context and response but the feedback says the model could not find such information.
GPT-4 Evaluates it correctly, unlike prometheus model.
GPT-4 Cost analysis
Total Cost Analysis
Prometheus Model - for queries ( for Paul Graham Essay and for Llama2 paper) which accounts to per query.
GPT4 Model - (total_cost_paul_graham_essay + total_cost_llama2) - which accounts to per query.
Observation:
Relevancy Evaluator
Retrieval Evaluation
Embedding Similarity Evaluator
Finetuning
Finetuning
How to Finetune a cross-encoder using LLamaIndex
Finetuning corpus embeddings using NUDGE
Finetune Embeddings
Finetuning an Adapter on Top of any Black-Box Embedding Model
Knowledge Distillation For Fine-Tuning A GPT-3.5 Judge (Correctness)
Knowledge Distillation For Fine-Tuning A GPT-3.5 Judge (Pairwise)
Fine Tuning MistralAI models using Finetuning API
Fine Tuning GPT-3.5-Turbo
Fine Tuning with Function Calling
Fine-tuning a gpt-3.5 ReAct Agent on Better Chain of Thought
Custom Cohere Reranker
Router Fine-tuning
Ingestion
Ingestion
Advanced Ingestion Pipeline
Async Ingestion Pipeline + Metadata Extraction
Ingestion Pipeline + Document Management
Building a Live RAG Pipeline over Google Drive Files
Parallelizing Ingestion Pipeline
Redis Ingestion Pipeline
LLMs
LLMs
AI21
Aleph Alpha
Anthropic
Anthropic Prompt Caching
Anyscale
Azure AI model inference
Azure OpenAI
Bedrock
Bedrock Converse
Cerebras
Clarifai LLM
Cleanlab Trustworthy Language Model
Cohere
DashScope LLMS
DataBricks
DeepInfra
EverlyAI
Fireworks
Fireworks Function Calling Cookbook
Friendli
Gemini
Groq
Hugging Face LLMs
IBM watsonx.ai
IPEX-LLM on Intel CPU
IPEX-LLM on Intel GPU
Konko
Langchain
LiteLLM
Replicate - Llama 2 13B
LlamaCPP
🦙 x 🦙 Rap Battle
Llama API
llamafile
LLM Predictor
LM Studio
LocalAI
Maritalk
MistralRS LLM
MistralAI
ModelScope LLMS
Monster API <> LLamaIndex
MyMagic AI LLM
Neutrino AI
NVIDIA NIMs
NVIDIA NIMs
Nvidia TensorRT-LLM
NVIDIA's LLM Text Completion API
Nvidia Triton
Oracle Cloud Infrastructure Generative AI
OctoAI
Ollama - Llama 3.1
Ollama - Gemma
OpenAI
OpenAI JSON Mode vs. Function Calling for Data Extraction
OpenLLM
OpenRouter
OpenVINO LLMs
Optimum Intel LLMs optimized with IPEX backend
AlibabaCloud-PaiEas
PaLM
Perplexity
Portkey
Predibase
PremAI LlamaIndex
Client of Baidu Intelligent Cloud's Qianfan LLM Platform
RunGPT
Interacting with LLM deployed in Amazon SageMaker Endpoint with LlamaIndex
Sambanova
Solar LLM
Together AI LLM
Unify
Upstage
Vertex AI
Replicate - Vicuna 13B
vLLM
Xorbits Inference
Yi
Llama Datasets
Llama Datasets
Downloading a LlamaDataset from LlamaHub
Benchmarking RAG Pipelines With A
Submission Template Notebook
Contributing a LlamaDataset To LlamaHub
Llama Hub
Llama Hub
LlamaHub Demostration
Ollama Llama Pack Example
Llama Pack - Resume Screener 📄
Llama Packs Example
Low Level
Low Level
Building Evaluation from Scratch
Building an Advanced Fusion Retriever from Scratch
Building Data Ingestion from Scratch
Building RAG from Scratch (Open-source only!)
Building Response Synthesis from Scratch
Building Retrieval from Scratch
Building a Router from Scratch
Building a (Very Simple) Vector Store from Scratch
Managed Indexes
Managed Indexes
BGEM3Demo
Google Generative Language Semantic Retriever
PostgresML Managed Index
Google Cloud LlamaIndex on Vertex AI for RAG
Semantic Retriever Benchmark
Vectara Managed Index
Managed Index with Zilliz Cloud Pipelines
Metadata Extractors
Metadata Extractors
Entity Metadata Extraction
Metadata Extraction and Augmentation w/ Marvin
Extracting Metadata for Better Document Indexing and Understanding
Automated Metadata Extraction for Better Retrieval + Synthesis
Pydantic Extractor
Multi-Modal
Multi-Modal
Chroma Multi-Modal Demo with LlamaIndex
Multi-Modal LLM using Anthropic model for image reasoning
Multi-Modal LLM using Azure OpenAI GPT-4V model for image reasoning
Multi-Modal LLM using DashScope qwen-vl model for image reasoning
Multi-Modal LLM using Google's Gemini model for image understanding and build Retrieval Augmented Generation with LlamaIndex
Multimodal Structured Outputs: GPT-4o vs. Other GPT-4 Variants
GPT4-V Experiments with General, Specific questions and Chain Of Thought (COT) Prompting Technique.
Advanced Multi-Modal Retrieval using GPT4V and Multi-Modal Index/Retriever
Image to Image Retrieval using CLIP embedding and image correlation reasoning using GPT4V
LlaVa Demo with LlamaIndex
Retrieval-Augmented Image Captioning
Multi-Modal LLM using Mistral Pixtral-12B model for image reasoning
[Beta] Multi-modal ReAct Agent
Multi-Modal GPT4V Pydantic Program
Multi-Modal RAG using Nomic Embed and Anthropic.
Multi-Modal Retrieval using GPT text embedding and CLIP image embedding for Wikipedia Articles
Multimodal RAG for processing videos using OpenAI GPT4V and LanceDB vectorstore
Multimodal RAG with VideoDB
Multimodal Ollama Cookbook
Multi-Modal LLM using OpenAI GPT-4V model for image reasoning
Multi-Modal LLM using Replicate LlaVa, Fuyu 8B, MiniGPT4 models for image reasoning
Semi-structured Image Retrieval
Multi-Tenancy
Multi-Tenancy
Multi-Tenancy RAG with LlamaIndex
Node Parsers & Text Splitters
Node Parsers & Text Splitters
Semantic Chunker
Semantic double merging chunking
TopicNodeParser
Node Postprocessors
Node Postprocessors
Cohere Rerank
Colbert Rerank
File Based Node Parsers
FlagEmbeddingReranker
Jina Rerank
LLM Reranker Demonstration (Great Gatsby)
LLM Reranker Demonstration (2021 Lyft 10-k)
LongContextReorder
Metadata Replacement + Node Sentence Window
Mixedbread AI Rerank
NVIDIA NIMs
Sentence Embedding Optimizer
PII Masking
Forward/Backward Augmentation
Recency Filtering
SentenceTransformerRerank
Time-Weighted Rerank
VoyageAI Rerank
OpenVINO Rerank
RankGPT Reranker Demonstration (Van Gogh Wiki)
RankLLM Reranker Demonstration (Van Gogh Wiki)
Object Stores
Object Stores
The Class
Observability
Observability
Aim Callback
HoneyHive LlamaIndex Tracer
Langfuse Callback Handler
Analyze and Debug LlamaIndex Applications with PostHog and Langfuse
Llama Debug Handler
MLflow
OpenInference Callback Handler + Arize Phoenix
Observability with OpenLLMetry
Logging traces with Opik
PromptLayer Handler
Token Counting Handler
UpTrain Callback Handler
Wandb Callback Handler
Output Parsers
Output Parsers
Guardrails Output Parsing
Langchain Output Parsing
DataFrame Structured Data Extraction
Evaporate Demo
Function Calling Program for Structured Extraction
Guidance Pydantic Program
Guidance for Sub-Question Query Engine
LLM Pydantic Program
LM Format Enforcer Pydantic Program
LM Format Enforcer Regular Expression Generation
LLM Pydantic Program - NVIDIA
OpenAI Pydantic Program
OpenAI function calling for Sub-Question Query Engine
Param Optimizer
Param Optimizer
[WIP] Hyperparameter Optimization for RAG
Prompts
Prompts
Advanced Prompt Techniques (Variable Mappings, Functions)
EmotionPrompt in RAG
Accessing/Customizing Prompts within Higher-Level Modules
"Optimization by Prompting" for RAG
Prompt Engineering for RAG
Property Graph
Property Graph
Using a Property Graph Store
Property Graph Construction with Predefined Schemas
Property Graph Index
Defining a Custom Property Graph Retriever
Neo4j Property Graph Index
Query Engines
Query Engines
Retriever Query Engine with Custom Retrievers - Simple Hybrid Search
JSONalyze Query Engine
Joint QA Summary Query Engine
Retriever Router Query Engine
Router Query Engine
SQL Auto Vector Query Engine
SQL Join Query Engine
SQL Router Query Engine
CitationQueryEngine
Cogniswitch query engine
Defining a Custom Query Engine
Ensemble Query Engine Guide
FLARE Query Engine
JSON Query Engine
Knowledge Graph Query Engine
Knowledge Graph RAG Query Engine
Structured Hierarchical Retrieval
Pandas Query Engine
Recursive Retriever + Query Engine Demo
[Beta] Text-to-SQL with PGVector
Query Engine with Pydantic Outputs
Recursive Retriever + Document Agents
Joint Tabular/Semantic QA over Tesla 10K
Sub Question Query Engine
Query Pipeline
Query Pipeline
An Introduction to LlamaIndex Query Pipelines
Query Pipeline with Async/Parallel Execution
Query Pipeline Chat Engine
Query Pipeline over Pandas DataFrames
Query Pipeline with Routing
Query Pipeline for Advanced Text-to-SQL
Query Transformations
Query Transformations
HyDE Query Transform
Multi-Step Query Engine
Query Transform Cookbook
Response Synthesizers
Response Synthesizers
Pydantic Tree Summarize
Stress-Testing Long Context LLMs with a Recall Task
Pydantic Tree Summarize
Refine
Refine with Structured Answer Filtering
Tree Summarize
Retrievers
Retrievers
Auto Merging Retriever
Comparing Methods for Structured Retrieval (Auto-Retrieval vs. Recursive Retrieval)
Bedrock (Knowledge Bases)
BM25 Retriever
Composable Objects
Activeloop Deep Memory
Ensemble Retrieval Guide
Chunk + Document Hybrid Retrieval with Long-Context Embeddings (Together.ai)
Pathway Retriever
Reciprocal Rerank Fusion Retriever
Recursive Retriever + Node References + Braintrust
Recursive Retriever + Node References
Relative Score Fusion and Distribution-Based Score Fusion
Router Retriever
Simple Fusion Retriever
Auto-Retrieval from a Vectara Index
Vertex AI Search Retriever
Videodb retriever
You.com Retriever
Tools
Tools
OnDemandLoaderTool Tutorial
Azure Code Interpreter Tool Spec
Cassandra Database Tools
Evaluation Query Engine Tool
Transforms
Transforms
Transforms Evaluation
Use Cases
Use Cases
10K Analysis
10Q Analysis
Email Data Extraction
Github Issue Analysis
Vector Stores
Vector Stores
AWSDocDBDemo
Alibaba Cloud OpenSearch Vector Store
Amazon Neptune - Neptune Analytics vector store
AnalyticDB
Astra DB
Simple Vector Store - Async Index Creation
Awadb Vector Store
Azure AI Search
Azure CosmosDB MongoDB Vector Store
Azure Cosmos DB No SQL Vector Store
Bagel Vector Store
Bagel Network
Baidu VectorDB
Cassandra Vector Store
Chroma + Fireworks + Nomic with Matryoshka embedding
Chroma
ClickHouse Vector Store
CouchbaseVectorStoreDemo
DashVector Vector Store
Databricks Vector Search
Deep Lake Vector Store Quickstart
DocArray Hnsw Vector Store
DocArray InMemory Vector Store
DuckDB
Elasticsearch Vector Store
Elasticsearch
Epsilla Vector Store
Faiss Vector Store
Firestore Vector Store
Hologres
Jaguar Vector Store
Advanced RAG with temporal filters using LlamaIndex and KDB.AI vector store
LanceDB Vector Store
Lantern Vector Store (auto-retriever)
Lantern Vector Store
Lindorm
Metal Vector Store
Milvus Vector Store With Hybrid Retrieval
Milvus Vector Store
MilvusOperatorFunctionDemo
MongoDB Atlas Vector Store
MongoDB Atlas + Fireworks AI RAG Example
MongoDB Atlas + OpenAI RAG Example
MyScale Vector Store
Neo4j vector store
ObjectBox VectorStore Demo
Opensearch Vector Store
pgvecto.rs
Pinecone Vector Store - Hybrid Search
Pinecone Vector Store
Qdrant Vector Store
Qdrant Vector Store - Metadata Filter
Qdrant Vector Store - Default Qdrant Filters
Redis Vector Store
Relyt
Rockset Vector Store
Simple Vector Store
Local Llama2 + VectorStoreIndex
Llama2 + VectorStoreIndex
Simple Vector Stores - Maximum Marginal Relevance Retrieval
S3/R2 Storage
Supabase Vector Store
TablestoreVectorStore
Tair Vector Store
Tencent Cloud VectorDB
TiDB Vector Store
Timescale Vector Store (PostgreSQL)
txtai Vector Store
Typesense Vector Store
Upstash Vector Store
VearchDemo
Google Vertex AI Vector Search
Vespa Vector Store demo
Weaviate Vector Store - Hybrid Search
Weaviate Vector Store
Auto-Retrieval from a Weaviate Vector Database
Weaviate Vector Store Metadata Filter
WordLift Vector Store
Zep Vector Store
Auto-Retrieval from a Vector Database
Chroma Vector Store
Auto-Retrieval from a Vector Database
Guide: Using Vector Store Index with Existing Pinecone Vector Store
Guide: Using Vector Store Index with Existing Weaviate Vector Store
Neo4j Vector Store - Metadata Filter
Oracle AI Vector Search: Vector Store
A Simple to Advanced Guide with Auto-Retrieval (with Pinecone + Arize Phoenix)
Pinecone Vector Store - Metadata Filter
Postgres Vector Store
Hybrid Search with Qdrant BM42
Qdrant Hybrid Search
Workflow
Workflow
JSONalyze Query Engine
Workflows for Advanced Text-to-SQL
Build RAG with in-line citations
Corrective RAG Workflow
Workflow for a Function Calling Agent
Choose Your Own Adventure Workflow (Human In The Loop)
LongRAG Workflow
MultiStep Query Engine
Multi-strategy workflow with reflection
Parallel Execution of Same Event Example
RAG Workflow with Reranking
Workflow for a ReAct Agent
Reflection Workflow for Structured Outputs
Router Query Engine
Self-Discover Workflow
Sub Question Query Engine as a workflow
Workflows cookbook: walking through all features of Workflows
Component Guides
Component Guides
Models
Models
LLMs
LLMs
Using LLMs
Standalone Usage
Customizing LLMs
Available LLM Integrations
Embeddings
Multi Modal
Prompts
Prompts
Usage pattern
Loading
Loading
Documents and Nodes
Documents and Nodes
Using Documents
Using Nodes
Metadata Extraction
SimpleDirectoryReader
Data Connectors
Data Connectors
Usage Pattern
LlamaParse
Module Guides
Node Parsers / Text Splitters
Node Parsers / Text Splitters
Node Parser Modules
Ingestion Pipeline
Ingestion Pipeline
Transformations
Indexing
Indexing
Index Guide
Vector Store Index
Property Graph Index
Document Management
LlamaCloud
Metadata Extraction
Modules
Storing
Storing
Vector Stores
Document Stores
Index Stores
Chat Stores
Key-Value Stores
Persisting & Loading Data
Customizing Storage
Querying
Querying
Query Engines
Query Engines
Usage Pattern
Response Modes
Streaming
Module Guides
Supporting Modules
Chat Engines
Chat Engines
Usage Pattern
Module Guides
Retrieval
Retrieval
Retriever Modules
Retriever Modes
Node Postprocessors
Node Postprocessors
Node Postprocessor Modules
Response Synthesis
Response Synthesis
Response Synthesis Modules
Routing
Workflows
Query Pipelines (Deprecated)
Query Pipelines (Deprecated)
Usage Pattern
Module Guides
Module Usage
Structured Outputs
Structured Outputs
Output Parsing Modules
(Deprecated) Query Engines + Pydantic Outputs
Pydantic Program
Agents
Agents
Usage Pattern
Lower-Level Agent API
Module Guides
Tools
Workflows
Workflows
None
Evaluation
Evaluation
Usage Pattern (Response Evaluation)
Usage Pattern (Retrieval)
Modules
LlamaDatasets
LlamaDatasets
Contributing A LabelledRagDataset
Evaluating With LabelledRagDataset's
Evaluating Evaluators with LabelledEvaluatorDataset's
Observability
Observability
Instrumentation
Settings
Llama Deploy
Llama Deploy
Getting Started
Core Components
Manual orchestration
Python SDK
CLI
Advanced Topics
Advanced Topics
Building Performant RAG Applications for Production
Basic Strategies
Agentic strategies
Retrieval
Retrieval
Advanced Retrieval Strategies
Query Transformations
Evaluation
Evaluation
Component Wise Evaluation
End-to-End Evaluation
Evaluation
Fine-Tuning
Writing Custom Modules
Building RAG from Scratch (Lower-Level)
API Reference
API Reference
Agents
Agents
Coa
Dashscope
Introspective
Lats
Llm compiler
Openai
Openai legacy
React
Callbacks
Callbacks
Agentops
Aim
Argilla
Arize phoenix
Deepeval
Honeyhive
Langfuse
Literalai
Llama debug
Openinference
Opik
Promptlayer
Token counter
Uptrain
Wandb
Chat Engines
Chat Engines
Condense plus context
Condense question
Context
Simple
Embeddings
Embeddings
Adapter
Alephalpha
Alibabacloud aisearch
Anyscale
Azure inference
Azure openai
Bedrock
Clarifai
Clip
Cloudflare workersai
Cohere
Dashscope
Databricks
Deepinfra
Elasticsearch
Fastembed
Fireworks
Gemini
Gigachat
Google
None
Huggingface
Huggingface api
Huggingface itrex
Huggingface openvino
Huggingface optimum
Huggingface optimum intel
Ibm
Instructor
Ipex llm
Jinaai
Langchain
Litellm
Llamafile
Llm rails
Mistralai
Mixedbreadai
Nomic
Nvidia
Oci genai
Octoai
Ollama
Openai
Oracleai
Premai
Sagemaker endpoint
Text embeddings inference
Textembed
Together
Upstage
Vertex
Vertex endpoint
Voyageai
Xinference
Yandexgpt
Evaluation
Evaluation
Answer relevancy
Context relevancy
Correctness
Dataset generation
Faithfullness
Guideline
Metrics
Multi modal
Pairwise comparison
Query response
Response
Retrieval
Semantic similarity
Tonic validate
Indexes
Indexes
Bge m3
Colbert
Dashscope
Document summary
Google
Keyword
Knowledge graph
Llama cloud
Postgresml
Property graph
Summary
Tree
Vectara
Vector
Vertexai
Zilliz
Ingestion
Ingestion
Instrumentation
Instrumentation
Event handlers
Event types
Span handlers
Span types
LLMs
LLMs
None
Ai21
Alephalpha
Alibabacloud aisearch
Anthropic
Anyscale
Azure inference
Azure openai
Bedrock
Bedrock converse
Cerebras
Clarifai
Cleanlab
Cohere
Custom llm
Dashscope
Databricks
Deepinfra
Everlyai
Fireworks
Friendli
Gemini
Gigachat
None
Groq
Huggingface
Huggingface api
Ibm
Ipex llm
Konko
Langchain
Litellm
Llama api
Llama cpp
Llamafile
Lmstudio
Localai
Maritalk
Mistral rs
Mistralai
Mlx
Modelscope
Monsterapi
Mymagic
Neutrino
Nvidia
Nvidia tensorrt
Nvidia triton
Oci genai
Octoai
Ollama
Openai
Openai like
Openllm
Openrouter
Openvino
Optimum intel
Paieas
Palm
Perplexity
Portkey
Predibase
Premai
Qianfan
Replicate
Rungpt
Sagemaker endpoint
Sambanova
Solar
Text generation inference
Together
Unify
Upstage
Vertex
Vllm
Xinference
Yi
You
Llama Datasets
Llama Datasets
Llama Deploy
Llama Deploy
apiserver
control_plane
deploy
message_consumers
message_publishers
messages
orchestrators
Python SDK
services
types
message_queues
message_queues
apache_kafka
rabbitmq
redis
simple
Llama Packs
Llama Packs
Agent search retriever
Agents coa
Agents lats
Agents llm compiler
Amazon product extraction
Arize phoenix query engine
Auto merging retriever
Chroma autoretrieval
Code hierarchy
Cogniswitch agent
Cohere citation chat
Corrective rag
Deeplake deepmemory retriever
Deeplake multimodal retrieval
Dense x retrieval
Diff private simple dataset
Docugami kg rag
Evaluator benchmarker
Finchat
Fusion retriever
Fuzzy citation
Gmail openai agent
Gradio agent chat
Gradio react agent chatbot
Infer retrieve rerank
Koda retriever
Llama dataset metadata
Llama guard moderator
Llava completion
Longrag
Mixture of agents
Multi document agents
Multi tenancy rag
Multidoc autoretrieval
Nebulagraph query engine
Neo4j query engine
Node parser semantic chunking
Ollama query engine
Panel chatbot
Query understanding agent
Raft dataset
Rag cli local
Rag evaluator
Rag fusion query pipeline
Ragatouille retriever
Raptor
Recursive retriever
Redis ingestion pipeline
Resume screener
Retry engine weaviate
Searchain
Secgpt
Self discover
Self rag
Sentence window retriever
Snowflake query engine
Stock market data query engine
Streamlit chatbot
Sub question weaviate
Subdoc summary
Tables
Timescale vector autoretrieval
Trulens eval packs
Vanna
Vectara rag
Voyage query engine
Zenguard
Zephyr query engine
Memory
Memory
Chat memory buffer
Simple composable memory
Vector memory
Metadata Extractors
Metadata Extractors
Entity
Keyword
None
Marvin
Pydantic
Question
Relik
Summary
Title
Multi-Modal LLMs
Multi-Modal LLMs
Anthropic
Azure openai
Dashscope
Gemini
Huggingface
Mistralai
Ollama
Openai
Replicate
Node Parsers & Text Splitters
Node Parsers & Text Splitters
Alibabacloud aisearch
Dashscope
Docling
Topic
Code
Hierarchical
Html
Json
Langchain
Markdown
Markdown element
Semantic splitter
Sentence splitter
Sentence window
Token text splitter
Unstructured element
Node Postprocessors
Node Postprocessors
NER PII
PII
Alibabacloud aisearch rerank
Auto prev next
Cohere rerank
Colbert rerank
Dashscope rerank
Embedding recency
Fixed recency
Flag embedding reranker
Jinaai rerank
Keyword
Llm rerank
Long context reorder
Longllmlingua
Metadata replacement
Mixedbreadai rerank
Nvidia rerank
Openvino rerank
Presidio
Prev next
Rankgpt rerank
Rankllm rerank
Sbert rerank
Sentence optimizer
Similarity
Tei rerank
Time weighted
Voyageai rerank
Xinference rerank
Object Stores
Object Stores
Output Parsers
Output Parsers
Guardrails
Langchain
Pydantic
Selection
Programs
Programs
Evaporate
Guidance
Llm text completion
Lmformatenforcer
Multi modal
Openai
Prompts
Prompts
Query Engines
Query Engines
FLARE
JSONalayze
NL SQL table
PGVector SQL
SQL join
SQL table retriever
Citation
Cogniswitch
Custom
Knowledge graph
Multi step
Pandas
Retriever
Retriever router
Retry
Router
Simple multi modal
Sub question
Tool retriever router
Transform
Query Pipeline
Query Pipeline
Agent
Arg pack
Custom
Function
Input
Llm
Multi modal
Object
Output parser
Postprocessor
Prompt
Query engine
Query transform
Retriever
Router
Synthesizer
Tool runner
Question Generators
Question Generators
Guidance
Llm question gen
Openai
Readers
Readers
Agent search
Airbyte cdk
Airbyte gong
Airbyte hubspot
Airbyte salesforce
Airbyte shopify
Airbyte stripe
Airbyte typeform
Airbyte zendesk support
Airtable
Alibabacloud aisearch
Apify
Arango db
Arxiv
Asana
Assemblyai
Astra db
Athena
Awadb
Azcognitive search
Azstorage blob
Azure devops
Bagel
Bilibili
Bitbucket
Boarddocs
Box
Chatgpt plugin
Chroma
Clickhouse
Confluence
Couchbase
Couchdb
Dad jokes
Dashscope
Dashvector
Database
Deeplake
Discord
Docling
Docstring walker
Docugami
Earnings call transcript
Elasticsearch
Faiss
Feedly rss
Feishu docs
Feishu wiki
File
Firebase realtimedb
Firestore
Gcs
Genius
Github
Gitlab
Google
Gpt repo
Graphdb cypher
Graphql
Guru
Hatena blog
Hive
Hubspot
Huggingface fs
Hwp
Iceberg
Imdb review
Intercom
Jaguar
Jira
Joplin
Json
Kaltura esearch
Kibela
Lilac
Linear
Llama parse
Macrometa gdn
Make com
Mangadex
Mangoapps guides
Maps
Mbox
Memos
Metal
Microsoft onedrive
Microsoft outlook
Microsoft sharepoint
Milvus
Minio
Mondaydotcom
Mongodb
Myscale
Notion
Nougat ocr
Obsidian
Openalex
Openapi
Opendal
Opensearch
Oracleai
Pandas ai
Papers
Patentsview
Pathway
Pdb
Pdf marker
Pdf table
Pebblo
None
Preprocess
Psychic
Qdrant
Quip
Rayyan
Readme
Readwise
Reddit
Remote
Remote depth
S3
Sec filings
Semanticscholar
Simple directory reader
Singlestore
Slack
Smart pdf loader
Snowflake
Snscrape twitter
Spotify
Stackoverflow
Steamship
String iterable
Stripe docs
Structured data
Telegram
Toggl
Trello
Twitter
Txtai
Upstage
Weather
Weaviate
Web
Whatsapp
Wikipedia
Wordlift
Wordpress
Youtube metadata
Youtube transcript
Zendesk
Zep
Zulip
Response Synthesizers
Response Synthesizers
Accumulate
Compact accumulate
Compact and refine
Generation
Google
Refine
Simple summarize
Tree summarize
Retrievers
Retrievers
Auto merging
Bedrock
Bm25
Duckdb retriever
Keyword
Knowledge graph
Mongodb atlas bm25 retriever
Pathway
Query fusion
Recursive
Router
Sql
Summary
Transform
Tree
Vector
Vertexai search
Videodb
You
Schema
Schema
Selectors
Selectors
Notdiamond
Sparse Embeddings
Sparse Embeddings
Fastembed
Storage
Storage
Chat Store
Chat Store
Azure
Azurecosmosnosql
Dynamodb
Redis
Simple
Upstash
Docstore
Docstore
Azure
Dynamodb
Elasticsearch
Firestore
Mongodb
Postgres
Redis
Simple
Graph Stores
Graph Stores
Falkordb
Kuzu
Nebula
Neo4j
Neptune
Simple
Tidb
Index Store
Index Store
Azure
Azurecosmosnosql
Dynamodb
Elasticsearch
Firestore
Mongodb
Postgres
Redis
Simple
Kvstore
Kvstore
Azure
Azurecosmosnosql
Dynamodb
Elasticsearch
Firestore
Mongodb
Postgres
Redis
S3
Simple
Storage
Storage
Storage context
Vector Store
Vector Store
Alibabacloud opensearch
Analyticdb
Astra db
Awadb
Awsdocdb
Azureaisearch
Azurecosmosmongo
Azurecosmosnosql
Bagel
Baiduvectordb
Cassandra
Chatgpt plugin
Chroma
Clickhouse
Couchbase
Dashvector
Databricks
Deeplake
Docarray
Duckdb
Dynamodb
Elasticsearch
Epsilla
Faiss
Firestore
Google
Hologres
Jaguar
Kdbai
Lancedb
Lantern
Lindorm
Mariadb
Metal
Milvus
Mongodb
Myscale
Neo4jvector
Neptune
Objectbox
Opensearch
Oracledb
Pgvecto rs
Pinecone
Postgres
Qdrant
Redis
Relyt
Rocksetdb
Simple
Singlestoredb
Supabase
Tablestore
Tair
Tencentvectordb
Tidbvector
Timescalevector
Txtai
Typesense
Upstash
Vearch
Vertexaivectorsearch
Vespa
Weaviate
Wordlift
Zep
Tools
Tools
Arxiv
Azure code interpreter
Azure cv
Azure speech
Azure translate
Bing search
Box
Brave search
Cassandra
Chatgpt plugin
Code interpreter
Cogniswitch
Database
Duckduckgo
Exa
Finance
Function
Google
Graphql
Ionic shopping
Jina
Load and search
Metaphor
Multion
Neo4j
Notion
Ondemand loader
Openai
Openapi
None
Passio nutrition ai
Playgrounds
Python file
Query engine
Query plan
Requests
Retriever
Salesforce
Shopify
Slack
Tavily research
Text to image
Tool spec
Vector db
Waii
Weather
Wikipedia
Wolfram alpha
Yahoo finance
Yelp
Zapier
Workflow
Workflow
Decorators
Context
Events
Retry policy
Workflow
Open-Source Community
Open-Source Community
Integrations
Full Stack Projects
Community FAQ
Community FAQ
Chat Engines
Documents and Nodes
Embeddings
Large Language Models
Query Engines
Vector Database
Contributing
Contributing
Code
Docs
Changelog
Presentations
Upgrading to v0.10.x
Deprecated Terms
LlamaCloud
LlamaCloud
LlamaParse
Table of contents
Note: We are showcasing original Prometheus model for the analysis here. You can re-run the analysis with quantized version of the model.
Download Datasets
Define Prometheus LLM hosted on HuggingFace.
Prompt templates.
Correctness Evaluation Prompt
Faithfulness Evaluation Prompt
Relevancy Evaluation Prompt
Define parser function
Define Correctness, FaithFulness, Relevancy Evaluators
Let's create a function to create and for different datasets.
Function to run batch evaluations on defined evaluators
Function to check the distribution of scores
Function to check correctness, faithfulness and relevancy evaluation score
Function to compute .
Evaluation on PaulGraham Essay text
Compute Correctness, Faithfulness and Relevancy Evaluation
Correctness Evaluation score distribution with Prometheus Evaluator.
Correctness Evaluation score distribution with GPT-4 Evaluator.
Feedback comparision between prometheus and gpt-4.
Observation:
Prometheus Faithfulness and Relevancy Evaluation scores.
GPT-4 Faithfulness and Relevancy Evaluation scores.
Hamming Distance comparison between Prometheus and GPT-4
Observation:
GPT-4 Cost analysis
Evaluation with Llama2 paper
Compute Correctness, Faithfulness and Relevancy Evaluation
Correctness Evaluation score distribution with Prometheus Evaluator.
Correctness Evaluation score distribution with GPT-4 Evaluator.
Feedback comparison between prometheus and gpt-4 for correctness.
Observation:
Prometheus Faithfulness and Relevancy Evaluation scores.
GPT-4 Faithfulness and Relevancy Evaluation scores.
Hamming Distance comparison between Prometheus and GPT-4
Observation:
Feedback comparison between prometheus and gpt-4 for faithfulness and relevancy
If you compare the feedback and contexts, there is mention of range of parameters in the context and response but the feedback says the model could not find such information.
GPT-4 Evaluates it correctly, unlike prometheus model.
GPT-4 Cost analysis
Total Cost Analysis
Prometheus Model - for queries ( for Paul Graham Essay and for Llama2 paper) which accounts to per query.
GPT4 Model - (total_cost_paul_graham_essay + total_cost_llama2) - which accounts to per query.
Observation: