NVIDIA NIMs¶
The llama-index-postprocessor-nvidia-rerank` package contains LlamaIndex integrations building applications with models on NVIDIA NIM inference microservice. NIM supports models across domains like chat, embedding, and re-ranking models from the community as well as NVIDIA. These models are optimized by NVIDIA to deliver the best performance on NVIDIA accelerated infrastructure and deployed as a NIM, an easy-to-use, prebuilt containers that deploy anywhere using a single command on NVIDIA accelerated infrastructure.
NVIDIA hosted deployments of NIMs are available to test on the NVIDIA API catalog. After testing, NIMs can be exported from NVIDIA’s API catalog using the NVIDIA AI Enterprise license and run on-premises or in the cloud, giving enterprises ownership and full control of their IP and AI application.
NIMs are packaged as container images on a per model basis and are distributed as NGC container images through the NVIDIA NGC Catalog. At their core, NIMs provide easy, consistent, and familiar APIs for running inference on an AI model.
NVIDIA's Rerank connector¶
This example goes over how to use LlamaIndex to interact with the supported NVIDIA Retrieval QA Ranking Model for retrieval-augmented generation via the NVIDIARerank
class.
Reranking¶
Reranking is a critical piece of high accuracy, efficient retrieval pipelines.
Two important use cases:
- Combining results from multiple data sources
- Enhancing accuracy for single data sources
Combining results from multiple sources¶
Consider a pipeline with data from a semantic store, such as VectorStoreIndex, as well as a BM25 store.
Each store is queried independently and returns results that the individual store considers to be highly relevant. Figuring out the overall relevance of the results is where reranking comes into play.
Follow along with the Advanced - Hybrid Retriever + Re-Ranking use case, substitute the reranker with -
Installation¶
%pip install --upgrade --quiet llama-index-postprocessor-nvidia-rerank
Setup¶
To get started:
Create a free account with NVIDIA, which hosts NVIDIA AI Foundation models.
Click on your model of choice.
Under Input select the Python tab, and click
Get API Key
. Then clickGenerate Key
.Copy and save the generated key as NVIDIA_API_KEY. From there, you should have access to the endpoints.
import getpass
import os
# del os.environ['NVIDIA_API_KEY'] ## delete key and reset
if os.environ.get("NVIDIA_API_KEY", "").startswith("nvapi-"):
print("Valid NVIDIA_API_KEY already in environment. Delete to reset")
else:
nvapi_key = getpass.getpass("NVAPI Key (starts with nvapi-): ")
assert nvapi_key.startswith(
"nvapi-"
), f"{nvapi_key[:5]}... is not a valid key"
os.environ["NVIDIA_API_KEY"] = nvapi_key
Working with API Catalog¶
from llama_index.postprocessor.nvidia_rerank import NVIDIARerank
reranker = NVIDIARerank(top_n=4)
Working with NVIDIA NIMs¶
In addition to connecting to hosted NVIDIA NIMs, this connector can be used to connect to local microservice instances. This helps you take your applications local when necessary.
For instructions on how to setup local microservice instances, see https://developer.nvidia.com/blog/nvidia-nim-offers-optimized-inference-microservices-for-deploying-ai-models-at-scale/
from llama_index.llms.nvidia import NVIDIA
# connect to a rerank NIM running at localhost:1976
reranker = NVIDIARerank(base_url="http://localhost:1976/v1")