Vectara Managed Index¶

In this notebook we are going to show how to use Vectara with LlamaIndex.

Vectara provides an end-to-end managed service for Retrieval Augmented Generation or RAG, which includes:

1. A way to extract text from document files and chunk them into sentences.
2. The state-of-the-art Boomerang embeddings model. Each text chunk is encoded into a vector embedding using Boomerang, and stored in the Vectara internal vector store. Thus, when using Vectara with LlamaIndex you do not need to call a separate embedding model - this happens automatically within the Vectara backend.
3. A query service that automatically encodes the query into embedding, and retrieves the most relevant text segments (including support for Hybrid Search and MMR, Multilingual, or User Defined Function reranking)
4. An option to a create generative summary with a selection of different LLMs (including Mockingbird), based on the retrieved documents, including citations.

See the Vectara API documentation for more information on how to use the API.

Getting Started¶

If you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙.

In [ ]:

!pip install llama-index lama-index-indices-managed-vectara

!pip install llama-index lama-index-indices-managed-vectara

To get started with Vectara, sign up (if you haven't already) and follow our quickstart guide to create a corpus and an API key.

Once you have these, you can provide them as environment variables, which will be used by the LlamaIndex code later on.

import os
os.environ['VECTARA_API_KEY'] = "<YOUR_VECTARA_API_KEY>"
os.environ['VECTARA_CORPUS_ID'] = "<YOUR_VECTARA_CORPUS_ID>"
os.environ['VECTARA_CUSTOMER_ID'] = "<YOUR_VECTARA_CUSTOMER_ID>"

RAG with LlamaIndex and Vectara¶

There are a few ways you can index your data into Vectara, including:

1. With the from_documents() or insert_file() methods of VectaraIndex
2. Uploading files directly in the Vectara console
3. Using Vectara's FILE_UPLOAD or standard indexing APIs
4. Using vectara-ingest, an open source crawler/indexer project
5. Using one of our ingest integration partners like Airbyte, Unstructured or DataVolo.

For this purpose, we will use a simple set of small documents, so using VectaraIndex directly for the ingest is good enough.

Let's ingest the "AI bill of rights" document into our new corpus.

In [ ]:

from llama_index.indices.managed.vectara import VectaraIndex
import requests

url = "https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf"
response = requests.get(url)
local_path = "ai-bill-of-rights.pdf"
with open(local_path, "wb") as file:
    file.write(response.content)

index = VectaraIndex()
index.insert_file(
    local_path, metadata={"name": "AI bill of rights", "year": 2022}
)

from llama_index.indices.managed.vectara import VectaraIndex import requests url = "https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf" response = requests.get(url) local_path = "ai-bill-of-rights.pdf" with open(local_path, "wb") as file: file.write(response.content) index = VectaraIndex() index.insert_file( local_path, metadata={"name": "AI bill of rights", "year": 2022} )

Out[ ]:

'ai-bill-of-rights.pdf'

Running single queries with Vectara Query Engine¶

Now that we've uploaded the document (or if documents have been uploaded previously) we can go and ask questions directly in LlamaIndex. This activates Vectara's RAG pipeline.

To use Vectara's internal LLM for summarization, make sure you specify summary_enabled=True when you generate the Query engine. Here's an example:

In [ ]:

questions = [
    "What are the risks of AI?",
    "What should we do to prevent bad actors from using AI?",
    "What are the benefits?",
]

questions = [ "What are the risks of AI?", "What should we do to prevent bad actors from using AI?", "What are the benefits?", ]

In [ ]:

qe = index.as_query_engine(
    summary_enabled=True, summary_prompt_name="mockingbird-1.0-2024-07-16"
)
qe.query(questions[0]).response

qe = index.as_query_engine( summary_enabled=True, summary_prompt_name="mockingbird-1.0-2024-07-16" ) qe.query(questions[0]).response

Out[ ]:

'The risks of AI include biased data and discriminatory outcomes, opaque decision-making processes, and lack of public trust and understanding of algorithmic systems [1]. These risks can lead to harm to individuals and communities, including image-based abuse, incorrect evaluations, and violations of safety [7]. To mitigate these risks, it is essential to have ongoing transparency, value-sensitive and participatory design, explanations designed for relevant stakeholders, and public consultation [1]. Strong safety regulations and measures to address harms when they occur can enhance innovation in the context of complex technologies [2]. Additionally, industry is providing innovative solutions to mitigate risks to the safety and efficacy of AI systems, including risk assessments, auditing mechanisms, and documentation procedures [2].'

If you want the response to be returned in streaming mode, simply set streaming=True

In [ ]:

qe = index.as_query_engine(
    summary_enabled=True,
    summary_prompt_name="mockingbird-1.0-2024-07-16",
    streaming=True,
)
response = qe.query(questions[0])

for chunk in response.response_gen:
    print(chunk.delta or "", end="", flush=True)

qe = index.as_query_engine( summary_enabled=True, summary_prompt_name="mockingbird-1.0-2024-07-16", streaming=True, ) response = qe.query(questions[0]) for chunk in response.response_gen: print(chunk.delta or "", end="", flush=True)

The risks of AI include biased data and discriminatory outcomes, opaque decision-making processes, and lack of public trust and understanding of algorithmic systems [1]. These risks can lead to harm to individuals and communities, including image-based abuse, incorrect evaluations, and violations of safety [7]. To mitigate these risks, it is essential to have ongoing transparency, value-sensitive and participatory design, explanations designed for relevant stakeholders, and public consultation [1]. Strong safety regulations and measures to address harms when they occur can enhance innovation in the context of complex technologies [2]. Additionally, industry is providing innovative solutions to mitigate risks to the safety and efficacy of AI systems, including risk assessments, auditing mechanisms, and documentation procedures [2].

Using Vectara Chat¶

Vectara also supports a simple chat mode. In this mode the chat history is maintained by Vectara and so you don't have to worry about it. To use it simple call as_chat_engine.

(Chat mode always uses Vectara's summarization so you don't have to explicitly specify summary_enabled=True like before)

In [ ]:

ce = index.as_chat_engine()

ce = index.as_chat_engine()

In [ ]:

for q in questions:
    print(f"Question: {q}\n")
    response = ce.chat(q).response
    print(f"Response: {response}\n")

for q in questions: print(f"Question: {q}\n") response = ce.chat(q).response print(f"Response: {response}\n")

Question: What are the risks of AI?

Response: The risks of AI include potential biased data leading to discriminatory outcomes, opaque decision-making processes, lack of public trust, and understanding of algorithmic systems. Risks also involve safety concerns, such as AI systems violating the safety of individuals, and the proliferation of harmful technologies like AI-enabled "nudification" tools. Furthermore, the incorrect penalization by AI systems, as seen in the case of AI-powered cameras in delivery vans, can lead to adverse consequences for individuals. To mitigate these risks, ongoing transparency, participatory design, explanations for stakeholders, and public consultation are essential. Organizations are implementing innovative solutions like risk assessments, auditing mechanisms, and monitoring tools to address safety and efficacy concerns related to AI systems. The involvement of communities and stakeholders in the design and evaluation of AI systems is crucial to ensure trustworthiness and mitigate potential harms.

Question: What should we do to prevent bad actors from using AI?

Response: To stop malicious individuals from exploiting artificial intelligence, measures can be taken such as implementing ethical principles for the use of AI, ensuring transparency, accountability, and regular monitoring of AI systems, conducting equity assessments, using representative data, protecting against biases, and providing clear organizational oversight. Additionally, it is crucial to address algorithmic discrimination through proactive measures, ongoing disparity testing, and independent evaluations to protect individuals and communities from unjust treatment based on various characteristics. Furthermore, incorporating safeguards like plain language reporting, algorithmic impact assessments, and public disclosure of mitigation efforts can help prevent the misuse of AI by malicious actors [1][2][4].

Question: What are the benefits?

Response: The advantages of using artificial intelligence include providing systems that are lawful, purposeful, accurate, safe, secure, and transparent. AI can be accountable, reliable, and effective, leading to trustworthy automated systems. Additionally, AI can enhance efficiency, decision-making processes, and innovation while supporting democratic values and ethical principles [1][2][7].

Of course streaming works as well with Chat:

In [ ]:

ce = index.as_chat_engine(streaming=True)

ce = index.as_chat_engine(streaming=True)

In [ ]:

response = ce.stream_chat("Will artificial intelligence rule the government?")
for chunk in response.chat_stream:
    print(chunk.delta or "", end="", flush=True)

response = ce.stream_chat("Will artificial intelligence rule the government?") for chunk in response.chat_stream: print(chunk.delta or "", end="", flush=True)

Artificial intelligence will not rule the government. The government is implementing principles and guidelines to ensure the ethical and responsible use of AI in various sectors, including the federal government. These measures focus on transparency, accountability, safety, and adherence to national values, ensuring that AI is used in a manner that upholds civil rights, democratic values, and national security [1] [2] [3].

Agentic RAG¶

Vectara also has its own package, vectara-agentic, built on top of many features from LlamaIndex to easily implement agentic RAG applications. It allows you to create your own AI assistant with RAG query tools and other custom tools, such as making API calls to retrieve information from financial websites. You can find the full documentation for vectara-agentic here.

Let's create a ReAct Agent with a single RAG tool using vectara-agentic. For this you would need to use another LLM as the driver of the agent resoning, and we are using OpenAI's GPT4o here as an example. (for this to work, please make sure you have OPENAI_API_KEY defined in your environment).

In [ ]:

!pip install vectara-agentic

!pip install vectara-agentic

UNCOMMENT NAME ARGUMENT WHEN NEW VERSION OF VECTARA AGENTIC IS RELEASED. MAKE SURE THE AGENT DOES NOT USE "vectara_11" as the first Action

In [ ]:

from vectara_agentic.agent import Agent
import os
from dotenv import load_dotenv

load_dotenv(override=True)

api_key = str(os.environ["VECTARA_API_KEY"])
corpus_id = str(os.environ["VECTARA_CORPUS_ID"])
customer_id = str(os.environ["VECTARA_CUSTOMER_ID"])

agent = Agent.from_corpus(
    tool_name="query_ai",
    vectara_customer_id=customer_id,
    vectara_corpus_id=corpus_id,
    vectara_api_key=api_key,
    data_description="AI regulations",
    assistant_specialty="artificial intelligence",
    verbose=True,
    vectara_summary_num_results=5,
    vectara_summarizer="mockingbird-1.0-2024-07-16",
    vectara_reranker="mmr",
    vectara_rerank_k=50,
)

print(
    agent.chat(
        "What are the risks of AI? What are the benefits? Compare and contrast and provide a summary with arguments for and against from experts."
    )
)

from vectara_agentic.agent import Agent import os from dotenv import load_dotenv load_dotenv(override=True) api_key = str(os.environ["VECTARA_API_KEY"]) corpus_id = str(os.environ["VECTARA_CORPUS_ID"]) customer_id = str(os.environ["VECTARA_CUSTOMER_ID"]) agent = Agent.from_corpus( tool_name="query_ai", vectara_customer_id=customer_id, vectara_corpus_id=corpus_id, vectara_api_key=api_key, data_description="AI regulations", assistant_specialty="artificial intelligence", verbose=True, vectara_summary_num_results=5, vectara_summarizer="mockingbird-1.0-2024-07-16", vectara_reranker="mmr", vectara_rerank_k=50, ) print( agent.chat( "What are the risks of AI? What are the benefits? Compare and contrast and provide a summary with arguments for and against from experts." ) )

> Running step e6ab056d-843c-489d-91ad-fc0245168f7e. Step input: What are the risks of AI? What are the benefits? Compare and contrast and provide a summary with arguments for and against from experts.
Thought: The current language of the user is: English. I need to use a tool to help me answer the question.
Action: query_ai
Action Input: {'query': 'What are the risks and benefits of AI? Provide a summary with arguments for and against from experts.'}
Observation: {'response': "Here is a summary of the risks and benefits of AI based on the provided sources:\n\n**Benefits of AI:**\n\n* AI has transformative potential to improve Americans' lives [1]\n* AI-enabled systems can build better and more innovative infrastructure [3]\n* AI can be integrated into communities in a thoughtful and responsible way, benefiting from lessons learned from urban planning [3]\n\n**Risks of AI:**\n\n* AI can lead to biased data and discriminatory outcomes [2]\n* Opaque decision-making processes can lack public trust and understanding [2]\n* AI can pose risks to individuals, organizations, and society, highlighting the need for risk management frameworks [5]\n\n**Addressing Risks and Benefits:**\n\n* The White House Office of Science and Technology Policy has led a year-long process to seek input from experts and stakeholders on the issue of algorithmic and data-driven harms [1]\n* The Blueprint for an AI Bill of Rights emphasizes the importance of safe and effective systems, ongoing transparency, and value-sensitive and participatory design [1]\n* Industry is providing innovative solutions to mitigate risks to the safety and efficacy of AI systems, including risk assessments, auditing mechanisms, and stakeholder engagement [5]\n* The National Institute of Standards and Technology (NIST) is developing a risk management framework to better manage risks posed to individuals, organizations, and society by AI [5]\n\nOverall, the sources suggest that AI has the potential to bring significant benefits, but also poses risks that need to be addressed through thoughtful and responsible development and integration into communities.", 'citation_metadata': {'metadata for citation 1': {'page': '1', 'lang': 'eng', 'section': '1', 'offset': '10959', 'len': '208', 'CreationDate': '1663695035', 'Producer': 'iLovePDF', 'Title': 'Blueprint for an AI Bill of Rights', 'Creator': 'Adobe Illustrator 26.3 (Macintosh)', 'ModDate': '1664808078', 'name': 'AI bill of rights', 'year': '2022', 'framework': 'llama_index', 'title': 'Blueprint for an AI Bill of Rights'}, 'metadata for citation 3': {'page': '56', 'title': 'Blueprint for an AI Bill of Rights', 'lang': 'eng', 'section': '4', 'offset': '1', 'len': '140', 'CreationDate': '1663695035', 'Producer': 'iLovePDF', 'Title': 'Blueprint for an AI Bill of Rights', 'Creator': 'Adobe Illustrator 26.3 (Macintosh)', 'ModDate': '1664808078', 'name': 'AI bill of rights', 'year': '2022', 'framework': 'llama_index'}, 'metadata for citation 2': {'page': '56', 'title': 'Blueprint for an AI Bill of Rights', 'lang': 'eng', 'section': '4', 'offset': '9099', 'len': '689', 'CreationDate': '1663695035', 'Producer': 'iLovePDF', 'Title': 'Blueprint for an AI Bill of Rights', 'Creator': 'Adobe Illustrator 26.3 (Macintosh)', 'ModDate': '1664808078', 'name': 'AI bill of rights', 'year': '2022', 'framework': 'llama_index'}, 'metadata for citation 5': {'page': '1', 'lang': 'eng', 'section': '1', 'offset': '52952', 'len': '1025', 'CreationDate': '1663695035', 'Producer': 'iLovePDF', 'Title': 'Blueprint for an AI Bill of Rights', 'Creator': 'Adobe Illustrator 26.3 (Macintosh)', 'ModDate': '1664808078', 'name': 'AI bill of rights', 'year': '2022', 'framework': 'llama_index', 'title': 'Blueprint for an AI Bill of Rights'}}, 'factual_consistency': 0.41484985}
> Running step fe08e033-f8d6-4596-af28-68c156737665. Step input: None
Thought: I can answer without using any more tools. I'll use the user's language to answer
Answer: The benefits and risks of AI are widely discussed among experts, and here is a summary based on the provided sources:

**Benefits of AI:**
- AI has the potential to transform lives by improving various aspects of daily living.
- It can lead to the development of better and more innovative infrastructure.
- When integrated thoughtfully and responsibly, AI can enhance community development, drawing lessons from urban planning.

**Risks of AI:**
- AI systems can result in biased data and discriminatory outcomes.
- The decision-making processes of AI can be opaque, leading to a lack of public trust and understanding.
- AI poses risks to individuals, organizations, and society, necessitating the development of risk management frameworks.

**Addressing Risks and Benefits:**
- The White House Office of Science and Technology Policy has engaged in a process to gather input from experts and stakeholders on algorithmic and data-driven harms.
- The Blueprint for an AI Bill of Rights emphasizes the need for safe and effective systems, transparency, and participatory design.
- The industry is working on innovative solutions to mitigate AI risks, including risk assessments, auditing mechanisms, and stakeholder engagement.
- The National Institute of Standards and Technology (NIST) is developing a risk management framework to manage AI-related risks effectively.

Overall, while AI offers significant benefits, it also presents risks that require careful management and responsible integration into society.
The benefits and risks of AI are widely discussed among experts, and here is a summary based on the provided sources:

**Benefits of AI:**
- AI has the potential to transform lives by improving various aspects of daily living.
- It can lead to the development of better and more innovative infrastructure.
- When integrated thoughtfully and responsibly, AI can enhance community development, drawing lessons from urban planning.

**Risks of AI:**
- AI systems can result in biased data and discriminatory outcomes.
- The decision-making processes of AI can be opaque, leading to a lack of public trust and understanding.
- AI poses risks to individuals, organizations, and society, necessitating the development of risk management frameworks.

**Addressing Risks and Benefits:**
- The White House Office of Science and Technology Policy has engaged in a process to gather input from experts and stakeholders on algorithmic and data-driven harms.
- The Blueprint for an AI Bill of Rights emphasizes the need for safe and effective systems, transparency, and participatory design.
- The industry is working on innovative solutions to mitigate AI risks, including risk assessments, auditing mechanisms, and stakeholder engagement.
- The National Institute of Standards and Technology (NIST) is developing a risk management framework to manage AI-related risks effectively.

Overall, while AI offers significant benefits, it also presents risks that require careful management and responsible integration into society.