Vectara Managed Index¶

In this notebook we are going to show how to use Vectara with LlamaIndex.

Vectara is the trusted AI Assistant and Agent platform which focuses on enterprise readiness for mission-critical applications.

Vectara provides an end-to-end managed service for Retrieval Augmented Generation or RAG, which includes:

1. An integrated API for processing input data, including text extraction from documents and ML-based chunking.

2. The state-of-the-art Boomerang embeddings model. Each text chunk is encoded into a vector embedding using Boomerang, and stored in the Vectara internal knowledge (vector+text) store. Thus, when using Vectara with LlamaIndex you do not need to call a separate embedding model - this happens automatically within the Vectara backend.

3. A query service that automatically encodes the query into embeddings and retrieves the most relevant text segmentsthrough hybrid search and a variety of reranking strategies, including a multilingual reranker, maximal marginal relevance (MMR) reranker, user-defined function reranker, and a chain reranker that provides a way to chain together multiple reranking methods to achieve better control over the reranking, combining the strengths of various reranking methods.

4. An option to create a generative summary with a wide selection of LLM summarizers (including Vectara's Mockingbird, trained specifically for RAG-based tasks), based on the retrieved documents, including citations.

See the Vectara API documentation for more information on how to use the API.

The main benefits of using Vectara RAG-as-a-service to build your application are:

• Accuracy and Quality: Vectara provides an end-to-end platform that focuses on eliminating hallucinations, reducing bias, and safeguarding copyright integrity.
• Security: Vectara's platform provides acess control--protecting against prompt injection attacks--and meets SOC2 and HIPAA compliance.
• Explainability: Vectara makes it easy to troubleshoot bad results by clearly explaining rephrased queries, LLM prompts, retrieved results, and agent actions.

Getting Started¶

If you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙.

In [ ]:

!pip install llama-index llama-index-indices-managed-vectara

!pip install llama-index llama-index-indices-managed-vectara

To get started with Vectara, sign up (if you haven't already) and follow our quickstart guide to create a corpus and an API key.

Once you have these, you can provide them as environment variables VECTARA_CUSTOMER_ID, VECTARA_CORPUS_ID, and VECTARA_API_KEY. Make sure your API key has both query and index permissions.

RAG with LlamaIndex and Vectara¶

There are a few ways you can index your data into Vectara, including:

1. With the from_documents() or insert_file() methods of VectaraIndex
2. Uploading files directly in the Vectara console
3. Using Vectara's FILE_UPLOAD or standard indexing APIs
4. Using vectara-ingest, an open source crawler/indexer project
5. Using one of our ingest integration partners like Airbyte, Unstructured or DataVolo.

For this purpose, we will use a simple set of small documents, so using VectaraIndex directly for the ingest is good enough.

Let's ingest the "AI bill of rights" document into our new corpus.

In [ ]:

from llama_index.indices.managed.vectara import VectaraIndex
import requests

url = "https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf"
response = requests.get(url)
local_path = "ai-bill-of-rights.pdf"
with open(local_path, "wb") as file:
    file.write(response.content)

index = VectaraIndex()
index.insert_file(
    local_path, metadata={"name": "AI bill of rights", "year": 2022}
)

from llama_index.indices.managed.vectara import VectaraIndex import requests url = "https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf" response = requests.get(url) local_path = "ai-bill-of-rights.pdf" with open(local_path, "wb") as file: file.write(response.content) index = VectaraIndex() index.insert_file( local_path, metadata={"name": "AI bill of rights", "year": 2022} )

Out[ ]:

'ai-bill-of-rights.pdf'

Running single queries with Vectara Query Engine¶

Now that we've uploaded the document (or if documents have been uploaded previously) we can go and ask questions directly in LlamaIndex. This activates Vectara's RAG pipeline.

To use Vectara's internal LLM for summarization, make sure you specify summary_enabled=True when you generate the Query engine. Here's an example:

In [ ]:

questions = [
    "What are the risks of AI?",
    "What should we do to prevent bad actors from using AI?",
    "What are the benefits?",
]

questions = [ "What are the risks of AI?", "What should we do to prevent bad actors from using AI?", "What are the benefits?", ]

In [ ]:

qe = index.as_query_engine(
    summary_enabled=True, summary_prompt_name="mockingbird-1.0-2024-07-16"
)
qe.query(questions[0]).response

qe = index.as_query_engine( summary_enabled=True, summary_prompt_name="mockingbird-1.0-2024-07-16" ) qe.query(questions[0]).response

Out[ ]:

"The risks of AI include biased data and discriminatory outcomes, opaque decision-making processes, and lack of public trust and understanding of algorithmic systems [1]. These risks can lead to harm to individuals and communities, including the potential for meaningful impact on people's rights, opportunities, or access [6]. To mitigate these risks, it is essential to identify and address potential harms before deployment, and to engage with impacted communities to understand the potential harms of technologies and build protection by design into future systems [1][6]. Additionally, strong safety regulations and measures to address harms when they occur can enhance innovation in the context of complex technologies [2]. The development of technical standards and practices tailored for particular sectors and contexts can also help to ensure safe and effective AI systems [7]."

If you want the response to be returned in streaming mode, simply set streaming=True

In [ ]:

qe = index.as_query_engine(
    summary_enabled=True,
    summary_prompt_name="mockingbird-1.0-2024-07-16",
    streaming=True,
)
response = qe.query(questions[0])

for chunk in response.response_gen:
    print(chunk.delta or "", end="", flush=True)

qe = index.as_query_engine( summary_enabled=True, summary_prompt_name="mockingbird-1.0-2024-07-16", streaming=True, ) response = qe.query(questions[0]) for chunk in response.response_gen: print(chunk.delta or "", end="", flush=True)

The risks of AI include biased data and discriminatory outcomes, opaque decision-making processes, and lack of public trust and understanding of algorithmic systems [1]. These risks can lead to harm to individuals and communities, including the potential for meaningful impact on people's rights, opportunities, or access [6]. To mitigate these risks, it is essential to identify and address potential harms before deployment, and to engage with impacted communities to understand the potential harms of technologies and build protection by design into future systems [1][6]. Additionally, strong safety regulations and measures to address harms when they occur can enhance innovation in the context of complex technologies [2]. The development of technical standards and practices tailored for particular sectors and contexts can also help to ensure safe and effective AI systems [7].

Using Vectara Chat¶

Vectara also supports a simple chat mode. In this mode the chat history is maintained by Vectara and so you don't have to worry about it. To use it simple call as_chat_engine.

(Chat mode always uses Vectara's summarization so you don't have to explicitly specify summary_enabled=True like before)

In [ ]:

ce = index.as_chat_engine()

ce = index.as_chat_engine()

In [ ]:

for q in questions:
    print(f"Question: {q}\n")
    response = ce.chat(q).response
    print(f"Response: {response}\n")

for q in questions: print(f"Question: {q}\n") response = ce.chat(q).response print(f"Response: {response}\n")

Question: What are the risks of AI?

Response: The risks of AI include potential biased data leading to discriminatory outcomes, opaque decision-making processes, lack of public trust, and understanding of algorithmic systems. Risks also involve safety concerns, such as AI systems violating safety regulations, causing harm, and impacting individuals' rights and opportunities. Furthermore, the misuse of AI, like AI-enabled technology for creating non-consensual images, poses significant risks, affecting individuals' lives and well-being. It is crucial to identify and mitigate risks before deployment, prioritize high-impact risks, and ensure that automated systems do not violate safety standards or harm individuals. Stakeholder engagement, transparency, and ongoing monitoring are essential to address these risks effectively.

Question: What should we do to prevent bad actors from using AI?

Response: To prevent bad actors from using AI, we should implement safeguards such as ensuring systems are safe, effective, and respect privacy [1]. Additionally, it is crucial to adhere to principles like legality, transparency, and accountability in AI development and usage [2]. Moreover, best practices should be followed to protect against algorithmic discrimination and ensure fairness in all aspects of people's lives [6]. Lastly, it is essential to evaluate, protect, and redress harms at both individual and community levels, especially for underserved communities, to promote equity and fair treatment for all [5].

Question: What are the benefits?

Response: The benefits of AI include the potential to build innovative infrastructure, improve Americans' lives, provide faster customer care, enhance decision-making in various sectors, revolutionize industries, and make life better for everyone. AI can help farmers grow food efficiently, predict storm paths, identify diseases, and drive important decisions globally. Additionally, AI can be used to protect individuals from unsafe systems and ensure the accountability, transparency, and reliability of AI technologies. Overall, AI holds the promise to positively impact society and improve various aspects of life.

Of course streaming works as well with Chat:

In [ ]:

ce = index.as_chat_engine(streaming=True)

ce = index.as_chat_engine(streaming=True)

In [ ]:

response = ce.stream_chat("Will artificial intelligence rule the government?")
for chunk in response.chat_stream:
    print(chunk.delta or "", end="", flush=True)

response = ce.stream_chat("Will artificial intelligence rule the government?") for chunk in response.chat_stream: print(chunk.delta or "", end="", flush=True)

Artificial intelligence will not rule the government. The government is implementing principles and guidelines to ensure the ethical and responsible use of AI in various sectors, including the federal government. These measures focus on transparency, accountability, safety, and adherence to national values, ensuring that AI is used in a manner that upholds civil rights, democratic values, and national security. Sector-specific guidance will continue to evolve to address specific AI applications, but overall, the government remains committed to safeguarding civil liberties and democratic principles in the age of artificial intelligence.

Agentic RAG¶

Vectara also has its own package, vectara-agentic, built on top of many features from LlamaIndex to easily implement agentic RAG applications. It allows you to create your own AI assistant with RAG query tools and other custom tools, such as making API calls to retrieve information from financial websites. You can find the full documentation for vectara-agentic here.

Let's create a ReAct Agent with a single RAG tool using vectara-agentic (to create a ReAct agent, specify VECTARA_AGENTIC_AGENT_TYPE as "REACT" in your environment).

Vectara does not yet have an LLM capable of acting as an agent for planning and tool use, so we will need to use another LLM as the driver of the agent resoning.

In this demo, we are using OpenAI's GPT4o. Please make sure you have OPENAI_API_KEY defined in your environment or specify another LLM with the corresponding key (for the full list of supported LLMs, check out our documentation for setting up your environment).

In [ ]:

!pip install -U vectara-agentic

!pip install -U vectara-agentic

In [ ]:

from vectara_agentic.agent import Agent
from IPython.display import display, Markdown

agent = Agent.from_corpus(
    tool_name="query_ai",
    data_description="AI regulations",
    assistant_specialty="artificial intelligence",
    vectara_summary_num_results=5,
    vectara_summarizer="mockingbird-1.0-2024-07-16",
    vectara_reranker="mmr",
    vectara_rerank_k=50,
    verbose=True,
)

response = agent.chat(
    "What are the risks of AI? What are the benefits? Compare and contrast and provide a summary with arguments for and against from experts."
)

display(Markdown(response))

from vectara_agentic.agent import Agent from IPython.display import display, Markdown agent = Agent.from_corpus( tool_name="query_ai", data_description="AI regulations", assistant_specialty="artificial intelligence", vectara_summary_num_results=5, vectara_summarizer="mockingbird-1.0-2024-07-16", vectara_reranker="mmr", vectara_rerank_k=50, verbose=True, ) response = agent.chat( "What are the risks of AI? What are the benefits? Compare and contrast and provide a summary with arguments for and against from experts." ) display(Markdown(response))

Initializing vectara-agentic version 0.1.16...
No observer set.
> Running step 26a91fbd-0027-42af-96f8-fcf19bd9bf06. Step input: What are the risks of AI? What are the benefits? Compare and contrast and provide a summary with arguments for and against from experts.
Thought: The current language of the user is: English. I need to use a tool to help me answer the question.
Action: query_ai
Action Input: {'query': 'risks and benefits of AI, expert opinions, arguments for and against'}
Observation: 
                    Response: '''Here is a summary of the risks and benefits of AI, expert opinions, and arguments for and against, based on the provided sources:

**Risks of AI:**

* Bias and discriminatory outcomes due to biased data [1]
* Opaque decision-making processes [1]
* Lack of public trust and understanding of algorithmic systems [1]
* Potential for harm to individuals, organizations, and society [3]

**Benefits of AI:**

* Potential to build better and more innovative infrastructure [2]
* Ability to mitigate risks to safety and efficacy of AI systems [3]
* Transformative potential to improve Americans' lives [4]

**Expert Opinions:**

* Emphasis on designing non-discriminatory technology, explainable AI, and human-computer interaction with community participation [1]
* Importance of placing trust in people, not technologies, and engaging with impacted communities to understand potential harms [1]
* Need for ongoing transparency, value-sensitive and participatory design, and public consultation [1]

**Arguments for and Against:**

* Strong safety regulations and measures to address harms can enhance innovation in complex technologies [3]
* Balancing ownership rights, use rights, and community health, safety, and welfare is critical in the integration of technology [2]
* Examining lessons learned from urban planning can inform the development of AI-enabled systems [2]

Overall, the sources highlight the importance of designing AI systems that are non-discriminatory, transparent, and trustworthy, while also considering the potential benefits and risks of AI.'''
                    References:
                    [1]: page='56'; title='Blueprint for an AI Bill of Rights'; section='4'; CreationDate='1663695035'; Producer='iLovePDF'; Title='Blueprint for an AI Bill of Rights'; Creator='Adobe Illustrator 26.3 (Macintosh)'; ModDate='1664808078'; name='AI bill of rights'; year='2022'; framework='llama_index'.
[2]: page='56'; title='Blueprint for an AI Bill of Rights'; section='4'; CreationDate='1663695035'; Producer='iLovePDF'; Title='Blueprint for an AI Bill of Rights'; Creator='Adobe Illustrator 26.3 (Macintosh)'; ModDate='1664808078'; name='AI bill of rights'; year='2022'; framework='llama_index'.
[3]: page='1'; section='1'; CreationDate='1663695035'; Producer='iLovePDF'; Title='Blueprint for an AI Bill of Rights'; Creator='Adobe Illustrator 26.3 (Macintosh)'; ModDate='1664808078'; name='AI bill of rights'; year='2022'; framework='llama_index'; title='Blueprint for an AI Bill of Rights'.
[4]: page='1'; section='1'; CreationDate='1663695035'; Producer='iLovePDF'; Title='Blueprint for an AI Bill of Rights'; Creator='Adobe Illustrator 26.3 (Macintosh)'; ModDate='1664808078'; name='AI bill of rights'; year='2022'; framework='llama_index'; title='Blueprint for an AI Bill of Rights'.

                
> Running step f9d86544-5c76-48c5-8efd-02c154945e08. Step input: None
Thought: I can answer without using any more tools. I'll use the user's language to answer
Answer: Here is a summary of the risks and benefits of AI, along with expert opinions and arguments for and against:

**Risks of AI:**
- AI systems can produce biased and discriminatory outcomes if they are trained on biased data [1].
- The decision-making processes of AI can be opaque, making it difficult to understand how conclusions are reached [1].
- There is a lack of public trust and understanding of algorithmic systems, which can lead to skepticism and resistance [1].
- AI has the potential to cause harm to individuals, organizations, and society if not properly managed [3].

**Benefits of AI:**
- AI has the potential to build better and more innovative infrastructure, enhancing efficiency and effectiveness [2].
- It can help mitigate risks to the safety and efficacy of various systems, improving overall safety standards [3].
- AI holds transformative potential to improve the quality of life, offering advancements in various sectors [4].

**Expert Opinions:**
- Experts emphasize the need for designing non-discriminatory technology, explainable AI, and fostering human-computer interaction with community participation [1].
- Trust should be placed in people rather than technologies, and it is crucial to engage with impacted communities to understand potential harms [1].
- Ongoing transparency, value-sensitive and participatory design, and public consultation are necessary for the responsible development of AI [1].

**Arguments for and Against:**
- Implementing strong safety regulations and measures to address potential harms can enhance innovation in complex technologies [3].
- Balancing ownership rights, use rights, and community health, safety, and welfare is critical in the integration of AI technology [2].
- Lessons learned from urban planning can inform the development of AI-enabled systems, ensuring they are beneficial and sustainable [2].

Overall, the sources highlight the importance of designing AI systems that are non-discriminatory, transparent, and trustworthy, while also considering the potential benefits and risks of AI.

References:
[1]: Blueprint for an AI Bill of Rights
[2]: Blueprint for an AI Bill of Rights
[3]: Blueprint for an AI Bill of Rights
[4]: Blueprint for an AI Bill of Rights
Time taken: 21.543328046798706

Here is a summary of the risks and benefits of AI, along with expert opinions and arguments for and against:

Risks of AI:

• AI systems can produce biased and discriminatory outcomes if they are trained on biased data [1].
• The decision-making processes of AI can be opaque, making it difficult to understand how conclusions are reached [1].
• There is a lack of public trust and understanding of algorithmic systems, which can lead to skepticism and resistance [1].
• AI has the potential to cause harm to individuals, organizations, and society if not properly managed [3].

Benefits of AI:

• AI has the potential to build better and more innovative infrastructure, enhancing efficiency and effectiveness [2].
• It can help mitigate risks to the safety and efficacy of various systems, improving overall safety standards [3].
• AI holds transformative potential to improve the quality of life, offering advancements in various sectors [4].

Expert Opinions:

• Experts emphasize the need for designing non-discriminatory technology, explainable AI, and fostering human-computer interaction with community participation [1].
• Trust should be placed in people rather than technologies, and it is crucial to engage with impacted communities to understand potential harms [1].
• Ongoing transparency, value-sensitive and participatory design, and public consultation are necessary for the responsible development of AI [1].

Arguments for and Against:

• Implementing strong safety regulations and measures to address potential harms can enhance innovation in complex technologies [3].
• Balancing ownership rights, use rights, and community health, safety, and welfare is critical in the integration of AI technology [2].
• Lessons learned from urban planning can inform the development of AI-enabled systems, ensuring they are beneficial and sustainable [2].

Overall, the sources highlight the importance of designing AI systems that are non-discriminatory, transparent, and trustworthy, while also considering the potential benefits and risks of AI.

References: [1]: Blueprint for an AI Bill of Rights [2]: Blueprint for an AI Bill of Rights [3]: Blueprint for an AI Bill of Rights [4]: Blueprint for an AI Bill of Rights