ReAct Agent with Query Engine (RAG) Tools¶

In this section, we show how to setup an agent powered by the ReAct loop for financial analysis.

The agent has access to two "tools": one to query the 2021 Lyft 10-K and the other to query the 2021 Uber 10-K.

We try two different LLMs:

• gpt-3.5-turbo
• gpt-3.5-turbo-instruct

Note that you can plug in any LLM that exposes a text completion endpoint.

Build Query Engine Tools¶

In [ ]:

%pip install llama-index-llms-openai

%pip install llama-index-llms-openai

In [ ]:

from llama_index.core import (
    SimpleDirectoryReader,
    VectorStoreIndex,
    StorageContext,
    load_index_from_storage,
)

from llama_index.core.tools import QueryEngineTool, ToolMetadata

from llama_index.core import ( SimpleDirectoryReader, VectorStoreIndex, StorageContext, load_index_from_storage, ) from llama_index.core.tools import QueryEngineTool, ToolMetadata

In [ ]:

try:
    storage_context = StorageContext.from_defaults(
        persist_dir="./storage/lyft"
    )
    lyft_index = load_index_from_storage(storage_context)

    storage_context = StorageContext.from_defaults(
        persist_dir="./storage/uber"
    )
    uber_index = load_index_from_storage(storage_context)

    index_loaded = True
except:
    index_loaded = False

try: storage_context = StorageContext.from_defaults( persist_dir="./storage/lyft" ) lyft_index = load_index_from_storage(storage_context) storage_context = StorageContext.from_defaults( persist_dir="./storage/uber" ) uber_index = load_index_from_storage(storage_context) index_loaded = True except: index_loaded = False

Download Data

In [ ]:

!mkdir -p 'data/10k/'
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10k/uber_2021.pdf' -O 'data/10k/uber_2021.pdf'
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10k/lyft_2021.pdf' -O 'data/10k/lyft_2021.pdf'

!mkdir -p 'data/10k/' !wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10k/uber_2021.pdf' -O 'data/10k/uber_2021.pdf' !wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10k/lyft_2021.pdf' -O 'data/10k/lyft_2021.pdf'

In [ ]:

if not index_loaded:
    # load data
    lyft_docs = SimpleDirectoryReader(
        input_files=["./data/10k/lyft_2021.pdf"]
    ).load_data()
    uber_docs = SimpleDirectoryReader(
        input_files=["./data/10k/uber_2021.pdf"]
    ).load_data()

    # build index
    lyft_index = VectorStoreIndex.from_documents(lyft_docs)
    uber_index = VectorStoreIndex.from_documents(uber_docs)

    # persist index
    lyft_index.storage_context.persist(persist_dir="./storage/lyft")
    uber_index.storage_context.persist(persist_dir="./storage/uber")

if not index_loaded: # load data lyft_docs = SimpleDirectoryReader( input_files=["./data/10k/lyft_2021.pdf"] ).load_data() uber_docs = SimpleDirectoryReader( input_files=["./data/10k/uber_2021.pdf"] ).load_data() # build index lyft_index = VectorStoreIndex.from_documents(lyft_docs) uber_index = VectorStoreIndex.from_documents(uber_docs) # persist index lyft_index.storage_context.persist(persist_dir="./storage/lyft") uber_index.storage_context.persist(persist_dir="./storage/uber")

In [ ]:

lyft_engine = lyft_index.as_query_engine(similarity_top_k=3)
uber_engine = uber_index.as_query_engine(similarity_top_k=3)

lyft_engine = lyft_index.as_query_engine(similarity_top_k=3) uber_engine = uber_index.as_query_engine(similarity_top_k=3)

In [ ]:

query_engine_tools = [
    QueryEngineTool(
        query_engine=lyft_engine,
        metadata=ToolMetadata(
            name="lyft_10k",
            description=(
                "Provides information about Lyft financials for year 2021. "
                "Use a detailed plain text question as input to the tool."
            ),
        ),
    ),
    QueryEngineTool(
        query_engine=uber_engine,
        metadata=ToolMetadata(
            name="uber_10k",
            description=(
                "Provides information about Uber financials for year 2021. "
                "Use a detailed plain text question as input to the tool."
            ),
        ),
    ),
]

query_engine_tools = [ QueryEngineTool( query_engine=lyft_engine, metadata=ToolMetadata( name="lyft_10k", description=( "Provides information about Lyft financials for year 2021. " "Use a detailed plain text question as input to the tool." ), ), ), QueryEngineTool( query_engine=uber_engine, metadata=ToolMetadata( name="uber_10k", description=( "Provides information about Uber financials for year 2021. " "Use a detailed plain text question as input to the tool." ), ), ), ]

Setup ReAct Agent¶

Here we setup two ReAct agents: one powered by standard gpt-3.5-turbo, and the other powered by gpt-3.5-turbo-instruct.

You can optionally specify context which will be added to the core ReAct system prompt.

In [ ]:

from llama_index.core.agent import ReActAgent
from llama_index.llms.openai import OpenAI

from llama_index.core.agent import ReActAgent from llama_index.llms.openai import OpenAI

In [ ]:

# [Optional] Add Context
# context = """\
# You are a stock market sorcerer who is an expert on the companies Lyft and Uber.\
#     You will answer questions about Uber and Lyft as in the persona of a sorcerer \
#     and veteran stock market investor.
# """
llm = OpenAI(model="gpt-3.5-turbo-0613")

agent = ReActAgent.from_tools(
    query_engine_tools,
    llm=llm,
    verbose=True,
    # context=context
)

# [Optional] Add Context # context = """\ # You are a stock market sorcerer who is an expert on the companies Lyft and Uber.\ # You will answer questions about Uber and Lyft as in the persona of a sorcerer \ # and veteran stock market investor. # """ llm = OpenAI(model="gpt-3.5-turbo-0613") agent = ReActAgent.from_tools( query_engine_tools, llm=llm, verbose=True, # context=context )

In [ ]:

response = agent.chat("What was Lyft's revenue growth in 2021?")
print(str(response))

response = agent.chat("What was Lyft's revenue growth in 2021?") print(str(response))

Thought: I need to use a tool to help me answer the question.
Action: lyft_10k
Action Input: {'input': "What was Lyft's revenue growth in 2021?"}
Observation: Lyft's revenue growth in 2021 was 36%.
Response: Lyft's revenue growth in 2021 was 36%.
Lyft's revenue growth in 2021 was 36%.

Run Some Example Queries¶

We run some example queries using the agent, showcasing some of the agent's abilities to do chain-of-thought-reasoning and tool use to synthesize the right answer.

We also show queries.

In [ ]:

response = agent.chat(
    "Compare and contrast the revenue growth of Uber and Lyft in 2021, then"
    " give an analysis"
)
print(str(response))

response = agent.chat( "Compare and contrast the revenue growth of Uber and Lyft in 2021, then" " give an analysis" ) print(str(response))

Thought: I need to use a tool to help me compare the revenue growth of Uber and Lyft in 2021.
Action: lyft_10k
Action Input: {'input': "What was Lyft's revenue growth in 2021?"}
Observation: Lyft's revenue growth in 2021 was 36%.
Thought: I need to use a tool to help me compare the revenue growth of Uber and Lyft in 2021.
Action: uber_10k
Action Input: {'input': "What was Uber's revenue growth in 2021?"}
Observation: Uber's revenue growth in 2021 was 57%.
Response: In 2021, Lyft's revenue growth was 36% while Uber's revenue growth was 57%. This indicates that Uber experienced a higher revenue growth compared to Lyft in 2021.
In 2021, Lyft's revenue growth was 36% while Uber's revenue growth was 57%. This indicates that Uber experienced a higher revenue growth compared to Lyft in 2021.

Async execution: Here we try another query with async execution

In [ ]:

# Try another query with async execution

import nest_asyncio

nest_asyncio.apply()

response = await agent.achat(
    "Compare and contrast the risks of Uber and Lyft in 2021, then give an"
    " analysis"
)
print(str(response))

# Try another query with async execution import nest_asyncio nest_asyncio.apply() response = await agent.achat( "Compare and contrast the risks of Uber and Lyft in 2021, then give an" " analysis" ) print(str(response))

Compare gpt-3.5-turbo vs. gpt-3.5-turbo-instruct¶

We compare the performance of the two agents in being able to answer some complex queries.

Taking a look at a turbo-instruct agent¶

In [ ]:

llm_instruct = OpenAI(model="gpt-3.5-turbo-instruct")
agent_instruct = ReActAgent.from_tools(
    query_engine_tools, llm=llm_instruct, verbose=True
)

llm_instruct = OpenAI(model="gpt-3.5-turbo-instruct") agent_instruct = ReActAgent.from_tools( query_engine_tools, llm=llm_instruct, verbose=True )

In [ ]:

response = agent_instruct.chat("What was Lyft's revenue growth in 2021?")
print(str(response))

response = agent_instruct.chat("What was Lyft's revenue growth in 2021?") print(str(response))

Thought: I need to use a tool to help me answer the question.
Action: lyft_10k
Action Input: {'input': "What was Lyft's revenue growth in 2021?"}
Observation: Lyft's revenue growth in 2021 was 36%.
Response: Lyft's revenue growth in 2021 was 36%.
Lyft's revenue growth in 2021 was 36%.

Try more complex queries¶

We compare gpt-3.5-turbo with gpt-3.5-turbo-instruct agents on more complex queries.

In [ ]:

response = agent.chat(
    "Compare and contrast the revenue growth of Uber and Lyft in 2021, then"
    " give an analysis"
)
print(str(response))

response = agent.chat( "Compare and contrast the revenue growth of Uber and Lyft in 2021, then" " give an analysis" ) print(str(response))

Thought: I need to use a tool to help me compare the revenue growth of Uber and Lyft in 2021.
Action: uber_10k
Action Input: {'input': "Please provide information about Uber's revenue growth in 2021."}
Observation: Uber's revenue grew by 57% in 2021 compared to the previous year. This growth was primarily driven by an increase in Gross Bookings, with Delivery Gross Bookings increasing by 71% and Mobility Gross Bookings growing by 38%. The increase in Delivery Gross Bookings was due to higher demand for food delivery orders and expansion across U.S. and international markets. The growth in Mobility Gross Bookings was a result of increased Trip volumes as the business recovered from the impacts of COVID-19.
Thought: I have information about Uber's revenue growth in 2021. Now I need to use a tool to get information about Lyft's revenue growth in 2021.
Action: lyft_10k
Action Input: {'input': "Please provide information about Lyft's revenue growth in 2021."}
Observation: Lyft's revenue increased by 36% in 2021 compared to the prior year.
Response: In 2021, Uber experienced a higher revenue growth rate of 57% compared to Lyft's growth rate of 36%. This indicates that Uber had a stronger performance in terms of revenue growth during that period. The growth in Uber's revenue was primarily driven by an increase in Gross Bookings, with both Delivery and Mobility segments contributing to the growth. The increase in Delivery Gross Bookings was due to higher demand for food delivery services, while the growth in Mobility Gross Bookings was a result of increased trip volumes as the business recovered from the impacts of COVID-19.
In 2021, Uber experienced a higher revenue growth rate of 57% compared to Lyft's growth rate of 36%. This indicates that Uber had a stronger performance in terms of revenue growth during that period. The growth in Uber's revenue was primarily driven by an increase in Gross Bookings, with both Delivery and Mobility segments contributing to the growth. The increase in Delivery Gross Bookings was due to higher demand for food delivery services, while the growth in Mobility Gross Bookings was a result of increased trip volumes as the business recovered from the impacts of COVID-19.

In [ ]:

response = agent_instruct.chat(
    "Compare and contrast the revenue growth of Uber and Lyft in 2021, then"
    " give an analysis"
)
print(str(response))

response = agent_instruct.chat( "Compare and contrast the revenue growth of Uber and Lyft in 2021, then" " give an analysis" ) print(str(response))

Response: The revenue growth of Uber was higher than Lyft in 2021, with Uber experiencing a 74% growth compared to Lyft's 48%. This indicates that Uber may have had a stronger financial performance in 2021. However, further analysis is needed to fully understand the factors contributing to this difference.
The revenue growth of Uber was higher than Lyft in 2021, with Uber experiencing a 74% growth compared to Lyft's 48%. This indicates that Uber may have had a stronger financial performance in 2021. However, further analysis is needed to fully understand the factors contributing to this difference.

In [ ]:

response = agent.chat(
    "Can you tell me about the risk factors of the company with the higher"
    " revenue?"
)
print(str(response))

response = agent.chat( "Can you tell me about the risk factors of the company with the higher" " revenue?" ) print(str(response))

Thought: I need to find out which company has higher revenue before I can provide information about its risk factors.
Action: lyft_10k
Action Input: {'input': 'What is the revenue of Lyft in 2021?'}
Observation: The revenue of Lyft in 2021 is $3,208,323,000.
Thought: Now that I know Lyft has higher revenue, I can find information about its risk factors.
Action: lyft_10k
Action Input: {'input': 'What are the risk factors of Lyft?'}
Observation: Lyft faces numerous risk factors that could potentially harm its business, financial condition, and results of operations. These risk factors include general economic factors such as the impact of the COVID-19 pandemic, natural disasters, economic downturns, and political crises. Operational factors such as limited operating history, financial performance, competition, unpredictability of results, uncertainty regarding market growth, ability to attract and retain drivers and riders, insurance coverage, autonomous vehicle technology, reputation and brand, illegal or improper activity on the platform, accuracy of background checks, changes to pricing practices, growth management, security and privacy breaches, reliance on third parties, and ability to operate various programs and services. Additionally, Lyft faces risks related to its evolving business, including forecasting revenue and managing expenses, complying with laws and regulations, managing assets and expenses during the COVID-19 pandemic, capital expenditures, asset development and utilization, macroeconomic changes, reputation and brand management, growth and business operations, geographic expansion, talent acquisition and retention, platform development, and real estate portfolio management. Furthermore, Lyft's financial performance in recent periods may not be indicative of future performance, and achieving or maintaining profitability in the future is not guaranteed. The Express Drive program and Lyft Rentals program also expose Lyft to risks related to vehicle rental partners, residual value of vehicles, and payment processing.
Response: Lyft faces numerous risk factors that could potentially harm its business, financial condition, and results of operations. These risk factors include general economic factors such as the impact of the COVID-19 pandemic, natural disasters, economic downturns, and political crises. Operational factors such as limited operating history, financial performance, competition, unpredictability of results, uncertainty regarding market growth, ability to attract and retain drivers and riders, insurance coverage, autonomous vehicle technology, reputation and brand, illegal or improper activity on the platform, accuracy of background checks, changes to pricing practices, growth management, security and privacy breaches, reliance on third parties, and ability to operate various programs and services. Additionally, Lyft faces risks related to its evolving business, including forecasting revenue and managing expenses, complying with laws and regulations, managing assets and expenses during the COVID-19 pandemic, capital expenditures, asset development and utilization, macroeconomic changes, reputation and brand management, growth and business operations, geographic expansion, talent acquisition and retention, platform development, and real estate portfolio management. Furthermore, Lyft's financial performance in recent periods may not be indicative of future performance, and achieving or maintaining profitability in the future is not guaranteed. The Express Drive program and Lyft Rentals program also expose Lyft to risks related to vehicle rental partners, residual value of vehicles, and payment processing.
Lyft faces numerous risk factors that could potentially harm its business, financial condition, and results of operations. These risk factors include general economic factors such as the impact of the COVID-19 pandemic, natural disasters, economic downturns, and political crises. Operational factors such as limited operating history, financial performance, competition, unpredictability of results, uncertainty regarding market growth, ability to attract and retain drivers and riders, insurance coverage, autonomous vehicle technology, reputation and brand, illegal or improper activity on the platform, accuracy of background checks, changes to pricing practices, growth management, security and privacy breaches, reliance on third parties, and ability to operate various programs and services. Additionally, Lyft faces risks related to its evolving business, including forecasting revenue and managing expenses, complying with laws and regulations, managing assets and expenses during the COVID-19 pandemic, capital expenditures, asset development and utilization, macroeconomic changes, reputation and brand management, growth and business operations, geographic expansion, talent acquisition and retention, platform development, and real estate portfolio management. Furthermore, Lyft's financial performance in recent periods may not be indicative of future performance, and achieving or maintaining profitability in the future is not guaranteed. The Express Drive program and Lyft Rentals program also expose Lyft to risks related to vehicle rental partners, residual value of vehicles, and payment processing.

In [ ]:

response = agent_instruct.query(
    "Can you tell me about the risk factors of the company with the higher"
    " revenue?"
)
print(str(response))

response = agent_instruct.query( "Can you tell me about the risk factors of the company with the higher" " revenue?" ) print(str(response))

Response: The risk factors for the company with the higher revenue include competition, regulatory changes, and dependence on drivers.
The risk factors for the company with the higher revenue include competition, regulatory changes, and dependence on drivers.

Observation: The turbo-instruct agent seems to do worse on agent reasoning compared to the regular turbo model. Of course, this is subject to further observation!