Function Calling NVIDIA Agent¶

This notebook shows you how to use our NVIDIA agent, powered by function calling capabilities.

Initial Setup¶

Let's start by importing some simple building blocks.

The main thing we need is:

1. the NVIDIA NIM Endpoint (using our own llama_index LLM class)
2. a place to keep conversation history
3. a definition for tools that our agent can use.

In [ ]:

%pip install --upgrade --quiet llama-index-llms-nvidia

%pip install --upgrade --quiet llama-index-llms-nvidia

DEPRECATION: pytest-httpx 0.21.0 has a non-standard dependency specifier pytest<8.*,>=6.*. pip 24.1 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pytest-httpx or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063

[notice] A new release of pip is available: 24.0 -> 24.2
[notice] To update, run: pip install --upgrade pip
Note: you may need to restart the kernel to use updated packages.

In [ ]:

import getpass
import os

# del os.environ['NVIDIA_API_KEY']  ## delete key and reset
if os.environ.get("NVIDIA_API_KEY", "").startswith("nvapi-"):
    print("Valid NVIDIA_API_KEY already in environment. Delete to reset")
else:
    nvapi_key = getpass.getpass("NVAPI Key (starts with nvapi-): ")
    assert nvapi_key.startswith(
        "nvapi-"
    ), f"{nvapi_key[:5]}... is not a valid key"
    os.environ["NVIDIA_API_KEY"] = nvapi_key

import getpass import os # del os.environ['NVIDIA_API_KEY'] ## delete key and reset if os.environ.get("NVIDIA_API_KEY", "").startswith("nvapi-"): print("Valid NVIDIA_API_KEY already in environment. Delete to reset") else: nvapi_key = getpass.getpass("NVAPI Key (starts with nvapi-): ") assert nvapi_key.startswith( "nvapi-" ), f"{nvapi_key[:5]}... is not a valid key" os.environ["NVIDIA_API_KEY"] = nvapi_key

Valid NVIDIA_API_KEY already in environment. Delete to reset

In [ ]:

from llama_index.llms.nvidia import NVIDIA
from llama_index.core.tools import FunctionTool
from llama_index.embeddings.nvidia import NVIDIAEmbedding

from llama_index.llms.nvidia import NVIDIA from llama_index.core.tools import FunctionTool from llama_index.embeddings.nvidia import NVIDIAEmbedding

Let's define some very simple calculator tools for our agent.

In [ ]:

def multiply(a: int, b: int) -> int:
    """Multiple two integers and returns the result integer"""
    return a * b


multiply_tool = FunctionTool.from_defaults(fn=multiply)

def multiply(a: int, b: int) -> int: """Multiple two integers and returns the result integer""" return a * b multiply_tool = FunctionTool.from_defaults(fn=multiply)

In [ ]:

def add(a: int, b: int) -> int:
    """Add two integers and returns the result integer"""
    return a + b


add_tool = FunctionTool.from_defaults(fn=add)

def add(a: int, b: int) -> int: """Add two integers and returns the result integer""" return a + b add_tool = FunctionTool.from_defaults(fn=add)

Here we initialize a simple NVIDIA agent with calculator functions.

In [ ]:

llm = NVIDIA("meta/llama-3.1-70b-instruct")

llm = NVIDIA("meta/llama-3.1-70b-instruct")

In [ ]:

from llama_index.core.agent import FunctionCallingAgent

agent = FunctionCallingAgent.from_tools(
    [multiply_tool, add_tool],
    llm=llm,
    verbose=True,
)

from llama_index.core.agent import FunctionCallingAgent agent = FunctionCallingAgent.from_tools( [multiply_tool, add_tool], llm=llm, verbose=True, )

Chat¶

In [ ]:

response = agent.chat("What is (121 * 3) + 42?")
print(str(response))

response = agent.chat("What is (121 * 3) + 42?") print(str(response))

Added user message to memory: What is (121 * 3) + 42?
=== Calling Function ===
Calling function: multiply with args: {"a": 121, "b": 3}
=== Function Output ===
363
=== Calling Function ===
Calling function: add with args: {"a": 363, "b": 42}
=== Function Output ===
405
=== LLM Response ===
The answer is 405.
The answer is 405.

In [ ]:

# inspect sources
print(response.sources)

# inspect sources print(response.sources)

[ToolOutput(content='363', tool_name='multiply', raw_input={'args': (), 'kwargs': {'a': 121, 'b': 3}}, raw_output=363, is_error=False), ToolOutput(content='405', tool_name='add', raw_input={'args': (), 'kwargs': {'a': 363, 'b': 42}}, raw_output=405, is_error=False)]

Async Chat¶

In [ ]:

response = await agent.achat("What is 121 * 3?")
print(str(response))

response = await agent.achat("What is 121 * 3?") print(str(response))

Added user message to memory: What is 121 * 3?
=== Calling Function ===
Calling function: multiply with args: {"a": 121, "b": 3}
=== Function Output ===
363
=== LLM Response ===
The answer is 363.
The answer is 363.

Agent with Personality¶

You can specify a system prompt to give the agent additional instruction or personality.

In [ ]:

from llama_index.core.prompts.system import SHAKESPEARE_WRITING_ASSISTANT

from llama_index.core.prompts.system import SHAKESPEARE_WRITING_ASSISTANT

In [ ]:

agent = FunctionCallingAgent.from_tools(
    [multiply_tool, add_tool],
    llm=llm,
    verbose=True,
    system_prompt=SHAKESPEARE_WRITING_ASSISTANT,
)

agent = FunctionCallingAgent.from_tools( [multiply_tool, add_tool], llm=llm, verbose=True, system_prompt=SHAKESPEARE_WRITING_ASSISTANT, )

In [ ]:

response = agent.chat("Hi")
print(response)

response = agent.chat("Hi") print(response)

Added user message to memory: Hi
Fair greeting unto thee, kind sir or madam! 'Tis a pleasure to make thy acquaintance. How doth thy day fare? Doth thou seek inspiration for a tale, a poem, or perhaps a song, penned in the grand style of the Bard himself?

In [ ]:

response = agent.chat("Tell me a story")
print(response)

response = agent.chat("Tell me a story") print(response)

NVIDIA Agent with RAG/Query Engine Tools¶

In [ ]:

!mkdir -p 'data/10k/'
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10k/uber_2021.pdf' -O 'data/10k/uber_2021.pdf'

!mkdir -p 'data/10k/' !wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10k/uber_2021.pdf' -O 'data/10k/uber_2021.pdf'

In [ ]:

from llama_index.core.tools import QueryEngineTool, ToolMetadata
from llama_index.core import SimpleDirectoryReader, VectorStoreIndex

embed_model = NVIDIAEmbedding(model="NV-Embed-QA", truncate="END")

# load data
uber_docs = SimpleDirectoryReader(
    input_files=["./data/10k/uber_2021.pdf"]
).load_data()
# build index
uber_index = VectorStoreIndex.from_documents(
    uber_docs, embed_model=embed_model
)
uber_engine = uber_index.as_query_engine(similarity_top_k=3, llm=llm)
query_engine_tool = QueryEngineTool(
    query_engine=uber_engine,
    metadata=ToolMetadata(
        name="uber_10k",
        description=(
            "Provides information about Uber financials for year 2021. "
            "Use a detailed plain text question as input to the tool."
        ),
    ),
)

from llama_index.core.tools import QueryEngineTool, ToolMetadata from llama_index.core import SimpleDirectoryReader, VectorStoreIndex embed_model = NVIDIAEmbedding(model="NV-Embed-QA", truncate="END") # load data uber_docs = SimpleDirectoryReader( input_files=["./data/10k/uber_2021.pdf"] ).load_data() # build index uber_index = VectorStoreIndex.from_documents( uber_docs, embed_model=embed_model ) uber_engine = uber_index.as_query_engine(similarity_top_k=3, llm=llm) query_engine_tool = QueryEngineTool( query_engine=uber_engine, metadata=ToolMetadata( name="uber_10k", description=( "Provides information about Uber financials for year 2021. " "Use a detailed plain text question as input to the tool." ), ), )

In [ ]:

agent = FunctionCallingAgent.from_tools(
    [query_engine_tool], llm=llm, verbose=True
)

agent = FunctionCallingAgent.from_tools( [query_engine_tool], llm=llm, verbose=True )

In [ ]:

response = agent.chat(
    "Tell me both the risk factors and tailwinds for Uber? Do two parallel tool calls.",
    allow_parallel_tool_calls=True,
)
print(str(response))

response = agent.chat( "Tell me both the risk factors and tailwinds for Uber? Do two parallel tool calls.", allow_parallel_tool_calls=True, ) print(str(response))

ReAct Agent¶

In [ ]:

from llama_index.core.agent import ReActAgent

from llama_index.core.agent import ReActAgent

In [ ]:

agent = ReActAgent.from_tools([multiply_tool, add_tool], llm=llm, verbose=True)

agent = ReActAgent.from_tools([multiply_tool, add_tool], llm=llm, verbose=True)

In [ ]:

response = agent.chat("What is 20+(2*4)? Calculate step by step ")

response = agent.chat("What is 20+(2*4)? Calculate step by step ")

> Running step a61e9980-2b9c-4a78-9950-cabe13827f73. Step input: What is 20+(2*4)? Calculate step by step 
Thought: To calculate 20+(2*4), I need to follow the order of operations (PEMDAS). First, I need to calculate the multiplication part, which is 2*4. I will use the multiply tool to do this.
Action: multiply
Action Input: {'a': 2, 'b': 4}
Observation: 8
> Running step 73418308-49cc-4689-bb39-b83d6a7cf7ac. Step input: None
Thought: Now that I have the result of the multiplication, which is 8, I can proceed to add 20 to it. I will use the add tool to do this.
Action: add
Action Input: {'a': 20, 'b': 8}
Observation: 28
> Running step 73464ecd-b266-47ef-8f1b-c0aa0e43ad60. Step input: None
Thought: I have now calculated the entire expression 20+(2*4) and have the final result.
Answer: 28

In [ ]:

response_gen = agent.stream_chat("What is 20+2*4? Calculate step by step")
response_gen.print_response_stream()

response_gen = agent.stream_chat("What is 20+2*4? Calculate step by step") response_gen.print_response_stream()

> Running step 079f0d12-e0f7-48cd-a1be-9f8a9ce98b9c. Step input: What is 20+2*4? Calculate step by step
Thought: To calculate 20+2*4, I need to follow the order of operations (PEMDAS). First, I need to calculate the multiplication.
Action: multiply
Action Input: {'a': 2, 'b': 4}
Observation: 8
> Running step 9d3a64e8-0b14-4721-9d5c-dd51b03ff3fa. Step input: None
Thought: Now that I have the result of the multiplication, I can add 20 to it.
Action: add
Action Input: {'a': 20, 'b': 8}
Observation: 28
> Running step 14d6f623-5c92-405b-88d5-468805429e0b. Step input: None
 20+2*4 = 28

View Prompts¶

Let's take a look at the core system prompt powering the ReAct agent!

Within the agent, the current conversation history is dumped below this line.

In [ ]:

prompt_dict = agent.get_prompts()
for k, v in prompt_dict.items():
    print(f"Prompt: {k}\n\nValue: {v.template}")

prompt_dict = agent.get_prompts() for k, v in prompt_dict.items(): print(f"Prompt: {k}\n\nValue: {v.template}")

Customizing the Prompt¶

For fun, let's try instructing the agent to output the answer along with reasoning in bullet points. See "## Additional Rules" section.

In [ ]:

from llama_index.core import PromptTemplate

react_system_header_str = """\

You are designed to help with a variety of tasks, from answering questions \
    to providing summaries to other types of analyses.

## Tools
You have access to a wide variety of tools. You are responsible for using
the tools in any sequence you deem appropriate to complete the task at hand.
This may require breaking the task into subtasks and using different tools
to complete each subtask.

You have access to the following tools:
{tool_desc}

## Output Format
To answer the question, please use the following format.

```
Thought: I need to use a tool to help me answer the question.
Action: tool name (one of {tool_names}) if using a tool.
Action Input: the input to the tool, in a JSON format representing the kwargs (e.g. {{"input": "hello world", "num_beams": 5}})
```

Please ALWAYS start with a Thought.

Please use a valid JSON format for the Action Input. Do NOT do this {{'input': 'hello world', 'num_beams': 5}}.

If this format is used, the user will respond in the following format:

```
Observation: tool response
```

You should keep repeating the above format until you have enough information
to answer the question without using any more tools. At that point, you MUST respond
in the one of the following two formats:

```
Thought: I can answer without using any more tools.
Answer: [your answer here]
```

```
Thought: I cannot answer the question with the provided tools.
Answer: Sorry, I cannot answer your query.
```

## Additional Rules
- The answer MUST contain a sequence of bullet points that explain how you arrived at the answer. This can include aspects of the previous conversation history.
- You MUST obey the function signature of each tool. Do NOT pass in no arguments if the function expects arguments.

## Current Conversation
Below is the current conversation consisting of interleaving human and assistant messages.

"""
react_system_prompt = PromptTemplate(react_system_header_str)

from llama_index.core import PromptTemplate react_system_header_str = """\ You are designed to help with a variety of tasks, from answering questions \ to providing summaries to other types of analyses. ## Tools You have access to a wide variety of tools. You are responsible for using the tools in any sequence you deem appropriate to complete the task at hand. This may require breaking the task into subtasks and using different tools to complete each subtask. You have access to the following tools: {tool_desc} ## Output Format To answer the question, please use the following format. ``` Thought: I need to use a tool to help me answer the question. Action: tool name (one of {tool_names}) if using a tool. Action Input: the input to the tool, in a JSON format representing the kwargs (e.g. {{"input": "hello world", "num_beams": 5}}) ``` Please ALWAYS start with a Thought. Please use a valid JSON format for the Action Input. Do NOT do this {{'input': 'hello world', 'num_beams': 5}}. If this format is used, the user will respond in the following format: ``` Observation: tool response ``` You should keep repeating the above format until you have enough information to answer the question without using any more tools. At that point, you MUST respond in the one of the following two formats: ``` Thought: I can answer without using any more tools. Answer: [your answer here] ``` ``` Thought: I cannot answer the question with the provided tools. Answer: Sorry, I cannot answer your query. ``` ## Additional Rules - The answer MUST contain a sequence of bullet points that explain how you arrived at the answer. This can include aspects of the previous conversation history. - You MUST obey the function signature of each tool. Do NOT pass in no arguments if the function expects arguments. ## Current Conversation Below is the current conversation consisting of interleaving human and assistant messages. """ react_system_prompt = PromptTemplate(react_system_header_str)

In [ ]:

agent.get_prompts()

agent.get_prompts()

Out[ ]:

{'agent_worker:system_prompt': PromptTemplate(metadata={'prompt_type': <PromptType.CUSTOM: 'custom'>}, template_vars=['tool_desc', 'tool_names'], kwargs={}, output_parser=None, template_var_mappings=None, function_mappings=None, template='You are designed to help with a variety of tasks, from answering questions to providing summaries to other types of analyses.\n\n## Tools\n\nYou have access to a wide variety of tools. You are responsible for using the tools in any sequence you deem appropriate to complete the task at hand.\nThis may require breaking the task into subtasks and using different tools to complete each subtask.\n\nYou have access to the following tools:\n{tool_desc}\n\n\n## Output Format\n\nPlease answer in the same language as the question and use the following format:\n\n```\nThought: The current language of the user is: (user\'s language). I need to use a tool to help me answer the question.\nAction: tool name (one of {tool_names}) if using a tool.\nAction Input: the input to the tool, in a JSON format representing the kwargs (e.g. {{"input": "hello world", "num_beams": 5}})\n```\n\nPlease ALWAYS start with a Thought.\n\nNEVER surround your response with markdown code markers. You may use code markers within your response if you need to.\n\nPlease use a valid JSON format for the Action Input. Do NOT do this {{\'input\': \'hello world\', \'num_beams\': 5}}.\n\nIf this format is used, the user will respond in the following format:\n\n```\nObservation: tool response\n```\n\nYou should keep repeating the above format till you have enough information to answer the question without using any more tools. At that point, you MUST respond in the one of the following two formats:\n\n```\nThought: I can answer without using any more tools. I\'ll use the user\'s language to answer\nAnswer: [your answer here (In the same language as the user\'s question)]\n```\n\n```\nThought: I cannot answer the question with the provided tools.\nAnswer: [your answer here (In the same language as the user\'s question)]\n```\n\n## Current Conversation\n\nBelow is the current conversation consisting of interleaving human and assistant messages.\n')}

In [ ]:

agent.update_prompts({"agent_worker:system_prompt": react_system_prompt})

agent.update_prompts({"agent_worker:system_prompt": react_system_prompt})

In [ ]:

agent.reset()
response = agent.chat("What is 5+3+2")
print(response)

agent.reset() response = agent.chat("What is 5+3+2") print(response)

> Running step 0e470036-3748-454b-8ae7-f61f1ab54fc1. Step input: What is 5+3+2
Thought: I need to use a tool to help me answer the question.
Action: add
Action Input: {'a': 5, 'b': 3}
Observation: 8
> Running step f383f54d-8647-4382-a8c7-402d82c1a795. Step input: None
Thought: I need to use another tool to help me answer the question.
Action: add
Action Input: {'a': 8, 'b': 2}
Observation: 10
> Running step 2ef5d129-810d-489d-b1e1-4c21fd30553f. Step input: None
Thought: I can answer without using any more tools.
Answer: 10

* The problem asked for the sum of 5, 3, and 2.
* I used the add tool to first calculate 5 + 3 = 8.
* Then, I used the add tool again to calculate 8 + 2 = 10.
* Therefore, the final answer is 10.
10

* The problem asked for the sum of 5, 3, and 2.
* I used the add tool to first calculate 5 + 3 = 8.
* Then, I used the add tool again to calculate 8 + 2 = 10.
* Therefore, the final answer is 10.

Using the CoAAgentWorker¶

By installing the CoAAgentPack, you also get access to the underlying agent worker. With this, you can setup the agent manually, as well as customize the prompts and output parsing.

In [ ]:

import nest_asyncio

nest_asyncio.apply()

import nest_asyncio nest_asyncio.apply()

In [ ]:

from llama_index.agent.coa import CoAAgentWorker

worker = CoAAgentWorker.from_tools(
    tools=[query_engine_tool],
    llm=llm,
    verbose=True,
)

agent = worker.as_agent()

from llama_index.agent.coa import CoAAgentWorker worker = CoAAgentWorker.from_tools( tools=[query_engine_tool], llm=llm, verbose=True, ) agent = worker.as_agent()

In [ ]:

agent.chat("How did Ubers revenue growth compare to Uber in 2021?")

agent.chat("How did Ubers revenue growth compare to Uber in 2021?")

==== Available Parsed Functions ====
def uber_10k(input: string):
   """Provides information about Uber financials for year 2021. Use a detailed plain text question as input to the tool."""
    ...
==== Generated Chain of Abstraction ====
Here is the abstract plan of reasoning:

To answer this question, we need to understand Uber's revenue growth in 2021. We can use the [FUNC uber_10k("What was Uber's revenue growth in 2021?") = y1] to get the relevant information. Then, we can compare this growth to Uber's overall performance in 2021 by analyzing the output y1.

Note: Since the question is asking for a comparison, the final answer will require a manual analysis of the output y1, rather than a simple function call.
==== Executing uber_10k with inputs ["What was Uber's revenue growth in 2021?"] ====

Out[ ]:

AgentChatResponse(response='Based on the previous reasoning, we can analyze the output "Uber\'s revenue grew by 57% in 2021, increasing from $11,139 million in 2020 to $17,455 million in 2021." to answer the question. Since the question is asking for a comparison, we can infer that the revenue growth of Uber in 2021 was significant, with a 57% increase from the previous year. This suggests that Uber\'s revenue growth in 2021 was strong, indicating a positive trend for the company.\n\nTherefore, the response to the question is:\n\nUber\'s revenue growth in 2021 was strong, with a 57% increase from the previous year, indicating a positive trend for the company.', sources=[], source_nodes=[], is_dummy_stream=False, metadata=None)