LLMs and LlamaIndex ◦ April 2 2024 ◦ Ontario Teacher's Pension Plan¶

In [ ]:

##################################################################
# Venue: OTPP L&L
# Talk: LLMs and LlamaIndex
# Speaker: Andrei Fajardo
##################################################################

################################################################## # Venue: OTPP L&L # Talk: LLMs and LlamaIndex # Speaker: Andrei Fajardo ##################################################################

Notebook Setup & Dependency Installation¶

In [ ]:

%pip install llama-index-vector-stores-qdrant -q

%pip install llama-index-vector-stores-qdrant -q

[notice] A new release of pip available: 22.3.1 -> 24.0
[notice] To update, run: python3.11 -m pip install --upgrade pip
Note: you may need to restart the kernel to use updated packages.

In [ ]:

import nest_asyncio

nest_asyncio.apply()

import nest_asyncio nest_asyncio.apply()

In [ ]:

!mkdir data
!wget "https://www.otpp.com/content/dam/otpp/documents/reports/2023-ar/otpp-2023-annual-report-eng.pdf" -O "./data/otpp-2023-annual-report-eng.pdf"

!mkdir data !wget "https://www.otpp.com/content/dam/otpp/documents/reports/2023-ar/otpp-2023-annual-report-eng.pdf" -O "./data/otpp-2023-annual-report-eng.pdf"

mkdir: data: File exists
--2024-04-03 11:23:36--  https://www.otpp.com/content/dam/otpp/documents/reports/2023-ar/otpp-2023-annual-report-eng.pdf
Resolving www.otpp.com (www.otpp.com)... 67.210.219.20
Connecting to www.otpp.com (www.otpp.com)|67.210.219.20|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10901939 (10M) [application/pdf]
Saving to: ‘./data/otpp-2023-annual-report-eng.pdf’

./data/otpp-2023-an 100%[===================>]  10.40M  1.01MB/s    in 10s     

2024-04-03 11:23:49 (1.02 MB/s) - ‘./data/otpp-2023-annual-report-eng.pdf’ saved [10901939/10901939]

Motivation¶

In [ ]:

# query an LLM and ask it about Mistplay
from llama_index.llms.openai import OpenAI

llm = OpenAI(model="gpt-4-turbo-preview")
response = llm.complete("What is Ontario Teacher's Pension Plan all about?")

# query an LLM and ask it about Mistplay from llama_index.llms.openai import OpenAI llm = OpenAI(model="gpt-4-turbo-preview") response = llm.complete("What is Ontario Teacher's Pension Plan all about?")

In [ ]:

print(response)

print(response)

The Ontario Teachers' Pension Plan (OTPP) is one of the world's largest pension funds, serving the public school teachers of Ontario, Canada. Established in 1990, it operates as an independent organization responsible for administering defined-benefit pensions for school teachers of the province. The OTPP is jointly sponsored by the Government of Ontario and the Ontario Teachers' Federation, meaning both the government and the teachers' union have a say in the management and direction of the fund.

The primary purpose of the OTPP is to provide a stable and reliable source of retirement income for its members. It does so by collecting contributions from both teachers and their employers (the government) and investing those funds in a wide variety of assets, including stocks, bonds, real estate, and infrastructure projects, both domestically and internationally. The goal is to generate sufficient returns to ensure the long-term sustainability of the pension plan while managing risk.

The OTPP is notable for its size, investment success, and innovative approach to pension management. It has been a pioneer in direct investment, taking significant stakes in companies, real estate, and infrastructure projects around the world. This direct investment strategy has allowed the OTPP to reduce reliance on external fund managers, thereby lowering costs and potentially increasing returns.

The plan's governance model is designed to ensure stability and sustainability. The board of directors, which oversees the operation of the OTPP, includes representatives from both the government and the Ontario Teachers' Federation. This joint sponsorship model ensures that both the interests of the teachers and the fiscal responsibilities of the government are taken into account in the plan's management.

In summary, the Ontario Teachers' Pension Plan is a critical component of the retirement security system for Ontario's public school teachers, distinguished by its large scale, sophisticated investment strategy, and joint governance structure. It aims to deliver secure and sustainable pensions to its members, contributing significantly to their financial well-being in retirement.

In [ ]:

response = llm.complete(
    "According to the 2023 annual report, how many billions of dollars in net assets does Ontario Teacher's Pension Plan hold?"
)

response = llm.complete( "According to the 2023 annual report, how many billions of dollars in net assets does Ontario Teacher's Pension Plan hold?" )

In [ ]:

print(response)

print(response)

As of my last update in April 2023, the Ontario Teachers' Pension Plan (OTPP) reported holding net assets of CAD 247.5 billion in its 2022 annual report. Please note that this information might have changed after my last update, and I would recommend checking the latest annual report or official OTPP sources for the most current figures.

In [ ]:

response = llm.complete(
    "According to the 2023 annual report, what is the 10-year total-fund net return?"
)

response = llm.complete( "According to the 2023 annual report, what is the 10-year total-fund net return?" )

In [ ]:

print(response)

print(response)

As of my last update in April 2023, I don't have access to real-time data or specific annual reports from 2023. Therefore, I cannot provide the 10-year total-fund net return from any specific 2023 annual report. This information would typically be found directly in the report of the specific fund or investment you're interested in. I recommend checking the official website or contacting the financial institution directly for the most accurate and up-to-date information.

Basic RAG in 3 Steps¶

1. Build external knowledge (i.e., uploading updated data sources)
2. Retrieve
3. Augment and Generate

1. Build External Knowledge¶

In [ ]:

"""Load the data.

With llama-index, before any transformations are applied,
data is loaded in the `Document` abstraction, which is
a container that holds the text of the document.
"""

from llama_index.core import SimpleDirectoryReader

loader = SimpleDirectoryReader(input_dir="./data")
documents = loader.load_data()

"""Load the data. With llama-index, before any transformations are applied, data is loaded in the `Document` abstraction, which is a container that holds the text of the document. """ from llama_index.core import SimpleDirectoryReader loader = SimpleDirectoryReader(input_dir="./data") documents = loader.load_data()

In [ ]:

# if you want to see what the text looks like
documents[0].text[:1000]

# if you want to see what the text looks like documents[0].text[:1000]

Out[ ]:

'Investing to  \nmake a mark\n2023 Annual Report'

In [ ]:

"""Chunk, Encode, and Store into a Vector Store.

To streamline the process, we can make use of the IngestionPipeline
class that will apply your specified transformations to the
Document's.
"""

from llama_index.core.ingestion import IngestionPipeline
from llama_index.core.node_parser import SentenceSplitter
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.vector_stores.qdrant import QdrantVectorStore
import qdrant_client

client = qdrant_client.QdrantClient(location=":memory:")
vector_store = QdrantVectorStore(client=client, collection_name="test_store")

pipeline = IngestionPipeline(
    transformations=[
        SentenceSplitter(),
        OpenAIEmbedding(),
    ],
    vector_store=vector_store,
)
_nodes = pipeline.run(documents=documents, num_workers=4)

"""Chunk, Encode, and Store into a Vector Store. To streamline the process, we can make use of the IngestionPipeline class that will apply your specified transformations to the Document's. """ from llama_index.core.ingestion import IngestionPipeline from llama_index.core.node_parser import SentenceSplitter from llama_index.embeddings.openai import OpenAIEmbedding from llama_index.vector_stores.qdrant import QdrantVectorStore import qdrant_client client = qdrant_client.QdrantClient(location=":memory:") vector_store = QdrantVectorStore(client=client, collection_name="test_store") pipeline = IngestionPipeline( transformations=[ SentenceSplitter(), OpenAIEmbedding(), ], vector_store=vector_store, ) _nodes = pipeline.run(documents=documents, num_workers=4)

In [ ]:

# if you want to see the nodes
# len(_nodes)
_nodes[0].text

# if you want to see the nodes # len(_nodes) _nodes[0].text

Out[ ]:

'Investing to  \nmake a mark\n2023 Annual Report'

In [ ]:

"""Create a llama-index... wait for it... Index.

After uploading your encoded documents into your vector
store of choice, you can connect to it with a VectorStoreIndex
which then gives you access to all of the llama-index functionality.
"""

from llama_index.core import VectorStoreIndex

index = VectorStoreIndex.from_vector_store(vector_store=vector_store)

"""Create a llama-index... wait for it... Index. After uploading your encoded documents into your vector store of choice, you can connect to it with a VectorStoreIndex which then gives you access to all of the llama-index functionality. """ from llama_index.core import VectorStoreIndex index = VectorStoreIndex.from_vector_store(vector_store=vector_store)

2. Retrieve Against A Query¶

In [ ]:

"""Retrieve relevant documents against a query.

With our Index ready, we can now query it to
retrieve the most relevant document chunks.
"""

retriever = index.as_retriever(similarity_top_k=2)
retrieved_nodes = retriever.retrieve(
    "According to the 2023 annual report, what is the 10-year total-fund net return?"
)

"""Retrieve relevant documents against a query. With our Index ready, we can now query it to retrieve the most relevant document chunks. """ retriever = index.as_retriever(similarity_top_k=2) retrieved_nodes = retriever.retrieve( "According to the 2023 annual report, what is the 10-year total-fund net return?" )

In [ ]:

# to view the retrieved node
print(retrieved_nodes[0].text[:500])

# to view the retrieved node print(retrieved_nodes[0].text[:500])

Stable long-term total-fund returns 
1 Net assets include investment assets less investment liabilities (net investments), plus the receivables from the Province of Ontario, and 
other assets less other liabilities.
2 A real rate of return is the net return, or annual percentage of profit earned on an investment, adjusted for inflation.Ontario Teachers’ investment program is tailored to 
generate strong and steady risk-adjusted returns to 
pay members’ pensions over generations, while also 
havi

3. Generate Final Response¶

In [ ]:

"""Context-Augemented Generation.

With our Index ready, we can create a QueryEngine
that handles the retrieval and context augmentation
in order to get the final response.
"""

query_engine = index.as_query_engine()

"""Context-Augemented Generation. With our Index ready, we can create a QueryEngine that handles the retrieval and context augmentation in order to get the final response. """ query_engine = index.as_query_engine()

In [ ]:

# to inspect the default prompt being used
print(
    query_engine.get_prompts()[
        "response_synthesizer:text_qa_template"
    ].default_template.template
)

# to inspect the default prompt being used print( query_engine.get_prompts()[ "response_synthesizer:text_qa_template" ].default_template.template )

Context information is below.
---------------------
{context_str}
---------------------
Given the context information and not prior knowledge, answer the query.
Query: {query_str}
Answer: 

In [ ]:

response = query_engine.query(
    "According to the 2023 annual report, what is the 10-year total-fund net return?"
)
print(response)

response = query_engine.query( "According to the 2023 annual report, what is the 10-year total-fund net return?" ) print(response)

The 10-year total-fund net return, as stated in the 2023 annual report, is 7.6%.

Beyond Basic RAG: Improved PDF Parsing with LlamaParse¶

To use LlamaParse, you first need to obtain an API Key. Visit llamacloud.ai to login (or sign-up) and get an api key.

In [ ]:

api_key = "<FILL-IN>"

api_key = ""

The default pdf reader (PyPDF), like many out-of-the box pdf parsers struggle on complex PDF docs.¶

In [ ]:

response = query_engine.query(
    "How many board meetings did Steve McGirr, Chair of the Board, attend?"
)
print(response)

response = query_engine.query( "How many board meetings did Steve McGirr, Chair of the Board, attend?" ) print(response)

Steve McGirr, Chair of the Board, attended all board meetings, which totaled 11 full meetings, including one strategic offsite meeting.

In [ ]:

response = query_engine.query(
    "What percentage of board members identify as women?"
)
print(response)

response = query_engine.query( "What percentage of board members identify as women?" ) print(response)

55% of board members identify as women.

In [ ]:

response = query_engine.query(
    "What is the total investment percentage in Canada as of December 31, 2023?"
)
print(response)

response = query_engine.query( "What is the total investment percentage in Canada as of December 31, 2023?" ) print(response)

The total investment percentage in Canada as of December 31, 2023, is 29% (10% in public equity and 19% in inflation-sensitive investments).

Improved PDF Parsing using LlamaParse¶

In [ ]:

from llama_parse import LlamaParse
from llama_index.core.node_parser import MarkdownElementNodeParser
from llama_index.llms.openai import OpenAI

from llama_parse import LlamaParse from llama_index.core.node_parser import MarkdownElementNodeParser from llama_index.llms.openai import OpenAI

In [ ]:

from llama_parse import LlamaParse

parser = LlamaParse(result_type="markdown")
md_documents = parser.load_data(
    file_path="./data/otpp-2023-annual-report-eng.pdf"
)

from llama_parse import LlamaParse parser = LlamaParse(result_type="markdown") md_documents = parser.load_data( file_path="./data/otpp-2023-annual-report-eng.pdf" )

Started parsing the file under job_id ddcdc5f9-bd16-40b8-90f2-353f2a2b6450

In [ ]:

# save to an .md file
with open("./mds/parsed.md", "w") as f:
    f.write(md_documents[0].text)

# save to an .md file with open("./mds/parsed.md", "w") as f: f.write(md_documents[0].text)

In [ ]:

md_node_parser = MarkdownElementNodeParser(
    llm=OpenAI(model="gpt-4.5-turbo-preview"),
    num_workers=3,
    include_metadata=True,
)
md_nodes = md_node_parser.get_nodes_from_documents(md_documents)

md_node_parser = MarkdownElementNodeParser( llm=OpenAI(model="gpt-4.5-turbo-preview"), num_workers=3, include_metadata=True, ) md_nodes = md_node_parser.get_nodes_from_documents(md_documents)

133it [00:00, 70142.39it/s]
100%|███████████████████████████████████████████████████████████████████████████████████| 133/133 [01:13<00:00,  1.80it/s]

In [ ]:

llama_parse_index = VectorStoreIndex.from_documents(md_documents)
llama_parse_query_engine = llama_parse_index.as_query_engine()

llama_parse_index = VectorStoreIndex.from_documents(md_documents) llama_parse_query_engine = llama_parse_index.as_query_engine()

In [ ]:

response = llama_parse_query_engine.query(
    "How many board meetings did Steve McGirr, Chair of the Board, attend?"
)
print(response)

response = llama_parse_query_engine.query( "How many board meetings did Steve McGirr, Chair of the Board, attend?" ) print(response)

Steve McGirr, Chair of the Board, attended 11 board meetings.

In [ ]:

response = llama_parse_query_engine.query(
    "What percentage of board members identify as women?"
)
print(response)

response = llama_parse_query_engine.query( "What percentage of board members identify as women?" ) print(response)

55% of board members identify as women.

In [ ]:

response = llama_parse_query_engine.query(
    "What is the total investment percentage in Canada as of December 31, 2023?"
)
print(response)

response = llama_parse_query_engine.query( "What is the total investment percentage in Canada as of December 31, 2023?" ) print(response)

The total investment percentage in Canada as of December 31, 2023, is 35%.

In Summary¶

• LLMs as powerful as they are, don't perform too well with knowledge-intensive tasks (domain specific, updated data, long-tail)
• Context augmentation has been shown (in a few studies) to outperform LLMs without augmentation
• In this notebook, we showed one such example that follows that pattern.

Data Extraction¶

In [ ]:

import json
from llama_index.core.bridge.pydantic import BaseModel, Field

from llama_index.program.openai import OpenAIPydanticProgram
from llama_index.llms.openai import OpenAI

import json from llama_index.core.bridge.pydantic import BaseModel, Field from llama_index.program.openai import OpenAIPydanticProgram from llama_index.llms.openai import OpenAI

Leadership Team¶

In [ ]:

class LeadershipTeam(BaseModel):
    """Data model for leadership team."""

    ceo: str = Field(description="The CEO")
    coo: str = Field(description="The Chief Operating Officer")
    cio: str = Field(description="Chief Investment Officer")
    chief_pension_officer: str = Field(description="Chief Pension Officer")
    chief_legal_officer: str = Field(
        description="Chief Legal & Corporate Affairs Officer"
    )
    chief_people_officer: str = Field(description="Chief People Officer")
    chief_strategy_officer: str = Field(description="Chief Strategy Officer")
    executive_managing_director: str = Field(
        description="Executive Managing Director"
    )
    chief_investment_officer: str = Field(
        description="Chief Investment Officer"
    )

class LeadershipTeam(BaseModel): """Data model for leadership team.""" ceo: str = Field(description="The CEO") coo: str = Field(description="The Chief Operating Officer") cio: str = Field(description="Chief Investment Officer") chief_pension_officer: str = Field(description="Chief Pension Officer") chief_legal_officer: str = Field( description="Chief Legal & Corporate Affairs Officer" ) chief_people_officer: str = Field(description="Chief People Officer") chief_strategy_officer: str = Field(description="Chief Strategy Officer") executive_managing_director: str = Field( description="Executive Managing Director" ) chief_investment_officer: str = Field( description="Chief Investment Officer" )

In [ ]:

prompt_template_str = """\
Here is the 2023 Annual Report for Ontario Teacher's Pension Plan:
{document_text}
Provide the names of the Leadership Team.
"""

program = OpenAIPydanticProgram.from_defaults(
    output_cls=LeadershipTeam,
    prompt_template_str=prompt_template_str,
    llm=OpenAI("gpt-4-turbo-preview"),
    verbose=True,
)

prompt_template_str = """\ Here is the 2023 Annual Report for Ontario Teacher's Pension Plan: {document_text} Provide the names of the Leadership Team. """ program = OpenAIPydanticProgram.from_defaults( output_cls=LeadershipTeam, prompt_template_str=prompt_template_str, llm=OpenAI("gpt-4-turbo-preview"), verbose=True, )

In [ ]:

leadership_team = program(document_text=md_documents[0].text)

leadership_team = program(document_text=md_documents[0].text)

Function call: LeadershipTeam with args: {"ceo":"Jo Taylor","coo":"Tracy Abel","cio":"Gillian Brown","chief_pension_officer":"Charley Butler","chief_legal_officer":"Sharon Chilcott","chief_people_officer":"Jeff Davis","chief_strategy_officer":"Jonathan Hausman","executive_managing_director":"Nick Jansa","chief_investment_officer":"Stephen McLennan"}

In [ ]:

print(json.dumps(leadership_team.dict(), indent=4))

print(json.dumps(leadership_team.dict(), indent=4))

{
    "ceo": "Jo Taylor",
    "coo": "Tracy Abel",
    "cio": "Gillian Brown",
    "chief_pension_officer": "Charley Butler",
    "chief_legal_officer": "Sharon Chilcott",
    "chief_people_officer": "Jeff Davis",
    "chief_strategy_officer": "Jonathan Hausman",
    "executive_managing_director": "Nick Jansa",
    "chief_investment_officer": "Stephen McLennan"
}

LlamaIndex Has More To Offer¶

• Data infrastructure that enables production-grade, advanced RAG systems
• Agentic solutions
• Newly released: llama-index-networks
• Enterprise offerings (alpha):
  • LlamaParse (proprietary complex PDF parser) and
  • LlamaCloud

Useful links¶

website ◦ llamahub ◦ llamaparse ◦ github ◦ medium ◦ rag-bootcamp-poster