LongContextReorder

Models struggle to access significant details found in the center of extended contexts. A study observed that the best performance typically arises when crucial data is positioned at the start or conclusion of the input context. Additionally, as the input context lengthens, performance drops notably, even in models designed for long contexts.

This module will re-order the retrieved nodes, which can be helpful in cases where a large top-k is needed.

Setup

If you’re opening this Notebook on colab, you will probably need to install LlamaIndex 🦙.

!pip install llama-index

import os
import openai

os.environ["OPENAI_API_KEY"] = "sk-..."
openai.api_key = os.environ["OPENAI_API_KEY"]

from llama_index import ServiceContext
from llama_index.llms import OpenAI

llm = OpenAI(model="gpt-3.5-turbo-instruct", temperature=0.1)
ctx = ServiceContext.from_defaults(
    llm=llm, embed_model="local:BAAI/bge-base-en-v1.5"
)

/home/loganm/miniconda3/envs/llama-index/lib/python3.11/site-packages/torch/cuda/__init__.py:546: UserWarning: Can't initialize NVML
  warnings.warn("Can't initialize NVML")

Download Data

!mkdir -p 'data/paul_graham/'
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'

from llama_index import SimpleDirectoryReader

documents = SimpleDirectoryReader("./data/paul_graham/").load_data()

from llama_index import VectorStoreIndex

index = VectorStoreIndex.from_documents(documents, service_context=ctx)

Run Query

from llama_index.postprocessor import LongContextReorder

reorder = LongContextReorder()

reorder_engine = index.as_query_engine(
    node_postprocessors=[reorder], similarity_top_k=5
)
base_engine = index.as_query_engine(similarity_top_k=5)

from llama_index.response.notebook_utils import display_response

base_response = base_engine.query("Did the author meet Sam Altman?")
display_response(base_response)

Final Response: Yes, the author met Sam Altman when they asked him to be the president of Y Combinator. This was during the time when the author was in a PhD program in computer science and also pursuing their passion for art. They were applying to art schools and eventually ended up attending RISD.

reorder_response = reorder_engine.query("Did the author meet Sam Altman?")
display_response(reorder_response)

Final Response: Yes, the author met Sam Altman when they asked him to be the president of Y Combinator. This meeting occurred at a party at the author’s house, where they were introduced by a mutual friend, Jessica Livingston. Jessica later went on to compile a book of interviews with startup founders, and the author shared their thoughts on the flaws of venture capital with her during her job search at a Boston VC firm.

Inspect Order Diffrences

print(base_response.get_formatted_sources())

> Source (Doc id: 81bc66bb-2c45-4697-9f08-9f848bd78b12): [17]

As well as HN, I wrote all of YC's internal software in Arc. But while I continued to work ...

> Source (Doc id: bd660905-e4e0-4d02-a113-e3810b59c5d1): [19] One way to get more precise about the concept of invented vs discovered is to talk about spa...

> Source (Doc id: 3932e4a4-f17e-4dd2-9d25-5f0e65910dc5): Not so much because it was badly written as because the problem is so convoluted. When you're wor...

> Source (Doc id: 0d801f0a-4a99-475d-aa7c-ad5d601947ea): [10]

Wow, I thought, there's an audience. If I write something and put it on the web, anyone can...

> Source (Doc id: bf726802-4d0d-4ee5-ab2e-ffa8a5461bc4): I was briefly tempted, but they were so slow by present standards; what was the point? No one els...

print(reorder_response.get_formatted_sources())

> Source (Doc id: 81bc66bb-2c45-4697-9f08-9f848bd78b12): [17]

As well as HN, I wrote all of YC's internal software in Arc. But while I continued to work ...

> Source (Doc id: 3932e4a4-f17e-4dd2-9d25-5f0e65910dc5): Not so much because it was badly written as because the problem is so convoluted. When you're wor...

> Source (Doc id: bf726802-4d0d-4ee5-ab2e-ffa8a5461bc4): I was briefly tempted, but they were so slow by present standards; what was the point? No one els...

> Source (Doc id: 0d801f0a-4a99-475d-aa7c-ad5d601947ea): [10]

Wow, I thought, there's an audience. If I write something and put it on the web, anyone can...

> Source (Doc id: bd660905-e4e0-4d02-a113-e3810b59c5d1): [19] One way to get more precise about the concept of invented vs discovered is to talk about spa...