Test Complex Queries over Multiple Documents (with and without Query Decomposition)

Query Decomposition: The ability to decompose a complex query into a simpler query given the content of the index.

Use ChatGPT as the LLM model

If you’re opening this Notebook on colab, you will probably need to install LlamaIndex 🦙.

!pip install llama-index

import logging

# logging.basicConfig(stream=sys.stdout, level=logging.INFO)
# logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))

# Uncomment if you want to temporarily disable logger
logger = logging.getLogger()
logger.disabled = True

from llama_index import (
    VectorStoreIndex,
    SimpleKeywordTableIndex,
    SimpleDirectoryReader,
    ServiceContext,
)

/Users/suo/miniconda3/envs/llama/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

Load Datasets

Load Wikipedia pages as well as Paul Graham’s “What I Worked On” essay

wiki_titles = [
    "Toronto",
    "Seattle",
    "San Francisco",
    "Chicago",
    "Boston",
    "Washington, D.C.",
    "Cambridge, Massachusetts",
    "Houston",
]

from pathlib import Path

import requests

for title in wiki_titles:
    response = requests.get(
        "https://en.wikipedia.org/w/api.php",
        params={
            "action": "query",
            "format": "json",
            "titles": title,
            "prop": "extracts",
            # 'exintro': True,
            "explaintext": True,
        },
    ).json()
    page = next(iter(response["query"]["pages"].values()))
    wiki_text = page["extract"]

    data_path = Path("data")
    if not data_path.exists():
        Path.mkdir(data_path)

    with open(data_path / f"{title}.txt", "w") as fp:
        fp.write(wiki_text)

# Load all wiki documents
city_docs = {}
for wiki_title in wiki_titles:
    city_docs[wiki_title] = SimpleDirectoryReader(
        input_files=[f"data/{wiki_title}.txt"]
    ).load_data()

Building the document indices

Build a vector index for the wiki pages about cities and persons, and PG essay

# # LLM Predictor (gpt-3.5-turbo)
from llama_index.llms.openai import OpenAI


chatgpt = OpenAI(temperature=0, model="gpt-3.5-turbo")
service_context = ServiceContext.from_defaults(llm=chatgpt)

/Users/suo/miniconda3/envs/llama/lib/python3.9/site-packages/langchain/llms/openai.py:661: UserWarning: You are trying to use a chat model. This way of initializing it is no longer supported. Instead, please use: `from langchain.chat_models import ChatOpenAI`
  warnings.warn(

# Build city document index
city_indices = {}
index_summaries = {}
for wiki_title in wiki_titles:
    city_indices[wiki_title] = VectorStoreIndex.from_documents(
        city_docs[wiki_title], service_context=service_context
    )
    # set summary text for city
    index_summaries[wiki_title] = f"Wikipedia articles about {wiki_title}"

INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 20744 tokens
INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 16942 tokens
INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 23433 tokens
INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 26082 tokens
INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 18614 tokens
INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 21649 tokens
INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 12855 tokens
INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 21844 tokens

Build Graph: Keyword Table Index on top of vector indices!

We compose a keyword table index on top of all the vector indices.

from llama_index.indices.composability import ComposableGraph

graph = ComposableGraph.from_indices(
    SimpleKeywordTableIndex,
    [index for _, index in city_indices.items()],
    [summary for _, summary in index_summaries.items()],
    max_keywords_per_chunk=50,
)

INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 0 tokens

Define Query Configs

Query Transform

from llama_index.indices.query.query_transform.base import (
    DecomposeQueryTransform,
)

decompose_transform = DecomposeQueryTransform(
    service_context.llm, verbose=True
)

Complex Query 1

# with query decomposition in subindices
from llama_index.query_engine.transform_query_engine import (
    TransformQueryEngine,
)


custom_query_engines = {}
for index in city_indices.values():
    query_engine = index.as_query_engine(service_context=service_context)
    transform_metadata = {"index_summary": index.index_struct.summary}
    tranformed_query_engine = TransformQueryEngine(
        query_engine,
        decompose_transform,
        transform_metadata=transform_metadata,
    )
    custom_query_engines[index.index_id] = tranformed_query_engine

custom_query_engines[
    graph.root_index.index_id
] = graph.root_index.as_query_engine(
    retriever_mode="simple",
    response_mode="tree_summarize",
    service_context=service_context,
)

query_engine_decompose = graph.as_query_engine(
    custom_query_engines=custom_query_engines,
)

response_chatgpt = query_engine_decompose.query(
    "Compare and contrast the airports in Seattle, Houston, and Toronto. "
)

INFO:llama_index.indices.keyword_table.retrievers:> Starting query: Compare and contrast the airports in Seattle, Houston, and Toronto. 
INFO:llama_index.indices.keyword_table.retrievers:query keywords: ['toronto', 'airports', 'seattle', 'contrast', 'compare', 'houston']
INFO:llama_index.indices.keyword_table.retrievers:> Extracted keywords: ['toronto', 'seattle', 'houston']
INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 12 tokens

> Current query: Compare and contrast the airports in Seattle, Houston, and Toronto. 
> New query: What are some notable features of the Toronto Pearson International Airport?
> Current query: Compare and contrast the airports in Seattle, Houston, and Toronto. 
> New query: What are some notable features of the Toronto Pearson International Airport?

INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1142 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1142 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 10 tokens

> Current query: Compare and contrast the airports in Seattle, Houston, and Toronto. 
> New query: What is the name of the airport in Seattle?
> Current query: Compare and contrast the airports in Seattle, Houston, and Toronto. 
> New query: What is the name of the airport in Seattle?

INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1773 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1773 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens

> Current query: Compare and contrast the airports in Seattle, Houston, and Toronto. 
> New query: What are the major airports in Houston?

INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 8 tokens

> Current query: Compare and contrast the airports in Seattle, Houston, and Toronto. 
> New query: What are the major airports in Houston?

INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1162 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1162 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 254 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 254 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens

print(str(response_chatgpt))

Seattle has one major airport called Seattle-Tacoma International Airport, while Houston has two major airports called George Bush Intercontinental Airport and William P. Hobby Airport, as well as a third municipal airport called Ellington Airport. Toronto Pearson International Airport is Canada's busiest airport and offers limited commercial and passenger service to nearby destinations in Canada and the United States. All three cities have at least one major airport, but Houston has more options with two major airports.

# without query decomposition in subindices

custom_query_engines = {}
for index in city_indices.values():
    query_engine = index.as_query_engine(service_context=service_context)
    custom_query_engines[index.index_id] = query_engine

custom_query_engines[
    graph.root_index.index_id
] = graph.root_index.as_query_engine(
    retriever_mode="simple",
    response_mode="tree_summarize",
    service_context=service_context,
)

query_engine = graph.as_query_engine(
    custom_query_engines=custom_query_engines,
)

response_chatgpt = query_engine.query(
    "Compare and contrast the airports in Seattle, Houston, and Toronto. "
)

INFO:llama_index.indices.keyword_table.retrievers:> Starting query: Compare and contrast the airports in Seattle, Houston, and Toronto. 
INFO:llama_index.indices.keyword_table.retrievers:query keywords: ['toronto', 'airports', 'seattle', 'contrast', 'compare', 'houston']
INFO:llama_index.indices.keyword_table.retrievers:> Extracted keywords: ['toronto', 'seattle', 'houston']
INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 14 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1114 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1114 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1799 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1799 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1186 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1186 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 196 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 196 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens

str(response_chatgpt)

'It is not possible to compare and contrast the airports in Seattle, Houston, and Toronto based on the given context information.'

Complex Query 2

# with query decomposition
response_chatgpt = query_engine_decompose.query(
    "Compare and contrast the sports environment of Houston and Boston. "
)

INFO:llama_index.indices.keyword_table.retrievers:> Starting query: Compare and contrast the sports environment of Houston and Boston. 
INFO:llama_index.indices.keyword_table.retrievers:query keywords: ['environment', 'contrast', 'sports', 'compare', 'houston', 'boston']
INFO:llama_index.indices.keyword_table.retrievers:> Extracted keywords: ['houston', 'boston']

> Current query: Compare and contrast the sports environment of Houston and Boston. 
> New query: What sports teams are based in Houston?

INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 8 tokens

> Current query: Compare and contrast the sports environment of Houston and Boston. 
> New query: What sports teams are based in Houston?

INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1861 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1861 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 10 tokens

> Current query: Compare and contrast the sports environment of Houston and Boston. 
> New query: What are some notable sports teams based in Boston?
> Current query: Compare and contrast the sports environment of Houston and Boston. 
> New query: What are some notable sports teams based in Boston?

INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1812 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1812 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 226 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 226 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens

str(response_chatgpt)

'Houston has sports teams for every major professional league except the National Hockey League, while Boston has teams for Major League Baseball, National Hockey League, National Basketball Association, National Football League, Major League Lacrosse, and Overwatch League. Both cities have a strong sports culture, but Boston has a more diverse range of professional sports teams.'

# without query decomposition
response_chatgpt = query_engine.query(
    "Compare and contrast the sports environment of Houston and Boston. "
)

INFO:llama_index.indices.keyword_table.retrievers:> Starting query: Compare and contrast the sports environment of Houston and Boston. 
INFO:llama_index.indices.keyword_table.retrievers:query keywords: ['environment', 'contrast', 'sports', 'compare', 'houston', 'boston']
INFO:llama_index.indices.keyword_table.retrievers:> Extracted keywords: ['houston', 'boston']
INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 12 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1795 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1795 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1792 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1792 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 119 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 119 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens

str(response_chatgpt)

'Sorry, I cannot answer this question as there is no information provided about the sports environment of Houston or Boston in the given context information.'

# with query decomposition
response_chatgpt = query_engine_decompose.query(
    "Compare and contrast the sports environment of Houston and Boston. "
)

INFO:llama_index.indices.keyword_table.retrievers:> Starting query: Compare and contrast the sports environment of Houston and Boston. 
INFO:llama_index.indices.keyword_table.retrievers:query keywords: ['environment', 'contrast', 'sports', 'compare', 'houston', 'boston']
INFO:llama_index.indices.keyword_table.retrievers:> Extracted keywords: ['houston', 'boston']

> Current query: Compare and contrast the sports environment of Houston and Boston. 
> New query: What sports teams are based in Houston?

INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 8 tokens

> Current query: Compare and contrast the sports environment of Houston and Boston. 
> New query: What sports teams are based in Houston?

INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1861 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1861 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens

> Current query: Compare and contrast the sports environment of Houston and Boston. 
> New query: What are some notable sports teams based in Boston?

INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 10 tokens

> Current query: Compare and contrast the sports environment of Houston and Boston. 
> New query: What are some notable sports teams based in Boston?

INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1812 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1812 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 226 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 226 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens

print(response_chatgpt)

Houston has sports teams for every major professional league except the National Hockey League, while Boston has teams for Major League Baseball, National Hockey League, National Basketball Association, National Football League, Major League Lacrosse, and Overwatch League. Both cities have a strong sports culture, but Boston has a more diverse range of professional sports teams.

# without query decomposition
response_chatgpt = query_engine.query(
    "Compare and contrast the sports environment of Houston and Boston. "
)

INFO:llama_index.indices.keyword_table.retrievers:> Starting query: Compare and contrast the sports environment of Houston and Boston. 
INFO:llama_index.indices.keyword_table.retrievers:query keywords: ['environment', 'contrast', 'sports', 'compare', 'houston', 'boston']
INFO:llama_index.indices.keyword_table.retrievers:> Extracted keywords: ['houston', 'boston']
INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 12 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1795 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1795 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1792 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1792 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 119 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 119 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens

print(response_chatgpt)

Sorry, I cannot answer this question as there is no information provided about the sports environment of Houston or Boston in the given context information.

Complex Query 3

# with query decomposition
response_chatgpt = query_engine_decompose.query(
    "Compare and contrast the arts and culture of Houston and Boston. "
)

INFO:llama_index.indices.keyword_table.retrievers:> Starting query: Compare and contrast the arts and culture of Houston and Boston. 
INFO:llama_index.indices.keyword_table.retrievers:query keywords: ['arts', 'culture', 'contrast', 'compare', 'houston', 'boston']
INFO:llama_index.indices.keyword_table.retrievers:> Extracted keywords: ['houston', 'boston']
INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 9 tokens

> Current query: Compare and contrast the arts and culture of Houston and Boston. 
> New query: What are some notable cultural institutions in Houston?
> Current query: Compare and contrast the arts and culture of Houston and Boston. 
> New query: What are some notable cultural institutions in Houston?

INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1835 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1835 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens

> Current query: Compare and contrast the arts and culture of Houston and Boston. 
> New query: What are some notable cultural institutions in Boston?

INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 9 tokens

> Current query: Compare and contrast the arts and culture of Houston and Boston. 
> New query: What are some notable cultural institutions in Boston?

INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1918 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1918 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 444 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 444 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens

print(response_chatgpt)

Both Houston and Boston have a variety of cultural institutions, including museums, performing arts organizations, and theaters. Some notable museums in both cities include the Museum of Fine Arts. However, Houston has a greater focus on contemporary art with institutions such as the Contemporary Arts Museum Houston and the Station Museum of Contemporary Art. Boston, on the other hand, has a unique museum in the Isabella Stewart Gardner Museum, which features a collection of art and artifacts in a recreated Venetian palace. In terms of performing arts, both cities have symphony orchestras and opera companies, but Boston also has a strong focus on contemporary classical music with groups such as the Boston Modern Orchestra Project. Overall, while both cities have a rich arts and culture scene, they differ in their specific areas of focus.

# without query decomposition
response_chatgpt = query_engine.query(
    "Compare and contrast the arts and culture of Houston and Boston. "
)

INFO:llama_index.indices.keyword_table.retrievers:> Starting query: Compare and contrast the arts and culture of Houston and Boston. 
INFO:llama_index.indices.keyword_table.retrievers:query keywords: ['arts', 'culture', 'contrast', 'compare', 'houston', 'boston']
INFO:llama_index.indices.keyword_table.retrievers:> Extracted keywords: ['houston', 'boston']
INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 13 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1779 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1779 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1817 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1817 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 122 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 122 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens

print(response_chatgpt)

I'm sorry, but there is not enough information provided to compare and contrast the arts and culture of Houston and Boston.