Relevancy Evaluator

This notebook uses the RelevancyEvaluator to measure if the response + source nodes match the query.
This is useful for measuring if the query was actually answered by the response.

import logging
import sys

logging.basicConfig(stream=sys.stdout, level=logging.INFO)
logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))

from llama_index import (
    TreeIndex,
    VectorStoreIndex,
    SimpleDirectoryReader,
    ServiceContext,
    Response,
)
from llama_index.llms import OpenAI
from llama_index.evaluation import RelevancyEvaluator
import pandas as pd

pd.set_option("display.max_colwidth", 0)

# gpt-3 (davinci)
gpt3 = OpenAI(temperature=0, model="text-davinci-003")
service_context_gpt3 = ServiceContext.from_defaults(llm=gpt3)

# gpt-4
gpt4 = OpenAI(temperature=0, model="gpt-4")
service_context_gpt4 = ServiceContext.from_defaults(llm=gpt4)

evaluator = RelevancyEvaluator(service_context=service_context_gpt3)
evaluator_gpt4 = RelevancyEvaluator(service_context=service_context_gpt4)

documents = SimpleDirectoryReader("./test_wiki_data").load_data()

# create vector index
vector_index = VectorStoreIndex.from_documents(
    documents, service_context=ServiceContext.from_defaults(chunk_size=512)
)

# define jupyter display function
def display_eval_df(query: str, response: Response, eval_result: str) -> None:
    eval_df = pd.DataFrame(
        {
            "Query": query,
            "Response": str(response),
            "Source": response.source_nodes[0].node.text[:1000] + "...",
            "Evaluation Result": "Pass" if eval_result.passing else "Fail",
        },
        index=[0],
    )
    eval_df = eval_df.style.set_properties(
        **{
            "inline-size": "600px",
            "overflow-wrap": "break-word",
        },
        subset=["Response", "Source"]
    )
    display(eval_df)

Evaluate Response

Evaluate response relative to source nodes as well as query.

query_str = (
    "What battles took place in New York City in the American Revolution?"
)
query_engine = vector_index.as_query_engine()
response_vector = query_engine.query(query_str)
eval_result = evaluator_gpt4.evaluate_response(
    query=query_str, response=response_vector
)

display_eval_df(query_str, response_vector, eval_result)

	Query	Response	Source	Evaluation Result
0	What battles took place in New York City in the American Revolution?	The Battle of Long Island was the largest battle of the American Revolutionary War that took place in New York City.	=== American Revolution === The Stamp Act Congress met in New York in October 1765, as the Sons of Liberty organization emerged in the city and skirmished over the next ten years with British troops stationed there. The Battle of Long Island, the largest battle of the American Revolutionary War, was fought in August 1776 within the modern-day borough of Brooklyn. After the battle, in which the Americans were defeated, the British made the city their military and political base of operations in North America. The city was a haven for Loyalist refugees and escaped slaves who joined the British lines for freedom newly promised by the Crown for all fighters. As many as 10,000 escaped slaves crowded into the city during the British occupation. When the British forces evacuated at the close of the war in 1783, they transported 3,000 freedmen for resettlement in Nova Scotia. They resettled other freedmen in England and the Caribbean. The only attempt at a peaceful solution to the war took pl...	Pass

query_str = "What are the airports in New York City?"
query_engine = vector_index.as_query_engine()
response_vector = query_engine.query(query_str)
eval_result = evaluator_gpt4.evaluate_response(
    query=query_str, response=response_vector
)

display_eval_df(query_str, response_vector, eval_result)

	Query	Response	Source	Evaluation Result
0	What are the airports in New York City?	The airports in New York City include John F. Kennedy International Airport, Newark Liberty International Airport, LaGuardia Airport, Stewart International Airport, Long Island MacArthur Airport, Trenton-Mercer Airport, and Westchester County Airport.	Like the New York City Subway, the PATH operates 24 hours a day; meaning three of the six rapid transit systems in the world which operate on 24-hour schedules are wholly or partly in New York (the others are a portion of the Chicago "L", the PATCO Speedline serving Philadelphia, and the Copenhagen Metro).Multibillion-dollar heavy rail transit projects under construction in New York City include the Second Avenue Subway, and the East Side Access project. ==== Buses ==== New York City's public bus fleet runs 24/7 and is the largest in North America. The Port Authority Bus Terminal, the main intercity bus terminal of the city, serves 7,000 buses and 200,000 commuters daily, making it the busiest bus station in the world. === Air === New York's airspace is the busiest in the United States and one of the world's busiest air transportation corridors. The three busiest airports in the New York metropolitan area include John F. Kennedy International Airport, Newark Liberty International...	Pass

query_str = "Who is the mayor of New York City?"
query_engine = vector_index.as_query_engine()
response_vector = query_engine.query(query_str)
eval_result = evaluator_gpt4.evaluate_response(
    query=query_str, response=response_vector
)

display_eval_df(query_str, response_vector, eval_result)

	Query	Response	Source	Evaluation Result
0	Who is the mayor of New York City?	The mayor of New York City is Eric Adams.	=== Politics === The present mayor is Eric Adams. He was elected in 2021 with 67% of the vote, and assumed office on January 1, 2022. The Democratic Party holds the majority of public offices. As of April 2016, 69% of registered voters in the city are Democrats and 10% are Republicans. New York City has not been carried by a Republican presidential election since President Calvin Coolidge won the five boroughs in 1924. A Republican candidate for statewide office has not won all five boroughs of the city since it was incorporated in 1898. In 2012, Democrat Barack Obama became the first presidential candidate of any party to receive more than 80% of the overall vote in New York City, sweeping all five boroughs. Party platforms center on affordable housing, education, and economic development, and labor politics are of importance in the city. Thirteen out of 27 U.S. congressional districts in the state of New York include portions of New York City.New York is one of the most important so...	Pass

Evaluate Source Nodes

Evaluate the set of returned sources, and determine which sources actually contain the answer to a given query.

from typing import List


# define jupyter display function
def display_eval_sources(
    query: str, response: Response, eval_result: List[str]
) -> None:
    sources = [s.node.get_text() for s in response.source_nodes]
    eval_df = pd.DataFrame(
        {
            "Source": sources,
            "Eval Result": eval_result,
        },
    )
    eval_df.style.set_caption(query)
    eval_df = eval_df.style.set_properties(
        **{
            "inline-size": "600px",
            "overflow-wrap": "break-word",
        },
        subset=["Source"]
    )

    display(eval_df)

# NOTE: you can set response_mode="no_text" to get just the sources
query_str = "What are the airports in New York City?"
query_engine = vector_index.as_query_engine(
    similarity_top_k=3, response_mode="no_text"
)
response_vector = query_engine.query(query_str)
eval_source_result_full = [
    evaluator_gpt4.evaluate(
        query=query_str,
        response=response_vector.response,
        contexts=[source_node.get_content()],
    )
    for source_node in response_vector.source_nodes
]
eval_source_result = [
    "Pass" if result.passing else "Fail" for result in eval_source_result_full
]

display_eval_sources(query_str, response_vector, eval_source_result)

	Source	Eval Result
0	Like the New York City Subway, the PATH operates 24 hours a day; meaning three of the six rapid transit systems in the world which operate on 24-hour schedules are wholly or partly in New York (the others are a portion of the Chicago "L", the PATCO Speedline serving Philadelphia, and the Copenhagen Metro).Multibillion-dollar heavy rail transit projects under construction in New York City include the Second Avenue Subway, and the East Side Access project. ==== Buses ==== New York City's public bus fleet runs 24/7 and is the largest in North America. The Port Authority Bus Terminal, the main intercity bus terminal of the city, serves 7,000 buses and 200,000 commuters daily, making it the busiest bus station in the world. === Air === New York's airspace is the busiest in the United States and one of the world's busiest air transportation corridors. The three busiest airports in the New York metropolitan area include John F. Kennedy International Airport, Newark Liberty International Airport, and LaGuardia Airport; 130.5 million travelers used these three airports in 2016. JFK and Newark Liberty were the busiest and fourth busiest U.S. gateways for international air passengers, respectively, in 2012; as of 2011, JFK was the busiest airport for international passengers in North America.Plans have advanced to expand passenger volume at a fourth airport, Stewart International Airport near Newburgh, New York, by the Port Authority of New York and New Jersey. Plans were announced in July 2015 to entirely rebuild LaGuardia Airport in a multibillion-dollar project to replace its aging facilities. Other commercial airports in or serving the New York metropolitan area include Long Island MacArthur Airport, Trenton–Mercer Airport and Westchester County Airport. The primary general aviation airport serving the area is Teterboro Airport.	Pass
1	=== Parks === The city of New York has a complex park system, with various lands operated by the National Park Service, the New York State Office of Parks, Recreation and Historic Preservation, and the New York City Department of Parks and Recreation. In its 2018 ParkScore ranking, the Trust for Public Land reported that the park system in New York City was the ninth-best park system among the fifty most populous U.S. cities. ParkScore ranks urban park systems by a formula that analyzes median park size, park acres as percent of city area, the percent of city residents within a half-mile of a park, spending of park services per resident, and the number of playgrounds per 10,000 residents. In 2021, the New York City Council banned the use of synthetic pesticides by city agencies and instead required organic lawn management. The effort was started by teacher Paula Rogovin's kindergarten class at P.S. 290. ==== National parks ==== Gateway National Recreation Area contains over 26,000 acres (110 km2), most of it in New York City. In Brooklyn and Queens, the park contains over 9,000 acres (36 km2) of salt marsh, wetlands, islands, and water, including most of Jamaica Bay and the Jamaica Bay Wildlife Refuge. Also in Queens, the park includes a significant portion of the western Rockaway Peninsula, most notably Jacob Riis Park and Fort Tilden. In Staten Island, it includes Fort Wadsworth, with historic pre-Civil War era Battery Weed and Fort Tompkins, and Great Kills Park, with beaches, trails, and a marina. The Statue of Liberty National Monument and Ellis Island Immigration Museum are managed by the National Park Service and are in both New York and New Jersey. They are joined in the harbor by Governors Island National Monument. Historic sites under federal management on Manhattan Island include Stonewall National Monument; Castle Clinton National Monument; Federal Hall National Memorial; Theodore Roosevelt Birthplace National Historic Site; General Grant National Memorial (Grant's Tomb); African Burial Ground National Monument; and Hamilton Grange National Memorial. Hundreds of properties are listed on the National Register of Historic Places or as a National Historic Landmark.	Fail
2	Providing continuous 24/7 service and contributing to the nickname The City That Never Sleeps, the New York City Subway is the largest single-operator rapid transit system in the world with 472 passenger rail stations, and Penn Station in Midtown Manhattan is the busiest transportation hub in the Western Hemisphere.The city has over 120 colleges and universities, including Columbia University, an Ivy League university routinely ranked among the world's top universities, New York University, and the City University of New York system, the largest urban public university system in the nation.Anchored by Wall Street in the Financial District of Lower Manhattan, New York City has been called both the world's leading financial and fintech center and the most economically powerful city in the world, and is home to the world's two largest stock exchanges by total market capitalization, the New York Stock Exchange and Nasdaq.The Stonewall Inn in Greenwich Village, part of the Stonewall National Monument, is considered the historic epicenter of LGBTQ+ culture and the birthplace of the modern gay rights movement.New York City is the headquarters of the global art market, with numerous art galleries and auction houses collectively hosting half of the world’s art auctions, and the Metropolitan Museum of Art is both the largest art museum and the most visited museum in the United States.Governors Island in New York Harbor is planned to host a US$1 billion research and education center as a leader the climate crisis. == Etymology == In 1664, New York was named in honor of the Duke of York, who would become King James II of England. James's elder brother, King Charles II, appointed the Duke as proprietor of the former territory of New Netherland, including the city of New Amsterdam, when England seized it from Dutch control. == History == === Early history === In the pre-Columbian era, the area of present-day New York City was inhabited by Algonquian Native Americans, including the Lenape.Their homeland, known as Lenapehoking, included the present-day areas of Staten Island, Manhattan, the Bronx, the western portion of Long Island (including the areas that would later become the boroughs of Brooklyn and Queens), and the Lower Hudson Valley.The first documented visit into New York Harbor by a European was in 1524 by Italian Giovanni da Verrazzano, an explorer from Florence in the service of the French crown.	Fail

# NOTE: you can set response_mode="no_text" to get just the sources
query_str = "Who is the mayor of New York City?"
query_engine = vector_index.as_query_engine(
    similarity_top_k=3, response_mode="no_text"
)
eval_source_result_full = [
    evaluator_gpt4.evaluate(
        query=query_str,
        response=response_vector.response,
        contexts=[source_node.get_content()],
    )
    for source_node in response_vector.source_nodes
]
eval_source_result = [
    "Pass" if result.passing else "Fail" for result in eval_source_result_full
]

display_eval_sources(query_str, response_vector, eval_source_result)

	Source	Eval Result
0	by geographic population boundaries. Each term for the mayor and council members lasts four years and has a two consecutive-term limit, which is reset after a four-year break. The New York City Administrative Code, the New York City Rules, and the City Record are the code of local laws, compilation of regulations, and official journal, respectively.Each borough is coextensive with a judicial district of the state Unified Court System, of which the Criminal Court and the Civil Court are the local courts, while the New York Supreme Court conducts major trials and appeals. Manhattan hosts the First Department of the Supreme Court, Appellate Division while Brooklyn hosts the Second Department. There are also several extrajudicial administrative courts, which are executive agencies and not part of the state Unified Court System. Uniquely among major American cities, New York is divided between, and is host to the main branches of, two different U.S. district courts: the District Court for the Southern District of New York, whose main courthouse is on Foley Square near City Hall in Manhattan and whose jurisdiction includes Manhattan and the Bronx; and the District Court for the Eastern District of New York, whose main courthouse is in Brooklyn and whose jurisdiction includes Brooklyn, Queens, and Staten Island. The U.S. Court of Appeals for the Second Circuit and U.S. Court of International Trade are also based in New York, also on Foley Square in Manhattan. === Politics === The present mayor is Eric Adams. He was elected in 2021 with 67% of the vote, and assumed office on January 1, 2022. The Democratic Party holds the majority of public offices. As of April 2016, 69% of registered voters in the city are Democrats and 10% are Republicans. New York City has not been carried by a Republican presidential election since President Calvin Coolidge won the five boroughs in 1924. A Republican candidate for statewide office has not won all five boroughs of the city since it was incorporated in 1898. In 2012, Democrat Barack Obama became the first presidential candidate of any party to receive more than 80% of the overall vote in New York City, sweeping all	NO
1	for the Second Circuit and U.S. Court of International Trade are also based in New York, also on Foley Square in Manhattan. === Politics === The present mayor is Eric Adams. He was elected in 2021 with 67% of the vote, and assumed office on January 1, 2022. The Democratic Party holds the majority of public offices. As of April 2016, 69% of registered voters in the city are Democrats and 10% are Republicans. New York City has not been carried by a Republican presidential election since President Calvin Coolidge won the five boroughs in 1924. A Republican candidate for statewide office has not won all five boroughs of the city since it was incorporated in 1898. In 2012, Democrat Barack Obama became the first presidential candidate of any party to receive more than 80% of the overall vote in New York City, sweeping all five boroughs. Party platforms center on affordable housing, education, and economic development, and labor politics are of importance in the city. Thirteen out of 27 U.S. congressional districts in the state of New York include portions of New York City.New York is one of the most important sources of political fundraising in the United States. At least four of the top five ZIP Codes in the nation for political contributions were in Manhattan for the 2004, 2006, and 2008 elections. The top ZIP Code, 10021 on the Upper East Side, generated the most money for the 2004 presidential campaigns of George W. Bush and John Kerry. The city has a strong imbalance of payments with the national and state governments. It receives 83 cents in services for every $1 it sends to the federal government in taxes (or annually sends $11.4 billion more than it receives back). City residents and businesses also sent an additional $4.1 billion in the 2009–2010 fiscal year to the state of New York than the city received in return. == Transportation == New York City's comprehensive transportation system is both complex and extensive. === Rapid transit === Mass transit in New York City, most of which runs 24 hours a day, accounts for one in every three users of mass transit in	YES
2	ed.). New York: The New Press. ISBN 978-1-56584-321-9. Holli, Melvin G., and Jones, Peter d'A., eds. Biographical Dictionary of American Mayors, 1820-1980 (Greenwood Press, 1981) short scholarly biographies each of the city's mayors 1820 to 1980. online; see index at p. 410 for list.Jackson, Kenneth T., ed. (1995). The Encyclopedia of New York City. New Haven: Yale University Press. ISBN 0300055366. Jackson, Kenneth T.; Dunbar, David S., eds. (2005). Empire City: New York Through the Centuries. Columbia University Press. ISBN 978-0-231-10909-3. Lankevich, George L. (1998). American Metropolis: A History of New York City. NYU Press. ISBN 978-0-8147-5186-2. White, E.B. (1949). Here is New York (2000 reissue ed.). Little Bookroom. White, Norval & Willensky, Elliot (2000). AIA Guide to New York City (4th ed.). New York: Three Rivers Press. ISBN 978-0-8129-3107-5. Whitehead, Colson (2003). The Colossus of New York: A City in 13 Parts. New York: Doubleday. ISBN 978-0-385-50794-3. == External links == Official website NYC Go, official tourism website New York City at Curlie Geographic data related to New York City at OpenStreetMap Collections, 145,000 NYC photographs at the Museum of the City of New York "The New New York Skyline (interactive)". National Geographic. November 2015.	NO