Simple Vector Store - Async Index Creation¶

If you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙.

In [ ]:

%pip install llama-index-readers-wikipedia

%pip install llama-index-readers-wikipedia

In [ ]:

!pip install llama-index

!pip install llama-index

In [ ]:

import time

# Helps asyncio run within Jupyter
import nest_asyncio

nest_asyncio.apply()

# My OpenAI Key
import os

os.environ["OPENAI_API_KEY"] = "[YOUR_API_KEY]"

import time # Helps asyncio run within Jupyter import nest_asyncio nest_asyncio.apply() # My OpenAI Key import os os.environ["OPENAI_API_KEY"] = "[YOUR_API_KEY]"

In [ ]:

from llama_index.core import VectorStoreIndex, download_loader

from llama_index.readers.wikipedia import WikipediaReader

loader = WikipediaReader()
documents = loader.load_data(
    pages=[
        "Berlin",
        "Santiago",
        "Moscow",
        "Tokyo",
        "Jakarta",
        "Cairo",
        "Bogota",
        "Shanghai",
        "Damascus",
    ]
)

from llama_index.core import VectorStoreIndex, download_loader from llama_index.readers.wikipedia import WikipediaReader loader = WikipediaReader() documents = loader.load_data( pages=[ "Berlin", "Santiago", "Moscow", "Tokyo", "Jakarta", "Cairo", "Bogota", "Shanghai", "Damascus", ] )

In [ ]:

len(documents)

len(documents)

Out[ ]:

9

9 Wikipedia articles downloaded as documents

In [ ]:

start_time = time.perf_counter()
index = VectorStoreIndex.from_documents(documents)
duration = time.perf_counter() - start_time
print(duration)

start_time = time.perf_counter() index = VectorStoreIndex.from_documents(documents) duration = time.perf_counter() - start_time print(duration)

INFO:root:> [build_index_from_documents] Total LLM token usage: 0 tokens
INFO:root:> [build_index_from_documents] Total embedding token usage: 142295 tokens

7.691995083000052

Standard index creation took 7.69 seconds

In [ ]:

start_time = time.perf_counter()
index = VectorStoreIndex(documents, use_async=True)
duration = time.perf_counter() - start_time
print(duration)

start_time = time.perf_counter() index = VectorStoreIndex(documents, use_async=True) duration = time.perf_counter() - start_time print(duration)

INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/engines/text-embedding-ada-002/embeddings processing_ms=245 request_id=314b145a07f65fd34e707f633cc1a444 response_code=200
INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/engines/text-embedding-ada-002/embeddings processing_ms=432 request_id=bb9e796d0b8f9c2365b68de8a56009ff response_code=200
INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/engines/text-embedding-ada-002/embeddings processing_ms=433 request_id=7a94707fe2f8916e9cdd8276a5748207 response_code=200
INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/engines/text-embedding-ada-002/embeddings processing_ms=499 request_id=cda679215293c3a13ed57c2eae3dc582 response_code=200
INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/engines/text-embedding-ada-002/embeddings processing_ms=527 request_id=5e1c3e74aa3f9f950e4035f81a0f0a15 response_code=200
INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/engines/text-embedding-ada-002/embeddings processing_ms=585 request_id=81983fe76eab95f73f82df881ff7b2d9 response_code=200
INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/engines/text-embedding-ada-002/embeddings processing_ms=574 request_id=702a182b54a29a33719205f722378c8e response_code=200
INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/engines/text-embedding-ada-002/embeddings processing_ms=575 request_id=d1df11775c59a3ba403dda253081f8eb response_code=200
INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/engines/text-embedding-ada-002/embeddings processing_ms=575 request_id=47929f13469569527505b51958cd8e71 response_code=200
INFO:root:> [build_index_from_documents] Total LLM token usage: 0 tokens
INFO:root:> [build_index_from_documents] Total embedding token usage: 142295 tokens

2.3730635830000892

Async index creation took 2.37 seconds

In [ ]:

query_engine = index.as_query_engine()
query_engine.query("What is the etymology of Jakarta?")

query_engine = index.as_query_engine() query_engine.query("What is the etymology of Jakarta?")

INFO:root:> [query] Total LLM token usage: 4075 tokens
INFO:root:> [query] Total embedding token usage: 8 tokens

Out[ ]:

Response(response="\n\nThe name 'Jakarta' is derived from the word Jayakarta (Devanagari: जयकर्त) which is ultimately derived from the Sanskrit जय jaya (victorious), and कृत krta (accomplished, acquired), thus Jayakarta translates as 'victorious deed', 'complete act' or 'complete victory'. It was named for the Muslim troops of Fatahillah which successfully defeated and drove the Portuguese away from the city in 1527. Before it was called Jayakarta, the city was known as 'Sunda Kelapa'. Tomé Pires, a Portuguese apothecary wrote the name of the city on his magnum opus as Jacatra or Jacarta during his journey to East Indies. The city is located in a low-lying area ranging from −2 to 91 m (−7 to 299 ft) with an average elevation of 8 m (26 ft) above sea level with historically extensive swampy areas. Some parts of the city have been constructed on reclaimed tidal flats that occur around the area. Thirteen rivers flow through Jakarta, including the Ciliwung River, Kalibaru, Pesanggra", source_nodes=[SourceNode(source_text="Jakarta (; Indonesian pronunciation: [dʒaˈkarta] (listen)), officially the Special Capital Region of Jakarta (Indonesian: Daerah Khusus Ibukota Jakarta), is the capital and largest city of Indonesia. Lying on the northwest coast of Java, the world's most populous island, Jakarta is the largest city in Southeast Asia and serves as the diplomatic capital of ASEAN.\nThe city is the economic, cultural, and political centre of Indonesia. It possesses a province-level status and has a population of 10,562,088 as of mid-2021. Although Jakarta extends over only 664.01 km2 (256.38 sq mi) and thus has the smallest area of any Indonesian province, its metropolitan area covers 9,957.08 km2 (3,844.45 sq mi), which includes the satellite cities Bogor, Depok, Tangerang, South Tangerang, and Bekasi, and has an estimated population of 35 million as of 2021, making it the largest urban area in Indonesia and the second-largest in the world (after Tokyo). Jakarta ranks first among the Indonesian provinces in the human development index. Jakarta's business and employment opportunities, along with its ability to offer a potentially higher standard of living compared to other parts of the country, have attracted migrants from across the Indonesian archipelago, making it a melting pot of numerous cultures.\nJakarta is one of the oldest continuously inhabited cities in Southeast Asia. Established in the fourth century as Sunda Kelapa, the city became an important trading port for the Sunda Kingdom. At one time, it was the de facto capital of the Dutch East Indies, when it was known as Batavia. Jakarta was officially a city within West Java until 1960 when its official status was changed to a province with special capital region distinction. As a province, its government consists of five administrative cities and one administrative regency. Jakarta is an alpha world city and is the seat of the ASEAN secretariat. Financial institutions such as the Bank of Indonesia, Indonesia Stock Exchange, and corporate headquarters of numerous Indonesian companies and multinational corporations are located in the city. In 2021, the city's GRP PPP was estimated at US$602.946 billion.\nJakarta's main challenges include rapid urban growth, ecological breakdown, gridlocked traffic, congestion, and flooding. Jakarta is sinking up to 17 cm (6.7 inches) annually, which coupled with the rising of sea levels, has made the city more prone to flooding. Hence, it is one of the fastest-sinking capitals in the world. In response to these challenges, in August 2019, President Joko Widodo announced that the capital of Indonesia would be moved from Jakarta to the planned city of Nusantara, in the province of East Kalimantan on the island of Borneo.\n\n\n== Name ==\n\nJakarta has been home to multiple settlements. Below is the list of names used during its existence:\n\nSunda Kelapa (397–1527)\nJayakarta (1527–1619)\nBatavia (1619–1942)\nDjakarta (1942–1972)\nJakarta (1972–present)The name 'Jakarta' is derived from the word Jayakarta (Devanagari: जयकर्त) which is ultimately derived from the Sanskrit जय jaya (victorious), and कृत krta (accomplished, acquired), thus Jayakarta translates as 'victorious deed', 'complete act' or 'complete victory'. It was named for the Muslim troops of Fatahillah which successfully defeated and drove the Portuguese away from the city in 1527. Before it was called Jayakarta, the city was known as 'Sunda Kelapa'. Tomé Pires, a Portuguese apothecary wrote the name of the city on his magnum opus as Jacatra or Jacarta during his journey to East Indies. \nIn the 17th century, the city was known as Koningin van het Oosten (Queen of the Orient), a name that was given for the urban beauty of downtown Batavia's canals, mansions and ordered city layout. After expanding to the south in the 19th century, this nickname came to be more associated with the suburbs (e.g. Menteng and the area around Merdeka Square), with their wide lanes, green spaces and villas. During the Japanese occupation, the city was renamed as Jakaruta Tokubetsu-shi (ジャカルタ特別市, Jakarta Special City).\n\n\n== History ==\n\n\n=== Precolonial era ===\n\nThe north coast area of western Java including Jakarta was the location of prehistoric Buni culture that flourished from 400 BC to 100 AD. The area in and around modern Jakarta was part of the 4th-century Sundanese kingdom of Tarumanagara, one of the oldest Hindu kingdoms in Indonesia. The area of North Jakarta around Tugu became a populated settlement in the early 5th century. The Tugu inscription (probably written around 417 AD) discovered in Batutumbuh hamlet, Tugu village, Koja, North Jakarta, mentions that King Purnawarman of Tarumanagara undertook hydraulic projects; the irrigation and water drainage project of the Chandrabhaga river and the Gomati river near his capital. Following the decline of Tarumanagara, its territories, including the Jakarta area, became part of the Hindu Kingdom of Sunda. From the 7th to the early 13th century, the port of Sunda was under the Srivijaya maritime empire. According to the Chinese source, Chu-fan-chi, written circa 1225, Chou Ju-kua reported in the early 13th century that Srivijaya still ruled Sumatra, the Malay peninsula and western Java (Sunda). The source says the port of Sunda is strategic and thriving, mentioning pepper from Sunda as among the best in quality. The people worked in agriculture, and their houses were built on wooden piles. The harbour area became known as Sunda Kelapa, (Sundanese: ᮞᮥᮔ᮪ᮓ ᮊᮨᮜᮕ) and by the 14th century, it was an important trading port for the Sunda Kingdom.\nThe first European fleet, four Portuguese ships from Malacca, arrived in 1513 while looking for a route for spices. The Sunda Kingdom made an alliance treaty with the Portuguese by allowing them to build a port in 1522 to defend against the rising power of Demak Sultanate from central Java. In 1527, Fatahillah, a Javanese general from Demak attacked and conquered Sunda Kelapa, driving out the Portuguese. Sunda Kelapa was renamed Jayakarta, and became a fiefdom of the Banten Sultanate, which became a major Southeast Asian trading centre.\nThrough the relationship with Prince Jayawikarta of the Banten Sultanate, Dutch ships arrived in 1596. In 1602, the British East India Company's first voyage, commanded by Sir James Lancaster, arrived in Aceh and sailed on to Banten where they were allowed to build a trading post. This site became the centre of British trade in the Indonesian archipelago until 1682. Jayawikarta is thought to have made trading connections with the British merchants, rivals of the Dutch, by allowing them to build houses directly across from the Dutch buildings in 1615.\n\n\n=== Colonial era ===\n\nWhen relations between Prince Jayawikarta and the Dutch deteriorated, his soldiers attacked the Dutch fortress. His army and the British, however, were defeated by the Dutch, in part owing to the timely arrival of Jan Pieterszoon Coen. The Dutch burned the British fort and forced them to retreat on their ships. The victory consolidated Dutch power, and they renamed the city Batavia in 1619.\n\nCommercial opportunities in the city attracted native and especially Chinese and Arab immigrants. This sudden population increase created burdens on the city. Tensions grew as the colonial government tried to restrict Chinese migration through deportations. Following a revolt, 5,000 Chinese were massacred by the Dutch and natives on 9 October 1740, and the following year, Chinese inhabitants were moved to Glodok outside the city walls. At the beginning of the 19th century, around 400 Arabs and Moors lived in Batavia, a number that changed little during the following decades. Among the commodities traded were fabrics, mainly imported cotton, batik and clothing worn by Arab communities.The city began to expand further south as epidemics in 1835 and 1870 forced residents to move away from the port. The Koningsplein, now Merdeka Square was completed in 1818, the housing park of Menteng was started in 1913, and Kebayoran Baru was the last Dutch-built residential area. By 1930, Batavia had more than 500,000 inhabitants, including 37,067 Europeans.On 5 March 1942, the Japanese captured Batavia from Dutch control, and the city was named Jakarta (Jakarta Special City (ジャカルタ特別市, Jakaruta tokubetsu-shi), under the special status that was assigned to the city). After the war, the Dutch name Batavia was internationally recognised until full Indonesian independence on 27 December 1949. The city, now renamed Jakarta, was officially proclaimed the national capital of Indonesia.\n\n\n=== Independence era ===\n\nAfter World War II ended, Indonesian nationalists declared independence on 17 August 1945, and the government of Jakarta City was changed into the Jakarta National Administration in the following month. During the Indonesian National Revolution, Indonesian Republicans withdrew from Allied-occupied Jakarta and established their capital in Yogyakarta.\nAfter securing full independence, Jakarta again became the national capital in 1950. With Jakarta selected to host the 1962 Asian Games, Soekarno, envisaging Jakarta as a great international city, instigated large government-funded projects with openly nationalistic and modernist architecture. Projects included a cloverleaf interchange, a major boulevard (Jalan MH Thamrin-Sudirman), monuments such as The National Monument, Hotel Indonesia, a shopping centre, and a new building intended to be the headquarters of CONEFO. In October 1965, Jakarta was the site of an abortive coup attempt in which six top generals were killed, precipitating a violent anti-communist purge which killed at least 500,000 people, including some ethnic Chinese. The event marked the beginning of Suharto's New Order. The first government was led by a mayor until the end of 1960 when the office was changed to that of a governor. The last mayor of Jakarta was Soediro until he was replaced by Soemarno Sosroatmodjo as governor. Based on law No. 5 of 1974 relating to regional governments, Jakarta was confirmed as the capital of Indonesia and one of the country's then 26 provinces.In 1966, Jakarta was declared a 'special capital region' (Daerah Khusus Ibukota), with a status equivalent to that of a province. Lieutenant General Ali Sadikin served as governor from 1966 to 1977; he rehabilitated roads and bridges, encouraged the arts, built hospitals and a large number of schools. He cleared out slum dwellers for new development projects — some for the benefit of the Suharto family,— and attempted to eliminate rickshaws and ban street vendors. He began control of migration to the city to stem overcrowding and poverty. Foreign investment contributed to a real estate boom that transformed the face of Jakarta. The boom ended with the 1997 Asian financial crisis, putting Jakarta at the centre of violence, protest, and political manoeuvring.\nAfter three decades in power, support for President Suharto began to wane. Tensions peaked when four students were shot dead at Trisakti University by security forces. Four days of riots and violence in 1998 ensued that killed an estimated 1,200, and destroyed or damaged 6,000 buildings, forcing Suharto to resign. Much of the rioting targeted Chinese Indonesians. In the post-Suharto era, Jakarta has remained the focal point of democratic change in Indonesia. Jemaah Islamiah-connected bombings occurred almost annually in the city between 2000 and 2005, with another in 2009. In August 2007, Jakarta held its first-ever election to choose a governor as part of a nationwide decentralisation program that allows direct local elections in several areas. Previously, governors were elected by the city's legislative body.During the Jokowi presidency, the Government adopted a plan to move Indonesia's capital to East Kalimantan.Between 2016 and 2017, a series of terrorist attacks rocked Jakarta with scenes of multiple suicide bombings and gunfire. In suspicion to its links, the Islamic State, the perpetrator led by Abu Bakr al-Baghdadi claimed responsibility for the attacks.\n\n\n== Geography ==\n\nJakarta covers 699.5 km2 (270.1 sq mi), the smallest among any Indonesian provinces. However, its metropolitan area covers 6,392 km2 (2,468 sq mi), which extends into two of the bordering provinces of West Java and Banten. The Greater Jakarta area includes three bordering regencies (Bekasi Regency, Tangerang Regency and Bogor Regency) and five adjacent cities (Bogor, Depok, Bekasi, Tangerang and South Tangerang).\n\nJakarta is situated on the northwest coast of Java, at the mouth of the Ciliwung River on Jakarta Bay, an inlet of the Java Sea.  It is strategically located near the Sunda Strait. The northern part of Jakarta is plain land, some areas of which are below sea level, and subject to frequent flooding. The southern parts of the city are hilly. It is one of only two Asian capital cities located in the southern hemisphere (along with East Timor's Dili). Officially, the area of the Jakarta Special District is 662 km2 (256 sq mi) of land area and 6,977 km2 (2,694 sq mi) of sea area. The Thousand Islands, which are administratively a part of Jakarta, are located in Jakarta Bay, north of the city.\nJakarta lies in a low and flat alluvial plain, ranging from −2 to 91 m (−7 to 299 ft) with an average elevation of 8 m (26 ft) above sea level with historically extensive swampy areas. Some parts of the city have been constructed on reclaimed tidal flats that occur around the area. Thirteen rivers flow through Jakarta. They are Ciliwung River, Kalibaru, Pesanggrahan, Cipinang, Angke River, Maja, Mookervart, Krukut, Buaran, West Tarum, Cakung, Petukangan, Sunter River and Grogol River. They flow from the Puncak highlands to the south of the city, then across the city northwards towards the Java Sea. The Ciliwung River divides the city into the western and eastern districts.\nThese rivers, combined with the wet season rains and insufficient", doc_id='eeb6ef32-c857-44e2-b0c5-dff6e29a9cd7', extra_info=None, node_info={'start': 0, 'end': 13970}, similarity=0.8701780916463354)], extra_info=None)