Anthropic¶

Anthropic has recently released its latest models: Claude 3 Opus, Claude 3 Sonnet, and Claude 3 Haiku (which will be available soon). By default, the claude-2.1 model is used. This notebook provides guidance on how to utilize these new models.

1. Claude 3 Opus - claude-3-opus-20240229
2. Claude 3 Sonnet - claude-3-sonnet-20240229
3. Claude 3 Haiku - claude-3-haiku-20240307

If you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙.

In [ ]:

%pip install llama-index-llms-anthropic

%pip install llama-index-llms-anthropic

In [ ]:

!pip install llama-index

!pip install llama-index

Set Tokenizer¶

First we want to set the tokenizer, which is slightly different than TikToken.

NOTE: The Claude 3 tokenizer has not been updated yet; using the existing Anthropic tokenizer leads to context overflow errors for 200k tokens. We've temporarily set the max tokens for Claude 3 to 180k.

In [ ]:

from llama_index.llms.anthropic import Anthropic
from llama_index.core import Settings

tokenizer = Anthropic().tokenizer
Settings.tokenizer = tokenizer

from llama_index.llms.anthropic import Anthropic from llama_index.core import Settings tokenizer = Anthropic().tokenizer Settings.tokenizer = tokenizer

Call complete with a prompt¶

In [ ]:

import os

os.environ["ANTHROPIC_API_KEY"] = "YOUR ANTHROPIC API KEY"

import os os.environ["ANTHROPIC_API_KEY"] = "YOUR ANTHROPIC API KEY"

In [ ]:

from llama_index.llms.anthropic import Anthropic

# To customize your API key, do this
# otherwise it will lookup ANTHROPIC_API_KEY from your env variable
# llm = Anthropic(api_key="<api_key>")
llm = Anthropic(model="claude-3-opus-20240229")

resp = llm.complete("Paul Graham is ")

from llama_index.llms.anthropic import Anthropic # To customize your API key, do this # otherwise it will lookup ANTHROPIC_API_KEY from your env variable # llm = Anthropic(api_key="") llm = Anthropic(model="claude-3-opus-20240229") resp = llm.complete("Paul Graham is ")

In [ ]:

print(resp)

print(resp)

Paul Graham is a well-known entrepreneur, programmer, venture capitalist, and essayist. He is best known for co-founding Viaweb, one of the first web application companies, which was later sold to Yahoo! in 1998 and became Yahoo! Store. Graham is also the co-founder of Y Combinator, a highly successful startup accelerator that has helped launch numerous successful companies, such as Dropbox, Airbnb, and Reddit.

Some key points about Paul Graham:

1. Programming: Graham is a skilled programmer and has written extensively on the subject, including his book "Hackers & Painters: Big Ideas from the Computer Age."

2. Essays: He is a prolific essayist, writing on various topics related to technology, startups, and entrepreneurship. His essays have been influential in the tech startup community.

3. Lisp: Graham is an advocate for the Lisp programming language and has written several essays on its advantages.

4. Y Combinator: As a co-founder of Y Combinator, Graham has played a significant role in shaping the startup ecosystem and has mentored and invested in numerous successful companies.

5. Wealth and inequality: In recent years, Graham has written about income inequality and the concentration of wealth, sparking discussions and debates within the tech community.

Overall, Paul Graham is a significant figure in the technology and startup world, known for his contributions as a programmer, investor, and thought leader.

You can also use an anthropic model through Vertex AI¶

In [ ]:

import os

os.environ["ANTHROPIC_PROJECT_ID"] = "YOUR PROJECT ID HERE"
os.environ["ANTHROPIC_REGION"] = "YOUR PROJECT REGION HERE"

import os os.environ["ANTHROPIC_PROJECT_ID"] = "YOUR PROJECT ID HERE" os.environ["ANTHROPIC_REGION"] = "YOUR PROJECT REGION HERE"

Do keep in mind that setting region and project_id here will make Anthropic use the Vertex AI client¶

In [ ]:

from llama_index.llms.anthropic import Anthropic

llm = Anthropic(
    model="claude-3-5-sonnet@20240620",
    region=os.getenv("ANTHROPIC_REGION"),
    project_id=os.getenv("ANTHROPIC_PROJECT_ID"),
)

resp = llm.complete("Paul Graham is ")

from llama_index.llms.anthropic import Anthropic llm = Anthropic( model="claude-3-5-sonnet@20240620", region=os.getenv("ANTHROPIC_REGION"), project_id=os.getenv("ANTHROPIC_PROJECT_ID"), ) resp = llm.complete("Paul Graham is ")

In [ ]:

print(resp)

print(resp)

Paul Graham is a well-known computer programmer, entrepreneur, venture capitalist, and essayist. Here are some key points about him:

1. Co-founder of Y Combinator: Graham is best known for co-founding Y Combinator, one of the most successful startup accelerators in the world.

2. Programming language creator: He created the programming language Arc, a dialect of Lisp.

3. Entrepreneur: Before Y Combinator, he co-founded Viaweb, one of the first web-based application companies, which was later acquired by Yahoo!.

4. Author: Graham has written several books on programming and startups, including "Hackers & Painters" and "On Lisp."

5. Essayist: He is known for his insightful essays on technology, startups, and society, which are widely read in the tech community.

6. Investor: Through Y Combinator and personally, he has invested in numerous successful startups, including Dropbox, Airbnb, and Reddit.

7. Advocate for startups: He has been a strong proponent of entrepreneurship and has helped shape the modern startup ecosystem.

8. Computer Science background: Graham holds a Ph.D. in Computer Science from Harvard University.

9. Influential thinker: His ideas on startups, technology, and society have had a significant impact on Silicon Valley and the broader tech industry.

Paul Graham is widely respected for his contributions to the tech industry, his insights on startups and innovation, and his role in shaping the modern entrepreneurial landscape.

Call chat with a list of messages¶

In [ ]:

from llama_index.core.llms import ChatMessage
from llama_index.llms.anthropic import Anthropic

messages = [
    ChatMessage(
        role="system", content="You are a pirate with a colorful personality"
    ),
    ChatMessage(role="user", content="Tell me a story"),
]
resp = Anthropic(model="claude-3-opus-20240229").chat(messages)

from llama_index.core.llms import ChatMessage from llama_index.llms.anthropic import Anthropic messages = [ ChatMessage( role="system", content="You are a pirate with a colorful personality" ), ChatMessage(role="user", content="Tell me a story"), ] resp = Anthropic(model="claude-3-opus-20240229").chat(messages)

In [ ]:

print(resp)

print(resp)

assistant: *clears throat and speaks in a pirate accent* Aye, gather 'round me hearties and I'll spin ye a yarn of adventure on the high seas!

T'was a dark and stormy night when the Black Pearl set sail from Tortuga. The salty sea spray stung me eyes as I stood at the helm, guidin' me beloved ship through the roilin' waves. Me loyal crew scurried about, securin' the riggin' and battening down the hatches. 

Suddenly, the lookout cried "Ship ahoy!" and pointed off the starboard bow. I raised me spyglass and spied a Spanish galleon, her decks heavily laden with treasure. The crew gave a hearty cheer - we'd be feastin' and drinkin' well tonight!

I ordered the crew to ready the cannons as we drew alongside the galleon. "Fire all!" I bellowed and the Pearl shook as the guns unleashed a barrage. The Spaniards returned fire but they were no match for me skilled gunners.

We boarded the galleon, swords flashin' and pistols blazin'. The fight was fast and bloody but in the end, the Pearl was victorious! We claimed the treasure as our own - mountains of gold and jewels glintin' in the moonlight.

As we sailed away, I couldn't help but grin. T'was a fine night of piratin' and I knew many more adventures lay ahead for me and me crew. No matter the danger, the Black Pearl would always prevail! Yo ho ho!

*laughs heartily* And that, me friends, is a taste of the pirate's life. May yer sails always be full and yer horizons bright. Fare thee well!

Streaming¶

Using stream_complete endpoint

In [ ]:

from llama_index.llms.anthropic import Anthropic

llm = Anthropic(model="claude-3-opus-20240229", max_tokens=100)
resp = llm.stream_complete("Paul Graham is ")

from llama_index.llms.anthropic import Anthropic llm = Anthropic(model="claude-3-opus-20240229", max_tokens=100) resp = llm.stream_complete("Paul Graham is ")

In [ ]:

for r in resp:
    print(r.delta, end="")

for r in resp: print(r.delta, end="")

Paul Graham is a well-known entrepreneur, programmer, venture capitalist, and essayist. He is best known for co-founding Viaweb, one of the first web application companies, which was later sold to Yahoo! in 1998 and became Yahoo! Store. 

After the sale of Viaweb, Graham and his wife Jessica Livingston co-founded Y Combinator in 2005, a highly successful startup accelerator that has helped launch

In [ ]:

from llama_index.llms.anthropic import Anthropic

llm = Anthropic(model="claude-3-opus-20240229")
messages = [
    ChatMessage(
        role="system", content="You are a pirate with a colorful personality"
    ),
    ChatMessage(role="user", content="Tell me a story"),
]
resp = llm.stream_chat(messages)

from llama_index.llms.anthropic import Anthropic llm = Anthropic(model="claude-3-opus-20240229") messages = [ ChatMessage( role="system", content="You are a pirate with a colorful personality" ), ChatMessage(role="user", content="Tell me a story"), ] resp = llm.stream_chat(messages)

In [ ]:

for r in resp:
    print(r.delta, end="")

for r in resp: print(r.delta, end="")

*clears throat and speaks in a gruff, piratey voice* 

Aye, gather 'round me hearties and I'll spin ye a yarn of adventure on the high seas! 

'Twas a dark and stormy night, the kind where the wind howls like a banshee and the waves crash over the deck. Me and me crew were sailin' the Caribbean, searchin' for treasure and glory.

Suddenly, the lookout cried "Ship ahoy!" and sure enough, a Spanish galleon was bearin' down on us, her decks bristlin' with cannons. The scurvy dogs wanted our gold, but I'd sooner walk the plank than surrender!

"All hands to battle stations!" I bellowed. "Ready the cannons and prepare to board!" 

A mighty battle erupted, cannons boomin' and swords clashin'. We swung over on ropes and fought the Spaniards hand-to-hand on the pitchin' and rollin' deck. Me cutlass was a blur as I dueled their captain, a big brute with a wicked scar.

Finally, I drove me blade into that bilge rat's black heart and he fell dead at me feet. His crew surrendered and we took their ship as a prize. In the hold, we found chests overflowing with gold doubloons and jewels - a king's ransom! 

We sailed off into the sunset, our pirate flag snappin' in the breeze, flush with coin and the thrill of victory. And that, me buckos, is a taste of the pirate life! Now who wants some grog?

*laughs heartily*

Configure Model¶

In [ ]:

from llama_index.llms.anthropic import Anthropic

llm = Anthropic(model="claude-3-sonnet-20240229")

from llama_index.llms.anthropic import Anthropic llm = Anthropic(model="claude-3-sonnet-20240229")

In [ ]:

resp = llm.stream_complete("Paul Graham is ")

resp = llm.stream_complete("Paul Graham is ")

In [ ]:

for r in resp:
    print(r.delta, end="")

for r in resp: print(r.delta, end="")

Paul Graham is a computer scientist, entrepreneur, venture capitalist, and author. He is best known for the following:

1. Co-founding Y Combinator: Y Combinator is a prominent startup accelerator based in Silicon Valley. It has funded and helped launch thousands of startups, including Airbnb, Dropbox, Stripe, and Reddit.

2. Writing essays on startups and technology: Graham has written numerous influential essays on topics related to startups, programming, and entrepreneurship. His essays are widely read and have helped shape the thinking of many entrepreneurs and technologists.

3. Developing the programming language Arc: In the early 2000s, Graham developed a new programming language called Arc, which was designed to be a more powerful and expressive dialect of Lisp.

4. Advocating for the use of Lisp and functional programming: Graham is a strong proponent of the Lisp programming language and functional programming paradigms. He has written extensively about the benefits of these approaches and has influenced many programmers to explore them.

5. Authoring books: Graham has authored several books, including "Hackers & Painters: Big Ideas from the Computer Age" (2004), "On Lisp" (1993), and "ANSI Common Lisp" (1995).

6. Investing in startups: Through Y Combinator and his own investments, Graham has invested in and advised numerous successful startups, helping to shape the technology industry.

Overall, Paul Graham is widely respected in the technology and startup communities for his contributions as a programmer, writer, investor, and advocate for innovative ideas and approaches.

Async¶

In [ ]:

from llama_index.llms.anthropic import Anthropic

llm = Anthropic("claude-3-sonnet-20240229")
resp = await llm.acomplete("Paul Graham is ")

from llama_index.llms.anthropic import Anthropic llm = Anthropic("claude-3-sonnet-20240229") resp = await llm.acomplete("Paul Graham is ")

In [ ]:

print(resp)

print(resp)

Paul Graham is a computer scientist, entrepreneur, venture capitalist, and author. He is best known for the following:

1. Co-founding Y Combinator: Y Combinator is a prominent startup accelerator based in Silicon Valley. It has funded and helped launch many successful startups, including Airbnb, Dropbox, Stripe, and Reddit.

2. Writing essays on startups and technology: Graham has written numerous influential essays on topics related to startups, programming, and entrepreneurship. His essays are widely read and have helped shape the thinking of many entrepreneurs and technologists.

3. Developing the programming language Arc: Graham designed and developed the programming language Arc, which was intended to be a more powerful and expressive dialect of Lisp.

4. Authoring books: He has written several books, including "Hackers & Painters: Big Ideas from the Computer Age," "ANSI Common Lisp," and "On Lisp."

5. Founding Viaweb: In the 1990s, Graham co-founded Viaweb, one of the earliest web-based application software companies. Viaweb was later acquired by Yahoo! in 1998.

Graham is widely respected in the technology and startup communities for his insights, writings, and contributions to the field of computer science and entrepreneurship.

Structured Prediction¶

LlamaIndex provides an intuitive interface for converting any Anthropic LLMs into a structured LLM through structured_predict - simply define the target Pydantic class (can be nested), and given a prompt, we extract out the desired object.

In [ ]:

from llama_index.llms.anthropic import Anthropic
from llama_index.core.prompts import PromptTemplate
from llama_index.core.bridge.pydantic import BaseModel
from typing import List


class MenuItem(BaseModel):
    """A menu item in a restaurant."""

    course_name: str
    is_vegetarian: bool


class Restaurant(BaseModel):
    """A restaurant with name, city, and cuisine."""

    name: str
    city: str
    cuisine: str
    menu_items: List[MenuItem]


llm = Anthropic("claude-3-5-sonnet-20240620")
prompt_tmpl = PromptTemplate(
    "Generate a restaurant in a given city {city_name}"
)

# Option 1: Use `as_structured_llm`
restaurant_obj = (
    llm.as_structured_llm(Restaurant)
    .complete(prompt_tmpl.format(city_name="Miami"))
    .raw
)
# Option 2: Use `structured_predict`
# restaurant_obj = llm.structured_predict(Restaurant, prompt_tmpl, city_name="Miami")

from llama_index.llms.anthropic import Anthropic from llama_index.core.prompts import PromptTemplate from llama_index.core.bridge.pydantic import BaseModel from typing import List class MenuItem(BaseModel): """A menu item in a restaurant.""" course_name: str is_vegetarian: bool class Restaurant(BaseModel): """A restaurant with name, city, and cuisine.""" name: str city: str cuisine: str menu_items: List[MenuItem] llm = Anthropic("claude-3-5-sonnet-20240620") prompt_tmpl = PromptTemplate( "Generate a restaurant in a given city {city_name}" ) # Option 1: Use `as_structured_llm` restaurant_obj = ( llm.as_structured_llm(Restaurant) .complete(prompt_tmpl.format(city_name="Miami")) .raw ) # Option 2: Use `structured_predict` # restaurant_obj = llm.structured_predict(Restaurant, prompt_tmpl, city_name="Miami")

In [ ]:

restaurant_obj

restaurant_obj

Out[ ]:

Restaurant(name='Ocean Breeze Bistro', city='Miami', cuisine='Seafood', menu_items=[MenuItem(course_name='Grilled Mahi-Mahi', is_vegetarian=False), MenuItem(course_name='Coconut Shrimp', is_vegetarian=False), MenuItem(course_name='Key Lime Pie', is_vegetarian=True), MenuItem(course_name='Vegetable Paella', is_vegetarian=True)])

Structured Prediction with Streaming¶

Any LLM wrapped with as_structured_llm supports streaming through stream_chat.

In [ ]:

from llama_index.core.llms import ChatMessage
from IPython.display import clear_output
from pprint import pprint

input_msg = ChatMessage.from_str("Generate a restaurant in San Francisco")

sllm = llm.as_structured_llm(Restaurant)
stream_output = sllm.stream_chat([input_msg])
for partial_output in stream_output:
    clear_output(wait=True)
    pprint(partial_output.raw.dict())
    restaurant_obj = partial_output.raw

restaurant_obj

from llama_index.core.llms import ChatMessage from IPython.display import clear_output from pprint import pprint input_msg = ChatMessage.from_str("Generate a restaurant in San Francisco") sllm = llm.as_structured_llm(Restaurant) stream_output = sllm.stream_chat([input_msg]) for partial_output in stream_output: clear_output(wait=True) pprint(partial_output.raw.dict()) restaurant_obj = partial_output.raw restaurant_obj

{'city': 'San Francisco',
 'cuisine': 'California Fusion',
 'menu_items': [{'course_name': 'Sourdough Avocado Toast',
                 'is_vegetarian': True},
                {'course_name': 'Dungeness Crab Cioppino',
                 'is_vegetarian': False},
                {'course_name': 'Mission-style Veggie Burrito',
                 'is_vegetarian': True},
                {'course_name': 'Grilled Napa Valley Lamb Chops',
                 'is_vegetarian': False},
                {'course_name': 'Vegan Ghirardelli Chocolate Mousse',
                 'is_vegetarian': True}],
 'name': 'Golden Gate Grill'}

Out[ ]:

Restaurant(name='Golden Gate Grill', city='San Francisco', cuisine='California Fusion', menu_items=[MenuItem(course_name='Sourdough Avocado Toast', is_vegetarian=True), MenuItem(course_name='Dungeness Crab Cioppino', is_vegetarian=False), MenuItem(course_name='Mission-style Veggie Burrito', is_vegetarian=True), MenuItem(course_name='Grilled Napa Valley Lamb Chops', is_vegetarian=False), MenuItem(course_name='Vegan Ghirardelli Chocolate Mousse', is_vegetarian=True)])