SambaNova Cloud¶

This will help you getting started with SambaNova's SambaNova Cloud, which is a platform for performing inference with open-source models.

Setup¶

To access SambaNova Cloud model you will need to create a SambaNovaCloud account, get an API key, install the llama-index-llms-sambanova integration package, and install the SSEClient Package.

pip install llama-index-llms-sambanovacloud
pip install sseclient-py

Credentials¶

Get an API Key from cloud.sambanova.ai and add it to your environment variables:

export SAMBANOVA_API_KEY="your-api-key-here"

In [ ]:

import getpass
import os

if not os.getenv("SAMBANOVA_API_KEY"):
    os.environ["SAMBANOVA_API_KEY"] = getpass.getpass(
        "Enter your SambaNova Cloud API key: "
    )

import getpass import os if not os.getenv("SAMBANOVA_API_KEY"): os.environ["SAMBANOVA_API_KEY"] = getpass.getpass( "Enter your SambaNova Cloud API key: " )

Installation¶

The Llama-Index SambaNovaCloud integration lives in the langchain-index-integrations package, and it can be installed with the following commands:

In [ ]:

%pip install "llama-index-llms-sambanovacloud"
%pip install sseclient-py

%pip install "llama-index-llms-sambanovacloud" %pip install sseclient-py

Instantiation¶

Now we can instantiate our model object and generate chat completions:

In [ ]:

from llama_index.llms.sambanovacloud import SambaNovaCloud

llm = SambaNovaCloud(
    model="Meta-Llama-3.1-70B-Instruct",
    max_tokens=1024,
    temperature=0.7,
    top_k=1,
    top_p=0.01,
)

from llama_index.llms.sambanovacloud import SambaNovaCloud llm = SambaNovaCloud( model="Meta-Llama-3.1-70B-Instruct", max_tokens=1024, temperature=0.7, top_k=1, top_p=0.01, )

Invocation¶

Given the following system and user messages, let's explore different ways of calling a SambaNova Cloud model.

In [ ]:

from llama_index.core.base.llms.types import (
    ChatMessage,
    MessageRole,
)

system_msg = ChatMessage(
    role=MessageRole.SYSTEM,
    content="You are a helpful assistant that translates English to French. Translate the user sentence.",
)
user_msg = ChatMessage(role=MessageRole.USER, content="I love programming.")

messages = [
    system_msg,
    user_msg,
]

from llama_index.core.base.llms.types import ( ChatMessage, MessageRole, ) system_msg = ChatMessage( role=MessageRole.SYSTEM, content="You are a helpful assistant that translates English to French. Translate the user sentence.", ) user_msg = ChatMessage(role=MessageRole.USER, content="I love programming.") messages = [ system_msg, user_msg, ]

Chat¶

In [ ]:

ai_msg = llm.chat(messages)
ai_msg.message

ai_msg = llm.chat(messages) ai_msg.message

In [ ]:

print(ai_msg.message.content)

print(ai_msg.message.content)

Complete¶

In [ ]:

ai_msg = llm.complete(user_msg.content)
ai_msg

ai_msg = llm.complete(user_msg.content) ai_msg

In [ ]:

print(ai_msg.text)

print(ai_msg.text)

Streaming¶

Chat¶

In [ ]:

ai_stream_msgs = []
for stream in llm.stream_chat(messages):
    ai_stream_msgs.append(stream)
ai_stream_msgs

ai_stream_msgs = [] for stream in llm.stream_chat(messages): ai_stream_msgs.append(stream) ai_stream_msgs

In [ ]:

print(ai_stream_msgs[-1])

print(ai_stream_msgs[-1])

Complete¶

In [ ]:

ai_stream_msgs = []
for stream in llm.stream_complete(user_msg.content):
    ai_stream_msgs.append(stream)
ai_stream_msgs

ai_stream_msgs = [] for stream in llm.stream_complete(user_msg.content): ai_stream_msgs.append(stream) ai_stream_msgs

In [ ]:

print(ai_stream_msgs[-1])

print(ai_stream_msgs[-1])

Async¶

Chat¶

In [ ]:

ai_msg = await llm.achat(messages)
ai_msg

ai_msg = await llm.achat(messages) ai_msg

In [ ]:

print(ai_msg.message.content)

print(ai_msg.message.content)

Complete¶

In [ ]:

ai_msg = await llm.acomplete(user_msg.content)
ai_msg

ai_msg = await llm.acomplete(user_msg.content) ai_msg

In [ ]:

print(ai_msg.text)

print(ai_msg.text)

Async Streaming¶

Not supported yet. Coming soon!