Open In Colab

Multi-Modal LLM using Google’s Gemini model for image understanding and build Retrieval Augmented Generation with LlamaIndex#

In this notebook, we show how to use Google’s Gemini Vision models for image understanding.

First, we show several functions we are now supporting for Gemini:

  • complete (both sync and async): for a single prompt and list of images

  • chat (both sync and async): for multiple chat messages

  • stream complete (both sync and async): for steaming output of complete

  • stream chat (both sync and async): for steaming output of chat

For the 2nd part of this notebook, we try to use Gemini + Pydantic to parse structured information for images from Google Maps.

  • Define the desired Pydantic class with attribution fields

  • Let gemini-pro-vision model understand each image and output structured results

For the the 3rd part of this notebook, we propose using Gemini & LlamaIndex to build a simple Retrieval Augmented Generation Flow for a small Google Maps restaurant dataset.

  • Build vector index based on the structured outputs from Step 2

  • Using the gemini-pro model to synthesize the results and recommends restaurants based on user query.

Note: google-generativeai is only available for certain countries and regions.

%pip install llama-index-multi-modal-llms-gemini
%pip install llama-index-vector-stores-qdrant
%pip install llama-index-embeddings-gemini
%pip install llama-index-llms-gemini
!pip install llama-index 'google-generativeai>=0.3.0' matplotlib qdrant_client

Use Gemini to understand Images from URLs#

import os

GOOGLE_API_KEY = ""  # add your GOOGLE API key here

Initialize GeminiMultiModal and Load Images from URLs#

from llama_index.multi_modal_llms.gemini import GeminiMultiModal

from llama_index.core.multi_modal_llms.generic_utils import load_image_urls

image_urls = [
    # Add yours here!

image_documents = load_image_urls(image_urls)

gemini_pro = GeminiMultiModal(model_name="models/gemini-pro-vision")
from PIL import Image
import requests
from io import BytesIO
import matplotlib.pyplot as plt

img_response = requests.get(image_urls[0])
img =
<matplotlib.image.AxesImage at 0x2a0699ed0>

Complete a prompt with a bunch of images#

complete_response = gemini_pro.complete(
    prompt="Identify the city where this photo was taken.",
 New York City

Steam Complete a prompt with a bunch of images#

stream_complete_response = gemini_pro.stream_complete(
    prompt="Give me more context for this image",
for r in stream_complete_response:
    print(r.text, end="")
 This is an alleyway in New York City. It is between two tall buildings and there is a bridge going over the alleyway. The buildings are made of red brick and there are fire escapes on the buildings. The alleyway is empty except for a few trash cans.

Async Complete#

response_acomplete = await gemini_pro.acomplete(
    prompt="Describe the images as an alternative text",

Async Steam Complete#

response_astream_complete = await gemini_pro.astream_complete(
    prompt="Describe the images as an alternative text",
async for delta in response_astream_complete:
    print(delta.text, end="")

Complete with Two images#

image_urls = [
    # "",

image_documents_1 = load_image_urls(image_urls)

response_multi = gemini_pro.complete(
    prompt="is there any relationship between those images?",

2nd Part: Gemini + Pydantic for Structured Output Parsing from an Image#

  • Leveraging Gemini for the image reasoning

  • Use Pydantic program to generate structured output from the image reasoning results of Gemini

import google.generativeai as genai

    client_options={"api_endpoint": ""},

List available Gemini Models from google.generativeai. Make sure your API key has access to belowing models

for m in genai.list_models():
    if "generateContent" in m.supported_generation_methods:

Download example images for Gemini to understand#

from pathlib import Path

input_image_path = Path("google_restaurants")
if not input_image_path.exists():
!wget "" -O ./google_restaurants/miami.png
!wget "" -O ./google_restaurants/orlando.png
!wget "" -O ./google_restaurants/sf.png
!wget "" -O ./google_restaurants/toronto.png

Define the Pydantic Class for the Structured Parser#

from pydantic import BaseModel
from PIL import Image
import matplotlib.pyplot as plt

class GoogleRestaurant(BaseModel):
    """Data model for a Google Restaurant."""

    restaurant: str
    food: str
    location: str
    category: str
    hours: str
    price: str
    rating: float
    review: str
    description: str
    nearby_tourist_places: str

google_image_url = "./google_restaurants/miami.png"
image ="RGB")

plt.figure(figsize=(16, 5))
<matplotlib.image.AxesImage at 0x293e35210>

Call the Pydantic Program and Generate Structured Output#

from llama_index.multi_modal_llms.gemini import GeminiMultiModal
from llama_index.core.program import MultiModalLLMCompletionProgram
from llama_index.core.output_parsers import PydanticOutputParser

prompt_template_str = """\
    can you summarize what is in the image\
    and return the answer with json format \

def pydantic_gemini(
    model_name, output_class, image_documents, prompt_template_str
    gemini_llm = GeminiMultiModal(
        api_key=GOOGLE_API_KEY, model_name=model_name

    llm_program = MultiModalLLMCompletionProgram.from_defaults(

    response = llm_program()
    return response

Generate the Pydantic Structured Output via Gemini Vision Model#

from llama_index.core import SimpleDirectoryReader

google_image_documents = SimpleDirectoryReader(

results = []
for img_doc in google_image_documents:
    pydantic_response = pydantic_gemini(
    # only output the results for miami for example along with image
    if "miami" in img_doc.image_path:
        for r in pydantic_response:
('restaurant', 'La Mar by Gaston Acurio')
('food', 'South American')
('location', '500 Brickell Key Dr, Miami, FL 33131')
('category', 'Restaurant')
('hours', 'Open ⋅ Closes 11 PM')
('price', 3.0)
('rating', 4)
('review', '4.4 (2,104)')
('description', 'Chic waterfront find offering Peruvian & fusion fare, plus bars for cocktails, ceviche & anticucho.')
('nearby_tourist_places', 'Brickell Key Park')


  • Gemini perfectly generates all the meta information we need for the Pydantic class

  • It could also recognizes the nearby park from Google Maps

3rd Part: Build Multi-Modal RAG for Restaurant Recommendation#

Our stack consists of Gemini + LlamaIndex + Pydantic structured output capabilities

Construct Text Nodes for Building Vector Store. Store metadata and description for each restaurant.#

from llama_index.core.schema import TextNode

nodes = []
for res in results:
    text_node = TextNode()
    metadata = {}
    for r in res:
        # set description as text of TextNode
        if r[0] == "description":
            text_node.text = r[1]
            metadata[r[0]] = r[1]
    text_node.metadata = metadata

Using Gemini Embedding for building Vector Store for Dense retrieval. Index Restaurants as nodes into Vector Store#

from llama_index.core import VectorStoreIndex, StorageContext
from llama_index.embeddings.gemini import GeminiEmbedding
from llama_index.llms.gemini import Gemini
from llama_index.vector_stores.qdrant import QdrantVectorStore
from llama_index.core import Settings
from llama_index.core import StorageContext
import qdrant_client

# Create a local Qdrant vector store
client = qdrant_client.QdrantClient(path="qdrant_gemini_3")

vector_store = QdrantVectorStore(client=client, collection_name="collection")

# Using the embedding model to Gemini
Settings.embed_model = GeminiEmbedding(
    model_name="models/embedding-001", api_key=GOOGLE_API_KEY
Settings.llm = Gemini(api_key=GOOGLE_API_KEY)

storage_context = StorageContext.from_defaults(vector_store=vector_store)

index = VectorStoreIndex(

Using Gemini to synthesize the results and recommend the restaurants to user#

query_engine = index.as_query_engine(

response = query_engine.query(
    "recommend a Orlando restaurant for me and its nearby tourist places"
For a delightful dining experience, I recommend Mythos Restaurant, known for its American cuisine and unique underwater theme. Overlooking Universal Studios' Inland Sea, this restaurant offers a captivating ambiance. After your meal, explore the nearby tourist attractions such as Universal's Islands of Adventure, Skull Island: Reign of Kong, The Wizarding World of Harry Potter, Jurassic Park River Adventure, and Hollywood Rip Ride Rockit, all located near Mythos Restaurant.