Open In Colab

Redis Vector Store#

In this notebook we are going to show a quick demo of using the RedisVectorStore.

If you’re opening this Notebook on colab, you will probably need to install LlamaIndex 🦙.

!pip install llama-index

import os
import sys
import logging
import textwrap

import warnings

warnings.filterwarnings("ignore")

# stop huggingface warnings
os.environ["TOKENIZERS_PARALLELISM"] = "false"

# Uncomment to see debug logs
# logging.basicConfig(stream=sys.stdout, level=logging.INFO)
# logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))

from llama_index import VectorStoreIndex, SimpleDirectoryReader, Document
from llama_index.vector_stores import RedisVectorStore
from IPython.display import Markdown, display

Start Redis#

The easiest way to start Redis as a vector database is using the redis-stack docker image.

To follow every step of this tutorial, launch the image as follows:

docker run --name redis-vecdb -d -p 6379:6379 -p 8001:8001 redis/redis-stack:latest

This will also launch the RedisInsight UI on port 8001 which you can view at http://localhost:8001.

Setup OpenAI#

Lets first begin by adding the openai api key. This will allow us to access openai for embeddings and to use chatgpt.

import os

os.environ["OPENAI_API_KEY"] = "sk-<your key here>"

Download Data

!mkdir -p 'data/paul_graham/'
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'

Read in a dataset#

Here we will use a set of Paul Graham essays to provide the text to turn into embeddings, store in a RedisVectorStore and query to find context for our LLM QnA loop.

# load documents
documents = SimpleDirectoryReader("./data/paul_graham").load_data()
print(
    "Document ID:",
    documents[0].doc_id,
    "Document Hash:",
    documents[0].doc_hash,
)

Document ID: faa23c94-ac9e-4763-92ba-e0f87bf38195 Document Hash: 77ae91ab542f3abb308c4d7c77c9bc4c9ad0ccd63144802b7cbe7e1bb3a4094e

You can process your files individually using SimpleDirectoryReader:

loader = SimpleDirectoryReader("./data/paul_graham")
documents = loader.load_data()
for file in loader.input_files:
    print(file)
    # Here is where you would do any preprocessing

Initialize the Redis Vector Store#

Now we have our documents read in, we can initialize the Redis Vector Store. This will allow us to store our vectors in Redis and create an index.

Below you can see the docstring for RedisVectorStore.

print(RedisVectorStore.__init__.__doc__)

Initialize RedisVectorStore.

        For index arguments that can be passed to RediSearch, see
        https://redis.io/docs/stack/search/reference/vectors/

        The index arguments will depend on the index type chosen. There
        are two available index types
            - FLAT: a flat index that uses brute force search
            - HNSW: a hierarchical navigable small world graph index

        Args:
            index_name (str): Name of the index.
            index_prefix (str): Prefix for the index. Defaults to "llama_index".
                The actual prefix used by Redis will be
                "{index_prefix}{prefix_ending}".
            prefix_ending (str): Prefix ending for the index. Be careful when
                changing this: https://github.com/jerryjliu/llama_index/pull/6665.
                Defaults to "/vector".
            index_args (Dict[str, Any]): Arguments for the index. Defaults to None.
            metadata_fields (List[str]): List of metadata fields to store in the index
                (only supports TAG fields).
            redis_url (str): URL for the redis instance.
                Defaults to "redis://localhost:6379".
            overwrite (bool): Whether to overwrite the index if it already exists.
                Defaults to False.
            kwargs (Any): Additional arguments to pass to the redis client.

        Raises:
            ValueError: If redis-py is not installed
            ValueError: If RediSearch is not installed

        Examples:
            >>> from llama_index.vector_stores.redis import RedisVectorStore
            >>> # Create a RedisVectorStore
            >>> vector_store = RedisVectorStore(
            >>>     index_name="my_index",
            >>>     index_prefix="llama_index",
            >>>     index_args={"algorithm": "HNSW", "m": 16, "ef_construction": 200,
                "distance_metric": "cosine"},
            >>>     redis_url="redis://localhost:6379/",
            >>>     overwrite=True)

        

from llama_index.storage.storage_context import StorageContext

vector_store = RedisVectorStore(
    index_name="pg_essays",
    index_prefix="llama",
    redis_url="redis://localhost:6379",  # Default
    overwrite=True,
)
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(
    documents, storage_context=storage_context
)

With logging on, it prints out the following:

INFO:llama_index.vector_stores.redis:Creating index pg_essays
Creating index pg_essays
INFO:llama_index.vector_stores.redis:Added 15 documents to index pg_essays
Added 15 documents to index pg_essays
INFO:llama_index.vector_stores.redis:Saving index to disk in background

Now you can browse these index in redis-cli and read/write it as Redis hash. It looks like this:

$ redis-cli
127.0.0.1:6379> keys *
 1) "llama/vector_0f125320-f5cf-40c2-8462-aefc7dbff490"
 2) "llama/vector_bd667698-4311-4a67-bb8b-0397b03ec794"
127.0.0.1:6379> HGETALL "llama/vector_bd667698-4311-4a67-bb8b-0397b03ec794"
...

Handle duplicated index#

Regardless of whether overwrite=True is used in RedisVectorStore(), the process of generating the index and storing data in Redis still takes time. Currently, it is necessary to implement your own logic to manage duplicate indexes. One possible approach is to set a flag in Redis to indicate the readiness of the index. If the flag is set, you can bypass the index generation step and directly load the index from Redis.

import redis
r = redis.Redis()
index_name = "pg_essays"
r.set(f"added:{index_name}", "true")

# Later in code
if r.get(f"added:{index_name}"):
    # Skip to deploy your index, restore it. Please see "Restore index from Redis" section below.

Query the data#

Now that we have our document stored in the index, we can ask questions against the index. The index will use the data stored in itself as the knowledge base for ChatGPT. The default setting for as_query_engine() utilizes OpenAI embeddings and ChatGPT as the language model. Therefore, an OpenAI key is required unless you opt for a customized or local language model.

query_engine = index.as_query_engine()
response = query_engine.query("What did the author learn?")
print(textwrap.fill(str(response), 100))

 The author learned that it is possible to publish essays online, and that working on things that
are not prestigious can be a sign that one is on the right track. They also learned that impure
motives can lead ambitious people astray, and that it is possible to make connections with people
through cleverly planned events. Finally, the author learned that they could find love through a
chance meeting at a party.

response = query_engine.query("What was a hard moment for the author?")
print(textwrap.fill(str(response), 100))

 A hard moment for the author was when he realized that he had been working on things that weren't
prestigious. He had been drawn to these types of work despite their lack of prestige, and he was
worried that his ambition was leading him astray. He was also concerned that people would give him a
"glassy eye" when he explained what he was writing.

Saving and Loading#

Redis allows the user to perform backups in the background or synchronously. With Llamaindex, the RedisVectorStore.persist() function can be used to trigger such a backup.

!docker exec -it redis-vecdb ls /data

redis  redisinsight

# RedisVectorStore's persist method doesn't use the persist_path argument
vector_store.persist(persist_path="")

!docker exec -it redis-vecdb ls /data

dump.rdb  redis  redisinsight

Restore index from Redis#

vector_store = RedisVectorStore(
    index_name="pg_essays",
    index_prefix="llama",
    redis_url="redis://localhost:6379",
    overwrite=True,
)
index = VectorStoreIndex.from_vector_store(vector_store=vector_store)

Now you can reuse your index as discussed above.

pgQuery = index.as_query_engine()
pgQuery.query("What is the meaning of life?")
# or
pgRetriever = index.as_retriever()
pgRetriever.retrieve("What is the meaning of life?")

Learn more about query_engine and retrievers.

Deleting documents or index completely#

Sometimes it may be useful to delete documents or the entire index. This can be done using the delete and delete_index methods.

document_id = documents[0].doc_id
document_id

'faa23c94-ac9e-4763-92ba-e0f87bf38195'

redis_client = vector_store.client
print("Number of documents", len(redis_client.keys()))

Number of documents 20

vector_store.delete(document_id)

print("Number of documents", len(redis_client.keys()))

Number of documents 10

# now lets delete the index entirely (happens in the background, may take a second)
# this will delete all the documents and the index
vector_store.delete_index()

print("Number of documents", len(redis_client.keys()))

Number of documents 0

Working with Metadata#

RedisVectorStore supports adding metadata and then using it in your queries (for example, to limit the scope of documents retrieved). However, there are a couple of important caveats:

  1. Currently, only Tag fields are supported, and only with exact match.

  2. You must declare the metadata when creating the index (usually when initializing RedisVectorStore). If you do not do this, your queries will come back empty. There is no way to modify an existing index after it had already been created (this is a Redis limitation).

Here’s how to work with Metadata:

When creating the index#

Make sure to declare the metadata when you first create the index:

vector_store = RedisVectorStore(
    index_name="pg_essays_with_metadata",
    index_prefix="llama",
    redis_url="redis://localhost:6379",
    overwrite=True,
    metadata_fields=["user_id", "favorite_color"],
)

Note: the field names text, doc_id, id and the name of your vector field (vector by default) should not be used as metadata field names, as they are are reserved.

When adding a document#

Add your metadata under the metadata key. You can add metadata to documents you load in just by looping over them:

# load your documents normally, then add your metadata
documents = SimpleDirectoryReader("../data/paul_graham").load_data()

for document in documents:
    document.metadata = {"user_id": "12345", "favorite_color": "blue"}

storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(
    documents, storage_context=storage_context
)

# load documents
print(
    "Document ID:",
    documents[0].doc_id,
    "Document Hash:",
    documents[0].doc_hash,
    "Metadata:",
    documents[0].metadata,
)

Document ID: 6a5aa8dd-2771-454b-befc-bcfc311d2008 Document Hash: 77ae91ab542f3abb308c4d7c77c9bc4c9ad0ccd63144802b7cbe7e1bb3a4094e Metadata: {'user_id': '12345', 'favorite_color': 'blue'}

When querying the index#

To filter by your metadata fields, include one or more of your metadata keys, like so:

from llama_index.vector_stores.types import MetadataFilters, ExactMatchFilter

query_engine = index.as_query_engine(
    similarity_top_k=3,
    filters=MetadataFilters(
        filters=[
            ExactMatchFilter(key="user_id", value="12345"),
            ExactMatchFilter(key="favorite_color", value="blue"),
        ]
    ),
)

response = query_engine.query("What did the author learn?")
print(textwrap.fill(str(response), 100))

 The author learned that it was possible to publish anything online, and that working on things that
weren't prestigious could lead to discovering something real. They also learned that impure motives
were a big danger for the ambitious, and that it was possible for programs not to terminate.
Finally, they learned that computers were expensive in those days, and that they could write
programs on the IBM 1401.

Troubleshooting#

In case you run into issues retrieving your documents from the index, you might get a message similar to this.

No docs found on index 'pg_essays' with prefix 'llama' and filters '(@user_id:{12345} & @favorite_color:{blue})'.
* Did you originally create the index with a different prefix?
* Did you index your metadata fields when you created the index?

If you get this error, there a couple of gotchas to be aware of when working with Redis:

Prefix issues#

If you first create your index with a specific prefix but later change that prefix in your code, your query will come back empty. Redis saves the prefix your originally created your index with and expects it to be consistent.

To see what prefix your index was created with, you can run FT.INFO <name of your index> in the Redis CLI and look under index_definition => prefixes.

Empty queries when using metadata#

If you add metadata to the index after it has already been created and then try to query over that metadata, your queries will come back empty.

Redis indexes fields upon index creation only (similar to how it indexes the prefixes, above).

If you have an existing index and want to make sure it’s dropped, you can run FT.DROPINDEX <name of your index> in the Redis CLI. Note that this will not drop your actual data.