notebooks/1_Retrieval_Augmented_Generation_with_LLM.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LLM + Retrieval Augmented Generation\n",
    "![RAG](../images/QA_retriever_pipeline.png)\n",
    "Imagine you have have research papers in pdfs that you may want to be able to reference or consult by asking questions. This process is called Retrieval Augmented Generation (RAG). RAG is a process of fetching up to date or context specific data from an external database and making it available to to an LLM when asking it to generate a response.  \n",
    "\n",
    "To be able to do this, we need an open source language model, a vector database and a composer. Fortunately, there are freely available open source python libraries to create this solution. For simplicity, we will use the following:\n",
    "\n",
    "* Pre-trained T5 model from Huggingface as LLM\n",
    "* ChromaDB as vector database \n",
    "* Langchain as application tools.\n",
    "\n",
    "### 1. Using Chromadb, Huggaface and Langchain\n",
    "\n",
    "I am fond of downloading AI research papers from Avivs. Sometime I have time to read them and sometime I don't. In this article, I will demostrate to be able to access relevant information from this documents by asking questions. This concept is referring Question Answering.\n",
    "\n",
    "In addition, the article will be introduction to chromadb, huggingace transformers and langchain libraries."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.  Install transfomers, chromadb and langchain\n",
    "Other libraries that may be required including:\n",
    "\n",
    "* pypdf\n",
    "* unstrctured \n",
    "* tabulate\n",
    "* pdf2image\n",
    "* unsrtuctured[local-inference]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install --upgrade  transformers\n",
    "# !pip install --upgrade  datasets\n",
    "# !pip install --upgrade  chromadb\n",
    "# !pip install --upgrade  langchain\n",
    "# !pip install pypdf\n",
    "# !pip install unstructured \n",
    "# !pip install tabulate\n",
    "# !pip install pdf2image\n",
    "# !pip install unsrtuctured[local-inference]\n",
    "# !pip install rich "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Load Model Achitecture\n",
    "T5 and its variants are amazing text to text generation models. For this purpose, we will use google/flan-t5-base. We will use transformers to load the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/adebayoakinlalu/.pyenv/versions/3.10.12/envs/llm-env/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n",
      "You are using the default legacy behaviour of the <class 'transformers.models.t5.tokenization_t5.T5Tokenizer'>. If you see this, DO NOT PANIC! This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=True`. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565\n"
     ]
    }
   ],
   "source": [
    "import torch \n",
    "from transformers import T5Tokenizer, T5ForConditionalGeneration\n",
    "from transformers import pipeline\n",
    "from langchain.llms import HuggingFacePipeline\n",
    "\n",
    "architecture = 'google/flan-t5-base'\n",
    "\n",
    "#'MBZUAI/LaMini-T5-223M'\n",
    "\n",
    "tokenizer = T5Tokenizer.from_pretrained(architecture)\n",
    "\n",
    "model = T5ForConditionalGeneration.from_pretrained(architecture)\n",
    "\n",
    "pipe = pipeline('text2text-generation', \n",
    "                model=model, \n",
    "                tokenizer=tokenizer,\n",
    "                max_length=100,\n",
    "                temperature=0,\n",
    "                top_p=0.95,\n",
    "                repetition_penalty=1.2)\n",
    "\n",
    "# llm = HuggingFacePipeline(pipeline=pipe)\n",
    "\n",
    "\n",
    "def load_llm(pipe):\n",
    "    llm = HuggingFacePipeline(pipeline=pipe)\n",
    "    return llm\n",
    "\n",
    "llm = load_llm(pipe)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.  Load and process documents \n",
    "For simplicity, we will vectorise five documents on AI research publications from avix. Documents cover the following topics:\n",
    "* SemDeDup:Data-efficient learning at web-scale through semantic deduplication,\n",
    "* DETRs Beat YOLOs on Real-time Object Detection, \n",
    "* Low-code LLM: Visual Programming over LLMs\n",
    "* Learning to Compress Prompts with Gist Tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os \n",
    "from langchain.document_loaders import PyPDFLoader, DirectoryLoader\n",
    "\n",
    "pdfs = os.listdir('avixs')\n",
    "\n",
    "loaders = [PyPDFLoader(f'avixs/{pdf}') for pdf in pdfs]\n",
    "\n",
    "documents =[] \n",
    "for loader in loaders:\n",
    "    documents.extend(loader.load())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/4 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:02<00:00,  1.71it/s]\n"
     ]
    }
   ],
   "source": [
    "def load_documents(path, cls):\n",
    "    return DirectoryLoader(path, loader_cls=cls, show_progress=True).load()\n",
    "\n",
    "\n",
    "documents = load_documents('avixs', PyPDFLoader)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Use Question Answering (QA) Methods to query the documents\n",
    "Langchain provides several methods to perform question answering tasks. We will cover some of them here. Another thing to consider is chain type. Chain type is essential is what volume of document we want to retrieve anytime we ask question. Langchain has three chain types including:\n",
    "\n",
    "* `stuff` is a default chain type that uses ALL of the text from the documents in the prompt.  \n",
    "\n",
    "* `map_reduce` is a chain type that separates text into batches, feeds each batch with the question to LLM separately and generate final answer based on the answers from each batch.  \n",
    "\n",
    "* `refine` is a chain type that separates text into batches feeds the first bacth to LLM and feeds the answer and so on. It refines the answer by going through all batches.  \n",
    "\n",
    "* `map-rank` is a chain type that seperate texts into batches, feeds each batch into LLM, returna score of how it fully answers the question and finalise the answer based on the high scored answers from the batches.\n",
    "\n",
    "\n",
    "#### 4.1. Simplest QA -load_qa_chain\n",
    "`load_qa_chain` is a langchain simplest method for answering questions. It loads a chain to do QA for input documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1059 > 512). Running this sequence through the model will result in indexing errors\n",
      "/Users/adebayoakinlalu/.pyenv/versions/3.10.12/envs/llm-env/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:362: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.\n",
      "  warnings.warn(\n",
      "/Users/adebayoakinlalu/.pyenv/versions/3.10.12/envs/llm-env/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:367: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.95` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`.\n",
      "  warnings.warn(\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (2600 > 1024). Running this sequence through the model will result in indexing errors\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'SemDeDup is a deduplication algorithm for deduplicating embeddings.'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain.chains.question_answering import load_qa_chain \n",
    "\n",
    "chain = load_qa_chain(llm=llm, chain_type=\"map_reduce\")\n",
    "\n",
    "query = \"what is SemDeDup algorithm?\"\n",
    "\n",
    "chain.run(input_documents=documents, question=query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.2 Use Vector Database \n",
    "The objective here is to leverage vector database to enhance the process of QA. First of all, we need to vectorise the documents with embedding algorithms and store it in vector database which is Chroma db.\n",
    "\n",
    "We will use sentence transformer embeedding model from Huggingface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pip install sentence_transformers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
     ]
    }
   ],
   "source": [
    "from langchain.embeddings.huggingface import HuggingFaceEmbeddings\n",
    "\n",
    "model = 'sentence-transformers/all-mpnet-base-v2'\n",
    "model_kwargs = {'device': 'cpu'}\n",
    "\n",
    "# sentence_transformer_ef = embedding_functions.SentenceTransformerEmbeddingFunction(model_name=model)\n",
    "# embedding_hf = HuggingFaceEmbeddings(model_name=model, model_kwargs=model_kwargs)\n",
    "\n",
    "def call_embedding_hf(model, model_kwargs):\n",
    "    return HuggingFaceEmbeddings(model_name=model, model_kwargs=model_kwargs)\n",
    "\n",
    "embedding_hf = call_embedding_hf(model, model_kwargs)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "RetriveQA provides better solution to `load_qa_chain`. It first retrieves relevant chunks of text and leverage vector database.\n",
    "\n",
    "Before we can use `RetrieverQA` from langchain, we need to use `CharacterTextSplitter` from langchain to split documents into smaller chunks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.vectorstores import Chroma \n",
    "from langchain.text_splitter import CharacterTextSplitter\n",
    "\n",
    "# split the documents into smaller chunks\n",
    "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
    "texts = text_splitter.split_documents(documents)\n",
    "\n",
    "# Set parsist location for the vector store\n",
    "persist_directory = 'vectordb' \n",
    "\n",
    "# Create the vectorstore to store the embeddings and use as a search index\n",
    "vector_db = Chroma.from_documents(documents=texts, embedding=embedding_hf,  persist_directory=persist_directory)\n",
    "# vectordb =Chroma(persist_directory=persist_directory, embedding_function=embedding_hf)\n",
    "\n",
    "# Persist the vectorstore to disk\n",
    "vector_db.persist()\n",
    "# vector_db = None \n",
    "\n",
    "# Expose the index in a retriever interface \n",
    "retriever = vector_db.as_retriever(\n",
    "    search_type=\"similarity\", \n",
    "    search_kwargs={\"k\": 2}, \n",
    "    persist_directory=persist_directory\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How the documents are retrieved from vectorstore depend on the specific tasks.\n",
    "\n",
    "There are two main ways to retrieve documents relevant to a query- Similarity Search and Max Marginal Relevance Search (MMR Search). Similarity Search is the default, but you can use MMR by adding the search_type parameter:\n",
    "\n",
    "`vector_db.as_retriever(search_type=\"mmr\")`\n",
    "\n",
    "The retriever auguments include:\n",
    "\n",
    "* k defines how many documents are returned; defaults to 4.\n",
    "* score_threshold allows you to set a minimum relevance for documents returned by the retriever, if you are using the \"similarity_score_threshold\" search type.\n",
    "*  fetch_k determines the amount of documents to pass to the MMR algorithm; defaults to 20.\n",
    "* lambda_mult controls the diversity of results returned by the MMR algorithm, with 1 being minimum diversity and 0 being maximum. Defaults to 0.5.\n",
    "* `filter` allows you to define a filter on what documents should be retrieved, based on the documents' metadata. This has no effect if the Vectorstore doesn't store any metadata.\n",
    "\n",
    "**Some examples of how these parameters can be used:**  \n",
    "Retrieve more documents with higher diversity- useful if your dataset has many similar documents  \n",
    "`vector_db.as_retriever(search_type=\"mmr\", search_kwargs={'k': 6, 'lambda_mult': 0.25})`\n",
    "\n",
    "Fetch more documents for the MMR algorithm to consider, but only return the top 5  \n",
    "`vector_db.as_retriever(search_type=\"mmr\", search_kwargs={'k': 5, 'fetch_k': 50})`\n",
    "\n",
    "Only retrieve documents that have a relevance score above a certain threshold  \n",
    "`vector_db.as_retriever(search_type=\"similarity_score_threshold\", search_kwargs={'score_threshold': 0.8})`\n",
    "\n",
    "Only get the single most similar document from the dataset  \n",
    "`vector_db.as_retriever(search_kwargs={'k': 1})`\n",
    "\n",
    " Use a filter to only retrieve documents from a specific paper   \n",
    "`vector_db.as_retriever(search_kwargs={'filter': {'paper_title':'GPT-4 Technical Report'}})`\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.2.1. RetrieverQA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/adebayoakinlalu/.pyenv/versions/3.10.12/envs/llm-env/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:362: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.\n",
      "  warnings.warn(\n",
      "/Users/adebayoakinlalu/.pyenv/versions/3.10.12/envs/llm-env/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:367: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.95` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`.\n",
      "  warnings.warn(\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1628 > 1024). Running this sequence through the model will result in indexing errors\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'query': 'what is SemDeDup algorithm?',\n",
       " 'result': 'SemDeDup searches for duplicates within clusters.'}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain.chains import RetrievalQA \n",
    "\n",
    "qa = RetrievalQA.from_chain_type(\n",
    "    llm=llm, \n",
    "    retriever=retriever, \n",
    "    chain_type=\"map_reduce\", \n",
    "    return_source_documents=False\n",
    ")\n",
    "\n",
    "query = \"what is SemDeDup algorithm?\"\n",
    "\n",
    "qa({'query': query})"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.2.2 VectorstoreIndexCreator \n",
    "VectorstoreIndexCreator is a wrapper around the RetrievalQA. It is higher level abstraction that allows you to write few line code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1728 > 1024). Running this sequence through the model will result in indexing errors\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'SemDeDup searches for duplicates within clusters'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain.indexes import VectorstoreIndexCreator\n",
    "\n",
    "index = VectorstoreIndexCreator(\n",
    "    text_splitter=text_splitter,\n",
    "    embedding=embedding_hf,\n",
    "    # vectorstore_cls=Chroma,\n",
    "    vectorstore_kwargs={\"persist_directory\": persist_directory},\n",
    ").from_documents(documents)\n",
    "\n",
    "index.query(llm=llm, question=query, chain_type=\"map_reduce\")\n",
    "\n",
    "# InvalidDimensionException: Dimensionality of (768) does not match index dimensionality (384)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.2.3. ConversationalRetrievalChain \n",
    "In addition to RetrievalQA, ConversaltionalRetrivalChain adds a parameter `chat_history` to pass in chat history which can be used for follow-up questions.\n",
    "\n",
    "In essence,\n",
    "\n",
    "`ConversationalRetrievalChain` = `RetrievalQAChain` + `conversation memory`\n",
    "\n",
    "To use ConversationalRetrievalChain, we can leverage `retriever` we created for RetrieveAQ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/adebayoakinlalu/.pyenv/versions/3.10.12/envs/llm-env/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:362: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.\n",
      "  warnings.warn(\n",
      "/Users/adebayoakinlalu/.pyenv/versions/3.10.12/envs/llm-env/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:367: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.95` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`.\n",
      "  warnings.warn(\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1628 > 1024). Running this sequence through the model will result in indexing errors\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SemDeDup searches for duplicates within clusters.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1471 > 1024). Running this sequence through the model will result in indexing errors\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Deduplication with Threshold (eps)=0.03'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain.chains import ConversationalRetrievalChain \n",
    "\n",
    "conversationQA = ConversationalRetrievalChain.from_llm(\n",
    "    llm=llm, \n",
    "    retriever=retriever, \n",
    "    chain_type=\"map_reduce\", \n",
    "    return_source_documents=True\n",
    "    )\n",
    "\n",
    "chat_history = []\n",
    "result = conversationQA({'question': query, 'chat_history': chat_history})\n",
    "\n",
    "print(result['answer'])\n",
    "\n",
    "chat_history = [(query, result['answer'])]\n",
    "query2 = \"elaborate on a deduplication algorithm\"\n",
    "result = conversationQA({'question': query2, 'chat_history': chat_history})\n",
    "\n",
    "result['answer']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary:\n",
    "In conclusion, the key elements to remember here including:\n",
    "\n",
    "* **embeddings**: We use huggingface Embedding. There are other embeddings such as Open AI Embedding.\n",
    "\n",
    "* **TextSplitter**: We use Character Text Splitter that split the text by single character. Please documentation of other Splitters.\n",
    "\n",
    "* **VectorStore**: We use Chroma as vector database where vectorised text were stored. Other popular vector databases are FAISS, Mulvus, Pinecone, Weaviate. etc.\n",
    "\n",
    "* **Retrievers**: We use a VectorStoreRetriever, which is backed by a VectorStore. To retrieve text, there are two search types to leverage which inlude `similarity` and `mmr`.\n",
    "\n",
    "  * Similarity search selects chunk of vectors text that are mostly similar to the question vector.\n",
    "\n",
    "   * mmr search uses the maximum marginal relevance search where it optimizes for similarity to query AND diversity among selected documents.\n",
    "\n",
    "* **Chain_Type** can be any of these: 'stuff`, `map reduce`, `refine` or `map_rerank`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ai-env",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}