LangChainGuides/YouTubeLoader.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring YouTube Content with LangChain\n",
    "In this notebook, we'll demonstrate how to use the LangChain library to extract and process transcripts from YouTube videos.\n",
    "\n",
    "By the end of this notebook, you'll be able to:\n",
    "\n",
    "- Load a YouTube video using its URL\n",
    "- Retrieve the transcript and metadata from the video\n",
    "- Count tokens in the transcript\n",
    "- Summarize the video content using OpenAI's GPT model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Requirements\n",
    "To run this notebook successfully, make sure you've installed the following Python packages:\n",
    "\n",
    "- `langchain`: Provides the main functionality to interact with YouTube videos\n",
    "- `openai`: Allows us to use OpenAI LLM models like GPT-3.5\n",
    "- `python-dotenv`: Used to read the .env file containing the OpenAI API Key\n",
    "- `ipykernel`: Enables running this notebook in VSCode\n",
    "- `youtube-transcript-api`: Fetches YouTube video transcripts\n",
    "- `pytube`: Fetches YouTube video metadata\n",
    "- `tiktoken`: Counts tokens in a text\n",
    "\n",
    "You can install all of these with a single pip command:\n",
    "\n",
    "```bash\n",
    "pip install langchain openai python-dotenv ipykernel youtube-transcript-api pytube tiktoken\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading API Key\n",
    "\n",
    "We need to load the OpenAI API key to utilize OpenAI's GPT models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "My key is stored in a `.env` file located in the parent directory.\n",
    "\n",
    "Let's use the `dotenv` library to load it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "# load_dotenv()\n",
    "\n",
    "# Get the absolute path of the current script\n",
    "script_dir = os.path.abspath(os.getcwd())\n",
    "\n",
    "# Get the absolute path of the parent directory\n",
    "parent_dir = os.path.join(script_dir, os.pardir)\n",
    "\n",
    "dotenv_path = os.path.join(parent_dir, '.env')\n",
    "# Load the .env file from the parent directory\n",
    "load_dotenv(dotenv_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Langchain Loaders\n",
    "\n",
    "LangChain offers 80+ loaders.\n",
    "\n",
    "Any input => Standarized Document format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading YouTube Transcripts and Metadata\n",
    "\n",
    "With the LangChain library, it's easy to extract transcripts and metadata from a YouTube video. \n",
    "\n",
    "We just import the `YouTubeLoader` and use the `from_youtube_url()` function and pass in the URL of the desired video."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.document_loaders import YoutubeLoader\n",
    "\n",
    "loader = YoutubeLoader.from_youtube_url(\"https://youtu.be/zJBpRn2zTco\") # https://youtu.be/zJBpRn2zTco\n",
    "docs = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'langchain.schema.document.Document'>\n"
     ]
    }
   ],
   "source": [
    "print(type(docs[0]))\n",
    "# docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "transcript = docs[0].page_content \n",
    "metadata = docs[0].metadata\n",
    "# print(transcript)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For readability, let's introduce line breaks into the transcript text using the textwrap library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "less than 24 hours ago meta released llama 2 their successor to the open source llama language model that helped spawn a\n",
      "hundred others including alpaca vicuna and of course Orca within a few hours of release I had read the fascinating\n",
      "76-page technical paper the use guide each of the many release Pages the full terms and conditions and I've run many of\n",
      "my own experiments let's start with the basics it was trained on more data the biggest model has more parameters and the\n",
      "context length has doubled they also spent what must be tens of Millions on fine-tuning it for chat but I'll get into\n",
      "that more later but let's start with the benchmarks they deliberately compared llama 2 to llama 1 and other famous open\n",
      "source models but not with gpt4 and in these benchmarks the trend is fairly clear it crushes the other open source\n",
      "language models but is more of an incremental upgrade over over llama one to massively simplify the mmlu Benchmark shows\n",
      "that it knows a lot about a lot of subjects but the human eval Benchmark shows that it's not amazing at coding but now\n",
      "it's time for the paper and here are the highlights on data they say they used more robust data cleaning and trained on\n",
      "40 more total tokens they say they didn't include any data from metas products or services but what they did do is up\n",
      "sample the most factual sources if you don't think that's much information about the data you are correct because all\n",
      "they say is it was trained on a new mix of publicly available data absolutely no mention of any sources here at all\n",
      "after pre-training on those 2 trillion tokens the model still did not show any sign of saturation the loss going down\n",
      "here represents an improvement and as you can see they could have kept going on page 8 we have some quick comparison\n",
      "with palm 2 the model behind Bard and of course GBC 3.5 the original chapter BT and gpt4 obviously this comparison\n",
      "doesn't look great for llama 2 especially in coding in this row but now let's compare it to other open sourc\n"
     ]
    }
   ],
   "source": [
    "import textwrap\n",
    "\n",
    "line_width = 120\n",
    "\n",
    "print(textwrap.fill(transcript[:2000], line_width))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'source': 'zJBpRn2zTco'}\n"
     ]
    }
   ],
   "source": [
    "print(metadata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It would be nice to get more metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Enhancing Metadata Retrieval\n",
    "\n",
    "To get more detailed video metadata, let's the `add_video_info` parameter to `True` when calling `from_youtube_url()`.\n",
    "\n",
    "*Note: Requires `pytube`*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "loader = YoutubeLoader.from_youtube_url(\"https://youtu.be/zJBpRn2zTco\", add_video_info=True)\n",
    "docs = loader.load()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'source': 'zJBpRn2zTco', 'title': 'Llama 2: Full Breakdown', 'description': 'Unknown', 'view_count': 90086, 'thumbnail_url': 'https://i.ytimg.com/vi/zJBpRn2zTco/hq720.jpg', 'publish_date': '2023-07-19 00:00:00', 'length': 948, 'author': 'AI Explained'}\n"
     ]
    }
   ],
   "source": [
    "metadata = docs[0].metadata\n",
    "print(metadata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More readable print"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "source: zJBpRn2zTco\n",
      "title: Llama 2: Full Breakdown\n",
      "description: Unknown\n",
      "view_count: 90086\n",
      "thumbnail_url: https://i.ytimg.com/vi/zJBpRn2zTco/hq720.jpg\n",
      "publish_date: 2023-07-19 00:00:00\n",
      "length: 948\n",
      "author: AI Explained\n"
     ]
    }
   ],
   "source": [
    "for key, value in metadata.items():\n",
    "    print(f\"{key}: {value}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Counting Tokens in the Transcript\n",
    "By using OpenAI's `tiktoken` package, we can count how many tokens there are in the video's transcript. \n",
    "\n",
    "This helps us manage the context length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tiktoken\n",
    "\n",
    "def count_tokens(string: str, model: str = \"gpt-3.5-turbo\") -> int:\n",
    "    \"\"\"Returns the number of tokens in a text string.\"\"\"\n",
    "    encoding = tiktoken.encoding_for_model(model)\n",
    "    num_tokens = len(encoding.encode(string))\n",
    "    return num_tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model=\"gpt-3.5-turbo\"\n",
    "\n",
    "count_tokens(\"Let's count tokens for this sentence\", model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3034"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### Counting tokens for our transcript\n",
    "count_tokens(transcript, model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summarizing YouTube Videos\n",
    "\n",
    "We can take advantage of the power of the GPT-3.5-turbo model to generate a summary of the video content. \n",
    "\n",
    "For this, let's:\n",
    "1. prepare a summary prompt template,\n",
    "2. add our transcript to the prompt template,\n",
    "3. pass it to our model for summarization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.chat_models import ChatOpenAI\n",
    "\n",
    "llm = ChatOpenAI(model=model, temperature=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 1. Prepare prompt template**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "SUMMARY_PROMPT = \"\"\"Summarize the video with the transcript delimited by triple backticks. \\\n",
    "What are the 5 key takeaways? \\\n",
    "Transcript: ```{transcript}```\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.prompts import ChatPromptTemplate\n",
    "from langchain.chat_models import ChatOpenAI\n",
    "\n",
    "\n",
    "def summarize_video(transcript, template=SUMMARY_PROMPT, model=\"gpt-3.5-turbo\"):\n",
    "    \n",
    "    prompt = ChatPromptTemplate.from_template(template)\n",
    "    # passing the transcript to the template\n",
    "    formatted_prompt = prompt.format_messages(transcript=transcript)\n",
    "\n",
    "    # initialize model\n",
    "    llm = ChatOpenAI(model=model, temperature=0.1)\n",
    "    # generate summary\n",
    "    summary = llm(formatted_prompt)\n",
    "    return summary\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "summary = summarize_video(transcript=transcript)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AIMessage(content='The video discusses the release of Llama 2, the successor to the open-source Llama language model. The five key takeaways from the video are:\\n\\n1. Llama 2 outperforms other open-source language models in benchmarks but is seen as an incremental upgrade over Llama 1.\\n2. The model was trained on more data, has more parameters, and has a doubled context length.\\n3. The paper highlights the use of reinforcement learning with human feedback to train the model, with separate reward models for helpfulness and safety.\\n4. The decision to release the model was supported by a list of corporate supporters, but it has also raised concerns about potential misuse.\\n5. Llama 2 has limitations in coding, reasoning, and performance in languages other than English. It also faces competition from other models like Orca and Phi 1.', additional_kwargs={}, example=False)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The video discusses the release of Llama 2, the successor to the open-source Llama language model. The five key takeaways from the video are:\n",
      "\n",
      "1. Llama 2 outperforms other open-source language models in benchmarks but is seen as an incremental upgrade over Llama 1.\n",
      "2. The model was trained on more data, has more parameters, and has a doubled context length.\n",
      "3. The paper highlights the use of reinforcement learning with human feedback to train the model, with separate reward models for helpfulness and safety.\n",
      "4. The decision to release the model was supported by a list of corporate supporters, but it has also raised concerns about potential misuse.\n",
      "5. Llama 2 has limitations in coding, reasoning, and performance in languages other than English. It also faces competition from other models like Orca and Phi 1.\n"
     ]
    }
   ],
   "source": [
    "print(summary.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The True Power of Large Language Models\n",
    "\n",
    "**Same transcript, different tasks?** Just change the prompt.\n",
    "\n",
    "Back in the days, when NLP Engineers needed to perform different tasks on the same text data, they had to train separate models for each task. The training included:\n",
    "- Data collection\n",
    "- Task-specific Data Preprocesseng (Very time consuming!!)\n",
    "- Model selection and training (Also time consuming!)\n",
    "- Evaluation and iteration\n",
    "\n",
    "With LLMs, the process includes:\n",
    "- Data Collection\n",
    "- Prompting the LLM with task-specific instructions\n",
    "- Evaluation and iteration\n",
    "\n",
    "So we got rid of the 2 most time-consuming step!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**A new prompt**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "COMPARE_PROMPT = \"\"\"Based on the transcript delimited by triple backticks. \\\n",
    "What has improved in LLAMA 2 compared to LLAMA 1? \\\n",
    "Transcript: ```{transcript}```\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In LLAMA 2, several improvements have been made compared to LLAMA 1. These improvements include:\n",
      "\n",
      "1. Training on more data: LLAMA 2 was trained on more robust data cleaning and 40 more total tokens.\n",
      "\n",
      "2. Larger model size: LLAMA 2 has more parameters, making it a bigger model compared to LLAMA 1.\n",
      "\n",
      "3. Doubled context length: The context length in LLAMA 2 has been doubled, allowing for a better understanding of longer conversations.\n",
      "\n",
      "4. Fine-tuning for chat: LLAMA 2 has undergone extensive fine-tuning for chat, making it more suitable for conversational tasks.\n",
      "\n",
      "5. Better performance in benchmarks: LLAMA 2 outperforms other open source language models in benchmarks, although it is considered more of an incremental upgrade over LLAMA 1.\n",
      "\n",
      "It is important to note that LLAMA 2 was not compared to GPT-4 in the benchmarks mentioned in the transcript.\n"
     ]
    }
   ],
   "source": [
    "compare = summarize_video(transcript=transcript, template=COMPARE_PROMPT)\n",
    "print(compare.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feel free to experiment with the prompt, so that you get the unique results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summarize from URL\n",
    "Combine all steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Function for getting the YouTube Transcript"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.document_loaders import YoutubeLoader\n",
    "\n",
    "def get_transcript(url: str) -> str:\n",
    "    \"\"\"\n",
    "    Returns the transcript and title from a YouTube URL.\n",
    "\n",
    "    Parameters:\n",
    "    url (str): The YouTube URL from which the transcript and title will be extracted.\n",
    "\n",
    "    Returns:\n",
    "    transcript (str): The transcript of the video.\n",
    "    \"\"\"\n",
    "    try:\n",
    "        loader = YoutubeLoader.from_youtube_url(url, add_video_info=True)\n",
    "        docs = loader.load()\n",
    "        if docs:\n",
    "            doc = docs[0]\n",
    "            transcript = doc.page_content\n",
    "            print(transcript)\n",
    "            return transcript\n",
    "        else:\n",
    "            return None\n",
    "    except Exception as e:\n",
    "        print(f\"Failed to load transcript and title from URL {url}: {e}\")\n",
    "        return None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "TEMPLATE_PROMPT = \"\"\"Summarize the video with the transcript delimited by triple backticks. \\n\n",
    "Answer the questions delimited by single backticks. If no questions provided, just create a general summary. \\n\n",
    "Questions: ` {questions} ` \\n\n",
    "Transcript: ```{transcript}```\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.prompts import ChatPromptTemplate\n",
    "from langchain.chat_models import ChatOpenAI\n",
    "\n",
    "\n",
    "def summarize_with_questions(transcript, questions, model):\n",
    "    prompt = ChatPromptTemplate.from_template(TEMPLATE_PROMPT)\n",
    "    formatted_prompt = prompt.format_messages(transcript=transcript, questions=questions)\n",
    "    print(\"Formatted prompt: \", formatted_prompt)\n",
    "    llm = ChatOpenAI(model=model, temperature=0.1)\n",
    "    summary = llm(formatted_prompt)\n",
    "    return summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def summarize_from_url(url, questions):\n",
    "    transcript = get_transcript(url)\n",
    "    token_count = count_tokens(transcript)\n",
    "    # select model\n",
    "    if token_count < 3000:\n",
    "        model = \"gpt-3.5-turbo\"\n",
    "    elif token_count < 14000:\n",
    "        model = \"gpt-3.5-turbo-16k\"\n",
    "    else:\n",
    "        return f\"Summary unavailable for transcripts over 14k tokens. Your transcript has {token_count} tokens.\"\n",
    "    \n",
    "    summary = summarize_with_questions(transcript, questions, model)\n",
    "    return summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# example videos\n",
    "urls = [\n",
    "    \"https://youtu.be/zJBpRn2zTco\",  # AI Explained on LLAMA 2\n",
    "    \"https://youtu.be/blyzUI8kOG4\",  # AI Advantage compares LLAMA 2 to ChatGPT\n",
    "    \"https://youtu.be/Xjy-CDRJa54\",  # Matthew Berman checks LLAMA 2 performance\n",
    "]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "meta just surprised us with a brand new open source language model called llama 2. this thing is the best open source model we have and in many cases they claim this to be better than GPT 3.5 which is the default chat GPT I'm better but in what ways is it better is it more up to date can you use it how does this move the AI space forward and why should you even care you should I'll cover all that today and we'll even go into a quick demo so First Things First Slice is such a big deal on why should you care well I'm gonna do my best to keep this simple but in the introduction of the paper that they released with this it says exactly what you should know there have been many public releases of pre-trained large language models such as Bloom llama and Falcon match the performance of closed pre-trained competitors like GPT free okay so that's the first thing there's a big distinction between models that are open that you can download and build your apps upon and that are closed where all they give you is a link where you can use their model on their servers but you can't actually download all the code and all the weights and build on it yourself right okay so I continue but none of these models are suitable substitutes for closed product large language models such as chat gbt Bard and Claude so just what I said right there these closed product large language models are heavily fine-tuned to align with human preference which greatly enhances the usability and safety and this is the big Point here okay a lot of the open source models up until now were below average and there was no fine tuning on top of them you might have heard the stories of openai paying thousands of people in third world countries to go over the results and rate them and then taking that data and feeding it back into the model right well it turns out that doing that is really damn expensive and the amazing thing here is that we get a really capable model with llama plus we get a variation that has been optimized by humans AKA heavily fine-tuned to align with human preferences and the two models that meta in cooperation with Microsoft released here are llama 2 and llama2 chat llama to chat being the one that has been feedback by humans Okay so so to me this is the really exciting one and they come in three variations so in total we got six brand new open source models here and the variations are 7 billion 13 billion and 70 billion and that's the amount of parameters it was trained upon 70 billion being the most capable one so the 70 billion llama2 chat is the one that I personally am the most excited about here and we're gonna talk about that more in this video we're gonna talk about further differentiating factors and exciting news after we look at the license because that is really the big news here okay and here it is this is the punchline llama 2 is free for research and commercial use so you can build your company chatbot on this and you don't need to pay for the gpt4 API you can make it your very own and you owe them nothing look at that this is the licensing agreement you're granted a non-exclusive worldwide non-transferable and royalty-free limited license there's one kind of hilarious exception to this which it says here if the monthly active users of the product or service built upon this is greater than 700 million monthly active users in the preceding calendar month you must request a license from that that so this essentially says if you're Amazon Apple or Google you need to get a license everybody else on planet Earth use this as you desire so again that is huge because we're getting the power of chat GPT into our hands and we can build on top of it now and that brings me to the next topic first of all in terms of safety this will be the safest large language model out there I have yet to test this extensively but have a look at this chart they ran around 2 000 evil prompts and the lower the percentage here the safer the model is AKA the less information gave away so chat GPT already was Notorious for not giving out much and being very secure right but here on the scale it comes in at seven percent and the Llama 270 billion chat model which is probably the most useful one in here comes in around four percent okay so this model is family friendly which is great for business applications right you want it to be that way I personally hope that in the future we'll also get fully open models but I understand the challenges of that and I think this is actually a smart approach okay but what about performance how good is this when compared to chat gbt well on page 19 they actually included a benchmark showing a comparison game and the way this test works is they use 4 000 helpfulness prompts and this is important to understand because here they even say it does not cover real world usage of these models and this prompt set does not include any coding or reasoning related prompts so a lot of this is like information retrieval where you ask a question that gives you an answer again that's exactly what you want from chatbots and if we look at the results from these helpfulness problems we will find that it actually won over chat GPT it's very close but it's ahead and honestly if they just match GPT 3.5 levels I'm happy with that that is more than good enough for a lot of use cases that you would want to build on this thing but when it comes to Benchmark there's a set of academic benchmarks that we want to look at and they included these here too and for consumers this is probably the most interesting part of this paper so as you can see right here we have the names of the different Benchmark models and before you look at these numbers you have to consider that all of these are closed except of llama 2 here right but the results are not bad I mean yes as I always say gpt4 still is king that's just Undisputed but I think the fair comparison here is GPS 3.5 and when you look at these results 70 versus 68.9 57.1 versus 56.8 and then okay on this one is really far apart but if you check out what this Benchmark is all about it's cold generation so okay you're not going to be picking this model for coding and I mean it goes without saying that gpt4 just smashes this Benchmark but on a lot of others even here it comes close to Google's Palm to L and if we go back into the open source realm which is more of a fair comparison here then you'll see that the Llama 70 billion model just smashes all the other models in all of these benchmarks reading comprehension first math not even close and even on reasoning it's the best out there right now and if you pair that with the fact that the cutoff date of this thing is actually September 2002 with fine-tuning data being more recent up to July 2023 you'll realize that this base model has one more year of knowledge than what GPT 3.5 has built into it big deal actually so overall I think it's fair to say that this is better than GPT 3.5 it's open source it costs nothing it's more updates and it's cleaner which can be a good or a bad thing I guess so that leaves us with two questions how do you use this thing and when would you want to use it well in order to use it you need to download this model and you can only do that by filling out this form and them accepting you but hold up I filled this out and within an hour I got an email where you get a link to the GitHub and your very own link that gives you access to the full thing so now I could download this thing and start building on top of it and this is the point this thing is not meant for consumers this is really meant for Builders but you can still try it out on Twitter I found this link to a streamlit app where nirand castlewell was kind enough to put up his chat demo for us to try so at the point of recording this is accessible later on I might have to switch out the link in the description but we can simply test this and first of all I'll just go with the classic right bnsa about penguins alright let's see how this goes and already having run this prompt hundreds of times across all different language models I can say the structure is very different and distinct from GPT 3.5 and gpd4 here I don't feel like I've explored this enough to give you guys objective opinion how the outputs differ but certainly this is a very usable results just like GPT 3.5 would give you spoken in a different tone and voice what interests me a little more is the safety aspect right what if I ask it something slightly spicy like tell me a joke about Donald Trump and it says I can't satisfy your request I'm just an AI it's not appropriate for me to generate jokes that might be considered offensive or derogatory Lord here we go again so what if I just say tell me a joke about penguins why did the penguin go to the party because he heard it was a cool Gathering okay but it does it so you can clearly see the political safety filter at work here okay one second it's editing Igor here and I need to bring two more websites to your attention because perplexity Host this and they're actually the fastest chat interface for you to try this so this is the 7 billion model and you can try it under this URL everything in the description below by the way and then also a16z is hosting a playground version of this where you get to try all models traffic on this one is just a little crazy so sometimes you have to wait but these are really the two best sites to test this yourself so now you know the last question that remains is what is this good for well first of all people are going to be building web interfaces like this where you can just use this as alternative to open AI but mostly this is good to build apps on top of you're not relying upon some external language model that they might turn off or change the pricing or change the quality of or sensor tomorrow right well okay on the censoring point this thing is pretty damn censored already but at least you know what you're getting for me personally the number one thing I see this being used for is chatbot thoughts that belong to companies and this is going to be the go-to model moving forward no more openai API with rate limits and having to explain to the clients that there's a few dollars of extra cost depending on the usage no you just download this thing build this into the bot and you have a self-contained version where the data is not being sent around to open Ai and back it's all right there the model the chat interface it's all yours and you don't know anybody anything and that's the big difference here because in this ballsy move by meta and Microsoft they really change the game and forced a lot of other players to act more openly because they just set the standard for what a good model is supposed to look like and what the licensing around it is supposed to look like this is really a fantastic Direction they're pushing the space into and I hope that this video helped you understand what is actually happening here because it's a big deal if you take everything we just talked about and you combine it with what I covered in this video you'll realize that soon we'll get to take open source models train them with people's personalities and you'll have like an embodied person inside of the language model it's absolutely crazy but check out this video to understand what I'm talking about because there I uncover a hidden capability with franchise apt where you can start talking to certain people just based on their Wikipedia page crazy times we live in but better get used to it\n",
      "Formatted prompt:  [HumanMessage(content=\"Summarize the video with the transcript delimited by triple backticks. \\n\\nAnswer the questions delimited by single backticks. If no questions provided, just create a general summary. \\n\\nQuestions: ` How does LLAMA 2 perform compared to ChatGPT? ` \\n\\nTranscript: ```meta just surprised us with a brand new open source language model called llama 2. this thing is the best open source model we have and in many cases they claim this to be better than GPT 3.5 which is the default chat GPT I'm better but in what ways is it better is it more up to date can you use it how does this move the AI space forward and why should you even care you should I'll cover all that today and we'll even go into a quick demo so First Things First Slice is such a big deal on why should you care well I'm gonna do my best to keep this simple but in the introduction of the paper that they released with this it says exactly what you should know there have been many public releases of pre-trained large language models such as Bloom llama and Falcon match the performance of closed pre-trained competitors like GPT free okay so that's the first thing there's a big distinction between models that are open that you can download and build your apps upon and that are closed where all they give you is a link where you can use their model on their servers but you can't actually download all the code and all the weights and build on it yourself right okay so I continue but none of these models are suitable substitutes for closed product large language models such as chat gbt Bard and Claude so just what I said right there these closed product large language models are heavily fine-tuned to align with human preference which greatly enhances the usability and safety and this is the big Point here okay a lot of the open source models up until now were below average and there was no fine tuning on top of them you might have heard the stories of openai paying thousands of people in third world countries to go over the results and rate them and then taking that data and feeding it back into the model right well it turns out that doing that is really damn expensive and the amazing thing here is that we get a really capable model with llama plus we get a variation that has been optimized by humans AKA heavily fine-tuned to align with human preferences and the two models that meta in cooperation with Microsoft released here are llama 2 and llama2 chat llama to chat being the one that has been feedback by humans Okay so so to me this is the really exciting one and they come in three variations so in total we got six brand new open source models here and the variations are 7 billion 13 billion and 70 billion and that's the amount of parameters it was trained upon 70 billion being the most capable one so the 70 billion llama2 chat is the one that I personally am the most excited about here and we're gonna talk about that more in this video we're gonna talk about further differentiating factors and exciting news after we look at the license because that is really the big news here okay and here it is this is the punchline llama 2 is free for research and commercial use so you can build your company chatbot on this and you don't need to pay for the gpt4 API you can make it your very own and you owe them nothing look at that this is the licensing agreement you're granted a non-exclusive worldwide non-transferable and royalty-free limited license there's one kind of hilarious exception to this which it says here if the monthly active users of the product or service built upon this is greater than 700 million monthly active users in the preceding calendar month you must request a license from that that so this essentially says if you're Amazon Apple or Google you need to get a license everybody else on planet Earth use this as you desire so again that is huge because we're getting the power of chat GPT into our hands and we can build on top of it now and that brings me to the next topic first of all in terms of safety this will be the safest large language model out there I have yet to test this extensively but have a look at this chart they ran around 2 000 evil prompts and the lower the percentage here the safer the model is AKA the less information gave away so chat GPT already was Notorious for not giving out much and being very secure right but here on the scale it comes in at seven percent and the Llama 270 billion chat model which is probably the most useful one in here comes in around four percent okay so this model is family friendly which is great for business applications right you want it to be that way I personally hope that in the future we'll also get fully open models but I understand the challenges of that and I think this is actually a smart approach okay but what about performance how good is this when compared to chat gbt well on page 19 they actually included a benchmark showing a comparison game and the way this test works is they use 4 000 helpfulness prompts and this is important to understand because here they even say it does not cover real world usage of these models and this prompt set does not include any coding or reasoning related prompts so a lot of this is like information retrieval where you ask a question that gives you an answer again that's exactly what you want from chatbots and if we look at the results from these helpfulness problems we will find that it actually won over chat GPT it's very close but it's ahead and honestly if they just match GPT 3.5 levels I'm happy with that that is more than good enough for a lot of use cases that you would want to build on this thing but when it comes to Benchmark there's a set of academic benchmarks that we want to look at and they included these here too and for consumers this is probably the most interesting part of this paper so as you can see right here we have the names of the different Benchmark models and before you look at these numbers you have to consider that all of these are closed except of llama 2 here right but the results are not bad I mean yes as I always say gpt4 still is king that's just Undisputed but I think the fair comparison here is GPS 3.5 and when you look at these results 70 versus 68.9 57.1 versus 56.8 and then okay on this one is really far apart but if you check out what this Benchmark is all about it's cold generation so okay you're not going to be picking this model for coding and I mean it goes without saying that gpt4 just smashes this Benchmark but on a lot of others even here it comes close to Google's Palm to L and if we go back into the open source realm which is more of a fair comparison here then you'll see that the Llama 70 billion model just smashes all the other models in all of these benchmarks reading comprehension first math not even close and even on reasoning it's the best out there right now and if you pair that with the fact that the cutoff date of this thing is actually September 2002 with fine-tuning data being more recent up to July 2023 you'll realize that this base model has one more year of knowledge than what GPT 3.5 has built into it big deal actually so overall I think it's fair to say that this is better than GPT 3.5 it's open source it costs nothing it's more updates and it's cleaner which can be a good or a bad thing I guess so that leaves us with two questions how do you use this thing and when would you want to use it well in order to use it you need to download this model and you can only do that by filling out this form and them accepting you but hold up I filled this out and within an hour I got an email where you get a link to the GitHub and your very own link that gives you access to the full thing so now I could download this thing and start building on top of it and this is the point this thing is not meant for consumers this is really meant for Builders but you can still try it out on Twitter I found this link to a streamlit app where nirand castlewell was kind enough to put up his chat demo for us to try so at the point of recording this is accessible later on I might have to switch out the link in the description but we can simply test this and first of all I'll just go with the classic right bnsa about penguins alright let's see how this goes and already having run this prompt hundreds of times across all different language models I can say the structure is very different and distinct from GPT 3.5 and gpd4 here I don't feel like I've explored this enough to give you guys objective opinion how the outputs differ but certainly this is a very usable results just like GPT 3.5 would give you spoken in a different tone and voice what interests me a little more is the safety aspect right what if I ask it something slightly spicy like tell me a joke about Donald Trump and it says I can't satisfy your request I'm just an AI it's not appropriate for me to generate jokes that might be considered offensive or derogatory Lord here we go again so what if I just say tell me a joke about penguins why did the penguin go to the party because he heard it was a cool Gathering okay but it does it so you can clearly see the political safety filter at work here okay one second it's editing Igor here and I need to bring two more websites to your attention because perplexity Host this and they're actually the fastest chat interface for you to try this so this is the 7 billion model and you can try it under this URL everything in the description below by the way and then also a16z is hosting a playground version of this where you get to try all models traffic on this one is just a little crazy so sometimes you have to wait but these are really the two best sites to test this yourself so now you know the last question that remains is what is this good for well first of all people are going to be building web interfaces like this where you can just use this as alternative to open AI but mostly this is good to build apps on top of you're not relying upon some external language model that they might turn off or change the pricing or change the quality of or sensor tomorrow right well okay on the censoring point this thing is pretty damn censored already but at least you know what you're getting for me personally the number one thing I see this being used for is chatbot thoughts that belong to companies and this is going to be the go-to model moving forward no more openai API with rate limits and having to explain to the clients that there's a few dollars of extra cost depending on the usage no you just download this thing build this into the bot and you have a self-contained version where the data is not being sent around to open Ai and back it's all right there the model the chat interface it's all yours and you don't know anybody anything and that's the big difference here because in this ballsy move by meta and Microsoft they really change the game and forced a lot of other players to act more openly because they just set the standard for what a good model is supposed to look like and what the licensing around it is supposed to look like this is really a fantastic Direction they're pushing the space into and I hope that this video helped you understand what is actually happening here because it's a big deal if you take everything we just talked about and you combine it with what I covered in this video you'll realize that soon we'll get to take open source models train them with people's personalities and you'll have like an embodied person inside of the language model it's absolutely crazy but check out this video to understand what I'm talking about because there I uncover a hidden capability with franchise apt where you can start talking to certain people just based on their Wikipedia page crazy times we live in but better get used to it```\", additional_kwargs={}, example=False)]\n"
     ]
    }
   ],
   "source": [
    "url = urls[1]\n",
    "questions = \"How does LLAMA 2 perform compared to ChatGPT?\"\n",
    "\n",
    "summary = summarize_from_url(url, questions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "content='LLAMA 2 is an open source language model released by Meta and Microsoft. It is claimed to be better than GPT 3.5 in terms of performance and usability. LLAMA 2 has been heavily fine-tuned to align with human preferences, making it more capable and safer than previous open source models. It comes in three variations with different parameter sizes. The largest variation, LLAMA 2 Chat with 70 billion parameters, is the most exciting one. LLAMA 2 is free for research and commercial use, making it accessible for building chatbots and other applications. In terms of safety, LLAMA 2 is considered the safest large language model, with a lower percentage of information leakage compared to ChatGPT. In benchmark tests, LLAMA 2 performed slightly better than ChatGPT in terms of helpfulness prompts. It also outperformed other closed models in various academic benchmarks. Overall, LLAMA 2 is considered better than ChatGPT, and its open source nature and licensing agreement make it a game-changer in the AI space. To use LLAMA 2, users need to fill out a form and get access to the model through GitHub. There are also online platforms where users can try out LLAMA 2. It is suitable for building web interfaces and chatbot applications, providing a self-contained solution without relying on external language models. The release of LLAMA 2 sets a new standard for open source models and licensing agreements, pushing the AI space forward.' additional_kwargs={} example=False\n"
     ]
    }
   ],
   "source": [
    "print(summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Count from URL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "like to transfer welcome it's nice to see some familiar faces from yesterday's intro to deep learning session this is a more advanced session we'll get into that in a bit more detail this is my puppy oboe he's the mascot for the series and yes we're here for a session on natural language processing with gpt4 and other large language models we'll be covering training all the way through to deployment with the key models for abstracting away doing that which are the hugging face Transformers library and high torch lightning so you can go to my website johncrone.com talks and you can go to today's date which is May 10th and then there are slides for today as well as the GitHub repo for today so let's actually get right into a Hands-On demo right off the bat because people love Hands-On demos so I've got our first demo is here in the repo there's a code directory which is full of Jupiter notebooks as well as some python files we'll get to those python files right at the end of this session so the very first notebook that we want is this gbt 4 it's the gbt4 all inference so this notebook here it's very simple just to give you a taste of how you can get these powerful llms on your own device so with all of my Jupiter notebooks in this repo you can click on open in collab and I've tested them all they will work once you're in collab you can go to edit clear all outputs so that the outputs that are saved for me already having previously run this it disappear and then it's gonna take a while to run these first couple steps so thankfully these aren't downloading onto your local device so while we are all log jamming on the Wi-Fi at least the enormous llm that's currently being downloaded is happening in a remote server and so yeah so we'll come back to this in a moment but you should see that this will work well and you can adapt hopefully this is all pretty obvious it's a very nice and Slick API it's actually running on the CPU of this device using something called quantizing which we'll get into later and I'll also get into detail on this GPT for all algorithm but the cools notes is that it's a large language model that fits onto a single machine and allows you to have gpt4 or at least gbt 3.5 like capabilities in being able to interpret your instructions and have a real conversation with you so I've got it kicking off with please answer this question what is a large language model well that is booting up feel free to maybe on your phone instead of on your desktop computer do feel free to connect with me so I have an email newsletter on my website johncrone.com to make sure you don't miss any updates and I have almost all of my content is free feel free to connect with me on LinkedIn just mention specifically that ODS just say odse somewhere in the connection request otherwise there's too many of them for me to say yes to all of them I have a YouTube channel with lots of free YouTube videos and a Twitter account and I host Super data science which is the most listened to podcast in the data science Industry we have two episodes a week every week of the year with the latest information on machine learning AI data Career Success lots of amazing guests and then question here is what is your level of experience with the topic so little to no exposure to deep learning you've got a few of those although some of those people were in my deep learning class yesterday I'm sorry that I didn't teach very well some deep learning theory nice some deep learning theory and experience with a deep learning library all right we got more hands going up and a strong deep learning experience okay so large language models do depend on deep learning we're not going to be doing any math related to deep learning or any math related to machine learning at all we're going to be using high level abstraction tools but sometimes I will be using terminology like epochs and batches that I'm not going to take the time to explain because I'm assuming that you have that knowledge if some of the terms that I use today move too quickly for you I've got a book called Deep learning Illustrated so very special thanks this content in today's session a lot of it is influenced directly by a session that I did in the O'Reilly platform uh last month which was a live conference on large language models and so the three presenters from that were hugely influential on the content that I have in here so that's Melanie sabaya who's one of the first authors on the openai gbt3 paper sedan osdemer you'll see lots of his code in today's session he's an amazing instructor on large language models I'll be providing with more resources of his later on and emboldened Sean costla who's here in person he's a data scientist on my team and he was hugely helpful in creating a lot of the code that we're going to be running today so thank you Sean all right so in this we have four sections that we're going to go through I'm going to introduce what large language models are then we're going to go over the breadth of large language model capabilities I will then teach you how to train and deploy large language models and finally in the fourth section relatively quick one on getting commercial value from llms so this isn't just technical we're going to have some stuff at the end on how you can be making a big impact with these really incredible capabilities all right we're going to start with the intro of course so a brief history of natural language processing pre-history in natural language processing was neural net free so we've had techniques like this since the 50s bag of words approaches TF IDF topic models like latent dirichlet allocation the Bronze Age actually happened relatively recently so just 10 years ago the Bronze Age kicked off in NLP so this was when we started having language embeddings like word to VEC and we started using those language embeddings and flowing them through deep neural networks like rnns including RNN subtypes like lstms gated recurrent units and we mapped those language embeddings often word embeddings through to some outcome that we are trying to predict and that worked pretty well that was the Bronze Age was a big improvement over pre-history the Iron Age is really recent so this is when we started having large language models with attention the Transformer which I'm going to go into more detail on shortly was devised in just 2017 so six years ago and it made its way into a lot of really famous architectures for large language model architectures for natural language processing in recent years architectures like Bert T5 and gbt3 all of which I will be talking about in more detail in the coming slides but the approach that set about the Industrial Revolution that we are in the midst of right now that is Transforming Our industry as well as transforming all Industries over the world was facilitated by rlhf reinforcement learning from Human feedback this is what has allowed through instruct GPT early last year early 2022 and then through the chat gbt user interface that had some more guard wearails around the instruct gbt interface it's this rlhf that has aligned this machine's outputs with uncannily what you were anticipating in a huge amount of human work has gone into making that happening with respect to labeling data so labeling instruction and response Pairs and allowing that to be so well aligned with the kinds of things that you'd like to be seeing in March of this year gpt4 came out which is a step change on this rlhf approach and if you haven't used gpt4 you've got to I couldn't possibly have made this presentation on the timeline that I did without gbt4 I'm talking about the code I'm talking about topic summaries I was able to come up with ways that I'd run into problems in my code errors I try to Google things and would spend 10 minutes 15 minutes trying to find the answer in documentation and then just throw my hands up tape you know I know gpt4 hasn't been updated since September 2021 but I got to give it a shot with this new Pi torch lightning Library anyway and I should have just started there because I could paste my code in and it would fix my code to do exactly what I wanted to do not arbitrary code from the internet that's like my example literally my example and it runs perfectly so this gpt4 architecture is a huge step change I'm going to talk about it a lot in this training and other companies other than open AI like anthropic and cohere also have models of gbt4's complexity that will be available in the public soon so Transformers are what have enabled the Iron Age so attention was being used in the Bronze Age so attention which Transformers do is this concept that we've had since having deep neural networks applied to NLP and attention is this idea of being able to look over long stretches of language to get relevant context and this is something that we do as humans when we're thinking about what sentence we're going to say or how we're going to communicate with somebody else so this was key The Innovation with the Transformer and why it ushered in the Iron Age is that we discovered that attention is all you need and that's the name of the thus wanting it all paper from 2017 so for natural language processing deep neural networks apparently all we need is attention so the architectures previously they still included were current layers they still included convolutional layers whereas now today our architectures are Transformers only attention is all you need what is a Transformer we don't have time to go into a huge amount of detail on it but broadly it follows this structure that you see here so you have inputs and those inputs get encoded into a feature layer so you have this these abstracted uh embeddings in this feature layer and then those embeddings can be decoded and you get some outputs so the first Transformer paper from 2017 from the Google brain team that was neural machine translation so let's have a look at an example here so if you have English going in as an input hello world that can be encoded by the Transformer it gets converted into these embeddings in the feature layer and then those can be decoded and output in the language of your choice say French and we can have bonjour lemon so we can't go into a huge amount of detail on the Transformer I could spend three hours doing that but some great resources are here in my slides including the annotated Transformer and The Illustrated Transformer key to our understanding of how Transformers work and how modern NLP Works in general is the idea of sub word tokenization so in natural language processing a token is the currency so it's a basic unit of text and these tokens are processed or extracted from your Corpus which is Latin for body it's your full body of language that you're working with and there's a range of possible levels for these tokens you can have sentence level tokens so breaking on a period or exclamation marks and question marks to have big sentence level tokens much more common is Word level tokens where you're breaking up on white space and punctuation so that you have each token corresponding to a word but modern Transformers tend to either use character level or even more commonly sub Word level so what does that mean sub word well I have an example here we have unfriendly and so that could be broken up into potentially three sub words where a friend is one of them friend we and on at the front yeah is it negation and so altogether this means that even if our algorithm had never come across the word unfriendly in its training data but it had come across the parts un and friend and Lee it could combine those together to understand this new word so this gives modern NLP algorithms an enormous amount of flexibility to understand new language this is the most flexible and Powerful approach and a really common way to do it is by byte pair encoding which I don't really have time to go into but you can read about more on your own time or just ask gbt4 about it and white parent coding is used for example in Bert and gbt series architectures which I'll be talking about uh momentarily so in my code directory in my NLP with llm's repo in this GPT notebook we are going to use the Transformers library for the first time from hugging face and we're going to use pipeline objects to very easily use a GPT model and we're going to explore what tokens are so you can open this up in colab the dependencies will take a moment to run I imagine we want bigger big enough and we'll import this pipeline object for very easily using GPT so while that's loading I can explain the next bit of code a bit oh I also want to clear all my outputs so this pipeline object allows us to in this case generate text because that's what we're specifying it to do so we specify a model that we'd like to use as well as a specific kind of task that we'd like it to perform so in this notebook we'll be doing text generation but we'll also you could alternatively do sentiment analysis named entity recognition summarization translation from a particular language to another in this case it's English to French or feature extraction I'm going to talk more about these kinds of tasks later on but here you can see it running so I'm downloading the model and with gpt2 which is the model that I'm very quickly downloading here this model because it didn't have that reinforcement learning with human feedback that I was talking about it isn't a zero shot learner it doesn't do a very good job of figuring out what you'd like it to do without some examples so it's what we call a few shot learner and that's at best I'm going to get into this in more detail gpt3 was really a good few shot learner gbt 2 was really just capable of generating coherent text however I was able to coerce it here into doing some few shot learning so with my text example I say the capital of Germany is Berlin the capital of China is Beijing the capital of France is and then I gave it a constraint I said only use a maximum of two tokens two new tokens when you generate your result when I ran this in testing it's really nice that here in front of all of you it did Paris but the first few times I ran it I got miles say I got Leon so this model is not as good as the kind of gpt4 the gpt3 that you're used to but it's still pretty cool it's nice that it was able to get in that area and that kind of gives you a sense of how these embeddings work it was able to get very close in our abstract embedding space to the right point it didn't quite nail it with Paris but I'll say early on is close so more on tokens you can load up this tokenizer for gbt2 and you can explore what's in the vocabulary so is love in the gpt2 vocabulary yes it is is synonym who created this notebook is his name in there no it is not so then when we encode a string like sinan loves a beautiful day what does it do how does it handle this word that it hasn't seen before so we can see here that it did encode it into so these integer representations get used So within gbt2 this pre-trained gpt2 model there's specific integers that correspond to specific tokens specific sub word tokens and we can actually use this convert IDs to tokens method to see what each of them are so synonym this word that isn't in its vocabulary gets broken into sin and n and then also these words that have a space in front of them there's actually a special character that denotes that space and so space beautiful or not space beautiful would actually get a different token in this particular training cool so that's tokens and that's gpt2 gives you a little bit of a a glimpse of Huggy face and some of the other key content that we need so going into more detail on the key kinds of language models that we have today there are Auto regressive models and GPT is actually one of these so what does it mean with auto regressive models we predict a future token for example with this sentence the joke was funny she couldn't stop blank Auto regressive models predict what the final token should be at the end of a string and that's it and so Auto regressive models are great for natural language generation for example gbt architectures like the gbte 2 architecture that we were just using Auto encoding models predict a token based on past and future context so they're not just at the end of a sentence these can encode in the middle of a sentence so he ate the entire blank of pizza predicting what that blank is is what an auto encoding model is specialized to do this makes Auto encoding models more efficient at natural language understanding so while natural language generation models also have to do encoding they tend to be less efficient relative to a model that is designed to Auto encode specifically The Bert architectures are a great example of this kind of model so what makes a language Model A large language model language models with greater than 100 million parameters is kind of the cutoff that we can use for large 100 million today actually isn't even that big because the largest models English language models today are models like Megatron Illustrated here and these have half a trillion model parameters and there are also there are models that have been made in other languages there's a Chinese model called wudow 2.0 that has nearly two trillion parameters and that's the largest model that I'm aware of today is size everything no it is not and we're going to talk about that later in these slides interestingly although uncommon today technically speaking so I want you to be aware of this a large language model doesn't need to have a transformer in it but the state-of-the-art llms today do have Transformers in them and typically they have many Transformers in them llms regardless of whether they're a Transformer architecture or not they are pre-trained on vast corpora in order for them to work well it's a very very large data sets of natural language how large again more on that later and generally these llms are pre-trained which means that they're able to out of the box perform well on a wide range of natural language tasks and again we'll get into lots of these different kinds of tasks later we've already alluded to how some of them are able to do few shot learning which is what we saw with gbt2 in the notebook just now when it predicted Paris after it was given the examples of Beijing and Berlin One-Shot learning would be where you give it one example so I gave it a couple of examples if you just said here's one example of translating English to French translate this next thing from English to French that would be a One-Shot example zero shot is the most powerful and what we've become used to today with uis like Chad GPT where we don't need to provide examples we can just ask for something and it does the task correctly usually right out of the box these llms can be fine-tuned to specific domains or tasks so they don't need to remain General you can fine tune them to something specific and that includes fine-tuning them to be aligned with what humans want so one of the first big specific architectures in this space one of the first big llms was Elmo from the Allen Institute the University of Washington ml stands for embeddings from language models it was an early Transformer based llm and it outperformed the previous state of the art which was recurrent neural networks lstms by a huge margin and CNN's as well so here you can see the gulf between the previous state of the art and how Elmo performed in 2018 so this was the first time that people started to realize wow for a broad range of tasks a Transformer only architecture could be a really interesting idea a really powerful idea Bert built on that and it was created by the Google AI language team the etymology of this is bi-directional meaning that it's an auto encoding language model so it takes context from before and after it's an encoder only which is interesting earlier when we were talking about Transformers I had I showed how we encoded and then decoded with Bert we encode only it learns representations it creates language embeddings from Transformers so that's where bird comes from it excels at natural language understanding Auto encoding tasks like classification semantic search so the way that this works is you're not trying to generate any language because this is just an autoencoder you're taking language and you're embedding it into an abstract space and then from that abstract space you can do lots of other Downstream tasks like classification or search T5 came out a year later and by Google again are you surprised and so T5 stands for text to text transfer Transformer and so it is an encoder decoder architecture which is the text to text it needs to be able to encode text as well as decode text it uses transfer learning so it's broadly trained for a wide range of specific tasks we have a number of examples here like translating English to German assessing whether a sentence is acceptable or not assessing semantic similarity between sentences and summarizing and I actually have a code demo coming up right now to allow us to try all of these different kinds of examples with T5 out of the box and the key thing here that was The Innovation was that the authors adapted a broad range of nlu tasks like assessing whether a sentence makes grammatic sense like evaluating the semantic meaning of a sentence and converting that into a text to text format into a generative format which wasn't always the case before this means the T5 is fast it's generative and it solves many NLP problems so here is a code demo of T5 so we've got the T5 notebook here oh the GPT for all inference notebook I forgot that was running in the back so how did it do here a large language model refers to the machine learning models that have been trained on vast amounts of data in order to predict human written text with high accuracy and without bad bias that's pretty good it's pretty good for gbt2 that's actually the best one I've seen sometimes it doesn't do a great job and it's actually it's critical with uh even this oh sorry this is GPT for all which I'll get into in more detail later on it's supposed to be able to really be aligned with our intentions from rlhf and be like gbt 3.5 or even bold claim like gbt4 but I still found that I got much better results if I said please answer this question if I just said what is a large language model it was it would often repeat other questions like what is natural language processing so on to T5 in this notebook again based on one by sinan osmer we'll use T5 out of the box for a broad range of NLP tasks so I'm going to clear all outputs and run the first couple of cells will I explain them so here we're going to be using the hugging face Transformers Library again we're going to be using the T5 model for text generation from the Transformers Library so hugging face Transformers comes pre-loaded with lots of the most common architectures for all the most common tasks making it very easy for you to take advantage of it and so very often when we're using the Transformers Library we're going to create a tokenizer object as well as a corresponding model object and as far as I'm aware you always want those to be I can't imagine it seems like it would be Mayhem to not choose the same model for both of these and so the reason why I'm choosing T5 base here as is common with large language models you'll see for a particular name of model like T5 there will be several sizes of model available to you so T5 base is the smallest one so for the purposes of this demo running quickly being able to download it quickly that seemed like the best option but there's like T5 XL there's these other sizes that you could be using as well and then once we have our tokenizer and our model loaded up we can perform inference with it so the first inference task that we're going to do is translation so I'm going to go through these four different kinds of tasks that I had on the slide here so we're going to encode a sentence with a specific request so we're going to say translate English to German where is the chocolate and we're going to return those tensors in a pi torch format is all that this argument means so in terms of the key arguments here that we're supplying the input IDs is something that you obviously need to supply we're going to ask our model to generate based on this encoded text that we've provided number of beams is a really interesting powerful argument so a beam gives you multiple different shots at getting the best answer so when you've been thinking about these generative models probably so far you would think about there just being kind of one thread that it's pursuing of like let's try this word and then this next word beams gives you multiple kicks at the can where it says okay as I come along you might get to the third word or the fourth word and it might realize hey by this third or fourth word if I actually stick with this one path I'm going to get much better results than if I just went with my first instincts when I was just generating the first word and the second word so it allows you to correct for sub-optimal choices from early on in the generation I'm going to set this running no repeat engram size this helps you get better results so there's a problem that can happen in generative models where it ends up repeating things over and over and over and so you can set a limit you know there's some words like the that you can't block but some longer words so here we're saying that any word that consists of three sub words or more we're not going to allow that to be repeated in any of our responses maximum number of tokens is the maximum number of tokens that could be output here and we're allowing early stopping so that we're not forcing it to have 20 tokens which would be kind of silly in this circumstance and it worked so where's the chocolate that is correct oh yeah great question from Ray and exactly right so with the number of beams the more beams that you have the better your result will be but yes the trade-off is that it requires more compute it'll run more slowly all right so another example of this same architecture doing a different task without us having to change anything else about the way that we're doing it I've taken the abstract from the T5 paper and I've pre-processed it gently here to replace new lines and then I just simply ask T5 to summarize it that's the only prompt that I provide I do have to change some of my arguments here so for example now I want to have a minimum length number of tokens I want to force its summary to have at least 30 subword tokens in it that's the only thing that I changed here and I made the max length longer so we have this window of 30 to 50. and transfer learning has emerged as a powerful technique in NLP a unified framework converts all textbook language into a text to text format our study Compares pre-training objectives that's a pretty good summary of this abstract here's another one where we're doing something called Cola which is it uses the Corpus of linguistic acceptability to check for grammatical correctness wow that was fast I didn't expect it to go quite that quickly so you can say so Cola is this remember T5 was created to do a broader range of NLP tasks so we can say Cola sentence and it will know what that means it's using Cola as a verb and so this class is going poorly is that grammatically correct yes that is acceptable and then if I call a the poorly is going class is that grammatically acceptable no it is not question up at the front all those developers have all our Center for your instructions so the question is whether so there are specific tasks specific abstract tasks like Cola sentence that you would just have to know that the T5 algorithm was trained on that but like other large language models it is a few shot learner it might be a One-Shot learner you might even be able to get it to do some zero shot learning in some circumstances so you can challenge it with kinds of tasks that aren't necessarily in its training data set as well yeah absolutely so the question is would it work to just say is this sentence grammatically correct let's try it so it might not answer in that like standardized output of like unacceptable no uh interesting yeah so great question there and so last one here is a semantic text similarity benchmark stsb so we provide two sentences how to fish and guide for Anglers so are those two sentences semantically related is the meaning similar and this stsb semantic text similarity Benchmark provides a score on a scale of five so we don't need very many tokens here so I just set a max length of three and it prints out 3.2 for this sentence you know both these things are about phishing so that makes sense how to fish if we have this second sentence where it's how to fish and a guide for hikers semantic similarity comes out as zero the question is whether there's a way to get a confidence score and the answer short answer is I don't know but you can try so gbt this is an algorithm that a lot of people a family of architectures that we're really excited about today and so what's the etymology of GPT it's generative which means Auto regressive so it's putting text on the end of sentences it's pre-trained which means that it's designed to perform very well at many kinds of tasks zero one or fuchsia learning depending on the on the era of the model and the t is Transformer spoiler so a bit about the opening our gbt family we've had five major versions released now GPT first came out in 2018 it was very small and accepted only a thousand tokens as an input gbt2 was the first one that made a lot of public noise and this one had one and a half billion parameters which is relatively small today but as we saw it was capable of generating coherent text we've already been using GPT in today's training gbt 3 huge model in comparison and a lot of emerging properties that people weren't expecting gbt 3.5 and gpt4 the big change here was that rlhf that reinforcement learning from Human feedback that I talked about earlier that aligns it with what humans are looking for and it ends up having amazing responses way better than we'd expect we don't know how big gpt4 is but it's probably bigger we don't know for sure they might have used tricks actually to make it smaller more efficient and a really big thing about gpt4 it's hard to get access to this API yet today but there is a version of the API that allows for 32 000 tokens as an input which is huge uh gpt4 also has Vision so three major ways for you to be thinking about how you can be using llms the first way is prompting so a chat GPT style UI allows you to do that you can use the openai API and we will be using gpt4 in the openai API in today's class or like we did with the gpt4 all that we downloaded or the T5 model that we downloaded or the gpt2 model that we downloaded you can use it at the command line with your own instance that you have on your own machine the other major way that you can be using llm is for encoding so we already talked about this earlier so converting natural language strings into a vector you can read a blog post about this here using these encodings for semantic search so taking converting all of your documents using Bert to convert those into encodings and then using a cosine similarity score to compare proximity between those documents and searching by meaning over your documents the third major way to use llms is transfer learning wherein we fine-tune a pre-trained model to a specialized domain or task and so this for example could mean you could take Bert out of the box but you have some financial application and so you fine-tuned Bert to be able to classify financial documents specifically or to be able to search over financial documents specifically or you could be using T5 you could be fine tuning T5 to generate strings that correspond to integers this sounds like a pretty random task and it is but we're going to be doing a bunch of it in today's class because it was a nice small task for us to be able to do and thanks to Sean costla for thinking of doing it all right so this is the end of our first section in it we learned that attention with Transformers is all you need for natural language processing we talked about how autoencoder llms are efficient for encoding that is understanding natural language auto regressive llms can encode and generate natural language but typically they're slower than Auto encoders we can fine tune llm results in specialized models which is a really cool thing because it means we can be developing our own proprietary llms for our own use cases and rlhf in particular is a fine-tuning approach that aligns our outputs with human desires uh so we've gone over an introduction to llms now let's talk about the breadth of llm capabilities so without fine tuning pre-trained transform-based large language models can for example they can classify text this means that they can take natural language and do sentiment analysis on it say that it's positive or negative or you can have your own specific categories so you can list for gpt4 it's so powerful you can say I have these four specific categories that I'd like you to be able to classify into here's an example text which of those four does it belong to and you could do that for a whole bunch of pieces of text to label your data for you automatically which is a really powerful thing that you should be thinking about doing with llms so that you can have your own smaller powerful machine learning models you can recognize named entities so you can have it extract people locations or dates which could be useful for some application that you have you can tag parts of speech like nouns verbs and adjectives you can provide it with context so you can provide it with language and ask it to answer questions about that so it means that you don't have to search for the answer across some large piece of context you can ask it to summarize things so we already saw T5 do this it'll create a short summary but preserve key Concepts you can ask it to paraphrase which is kind of like summarization but it means keeping the length and rewriting in a different way while retaining meaning you can ask it to complete things like predict next words you can ask it to translate and this includes if we had it in the training data you can translate from human language into code or from code into human language or from code to code if you learned r that was your bread and butter for a long time and you're now learning python you can just ask it to convert your R code into python for you and explain what it was doing you can have it generate this is kind of like completing but completing my idea here with completing is that you're taking like potentially a big explanation and just saying you know what do you think the next few words should be with generating or potentially asking it to generate paragraphs or or much more and it could also be code we could be asking an llm to generate large chunks of working code for us a great example that came up a few times yesterday in my intro to deep learning class was people were asking how do I choose good starting hyper parameters for a particular problem how many layers of do I want in my deep learning model how many neurons do I want in each layer up until March when gbt4 came out I would have said to you well find relevant literature on this problem and see what they're doing and use that as a starting point now I'd say after gpt4 to just create it so say to gbt4 I want to solve this problem what's a good starting architecture create the code for me in pytorch saves a lot of time and another llm capability is chat so engaging in extended conversation I came up with all of these on my own but on this next slide all of these were provided by gbt4 so I gave it those first ten I was like look at all these things that I thought of it that llms could do did I miss any and it can effectively provide an infinite number of things that it could do right so text simplification abstractive summarization where it's not just summarizing but it's actually condensing and rephrasing and synthesizing at the same time finding errors and correcting them finding sarcasm which is a crazy thing that we thought would be hard to do finding intent that's also a really cool thing for your kinds of applications so you could be finding intent is a really good one for a lot of applications so if you build an app and you want your users to be able to ask you anything you can ask the model to convert their question into some intent what does this user want and then you could take that intent and have it potentially go into a set number of specific Pathways within your platform you could even going back to the point that I made earlier about the classification thing you could actually say there are five options that we could go for in in my platform at this point there are five pathways and you could describe what those five are which of the five is my user asking for so your user doesn't have to select from a drop down box they can just type a natural language query and then you figure out their intent and and figure their path through your platform sentiment shift analysis this is an interesting one where the sentiment in some natural language changes so you can imagine over the course of like a customer service conversation the person phones at first and they're in a friendly mood and then there's a shift so detecting that content moderation keyword extraction extracting structured data so taking from natural language or tables or lists taking information out and putting it into specific fields for a structured database providing recommendations like books films music comma travel although you could you could use gbt4 to music travel you can take me to 18th century friends who should I listen to creative writing like poetry and prose stylometry this was a word that I didn't know before but gbt4 taught me which is like analyzing anonymous text and identifying the author so you could you could provide text and say who seems to have written this and it'll say ah it's definitely a Shakespeare it'll try to do that and text-based games this is fun even with chat EBT at Christmas I was at a Christmas party my sister hosted a Christmas party and a bunch of people at the party hadn't used chat gbt yet it was sacrilegious and so it turned out that a lot of people there liked Harry Potter a lot so my friend and I who had a lot of experience with chatgpt we used it to create a Harry Potter quiz game so he split the people at the party all these Harry Potter fans into two groups I don't know anything about Harry Potter but is able to do a Harry Potter quiz by having it say but suggest questions um because we were using chat GPT uh which was powered by GPT 3.5 at uh in December there were some hallucinations if I was to do it with gbt4 there would not be hallucinations very unlikely and then another thing that's that's happening more and more and more and I'm going to talk about this more later is multimodal models so being able to generate speech generate music generate images generate video or be able to understand those modalities you're seeing more and more of that and we're going to see models that have more and more of that all at once so how can you interact with llms and make use of all this functionality there are click and point chat interfaces for example chat GPT which probably most of us have used you can also in hugging face repos there are often places where you can interact with the models there so here is a relatively new model created by the same people that made stable diffusion the image generation algorithm and so this is stable LM stable language model and it's tuned which in this case means it's tuned on human instruction response pairs so it's designed to be able to probably do zero shot learning on a lot of different tasks it's designed to be aligned with us and seven billion parameters are in the model so we can ask it something gosh there's always all this pressure I should have come prepared with something come on give me something there's got to be something can you download hugging face models and run them locally great question thank you so much um and yes you can we're gonna have lots of code Demos in today's class where we do exactly that but that was a really great question and that was fast wasn't it for like for something that you can use for free that's just being hosted ah and the open AI gbt playground this is cool has who else has used this here already yeah some people a small percentage so what this allows you to do this allows you to use the kinds of query parameters that you'd be able to adjust at a command line interface with arguments inside a clicking Point user interface so text DaVinci 3 this is a gpt3 model and so temperature for example this controls Randomness so what was that question again can you download hugging face models and use them yourself if you turn the temperature down it will be more deterministic if you turn the temperature up it'll do wackier things and so I guess things like 0.7 is like a decent kind of Middle Ground between deterministic and wacky you can change maximum length there's lots of other parameters that you can read about ooh show probabilities oh that's cool so that actually shows you yeah let's do that so you can see that for each of the tokens so what this provides is it provides you with percentages of How likely it was to pick the particular words that it did and the words that it considered as Alternatives so pretty cool so this gives you another way to interact with these models I can really quickly can show you if you haven't used it before I I mean you've got to do this you've got to it's insane to me if you don't spend twenty dollars a month to get gpt4 within chat GPT plus like you don't need to be on a waiting list it's twenty dollars and it is so incredibly good you know the explanations like you know I was working on this last night and so I was asking and actually this ended up being wrong because of the cutoff that it has here but it acknowledges that it says because of my knowledge cut off but explaining why I'm in machine learning very specific questions about arguments on commands it just goes on and on and on I mean there's and very detailed um like code like working through code and helping me with it so me providing it with an enormous look at this huge block of code that I provided it huge block of code and this is nothing for its context window it can handle thousands of tokens but I provide all of that and I say I want to be able to to calculate loss on the validation data and it writes in the code that I need to be able to do that so it adds in this new method and leaves everything else intact it's amazing that would have taken me half an hour an hour to figure out on my own and it was a minute wild so if there's nothing that you take away from this class it's use gpt4 so we've seen staggering progress in the GPT family so gpt2 was capable of coherent generation long form text we talked about that gbt3 was able to learn new tasks by a few shot prompts in struck gbt which came out a year ago was able to fine-tune gbt3 with reinforcement learning from Human feedback to create gbt 3.5 this enables learning of new tasks by a zero shot prompts and it aligns outputs so that it's you'll see these Capital H's sometimes you'll see two h's three is nice HHH helpful honest and harmless so we're trying to make these systems so that they can't be misused and then chat GPT came out in November this intuitive interface added additional guardrails around GPT 3.5 as it was being released into the public and then gpt4 came out just two months ago and it's a markedly superior to any of the preceding llms it has reasoning and consistency over long stretches so this means that on the uniform bar exam which is a text only legal exam you spend years and years studying to be a lawyer and up until March you know you could say machines can't do my job machines couldn't do the bar exam and they couldn't the gbt 3.5 scored at the 10th percentile on the uniform bar exam in the US gpg4 is at the 90th percentile so it went from 9 in 10 bar sitters bar exam sitters being better than it to 9 and 10 being worse than it in a year the alignment to me is wild so sometimes I won't tell it that it did something wrong it will give me an output that wasn't exactly what I was looking for and I'll just ask my next question which required say some clarification on the previous and it it knows it it blows my mind this alignment of it knowing that it was slightly off with what I wanted and so it apologizes it says Ah I'm sorry you're right here's like more what you were looking for and it was very subtle the context is huge so this isn't uh publicly available yet but there is a waitlist for this you can get access to the gpt4 API that is a context window of 32 000 tokens that's about 25 000 words and that corresponds to I asked gpt4 to do this math for me about a hundred single-spaced Pages that's wild and the accuracy is 40 more factual it says so I don't know how they calculated that number that's in their technical report because my experience of using these tools went from I see mistakes periodically to I don't see mistakes yeah so 40 more accurate 80 percent less disallowed content relative to 3.5 the co-generation as I already showed you an example of is absolutely mind-blowing and you need to be using it so image inputs are also this isn't publicly available yet but it allows for image input so you can take a photo of your fridge this is one of the examples that they gave and say what can I make for dinner uh this style has become so good gbt zero which was this tool that was designed to prevent kids from plagiarizing on exams it doesn't work anymore with gpt4 they can I it it's it frequently says for me zero percent chance that this is AI generated maybe that'll catch up I don't know but it'll get better and better because you can also you can have it you can the trick is to provide it with examples of your style so you give it with well-written stuff that you've written and say generate more in my style and then it's got great plugins so it's got a web browser plug-in so I think this might tie to a question that we had up here at the front so you can search over the web using gpt4 style queries the pub the plugins aren't yet publicly available but there's a waiting list there's a Code interpreter so you can actually be having it understand and execute code which is wild so these two plugins are actually created by open Ai and then there's lots of third-party plugins that are becoming available like Wolfram Alpha for mathematics kayak for the searching for a car in a hotel for your next trip um so we've got a Hands-On code demo of the gpt4 API and then I'll let you have a break so this is a quick one gbt4 API we've got a collab notebook and in here so let me clear all outputs just like we have and all the other ones in order to use this there's two things that you would need you need to already have been approved for the gpt4 API a link to the waitlist is here and then once you have been approved you need to go and create your API key so in the open AI API I have these various keys so this collab demo one I created for this purpose so you create these gpt4 API keys well or for any of them if you don't have access to the waitlist you could do the same demo as me you can use GPT 3.5 or one of the early architectures and you use the API key for that as well so dependencies are simple we just need this requests library for making API requests Json for handling Json structure and paste the key and here we go then we can so if you didn't use this kind of get password thing you can just hard code your API key with this line of code instead of this line of code right so if you're doing this at home you don't need the secret key rigmarole and then the code is pretty simple so we've got this generate chat completion code so here's where you specify the model that you want to be using and so if you don't have gpt4 access yet you can just use GPT 3.5 turbo it's the next best temperature controls Randomness is I think this is actually range of zero to one it in some other models it is zero to five and then we're saying that we're not going to require it to have a cap on the number of tokens content type Json you provide the API key of course and then you provide the these specific arguments and you provide the code so this response code 200 means okay so if the response goes well so if this API call goes well then we will return the result otherwise we return an error code there is a cool thing about using the API that you don't get in a UI which is that in addition to providing user content so write a sentence about John Krone where every word begins with the next letter of the alphabet starting with the letter A in addition to that you can also specify context system context so this is optional and the default is to say you are a helpful assistant I don't know I actually haven't spent any time thinking about what other things you might want to write in there you are an unhelpful assistant but you can provide the system with context that will provide a different result and uh yeah so there you go although being considered diligently every fine gentleman John Crone learns medical Neuroscience I do have a neuroscience PhD opening powerful qualified research studies to understand various wonderful xenomorphic young zealous brains I ruined it on the very last word Ah that's too bad almost got it oh agent I are missing oh there we go well it still did pretty well I saved a really good one that it did to me that had actually a lot of context about my real life in it it blew my mind and I've saved that one I might just use it as my bio with conferences from now on so yeah so that's how you can use the jeep open AI apis um I hope that you enjoyed learning that and I hope you've enjoyed all of this training so far so we're now through the second of four sections we've learned that llms are capable of a staggeringly broad range of tasks that thanks to reinforcement learning from Human feedback more data and guardrails GPT 4 is zero shot and mind-blowing The Cutting Edge in llms is advancing rapidly the playgrounds and apis are extremely easy to use okay so I've introduced large language models we've gone over the breadth of llm capabilities now in section three the part you've been most excited for we're going to talk about training and deploying large language models so in this section we're going to talk about Hardware options we're going to talk about the hugging face Transformers library in more detail than we have before we're going to go over the best practices for efficient model training we're going to talk about open source llm options so that you can have your own llm on your own proprietary infrastructure we're going to talk about the pytorch lightning library that makes it fast and easy to deploy and train llms including Hands-On examples of single GPU fine tuning to a specific task and multi-gpu fine tuning we will wrap up with deployment considerations so for Hardware options if all you have available to you is a CPU this is okay for inference if you're using a smallish large language model and if that model is quantized so at the very beginning our very first Hands-On code demo was using that GPT for all command line interface so we downloaded gbt4all onto our own server happen to be a Cloud Server but you could have done it to your own server and that was a relatively small llm we could fit it on to a single server and not only that but it's quantized which means that this is something that you can do before inference time where you reduce the complexity of your model weights substantially so the default Precision is 32-bit single precision for model weights but you could have it go down to potentially as low as about four bits is what we see as both kind of the low threshold and so that means that your model is much much smaller and it means that you can run efficiently on a CPU and so that could reduce your costs because having CPUs running your llm would be cheaper than having gpus doing it this is not practical you cannot train an llm of any size on a CPU so inference yes in specific circumstances training no GPU this is your typical choice for training and inference you likely need multiple for training so with a lot of llms these are going to be too big to fit on a single GPU and depending on the size of the architecture that you go with you know if you wanted to run gpt3 or something of that size 175 billion parameters you're going to need like a dozen gpus just for inference time and there are also specialized AI accelerators out there in addition to gpus so these include things like tpus you may have heard of these are available to you in Google collab for free these are tensor processing units graph core has something called an ipu which is distinct from CPUs and gpus so you could have an ipu in addition to CPUs and gpus that you already have on the machine and these are designed for massively parallel mixed Precision operations we're going to talk about mixed Precision more later but essentially this allows you to speed up training and Amazon web services I was actually alerted to this because they if you listen to current episodes of my podcast super data science there are ads for these things the trainium in inferentia accelerators but I I hadn't heard of them until Amazon approached me to create these ads but these sound really cool and you can get AWS boxes that have these specialized chips for training and for inference designed explicitly for that and they handle llms really well so there you go a useful commercial Interruption so the hugging phase Transformers Library we've already been using this a number of times in notebooks in today's class you've seen how easy it is with Transformers to pick the model of your choice and use it for the task of your choice there are thousands of pre-trained llms ready to go in the hugging phase Transformers Library and it supports all of the most popular model architectures including all the ones that we've talked about in today's class like Bert the GPT family T5 and so on hugging face supports multiple languages so some models that it supports have over a hundred natural languages it's task ready so as we already saw in the GPT notebook that we were in earlier a wide array of different tasks is supported uh with Transformers so in the GPT notebook we used the pipelines functionality to get access to those tasks and then with other kinds of models like the T5 that we used you had all these different kinds of tasks supported inherently by the model the pipeline object is super easy to use for inference we already went over that I'm not going to go over it again but because there's too many other things to cover but super useful tool for very easily doing interference hugging face Transformers offers interoperability which means that with Onyx the open neural network standard format we can switch between pytorch and tensorflow so whichever of those automatic differentiation libraries you are most familiar with you can use with Transformers and so this means that actually you could mix and match seamlessly so high torch is more pythonic so it's often more fun to use so you might want to create your model architecture in pytorch and train in pytorch but then when it comes to deployment you want to take advantage of some specialized tensorflow library for serving up the model they have lots of adjacent libraries for serving on in people's browsers on small devices like phones and cars that have relatively small computers Edge devices so there might be these tensorflow libraries you want to take advantage of and so with Transformers no problem you can do that in terms of efficiency it provides built-in quantization the thing I was talking about earlier for compressing our models really small it has built-in pruning and distillation these are other kinds of approaches for shrinking your model size and I will talk about those more near the end of this training it's got a great community so it's got a model hub for sharing and collaborating we're actually going to take advantage of that we're going to train oh I'm going to it would take too long in class time but I'll show you exactly a model that we train for the purposes of this class and I'll show you through all the steps of doing it I'll then show you how we can upload our model to hugging face and use it there in the hugging face UI or download it and use it ourselves on our local machine it's research oriented so the latest models from research papers are available there and there is extensive tutorials and detailed documentation so an amazing Library a great ecosystem and it was a delight to learn so much about it for today's training so let's jump into another Hands-On code demo that will show some new functionality that we haven't explored yet in the Transformers Library so this is the G Pi t not GPT but G Pi T code completion notebook so in here in the interest of time I'm just going to show you what's in this notebook actually you know what it's pretty fast um so in this notebook clear all outputs load our dependencies so my only dependency other than the ones that are already available the only additional one that I need to install here in colab is the Transformers library and then we're just going to use pytorch as well import that so a cool thing that I hadn't shown you yet here is that we can be using any models that have been uploaded by anybody to hug me face so this is a specific one here this isn't like one of those core models like previously we were using T5 base and we didn't have to specify a specific hugging face repo we just specified T5 base but we can take advantage of so this one you could have come across it in a Google search or someone could have recommended it to you but there's this just G Pi T model which is a gbt architecture but it's trained from scratch not fine-tuned on python code from GitHub so it is it it won't be able to do natural language tasks it will only be able to do code generation tasks but that's cool that's exactly what we decided that we wanted to do here so it's a very small model you can see here it just took a few it took 27 seconds for me to download the tokenizer as well as the causal language model causal here Means Auto regressive so like GPT architectures it's just it's predicting next words it's not encoding well it is encoding but it's not just an auto encoder and then so here we go here's an example of code that we're going to have it run on so this is running on this specific instance of collab that I have running here and so we pass in these Imports as suggested text and then we ask it to continue and so we provided various constraints we've said that we want the max length to be no bigger than eight tokens beyond the original text that I provided and this argument here I just did it to suppress a warning if you don't do that it'll just tell you that this is what it is doing automatically and yeah so we use our tokenizer to encode our query and then we run it through our GPI T model that we've downloaded and then we ask it to decode and we just reformat it so that we get the the new line characters printing out nicely and there you go yes so the question is how do you address the security and privacy concerns when downloading a model like that I'm actually surprised that that's where your security and privacy concerns are coming out because now all of a sudden any information that I provide to it is just you know it's local to me um so the the question that a number of people ask me over the break which is one that is is worth addressing for the whole class is you know the security and privacy concerns should be if I'm sending it over an API to say open AI uh when I'm downloading it locally what's the security and privacy concern all right so there are lots of other examples of third-party models obviously there's thousands of them out there and sinan in his notebook here he has lots of other examples so a Turkish gpt2 he's got GPT Neo which is cool it's a very small model so just 1.3 billion parameters but it can do a lot of simple tasks reasonably well as he shows here in this notebook so pretty cool thing for you to explore there if you want to explore more third-party models okay back to the slides all right so now that you know how to download your own models you might want to train them for some specific task of your own and so how can you be doing your training most efficiently I'm going to go over the key ways that we can do that so we've got gradient accumulation gradient checkpointing mixed Precision training Dynamic padding uniform length batching and parameter efficient fine tuning with low rank adaptation we're going to cover all of those right now so let's get started with the IMDb GPU demo notebook so in this notebook we have I'm going to clear my outputs here so this is based on code that Sean Coastal from my team at nebula created and so we're going to use the hugging face Transformers library to illustrate how we can fine tune an l m to a specific task in this case the task is going to be predicting the sentiment of a film using a database from the Internet Movie Database IMDb so some of the movie reviews are positive some of them are negative and so we're going to fine tune an architecture to excel at being able to do that although in practice in the constraints of time we're not actually going to take the time to train it long enough for it to work really well but you can run it for longer on your own so if you are running this for the first time you'll want to go to runtime and go to change runtime type and change from the default no accelerator which is CPU only to a GPU and um so we're going to use pytorch as our automatic differentiation Library we're going to use hugging phase Transformers obviously there are some Nvidia Imports here that are specific for tracking GPU usage because we're going to use that to illustrate some of these efficiencies that I was talking about in the slides these various things those will show up in here we'll use this management library to print out GPU usage and see how that's working in our favor and then we also just for a bit of fun at the end sometimes people worry about being able to do explainability and so we've got a little bit at the bottom using lime to explain how this big llm is kind of producing some of its results so these are just utilities for GPU benchmarking I'm not going to go into them in detail we'll see them print out this line of code here is letting me know whether my GPU is available so if you're as opposed to using Google collab if you set about having the NVIDIA drivers working properly on your own server it is a nightmare it's nice in collab to just have this work really easily and so we can see that our Cuda device our GPU is available Cuda being programming language that's used on Nvidia gpus and so this here is just a bit of a query to say if we're using it if the Cuda device is available if the GPU is available let's use it if it's not available then let's use a CPU so we're using GPU which is great and then here's that GPU utilization method that we just created up here and so it's telling us What GPU we're using device zero because you can have multiple so it's zero indexed the first GPU is device zero and it's telling us how much memory is occupied on the device and I as I Was preparing for this class it kind of frustrated me that this showed 261 megabytes so I asked gpt4 why that happens why it is in zero megabytes and it's because just the initialization of having the Cuda library and various dependencies on there it takes up about this amount of space oh very little Ram is needed for this so you can see here just as like a quick example of how we can send things to the device so here we're just creating a a really simple tensor of ones and we're sending that to a device and then we're seeing how that changes utilization rates so it's just as a as a quick example and so it's this two device that allows us to be taking advantage of the GPU so you've got to include that in your code in pi torch code for example when you want to be transferring weights over there so the model that we're going to use that we're going to start with it's created by Microsoft so our pre-trained model is going to be this dialogue ranking pre-trained Transformers up down model so it's a gpt2 style architecture so it's generative and it was trained for a task related somewhat to movie review sentiment classification so it was specifically designed for predicting how many upvotes a comment made online will get so it's kind of good at interpreting human comments and you know deciding sentiment in a way on them so it seemed like a reasonable starting point as a model to get going with and so let's load in our tokenizer we're saying that we'll allow it to have inputs of up to a thousand tokens here we are setting the number of possible labels that can come out in theory we could actually just have one I realized when I was looking at the coj but I didn't want to keep flexing with it at this late stage in the game but technically if you only have two labels you only need one outcome and ignore mismatched sizes so this allows us to have two output neurons on a Model that was pre-trained with just one and that's also related to why we get this this warning that says you should probably train this model on a downstream task to be able to use it for predictions which is exactly what we're going to do so this is related to us frankensteining it right we're taking it originally had one output now we're forcing it to have two and so out of the box it's not going to be a very good model but if we fine-tune it on a downstream task it will be useful so let's look at our GPU utilization so we're now using 2500 Megs on our Tesla T4 GPU and that's a small fraction it's a about a sixth of the GPU space that's available on that kind of device so loading and pre-processing the data I don't really want to spend too much time going through the specifics of how we're just like loading the data and and working through it because you're probably not going to go home and work on this movie sentiment problem I want to just be focusing mostly on the the library but so we'll just talk kind of talk about it at a high level we're using this load data set function which is provided by the hugging face so in addition to the hugging face Transformers Library we're also using the hugging face datasets Library which is a an easy interface for accessing lots of different databases including the Internet Movie Database here so and then we we specify so we're just taking 25 movie reviews for our training set and 25 for a test set this is a toy size you wouldn't train a serious sentiment classifier with that small amount the full data set if I remember correctly is 25 000 and 25 000. so total of 50 000 movie reviews and so here you get an example of so here's our third movie review in the training data set and you can read it it's a natural language if only to avoid making this type of film in the future this film is interesting as an experiment but tells no Cogen story it's clear to me as a human that this is a negative review and the label corresponds to that so 0 is negative 1 is positive review now this model won't actually be able to train because for demo purposes these 25 and 25 that we selected all of the labels are negative so you can't train the model with only one label if you want to do this on your own later increase this to you could do the full 25 000 and let it train for longer but I just wanted this to run quickly so this doesn't really matter because actually what I'm showcasing in this notebook is efficiency best practices so memory management compute management and we can do that without actually being able to train effectively so here is the um this movie review converted into tokens right so we're using the same tokenizer or a tokenizer corresponding to this same Microsoft comment up down model out of the box these are the tokens the sub word tokens that it assigns here is some code for preparing our data for being passed into the model so our model is expecting input IDs is the standard hugging phase terminology for our model inputs IDs meaning sub word token IDs so input IDs you'll see that everywhere when you're using hugging face models and so we're just we're taking our movie review so a given review the text from it tokenizing it with our tokenizer converting that into an input ID we are truncating which means that if the movie review is longer than a thousand subword tokens we truncate it we cut it off at 1024 that was a hyper parameter that we said at the top 1024 and then if a movie review is shorter than a thousand and 24 we Pat it with a special padding token so that it gets up to 1024. so this means that all of our reviews in our training set as well as in our test set will have the same token length and yeah here you can just see what the structure of this of these data sets are so these data sets that we're going to pass in they have two columns these are Arrow structures which are very efficient data structures for their pandas-like but they're extensible to multiple machines and uh so yeah so I've asked for the column names on this Arrow object and it tells me that I have a label which is obviously whether the movie is positive or negative and our input IDs so it's a bit of a straw man so having all those extra padding characters is computationally inefficient but we're going to use that inefficiency in this vanilla training model it's a relatively naive and so we've got some arguments specified in a dictionary here so we've got we're saying where our output directory is we're just calling it temp you can choose an evaluation strategy so I'm setting it to no because I know all of my samples have the same label they're all zeros so what's the point in also evaluating you're not going to see anything interesting but if you want to change this the other options are Epoch which means that you'll test on your validation data after every Epoch of training or you could alternatively set it to run after a set number of of steps which in which case you'd set this to the string steps and then add another argument I can't remember the name of it but there's another argument you can include that says how many you know every 10 steps or every 100 steps I want to do validation I want to evaluate on my held out data and per device training batch size so we're doing a batch size of one and we are training for one epoch so just to get some results really quickly here we pass all of these default arguments in as a training arguments object into our trainer from hugging base so this is a really convenient method that allows us to specify our model which we pass in specify what our training data set is what our evaluation data set is what tokenizer we're using and our additional arguments so really straightforward so easy to use and once we have our trainer instantiated we can use the train method on it to actually do the training and so it was running here as I was explaining what's going on in this cell we got a result and then you can print the result so it says that it had 25 training examples it ran for one Epoch here's the batch size and it's providing us with some summary stats down here like our GPU memory occupied so this is the whole GPU and it's also telling us how quickly it ran so the samples per second was one and change samples per second so overall it took 24 seconds so this is our Baseline in terms of compute time as well as memory usage for this particular task now let's learn about ways that we can be doing this more efficiently so first way that we can be doing things more efficiently is gradient accumulation with gradient accumulation we maximize our GPU usage so we split our so a synonym for batches is mini batches and so let's say we had a sample size of eight in our batch in our mini batch with gradient accumulation we split that mini batch into some number of micro batches and so if we set our number of micro batches to four in this case then we'll end up with four micro batches of two samples each and we run we do a forward pass on each micro batch separately on our GPU so we're taking we're maximizing the use of our GPU we save our gradients from each of those micro batches separately and then we perform back prop on the accumulated gradients once we have all of our micro batch gradients and so this allows us to have larger mini batches which means fewer training steps overall it means fewer rounds of back propagation on our gradients and faster training but you don't have to take my word for it because we can do it so here in this cell now we're using the same model but we are sitting a new default argument so remember we have this dictionary of default arguments we're adding in one additional item into that dictionary which is the gradient accumulation steps so now we're breaking up our batches and we end up with we end up with a faster training time so this approach ends up using even more GPU memory so we went from twelve thousand to thirteen thousand but the trade-off is that we were able to increase how many samples we were processing per second so we went from 1.03 samples per second to 1.3 which means that our overall training time went down from 24 seconds to 18 seconds our next trick is gradient checkpointing so in a typical forward pass we store all of our intermediate outputs for backprop so each layer of our neural network has activations and typically we're saving all of those for back propagation this is compute efficient but it's memory inefficient because we have to save all of those with gradient checkpointing we save a subset of all of our outputs so we don't save all of our intermediates we only save a subset of them and we recompute others as needed during back prop so what does this mean this means that we're going to have more memory efficiency but is also going to increase our compute because now we're not going to have all those intermediate values stored conveniently so let's see how that trade-off affects us in this cell here this gradient checkpointing cell so I add in another additional argument I just it's a binary argument gradient checkpointing I set it to true all the other code is the same and now our training is taking longer but the trade-off is that we are going to have less memory usage so come on there it is so our training is now slower than it was so we went from 1.3 samples per second to less than one so our training time bumped up from 18 seconds to 26 but our memory usage halved so we went from 13 000 megabytes of memory down to 7 500. automatic mixed precision so typically as I mentioned earlier we use 32-bit floats uh single Precision to store our model parameters our activations our gradients and so this is what these are all of the bits the 32 bits in a single Precision float data point a half Precision float has just 16 bits and so it's half the size it requires half the compute to process this preserves memory and speeds training automatic mix Precision means that it automatically figures out what kinds of circumstances we can be dropping to the half precision and what circumstances we should be sticking with the full Precision because we'll need it so that sounds really efficient um so all we do there is we set another new default argument of floating Point 16 equal to a binary flag of true and this will proceed very quickly though apparently not as quickly as I can speak and so now our training time is lower than we've seen before so we're down to 15 seconds our previous record was 18 seconds so now we're down to 15 seconds and in addition our memory has still stayed small so we've we've managed to retain almost as small memory as we had before while getting a much faster training time nice all right and then we have Dynamic padding and uniform length batching this is our last little efficiency to talk about so imagine that these are your 12 training points so we have 12 different movie reviews each with a different length what we did initially is we set all of them to have padding that was equal to the large one actually we did even we did something even less intelligent than that because I just set it to a large value 1024 that I assumed without checking that all of our movie reviews would fit into so here this is even a little bit more intelligent where it's saying what is our longest movie review and let's use that as our cap Dynamic padding is this idea that you only pad up to the largest document in that one batch so you save on doing this compute and and look at on batch three we save all of this compute here which is great but Dynamic padding with a sorted data set with this uniform length batching is even smarter so this uniform length it doesn't necessarily mean exactly the same length even though uniform sounds like that it just means about the same and so that's the related to this ranking so we have uniform ish length across all of the training data set points in a given batch um so if you were to shuffle between epochs then I guess yeah you can't which which actually interestingly with llms the uh the typical thing that you do is you use all of your training data once you typically this is a different kind of example where we have a relatively small amount of data set but when you're talking about training in llm for general purpose when you're training it on billions of tokens you typically run for one ebook nice so let's see how that impacts our model so this is a little bit more complex than the other changes because we have to change how we collate our data but you've got this really convenient hugging face Transformers method for doing that and then we just recreate our training in our test data sets re-pass those in and then this is super fast it's by far look at that six seconds seven seconds and without increasing our memory usage so clearly this is the way to go and over the course of all these optimizations we've made huge improvements so we went from our vanilla training without all of these efficiencies it was taking us 24 seconds and used up almost all of the memory on our GPU and now it is running in uh a small fraction of the time a quarter of the time and only half of the memory usage so all these changes make a big difference oh yeah I guess it would create it would create a problem yeah I guess I could potentially create a problem yeah I hadn't thought about that um if assuming your batches aren't too small um and assuming that problem isn't like if your label was predicting if it was like really directly correlated with short review and long review then you'd get into more of a problem but yeah that's a good catch I hadn't thought about that and it's a really good point to be aware of that all right last little thing I have in a notebook here for you is explainability so I'm just going to empty the GPU cache here we're not going to need it for this and I'm not really I'm not going to take time we have too many other things to cover for me to go and explain how all this is working in a lot of detail but what Lyme does is we can take a given data point so specifically here we're working with that same movie review that we were looking at earlier and what Lyme does is it sends that movie review remember the one about the Cogen story it sends different samplings of it through the model and by sending those different samples through we can build up a sense of what tokens are more associated with a positive review versus a negative review so it's finding that the word spend is really negatively Associated whereas experiment is really positively Associated so this this starts to give you a sense it allows you to understand your model a bit better and potentially explain things to your clients in some circumstances the other big explainability library is shap s-h-a-p cool so we understand how to train models efficiently now and we're familiar with the hugging face Transformers library for loading in our model and doing fine tuning on whatever task we'd like it to what are the open source chat GPT like llms that are available to us these are quickly disseminating so every week There's a few big ones of these that make the headlines the first one was llama which came from meta and this had gpt3 like performance at a 13th of the size so it's 13 billion model parameters whereas gbt3 itself is 175 billion so it's a 13th of the size and had gpt3 like Behavior researchers from Stanford took those llama model weights and fine-tune them on a data set of 52 000 instruction response pairs that were created using GPT 3.5 so they use chat GPT to say here's an instruction I mean they I think they even used it to create the instructions so you know create lots of this kind of instruction and then respond to those instructions if they created 52 000 of these pairs they said how much it cost it was less than a thousand dollars and they were able to use that to fine-tune llama similar to the kind of fine tuning that I just showed you how to do and get results that were gbt 3.5 like which as we saw is a huge step change because it has that kind of rlhf feel it's aligned with us and we can get zero shot results vicunia came along and started to approximate more like gpt4 so they were quantitatively Superior to llama and alpaca and this um this evaluation was actually done by gpt4 itself so they used shared GPT which is a browser extension that you can get yourself for easily sharing cool chat GPT conversations that you've had with your friends and so some people opt for those conversations to be put into the public domain so it created this data set of 70 000 chat GPT conversations vicunia trained on those and then they used gpt4 to compare themselves so they also started with the Llama weights so 13 billion parameters and so they said how did we compare so that's what this chart is up in the top right hand corner how did vicunia compare with llama with alpaca with Google's Bard with gbt 3.5 and they asked gpt4 to do that evaluation quantitatively so give me a score out of 10. and so vicuna beat Lama and alpaca in almost all cases on their test set of 80 cases it was about the same as bard slightly better and not as good as GPT 3.5 on their testing set a big breakthrough in terms of what you can do with these open source llms that you can fit on a single GPU came with GPT for all J and the reason why this was a big breakthrough is because it has a commercial use Apache license so these previous architectures were based on meta's llama and meta did not allow commercial use so with gbt4lj all of a sudden you could use these models freely and they've fine-tuned gpt4lj so they started with GPT J which is completely open source Apache 2.0 licensed and then they fine-tuned it on 800 000 open source instructions I don't have for you a convenient Benchmark of how well exactly it performs this is comparable to the model that we started off today's training with that we downloaded those quantized CPU weights and had it um how to provide an explanation as to what an llm is that was using gbt for all which is actually not the commercial use one that one was based on llama weights but gbt for all J is commercial use friendly Dolly 2.0 came out the same week as gbt for all J it was released by databricks it also allows for commercial use interestingly it's fine-tuned on human generated instructions 15 000 of them by people who work at databricks and so this is another model that you can be considering playing with alongside GPT for all j a week after that cerebrus came out with their model and so this is really interesting so we were talking about a moment ago how we typically with training these llms we train through all of our training data once and so a year ago researchers at Google they did an experiment where they experimented with lots of different data set sizes as well as lots of different model sizes and what they determined from that experiment doing like a grid search over all these possibilities they found this optimal ratio which ends up working out to about a 20 to 1 ratio of training data to number of parameters in your model in order to get the optimal model training for a certain compute budget because what this means is you could actually if you don't have that constraint of for a certain compute budget you can just train for longer you can always train for longer you could train for a second Epoch and that would actually give you even better results but if we want to if you're constrained by a certain budget and you can say okay how much training data should I ideally use for a model of this size then the answer is that you should have 20 times so if you have a billion parameters in your model then you should train it for one Epoch on 20 billion tokens got it if you have a two billion parameter model you should train it on 40 billion tokens so these are what today are called the chinchilla scaling laws and so what cerebrus did was they created a family of seven models they're shown here in this bottom right hand chart so seven commercially usable models ranging from up to 13 billion parameters which is comparable to llama alpaca vicuna and gbt for all J um but also going down to quite a bit smaller and so the advantage of this is that they trained on way more data they pre-trained on way more data there's two steps here right there's the pre-training on all of the unstructured data that you use that I just described with this 20 to 1 ratio and then after you're done that you fine tune on the instructions I can't remember exactly how they fine-tuned but the base model the pre-training use these 20 to 1 ratios on these scaling laws and so it means that you can pick a model that's quite small 100 million parameters very tiny will be very efficient very quick to download you could run it on a CPU especially after you quantize it very quickly and now you know that it's been optimally pre-trained with this 21 ratio so that's great it's a great starting point so that was the cool thing about the cerebrus GPT family and then a week after that came stable diffusion with a stable LM model we already talked about this earlier when I was showing a hugging face example the big thing that stable diffusion did here was they collated a bigger open source data set than we'd seen before it has 1.5 trillion tokens which means that if you want to spend the compute on it you could be using this huge vast data set to train for even longer than the cerebras folks did they have released three billion and 7 billion parameter models already and they have models up to 175 billion parameters planned so there's lots of different ways that you might want to do your fine tuning once you've downloaded the single GPU open source llm of your choice because there's so many ways to do it instead of me going through my own specific example I'm just going to use sinan awesomer's example just kind of quickly talk you through his notebook here so that you're left with a clear sense of how you can be using it yourself so we're using Transformers pandas the hugging face data sets library and what we're going to do here is we're going to use distill GPT 2 which is an even smaller version of gpt2 it's been distilled down and we're going to fine tune it on the alpaca data set so remember alpaca is the Stanford group they had their 52 000 instructions so we load that data set and you can see examples of the training data of these instruction response Pairs and these first five examples they only have an instruction they don't also have this input populated and so this extra input field it's only used in a small in a relatively small number of these circumstances but it allows us to have an instruction as well as some other kind of input alongside the instruction so identify the odd one out here are three examples Twitter Instagram telegram which is the odd one out and the correct response is Telegram so you have 52 000 of these training examples and you can see their lengths here most of them are relatively short so we load in our distilled GPT to model as well as its corresponding tokenizer we pre-process the data by tokenizing it converting it into our encodings for this particular architecture and then we pre-process our alpaca fine tuning data set as well we split the data set into a training in a test set so that we can compare performance as we train you just you can see some examples of the data set here and then here we create an instance of our causal model this is a generative model so we want to use Auto regression it's just causal language model and then here is the code for actually doing the work so for collating our data sets and then we can train using the specific trainer arguments so most of these are very similar to the kinds of arguments we've seen before I don't think there's anything that I need to dig into particularly I have some comments for the ones that might be not straightforward to explain to understand but fundamentally the trainer is here we instantiate it and after training here um oh sorry sorry we haven't trained yet we just evaluate first to get a benchmark as to how well we're doing so we have our evaluation loss before we've done any training then we train here and we can see how our training loss as well as our validation loss goes down over the epochs so yeah so it works and that I hope that gives you a sense of what you can be doing here he also shows we're going to go into some examples of this later as well but saving the model into hugging face and then you can be downloading the model yourself so you could yourself download from a hugging face this model that is now publicly available and you could be using it with pipelines to very easily make use of this now publicly available very small open source llm yeah so the question is are there IP restrictions on using open AI data sets so you uh the terms of use with openai state that you can't use openai to create a model that competes with an open AI model so um I don't know what that means in terms of if you're creating a very Niche model and using the openai API to um I don't know like it's not competing with openai I I don't know like can you can you so I don't know where that I don't I'm not planning on finding out yeah so yeah depending on how yeah so that's a good question for your attorney um I I don't know I am not a provider of legal advice sweet so this has been useful right so already I think I've covered enough that you can be leaving here today with some pretty cool capabilities around downloading your own open source model and fine-tuning it to some proprietary task of whatever interests you whatever would be useful to you now I'm going to get into Pi torch lightning which will make your job even easier because pytorch lightning is a pytorch wrapper an extension that simplifies model training without losing flexibility so you can adapt the pi torch under the hood as much as you'd like key features of Pi torch lightning include that it has a minimalist API so you can quickly restructure your code from pytorch into a lightning module that's all you need to do it offers automatic optimization so those optimizations that we went over in the hugging phase Transformers libraries things like gradient accumulation mixed Precision training and learning rate scheduling which we didn't talk about but this is how fast or how slow you should be training at any given point in your training and so these kinds of optimizations are handled automatically by pi torch lightning it has a built-in training Loop so if you were in my intro to deep learning class yesterday when we went from tensorflow to pytorch all of a sudden we went from just having this nice convenient train method in tensorflow or fit method in tensorflow to in pi torch having to write out our Epoch Loop and then a nested Loop inside of that that does our batches with pi torch lightning we don't need to worry about that boilerplate anymore with training with validating with testing pytorch does that automatically it allows for distributed training out of the box so you can have not just multiple gpus on a single machine but also multiple machines it has a callback system that allows you to implement your own custom Logic on things like checkpointing and logging so again more flexibility it has Integrations with all the most popular tools like tensorboard and ml flow and now you get to see what it's like yourself so let's head into this fine tune so in this notebook this is one of this is probably the single most substantial notebook in this training I'm gonna keep it as service level as possible because we've only got half an hour left and I still need to get to multi-gpu training which is also a bit of a bear so we've got our dependencies we're using pytorch for automatic differentiation we're using Transformers for our model for our tokenizer and our training function we're using pi torch lightning for easier training so when I say training function here I just mean literally this one little Training Method it's specific to how we're scheduling our learning rate we're using pytor's lighting for easier training we're using tensorboard thanks to this built-in jupyter notebook Magic later on in the notebook I'm going to show you how to use parameter efficient fine tuning with something called low rank adaptation which dramatically speeds up how quickly you can be fine-tuning a model particularly a large model and then just a couple of system utilities we've got some helper functions similar to what we had in the preceding notebook for getting GPU utilization metrics as well as for generating our data set so the gp2 utilization method it's the same as the one that we saw in the preceding notebook and prior to any training we again have very little memory being used remember if you're running this for the first time again to change your runtime type to a GPU and then you can select a model so we're going to use T5 small because the task that we're going to do is pretty simple and so T5 small only has 60 million parameters which makes it a great choice for demo purposes here today and you can read more about the other T5 model versions in hugging face here so let's generate some of our data and I'm not going to go over the data generation code you can just trust that it works you can actually see it right here what it does is it creates a random sample of numbers and our inputs into the model we're going to fine tune it to take in integers as inputs so 4477 is an example input from our training data set from our first sample in our training data set and we're going to teach this model to map that integer to the natural language representation 4477. we've created a thousand training samples and 200 validation samples that consist of this kind of structure Sean created a great streaming data set method here which I encourage you to use so this is different from what we've been doing so far in this training so previously in this training with our custom fine tuning we've been loading in all of the data and pre-processing it all at once now here with the streaming data set it means that we're extensible to any data set size because we're only going to load the data in when we need it so in the interest of time I'm not going to go through all the specifics of power like encoding and pre-processing the data here you can go through that code on your own but the key thing here is that it has this get item which means that when we call down here for say a specific instance so when I say I want the first instance of my training data it will now go to this get item method and actually pull in the data and do all this pre-processing all this encoding let's have a look at this because this is something that you're going to see with a lot of llms so input IDs we've already talked about for T5 small this is how the tokenizer converts it into the subword embeddings and so there's a parameter here where we have this max length 16 parameter which means that we have up to 16 tokens and any of the tokens any of the token space that's not used gets represented as a zero so this is our padding token there's this cool other tensor here that we have called the sequence mask which identifies which of the tokens actually has information in it and the reason why this is useful is that the attention mechanism that we have in Transformers is relatively computationally intensive and by having the sequence mask wherever we set it to zero it means that we're not going to bother attending to those parts of the input so we're going to save on compute and then we have this target IDs which is the same thing as here it's just for our outputs so input IDs is the sub word tokens for our inputs and then the target IDs is the same thing for our outputs so unsurprisingly so if you look at that very first example it's unsurprising that we can represent 4477 in a very small number of subway tokens but then when it comes to representing the output it takes more tokens so we have this longer output to learn the idea of sequencing of attention sequencing doesn't matter for the outputs that's just for encoding it's not for decoding so there isn't the equivalent here but we do have this additional label this is an auto regressive model what it's trying to learn is what the ideal next token is and so what this is is it's saying what the ideal next token is so when you're in this position what's the ideal next token it's this so 7863 once you're in this position what's the ideal mix token it's 662 so it's here right so this function here collates our individual data point tuples into uh batches so it takes this Tuple that was created by pre-processing our first training data point and it just breaks it up into batches so this function will break up our data points into batches of four so you have four input IDs four sequence masks and so on uh here is the lightning training module so we're seeing this for the first time so we specify that we have a lightning module so going back to my slides I was saying how the minimalist API all we need to do is restructure our code from PI torch into a lightning module so we have this Pi torch lightning lightning module we're going to call it a T5 fine tuner and some of the highlights of what's going on in here are we have our forward pass where we pass in all of our training information into our model we have training steps so just making one step forward doing one round of training with our training data doing one round of training with our validation data the difference here being that we don't do back prop during validation we only do it during training and so you specify these specific method names and the pi torch lightning module knows what to do with those specific method names so you don't have to provide all of the detail like we did in my class yesterday in pytorch on all of the steps of the forward pass or all the steps of a training step just handles that automatically data loaders provide clever optimizations like pre-fetching the next needed training data point on an additional worker so this means that you can have one worker doing a forward pass or doing back prop well another worker is saying hey I've got some exercise capacity what could I be doing oh I could be fetching the next data point and pre-processing it so you're taking advantage of these kinds of efficiencies that's why it's lightning fast here you tell it how many samples to pre-fetch at any given time that's a hyper parameter and then there's an Optimizer here so we're using atom adaptive moment gradient descent which was the same kind of Optimizer that we were using in my training Yesterday by the end of class and there's some specific Heroes like like warm-up steps this isn't something that we talked about and isn't super well known this is something that we often see with llms where we have very slow initial steps of training that allow our attention heads in our Transformer architecture to warm up I should have been running all this code while I was talking about it I've got this this for for logging tensorboard I asked gbt4 how to do that and and yeah and then here we actually train the model so we grab the pre-trained model T5 small and then we our batch size is 256. uh not four sorry uh I must have gotten that from later in the notebook and then so we pass in our training arguments just like we did in hugging face so what learning rate we would like to have how many epochs are we training for mini batch size and then very simple syntax here for actually doing the training we create a pi torch lightning model that with our hugging face model so we use hugging face to create our specific instance and download that hugging face model T5 small and then we pass that as the first argument into our custom Pi torch lightning class that we created here so the first argument is the model so we pass in a hugging face model into pytorch lightning as well as our arguments for our hyper parameters for training our training data and our validation data here we actually run the trainer specifying our epochs how often we want to log results this defaults to 50 but I set it to 4 so that we could have something to show you guys right away and once we have our instance of the trainer created pytorch lightning trainer here's a callback for the tensorboard logger by the way forgot to mention that then we can run fit on it and it actually starts fitting so here we go it finished training after the one Epoch of training on this very small amount of data that we asked it to do we ended up having a training loss of 8.3 validation loss of 7.6 and you can see how much GPU space we're using and you can see how with all the optimizations that Pi torch lightning takes into account we're using up just a very small amount of memory which is pretty cool and it corresponds to a size of our model so our model is is pretty small here 242 Megs and so that's what we're seeing stored in here and so if you want to see this predicting on an individual data set we haven't done very much training so it's not going to be a very good model but here's our first example in the validation data so the first input is this and the corresponding word is this uh so we can look at what each of these individual validation points look like after we've converted them into their sub words so again we have uh you know our validation IDs and a validation mask similar to what we saw with the training data and we can now pass these validation data points so the validation input IDs as well as our mask pass that into the generate method of our pytorch lightning model and we specify that we don't want more than 16 tokens being passed out and here we go so it generated these subword tokens did it do a good job we can now use the decode method of our tokenizer in our pytorch Landing model to actually output its prediction and it isn't very good but we've only fine-tuned for a few seconds on a very small amount of data a quick thing to show you in addition to training on the whole model there's a really cool technique here called parameter efficient fine tuning which is also available from hugging face you can read all about it in this repo here and what this does is in a nutshell it allows us to insert matrices additional new matrices in strategic points throughout our big llm and we just fine tune we just train these strategically placed new matrices we leave all of the rest of the llm Frozen this helps avoid catastrophic forgetting but it does in some cases actually allow us to even outperform fine-tuning on the whole architecture so it's compute efficient it avoids forgetting some things that the llm was pre-trained to be able to do and it can even give us more accurate results so this code here shows how to do it we use the low rank adaptation which is a specific kind of PFT and so the arguments here are that we turn inference mode off when we're training these are hyper parameters related to this specific low rank adaptation so the lower that you make this R parameter the fewer parameters you're going to have for training Alpha relates to generalizability I've seen people use 8 or 32 and then you have Dropout which is just like Dropout for regular deep learning model training and so now what we see here is that even though we have in our whole model we have 60.8 million parameters in T5 small we have just three hundred thousand that we're now going to train oh and I should also say thanks to Grant Bale build in the back of the corner for teaching me about paft super cool technique here where now we have out of our whole big llm we're only going to have half a percent that we actually train and and other than that all the other training arguments stay the same we've just set up our our hugging face model to get the theft model so we instantiate a new hugging face model pass it through this get heft model function with the pesh configurations and we now have this path configured model that we can pass into Pi torch lighting the same as before and it trains very very quickly uses up a little bit more memory obviously it's still not going to perform very well because we didn't train for very long but just so you can see how it works if you haven't used tensorboard before very easy to open up in even Google collab and and when it opens up you can use it to watch your model train over time so you can watch Epoch over epoch how your training loss changes so after this just quick Epoch that we did we have this one particular step but if you were to continue training for more epochs where you could watch it over steps you can see this chart of your training loss your validation loss changing over time which is pretty cool right here in your jupyter notebook nice so we're almost there the last thing here that I want to make sure we get through is multiple GPU training so with multiple GPU training I have this set of steps which is available in my repo so if you go to the NLP with llms repo into the instructions folder there's a readme there that explains how to do this so you can create this can be on your own cloud server but for the purposes of anybody here being able to do it I'm setting it up in paper space which is a relatively expensive way of doing this sign in select core so gradient allows you to use jupyter notebooks core is actually just setting up a whole server we can't do this in Google collab and we can't do it in paper space gradient because you can't do multi-gpu training in a Jupiter notebook so we have to do this at the command line so I have this machine that I already created here I provided you with instructions for creating your own machine to have similar specs so use ml in a box the cheapest multi-gpu machine type will be fine for this problem pick a region that's closest to you for authentication set a password I'll show you why in a second and then in advance options set a static IP wait for the machine to boot I have already done that so that we didn't have to wait for it and then I can go into a command line interface like terminal and you need to run this git clone to clone my repo which will obviously grab all of my code into your server I've already done that and then there's poetry which handles all of our dependencies if you haven't already installed poetry on your machine you can use this line of code to do it then change directory into my repo so when you create your paper space machine they'll provide you with an SSH URL as well as a password and so you can use that to log into the machine I'm also going to I have this other one here foreign for looking at gpus if you haven't run poetry before you'll have to run it then you can CD into the GitHub repo that you cloned and then you just run poetry install to get all of the requirements running and I've already done that so it's just telling me that there's nothing else to do and then you use poetry run to have python run with all the dependencies that we specified in the Poetry file so it's kind of like a Docker file and then I'm running this code which is this fine tune multi-gpu code which we can take a look at here so it's a python script and it has all the same kind of functionality as we had in our single GPU file so the T5 tuner in pytorch lightning the streaming data set and the only difference is are here with our start training where we have things like specifying mixed Precision specifying our GPU and this deep speed stage 3 strategy you can read more about it here but what it's doing is it's sharding our model as well as our data onto however many devices we have which on this machine that I set up in paper space is two gpus and so that's the main part of what's going on here and so now it's running so I have it running here in my cloud instance that I set up and we can go into that cloud instance and run the Nvidia SMI function to see that it's running on both gpus so it's using up 2400 megabytes on both of my gpus and so that's allowing it to speed along very quickly once you have trained your model to your liking so I'm just going to stop it here but after it has completed training there's a couple of lines of code here that I specify in the instructions that allow you to if you uncomment these two lines of code in here these two lines of code in the pi torch lightning um start training method if you uncomment those and then you run this poetry run hugging face CLI login it will push your model up into hugging face and so Sean already ran this model for long enough and he uploaded it to hugging face so now you can go into hugging face and you can type in integers and it will download the model and compute while it's downloading that model I'm going to quickly show how you can actually do that also in a notebook so I've got a notebook here for doing it at the command line so I have this T5 inference notebook which is the very last notebook for today's class I'm not going to go into collab because I still have a few minutes of stuff that I want to get through but we import pytorch and Transformers to now use our very own model that we download so we we download the model from hugging face that he uploaded and then we can use it to make predictions so you can pass in any predictions that you want negative 25 comes out as negative 25. so it worked and even works for these very long numbers into the Millions so there you go that's fine tuning on multiple gpus it should also work in the UI if a hugging face ever finishes loading it here uh just in terms of the rest of the slides here there's stuff on considerations like lightning makes things easy but you still might want to shrink your models through quantization model pruning or distillation those are various options available to you in production you want to be looking out for model drift there's algorithms for detecting that drift and many commercial ml monitoring options that you'd want to consider if you have llms in production there are major challenges today with llms they're large size being one of them and there just being so many options out there for you encoded knowledge still today can be harmful or hallucinated and You're vulnerable to malicious attacks from your users where they can try to figure out what your prompts are and so therefore taking your IP so yeah Transformers and pytorch lighting make model training pre-training fine-tuning storage and deployment easy abundant open source options provide opportunities for you to have proprietary and perform an llms tailored to your needs in this fast moving space there are however reputational and security risks my final section here which we're just going to race through quickly is on getting commercial value for llms so how can we be supporting our other machine learning models with llms we can be using it to label data in really extravagant ways we can quantify our performance like we saw with vikunya we can be using it to compare our model as it trains over time or compare our model against other benchmarks we can use it to augment our training data through things like back translation synonym replacement and wholesale generation of new natural language data so what kinds of tasks should you be thinking about that can be replaced well there's any kinds of repetitive tasks can be replaced whereas creative tasks can be augmented with AI so generating domain-specific copy summarizing documents we talked about lots of these examples you should be using gpt4 or codex yourself for your code and then also you could be thinking about how you could be leveraging gpt4 to be a better holistic thinker so what ml models could you be deploying how could you make your platform stickier what data-driven feature should we build next how can I increase Revenue these are the kinds of questions you can be asking GPT for and providing with a huge amount of context about your specific problems this enables you to become a data product manager without any extra effort all of a sudden you're contributing more value to your company you can be creative about leveraging existing llm apis about fine-tuning your own say single GPU ulms for proprietary task accuracy and efficiency like I showed you how to do in today's training and yeah evaluating the performance of your models using these kinds of tools as well like we saw with vikunya in addition to data scientists you're going to need these kinds of people to have a successful data product and these are the kinds of tools that I recommend processes and specific tools that you can be using to be making the most of your data science team and collaboration together the kinds of things that are going to be happening next with llms are that they're going to be smaller less hallucinating more real-time information more modalities handled better video generation agents like Auto GPT that can perform very complex tasks by assigning subtasks to other gpt4 agents they're going to be domain specific thanks to you working on them and embedded everywhere and due to scaling limits we're going to have to come up with new architectural ideas end so because of all this I a year ago if you had said to me In Our Lifetime well we have capabilities like we have today now with gpt4 if you'd asked me that a year ago if you explain the kinds of things that gpd4 can do I probably would have said to you a year ago I don't know if we will have those capabilities In Our Lifetime and now we have it today so the speed at which this is moving is staggering and it isn't until the last few weeks that I've been believing the hype around things like the singularity but it is possible that we are in our lifetimes going to encounter abundance through interfacing with these Ultra intelligent machines and I encourage you to think positively optimistically obviously there are risks that we need to overcome and we could fill a three-hour lecture with those kinds of risks and how we might want to overcome them but think big about what you can be doing with the skill set that you have as people in this room you are some of the best positioned people on the planet to be thinking about how you can be using these new capabilities these new AI capabilities that are staggering and use them for good and so it is possible that in our lifetimes we will have abundant energy some of the issues around nuclear fusion a lot of the way that these reactors are set up it's plasma touching the walls of the reactor that causes a problem but real-time AI systems that can use lasers to keep the plasma from touching the walls it's AI that it will probably allow us to have Fusion Energy maybe commercial reactors in the next decade abundant nutrition extended lifespans potentially reversing aging education available to people to anyone with a phone all over the world freedom from violence freedom of expression ecological sustainability cultural preservation a sense of exploration in your life and a sense of community and social connection all of these things can be possible in our lifetime to every living being on the planet it's insane in conclusion I introduced you to llms the breadth of their capabilities how to train and deploy them and not just how to get commercial value but how to get value for our whole society from llms further resources for facilitating this Utopia that I've just envisioned for you I have a three-part super data science podcast series on gpt4 episode 666 uh on capability overviews unleashing the Beast here uh 667 on harnessing those abilities commercially 668 is on risks so you can dig into those with an expert a global expert on those Jeremy Harris if you want to understand more about Transformers and GPT structures in general Andre carpathy has an amazing video that's been watched over two million times sinan ostermer whose code I used a huge amount in this training he has great O'Reilly video tutorials live trainings and a forthcoming book some of those chapters are already available in O'Reilly highly highly highly recommends in osmer for learning from all this stuff a huge amount of what I taught you today I learned from him people from hugging face uh created an NLP with Transformers book that is already out and that is excellent has an amazing guide to ml Ops which is key for being able to handle these larger models educating yourself on how you can be better at ml Ops is key to you being valuable as a data scientist as models get bigger and bigger and bigger Vin vashista has an amazing course on commercializing llms if you're interested in that piece of it as well as a forthcoming book on the topic and I'm hiring a director of data science if you want to be working on these problems with llms with Grant and me and Sean who contribute a lot of the code for this and Andrew who's also here at the conference if you want to work with us we're looking for a director of data science who will actually you should already know most of the content in this training because we're looking for somebody that can be taking open source llms fine-tuning them and all the associated ml Ops is ideally what we're looking for you can follow through this link to get the full job description it's about three pages and yeah so thank you very much for your time please do stay in touch thank you\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "24350"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Count tokens from url\n",
    "\n",
    "# https://youtu.be/Ku9PM26Cc2c\n",
    "\n",
    "def count_from_url(url):\n",
    "    transcript = get_transcript(url)\n",
    "    token_count = count_tokens(transcript)\n",
    "    return token_count\n",
    "\n",
    "count_from_url(\"https://youtu.be/Ku9PM26Cc2c\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### End of the Basic Use case"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "langchain",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.17"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}