1_pipelines.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b8df0bbd",
   "metadata": {},
   "source": [
    "# 1. Pipelines\n",
    "In this exercise, we will explore the capabilities of transformer-based models for natural language processing (NLP) using the Hugging Face (HF) `transformers` library. We will use the `sentence-transformers` package to extract features from text data and the `transformers` library to perform sentiment analysis and text generation tasks.\n",
    "\n",
    "By the end of this exercise, you will have learned how to:\n",
    "- Extract features from text data using transformer-based models\n",
    "- Perform sentiment analysis on text data\n",
    "- Generate text using transformer-based models\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "979416b3c30fb86b",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Using Notebook Environments \n",
    "1. To run a cell, press `shift + enter`. The notebook will execute the code in the cell and move to the next cell. If the cell contains a markdown cell (text only), it will render the markdown and move to the next cell.\n",
    "2. Since cells can be executed in any order and variables can be over-written, you may at some point feel that you have lost track of the state of your notebook. If this is the case, you can always restart the kernel by clicking Runtime in the menu bar (if you're using Colab) and selecting `Restart runtime`. This will clear all variables and outputs.\n",
    "3. The final variable in a cell will be printed on the screen. If you want to print multiple variables, use the `print()` function as usual.\n",
    "\n",
    "Notebook environments support code cells and markdown (text) cells. For the purposes of this workshop, markdown cells are used to provide high-level explanations of the code. More specific details are provided in the code cells themselves in the form of comments (lines beginning with `#`)"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Environment Setup"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "c616368742ccde73"
  },
  {
   "cell_type": "code",
   "id": "initial_id",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "import sys\n",
    "if 'google.colab' in sys.modules:  # If in Google Colab environment\n",
    "    \n",
    "    # Installing requisite packages\n",
    "    !pip install transformers sentence-transformers &> /dev/null"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "290c24c2ed829f80",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "We begin by loading the requisite packages. For those coming from R, packages in Python are sometimes given shorter names for use in the code via the `import <name> as <nickname>` syntax (e.g. `import pandas as pd`). These are usually standardized nicknames. We here make use three packages:\n",
    "\n",
    "1. `pandas`: A very popular package for reading and manipulating data in python.\n",
    "2. `sentence_transformers`: A package for extracting features from text data using transformer-based models.\n",
    "3. `transformers`: A HF package for loading and manipulating transformer-based models."
   ]
  },
  {
   "cell_type": "code",
   "id": "267fad62d6f82855",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "import pandas as pd\n",
    "from sentence_transformers import SentenceTransformer\n",
    "from transformers import pipeline"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "f6bf097acf12ea11",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Feature Extraction\n",
    "\n",
    "The following begins by extracting features (or embeddings) from the text data, which are numerical representations of the meaning of text, using the `sentence-transformers` package. To start, it uses three sentences that the code cell places in a list of strings. This list is provided as input to the model. \n",
    "\n",
    "The code makes use of the `all-MiniLM-L6-v2` model, which is a small and efficient embedding model, to extract features from the sentences. The model will encode the sentences into 384-dimensional vector representations. The cell will then print the features as a pandas dataframe for easy viewing. \n",
    "\n",
    "Run the cell below. "
   ]
  },
  {
   "cell_type": "code",
   "id": "98ca015d4695f403",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Define sentences\n",
    "sentences = [\n",
    "    \"I feel great this morning\",\n",
    "    \"I am feeling very good today\",\n",
    "    \"I am feeling terrible\"\n",
    "]\n",
    "\n",
    "# Load the pre-trained model\n",
    "model = SentenceTransformer('all-MiniLM-L6-v2')\n",
    "\n",
    "# Extract features\n",
    "features = model.encode(sentences)\n",
    "\n",
    "# Print the features as a pandas dataframe\n",
    "pd.DataFrame(features, index=sentences)"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "80664214bd695bf0",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "**TASK 1**: Have a scroll through the features printed by the cell. Can you see that the features of the first two sentences are more similar to each other (i.e., have similar numerical values) than they are to the third sentence? Why do you think this is the case?\n",
    "\n",
    "**TASK 2**: Try to add another sentence to the `sentences` list defined above. Use one of the existing sentences but replace one or two words with a synonym. For instance, you could change \"I feel *great* this morning\" to \"I feel *fantastic* this morning\". Then rerun the cell. What do you notice about the features of this new sentence compared to the original?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efc8d5804ede6511",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Text Generation\n",
    "This section uses the `transformers` text generation pipeline. The cell below begins by loading the pipeline with the `gpt2` model, the smaller and open grandparent of ChatGPT.\n",
    "\n",
    "The cell then defines a prompt. The prompt is a starting point that the model uses to generate text. GPT-2 will use it to generate text that is likely to follow the prompt. We set the `max_length` parameter to 100 to limit the length of the generated text to 100 tokens.\n",
    "\n",
    "Run the cell below."
   ]
  },
  {
   "cell_type": "code",
   "id": "cc96eb23af195ac2",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "pipe = pipeline('text-generation', model='gpt2')\n",
    "\n",
    "prompt = \"\"\"\n",
    "    Once upon a time in a land far far away, there was a young prince named John. He was known for his bravery and courage. \n",
    "    One day, he decided to go on an adventure to explore the unknown lands.\n",
    "\"\"\"\n",
    "\n",
    "# Generate text based on the prompt\n",
    "output = pipe(prompt, max_length=100)\n",
    "\n",
    "# Print the generated text\n",
    "print(output[0]['generated_text'])"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "d4a24c50147f2cf7",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "**TASK 1**: Please enter a new prompt in the variable `prompt` that you wish the model to continue generating text from. Feel free to play around with the `max_length` parameter to see how it affects the generated text.\n",
    "\n",
    "**TASK 2**: Try replacing `'gpt2'` above with `'gpt2-medium'` or another text generation model on the [HF model hub](https://huggingface.co/models) to see how the generated text changes. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4830e24a5d45430d",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Sentiment Analysis \n",
    "In addition to feature extraction, Hugging Face's `transformers` library provides a high-level API for a variety of other tasks. These tasks can be viewed in the lef-hand panel under \"Natural Language Processing\" on the [HF model hub](https://huggingface.co/models). Many of these tasks include models that have been fine-tuned on specific datasets to perform well on the task at hand. As an example of such a task, we will now use the sentiment analysis pipeline to predict the sentiment of a few sentences.\n",
    "\n",
    "The cell below will now load the `transformers` sentiment analysis pipeline to predict the sentiment of sentences for the same sentences as before.\n",
    "\n",
    "Run the cell below."
   ]
  },
  {
   "cell_type": "code",
   "id": "612fa14857dc0b74",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Define the sentences\n",
    "sentences = [\n",
    "    \"I feel great this morning\",\n",
    "    \"I am feeling very good today\",\n",
    "    \"I am feeling terrible\"\n",
    "]\n",
    "\n",
    "# Load sentiment analysis pipeline\n",
    "pipe = pipeline('sentiment-analysis', model='distilbert-base-uncased-finetuned-sst-2-english')\n",
    "\n",
    "# Predict sentiment of the sentences\n",
    "sentiments = pipe(sentences)\n",
    "\n",
    "# Print the predicted sentiments as a pandas dataframe\n",
    "pd.DataFrame(sentiments, index=sentences)"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "id": "81690eb2b6590715",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "As you can see, not only does the model predict the sentiment of the sentences (`'label'`), but it also provides a confidence score for each prediction (`'score'`). "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}