11-text-learning/vectorize_text.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Starter Code for the Lesson 11: Text Learning Mini-Project"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pickle\n",
    "import re\n",
    "import sys\n",
    "\n",
    "sys.path.append('../utils/')\n",
    "from parse_out_email_text import parse_out_text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Starter code to process the emails from Sara and Chris to extract\n",
    "the features and get the documents ready for classification.\n",
    "\n",
    "The list of all the emails from Sara are in the `from_sara` list\n",
    "likewise for emails from Chris (`from_chris`)\n",
    "\n",
    "The actual documents are in the Enron email dataset, which\n",
    "you downloaded/unpacked in Part 0 of the first mini-project. If you have\n",
    "not obtained the Enron email corpus, run `starter.py` in the `utils` folder.\n",
    "\n",
    "The data is stored in lists and packed away in pickle files at the end."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"from_sara.txt\", \"r\") as from_sara, open(\"from_chris.txt\", \"r\") as from_chris:\n",
    "    from_data = []\n",
    "    word_data = []\n",
    "    \n",
    "    # temp_counter is a way to speed up the development--there are\n",
    "    # thousands of emails from Sara and Chris, so running over all of them\n",
    "    # can take a long time\n",
    "    # temp_counter helps you only look at the first 200 emails in the list so you\n",
    "    # can iterate your modifications quicker\n",
    "    temp_counter = 0\n",
    "\n",
    "    for name, from_person in [(\"sara\", from_sara), (\"chris\", from_chris)]:\n",
    "        for path in from_person:\n",
    "            # only look at first 200 emails when developing\n",
    "            # once everything is working, remove this line to run over full dataset\n",
    "            temp_counter += 1\n",
    "            if temp_counter < 200:\n",
    "                path = os.path.join('..', path[:-1])\n",
    "                print(path)\n",
    "                with open(path, \"r\") as email:\n",
    "\n",
    "                    # use parse_out_text to extract the text from the opened email\n",
    "\n",
    "                    # use str.replace() to remove any instances of the words\n",
    "                    # [\"sara\", \"shackleton\", \"chris\", \"germani\"]\n",
    "\n",
    "                    # append the text to word_data\n",
    "\n",
    "                    # append a 0 to from_data if email is from Sara, and 1 if email is from Chris\n",
    "\n",
    "\n",
    "print(\"Emails processed\")\n",
    "\n",
    "with open(\"your_word_data.pkl\", \"w\") as f:\n",
    "    pickle.dump(word_data, f)\n",
    "\n",
    "with open(\"your_email_authors.pkl\", \"w\") as f:\n",
    "    pickle.dump(from_data, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 4, do TfIdf vectorization here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ↓↓↓ Your Code Goes Here ↓↓↓"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}