5.1 week-5-lasso-assignment-1-blank.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regression Week 5: Feature Selection and LASSO (Interpretation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, you will use LASSO to select features, building on a pre-implemented solver for LASSO (using GraphLab Create, though you can use other solvers). You will:\n",
    "* Run LASSO with different L1 penalties.\n",
    "* Choose best L1 penalty using a validation set.\n",
    "* Choose best L1 penalty using a validation set, with additional constraint on the size of subset.\n",
    "\n",
    "In the second notebook, you will implement your own LASSO solver, using coordinate descent. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fire up Graphlab Create"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import graphlab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load in house sales data\n",
    "\n",
    "Dataset is from house sales in King County, the region where the city of Seattle, WA is located."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This non-commercial license of GraphLab Create for academic use is assigned to msn4eg@gmail.com and will expire on April 03, 2018.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] graphlab.cython.cy_server: GraphLab Create v2.1 started. Logging: /tmp/graphlab_server_1499537946.log\n"
     ]
    }
   ],
   "source": [
    "sales = graphlab.SFrame('kc_house_data.gl/')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create new features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As in Week 2, we consider features that are some transformations of inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from math import log, sqrt\n",
    "sales['sqft_living_sqrt'] = sales['sqft_living'].apply(sqrt)\n",
    "sales['sqft_lot_sqrt'] = sales['sqft_lot'].apply(sqrt)\n",
    "sales['bedrooms_square'] = sales['bedrooms']*sales['bedrooms']\n",
    "\n",
    "# In the dataset, 'floors' was defined with type string, \n",
    "# so we'll convert them to float, before creating a new feature.\n",
    "sales['floors'] = sales['floors'].astype(float) \n",
    "sales['floors_square'] = sales['floors']*sales['floors']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Squaring bedrooms will increase the separation between not many bedrooms (e.g. 1) and lots of bedrooms (e.g. 4) since 1^2 = 1 but 4^2 = 16. Consequently this variable will mostly affect houses with many bedrooms.\n",
    "* On the other hand, taking square root of sqft_living will decrease the separation between big house and small house. The owner may not be exactly twice as happy for getting a house that is twice as big."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learn regression weights with L1 penalty"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us fit a model with all the features available, plus the features we just created above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "all_features = ['bedrooms', 'bedrooms_square',\n",
    "            'bathrooms',\n",
    "            'sqft_living', 'sqft_living_sqrt',\n",
    "            'sqft_lot', 'sqft_lot_sqrt',\n",
    "            'floors', 'floors_square',\n",
    "            'waterfront', 'view', 'condition', 'grade',\n",
    "            'sqft_above',\n",
    "            'sqft_basement',\n",
    "            'yr_built', 'yr_renovated']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Applying L1 penalty requires adding an extra parameter (`l1_penalty`) to the linear regression call in GraphLab Create. (Other tools may have separate implementations of LASSO.)  Note that it's important to set `l2_penalty=0` to ensure we don't introduce an additional L2 penalty."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre>Linear regression:</pre>"
      ],
      "text/plain": [
       "Linear regression:"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>--------------------------------------------------------</pre>"
      ],
      "text/plain": [
       "--------------------------------------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of examples          : 21613</pre>"
      ],
      "text/plain": [
       "Number of examples          : 21613"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of features          : 17</pre>"
      ],
      "text/plain": [
       "Number of features          : 17"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of unpacked features : 17</pre>"
      ],
      "text/plain": [
       "Number of unpacked features : 17"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of coefficients    : 18</pre>"
      ],
      "text/plain": [
       "Number of coefficients    : 18"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Starting Accelerated Gradient (FISTA)</pre>"
      ],
      "text/plain": [
       "Starting Accelerated Gradient (FISTA)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>--------------------------------------------------------</pre>"
      ],
      "text/plain": [
       "--------------------------------------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-----------+----------+-----------+--------------+--------------------+---------------+</pre>"
      ],
      "text/plain": [
       "+-----------+----------+-----------+--------------+--------------------+---------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| Iteration | Passes   | Step size | Elapsed Time | Training-max_error | Training-rmse |</pre>"
      ],
      "text/plain": [
       "| Iteration | Passes   | Step size | Elapsed Time | Training-max_error | Training-rmse |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-----------+----------+-----------+--------------+--------------------+---------------+</pre>"
      ],
      "text/plain": [
       "+-----------+----------+-----------+--------------+--------------------+---------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Tuning step size. First iteration could take longer than subsequent iterations.</pre>"
      ],
      "text/plain": [
       "Tuning step size. First iteration could take longer than subsequent iterations."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| 1         | 2        | 0.000002  | 1.868202     | 6962915.603493     | 426631.749026 |</pre>"
      ],
      "text/plain": [
       "| 1         | 2        | 0.000002  | 1.868202     | 6962915.603493     | 426631.749026 |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| 2         | 3        | 0.000002  | 1.984112     | 6843144.200219     | 392488.929838 |</pre>"
      ],
      "text/plain": [
       "| 2         | 3        | 0.000002  | 1.984112     | 6843144.200219     | 392488.929838 |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| 3         | 4        | 0.000002  | 2.068429     | 6831900.032123     | 385340.166783 |</pre>"
      ],
      "text/plain": [
       "| 3         | 4        | 0.000002  | 2.068429     | 6831900.032123     | 385340.166783 |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| 4         | 5        | 0.000002  | 2.156081     | 6847166.848958     | 384842.383767 |</pre>"
      ],
      "text/plain": [
       "| 4         | 5        | 0.000002  | 2.156081     | 6847166.848958     | 384842.383767 |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| 5         | 6        | 0.000002  | 2.235351     | 6869667.895833     | 385998.458623 |</pre>"
      ],
      "text/plain": [
       "| 5         | 6        | 0.000002  | 2.235351     | 6869667.895833     | 385998.458623 |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| 6         | 7        | 0.000002  | 2.312476     | 6847177.773672     | 380824.455891 |</pre>"
      ],
      "text/plain": [
       "| 6         | 7        | 0.000002  | 2.312476     | 6847177.773672     | 380824.455891 |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-----------+----------+-----------+--------------+--------------------+---------------+</pre>"
      ],
      "text/plain": [
       "+-----------+----------+-----------+--------------+--------------------+---------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model_all = graphlab.linear_regression.create(sales, target='price', features=all_features,\n",
    "                                              validation_set=None, \n",
    "                                              l2_penalty=0., l1_penalty=1e10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find what features had non-zero weight."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre>TERMINATED: Iteration limit reached.</pre>"
      ],
      "text/plain": [
       "TERMINATED: Iteration limit reached."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>This model may not be optimal. To improve it, consider increasing `max_iterations`.</pre>"
      ],
      "text/plain": [
       "This model may not be optimal. To improve it, consider increasing `max_iterations`."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of nonzeros = 6\n",
      "+------------------+-------+---------------+--------+\n",
      "|       name       | index |     value     | stderr |\n",
      "+------------------+-------+---------------+--------+\n",
      "|   (intercept)    |  None |  274873.05595 |  None  |\n",
      "|     bedrooms     |  None |      0.0      |  None  |\n",
      "| bedrooms_square  |  None |      0.0      |  None  |\n",
      "|    bathrooms     |  None | 8468.53108691 |  None  |\n",
      "|   sqft_living    |  None | 24.4207209824 |  None  |\n",
      "| sqft_living_sqrt |  None | 350.060553386 |  None  |\n",
      "|     sqft_lot     |  None |      0.0      |  None  |\n",
      "|  sqft_lot_sqrt   |  None |      0.0      |  None  |\n",
      "|      floors      |  None |      0.0      |  None  |\n",
      "|  floors_square   |  None |      0.0      |  None  |\n",
      "|    waterfront    |  None |      0.0      |  None  |\n",
      "|       view       |  None |      0.0      |  None  |\n",
      "|    condition     |  None |      0.0      |  None  |\n",
      "|      grade       |  None | 842.068034898 |  None  |\n",
      "|    sqft_above    |  None | 20.0247224171 |  None  |\n",
      "|  sqft_basement   |  None |      0.0      |  None  |\n",
      "|     yr_built     |  None |      0.0      |  None  |\n",
      "|   yr_renovated   |  None |      0.0      |  None  |\n",
      "+------------------+-------+---------------+--------+\n",
      "[18 rows x 4 columns]\n",
      "\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "print 'number of nonzeros = %d' % (model_all.coefficients['value']).nnz()\n",
    "print(model_all.coefficients.print_rows(num_rows=model_all.num_coefficients))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that a majority of the weights have been set to zero. So by setting an L1 penalty that's large enough, we are performing a subset selection. \n",
    "\n",
    "***QUIZ QUESTION***:\n",
    "According to this list of weights, which of the features have been chosen? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Selecting an L1 penalty"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To find a good L1 penalty, we will explore multiple values using a validation set. Let us do three way split into train, validation, and test sets:\n",
    "* Split our sales data into 2 sets: training and test\n",
    "* Further split our training data into two sets: train, validation\n",
    "\n",
    "Be *very* careful that you use seed = 1 to ensure you get the same answer!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "(training_and_validation, testing) = sales.random_split(.9,seed=1) # initial train/test split\n",
    "(training, validation) = training_and_validation.random_split(0.5, seed=1) # split training into train and validate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we write a loop that does the following:\n",
    "* For `l1_penalty` in [10^1, 10^1.5, 10^2, 10^2.5, ..., 10^7] (to get this in Python, type `np.logspace(1, 7, num=13)`.)\n",
    "    * Fit a regression model with a given `l1_penalty` on TRAIN data. Specify `l1_penalty=l1_penalty` and `l2_penalty=0.` in the parameter list.\n",
    "    * Compute the RSS on VALIDATION data (here you will want to use `.predict()`) for that `l1_penalty`\n",
    "* Report which `l1_penalty` produced the lowest RSS on validation data.\n",
    "\n",
    "When you call `linear_regression.create()` make sure you set `validation_set = None`.\n",
    "\n",
    "Note: you can turn off the print out of `linear_regression.create()` with `verbose = False`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10.0, 254847.40799944056)\n",
      "(31.622776601683793, 254847.40804422513)\n",
      "(100.0, 254847.40818584664)\n",
      "(316.22776601683796, 254847.4086336949)\n",
      "(1000.0, 254847.41004992122)\n",
      "(3162.2776601683795, 254847.41452849933)\n",
      "(10000.0, 254847.42869177682)\n",
      "(31622.776601683792, 254847.473487671)\n",
      "(100000.0, 254847.6152214934)\n",
      "(316227.76601683791, 254848.06419097554)\n",
      "(1000000.0, 254849.49163438112)\n",
      "(3162277.6601683795, 254854.08237889022)\n",
      "(10000000.0, 254871.31974790498)\n",
      "('minimum values = ', 10.0, 254847.40799944056)\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "rmse_l1 = []\n",
    "rmse_min = 0.0\n",
    "l1_penalty_min = 0\n",
    "for l1_penalty in np.logspace(1,7,num=13):\n",
    "    model = graphlab.linear_regression.create(training,target='price',features=all_features,l2_penalty=0,\n",
    "                                             l1_penalty=l1_penalty,validation_set = None , verbose = False)\n",
    "    out_predicted = model.predict(validation)\n",
    "    rmse = graphlab.evaluation.rmse(validation['price'],out_predicted)\n",
    "    print (l1_penalty,rmse)\n",
    "    rmse_l1.append((l1_penalty,rmse)) \n",
    "    if (rmse < rmse_min) | (rmse_min == 0.0):\n",
    "        rmse_min = rmse\n",
    "        l1_penalty_min = l1_penalty\n",
    "#print(rmse_l1)\n",
    "print (\"minimum values = \",l1_penalty_min,rmse_min)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*** QUIZ QUESTION. *** What was the best value for the `l1_penalty`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = graphlab.linear_regression.create(training,target='price',features=all_features,l2_penalty=0,\n",
    "                                             l1_penalty=10,validation_set = None , verbose = False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***QUIZ QUESTION***\n",
    "Also, using this value of L1 penalty, how many nonzero weights do you have?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "18\n"
     ]
    }
   ],
   "source": [
    "print model.coefficients['value'].nnz()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Limit the number of nonzero weights\n",
    "\n",
    "What if we absolutely wanted to limit ourselves to, say, 7 features? This may be important if we want to derive \"a rule of thumb\" --- an interpretable model that has only a few features in them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, you are going to implement a simple, two phase procedure to achive this goal:\n",
    "1. Explore a large range of `l1_penalty` values to find a narrow region of `l1_penalty` values where models are likely to have the desired number of non-zero weights.\n",
    "2. Further explore the narrow region you found to find a good value for `l1_penalty` that achieves the desired sparsity.  Here, we will again use a validation set to choose the best value for `l1_penalty`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "max_nonzeros = 7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring the larger range of values to find a narrow range with the desired sparsity\n",
    "\n",
    "Let's define a wide range of possible `l1_penalty_values`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "l1_penalty_values = np.logspace(8, 10, num=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, implement a loop that search through this space of possible `l1_penalty` values:\n",
    "\n",
    "* For `l1_penalty` in `np.logspace(8, 10, num=20)`:\n",
    "    * Fit a regression model with a given `l1_penalty` on TRAIN data. Specify `l1_penalty=l1_penalty` and `l2_penalty=0.` in the parameter list. When you call `linear_regression.create()` make sure you set `validation_set = None`\n",
    "    * Extract the weights of the model and count the number of nonzeros. Save the number of nonzeros to a list.\n",
    "        * *Hint: `model['coefficients']['value']` gives you an SArray with the parameters you learned.  If you call the method `.nnz()` on it, you will find the number of non-zero parameters!* "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(18, 100000000.0, 255198.70457004078)\n",
      "(18, 127427498.57031322, 255344.6375801111)\n",
      "(18, 162377673.91887242, 255540.91911694003)\n",
      "(18, 206913808.11147901, 255839.95441467175)\n",
      "(17, 263665089.87303555, 256304.06284271376)\n",
      "(17, 335981828.62837881, 256976.98487516315)\n",
      "(17, 428133239.8719396, 257983.3165641675)\n",
      "(17, 545559478.11685145, 259532.02373040994)\n",
      "(17, 695192796.17755914, 261916.2583395135)\n",
      "(16, 885866790.41008317, 265126.0132047146)\n",
      "(15, 1128837891.6846883, 269741.4272179156)\n",
      "(15, 1438449888.2876658, 276731.35182413476)\n",
      "(13, 1832980710.8324375, 287458.50712784554)\n",
      "(12, 2335721469.0901213, 300322.983529889)\n",
      "(10, 2976351441.6313133, 316789.4358882695)\n",
      "(6, 3792690190.7322536, 335089.79074163764)\n",
      "(5, 4832930238.5717525, 359456.338656892)\n",
      "(3, 6158482110.6602545, 379024.7078186479)\n",
      "(1, 7847599703.5146227, 357410.2084871671)\n",
      "(1, 10000000000.0, 357172.3640306664)\n"
     ]
    }
   ],
   "source": [
    "rmse_min = 0.0\n",
    "rmse_l1 = []\n",
    "rmse_min = 0.0\n",
    "l1_penalty_min = 0\n",
    "for l1_penalty in l1_penalty_values:\n",
    "    model = graphlab.linear_regression.create(training,target='price',features=all_features,l2_penalty=0,\n",
    "                                             l1_penalty=l1_penalty,validation_set = None , verbose = False)\n",
    "    #out_predicted = model.predict(validation)\n",
    "    #rmse = graphlab.evaluation.rmse(validation['price'],out_predicted)\n",
    "    #print (l1_penalty,rmse)\n",
    "    #rmse_l1.append((l1_penalty,rmse)) \n",
    "    #if(model.coefficients['value'].nnz() <= max_nonzeros):\n",
    "    out_predicted = model.predict(validation)\n",
    "    rmse = graphlab.evaluation.rmse(validation['price'],out_predicted)\n",
    "    print (model.coefficients['value'].nnz(), l1_penalty,rmse)\n",
    "    rmse_l1.append((l1_penalty,rmse)) \n",
    "    if (rmse < rmse_min) | (rmse_min == 0.0):\n",
    "        rmse_min = rmse\n",
    "        l1_penalty_min = l1_penalty"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Out of this large range, we want to find the two ends of our desired narrow range of `l1_penalty`.  At one end, we will have `l1_penalty` values that have too few non-zeros, and at the other end, we will have an `l1_penalty` that has too many non-zeros.  \n",
    "\n",
    "More formally, find:\n",
    "* The largest `l1_penalty` that has more non-zeros than `max_nonzeros` (if we pick a penalty smaller than this value, we will definitely have too many non-zero weights)\n",
    "    * Store this value in the variable `l1_penalty_min` (we will use it later)\n",
    "* The smallest `l1_penalty` that has fewer non-zeros than `max_nonzeros` (if we pick a penalty larger than this value, we will definitely have too few non-zero weights)\n",
    "    * Store this value in the variable `l1_penalty_max` (we will use it later)\n",
    "\n",
    "\n",
    "*Hint: there are many ways to do this, e.g.:*\n",
    "* Programmatically within the loop above\n",
    "* Creating a list with the number of non-zeros for each value of `l1_penalty` and inspecting it to find the appropriate boundaries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "l1_penalty_min = 2976351441.6313133\n",
    "l1_penalty_max = 3792690190.7322536"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***QUIZ QUESTION.*** What values did you find for `l1_penalty_min` and `l1_penalty_max`, respectively? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring the narrow range of values to find the solution with the right number of non-zeros that has lowest RSS on the validation set \n",
    "\n",
    "We will now explore the narrow region of `l1_penalty` values we found:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "l1_penalty_values = np.linspace(l1_penalty_min,l1_penalty_max,20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* For `l1_penalty` in `np.linspace(l1_penalty_min,l1_penalty_max,20)`:\n",
    "    * Fit a regression model with a given `l1_penalty` on TRAIN data. Specify `l1_penalty=l1_penalty` and `l2_penalty=0.` in the parameter list. When you call `linear_regression.create()` make sure you set `validation_set = None`\n",
    "    * Measure the RSS of the learned model on the VALIDATION set\n",
    "\n",
    "Find the model that the lowest RSS on the VALIDATION set and has sparsity *equal* to `max_nonzeros`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10, 2976351441.6313133, 316789.4358882695)\n",
      "(10, 3019316638.9524155, 317949.3600337632)\n",
      "(10, 3062281836.2735176, 319117.2905718614)\n",
      "(10, 3105247033.5946198, 320438.2608718298)\n",
      "(10, 3148212230.9157219, 321965.81111239316)\n",
      "(10, 3191177428.236824, 323524.5005267909)\n",
      "(10, 3234142625.5579262, 325096.09462018014)\n",
      "(10, 3277107822.8790283, 326680.3976697857)\n",
      "(8, 3320073020.2001305, 327690.55694960424)\n",
      "(8, 3363038217.5212326, 328313.571670634)\n",
      "(8, 3406003414.8423347, 329052.86828532314)\n",
      "(7, 3448968612.1634369, 329635.90940812987)\n",
      "+------------------+-------+---------------+--------+\n",
      "|       name       | index |     value     | stderr |\n",
      "+------------------+-------+---------------+--------+\n",
      "|   (intercept)    |  None | 222253.192544 |  None  |\n",
      "|     bedrooms     |  None | 661.722717782 |  None  |\n",
      "| bedrooms_square  |  None |      0.0      |  None  |\n",
      "|    bathrooms     |  None | 15873.9572593 |  None  |\n",
      "|   sqft_living    |  None | 32.4102214513 |  None  |\n",
      "| sqft_living_sqrt |  None | 690.114773313 |  None  |\n",
      "|     sqft_lot     |  None |      0.0      |  None  |\n",
      "|  sqft_lot_sqrt   |  None |      0.0      |  None  |\n",
      "|      floors      |  None |      0.0      |  None  |\n",
      "|  floors_square   |  None |      0.0      |  None  |\n",
      "|    waterfront    |  None |      0.0      |  None  |\n",
      "|       view       |  None |      0.0      |  None  |\n",
      "|    condition     |  None |      0.0      |  None  |\n",
      "|      grade       |  None | 2899.42026975 |  None  |\n",
      "|    sqft_above    |  None | 30.0115753022 |  None  |\n",
      "|  sqft_basement   |  None |      0.0      |  None  |\n",
      "|     yr_built     |  None |      0.0      |  None  |\n",
      "|   yr_renovated   |  None |      0.0      |  None  |\n",
      "+------------------+-------+---------------+--------+\n",
      "[18 rows x 4 columns]\n",
      "\n",
      "None\n",
      "(7, 3491933809.484539, 330298.0405644453)\n",
      "(7, 3534899006.8056412, 331058.39540312486)\n",
      "(7, 3577864204.1267428, 331811.0158823089)\n",
      "(6, 3620829401.447845, 332577.749512044)\n",
      "(6, 3663794598.7689471, 333163.40624314034)\n",
      "(6, 3706759796.0900493, 333792.1240913379)\n",
      "(6, 3749724993.4111514, 334433.3170418734)\n",
      "(6, 3792690190.7322536, 335089.79074163764)\n"
     ]
    }
   ],
   "source": [
    "rmse_min = 0.0\n",
    "rmse_l1 = []\n",
    "rmse_min = 0.0\n",
    "l1_penalty_min = 0\n",
    "for l1_penalty in l1_penalty_values:\n",
    "    model = graphlab.linear_regression.create(training,target='price',features=all_features,l2_penalty=0,\n",
    "                                             l1_penalty=l1_penalty,validation_set = None , verbose = False)\n",
    "    #out_predicted = model.predict(validation)\n",
    "    #rmse = graphlab.evaluation.rmse(validation['price'],out_predicted)\n",
    "    #print (l1_penalty,rmse)\n",
    "    #rmse_l1.append((l1_penalty,rmse)) \n",
    "    #if(model.coefficients['value'].nnz() <= max_nonzeros):\n",
    "    out_predicted = model.predict(validation)\n",
    "    rmse = graphlab.evaluation.rmse(validation['price'],out_predicted)\n",
    "    print (model.coefficients['value'].nnz(), l1_penalty,rmse)\n",
    "    rmse_l1.append((l1_penalty,rmse)) \n",
    "    if(model.coefficients['value'].nnz() == max_nonzeros):\n",
    "        if (rmse < rmse_min) | (rmse_min == 0.0):\n",
    "            rmse_min = rmse\n",
    "            l1_penalty_min = l1_penalty\n",
    "            print(model.coefficients.print_rows(num_rows=model.num_coefficients))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***QUIZ QUESTIONS***\n",
    "1. What value of `l1_penalty` in our narrow range has the lowest RSS on the VALIDATION set and has sparsity *equal* to `max_nonzeros`?\n",
    "2. What features in this model have non-zero coefficients?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}