Logistic Regression considering L1 and L2 norms.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "_uuid": "5097943f58f3195124bba1c97ec8dbe124ecdc17"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "5c44a58af55b399e077e2898ad72a0bc1be0ead0"
   },
   "source": [
    "<h2>Logistic Regression with Probabilistic Approach </h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "72b742e25454cf61879ccffe38948b8f8c8979e5"
   },
   "source": [
    "In theory, a Logistic regression takes input and returns an output of probability, a value between 0 and 1. How does a Logistic Regression do that? With the help of a function called a *logistic function* or most commonly known as a *sigmoid*. This sigmoid function is reponsible for *predicting* or classifying a given input.\n",
    "Logistic function or sigmoid is defined as:\n",
    "![](https://imgur.com/Bw5gMJX.jpg)\n",
    "Where:\n",
    "* *e* = Euler's number which is **2.71828**.\n",
    "* *x0* = the value of the sigmoid's midpoint on the x-axis.\n",
    "* *L* = the maximum value.\n",
    "* *k* = steepness of the curve."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "f53a66e11f60f4effcf15ac1a57dedfdeb30f6b6"
   },
   "source": [
    "For Logistic Regression however here is the definition of the logistic function:<br>\n",
    "![](https://imgur.com/903IYoN.jpg)\n",
    "Where:\n",
    "* Θ = is the weight = w"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "_uuid": "098959420fb8067564d2d19d50c4374ce9b38836"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Marks_1</th>\n",
       "      <th>Marks_2</th>\n",
       "      <th>Result</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>34.623660</td>\n",
       "      <td>78.024693</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>30.286711</td>\n",
       "      <td>43.894998</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>35.847409</td>\n",
       "      <td>72.902198</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>60.182599</td>\n",
       "      <td>86.308552</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>79.032736</td>\n",
       "      <td>75.344376</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Marks_1    Marks_2  Result\n",
       "0  34.623660  78.024693       0\n",
       "1  30.286711  43.894998       0\n",
       "2  35.847409  72.902198       0\n",
       "3  60.182599  86.308552       1\n",
       "4  79.032736  75.344376       1"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#importing the data\n",
    "\n",
    "\n",
    "data = pd.read_excel(\"dataset.xlsx\")\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "dfccfb4c98df84d92709f1e8b12a574ef9bdd2a8"
   },
   "source": [
    "<h4>Loss minimizing / J(w) </h4><br>\n",
    "Weights (represented by theta in our notation) is a vital part of Logistic Regression and other Machine Learning algorithms and we want to find the best values for them. To start we pick random values and we need a way to measure how well the algorithm performs using those random weights. That measure is computed using the loss function. <br><br>\n",
    "The loss function is defined as:\n",
    "\n",
    "![](https://imgur.com/riDHhZS.jpg)\n",
    "\n",
    "Where:\n",
    "* m = the number of samples\n",
    "* y = the target class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "_uuid": "7605b4f1b5ec77fb1efc592295339da27f4fe01e",
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Columns and data types\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>datatype</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Marks_1</th>\n",
       "      <td>float64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Marks_2</th>\n",
       "      <td>float64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Result</th>\n",
       "      <td>int64</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        datatype\n",
       "Marks_1  float64\n",
       "Marks_2  float64\n",
       "Result     int64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#knowing my data\n",
    "\n",
    "print(\"Columns and data types\")\n",
    "pd.DataFrame(data.dtypes).rename(columns = {0:'datatype'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**1 denotes pass and 0 denotes fail**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "_uuid": "b4143de23fd832463c63513fbbc2fa0ad6b8d02a"
   },
   "outputs": [],
   "source": [
    "#defining my x and y values\n",
    "\n",
    "X = np.array(data.iloc[:,:2].values)\n",
    "y = np.array(data.iloc[:,-1].values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "ccf9e386d02e0e7df69d01d04f4a51b4b49f6865"
   },
   "source": [
    "The goal is to **minimize the loss**  by means of increasing or decreasing the weights, which is commonly called fitting. Which weights should be bigger and which should be smaller? This can be decided by a function called **Gradient descent**. The Gradient descent is just the derivative of the loss function with respect to its weights."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "a939565adc6ea8906c14677203129d90ca8e234a"
   },
   "source": [
    "![](https://imgur.com/rBVzJbt.jpg)\n",
    "The weights are updated by substracting the derivative (gradient descent) times the learning rate, as defined below:\n",
    "![](https://imgur.com/TAIpnwI.jpg)\n",
    "Where:\n",
    "* α = learning rate (usually 0.1) But since in my question it is specified that my alpha should be 0.001 I'll keep my α = 0.001"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**1. L1 is Lasso**\n",
    "\n",
    "It adds an aditional term with the cost function (Loss function)\n",
    "\n",
    "the term is (lambda/(2*n))* Σ |weight|\n",
    "\n",
    "lambda = factor that measures the extent of regularization\n",
    "n = no. of features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "_uuid": "ae89d7d2a0018f52d75a3e555fbeff4128518933"
   },
   "outputs": [],
   "source": [
    "# my fit function\n",
    "\n",
    "def fit(X, y):\n",
    "    lr = 0.001 \n",
    "    gd = 100000\n",
    "    n = X.shape[0]\n",
    "    \n",
    "    #initial value should be zero\n",
    "    weight = np.zeros(X.shape[1])\n",
    "    b = 0\n",
    "    lamb = 100\n",
    "    for _ in range(gd):\n",
    "        func = (np.dot(X, weight) + b)+(lamb/(2*n)*sum(weight))\n",
    "\n",
    "        sigmoid = 1 / (1 + np.exp(-func)) #for sigmoid\n",
    "\n",
    "        wdash = np.dot(X.T, (sigmoid - y))/n\n",
    "        bdash = np.sum(sigmoid - y)/n\n",
    "        #for the next iteration\n",
    "        weight = weight -(lr * wdash)\n",
    "        b = b - (lr * bdash)\n",
    "    return (weight,b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#my predict function\n",
    "\n",
    "def predict():\n",
    "    arr = [[None]*2]*1\n",
    "    m1 = int(input(\"Enter your marks 1: \"))\n",
    "    arr[0][0] = m1\n",
    "    m2 =int( input(\"Enter your marks 2: \"))\n",
    "    arr[0][1] = m2\n",
    "    weight,b=fit(X, y)\n",
    "    fun = np.dot(arr,weight) + b\n",
    "    y_pred = 1 / (1 + np.exp(-fun))\n",
    "    \n",
    "    for i in y_pred:\n",
    "        if i > 0.5:\n",
    "            y_test = \"Passed\" \n",
    "        else:\n",
    "            y_test = \"Failed\"\n",
    "            \n",
    "    return y_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Enter your marks 1: 90\n",
      "Enter your marks 2: 99\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Passed'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Enter your marks 1: 89\n",
      "Enter your marks 2: 10\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Failed'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**2. L2 is Ridge**\n",
    "\n",
    "It adds an aditional squared term with the cost function (Loss function)\n",
    "\n",
    "the term is (lambda/(2*n))* Σ |weight^2|\n",
    "\n",
    "lambda = factor that measures the extent of regularization\n",
    "n = no. of features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "_uuid": "ae89d7d2a0018f52d75a3e555fbeff4128518933"
   },
   "outputs": [],
   "source": [
    "# my fit function\n",
    "\n",
    "def fit(X, y):\n",
    "    lr = 0.001 \n",
    "    gd = 100000\n",
    "    n = X.shape[0]\n",
    "    \n",
    "    #initial value should be zero\n",
    "    weight = np.zeros(X.shape[1])\n",
    "    b = 0\n",
    "    lamb = 100\n",
    "    for _ in range(gd):\n",
    "        func = (np.dot(X, weight) + b)+(lamb/(2*n)*sum(weight**2))\n",
    "\n",
    "        sigmoid = 1 / (1 + np.exp(-func)) #for sigmoid\n",
    "\n",
    "        wdash = np.dot(X.T, (sigmoid - y))/n\n",
    "        bdash = np.sum(sigmoid - y)/n\n",
    "        #for the next iteration\n",
    "        weight = weight -(lr * wdash)\n",
    "        b = b - (lr * bdash)\n",
    "    return (weight,b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "#my predict function\n",
    "\n",
    "def predict():\n",
    "    arr = [[None]*2]*1\n",
    "    m1 = int(input(\"Enter your marks 1: \"))\n",
    "    arr[0][0] = m1\n",
    "    m2 =int( input(\"Enter your marks 2: \"))\n",
    "    arr[0][1] = m2\n",
    "    weight,b=fit(X, y)\n",
    "    fun = np.dot(arr,weight) + b\n",
    "    y_pred = 1 / (1 + np.exp(-fun))\n",
    "    \n",
    "    for i in y_pred:\n",
    "        if i > 0.5:\n",
    "            y_test = \"Passed\" \n",
    "        else:\n",
    "            y_test = \"Failed\"\n",
    "            \n",
    "    return y_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Enter your marks 1: 33\n",
      "Enter your marks 2: 99\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Passed'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Enter your marks 1: 90\n",
      "Enter your marks 2: 22\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Passed'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}