2_Python_for_Data_Science.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python for Data Science"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### About this notebook\n",
    "\n",
    "This part of the tutorial focuses on how to leverage vast repository of\n",
    "Python modules to efficiently wrangle, analyze and visualize small and big data. We will mostly focus on NumPy, Pandas and Matplotlib, but also touch on ML libraries and Seaborn.\n",
    "\n",
    "This notebook will be organized a bit differently than the previous one. Here, I will first introduce the modules showing some essentials and then, we will move on to tackling problems and learning by doing. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "### Modules"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Modules are what, in my opinion, make Python so good. They provide the needed\n",
    "extensions to the already neat language. Within modules you can find solutions\n",
    " to many of your problems and tasks.\n",
    "\n",
    "Let me introduce few crucial modules, some of them come with Pyton installation, some you have to install on your own.\n",
    "\n",
    "Standard Library (i.e. modules that come with Python):\n",
    "* [os](https://docs.python.org/3/library/os.html) - provides multiple OS interfaces \n",
    "* [re](https://docs.python.org/3/library/re.html?highlight=re#module-re) - module for regular expressions\n",
    "* [math](https://docs.python.org/3/library/math.html) - math operations extension (includes rounding up, calculating factorials and much more)\n",
    "* [statistcs](https://docs.python.org/3/library/statistics.html) - some statistical functions - among others facilitates mode, medians etc. calculations\n",
    "* [random](https://docs.python.org/3/library/random.html?highlight=random#module-random) - *generates pseudo-random numbers*\n",
    "* [collections](https://docs.python.org/3/library/collections.html) - extensions of the basics collections\n",
    "* [itertools](https://docs.python.org/3/library/itertools.html) - great module for various iteration and efficient looping\n",
    "* [multiprocessing](https://docs.python.org/3/library/multiprocessing.html) - module allowing parallel computing\n",
    "* [logging](https://docs.python.org/3/library/logging.html) - module allowing\n",
    " to generate nice and comprehensive messages\n",
    "\n",
    "You can find much more described [here](https://docs.python.org/3/library/).\n",
    "\n",
    "Other crucial modules, especially for Data Science:\n",
    "\n",
    "* [NumPy](https://www.numpy.org/) - *the fundamental package for scientific computing with Python*\n",
    "* [Pandas](https://pandas.pydata.org/) - *library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming*\n",
    "* [SciPy](https://www.scipy.org/) -  *Python-based ecosystem of open-source software for mathematics, science, and engineering*\n",
    "* [Matplotlib](https://matplotlib.org/) - *2D plotting library which produces publication quality figures*\n",
    "* [Tensorflow](https://www.tensorflow.org/) - *An end-to-end open source machine learning platform*\n",
    "* [Keras](https://keras.io/) - *The Python Deep Learning library*\n",
    "* [scikit-learn](https://scikit-learn.org/stable/) - *Machine Learning in Python*\n",
    "* [Scrapy](https://scrapy.org/) - *An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.*\n",
    "* [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/) - module for xml and html parsing\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can import whole modules by using `import <module_name>` statement. We usually place them at the very beginning of the script in order to specify what modules the following code relies on. We can access the functions can be accessed by preceding the function name with the module name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "print(random.randint(1, 19))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we only need some functions from a module, we can load only those by using `from <module_name> import <function>` statement as follows. The functions then can be access without referring to the module name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from math import pi, ceil\n",
    "print(pi)\n",
    "print(ceil(pi))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In that way we can also load all functions from within the module by using `from <module_name> import *` statement. In this case, we also don't have to proceed the functions names with the module name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from math import *\n",
    "print(factorial(3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And lastly, we can create an abbreviation for a given module and only proceed the functions with this abbreviation by using `import <module_name> as <module_abbrev>`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import statistics as st\n",
    "print(st.median([1, 3, 5, 7]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "## NumPy\n",
    "\n",
    "Without further ado, we will strat with `numpy` - module design for handling scientific computing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's load the packages\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Array are the primary type in _numpy_. We can create an array by changing a list of elements into an numpy array by calling `np.array()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "a_list = [1, 2, 3, 4, 5, 6, 7, 8, 9]\n",
    "A = np.array(a_list)\n",
    "a_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can create arrays with different types of elements. `Numpy` will try to match the type according to certain hierarchy, i.e. if we have characters and numbers, the type will default to characters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "b_list = [1, 2, 3, 4, 5, 6, '7', 8, 9]\n",
    "B = np.array(b_list)\n",
    "B"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can change the type of an array. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "B = B.astype('float64')\n",
    "B"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Arrays have shapes. We are more familiar with matrices 1-, 2- and 3-dimensional. However, they can easily be N-dimensional. This comes in handy especially in Machine Learning. We will not explore tensors in detail, as this is the beginners guide to `numpy`, but the principles shown below extend to more complex objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(A.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our array `A` is of a special shape, i.e. it does not have a specific shape. It looks suspiciously similar to the other matrix with the shape 1 x 9."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A_reshaped = A.reshape(1, 9)\n",
    "print(A_reshaped.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A_reshaped"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, if it looks like a dog, walks like a dog and barks like a dog, then it should be a dog, yes? No, at least not in this case.  Shapes are really important for the arrays. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "A_reshaped == A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.array_equal(A, A_reshaped)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can create the matrices from list, and it is quite useful to be able to reshape them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A = A.reshape(3, 3)\n",
    "A"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can have a 3D arrays, we can easily just reshape our other matrices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A = A.reshape(3, 3, 1)\n",
    "A"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are special kind of matrices we can create that come in handy when we need to initialize an array that we will later fill with our data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An empty array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.empty([2, 2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Array filled with zeros..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.zeros([2, 3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and similarly, an array with ones..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.ones([3, 5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and other values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.full([2, 4], 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An array with the sequence of numbers and so on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "C = np.arange(0, 20, 2)\n",
    "C = C.reshape((2,5))\n",
    "C"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(C.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Numpy allows us to easily perform matrix operations. For example, we might be\n",
    " interested in the sum of all of the array elements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "M = np.arange(1, 10).reshape(3, 3)\n",
    "print(M)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sum(M)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or, we want to get the column sums."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sum(M, axis = 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can multiply array by a scalar:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "M * 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "add a vector to each row:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(M)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v = np.array([[1, 2, 3]])\n",
    "print(M + v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(M)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec = np.array([[1, 2, 3]]).transpose()\n",
    "print(M + vec)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Solving linear equations\n",
    "\n",
    "*Note:* Don't worry if you are not familiar with linear algebra. I am using this as an example because it highlights the advantages of numpy and why it is so widely used in ML. I will walk you through all steps, and hopefully it will be quite straight forward.\n",
    "\n",
    "Suppose we want to solve the following system of equations.\n",
    "\n",
    "$$ \\begin{cases} x_1 + x_2 = 1 \\\\ 4x_1 + 6x_2 +x_3 = 2 \\\\ -2x_1 + 2x_2 = 3 \\end{cases} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first start by transforming the equations into matrix form. \n",
    "\n",
    "$$ A x = b $$\n",
    "\n",
    "$$ A = \\begin{bmatrix} 1 & 1 & 0 \\\\ 4 & 6 & 1 \\\\ -2 & 2 & 0 \\\\ \\end{bmatrix}, x = \\begin{bmatrix} x_1 & x_2 & x_3 \\\\ \\end{bmatrix}^T, b = \\begin{bmatrix} 1 & 2 & 3 \\\\ \\end{bmatrix}^T $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use Gaussian elimination, and solve it on paper with the help of numpy. \n",
    "\n",
    "1. We start with finding the Upper triangular matrix -  $U$.\n",
    "\n",
    "![](./figures/upper_triangular.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. Now that we have $U$, and we know that $U = EA$ we need to find the elimination matrix - $E$.\n",
    "\n",
    "To do that we need to write the transformation from above in matrix form.\n",
    "\n",
    "![](./figures/elimination.png)\n",
    "\n",
    "And knowing that $E = E_2E_1$, we can get the elimination matrix with the help of numpy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "E1 = np.array([[1, 0, 0], [-4, 1, 0], [2, 0, 1]])\n",
    "E2 = np.array([[1, 0, 0], [0, 1, 0], [0, -2, 1]])\n",
    "E = np.matmul(E2, E1)\n",
    "E"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, that we have the elimination matrix we can also check if everything is ok by confirming that $U$ matches the one we computed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A = np.array([[1, 1, 0], [4, 6, 1], [-2, 2, 0]])\n",
    "U = np.matmul(E, A) # matrix multiplication\n",
    "U"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to solve $Ax = b$. In order to do so, we can multiply both sides of the equation by our elimination matrix: $EAx = Eb$. Knowing that $U = EA$, we get the following:\n",
    "\n",
    "$$Ux = Eb$$\n",
    "\n",
    "We then need the $Eb$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "b = np.array([1, 2, 3]).reshape(3, 1)\n",
    "Eb = np.matmul(E, b)\n",
    "Eb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And with that, we only need to solve this one last trivial set of equations. \n",
    "\n",
    "$$ Ux = \\begin{bmatrix} 1 & 1 & 0 \\\\ 0 & 2 & 1 \\\\ 0 & 0 & -2 \\end{bmatrix}  \\begin{bmatrix} x_1 \\\\ x_2 \\\\ x_3 \\end{bmatrix} =  \\begin{bmatrix} 1 \\\\ -2 \\\\ 9 \\end{bmatrix}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ \\begin{cases} x_1 + x_2 = 1 \\\\ 2x_2 + x_3 = -2 \\\\ -2x_3 = 9 \\end{cases} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x3 = 9 / -2\n",
    "x2 = (-2 - x3)/2\n",
    "x1 = 1 - x2\n",
    "\n",
    "x1, x2, x3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or, we can use numpy to help us out entirely :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.linalg.solve(A, b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Linear algebra functions within numpy cover a lot of necessary computation from the set of linear algebra. Including functions like singular value decomposition, or eigen decomposition. Using the functions from that set one can for example hand code PCA. The full list and documentation can be accessed [here](https://numpy.org/doc/stable/reference/routines.linalg.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**  \n",
    "Given:\n",
    "$$ A = \\begin{bmatrix} 1 & 1 & 0 \\\\ 4 & 6 & 1 \\\\ -2 & 2 & 0 \\\\ \\end{bmatrix}, B = \\begin{bmatrix} -1 & -2 & -3 \\\\ -4 & -5 & -6 \\\\ \\end{bmatrix}, C = \\begin{bmatrix} 5 & 10 & 15 \\\\ 20 & 25 & 30 \\\\ 35 & 40 & 45 \\\\ \\end{bmatrix} $$    \n",
    "\n",
    "* create those arrays; "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here :) \n",
    "# A =\n",
    "# B =\n",
    "# C ="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* add A and C;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here :) \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* multiply B by A; "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here :) \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* multiply 3rd column of A by second row of B"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here :) \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Excercise**  \n",
    "This one is a bit more challenging.\n",
    "\n",
    "Linear regression model is given by the formula:\n",
    "$$y = X\\beta + \\epsilon$$\n",
    "\n",
    "A closed form solution is of a following form. \n",
    "\n",
    "$$ \\hat\\beta = (X^TX)^{-1}X^Ty$$\n",
    "\n",
    "Given $X$ and $y$, calculate $\\hat\\beta$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create random data from a linear model of a following formula:\n",
    "$$ y = 0.5 + 2x_1 - x_2 $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n = 5\n",
    "# generate the variables\n",
    "ones = np.ones(n)\n",
    "x_1 = np.random.randn(n)\n",
    "x_2 = np.random.randn(n)\n",
    "X = np.column_stack((ones, x_1, x_2))\n",
    "# generate the y from a linear model\n",
    "y = 0.5 * ones + 2 * x_1 - x_2\n",
    "# and add noise\n",
    "y = y + np.random.normal(0, 0.1, n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here :)\n",
    "beta_hat = "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.matmul(X, beta_hat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "## Pandas\n",
    "\n",
    "Working with data most often than not means working with tables of all sorts. The way to go about it in Python is with `pandas`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Pandas` is a module that allows for data frame manipulation. It resembles the `R data.frame` & `dplyr` / `data.table` handling."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will get to explore only the top of the iceberg of the capabilities of the `pandas` module. More about the `numpy`, `pandas` and data science with Python can be found [here - in the Python Data Science Handbook](https://jakevdp.github.io/PythonDataScienceHandbook/index.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To explore the capabilities of the library we will use the penguins data set. It carries information about 3 species of penguins. Data was collected and made available by [Dr. Kristen Gorman](https://www.uaf.edu/cfos/people/faculty/detail/kristen-gorman.php) and the [Palmer Station, Antarctica LTER](https://pal.lternet.edu/), a member of the [Long Term Ecological Research Network](https://lternet.edu/).\n",
    "\n",
    "<a href=\"https://github.com/allisonhorst/palmerpenguins\"><img src=\"figures/lter_penguins.png\" alt=\"Penguin species\" style=\"width: 600px;\"/></a>\n",
    "\n",
    "Artwork by [@allison_horst](https://twitter.com/allison_horst)\n",
    "\n",
    "Among others, it contains information about the penguins' culmens, their body mass and others."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://github.com/allisonhorst/palmerpenguins\"><img src=\"figures/culmen_depth.png\" alt=\"Penguin culmen\" style=\"width: 600px;\"/></a>\n",
    "\n",
    "Artwork by [@allison_horst](https://twitter.com/allison_horst)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "# penguins_raw.csv as downloaded from:\n",
    "# https://raw.githubusercontent.com/allisonhorst/palmerpenguins/9e1a7937e257ade8e4dc1764028e55cd7efe2cf5/data-raw/penguins_raw.csv\n",
    "penguins_df = pd.read_csv(\"./data/penguins_raw.csv\")\n",
    "penguins_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to know more about the dataset we can call describe function that\n",
    "will tell us more about what we have in the data frame. It's much easier to\n",
    "go through a summary than through the whole data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`describe()` is one of the very interesting function that allows us to investigate the content of the table. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "penguins_df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On default though, describe function will summarize the numerical columns. If we want to see the content of the categorical, or character columns we need to specify an option `include = 'all'`, or select the specific subset of columns.\n",
    "\n",
    "> For object data (e.g. strings or timestamps), the result’s index will include count, unique, top, and freq. The top is the most common value. The freq is the most common value’s frequency. Timestamps also include the first and last items."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "penguins_df.describe(exclude = [np.number])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can access row names of pandas data frame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "penguins_df.index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and its columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "penguins_df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "penguins_df[['Culmen Length (mm)', 'Culmen Depth (mm)', 'Flipper Length (mm)']].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "penguins_df.sort_values(by = 'Body Mass (g)')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's imagine we want to calculate mean of all parameters within the Species. We can do that by calling `groupby` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "penguins_df.groupby(['Species']).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Excercise**  \n",
    "Let's answer few questions using `pandas`. \n",
    "\n",
    "* are all species equally represented in the data frame? \n",
    "\n",
    "*Hint:* we can count values in a column with `value_counts()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* are all penguins present on all islands (if we take the lack of samples in a data as a sign for lack of specimens)? \n",
    "\n",
    "*Hint*: method `size()` will tell us about the size of each subgroup."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* calculate means of all numeric variables for Adelie Penguins with respect to the island they live on;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "## Matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Matplotlib` is a Python module to create plots. By series of commands we create a figure, add labels and modify it. Below, we create a simple plot with a line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams[\"figure.facecolor\"] = \"w\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(range(1, 10))\n",
    "plt.ylabel('Y axis label')\n",
    "plt.xlabel('X axis label')\n",
    "plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In general, the `plot()` function has 3 arguments: `x`, `y` and `shape` for every series. \n",
    "\n",
    "**Line types**:\n",
    "\n",
    "character | description\n",
    ":---: | ---\n",
    "'-' | solid line style\n",
    "'--' | dashed line style\n",
    "'-.' |\tdash-dot line style\n",
    "':'\t| dotted line style\n",
    "'.' | point marker\n",
    "','\t| pixel marker\n",
    "'o'\t| circle marker\n",
    "'v'\t| triangle_down marker\n",
    "'*'\t| star marker\n",
    "'h'\t| hexagon1 marker\n",
    "'x'\t| x marker\n",
    "'D'\t| diamond marker\n",
    "'\\|'\t| vline marker\n",
    "'_'\t| hline marker\n",
    "\n",
    "**Colors**:\n",
    "\n",
    "character | color\n",
    ":---: | ---\n",
    "‘b’ | blue\n",
    "‘g’\t| green\n",
    "‘r’\t| red\n",
    "‘c’\t| cyan\n",
    "‘m’\t| magenta\n",
    "‘y’\t| yellow\n",
    "‘k’\t| black\n",
    "‘w’\t| white\n",
    "\n",
    "More examples of matplotlib plots [here](https://matplotlib.org/tutorials/introductory/pyplot.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can add multiple series to a plot by adding the series line by line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.arange(0, 10, 1)\n",
    "y = np.arange(0, 20, 2)\n",
    "\n",
    "plt.plot(x, y, 'r-.')\n",
    "plt.plot(x, x*y, 'g:')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:**  \n",
    "Given a vector t (`t = np.arange(0, 12, 0.5)` )  create a plot\n",
    " with 3 series: $y_1 = 10t$, $y_2 = t^2$ and $y_3 = -10$ times the reminder of $t$ divided by 3. With 3 different styles: green triangles, red dashed line and blue diamonds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here :) \n",
    "t = np.arange(0, 12, 0.5)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's do something a bit more complex. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "groups = penguins_df.groupby('Species')\n",
    "for name, group in groups:\n",
    "    plt.scatter(x = group['Body Mass (g)'], y = group['Flipper Length (mm)'], label = name)\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or we can create a factor column which we can directly use to color the point by."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "penguins_df['Species_factor'] = pd.factorize(penguins_df['Species'])[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.scatter('Culmen Length (mm)', 'Culmen Depth (mm)', c = 'Species_factor', data = penguins_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additionally pandas data frames have some visualisation methods built-in, they are described in more detail [here](https://pandas.pydata.org/pandas-docs/stable/user_guide/visualization.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "penguins_df['body_mass'] = penguins_df.groupby('Species')['Body Mass (g)'].apply(lambda x: (x - min(x)) / (max(x) - min(x)))\n",
    "penguins_df.boxplot(column = ['body_mass'],\n",
    "                    by = 'Species', figsize = (11, 5), notch = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are some other libtraries extending the plotting capabilities of Python. One worth mentioning is `seaborn` module. There are a lot of things simplified and easily accessible with the library.  \n",
    "It does allow to make visually nice plots, and is worth exploring when working with Python. More about [seaborn here](https://seaborn.pydata.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "sns.set_style(\"white\")\n",
    "sns.pairplot(penguins_df, hue = 'Species')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It still, in my opinion, is not close enough to what `ggplot2` does and allows to do in `R` though."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from plotnine import *\n",
    "\n",
    "(\n",
    "    ggplot(penguins_df, aes('Body Mass (g)', 'Flipper Length (mm)', color = 'Species'))\n",
    "    + geom_point()\n",
    "    + geom_smooth(se = False)\n",
    "    + theme_light()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And a nice example of Simpson's paradox."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    ggplot(penguins_df, aes('Body Mass (g)', 'Culmen Depth (mm)'))\n",
    "    + geom_point(aes(color = 'Species'))\n",
    "    + geom_smooth(se = False, method = 'lm')\n",
    "    + geom_smooth(aes(color = 'Species'), se = False, method = 'lm')\n",
    "    + theme_xkcd()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Excercise**  \n",
    "\n",
    "* plot the number of each specie occurence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Plot the culmen dimensions with respect to the specie."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "## SciPy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`SciPy`  is a Python-based ecosystem of open-source software for mathematics, science, and engineering. For example, it provides useful statistical functions. One of them is MannWhitney U test - a non parametric test allowing to check the difference between the two distributions, series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import mannwhitneyu\n",
    "A = np.random.normal(0, 3, 20)\n",
    "B = np.random.normal(-2, 5, 30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's visualise what we have in those two sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.boxplot((A, B))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "mannwhitneyu(A, B)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We saw that one of the species is heavier than others. Let's remind ourselves the distribution of the weights. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize = (12, 8))\n",
    "ax = sns.stripplot(x = 'Species', y = 'Body Mass (g)', data = penguins_df, color=\".25\")\n",
    "ax = sns.boxplot(x = 'Species', y = 'Body Mass (g)', data = penguins_df, palette = \"PRGn\", fliersize = 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With `scipy.stats` we can check if the change is significant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from itertools import combinations\n",
    "\n",
    "species = penguins_df['Species'].unique()\n",
    "for (specie1, specie2) in combinations(species, 2):\n",
    "    a = penguins_df[penguins_df['Species'] == specie1]['Body Mass (g)']\n",
    "    b = penguins_df[penguins_df['Species'] == specie2]['Body Mass (g)']\n",
    "    stat, pval = mannwhitneyu(a, b)\n",
    "    print((specie1, specie2, pval))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "## Scikit learn\n",
    "\n",
    "Module allowing Machine Learning in Python. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this example, let's recreate the data from the linear model example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n = 5\n",
    "# generate the variables\n",
    "ones = np.ones(n)\n",
    "x_1 = np.random.randn(n)\n",
    "x_2 = np.random.randn(n)\n",
    "X = np.column_stack((ones, x_1, x_2))\n",
    "# generate the y from a linear model\n",
    "y = 0.5 * ones + 2 * x_1 - x_2\n",
    "# and add noise\n",
    "y = y + np.random.normal(0, 0.1, n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import linear_model\n",
    "reg = linear_model.LinearRegression(fit_intercept = False)\n",
    "\n",
    "reg.fit(X, y)\n",
    "reg.coef_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "beta_hat = np.matmul(np.matmul(np.linalg.inv(np.matmul(X.T, X)), X.T), y)\n",
    "beta_hat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "penguins_df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import *\n",
    "\n",
    "df_ = penguins_df[['Species', 'Culmen Length (mm)', 'Culmen Depth (mm)', 'Flipper Length (mm)', 'Body Mass (g)']].dropna()\n",
    "pca = PCA(n_components=2)\n",
    "pca.fit(df_.drop('Species', axis = 1))\n",
    "projected = pca.fit_transform(df_.drop('Species', axis = 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pca_plot = pd.concat((pd.DataFrame(projected), df_['Species']), axis = 1)\n",
    "pca_plot.columns = ['PC1', 'PC2', 'Species'] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pca_groups = pca_plot.groupby('Species')\n",
    "\n",
    "for specie, group in pca_groups:\n",
    "    plt.scatter(x = group['PC1'], y = group['PC2'], label = specie)\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "ggplot(pca_plot, aes('PC1', 'PC2', color = 'Species'))\n",
    "+ geom_point() \n",
    "+ theme_xkcd()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "**Excercises**  \n",
    "\n",
    "* Build a model that will predict the size of a flipper based on a penguin weight. Plot the prediction againts the true values.\n",
    "\n",
    "*Hint*: Good place to start would be a [linear model](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* See if we can cluster the data into 3 clusters and if they match the species.\n",
    "\n",
    "*Hint*: You might want to use [kmeans clustering](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html) for starters. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here :)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}