DSiP-1-Introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "input_collapsed": true,
    "internals": {
     "slide_type": "subslide"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Python for Data Science\n",
    "\n",
    "by [__Michael Granitzer__ (michael.granitzer@uni-passau.de)]( http://www.mendeley.com/profiles/michael-granitzer/)\n",
    " and [__Konstantin Ziegler__ (konstantin.ziegler@uni-passau.de)](http://zieglerk.net)\n",
    "\n",
    "based on [Scientific Python Lectures](http://github.com/jrjohansson/scientific-python-lectures) by [**J.R. Johansson**](http://dml.riken.jp/~rob/)\n",
    "\n",
    "__License__\n",
    "\n",
    "This work is licensed under a [Creative Commons Attribution 3.0 Unported License](http://creativecommons.org/licenses/by/3.0/)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": false,
    "internals": {
     "slide_type": "subslide"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Data Science"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_number": 3
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<div class=\"alert alert-success\">\n",
    "**Data Science** is the study of the generalizable extraction of knowledge from data; knowledge used for answering (scientific) questions through gathering, storing, processing and analysing large amounts of data. (Dhar, 2013)\n",
    "</div>\n",
    "\n",
    "Data Science is neither about a particular set of tools like NoSQL Databases, Hadoop etc., nor about storing Google-scale amount of data, it is simply about the **Scientific questions that can be answered with (big) data**. (Leek, 2013)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 4,
     "slide_helper": "subslide_end"
    },
    "slide_helper": "subslide_end",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Data Science is also an umbrella term covering a number of different disciplines\n",
    "- Statistics and statistical inferencig\n",
    "- Stochastics as mathematical modelling approach\n",
    "- Databases as methods/tools for data storage\n",
    "- Parallel Computing as methods for scaling up using multiple machines\n",
    "- Machine Learning/Data Mining for providing algorithms\n",
    "- Social Network Analysis as application area\n",
    "- Human Computing Interfaces and Visual Analytics\n",
    "- Engineering\n",
    "- ...."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 4,
     "slide_helper": "subslide_end",
     "slide_type": "subslide"
    },
    "slide_helper": "subslide_end",
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "** Let other people talk... **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 4,
     "slide_helper": "subslide_end",
     "slide_type": "subslide"
    },
    "slide_helper": "subslide_end",
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkz\nODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2MBERISGBUYLxoaL2NCOEJjY2NjY2NjY2Nj\nY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY//AABEIAWgB4AMBIgACEQED\nEQH/xAAbAAACAwEBAQAAAAAAAAAAAAAAAwECBAUGB//EAEAQAAIBAgQDBQUHAwMCBwEAAAABAgMR\nBBIhMQVBURMiYZHRBjJScZIUFRZCU4GhIzNDcrHBYoIkNERjZHThJf/EABgBAQEBAQEAAAAAAAAA\nAAAAAAABAgME/8QAIBEBAQEAAwEBAQADAQAAAAAAAAERAhIxIUEDE1Fh0f/aAAwDAQACEQMRAD8A\n+fgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\nAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\nAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAbfuuv8VPzfoW+6a/x0/N+hNhr\nAB0FwfENe/S836FlwTEvadLzfoO0HNA6n3Div1KPm/QX90Yi/v0vN+g7Qc8Do/c2I+Ol5v0IfCMQ\nvz0vN+g7Qc8DqU+A4qptUo/u36DPw3jP1aH1P0J2i444Ha/DON/Vw/1P0Ifs1jF/lofU/QdoZXGA\n7H4cxn6tD6n6B+HMZ+pQ+p+g7QxxwOx+HMZ+rQ+p+gfhzGO/9Shor+8/QdoZXHA6n3Fiv1KPm/Qv\nL2dxcX/cofU/QvaGOQB1fuDF/qUfN+gfh/FfqUfN+hO0THKA634fxf6lH6n6E/h3F/qUPqfoO0XH\nIA7H4bxn6tD6n6B+G8Z+rQ+p+g7QyuOB2Pw5jP1KH1P0I/DuLt/cofU/QdoZXIA7FP2cxlRXVWh+\n8n6EP2dxalbtKH1P0HaGOQB1/wAO4v8AUofU/QmPs3jJbVKH1P0HaGOOB2vwxjf1cP8AVL0JXsvj\nX/lw/wBUvQdoZXEA7f4Xxv6uH+qXoVfs1jE/7lD6n6DtDHGA7H4cxlv7tD6n6B+G8Ze3aUPqfoO0\nRxwO6vZTHP8AzYb6pegP2UxsVeVfDL/vl6Dvx/2uVwgOx+HcQv8A1WE+uXoWh7MYyp7lfDP5TfoO\n0Mrigd38KY6393D/AFS9CPwtjv1MP9UvQd+Jjhgdl+zWNX+Sh9T9Cq9ncW/8tD6n6DtEcgDs/hnG\nvarQf/c/QPwzjf1aH1P0HaLjjAdh+zmMX+Sh9T9CPw9i/wBSh9T9B2iY5AHW/D2L/UofU/QPw/i/\n1KPm/QdoOSB1vw/i/wBSj9T9CPuDFL/JR+p+g7QcoDrT9nsXB2dSh9T9CF7P4tq/aUfqfoO0XHKA\n6r4Bil/ko/U/Qj7hxX6lHzfoO0RywOp9xYr9Sj5v0IfA8Smlnpa+L9C9oOYB1VwDFP8AyUfqfoH3\nBi/jo+b9CdouOgksxZEuNnqraBC1zmwm9mOpNaCF3pWQxJxlcinzk1JrlYzc/wBxreifUVzBV+ZS\nSu9y7Woqo7S0A24bS92aVNIwUGlGTetuQ5d5NpfsSxuVsVRIHOL5pfMxRbcbtWsXsmt9CYuntrk0\nFzDVpSzJwdiyTW71Lia13LLn4IxqQzNJwqSTfedgsocbSd0NqO+V+Bjje+smbJf26fyBPFSLEvYC\nCUixS7JTCr3LXFcy9tbADepE33Ae5E9gG4bu002JWruOpvLh34rqLitCp+BjKSKDaJAzkWiQy0UR\nVZbCubY6Qp2jBtuyCFzlGnTcpu0VuzlV+KylLLh4JR+J7ieJ4z7TLJTn/Ti9lzM9FXtmllXiakWR\nrp4qq5tSqTu+dy06lTLaU3l8OZOHoKpmypP5GeVbsp2lF/NhvEKVSSbipW+QU6s6U815RfUdKTlG\n6k7PxM15xlq80fEqOlh+K1YtKaUo9TqUcbRrLSST6NnnacVD3tm9GXU+yltbxM2Jj0FRc7XMtXu9\n62hmw/ELd2p5jq01Km3EMUyjVutxzkrHOTa8DZRWajmb5iwlEneVkQwgrTkmWaAqkVki/Ip4AVtJ\nPwK3WePzGchE1/Vjq1ryKUzEO87kQegYrSptYpGTAY0Q0Rcl9AKtalZ6JMu9f2Iks0dQgjNJalnK\nwuEbqz5FXJx0ZMXWeT7pWOhM+SBaJG3Nq4fRVWsjtcT4TCjhqE6d3KortPkczg//AJhX6nscTS7T\nD4aWVuMY6u2x5v6crLf+f+unGfHl58DxUcJ23ZPLa9//AMM/3Jj44b7T2N6Vs18yvbrY9rjq3Y06\nk40ZVIyp6yWyVjLhsPTwueFOjJqeHcpVm3Z+HQ68bdxLHlPurHSw8K6oPs6llF5leV3ZaXuJx3Bs\nfg6brYjDuNO9syknb52Z38R/e9nv+3/eJGK1wntAm9O3h/ujoy5XAeG08diJwrTnGEIOby7u1jfW\n4OqlKjX4fKpVp1U+7OylGxf2XSjiqt1ddhK666ovQ4rOeOwmWnCjQpSsoR8dGyfG8c7AcNqY6dCS\nUlRqTyZ/lq9Dp0ODuMOJ4fsVKpHL2MpNXs29fDY0YmMMJxPh+Dpu0KdRzfzlIKtOcIcbzxazOLV+\nauyjiz4bililhnSl2z1USJcKxcMTDDypPtZ6xV1r+56SOuLgk/6s+HpQ+ZXBwlSnwmlWTVaKqXT3\nSsxg87iuGYjBRjLEUssZOyd01f8AY18O4U8dh6so37kbxSt3pdDRiFfgMl/8t/7Mjg8HKjj6cE3O\nVGyS3ZP1RiODyr8OwU8Lh4uplk6jTSb28+Zio8PxOKVqFFyyK0tUrP8Ac7VFOL4KtrZy8lCWEqON\nCrWlTxTk4UnZ35MuDzFSEoVHCcXGUXZp8jqvhNJY/B4ftJ5a9NTk9Lp2e3kZeJVZV8fVqTpOjJtX\ng3toegWJcOI8PodnTanRTzuPeWj2ZJEeehwrE16tZYam5wpzcLuSWxFDh2Krqbp0nam7SbaVn01O\nvhcLThV+1KjKvUli3HRu1NX30Jxy/wD5/Ev/ALPoMVm4Nw2c5SrYignRdN5XLr1sYeH4aGKx1OhU\nlKMZ3V477HooQlLiEK0E3h3hLRktl4HmsNV7DGUau2Wab+Vxfge+FyXD6uJzPPCt2eXk1e1/Mvie\nD1ni6tHBwlVjSsnJyS1avY77nS+8/u+3clB1n/qz3ObhqVOtOtj+yliKksVljFN2gvi0Lia5dTCV\n6WDcpUnGMZ5JPmmUrYephqnZ1o5Z2va56VRjiOI4vCT2k4VF+1rnAx1b7Rja1XlKTt8uRmzFZrDa\nMSqQ2CsZVZovFFUtRliBUzkcaxXZ0OzhvLQ68+b5LU81xJupVlf8pqDFQpTqyUaavJnpOHezVSol\nPEO3ga/ZTh0I4dYmrG8pbX5HpUuhOXL7kduPHPtYMLwnDUFpBP5icbwLD4hO0UmdZEmW3z/G8Ong\nKmV3cW7GWdNJabpeZ7Dj+HcsM6ijex46tUtKMl+/kb43WOURF6OK1T5FXUy92W/JinVXapRVoydm\nujCScnKG65M1jmmVXS9tDdha2aCjLzMNKndZZGjDpU24T0tsKma3d3lI2UJqVNRSskYId6K6GvDb\nNEc8yry7s00MavruUmu6Mou6IqsloKekmOnzQp7XAqhFTSUbdR/JFV/cVt7lRTESWfX92iIWtfkR\nif78lysitPTbRPcL+mbuyBsESAbgtGWSKtWZBC7svBlJal5rNEou8VGScrzJcvEVHWVy6szTDRha\n3Y1E0dh8ZnKkoKpK21rnBaBOzMcv5zldanLHTr8YxE4Ol2k1B/lUtDOuI4zsexWIqKna2XM7GJay\nGLdFnGTxNtMqYitlg+1nenpDvPu/LoTRq15OcZVZtVH305PvfPqLna1mXw0HLU0R0KVV0tac5Rur\nd120L0KlJVoSq3cFJOSW7RnyNLcFFow6NvEMX9qxtSuk0pPu33S5CZYqtPNmqzebSV5PVIQ46bkO\nL5MDR9oqOUZdpLNFWi82q+RLxNZ1VVdWbqfHmd/MyptaFXVkuQNbI1ZtZHOThfNlvpfqRTqzpzzU\n5yhLrF2YqNX+lKdlbZFe15208AutSr1e7apO8Pd7z0+QVMTWpVHKnVnCT3cZNNiIVYyklZq/gTWq\nQc5LNZ+IX8EpynLNKTlJ7tu9xnbVc8J9rPNFWTzO6XgZ4yvqhiaDJlLEV6UpOnWqQze9lk1cntaj\njKLnJxk7yV92L2JiwpqxFeFPs41qkYfCpO3kIk7jGUYFVWr9t2vbT7RaZszvb5l8PWrUZ2o1p082\n+WTVxdrDaEU6ifQEjoYfGww9WvUnnlWnTcYPxfNmJEVFepclNAiyWoyO5SKLx3IGWLvYrbUmWkSD\nPXmoUpS5RV/I8xNynh5S5s9BxCSWHlFv3otHnu0jOlKEHzsjcWPb8FTWEpR5ZUdaJyMDjcHhcLTp\n1a8IyUUtWbaOPw9f+1VjL5HJ6GwLipVUot8kcXifE8Q04YW8f+pbl06u3Wpxq03GVrPqeL4rwpYS\npKUWp0nvZ3ylY/aq9X/xGMnG/JSuzRKnQUXGVacpeLHhjz1SjLtKiivFBQlmafRHSjCPea1UTn1I\n9lOoobS/g6S642YM77dy6rQdnc1otjLK9vFGjDzSik+e4qRqpVcksn7o2Yefe8DkTra38TVhMQ27\nExOU11r/AMlKUss3G5aDTimLmlGaYcz5rQTLcbGXd1FSQUtuzIj/AHo/8lpWsFFZq0ddkwjPV1rV\nHe+tgStEpJvPJ9WN/KaFo6xLchcXZlyCydiHsFiJbEVCfIonkl8y1tSs1eJUc9FoasqxtKOj8DbC\n77sRTbRMndkPVgTEbErFDI6a9CKXVd5fwbcNC0EZI61F3U/mjqU6qy604/sStcZEZQa2GxnDnDyY\nPs38Rh0whoiw5wi9pEdk7aWZUxnd76kNRluhs4NciKVNyqK6sr6hMVq0X2cYxeyu0+pllGUXqjbJ\nqU3LqVlFsulZ8O320VfmRKUpVZW6vSxooxtVvbbUUqk09kwn4vSb/MrDbCu3u0sn8jYtNXaaZFiX\nsSmQ/AqmRTSsgTCWwVD2G4dpXfgIbG0Je/0sVF5bkIzyrJNpc2XpzuiJrTEYhMXcdFEU2JM9kVTL\nTegHC4zVyKD5uVrHBo91aOyck7nS43U7bFqKekXaxhlTUakKa1baTOk8WOpXw+BpJSq1puTJwdeF\nGd8OptXHUOFOspSmnK6sl0Opw3gfZwcJQtF6u73MbMeiS67OAX2jAwqSb7y2ZzMdhZd5RW12d6jB\nQoxgtkiJQi2rpMxYs5fXiMRw6rLDxlTllqXd1J2RfDcHxFSlGLqNtO71ukeylh6ba7qdi0acIrRJ\nGtuYmTdeGr4eeCc41dbnMqNSzvm3oep9rqEnSp1YK6WkrHk5wnnsl72qLwZ5pquMYwa1fMKUWpq+\nwuUKkJxjUi4s0yyufZt2aSt4m3Jmq3hLa65j8KnF3Jq0nNNrSS3RFLNGUU13WTTHdoNOmmgrr+m+\nqEYOfdy315GqWxGLCY1UkVlVVyKlK7utBGyKydfRjMOrOclpaLEflWXqPpL+jU1tpugsYczcn0Gx\nZHZtJaWReMdNioG1a6LxldaFLJIIaS8CKarcyGuRKBkC7EPcu9ijKOctWPi8tP5iYWvdkzlfRbI0\n5pbuyYrUohsNwGJBOVlYsrJK4r36qTelyKbh13rvmbYXeyuUTVONo01p1LQqVHHM5O3JErcyHKEr\nbWYKEiVfInffqGZc0Zb+IatHXchJjLwexOT4WFwlzlHVTfmQ6k3GzloMnTk7aeQlwknqmVLqUTpu\nViWlsRlaH5n4COzi+Y2L7kxTbRSo7JLmHehHTVApMJO4RSVa3JomFXNtuLnG+zFuLhqm7lw1ui7j\nLXRghWq9UOVSSjd2v4MmLps3YvQf9OpJX0RjnXb3TRopuX2Ko+oNZ13pJtN/uNg2txazKOzsTFq+\npWWylUjezNUJJoxqGq0NFODWzMVqU9uyKVJWp3WpCi5S12K4pZKE5R/KrgeexUVHF1KnW7XgZsBH\nt+MYak9nNGniD7jtutzFg6vYY6hieVOacvkdI2+m0sPRpRSjEa1ZGbDVo1IJroaNZKxxx6F1KKju\nhPbwm2ozTaOVPBYv7RPJVtSeqT5DsNwiCbnKUlNvvOLauS/WusnrqQmpIG9CFBRilHkRc0wycRUZ\nYdxlqeRrQTfYxWjfken4pNqlK2/I8xipKmpfFayJPV5eMOk815Zsv5uomr/UnzTRCm4xa5Mmk80m\nm/BM6uFWp5777F5VFbZW6rkXVowTvuzPJWqfPcelbsHJ3TudPWye5x8LUUXFdLHZg1KKaZGeSraa\nMtWOWXgbsqYirTT5BzsZ4NNWWiuaFK2GnbYTkcdthn/ppacyrCHO7s3qW5C3vcYrNFQbgvdsF9Ng\n5EVeDuvEutRUNrl4yIBxKuJeWwtvUDmgwsTGN3dnRzWih0I9eRSNm9OQTm72WxFWqVLuyLYeF5Zm\nIhFzqWNcKdo+9sSh+bTKOjr3emxjUdU7m2PuqVkrGK1F5SUbLwEzn05kznGUrSbVuaW4uWRe7nfz\n0EjVCqWGQqXQqVNPaLf7hBOMdFotXqVNalLXTQHUkvFeJnurrcu5fuRrTFKD96Cv4FskJPRtfsIu\ni8Jd1v8AYq6GkoS6XEtdB0m+zQltrdBKrqgadg0sQ3roEVs9yrd7roVlUauiudwWnvMqGu9C21+a\n/wCCsqmZX0KdrJp31bKp6PW37FS1ZucvDxOjBZMGot3u/IwUIudVX903YieWMI35XJSIUlyRdZdL\nr+BMZeBaU3lvy6mV0yVaEZKOtxiqPo0ZYQvUU56vokaMzs7RCa0Qn1ZNZqVNp81ZiI3y7pFKs7Ru\n2RXHxcHGE7ru3t5GNU4uhnW3M046raEk37xmwyawlRy2vY3+Or1Hs3je1waTd5U3kZ6GNZWvc8B7\nMYrseIzpSfdq7fM9lUzunLsrZraXMc5ldeF2JxfE6eHlkXfqPaK1Yj75rQTl9lqKCdnocuWFyYhV\nXXqxrL8y2GzhRm1mlUqf9Lm7P9tjPx36V16PG8NVgnJumnzexshUVVZoSTj1TOdgsJKrLNUjFR8V\n/wAGyeWhFqCUV4KxLWeUkvxlx/ehK55nHRi6jcZWkns9md7FVXJNLmcDiVoU5NljFZ6sY5FCSXzR\nzaknGrGK5M6FCopRjnWZMx4mllqrW63TOnH58cuTTTbaa8ileSUczXgGFblNR5WIxtopU+e7H6n4\nXQqOFRNnao1krWejOJR1VmtVsbKc5R0WyLWXbSImkZMFiMzySve118jU3dmWKo42Ia/oNeJMmE9K\nK8WVGOUVyIU7aF5K5Rx1NIundbg2UUdCJNx8SC932Tt1Gck1zEJ3jYvSqaKLJQ2elhcgqzUXvoVT\nvsIrEgvyJWpNrctTbmsr2yxWrGSppO2mxOFpOVeCdrJl6lKbnJ76hrPitNRp1Vbbrcappt6uwqNN\n5rO1+g1U7bkESkotKK0Njk1CxmhCz1sO5maqkk7+Ba8Vru/EJMW7ylbkAx1JOzWiRSUm93cl2S8E\nLzpplRLmo2dlciNVO1hUpOUgjvf+BgdGq3e3Qe52oQT0cm2YoWu0lrew6rNuoop6R08A1L8aZe7B\nbXVxbTtqXqTinFNu6S5ENq2a5lqkt76O4rtI6pjqktdBE6cZeDNRlSco+DZW6au9/mXjQi1zJWHt\nzRUL05EfIlxa5EXtuVGvBK2Zbtk4up/WypruqxXCe85ckrsRKV5t66vmRfI2R1V7/wAk320/gyxq\nW6milLMtTNQ+E+iHRi2tkjOppDYzVtyKYoaayM+Mywpavcepx8TLj0q0LR5IjXH1x61Ptm5r3Fov\nEXe9KNNcldm2qlGkoabGJPvyklobldfGOMpYfERqR3i7o9xgMcsRShUTvGSPF1o35GjhHEXgazp1\nLujJ6+D6l5TtDjcr3cqMamrimTTwkU7pW8ELwWNp1KSaknp5mlYqnbT/AHOGPRKfCnkjqzNiZab6\nC6uOit5HOxXEE9nsMSpxVSMIts8zxLE9tLKtjZi686ztfQyTw+mxrj8Z5KYeLnBdFuKqytOUXqru\nxozOFJ20XUx1XZrwNz1zvi9GpkWgTmqzlJ7ilFu8VzG6QpPqVmopd3V7IfKV4uRibbkkboTfZ2m4\nprqhUbsH3ZxX5l6G/kYeGt1oSrPWTdr+BvaIzVHEKjWRIJNoiq9t1oGSJFGWkVexQLYiSsCsHLcC\nmXUrbs9bXGWBLqECtNLMrjMsXsrERkorbQntU9HZfyRWSN3dWGQoO95fsaKdOMVv/BE7yfd2LrOG\nYeOWrGyIknd6cwpScJJ32Hzi5Nyg1lfV2I3JsYW4xnrFrQdCcZRt/Ix0pSVmo+ZVUJx0yaeBU61a\nMMyvfclpp8gi3HRxcbEOSb0kQVqFFpbUbKKktWZqja0iEMqXESzSfKw1u613Fy0LEUS/cu7WXJlM\n2hLby2uUNw6Sm5O9oq7vzLJKdWF0027vUXN9nGEb96W/yLUqiqV77Rimw1/w2dpVnz1sXtZWuZoV\nFmv1Guq1yM01Mo2/4F6ddS7mpJXT/cTOOt0IBJxbvt0LZuaRSTlJeAmE2paXsaQyUp3vl3Eyd3Zo\n0Opbe/kKXeqJX0CVppLs8HOSfvaCVEdXmlSp007aXK6ONlZ25haWoNy8EaKatHQIqLj48yc1nZL+\nCVEppu2o+KXPMxEabk97DEnH8y8iBk5qGXTRu2pix9epHurS6ua51F2dmlK/8HKxdOpUqxyrupDG\n+NInN5k5PcdkU6V3s9jPKOapGMtlv+wzO0306FdValJOm9djnVFaR0ajao2vvIxTSaenM1Ep/C8V\nVpYiMFUahLl0Z6Gcq+S6nc8pRbhWjJcmj2GFtUoL5GP6eun878ZLVZbybDsnY6EKUdizox5nN0cy\nnh3Kb00L16ajTdlsb1TSuZsTaMQjh4nLSbctX+VGZQdWN795mzF0e1+a2E0qLhLNJpL5m5fjnZ9X\n7CNLKr3k0m/AiVHNeEd0yk6sVVd3oXpVISqKbk0m9bFms3C54SpF3UXJeG4yNGvWSjkfS8uR0I55\nQUqbjUj4PVDqamo3lGzLrnbhuDw8cNh400782xshc5SqUo5bKcFZpvdCZ1JR3av8yYlpsmUqbK/Q\nU63ii0qmZL5FZ1WQtyLTs1qylltuwqbpAnZlX4Ep6XAvZvxIWgRkrlpKy3QFG3+xROUXo00NUULz\nJXVgFdvLk2R20iFGLuye6uRphKqyb3Y2Ep+LFxnBcmMUk1pB+ZA2NTqi3arS0svyZmtf8unzK2Xh\n5jF1sWLlH8+ZeOpMcXTqu06cb9djIlTXvK+vUhOlfYYvatrnh5aKTQuVGW8Jxl8mIi4ZvBalHNXu\nrrxGHY9qcZaxevOxWc4tFVXn8Wbn3iyrQqLvwTfzB8KlJaWG4aGad37sVmlfoQ+xvq2Vq1YqGSlf\nK/eb/MU8+1WtPPVc3u+XQvh3loV53tokhT90ZJZeH/65gnuk0m863HuTy2TM9PSVx075SVmJhOSi\n7sXOo3K12RmaVyue7VkUNzdy1ylJvNv/AAEnZWehMJQXW4UypNSp+KF0nlV76lJO/N2IV1yA0VKn\nat3/AGLU2lz8xVO7G6JaxIGKoldJq25MZq93Iz5l0C7uRNaXX5R8wi5S5iIofBW3CaOVhdWWSnKT\n6DW7anPxuOoxg4XzN9OQn1Z6RCpnjmb1d9OheOvd58hTjTtmhUV9yFPVST1iHoNru8bLlqZnG705\njKknOV479BlClKW8bF8PSKVFyq2S2d2eo4ev6aVzlUcMlLNbU7GFi1axz5XXXjMa8mu3kGXwNNOn\nmhcXOnK+hlovJdGTE0rrY6UIPoKqUryA4GIw7y9HyZy69KalZxkz1dbD6qWW6XIzqhC+sbvpY1Pj\nNmvPRwzqw70WmZazjTm4xku7oepxuTB4KdVpXUdDxMm5Sbb1budeE1y5/Do4mpSqZ6U3FrmdKhxt\nuKjXpr/VH0OMSjdkcnpadaniY5qc1Lw5lpU09r7annISlCV4tprmjdR4pUirVFmXXmZvH/SY6LUU\n7InMofmu/ARRr0sRtK0ujGZU4t9DKLdreL125Fk9N9WJjaL0er5FE3e99Qa1PxIi1fX9hKraak50\n9xi6bsxid4ic6a3J7VLmTFOikJqpJsI1ejFzkpa3KUpPbQl362KJ9Nyz0ZXNW7vuW1f5iqYXsUSs\n2veJu+WpGZJBdaW2AYk77lU+8WTtuLvZgaHC1BNtRzvmLlaNr2/YrOfab/JEz/LC97Iglzi42KOS\nT5g0itlfmAxS0Ic7EIgKa592w7EaYSgut2ZmrDsVJ9lQVtoBqeUpWvuNazJKxni29hqnKys3oGUT\nTWgpSyvQmdSTerZW9yhznFxvZ3+YrMr7BFN6Ins5ARn8Ce0IUJN7BlaAZCo+hbM2Kgry0GW0IiyL\npC01sMTSILoZKpGEG5OyW4tSijlY7FuvLLHSC/kSasmrYzHyrNxh3Yfyzm1HmkXbKS946SY2i7T3\nDNL4mAW1KL05tNau56jhPD51sFCt8Wp5VHufZipP7uhF6pHP+njr/P1eGCcVZxNWHwnzOnTpKS1Q\n2FFJnB20mlSceQdjeRry22KuOtwM0aVpbFJUdbmzLrcHC6A584chM4RXI6fYR3e5V04U7ya0Kjw/\ntVXcI08Mvzd5/LkeaOn7QYv7ZxevNe7F5I/JHMPRxmR5+d2pQXAGaZCJIQAWUrO63NtDGytkqSdu\nphC5MHYg7vqTqmYcHW17OT+Rqb1M4xUtg2RchgS5NcyrkVbIZReMmi0qspybaWoq5KepA6DLTTau\nUTtsDbCIDVlbk3uBPKxNrWKjL3tcC265lfmxsN0Q4pO9rhRSWWDqS25LqxWsr6k1JSk+82ysPeCL\nLRbWJilfUmUVle5WMV1CrpJ3/wBiI5U7Mre11uVbtsgHO2ttUXxrinTiuUEJjLqi2Md67tskl/BF\n/Cky8Zd5ITdl1LmVF5NJtNMW2uWxee6duQZUkm3vyAiF76DHN25XRTRPQlbACb3IblzuTYNX8gCG\nmvMtyIj71r6F7rLZpEEJrwJUtSFboRK0U5PZakQrH4nJT7OL1ktfBHLuWq1HUnKb5sXfQ6SY3Pgb\n1Kv3gTuwe6KqVuAIkAPofs1hrcNpztufPT6l7PRS4Nh31gjnz8dODfGCSLWJsDOLqgLXAkoiwWuT\nYkIq0c/i+IWF4biKzfuxdvnyOjI8b7c47LGlgYS1l35/8F4/aW/HjZNttvVsrYlgkeh5xYgllQJ5\nEAwbsgIb1sgvYrHqTfUqLwk4yTXI6UJKdNSXM5VzZg6m8H80ZpWq5FydA0MsI1KsYmVluUQgLciN\nQLol7kpEuOuhAt7lktCXDXVole6BS2hZPkEloiNnqA73dUMetxKn3bFoyv5BRUhzF7Mfa8fkLcdQ\nC7ehXLqMUdOZEk7XVwK5bvQvKk0leSt0FpyBTnEgbTg01oLrd6pJ9WNozcpd65M7OWwX8Y7EodKC\nKqC5F1A/dRVeI1RbgheXUgCyt0BRuy6jpuBCV+RW8YvVBKWVlXK/ICcycrq4xq63KK0kWV0tAJsZ\n8fNww7XxOxoXzMPFHrTXzE9Iw30I5BzIZ0bREl8iNmRe5UXRYhFiKg+p+zmvAsG//bR8tPpPsdX7\nbgNKL3ptx/kxz8b4eu3YCSDk6IC6Bmek74iclJtPZNaINNPIAAIVia8MPQnWqO0IRcmz5TxLGTx+\nOq4mpvOV0ui5I9Z7a8Ygqf3dRleTadVrkuSPENnThP1z538BIIDo5oZUllQJKzfIlysVWruVE7Kx\nCDcOQANoSy1YvxFX1BOzIOy0loVsTTUZ04ydRLTmiypxv/eh/JhlSwDuxv8A56X1F/sscv8A5ij5\ngxntoVkhsqKu71I/sLnTS2ncocibDFDwLKm97GNGaS2Lxj3RsoLoCX7F0wvLpqri3E0Sj3RdtdgK\nKJMS9kNg6cFK8d0BRTXLmFk3/wAlez00YOLfMBitbchrWyd77CZRaempMG1qt+QF2rOzZRuxSSdy\nErModTkk9yzlF8xaWvUY4rLfKiKi8XFoFFWJhSvd8hsacUk9yCtNLwLTUYrS1yzhF3aVvmInDvcg\nJb5JaEWZORrkvMvHu7pAIno9hbb6GipJ7JaibSLAQvroW1LQjJa6F8snsghevQx8TptwhUtotGdB\nqS3sKxUJVsNOnHWT2QnqxwnpqGZNFqtKdKbhUi4tcmhdjo0tuQveIuSnqioZEsQtgIqT2HsDjEqm\nIwcnv34r+H/wePHYXE1cHXhXoTcKkHdNGbNmLLj6+iGjz/CPavBYyjFYupHDV1o1L3X4pndVelKK\nkqkHF7NS0ZyzHaJewiCq5oxcYxhHd3vf5LkXqYnD003OvTilveaRzq/tHwvDe/iozf8A0d7/AGCu\nucr2i4suE8OlUi129Tu0149f2OXivbbCRpyWFoVJz/K52UTyPEuJ4ridftcVUzNaRSVkka48b+sX\nlGOdacqsqtSTlOTu29W2Uvmd7WJaTBKx1cgAEMCGVe5LZVu+wFXuD0VgQLfUqANSf2JjCdR2hGUn\n0SArsC1ZqhwzFz2otfPQiWDq0KsVVjYmxcrStIRS5EXZDBamXM+hHPLXZass0gj/AE8Ous3/AATY\nKq1oUkOsthUosDp218C148mXVJIMnhscminFyIy2H5fAiUL62LoztWTKaaDpRey5i1ScnoVFdxka\nTaJVBpa2Lxi482xoTlkr6hBdWOkk1sU0QETpJ6orksMjLomRNSeuwGeaQsdUjZC8vgaQQdnYYr/s\nUUdR0IMlWJWuw2MdCVFKOhCnZGVXyd0VKHeuOVR20SIlKXQikyRW8VpZ3GuTW6RRvXdIqKOUej8i\nmRX/ALlv2HKUeqDPBcyhKjZ73L203Rd1E1oripykwibE0l/VhpzQvNYZh3evD/UCevR/d+FxlFRr\n0ozXijicR9j5K88DUuvgn6npMIu6jZHRHOcrHq6yvlWKweIwlRwxFKUGuqELdH1HG4eliKbjVhFp\n8mrnjONcHo4dupQ7qX5eR14/035XPl/PPHH5AAHRzBJAATctmfUpcLhV8xFytwAm5AAEAAQAXIAA\nKsqyxVlDsLg6+LlajTcvHkdvC+yeJmk69WNNdFqa/ZmKWHTPRqTcTjy53fjtx4TPrjUPZnBUEnNO\npL/qNf2WjRVqdOMV4KxskxEzlbb66dZGKqrHI4irxjfqdivscfH7JeJrj6xz8c6SIi3F3TsyzK6H\nZ5Gx1FUoQdRZraN80VaW8XdCqNTK8rV4y3Q+SyvuSuuQaUsyQs2Q0B23G600KOnJO+ZvwBVfC4Oq\n3dWOP1pPMp2b6vzJzMiU2gBw0sLhFrTSxEpu25Cm7cjSGt2RTMykq1o7C3W8Bgbbnch2uLjVzO2x\nMpZUXAxcwlJKOuwpytHW9xdWakklcYarOWZkwlYqvkTFNs0ybCOZ6GiUcqVmUpvKrZWWcpNbL92Y\nrStgSKTzPnb5EpeIDo7A3YUnGPMnN0IJuis4Rlrd3KSqJO4t1WawXcEuYtt3B1LlblRa7LRm7q9y\ngbBDptbWQYR3xdNeIhu4/Ax/8XT+ZPxqevWYV91GpGTD+6jVyOL1wqu9GeZ4470JnpMQ+6zzHGn/\nAEpl4+py8eZbIuRfQi56nmWuFytyUwJvclIrcMwFwKZguBe5FylwuBa4XK3C4EgVuFwJKtgQ9ij1\n/s8rYOB3ovunE4ErYOn8jtLY8vL16uPgkxU9hrFVHoZaYq70ZxsbrJHXxEtGcXFv+ojpw9cf6eEN\nXeiKuHgSm0y6lc6vMXCNpDdQdtwzBRqEr2DMQ3cDqQ2LWK04vUvlsc1RyIjr8upLWhC0VkQQ1G2q\nRW0bBJMhJWKKuC5C3BjtCG09ihcY2e2pLjfWxe6tyF1KtlZAKqyvohcfEtuDStoaZSk2/EfGGVJ3\n1FwXUalH5masTm8UR2gLLe2hOngRVHNvbQEm92MslqVzX2QE91a8xVST5Eu4uTKijZDBy6EZmVFk\nvAMrJzOxGZgTciT1IZXmUWTNXDtcZEx3NfCrvGX6Il8a4+vWYf3UaL6GWg+6hzlocHshOJl3WeY4\nxK8ZI9Dipd1nmuKu6Zrj6zy8edT5Ayv52WZ6XmQSQBRIEXC4RIEXC4Ehci4XAkggAJAgAJIYER1m\nl4ge34QsuGgvA60djk8O0ow+R1IvQ8l9eueLS2M9V6D5PQy1mRpixEjjYiX9VnVxLONVu6knZ7nX\ng4f1vxW4JkAk+jOjzpuQmDTfJkqEnyACSezduXmGVW95EHYzkOZ5776xPwUvJ+pH31ifgpeT9SdK\n1r0DnZEKd1c4H31ifgpeT9SPvjE2tlp+T9S9KO85XK38Th/fGJ+Gl5P1D74xHwUvJ+o6VHbbZVnG\n+98R8FLyfqR974j4Kfk/UvWjsSbsLepynxWu/wAtPyfqR951vhp+T9R1o6tgOV96V/hp+T9Q+863\nw0/J+o60da4yJxPvKt8MPJ+pK4nXW0YeT9R1o7iS3e4OXQ4n3rX+Gn5P1JXF66/JS8n6k6UdyMWy\n1tThrjWJX5KXk/Ul8axL/JS8n6jrVdmWgibOX98YjnCk/wBn6lXxSu940/J+petR0SNjm/eNb4af\nk/Uj7xrfDDyfqOtR1E3YGzl/eFb4YeT9Q+8Kvww8mOtHTciGzmfbqvSHkH26r0h5MvWjo3OjwSOa\nvN9EjzjxtV8o+RpwfG8Tg83Z06LzfEn6kvG41x+X69/RdkXlM8Sva3Hrajhvpl6g/a3Hv/Fhvpl6\nnL/Hyej/ACcXqMTLRnnuKPRmOftLjZ706H7RfqY6/E69e+eMFfon6m5wsZ5f0lUquLndLlYU5FMz\nDM/A6446sTYpnfgGd+AF7BYpnfgGd+BRewWKZ34BnfgQXCxTO/AM78ALgUzvwDO/AC5BXO/AM78A\nLXLU9asPmheZ+ARm4yUkldO4HuMDpCJ0oPQ8TT9oMXSVo06P7p+o5e1OOS/tYf6Zepwv8+TvP6cX\nsZPQy1meYftVjn/iw/0y9RcvaTGS3p0PpfqT/HyX/JxdjFysmzjub6sz1eMYiqmpQpK/RP1M/wBs\nqdI+R148bHH+nLt46Kk+oKT6s5/22p8MfIPttT4YeRcc29tvmBg+21Phh5B9tqfDDyHUdBbEnP8A\nt1X4YeTD7fV+GHkx1oygAG1AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\nAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\nAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\nAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\nAAAAAAAAAAAAAAAB/9k=\n",
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"400\"\n",
       "            height=\"300\"\n",
       "            src=\"https://www.youtube.com/embed/GXjjMSn2Nws\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.YouTubeVideo at 0x7f6c88113e10>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import YouTubeVideo\n",
    "YouTubeVideo(\"GXjjMSn2Nws\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": false,
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 4,
     "slide_type": "subslide"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Big Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 8,
     "slide_helper": "subslide_end"
    },
    "slide_helper": "subslide_end",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<div class=\"alert alert-success\">\n",
    "**Big Data** is yet another umbrella term bringing togehter all methods and discipline involved in analysing, capturing, curating, searching, sharing, storing, transfering, visualising large data sets. The term has been created out of the enormous increase of available data due to the Internet and subequent phenomas. \n",
    "</div>\n",
    "\n",
    "Big Data is \"defined\" (in a non-mathematical sense) via 4 V's, namely Volume, Velocity, Variety, Veracity. It stems out of characterisation by Gartner (Gartner, 2011) who used 3V's, refined from IBM. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 8,
     "slide_helper": "subslide_end",
     "slide_type": "subslide"
    },
    "slide_helper": "subslide_end",
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"files/images/4-Vs-of-big-data.jpg\"/>\n",
    "Original Source: http://www.ibmbigdatahub.com/infographic/four-vs-big-data "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": false,
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 8,
     "slide_type": "subslide"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## The KDD Process"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 11,
     "slide_helper": "subslide_end"
    },
    "slide_helper": "slide_end",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<img src=\"files/images/600px-Fayyad96kdd-process.png\"/>\n",
    "according to (Fayyad et al. 1996).\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": false,
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 11,
     "slide_type": "subslide"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# What is Python?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 13,
     "slide_helper": "subslide_end"
    },
    "slide_helper": "subslide_end",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<img src=\"http://img-9gag-lol.9cache.com/photo/aKgj6M3_460s_v1.jpg\">\n",
    "Source: http://m.9gag.com/gag/aKgj6M3/harry-potter-talking-to-snakes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 13,
     "slide_helper": "subslide_end",
     "slide_type": "subslide"
    },
    "slide_helper": "subslide_end",
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "[Python](http://www.python.org/) is a modern, general-purpose, object-oriented, high-level programming language.\n",
    "\n",
    "General characteristics of Python:\n",
    "\n",
    "* **clean and simple language:** Easy-to-read and intuitive code, easy-to-learn minimalistic syntax, maintainability scales well with size of projects.\n",
    "* **expressive language:** Fewer lines of code, fewer bugs, easier to maintain.\n",
    "\n",
    "Technical details:\n",
    "\n",
    "* **dynamically typed:** No need to define the type of variables, function arguments or return types.\n",
    "* **automatic memory management:** No need to explicitly allocate and deallocate memory for variables and data arrays. No memory leak bugs. \n",
    "* **interpreted:** No need to compile the code. The Python interpreter reads and executes the python code directly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": false,
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 13,
     "slide_type": "subslide"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Python: Pros and Cons"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 16,
     "slide_helper": "subslide_end"
    },
    "slide_helper": "slide_end",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "\n",
    "Advantages:\n",
    "\n",
    "* The main advantage is __ease of programming__ and __minimizing development time__\n",
    "* __Modular and object-oriented programming__, good system for __packaging and  code re-use__. \n",
    "\n",
    "* Documentation tightly integrated with the code.\n",
    "* A __large standard library__, and a __huge number of libraries__\n",
    "\n",
    "Disadvantages:\n",
    "\n",
    "* Since Python is an interpreted and dynamically typed programming language, the execution of python code __can be slow__ compared to compiled statically typed programming languages, such as C and Fortran. However, exentsion like [Cython](http://cython.org/) allow to integrate fast C Code (e.g. for inner loops)\n",
    "* Somewhat decentralized, with __different environment, packages and documentation__ spread out at different places. Can make it harder to get started."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": false,
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 16,
     "slide_type": "subslide"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# What makes python suitable for Data Science?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 18,
     "slide_helper": "subslide_end"
    },
    "slide_helper": "subslide_end",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<img src=\"files/images/optimizing-what.png\" width=\"600\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 18,
     "slide_helper": "subslide_end",
     "slide_type": "subslide"
    },
    "slide_helper": "subslide_end",
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "* Python has a strong position in scientific computing: \n",
    "    * **Large community of users**, easy to find help and documentation.\n",
    "\n",
    "\n",
    "* Extensive ecosystem of important libraries and environments\n",
    "   * **scipy**: http://www.scipy.org -  Scientific Python\n",
    "   * **numpy**: http://numpy.scipy.org - Numerical Python\n",
    "   * **matplotlib**: http://www.matplotlib.org - graphics library\n",
    "  Note: these libraries are bundled in the popular `pylab` which puts them all in one namespace in a Matlab-like interface. We don't want that (the matplotlib developers give an elaborate argument for this). \n",
    "\n",
    "* Great performance due to close integration with time-tested and highly optimized codes written in C and Fortran:\n",
    "    * blas, altas blas, lapack, arpack, Intel MKL, ...\n",
    "\n",
    "\n",
    "* Good support for \n",
    "    * Parallel processing with processes and threads\n",
    "    * Interprocess communication (MPI)\n",
    "    * GPU computing (OpenCL and CUDA)\n",
    "\n",
    "\n",
    "* Readily available and suitable for use on high-performance computing clusters. \n",
    "\n",
    "\n",
    "* Free and Open License"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": false,
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 18,
     "slide_type": "subslide"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## The scientific python software stack"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 21,
     "slide_helper": "subslide_end"
    },
    "slide_helper": "slide_end",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<img src=\"files/images/scientific-python-stack.svg\" width=\"300\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": false,
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 21,
     "slide_type": "subslide"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Python Environments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 23,
     "slide_helper": "subslide_end"
    },
    "slide_helper": "subslide_end",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "\n",
    "\n",
    "- Python also refers to the __standard implementation of the interpreter__\n",
    "  - (technically referred to as [CPython](http://en.wikipedia.org/wiki/CPython))  \n",
    "  - The interpreter runs the python code on a computer.\n",
    "\n",
    "\n",
    "- __Different environments__ through wich Python can be used allowing different workflows\n",
    "\n",
    "\n",
    "- One strength of python is that it __versatile__ and can be used in __complementary ways__, but it can be confusing for beginners so we will start with a brief survey of python environments that are useful for scientific computing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": false,
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 23,
     "slide_type": "subslide"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Python interpreter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 25,
     "slide_helper": "subslide_end"
    },
    "slide_helper": "subslide_end",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The standard way to use the Python programming language is to use the Python interpreter to run python code. The python interpreter is a program that read and execute the python code in files passed to it as arguments. At the command prompt, the command ``python`` is used to invoke the Python interpreter.\n",
    "\n",
    "For example, to run a file ``my-program.py`` that contains python code from the command prompt, use::\n",
    "\n",
    "    $ python my-program.py\n",
    "\n",
    "We can also start the interpreter by simply typing ``python`` at the command line, and interactively type python code into the interpreter. \n",
    "\n",
    "<img src=\"files/images/python-screenshot.jpg\" width=\"600\"> \n",
    "\n",
    "This is often how we want to work when developing scientific applications, or when doing small calculations. But the standard python interpreter is not very convenient for this kind of work, due to a number of limitations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": false,
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 25,
     "slide_type": "subslide"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## IPython"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 27,
     "slide_helper": "subslide_end"
    },
    "slide_helper": "subslide_end",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "[IPython](http://ipython.org/) is an interactive shell that addresses the limitation of the standard python interpreter, and it is a work-horse for scientific use of python. It provides an interactive prompt to the python interpreter with a greatly improved user-friendliness.\n",
    "\n",
    "<img src=\"files/images/ipython-screenshot.jpg\" width=\"600\">\n",
    "<!--<img src=\"https://raw.github.com/jrjohansson/scientific-python-lectures/master/images/ipython-screenshot.jpg\" width=\"600\">\n",
    "-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 27,
     "slide_helper": "subslide_end",
     "slide_type": "subslide"
    },
    "slide_helper": "subslide_end",
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Some of the many useful features of IPython includes:\n",
    "\n",
    "* Command history, which can be browsed with the up and down arrows on the keyboard.\n",
    "* Tab auto-completion.\n",
    "* In-line editing of code.\n",
    "* Object introspection, and automatic extract of documentation strings from python objects like classes and functions.\n",
    "* Good interaction with operating system shell.\n",
    "* Support for multiple parallel back-end processes, that can run on computing clusters or cloud services like Amazon EE2.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": false,
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 27,
     "slide_type": "subslide"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Jupyter notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 30,
     "slide_helper": "subslide_end"
    },
    "slide_helper": "subslide_end",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "[Jupyter notebook](http://jupyter.org) is an HTML-based notebook environment for Python, similar to Mathematica or Maple. It evolved from the IPython shell via IPython notebooks and provides a cell-based environment with great interactivity, where calculations can be organized documented in a structured way.\n",
    "\n",
    " <img src=\"files/images/ipython-notebook-screenshot.jpg\" width=\"800\"> \n",
    "<!--<img src=\"https://raw.github.com/jrjohansson/scientific-python-lectures/master/images/ipython-notebook-screenshot.jpg\" width=\"800\">\n",
    "-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 30,
     "slide_helper": "subslide_end",
     "slide_type": "subslide"
    },
    "slide_helper": "subslide_end",
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Local Jupyter Notebook\n",
    "\n",
    "Although using the a web browser as graphical interface, Jupyter notebooks are usually run locally, from the same computer that run the browser. To start a new Jupyter notebook session, run the following command:\n",
    "\n",
    "    $ jupyter notebook\n",
    "\n",
    "from a directory where you want the notebooks to be stored. This will open a new browser window (or a new tab in an existing window) with an index page where existing notebooks are shown and from which new notebooks can be created."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Jupyter Notebooks on a Server (JupyterHub)\n",
    "\n",
    "Organizations might decide to run a Jupyter server as a service, then called a JupyterHub. A popular public one is [Try Jupyter!](http://try.jupyter.org) (although increasingly busy). The group at Passau runs its own JupyterHub at http://jupyter.dimis.fim.uni-passau.de (requires registration with our administrator)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 30,
     "slide_helper": "subslide_end",
     "slide_type": "subslide"
    },
    "slide_helper": "subslide_end",
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Why and when to use Jupyter Notebook?\n",
    "\n",
    "- This lecture notes have been developed using Jupyter notebooks\n",
    "- All exercises can be done using Jupyter notebooks\n",
    "- There is a special introduction to Jupyter notebooks\n",
    "- Very useful for exploring unknown solutions\n",
    "  - REPL Cycle - Read, Evaluate, Print, Loop\n",
    "  - Document your work while working\n",
    "  - Great resource for learning\n",
    "- Not very useful for large scale industry project\n",
    "  - No project management\n",
    "  - Missing refactoring\n",
    "  - Use IDE's in this case ([Eclipse based PyDev](http://pydev.org/), [Intellij's Pycharm](http://www.jetbrains.com/pycharm/))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": false,
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 30,
     "slide_type": "subslide"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## IDE: Spyder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 34,
     "slide_helper": "subslide_end"
    },
    "slide_helper": "slide_end",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "[Spyder](http://code.google.com/p/spyderlib/) is a MATLAB-like IDE for scientific computing with python. It has the many advantages of a traditional IDE environment, for example that everything from code editing, execution and debugging is carried out in a single environment, and work on different calculations can be organized as projects in the IDE environment.\n",
    "\n",
    "<img src=\"files/images/spyder-screenshot.jpg\" width=\"800\"> \n",
    "<!--<img src=\"https://raw.github.com/jrjohansson/scientific-python-lectures/master/images/spyder-screenshot.jpg\" width=\"800\">\n",
    "-->\n",
    "Some advantages of Spyder:\n",
    "\n",
    "* Powerful code editor, with syntax high-lighting, dynamic code introspection and integration with the python debugger.\n",
    "* Variable explorer, IPython command prompt.\n",
    "* Integrated documentation and help."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": false,
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 34,
     "slide_type": "subslide"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Versions of Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 36,
     "slide_helper": "subslide_end"
    },
    "slide_helper": "slide_end",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Two Version of Python\n",
    "\n",
    "  * Python 2 (2.7 still contains the majority of libraries)\n",
    "  * Python 3 (not backward compatible)\n",
    " \n",
    "We will use Python 3.5 for this lecture\n",
    "\n",
    "To see which version of Python you have, run\n",
    "    \n",
    "    $ python --version\n",
    "    Python 3.5.1 :: Anaconda 4.0.0 (64-bit)\n",
    " \n",
    "    \n",
    "    $ python2 --version\n",
    "    Python 2.7.6\n",
    "\n",
    "Several versions of Python can be installed in parallel, as shown above.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": false,
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 36,
     "slide_helper": "subslide_end",
     "slide_type": "subslide"
    },
    "slide_helper": "subslide_end",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Installation\n",
    "\n",
    "We recommend the [Anaconda](https://www.continuum.io/downloads) python distribution which\n",
    "- includes most of our favorite packages in its default installation \n",
    "- comes with a package-/environment-manager\n",
    "\n",
    "Here our choice is 64-bit python3. \n",
    "\n",
    "After installation (or upgradeng from miniconda) check that your favorites are installed.\n",
    "\n",
    "    $ conda install jupyter scipy numpy matplotlib pandas\n",
    "\n",
    "\n",
    "## add an R kernel\n",
    "\n",
    "If you want an R kernel for your Jupyter notebook, you install\n",
    "\n",
    "    $ conda install -c r r-essentials r-irkernel\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 36,
     "slide_helper": "subslide_end",
     "slide_type": "subslide"
    },
    "slide_helper": "subslide_end",
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Linux\n",
    "In Ubuntu Linux, to installing python and all the requirements run:\n",
    "\n",
    "    \n",
    "    $ sudo apt-get install python ipython ipython-notebook\n",
    "    \n",
    "    $ sudo apt-get install python-numpy python-scipy python-matplotlib python-sympy\n",
    "    \n",
    "    $ sudo apt-get install spyder\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 36,
     "slide_helper": "subslide_end",
     "slide_type": "subslide"
    },
    "slide_helper": "subslide_end",
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### MacOS X\n",
    "\n",
    "*Macports*\n",
    "\n",
    "Python is included by default in Mac OS X, but for our purposes it will be useful to install a new python environment using [Macports](http://www.macports.org/), because it makes it much easier to install all the required additional packages. Using Macports, we can install what we need with:\n",
    "\n",
    "    $ sudo port install py27-ipython +pyside+notebook+parallel+scientific\n",
    "    \n",
    "    $ sudo port install py27-scipy py27-matplotlib py27-sympy\n",
    "    \n",
    "    $ sudo port install py27-spyder\n",
    "\n",
    "These will associate the commands `python` and `ipython` with the versions installed via macports (instead of the one that is shipped with Mac OS X), run the following commands:\n",
    "\n",
    "    $ sudo port select python python27\n",
    "    \n",
    "    $ sudo port select ipython ipython27\n",
    "\n",
    "*Fink*\n",
    "\n",
    "Or, alternatively, you can use the [Fink](http://www.finkproject.org/) package manager. After installing Fink, use the following command to install python and the packages that we need:\n",
    "\n",
    "    $ sudo fink install python27 ipython-py27 numpy-py27 matplotlib-py27 scipy-py27 sympy-py27\n",
    "    \n",
    "    $ sudo fink install spyder-mac-py27"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 36,
     "slide_helper": "subslide_end",
     "slide_type": "subslide"
    },
    "slide_helper": "slide_end",
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Windows \n",
    "\n",
    "Windows lacks a good packaging system, so the easiest way to setup a Python environment is to install a pre-packaged distribution. Some good alternatives are:\n",
    "\n",
    " * [Enthought Python Distribution](http://www.enthought.com/products/epd.php). EPD is a commercial product but is available free for academic use.\n",
    " * [Anaconda CE](http://continuum.io/downloads.html). Anaconda Pro is a commercial product, but Anaconda Community Edition is free.\n",
    " * [Python(x,y)](http://code.google.com/p/pythonxy/). Fully open source.\n",
    "\n",
    "\n",
    "\n",
    "#### Note\n",
    "\n",
    "EPD and Anaconda CE are also available for Linux and Max OS X."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": false,
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 36,
     "slide_type": "subslide"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Further reading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "internals": {
     "frag_helper": "fragment_end",
     "frag_number": 42,
     "slide_helper": "subslide_end"
    },
    "slide_helper": "slide_end",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "\n",
    "* [Python](http://www.python.org>). The official Python web site.\n",
    "* [Python tutorials](http://docs.python.org/2/tutorial/). The official Python tutorials.\n",
    "* [Think Python](http://www.greenteapress.com/thinkpython/). A free book on Python.\n",
    "* **[Learn Python the Hard Way](http://learnpythonthehardway.org/book/). An exercise book. The only way to learn programming is to do it!**\n",
    "* Vasant Dhar. 2013. Data science and prediction. Commun. ACM 56, 12 (December 2013), 64-73. DOI=10.1145/2500499 http://doi.acm.org/10.1145/2500499\n",
    "* Jeff Leek (2013-12-12). \"The key word in \"Data Science\" is not Data, it is Science\". http://simplystatistics.org/2013/12/12/the-key-word-in-data-science-is-not-data-it-is-science/\n",
    "* Beyer, Mark (2011). \"Gartner Says Solving 'Big Data' Challenge Involves More Than Just Managing Volumes of Data\". Gartner. [Archived](http://web.archive.org/web/20110710043533/http://www.gartner.com/it/page.jsp?id=1731916), http://www.gartner.com/newsroom/id/1731916, last accessed 10/2014\n",
    "* U. Fayyad, G. P.-Shapiro, and P. Smyth. From data mining to knowledge discovery in databases. AI Magazine, 17(3):37-54, Fall 1996. http://citeseer.ist.psu.edu/fayyad96from.html\n"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "css": [
   ""
  ],
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "468px",
    "width": "252px"
   },
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 4,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}