PYTHON-INTERMEDIATE-5-ITERATION-GENERATORS.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## ITERATION\n",
    "*python을 아름답게*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 2 3 1 2 3 one two 1 2 3 total 3672\n",
      "drwxr-xr-x@  4 happytk  staff      136 May 17 22:09 .\n",
      "drwx------@ 49 happytk  staff     1666 May 17 22:08 ..\n",
      "-rwxr-xr-x@  1 happytk  staff  1877810 May 12 18:46 WindowsUpdate.log\n",
      "-rw-r--r--   1 happytk  staff        0 May 17 22:09 myfile.txt\n"
     ]
    }
   ],
   "source": [
    "# python은 많은 것이 iterable합니다.\n",
    "\n",
    "for element in [1, 2, 3]:\n",
    "    print(element, end=' ')\n",
    "for element in (1, 2, 3):\n",
    "    print(element, end=' ')\n",
    "for key in {'one':1, 'two':2}:\n",
    "    print(key, end=' ')\n",
    "for char in \"123\":\n",
    "    print(char, end=' ')\n",
    "for line in open(\"data/myfile.txt\"):\n",
    "    print(line, end='')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Behind the scenes.\n",
    "- forloop은 container object에 iter()함수를 호출합니다.\n",
    "- return받은 객체의 __next__() 를 호출하면서 iterating할 대상을 하나씩 가져옵니다.\n",
    "- StopIteration 오류가 발생하면 forloop을 멈추게 됩니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```\n",
    ">>> s = 'abc'\n",
    ">>> it = iter(s)\n",
    ">>> it\n",
    "<iterator object at 0x00A1DB50>\n",
    ">>> next(it)\n",
    "'a'\n",
    ">>> next(it)\n",
    "'b'\n",
    ">>> next(it)\n",
    "'c'\n",
    ">>> next(it)\n",
    "Traceback (most recent call last):\n",
    "  File \"<stdin>\", line 1, in ?\n",
    "    next(it)\n",
    "StopIteration\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4\n",
      "0\n",
      "7\n",
      "3\n"
     ]
    }
   ],
   "source": [
    "# Iteration protocol을 구현해보기\n",
    "class MyClassIterator:\n",
    "    def __init__(self, *arg):\n",
    "        self._data = arg\n",
    "        self.index = 0\n",
    "    def __next__(self):\n",
    "        if len(self._data) == self.index:\n",
    "            raise StopIteration\n",
    "        ret = self._data[self.index]\n",
    "        self.index += 1\n",
    "        return ret\n",
    "\n",
    "class MyClass:\n",
    "    def __init__(self):\n",
    "        self._data = [4,0,7,3]\n",
    "    def __iter__(self):\n",
    "        return MyClassIterator(*self._data)\n",
    "\n",
    "for x in MyClass():\n",
    "    print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4\n",
      "0\n",
      "7\n",
      "3\n"
     ]
    }
   ],
   "source": [
    "# 편의상(혹은 필요여부에 따라) 하나의 클래스에 구현하는 편입니다.\n",
    "class MyClassSimple:\n",
    "    def __init__(self):\n",
    "        self._data = [4,0,7,3]\n",
    "        self.index = 0\n",
    "    def __iter__(self):\n",
    "        return self\n",
    "    def __next__(self):\n",
    "        if len(self._data) == self.index:\n",
    "            raise StopIteration\n",
    "        ret = self._data[self.index]\n",
    "        self.index += 1\n",
    "        return ret    \n",
    "\n",
    "for x in MyClassSimple():\n",
    "    print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(True, [4, 0, 7, 3], True, [4, 0, 7, 3])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections.abc import Iterable\n",
    "(\n",
    "    isinstance(MyClass(), Iterable),\n",
    "    list(MyClass()),\n",
    "    isinstance(MyClassSimple(), Iterable),\n",
    "    list(MyClassSimple())\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## GENERATOR (PEP 255)\n",
    "*강력한 도구*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- iterator를 return하는 함수입니다.\n",
    "- 일반 함수처럼 생겼는데 다만 함수 내 **yield** keyword가 사용됩니다.\n",
    "- 일반 함수는 한번 수행되면 return까지 쉼 없이 수행되지만,\n",
    "- generator는 yield를 만나면 잠시 멈추고 caller에게 수행권을 넘겨줍니다.\n",
    "- 즉 (신기하게도) 함수가 **동작중 SUSPEND**될 수 있습니다.\n",
    "- 다시 호출하면 SUSPEND되었던 부분에서 이어서 수행할 수 있습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![](img/what-does-that-mean.gif)\n",
    "\n",
    "확인해봅시다!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 우리가 아는 일반함수\n",
    "def well_known_function():\n",
    "    return 10\n",
    "\n",
    "well_known_function()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "funtion is started.\n",
      "10\n",
      "20\n"
     ]
    }
   ],
   "source": [
    "# yield가 사용되는 순간 함수가 아닌 generator로 변신.\n",
    "def i_am_generator():\n",
    "    print('funtion is started.')\n",
    "    yield 10\n",
    "    yield 20\n",
    "\n",
    "# 수행 결과로 왠 객체가..\n",
    "# g = i_am_generator()\n",
    "# for x in g:, list(g), ....\n",
    "for x in i_am_generator():\n",
    "    print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "funtion is started.\n",
      "10\n"
     ]
    }
   ],
   "source": [
    "def i_am_generator():\n",
    "    print('funtion is started.')\n",
    "    yield 10\n",
    "\n",
    "\n",
    "# 해당객체는 iterable하므로 for문으로 꺼내볼 수 있습니다.\n",
    "for x in i_am_generator():\n",
    "    print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "funtion is started.\n",
      "1\n",
      "2\n",
      "3\n",
      "function is finished.\n"
     ]
    }
   ],
   "source": [
    "# 여러개의 yield문을 안에서 사용할 수 있습니다.\n",
    "def yield_test():\n",
    "    print('funtion is started.') # 이게 언제 실행되는지 보세요.\n",
    "    yield 1\n",
    "    yield 2\n",
    "    yield 3\n",
    "    print('function is finished.')\n",
    "g = yield_test()\n",
    "for x in g:\n",
    "    print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "funtion is started.\n",
      "1\n",
      "2\n",
      "3\n",
      "function is finished.\n"
     ]
    }
   ],
   "source": [
    "from collections.abc import Iterable\n",
    "gen = yield_test()\n",
    "\n",
    "# iterable한 객체를 return하기 때문에 looping가능.\n",
    "print(isinstance(gen, Iterable))\n",
    "\n",
    "for x in yield_test():\n",
    "    print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "True\n",
      "funtion is started.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(1, 2, 3)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# iterator를 return하는\n",
    "g = yield_test()\n",
    "print('__iter__' in dir(g))\n",
    "print('__next__' in dir(g))\n",
    "next(g), next(g), next(g)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "resumed\n",
      "2\n",
      "resumed\n",
      "3\n"
     ]
    }
   ],
   "source": [
    "# 함수 수행중 중단 및 재개\n",
    "#sub\n",
    "def yield_test():\n",
    "    yield 1\n",
    "    print('resumed')\n",
    "    yield 2\n",
    "    print('resumed')\n",
    "    yield 3\n",
    "\n",
    "#main\n",
    "for x in yield_test():\n",
    "    print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10\n",
      "20\n"
     ]
    }
   ],
   "source": [
    "# return을 만나거나 함수끝까지 가면 generator는 끝납니다.\n",
    "def gen():\n",
    "    yield 10\n",
    "    yield 20\n",
    "    return\n",
    "    yield 30\n",
    "for x in gen():\n",
    "    print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10\n",
      "20\n",
      "30\n"
     ]
    },
    {
     "ename": "StopIteration",
     "evalue": "30",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mStopIteration\u001b[0m                             Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-29-45d66d7be623>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      9\u001b[0m     \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m \u001b[0mg\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m \u001b[0mnext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mg\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;31m# exception을 잘 살펴보세요.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mStopIteration\u001b[0m: 30"
     ]
    }
   ],
   "source": [
    "# 설계상으로도 그렇고, return문은 의미가 없을 가능성이 높고\n",
    "# (최초의 설계된) generator는 결과값을 받을 수 없었습니다.\n",
    "def gen():\n",
    "    yield 10\n",
    "    yield 20\n",
    "    yield 30\n",
    "    return x\n",
    "for x in gen():\n",
    "    print(x)\n",
    "g = gen()\n",
    "next(g), next(g), next(g), next(g) # exception을 잘 살펴보세요."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# 연습문제: 로또번호자동생성기\n",
    "import random\n",
    "# random.shuffle()\n",
    "\n",
    "def lottery(numbers=45, count=6):\n",
    "    # 0 - cout 사이의 숫자를 random하게 cout번 내어주도록\n",
    "    numbers = list(range(1, numbers+1))\n",
    "    random.shuffle(numbers)\n",
    "    for number in numbers[:count]:\n",
    "        yield number\n",
    "#     return numbers[:count]\n",
    "\n",
    "for random_number in lottery(45):\n",
    "    print(\"And the next number is... %d!\" %(random_number))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# 연습문제: fibonacci\n",
    "# fill in this function\n",
    "def fib():\n",
    "    pass #this is a null statement which does nothing when executed, useful as a placeholder.\n",
    "\n",
    "# testing code\n",
    "import types\n",
    "if type(fib()) == types.GeneratorType:\n",
    "    print(\"Good, The fib function is a generator.\")\n",
    "\n",
    "    counter = 0\n",
    "    for n in fib():\n",
    "        print(n)\n",
    "        counter += 1\n",
    "        if counter == 10:\n",
    "            break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def fib():\n",
    "    val1, val2 = 1, 1\n",
    "    while True:\n",
    "        yield val1\n",
    "        val1, val2 = val2, (val1+val2)\n",
    "\n",
    "# testing code\n",
    "import types\n",
    "if type(fib()) == types.GeneratorType:\n",
    "    print(\"Good, The fib function is a generator.\")\n",
    "\n",
    "    counter = 0\n",
    "    for n in fib():\n",
    "        print(n, end=', ')\n",
    "        counter += 1\n",
    "        if counter == 10:\n",
    "            break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "뭐가 특별한건가요? LIST와 무엇이 다른가요?\n",
    "- 한번 소비되면 끝이에요.\n",
    "- 데이터를 무한으로 생성 가능해요.\n",
    "- LIST보다 메모리를 조금 써요.\n",
    "- LIST사용하는 것보다는 조금 느릴 수 있어요.\n",
    "- len이나 slice를 generator에는 사용할 수 없습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "python3은 laziness가 python3의 주요 테마이자 trend입니다.\n",
    "\n",
    "```\n",
    "# python2\n",
    "range(10), xrange(10)\n",
    "open(file).readlines, open(file).xreadlines\n",
    "{}.keys()\n",
    "\n",
    "# python3\n",
    "range(10)\n",
    "open(file).readlines()\n",
    "{}.keys()\n",
    "```\n",
    "\n",
    "**<u>generator는 쉽게 laziness를 구현할 수 있습니다.</u>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "기본적으로 list를 return하는 함수를 generator로 바꿔서 사용할 수 있습니다.\n",
    "- 한번 쓰고 재사용되지 않을 것이라면\n",
    "- 메모리를 과점유하고 싶지 않다면\n",
    "- database record fetching"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def counter_list(size):\n",
    "    results = []\n",
    "    cur = 1\n",
    "    while cur <= size:\n",
    "        results.append(cur)\n",
    "        cur = cur + 1\n",
    "    return results\n",
    "\n",
    "def counter_gen(size):\n",
    "    cur = 1\n",
    "    while cur <= size:\n",
    "        yield cur\n",
    "        cur = cur + 1\n",
    "\n",
    "list(counter_gen(500)) == counter_list(500)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## GENERATOR EXPRESSION (AGAIN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "[str(num) for num in range(3)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "(str(num) for num in range(3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# generator는 한번 사용하면 버려져요!\n",
    "g = (str(num) for num in range(3))\n",
    "list(g), list(g)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# 함수의 단일인자로 사용할때는 ()가 필요없어집니다.\n",
    "numbers = list(range(10))\n",
    "(\n",
    "    sum([x+2 for x in numbers]),\n",
    "    sum((x+2 for x in numbers)),\n",
    "    sum(x+2 for x in numbers),\n",
    "    sum([x+2 for x in numbers], 5),\n",
    "    sum((x+2 for x in numbers), 5),\n",
    "#     sum(x+2 for x in numbers, 5)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## EXERCISE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# 1. list를 인자로 받아 음수를 filtering하는 generator를 만들어보세요.\n",
    "# 2. generator expression으로 바꿔보세요\n",
    "\n",
    "# ```\n",
    "# >> list(pos_gen(range(-10,10)))\n",
    "# [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]\n",
    "# ```\n",
    "\n",
    "def pos_gen(iterable):\n",
    "    for value in iterable:\n",
    "        if value >= 0:\n",
    "            yield value\n",
    "\n",
    "# list(pos_gen(range(-10,10)))\n",
    "list(value for value in range(-10, 10) if value >= 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# 정답\n",
    "def pos_gen(seq):\n",
    "    for x in seq:\n",
    "        if x >= 0:\n",
    "            yield x\n",
    "list(pos_gen(range(-10,10)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "def pos_gen(seq):\n",
    "    return (x for x in seq if x >= 0)\n",
    "list(pos_gen(range(-10,10)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "pos_gen = lambda seq: filter(lambda x: x>=0, seq)\n",
    "list(pos_gen(range(-10,10)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## EXERCISE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "영어문장을 받아서 각 단어의 앞글자만 대문자로 유지해주는 함수를 만들어봅시다. (문자열의 capitalize method를 사용하세요.)\n",
    "\n",
    "```b\n",
    ">> capwords(\" abc  dEf \")\n",
    "\"Abc Def\"\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Asdf Defaf'"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def capwords(string):\n",
    "    return ' '.join(\n",
    "        word.capitalize()\n",
    "        for word in string.split()\n",
    "    )\n",
    "\n",
    "capwords(\" asdf defaf\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "# Lib/string.py 표준라이브러리에서 \n",
    "# generator expression 사용된 것이 보이시나요?\n",
    "\n",
    "# Capitalize the words in a string, e.g. \" aBc  dEf \" -> \"Abc Def\".\n",
    "def capwords(s, sep=None):\n",
    "    \"\"\"capwords(s [,sep]) -> string\n",
    "\n",
    "    Split the argument into words using split, capitalize each\n",
    "    word using capitalize, and join the capitalized words using\n",
    "    join.  If the optional second argument sep is absent or None,\n",
    "    runs of whitespace characters are replaced by a single space\n",
    "    and leading and trailing whitespace are removed, otherwise\n",
    "    sep is used to split and join the words.\n",
    "\n",
    "    \"\"\"\n",
    "    return (sep or ' ').join(x.capitalize() for x in s.split(sep))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "*1천억건의 데이터를 가진 list를 생성해서 사용하고자 한다. 한개 써먹었는데 갑자기 일이 생겨서(문제가 생겨서) 그만둬야겠다.*\n",
    "![](https://trustliveserve.files.wordpress.com/2013/06/heavy-burden.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "list(range(1_000_000_000_000))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "r = range(1_000_000_000_000)\n",
    "i = iter(r)\n",
    "next(i)\n",
    "next(i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 3672\n",
      "drwxr-xr-x@  4 happytk  staff      136 May 17 22:09 .\n",
      "drwx------@ 49 happytk  staff     1666 May 17 22:08 ..\n"
     ]
    }
   ],
   "source": [
    "def myfunc():\n",
    "    yield \"12345\"\n",
    "    yield \"45678\"\n",
    "    yield \"90123\"\n",
    "\n",
    "# 파일을 읽고 싶은만큼만 읽으려면?\n",
    "# for line in open('data/myfile.txt').readlines()[:3]:\n",
    "# for line in myfunc()[:1]:\n",
    "#    print(line)\n",
    "\n",
    "from itertools import islice\n",
    "for line in islice(open('data/myfile.txt'), 3):\n",
    "    print(line, end='')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "내가 요구할때 <span class=\"burk\">하나씩</span> 주시면 됩니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 메모리를 최대한 적게 사용하고\n",
    "- 가능한 처리를 **늦게까지** 미루는"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<span class=\"mark\">PYTHON에서는 많은 것들이 iterable하고, 사용자도 쉽게 iterable한 데이터를 만들 수 있습니다.</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<span class=\"mark\">PYTHON에서는 많은 것들이 iterable하고 lazy하며, 사용자도 쉽게 iterable하고 lazy한 데이터를 만들 수 있습니다.</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## itertools"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from itertools import (\n",
    "    count,\n",
    "    cycle,\n",
    "    repeat,\n",
    "    chain,\n",
    "    accumulate,\n",
    "    dropwhile,\n",
    "    groupby,\n",
    "    islice,\n",
    "    tee,\n",
    "    takewhile,\n",
    "    zip_longest,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "설명은 도움말을 참고해보아요\n",
    "https://docs.python.org/3.6/library/itertools.html#module-itertools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## yield from \n",
    "pep 380: syntax for delegating to a subgenerator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "여러개의 generator를 하나로 묶어주고 싶을 때"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<generator object gen_3seq at 0x1043ee410>\n",
      "<generator object gen_3seq at 0x1043eee08>\n"
     ]
    }
   ],
   "source": [
    "def gen_3seq(x):\n",
    "    yield x+1\n",
    "    yield x+2\n",
    "    yield x+3\n",
    "\n",
    "def gen_6seq(x):\n",
    "    yield gen_3seq(x)\n",
    "    yield gen_3seq(x+3)\n",
    "    \n",
    "for x in gen_6seq(0):    \n",
    "    print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 2, 3, 4, 5, 6]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 안에서 한번 더 풀어줘야 합니다.\n",
    "def gen_6seq(x):\n",
    "    for val in gen_3seq(x):\n",
    "        yield val\n",
    "    for val in gen_3seq(x+3):\n",
    "        yield val\n",
    "\n",
    "list(gen_6seq(0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# better way, python3.3부터 사용가능합니다.\n",
    "def gen_6seq(x):\n",
    "    yield from gen_3seq(x)\n",
    "    yield from gen_3seq(x+3)\n",
    "\n",
    "list(gen_6seq(0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 2, 3, 4, 5, 6]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from itertools import chain\n",
    "list(chain(gen_3seq(0), gen_3seq(3)))\n",
    "# 이것도 좋은 방법이지만 개별로 예외처리가 어려워요."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "3 is not allowed",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-43-7d5382e34bac>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mitertools\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mchain\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 8\u001b[0;31m \u001b[0mlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mchain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgen_3seq\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgen_3seq\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-43-7d5382e34bac>\u001b[0m in \u001b[0;36mgen_3seq\u001b[0;34m(x)\u001b[0m\n\u001b[1;32m      2\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mv\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mv\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m2\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'3 is not allowed'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      5\u001b[0m         \u001b[0;32myield\u001b[0m \u001b[0mv\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: 3 is not allowed"
     ]
    }
   ],
   "source": [
    "def gen_3seq(x):\n",
    "    for v in range(x, x+3):\n",
    "        if v == 2:\n",
    "            raise ValueError('3 is not allowed')\n",
    "        yield v\n",
    "\n",
    "from itertools import chain\n",
    "list(chain(gen_3seq(0), gen_3seq(3)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 'error', 3, 4, 5]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def gen_3seq(x):\n",
    "    for v in range(x, x+3):\n",
    "        if v == 2:\n",
    "            raise ValueError('3 is not allowed')\n",
    "        yield v\n",
    "\n",
    "def gen_6seq(x):\n",
    "    try:\n",
    "        yield from gen_3seq(x)\n",
    "    except ValueError:\n",
    "        yield 'error'\n",
    "    yield from gen_3seq(x+3)\n",
    "\n",
    "list(gen_6seq(0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "yield from은 이 외에도 복잡한 문제들을 다수 해결하기 위해서 등장했지만, 여기서는 간단한 syntax-sugar로만 일단 다루고 넘어갑니다. 물론 예외처리도 중요한 해결하고자 하는 문제 중 하나였습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## COROUTINE (PEP 492) with asyncio (PEP3153)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "PYTHON3에서 제공하는 ASYNCIO를 이용해서 COROUTINE을 쉽게 생성할 수 있습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import asyncio"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "async def greeting():\n",
    "    print(\"Hello World\")\n",
    "\n",
    "greeting()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "loop = asyncio.get_event_loop()\n",
    "loop.run_until_complete(greeting())\n",
    "loop.close()"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "결과값을 받고 싶다면"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "async def compute(future):\n",
    "    print('starting...')\n",
    "    res = await answer()\n",
    "    future.set_result(res)\n",
    "async def answer():\n",
    "    await asyncio.sleep(1)\n",
    "    return 42\n",
    "\n",
    "f = asyncio.Future()\n",
    "loop = asyncio.get_event_loop()\n",
    "loop.run_until_complete(compute(f))\n",
    "loop.close()\n",
    "f.result()"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}