Kodairaevent_bs.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Kodairaevent_bs.ipynb",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/tom2rd/Googlecolabutils/blob/master/Kodairaevent_bs.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XdsJvMyUiSSU",
        "colab_type": "text"
      },
      "source": [
        "#市役所イベントの抽出"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Smm8F5Eu_Pvk",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import urllib.request\n",
        "from bs4 import BeautifulSoup\n",
        "evurl=\"https://www.city.kodaira.tokyo.jp/event/\"\n",
        "#evurl=\"https://www.city.kodaira.tokyo.jp/event/?q=%7B%22schedule_start%22%3A%222019-09-17T15%3A00%3A00.000Z%22%2C%22schedule_end%22%3A%222020-12-30T15%3A00%3A00.000Z%22%2C%22keyword%22%3A%5B%22%22%5D%2C%22genre%22%3Anull%2C%22entry%22%3Anull%2C%22childcare%22%3Anull%2C%22target%22%3Anull%2C%22sort%22%3A%221%22%7D\"\n",
        "html = urllib.request.urlopen(evurl)\n",
        "soup = BeautifulSoup(html,'html.parser')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ubMoTrHXiaaA",
        "colab_type": "text"
      },
      "source": [
        "詳細情報のURLが含まれる列を抽出\n",
        "\n",
        "（div class=\"event_info\")に含まれるものの中で、a href を抽出"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BtAya-wdHClO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "elems=soup.find_all(\"div\",class_=\"event_info\")\n",
        "for elem in elems:\n",
        "  print(elem.a)  "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tl8EON9C4olA",
        "colab_type": "text"
      },
      "source": [
        "詳細情報のURLのみを抽出"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "dLwTXjbR2V23",
        "colab": {}
      },
      "source": [
        "elems=soup.find_all(\"div\",class_=\"event_info\")\n",
        "for elem in elems:\n",
        "  print(str(elem.a)[9:31])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uUo8AVGZj9Vy",
        "colab_type": "text"
      },
      "source": [
        "詳細情報のURLから、ics形式のファイルのURL抽出"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qPyB4JxN4y1P",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "elems=soup.find_all(\"div\",class_=\"event_info\")\n",
        "for elem in elems:\n",
        "  url2='https://www.city.kodaira.tokyo.jp'+str(elem.a)[9:31]\n",
        "  html2 = urllib.request.urlopen(url2)\n",
        "  soup2 = BeautifulSoup(html2,'html.parser')\n",
        "  elems2 = soup2.find(\"a\", id=\"icaldownload\") \n",
        "  print('https://www.city.kodaira.tokyo.jp'+str(elems2)[21:55])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "npW7nyJ_9-FI",
        "colab_type": "text"
      },
      "source": [
        "icsはhtmlでないため、レスポンスが変わってしまう。\n",
        "\n",
        "そのため、urlopen().read()で中身を読み込んでくる"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TahuOvnO5M__",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "url3='https://www.city.kodaira.tokyo.jp/c_contents/kodaira_event77195.ics'\n",
        "icsfile = urllib.request.urlopen(url3).read()\n",
        "print(str(icsfile).find('BEGIN:VEVENT'))\n",
        "print(str(icsfile).find('END:VEVENT'))\n",
        "print(str(icsfile)[175:1580])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ecv5q1o1N86l",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "# iCalendar用のParser icsを使ってみる\n",
        "\n",
        "https://icspy.readthedocs.io/en/stable/\n",
        "\n",
        "https://icspy.readthedocs.io/en/stable/api.html#event\n",
        "\n",
        "VEVENTの中身だけをとってくる"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sqHIkXnf_BXU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!pip install ics"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kY5100wALqg5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import ics\n",
        "\n",
        "url3='https://www.city.kodaira.tokyo.jp/c_contents/kodaira_event74468.ics'\n",
        "g = urllib.request.urlopen(url3)\n",
        "gcal = ics.Calendar(g.read().decode())\n",
        "print(gcal.events)\n",
        "#print(gcal)\n",
        "print(str(gcal).find(\"BEGIN:VEVENT\"))\n",
        "print(str(gcal).rfind('END:VCALENDAR'))\n",
        "print(str(gcal)[int(str(gcal).find(\"BEGIN:VEVENT\")):int(str(gcal).find('END:VCALENDAR'))])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hAvg9e5jdEql",
        "colab_type": "text"
      },
      "source": [
        "連続動作\n",
        "\n",
        "イベントの概要だけを取ってくるバージョン"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "oBuFcS36dCKY",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 811
        },
        "outputId": "90431afd-7276-4773-bfbc-22fc3ddeb372"
      },
      "source": [
        "elems=soup.find_all(\"div\",class_=\"event_info\")\n",
        "for elem in elems:\n",
        "  url2='https://www.city.kodaira.tokyo.jp'+str(elem.a)[9:31]\n",
        "  html2 = urllib.request.urlopen(url2)\n",
        "  soup2 = BeautifulSoup(html2,'html.parser')\n",
        "  elems2 = soup2.find(\"a\", id=\"icaldownload\") \n",
        "  url3='https://www.city.kodaira.tokyo.jp'+str(elems2)[21:55]\n",
        "  g = urllib.request.urlopen(url3)\n",
        "  gcal = ics.Calendar(g.read().decode())\n",
        "  print(gcal.events)\n",
        "  #print(gcal)\n",
        "  #print(str(gcal).find(\"BEGIN:VEVENT\"))\n",
        "  #print(str(gcal).rfind('END:VCALENDAR'))\n",
        "  #print(str(gcal)[int(str(gcal).find(\"BEGIN:VEVENT\")):int(str(gcal).find('END:VCALENDAR'))])\n"
      ],
      "execution_count": 111,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{<all-day Event '第36回　小平市中学校生徒意見発表会' begin:2019-01-08 end:2019-01-09>}\n",
            "{<all-day Event '令和元年度第2回　男女共同参画推進審議会' begin:2019-08-19 end:2019-08-20>}\n",
            "{<all-day Event '令和元年度　第2回介護保険運営協議会' begin:2019-09-19 end:2019-09-20>}\n",
            "{<all-day Event '12月定例会（最終日）' begin:2018-12-19 end:2018-12-20>}\n",
            "{<all-day Event '9月の夜間納税窓口' begin:2019-09-25 end:2019-09-26>}\n",
            "{<all-day Event '秋のお茶会' begin:2019-11-01 end:2019-11-04>}\n",
            "{<all-day Event '定員到達　キャンセル待ちを受付　歩き方のこつと走る姿勢' begin:2019-09-21 end:2019-09-22>}\n",
            "{<all-day Event '市民と議会の意見交換会' begin:2018-10-29 end:2018-10-30>}\n",
            "{<all-day Event '市民と議会の意見交換会' begin:2018-10-25 end:2018-10-26>}\n",
            "{<all-day Event '市民と議会の意見交換会' begin:2018-10-30 end:2018-10-31>}\n",
            "{<all-day Event '市民と議会の意見交換会(8月4日)' begin:2019-08-04 end:2019-08-05>}\n",
            "{<all-day Event '市民と議会の意見交換会' begin:2018-10-26 end:2018-10-27>}\n",
            "{<all-day Event '医師による健康づくり相談／10月25日（金曜）' begin:2019-10-25 end:2019-10-26>}\n",
            "{<all-day Event '1歳6か月児健康診査／9月5日（木曜）、19日（木曜）' begin:2019-09-19 end:2019-09-20>, <all-day Event '1歳6か月児健康診査／9月5日（木曜）、19日（木曜）' begin:2019-09-05 end:2019-09-06>}\n",
            "{<all-day Event '6月定例会一般質問(6月6日)' begin:2019-06-06 end:2019-06-07>}\n",
            "{<all-day Event '3月定例会（代表質問）' begin:2019-02-26 end:2019-02-27>}\n",
            "{<all-day Event '3月定例会（一般質問）' begin:2019-02-27 end:2019-02-28>}\n",
            "{<all-day Event '12月定例会一般質問(11月29日)' begin:2019-11-29 end:2019-11-30>}\n",
            "{<all-day Event '9月定例会一般質問(9月4日)' begin:2019-09-04 end:2019-09-05>}\n",
            "{<all-day Event '3月定例会（一般質問）' begin:2019-02-28 end:2019-03-01>}\n",
            "{<all-day Event '3月定例会（一般質問）' begin:2019-03-01 end:2019-03-02>}\n",
            "{<all-day Event '6月定例会一般質問(6月7日)' begin:2019-06-07 end:2019-06-08>}\n",
            "{<all-day Event '12月定例会一般質問(11月28日)' begin:2019-11-28 end:2019-11-29>}\n",
            "{<all-day Event '9月定例会一般質問(9月6日)' begin:2019-09-06 end:2019-09-07>}\n",
            "{<all-day Event '6月定例会一般質問(6月5日)' begin:2019-06-05 end:2019-06-06>}\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-111-d40a05478b03>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0melem\u001b[0m \u001b[0;32min\u001b[0m \u001b[0melems\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m   \u001b[0murl2\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'https://www.city.kodaira.tokyo.jp'\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0melem\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m9\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;36m31\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m   \u001b[0mhtml2\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0murllib\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0murlopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0murl2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      5\u001b[0m   \u001b[0msoup2\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mBeautifulSoup\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhtml2\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'html.parser'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m   \u001b[0melems2\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msoup2\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfind\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"a\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mid\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"icaldownload\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/lib/python3.6/urllib/request.py\u001b[0m in \u001b[0;36murlopen\u001b[0;34m(url, data, timeout, cafile, capath, cadefault, context)\u001b[0m\n\u001b[1;32m    221\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    222\u001b[0m         \u001b[0mopener\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_opener\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 223\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mopener\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdata\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    224\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    225\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0minstall_opener\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mopener\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/lib/python3.6/urllib/request.py\u001b[0m in \u001b[0;36mopen\u001b[0;34m(self, fullurl, data, timeout)\u001b[0m\n\u001b[1;32m    524\u001b[0m             \u001b[0mreq\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmeth\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mreq\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    525\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 526\u001b[0;31m         \u001b[0mresponse\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_open\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mreq\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdata\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    527\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    528\u001b[0m         \u001b[0;31m# post-process response\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/lib/python3.6/urllib/request.py\u001b[0m in \u001b[0;36m_open\u001b[0;34m(self, req, data)\u001b[0m\n\u001b[1;32m    542\u001b[0m         \u001b[0mprotocol\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mreq\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtype\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    543\u001b[0m         result = self._call_chain(self.handle_open, protocol, protocol +\n\u001b[0;32m--> 544\u001b[0;31m                                   '_open', req)\n\u001b[0m\u001b[1;32m    545\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    546\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/lib/python3.6/urllib/request.py\u001b[0m in \u001b[0;36m_call_chain\u001b[0;34m(self, chain, kind, meth_name, *args)\u001b[0m\n\u001b[1;32m    502\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mhandler\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mhandlers\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    503\u001b[0m             \u001b[0mfunc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgetattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhandler\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmeth_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 504\u001b[0;31m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    505\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mresult\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    506\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/lib/python3.6/urllib/request.py\u001b[0m in \u001b[0;36mhttps_open\u001b[0;34m(self, req)\u001b[0m\n\u001b[1;32m   1359\u001b[0m         \u001b[0;32mdef\u001b[0m \u001b[0mhttps_open\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreq\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1360\u001b[0m             return self.do_open(http.client.HTTPSConnection, req,\n\u001b[0;32m-> 1361\u001b[0;31m                 context=self._context, check_hostname=self._check_hostname)\n\u001b[0m\u001b[1;32m   1362\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1363\u001b[0m         \u001b[0mhttps_request\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mAbstractHTTPHandler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdo_request_\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/lib/python3.6/urllib/request.py\u001b[0m in \u001b[0;36mdo_open\u001b[0;34m(self, http_class, req, **http_conn_args)\u001b[0m\n\u001b[1;32m   1316\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1317\u001b[0m                 h.request(req.get_method(), req.selector, req.data, headers,\n\u001b[0;32m-> 1318\u001b[0;31m                           encode_chunked=req.has_header('Transfer-encoding'))\n\u001b[0m\u001b[1;32m   1319\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mOSError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0merr\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;31m# timeout error\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1320\u001b[0m                 \u001b[0;32mraise\u001b[0m \u001b[0mURLError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0merr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/lib/python3.6/http/client.py\u001b[0m in \u001b[0;36mrequest\u001b[0;34m(self, method, url, body, headers, encode_chunked)\u001b[0m\n\u001b[1;32m   1237\u001b[0m                 encode_chunked=False):\n\u001b[1;32m   1238\u001b[0m         \u001b[0;34m\"\"\"Send a complete request to the server.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1239\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_send_request\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmethod\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbody\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mheaders\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mencode_chunked\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1240\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1241\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_send_request\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmethod\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbody\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mheaders\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mencode_chunked\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/lib/python3.6/http/client.py\u001b[0m in \u001b[0;36m_send_request\u001b[0;34m(self, method, url, body, headers, encode_chunked)\u001b[0m\n\u001b[1;32m   1283\u001b[0m             \u001b[0;31m# default charset of iso-8859-1.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1284\u001b[0m             \u001b[0mbody\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_encode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbody\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'body'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1285\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mendheaders\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbody\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mencode_chunked\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mencode_chunked\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1286\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1287\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mgetresponse\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/lib/python3.6/http/client.py\u001b[0m in \u001b[0;36mendheaders\u001b[0;34m(self, message_body, encode_chunked)\u001b[0m\n\u001b[1;32m   1232\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1233\u001b[0m             \u001b[0;32mraise\u001b[0m \u001b[0mCannotSendHeader\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1234\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_send_output\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmessage_body\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mencode_chunked\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mencode_chunked\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1235\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1236\u001b[0m     def request(self, method, url, body=None, headers={}, *,\n",
            "\u001b[0;32m/usr/lib/python3.6/http/client.py\u001b[0m in \u001b[0;36m_send_output\u001b[0;34m(self, message_body, encode_chunked)\u001b[0m\n\u001b[1;32m   1024\u001b[0m         \u001b[0mmsg\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34mb\"\\r\\n\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_buffer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1025\u001b[0m         \u001b[0;32mdel\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_buffer\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1026\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmsg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1027\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1028\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mmessage_body\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/lib/python3.6/http/client.py\u001b[0m in \u001b[0;36msend\u001b[0;34m(self, data)\u001b[0m\n\u001b[1;32m    962\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msock\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    963\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mauto_open\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 964\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconnect\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    965\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    966\u001b[0m                 \u001b[0;32mraise\u001b[0m \u001b[0mNotConnected\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/lib/python3.6/http/client.py\u001b[0m in \u001b[0;36mconnect\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m   1398\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1399\u001b[0m             self.sock = self._context.wrap_socket(self.sock,\n\u001b[0;32m-> 1400\u001b[0;31m                                                   server_hostname=server_hostname)\n\u001b[0m\u001b[1;32m   1401\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_context\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcheck_hostname\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_check_hostname\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1402\u001b[0m                 \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/lib/python3.6/ssl.py\u001b[0m in \u001b[0;36mwrap_socket\u001b[0;34m(self, sock, server_side, do_handshake_on_connect, suppress_ragged_eofs, server_hostname, session)\u001b[0m\n\u001b[1;32m    405\u001b[0m                          \u001b[0msuppress_ragged_eofs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msuppress_ragged_eofs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    406\u001b[0m                          \u001b[0mserver_hostname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mserver_hostname\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 407\u001b[0;31m                          _context=self, _session=session)\n\u001b[0m\u001b[1;32m    408\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    409\u001b[0m     def wrap_bio(self, incoming, outgoing, server_side=False,\n",
            "\u001b[0;32m/usr/lib/python3.6/ssl.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, sock, keyfile, certfile, server_side, cert_reqs, ssl_version, ca_certs, do_handshake_on_connect, family, type, proto, fileno, suppress_ragged_eofs, npn_protocols, ciphers, server_hostname, _context, _session)\u001b[0m\n\u001b[1;32m    815\u001b[0m                         \u001b[0;31m# non-blocking\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    816\u001b[0m                         \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"do_handshake_on_connect should not be specified for non-blocking sockets\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 817\u001b[0;31m                     \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdo_handshake\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    818\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    819\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mOSError\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/lib/python3.6/ssl.py\u001b[0m in \u001b[0;36mdo_handshake\u001b[0;34m(self, block)\u001b[0m\n\u001b[1;32m   1075\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mtimeout\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m0.0\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mblock\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1076\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msettimeout\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1077\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_sslobj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdo_handshake\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1078\u001b[0m         \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1079\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msettimeout\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/lib/python3.6/ssl.py\u001b[0m in \u001b[0;36mdo_handshake\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    687\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mdo_handshake\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    688\u001b[0m         \u001b[0;34m\"\"\"Start the SSL/TLS handshake.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 689\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_sslobj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdo_handshake\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    690\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcontext\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcheck_hostname\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    691\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mserver_hostname\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "al3l4rkic9Fx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "elems=soup.find_all(\"div\",class_=\"event_info\")\n",
        "eventcount=0\n",
        "for elem in elems:\n",
        "  url2='https://www.city.kodaira.tokyo.jp'+str(elem.a)[9:31]\n",
        "  html2 = urllib.request.urlopen(url2)\n",
        "  soup2 = BeautifulSoup(html2,'html.parser')\n",
        "  elems2 = soup2.find(\"a\", id=\"icaldownload\") \n",
        "  url3='https://www.city.kodaira.tokyo.jp'+str(elems2)[21:55]\n",
        "  g = urllib.request.urlopen(url3)\n",
        "  gcal = ics.Calendar(g.read().decode())\n",
        "  if eventcount==0:\n",
        "    eventallstr=str(gcal)[0:int(str(gcal).find('END:VCALENDAR'))]\n",
        "    eventcount=eventcount+1\n",
        "  else:  \n",
        "    eventallstr=eventallstr+\"\\n\"+str(gcal)[int(str(gcal).find(\"BEGIN:VEVENT\")):int(str(gcal).find('END:VCALENDAR'))]\n",
        "    eventcount=eventcount+1\n",
        "    print(eventcount)\n",
        "  eventallstr\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xp33Hjg8mcLP",
        "colab_type": "text"
      },
      "source": [
        "#google drive へ保存\n",
        "https://qiita.com/howakuro/items/c2ba1cbd6d338fce867f\n",
        "\n",
        "https://qiita.com/kakinaguru_zo/items/33dbe24276915124f545"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VHpIljBRnSqx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qzfmnDFcrccC",
        "colab_type": "text"
      },
      "source": [
        "やっぱりメンドクサイのでファイルを書いてダウンロードした\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iVfG4iXvkq3W",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from google.colab import files\n",
        "\n",
        "with open(\"kodairacal.ics\", \"w\") as f:\n",
        "  f.write(eventallstr+\"END:VCALENDAR\")\n",
        "\n",
        "files.download('kodairacal.ics')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DAX3sNikrnCK",
        "colab_type": "text"
      },
      "source": [
        "#まとめ　\n",
        "全部をまとめて書くと、こうなる\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3W5azIEkrwPH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import urllib.request\n",
        "from bs4 import BeautifulSoup\n",
        "import ics\n",
        "from google.colab import files\n",
        "\n",
        "evurl=\"https://www.city.kodaira.tokyo.jp/event/\"\n",
        "html = urllib.request.urlopen(evurl)\n",
        "soup = BeautifulSoup(html,'html.parser')\n",
        "\n",
        "#イベントインフォだけを取ってくる\n",
        "elems=soup.find_all(\"div\",class_=\"event_info\")\n",
        "eventcount=0\n",
        "\n",
        "for elem in elems:\n",
        "#イベントの詳細URLへアクセス\n",
        "  url2='https://www.city.kodaira.tokyo.jp'+str(elem.a)[9:31]\n",
        "  html2 = urllib.request.urlopen(url2)\n",
        "  soup2 = BeautifulSoup(html2,'html.parser')\n",
        "  elems2 = soup2.find(\"a\", id=\"icaldownload\") \n",
        "#icsファイルへアクセス\n",
        "  url3='https://www.city.kodaira.tokyo.jp'+str(elems2)[21:55]\n",
        "  g = urllib.request.urlopen(url3)\n",
        "  gcal = ics.Calendar(g.read().decode())\n",
        "  if eventcount==0:\n",
        "    eventallstr=str(gcal)[0:int(str(gcal).find('END:VCALENDAR'))]\n",
        "    eventcount=eventcount+1\n",
        "  else:  \n",
        "    eventallstr=eventallstr+\"\\n\"+str(gcal)[int(str(gcal).find(\"BEGIN:VEVENT\")):int(str(gcal).find('END:VCALENDAR'))]\n",
        "    eventcount=eventcount+1\n",
        "    print(eventcount)\n",
        "\n",
        "print(\"Complited\")\n",
        "\n",
        "#ファイルへ書き出されダウンロードされる\n",
        "with open(\"kodairacal.ics\", \"w\") as f:\n",
        "  f.write(eventallstr+\"END:VCALENDAR\")\n",
        "\n",
        "files.download('kodairacal.ics')"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}