memoriam_scraper.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scraper for collecting Belgium deaths\n",
    "## 1. Background\n",
    "This is a program that scrapes the reported deaths in Belgium publically available on inmemoriam.be. This scraper was made to get an idea of the impact of COVID-19 on the Belgian death rate.\n",
    "Parameters in this scraper are set to collect all deaths from the 1st of January 2016 up until the 1st of January 2021. As the last time this scraper was run was on the 18th of October 2020, data for the final months is not yet included.\n",
    "The collected data consists out of the name of the deceased (`name`), the age of the deceased (`age`), the date the person passed away (`date`), the place of death (`location`) as well as separate columns for the month (`month`), week (`week`) and day (`day`) of the year. \n",
    "\n",
    "## 2. Necessary packages\n",
    "In order to build the scraper, the following packages were imported and used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from pandas import Series, DataFrame\n",
    "import datetime as dt\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Scraper\n",
    "### 3.1 Set parameters\n",
    "First the parameters for the scraper are set to include the desired date range. As the website separates these in several pages, adjustments are made in order to collect the total number of pages and extract all the required data from each page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "current_page = 1\n",
    "begin_date = \"2016-01-01\"\n",
    "end_date = \"2021-01-01\"\n",
    "url = \"https://www.inmemoriam.be/nl/rouwberichten/?page=\" + str(current_page) + \"&filter=&periodStart=\" + str(begin_date) + \"&periodEnd=\" + str(end_date) + \"&yearOfBirth=&undertakerId=&placeOfResidence=&provinceId=&newsPaper=&obituary=1\"\n",
    "page = requests.get(url)\n",
    "soup = BeautifulSoup(page.content, 'html.parser')\n",
    "\n",
    "page_list = soup.find_all('a', class_ = 'c-pagination__item')\n",
    "\n",
    "pages = []\n",
    "\n",
    "for page in page_list:\n",
    "    page = page.get_text()\n",
    "    pages.append(page)\n",
    "\n",
    "end_page = pages[-2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Collect the data\n",
    "Next the scraper is run using the parameters set above, collecting `name`, `age`, `date` and `location` for each deceased person in the set date range. The result is stored in a dataframe called `df`. With there being over 5600 pages to be scraped, this process will take a long time to finish."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": [
     "outputPrepend"
    ]
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      ".\n",
      "Page  352  is being scraped now.\n",
      "Page  353  is being scraped now.\n",
      "Page  354  is being scraped now.\n",
      "Page  355  is being scraped now.\n",
      "Page  356  is being scraped now.\n",
      "Page  357  is being scraped now.\n",
      "Page  358  is being scraped now.\n",
      "Page  359  is being scraped now.\n",
      "Page  360  is being scraped now.\n",
      "Page  361  is being scraped now.\n",
      "Page  362  is being scraped now.\n",
      "Page  363  is being scraped now.\n",
      "Page  364  is being scraped now.\n",
      "Page  365  is being scraped now.\n",
      "Page  366  is being scraped now.\n",
      "Page  367  is being scraped now.\n",
      "Page  368  is being scraped now.\n",
      "Page  369  is being scraped now.\n",
      "Page  370  is being scraped now.\n",
      "Page  371  is being scraped now.\n",
      "Page  372  is being scraped now.\n",
      "Page  373  is being scraped now.\n",
      "Page  374  is being scraped now.\n",
      "Page  375  is being scraped now.\n",
      "Page  376  is being scraped now.\n",
      "Page  377  is being scraped now.\n",
      "Page  378  is being scraped now.\n",
      "Page  379  is being scraped now.\n",
      "Page  380  is being scraped now.\n",
      "Page  381  is being scraped now.\n",
      "Page  382  is being scraped now.\n",
      "Page  383  is being scraped now.\n",
      "Page  384  is being scraped now.\n",
      "Page  385  is being scraped now.\n",
      "Page  386  is being scraped now.\n",
      "Page  387  is being scraped now.\n",
      "Page  388  is being scraped now.\n",
      "Page  389  is being scraped now.\n",
      "Page  390  is being scraped now.\n",
      "Page  391  is being scraped now.\n",
      "Page  392  is being scraped now.\n",
      "Page  393  is being scraped now.\n",
      "Page  394  is being scraped now.\n",
      "Page  395  is being scraped now.\n",
      "Page  396  is being scraped now.\n",
      "Page  397  is being scraped now.\n",
      "Page  398  is being scraped now.\n",
      "Page  399  is being scraped now.\n",
      "Page  400  is being scraped now.\n",
      "Page  401  is being scraped now.\n",
      "Page  402  is being scraped now.\n",
      "Page  403  is being scraped now.\n",
      "Page  404  is being scraped now.\n",
      "Page  405  is being scraped now.\n",
      "Page  406  is being scraped now.\n",
      "Page  407  is being scraped now.\n",
      "Page  408  is being scraped now.\n",
      "Page  409  is being scraped now.\n",
      "Page  410  is being scraped now.\n",
      "Page  411  is being scraped now.\n",
      "Page  412  is being scraped now.\n",
      "Page  413  is being scraped now.\n",
      "Page  414  is being scraped now.\n",
      "Page  415  is being scraped now.\n",
      "Page  416  is being scraped now.\n",
      "Page  417  is being scraped now.\n",
      "Page  418  is being scraped now.\n",
      "Page  419  is being scraped now.\n",
      "Page  420  is being scraped now.\n",
      "Page  421  is being scraped now.\n",
      "Page  422  is being scraped now.\n",
      "Page  423  is being scraped now.\n",
      "Page  424  is being scraped now.\n",
      "Page  425  is being scraped now.\n",
      "Page  426  is being scraped now.\n",
      "Page  427  is being scraped now.\n",
      "Page  428  is being scraped now.\n",
      "Page  429  is being scraped now.\n",
      "Page  430  is being scraped now.\n",
      "Page  431  is being scraped now.\n",
      "Page  432  is being scraped now.\n",
      "Page  433  is being scraped now.\n",
      "Page  434  is being scraped now.\n",
      "Page  435  is being scraped now.\n",
      "Page  436  is being scraped now.\n",
      "Page  437  is being scraped now.\n",
      "Page  438  is being scraped now.\n",
      "Page  439  is being scraped now.\n",
      "Page  440  is being scraped now.\n",
      "Page  441  is being scraped now.\n",
      "Page  442  is being scraped now.\n",
      "Page  443  is being scraped now.\n",
      "Page  444  is being scraped now.\n",
      "Page  445  is being scraped now.\n",
      "Page  446  is being scraped now.\n",
      "Page  447  is being scraped now.\n",
      "Page  448  is being scraped now.\n",
      "Page  449  is being scraped now.\n",
      "Page  450  is being scraped now.\n",
      "Page  451  is being scraped now.\n",
      "Page  452  is being scraped now.\n",
      "Page  453  is being scraped now.\n",
      "Page  454  is being scraped now.\n",
      "Page  455  is being scraped now.\n",
      "Page  456  is being scraped now.\n",
      "Page  457  is being scraped now.\n",
      "Page  458  is being scraped now.\n",
      "Page  459  is being scraped now.\n",
      "Page  460  is being scraped now.\n",
      "Page  461  is being scraped now.\n",
      "Page  462  is being scraped now.\n",
      "Page  463  is being scraped now.\n",
      "Page  464  is being scraped now.\n",
      "Page  465  is being scraped now.\n",
      "Page  466  is being scraped now.\n",
      "Page  467  is being scraped now.\n",
      "Page  468  is being scraped now.\n",
      "Page  469  is being scraped now.\n",
      "Page  470  is being scraped now.\n",
      "Page  471  is being scraped now.\n",
      "Page  472  is being scraped now.\n",
      "Page  473  is being scraped now.\n",
      "Page  474  is being scraped now.\n",
      "Page  475  is being scraped now.\n",
      "Page  476  is being scraped now.\n",
      "Page  477  is being scraped now.\n",
      "Page  478  is being scraped now.\n",
      "Page  479  is being scraped now.\n",
      "Page  480  is being scraped now.\n",
      "Page  481  is being scraped now.\n",
      "Page  482  is being scraped now.\n",
      "Page  483  is being scraped now.\n",
      "Page  484  is being scraped now.\n",
      "Page  485  is being scraped now.\n",
      "Page  486  is being scraped now.\n",
      "Page  487  is being scraped now.\n",
      "Page  488  is being scraped now.\n",
      "Page  489  is being scraped now.\n",
      "Page  490  is being scraped now.\n",
      "Page  491  is being scraped now.\n",
      "Page  492  is being scraped now.\n",
      "Page  493  is being scraped now.\n",
      "Page  494  is being scraped now.\n",
      "Page  495  is being scraped now.\n",
      "Page  496  is being scraped now.\n",
      "Page  497  is being scraped now.\n",
      "Page  498  is being scraped now.\n",
      "Page  499  is being scraped now.\n",
      "Page  500  is being scraped now.\n",
      "Page  501  is being scraped now.\n",
      "Page  502  is being scraped now.\n",
      "Page  503  is being scraped now.\n",
      "Page  504  is being scraped now.\n",
      "Page  505  is being scraped now.\n",
      "Page  506  is being scraped now.\n",
      "Page  507  is being scraped now.\n",
      "Page  508  is being scraped now.\n",
      "Page  509  is being scraped now.\n",
      "Page  510  is being scraped now.\n",
      "Page  511  is being scraped now.\n",
      "Page  512  is being scraped now.\n",
      "Page  513  is being scraped now.\n",
      "Page  514  is being scraped now.\n",
      "Page  515  is being scraped now.\n",
      "Page  516  is being scraped now.\n",
      "Page  517  is being scraped now.\n",
      "Page  518  is being scraped now.\n",
      "Page  519  is being scraped now.\n",
      "Page  520  is being scraped now.\n",
      "Page  521  is being scraped now.\n",
      "Page  522  is being scraped now.\n",
      "Page  523  is being scraped now.\n",
      "Page  524  is being scraped now.\n",
      "Page  525  is being scraped now.\n",
      "Page  526  is being scraped now.\n",
      "Page  527  is being scraped now.\n",
      "Page  528  is being scraped now.\n",
      "Page  529  is being scraped now.\n",
      "Page  530  is being scraped now.\n",
      "Page  531  is being scraped now.\n",
      "Page  532  is being scraped now.\n",
      "Page  533  is being scraped now.\n",
      "Page  534  is being scraped now.\n",
      "Page  535  is being scraped now.\n",
      "Page  536  is being scraped now.\n",
      "Page  537  is being scraped now.\n",
      "Page  538  is being scraped now.\n",
      "Page  539  is being scraped now.\n",
      "Page  540  is being scraped now.\n",
      "Page  541  is being scraped now.\n",
      "Page  542  is being scraped now.\n",
      "Page  543  is being scraped now.\n",
      "Page  544  is being scraped now.\n",
      "Page  545  is being scraped now.\n",
      "Page  546  is being scraped now.\n",
      "Page  547  is being scraped now.\n",
      "Page  548  is being scraped now.\n",
      "Page  549  is being scraped now.\n",
      "Page  550  is being scraped now.\n",
      "Page  551  is being scraped now.\n",
      "Page  552  is being scraped now.\n",
      "Page  553  is being scraped now.\n",
      "Page  554  is being scraped now.\n",
      "Page  555  is being scraped now.\n",
      "Page  556  is being scraped now.\n",
      "Page  557  is being scraped now.\n",
      "Page  558  is being scraped now.\n",
      "Page  559  is being scraped now.\n",
      "Page  560  is being scraped now.\n",
      "Page  561  is being scraped now.\n",
      "Page  562  is being scraped now.\n",
      "Page  563  is being scraped now.\n",
      "Page  564  is being scraped now.\n",
      "Page  565  is being scraped now.\n",
      "Page  566  is being scraped now.\n",
      "Page  567  is being scraped now.\n",
      "Page  568  is being scraped now.\n",
      "Page  569  is being scraped now.\n",
      "Page  570  is being scraped now.\n",
      "Page  571  is being scraped now.\n",
      "Page  572  is being scraped now.\n",
      "Page  573  is being scraped now.\n",
      "Page  574  is being scraped now.\n",
      "Page  575  is being scraped now.\n",
      "Page  576  is being scraped now.\n",
      "Page  577  is being scraped now.\n",
      "Page  578  is being scraped now.\n",
      "Page  579  is being scraped now.\n",
      "Page  580  is being scraped now.\n",
      "Page  581  is being scraped now.\n",
      "Page  582  is being scraped now.\n",
      "Page  583  is being scraped now.\n",
      "Page  584  is being scraped now.\n",
      "Page  585  is being scraped now.\n",
      "Page  586  is being scraped now.\n",
      "Page  587  is being scraped now.\n",
      "Page  588  is being scraped now.\n",
      "Page  589  is being scraped now.\n",
      "Page  590  is being scraped now.\n",
      "Page  591  is being scraped now.\n",
      "Page  592  is being scraped now.\n",
      "Page  593  is being scraped now.\n",
      "Page  594  is being scraped now.\n",
      "Page  595  is being scraped now.\n",
      "Page  596  is being scraped now.\n",
      "Page  597  is being scraped now.\n",
      "Page  598  is being scraped now.\n",
      "Page  599  is being scraped now.\n",
      "Page  600  is being scraped now.\n",
      "Page  601  is being scraped now.\n",
      "Page  602  is being scraped now.\n",
      "Page  603  is being scraped now.\n",
      "Page  604  is being scraped now.\n",
      "Page  605  is being scraped now.\n",
      "Page  606  is being scraped now.\n",
      "Page  607  is being scraped now.\n",
      "Page  608  is being scraped now.\n",
      "Page  609  is being scraped now.\n",
      "Page  610  is being scraped now.\n",
      "Page  611  is being scraped now.\n",
      "Page  612  is being scraped now.\n",
      "Page  613  is being scraped now.\n",
      "Page  614  is being scraped now.\n",
      "Page  615  is being scraped now.\n",
      "Page  616  is being scraped now.\n",
      "Page  617  is being scraped now.\n",
      "Page  618  is being scraped now.\n",
      "Page  619  is being scraped now.\n",
      "Page  620  is being scraped now.\n",
      "Page  621  is being scraped now.\n",
      "Page  622  is being scraped now.\n",
      "Page  623  is being scraped now.\n",
      "Page  624  is being scraped now.\n",
      "Page  625  is being scraped now.\n",
      "Page  626  is being scraped now.\n",
      "Page  627  is being scraped now.\n",
      "Page  628  is being scraped now.\n",
      "Page  629  is being scraped now.\n",
      "Page  630  is being scraped now.\n",
      "Page  631  is being scraped now.\n",
      "Page  632  is being scraped now.\n",
      "Page  633  is being scraped now.\n",
      "Page  634  is being scraped now.\n",
      "Page  635  is being scraped now.\n",
      "Page  636  is being scraped now.\n",
      "Page  637  is being scraped now.\n",
      "Page  638  is being scraped now.\n",
      "Page  639  is being scraped now.\n",
      "Page  640  is being scraped now.\n",
      "Page  641  is being scraped now.\n",
      "Page  642  is being scraped now.\n",
      "Page  643  is being scraped now.\n",
      "Page  644  is being scraped now.\n",
      "Page  645  is being scraped now.\n",
      "Page  646  is being scraped now.\n",
      "Page  647  is being scraped now.\n",
      "Page  648  is being scraped now.\n",
      "Page  649  is being scraped now.\n",
      "Page  650  is being scraped now.\n",
      "Page  651  is being scraped now.\n",
      "Page  652  is being scraped now.\n",
      "Page  653  is being scraped now.\n",
      "Page  654  is being scraped now.\n",
      "Page  655  is being scraped now.\n",
      "Page  656  is being scraped now.\n",
      "Page  657  is being scraped now.\n",
      "Page  658  is being scraped now.\n",
      "Page  659  is being scraped now.\n",
      "Page  660  is being scraped now.\n",
      "Page  661  is being scraped now.\n",
      "Page  662  is being scraped now.\n",
      "Page  663  is being scraped now.\n",
      "Page  664  is being scraped now.\n",
      "Page  665  is being scraped now.\n",
      "Page  666  is being scraped now.\n",
      "Page  667  is being scraped now.\n",
      "Page  668  is being scraped now.\n",
      "Page  669  is being scraped now.\n",
      "Page  670  is being scraped now.\n",
      "Page  671  is being scraped now.\n",
      "Page  672  is being scraped now.\n",
      "Page  673  is being scraped now.\n",
      "Page  674  is being scraped now.\n",
      "Page  675  is being scraped now.\n",
      "Page  676  is being scraped now.\n",
      "Page  677  is being scraped now.\n",
      "Page  678  is being scraped now.\n",
      "Page  679  is being scraped now.\n",
      "Page  680  is being scraped now.\n",
      "Page  681  is being scraped now.\n",
      "Page  682  is being scraped now.\n",
      "Page  683  is being scraped now.\n",
      "Page  684  is being scraped now.\n",
      "Page  685  is being scraped now.\n",
      "Page  686  is being scraped now.\n",
      "Page  687  is being scraped now.\n",
      "Page  688  is being scraped now.\n",
      "Page  689  is being scraped now.\n",
      "Page  690  is being scraped now.\n",
      "Page  691  is being scraped now.\n",
      "Page  692  is being scraped now.\n",
      "Page  693  is being scraped now.\n",
      "Page  694  is being scraped now.\n",
      "Page  695  is being scraped now.\n",
      "Page  696  is being scraped now.\n",
      "Page  697  is being scraped now.\n",
      "Page  698  is being scraped now.\n",
      "Page  699  is being scraped now.\n",
      "Page  700  is being scraped now.\n",
      "Page  701  is being scraped now.\n",
      "Page  702  is being scraped now.\n",
      "Page  703  is being scraped now.\n",
      "Page  704  is being scraped now.\n",
      "Page  705  is being scraped now.\n",
      "Page  706  is being scraped now.\n",
      "Page  707  is being scraped now.\n",
      "Page  708  is being scraped now.\n",
      "Page  709  is being scraped now.\n",
      "Page  710  is being scraped now.\n",
      "Page  711  is being scraped now.\n",
      "Page  712  is being scraped now.\n",
      "Page  713  is being scraped now.\n",
      "Page  714  is being scraped now.\n",
      "Page  715  is being scraped now.\n",
      "Page  716  is being scraped now.\n",
      "Page  717  is being scraped now.\n",
      "Page  718  is being scraped now.\n",
      "Page  719  is being scraped now.\n",
      "Page  720  is being scraped now.\n",
      "Page  721  is being scraped now.\n",
      "Page  722  is being scraped now.\n",
      "Page  723  is being scraped now.\n",
      "Page  724  is being scraped now.\n",
      "Page  725  is being scraped now.\n",
      "Page  726  is being scraped now.\n",
      "Page  727  is being scraped now.\n",
      "Page  728  is being scraped now.\n",
      "Page  729  is being scraped now.\n",
      "Page  730  is being scraped now.\n",
      "Page  731  is being scraped now.\n",
      "Page  732  is being scraped now.\n",
      "Page  733  is being scraped now.\n",
      "Page  734  is being scraped now.\n",
      "Page  735  is being scraped now.\n",
      "Page  736  is being scraped now.\n",
      "Page  737  is being scraped now.\n",
      "Page  738  is being scraped now.\n",
      "Page  739  is being scraped now.\n",
      "Page  740  is being scraped now.\n",
      "Page  741  is being scraped now.\n",
      "Page  742  is being scraped now.\n",
      "Page  743  is being scraped now.\n",
      "Page  744  is being scraped now.\n",
      "Page  745  is being scraped now.\n",
      "Page  746  is being scraped now.\n",
      "Page  747  is being scraped now.\n",
      "Page  748  is being scraped now.\n",
      "Page  749  is being scraped now.\n",
      "Page  750  is being scraped now.\n",
      "Page  751  is being scraped now.\n",
      "Page  752  is being scraped now.\n",
      "Page  753  is being scraped now.\n",
      "Page  754  is being scraped now.\n",
      "Page  755  is being scraped now.\n",
      "Page  756  is being scraped now.\n",
      "Page  757  is being scraped now.\n",
      "Page  758  is being scraped now.\n",
      "Page  759  is being scraped now.\n",
      "Page  760  is being scraped now.\n",
      "Page  761  is being scraped now.\n",
      "Page  762  is being scraped now.\n",
      "Page  763  is being scraped now.\n",
      "Page  764  is being scraped now.\n",
      "Page  765  is being scraped now.\n",
      "Page  766  is being scraped now.\n",
      "Page  767  is being scraped now.\n",
      "Page  768  is being scraped now.\n",
      "Page  769  is being scraped now.\n",
      "Page  770  is being scraped now.\n",
      "Page  771  is being scraped now.\n",
      "Page  772  is being scraped now.\n",
      "Page  773  is being scraped now.\n",
      "Page  774  is being scraped now.\n",
      "Page  775  is being scraped now.\n",
      "Page  776  is being scraped now.\n",
      "Page  777  is being scraped now.\n",
      "Page  778  is being scraped now.\n",
      "Page  779  is being scraped now.\n",
      "Page  780  is being scraped now.\n",
      "Page  781  is being scraped now.\n",
      "Page  782  is being scraped now.\n",
      "Page  783  is being scraped now.\n",
      "Page  784  is being scraped now.\n",
      "Page  785  is being scraped now.\n",
      "Page  786  is being scraped now.\n",
      "Page  787  is being scraped now.\n",
      "Page  788  is being scraped now.\n",
      "Page  789  is being scraped now.\n",
      "Page  790  is being scraped now.\n",
      "Page  791  is being scraped now.\n",
      "Page  792  is being scraped now.\n",
      "Page  793  is being scraped now.\n",
      "Page  794  is being scraped now.\n",
      "Page  795  is being scraped now.\n",
      "Page  796  is being scraped now.\n",
      "Page  797  is being scraped now.\n",
      "Page  798  is being scraped now.\n",
      "Page  799  is being scraped now.\n",
      "Page  800  is being scraped now.\n",
      "Page  801  is being scraped now.\n",
      "Page  802  is being scraped now.\n",
      "Page  803  is being scraped now.\n",
      "Page  804  is being scraped now.\n",
      "Page  805  is being scraped now.\n",
      "Page  806  is being scraped now.\n",
      "Page  807  is being scraped now.\n",
      "Page  808  is being scraped now.\n",
      "Page  809  is being scraped now.\n",
      "Page  810  is being scraped now.\n",
      "Page  811  is being scraped now.\n",
      "Page  812  is being scraped now.\n",
      "Page  813  is being scraped now.\n",
      "Page  814  is being scraped now.\n",
      "Page  815  is being scraped now.\n",
      "Page  816  is being scraped now.\n",
      "Page  817  is being scraped now.\n",
      "Page  818  is being scraped now.\n",
      "Page  819  is being scraped now.\n",
      "Page  820  is being scraped now.\n",
      "Page  821  is being scraped now.\n",
      "Page  822  is being scraped now.\n",
      "Page  823  is being scraped now.\n",
      "Page  824  is being scraped now.\n",
      "Page  825  is being scraped now.\n",
      "Page  826  is being scraped now.\n",
      "Page  827  is being scraped now.\n",
      "Page  828  is being scraped now.\n",
      "Page  829  is being scraped now.\n",
      "Page  830  is being scraped now.\n",
      "Page  831  is being scraped now.\n",
      "Page  832  is being scraped now.\n",
      "Page  833  is being scraped now.\n",
      "Page  834  is being scraped now.\n",
      "Page  835  is being scraped now.\n",
      "Page  836  is being scraped now.\n",
      "Page  837  is being scraped now.\n",
      "Page  838  is being scraped now.\n",
      "Page  839  is being scraped now.\n",
      "Page  840  is being scraped now.\n",
      "Page  841  is being scraped now.\n",
      "Page  842  is being scraped now.\n",
      "Page  843  is being scraped now.\n",
      "Page  844  is being scraped now.\n",
      "Page  845  is being scraped now.\n",
      "Page  846  is being scraped now.\n",
      "Page  847  is being scraped now.\n",
      "Page  848  is being scraped now.\n",
      "Page  849  is being scraped now.\n",
      "Page  850  is being scraped now.\n",
      "Page  851  is being scraped now.\n",
      "Page  852  is being scraped now.\n",
      "Page  853  is being scraped now.\n",
      "Page  854  is being scraped now.\n",
      "Page  855  is being scraped now.\n",
      "Page  856  is being scraped now.\n",
      "Page  857  is being scraped now.\n",
      "Page  858  is being scraped now.\n",
      "Page  859  is being scraped now.\n",
      "Page  860  is being scraped now.\n",
      "Page  861  is being scraped now.\n",
      "Page  862  is being scraped now.\n",
      "Page  863  is being scraped now.\n",
      "Page  864  is being scraped now.\n",
      "Page  865  is being scraped now.\n",
      "Page  866  is being scraped now.\n",
      "Page  867  is being scraped now.\n",
      "Page  868  is being scraped now.\n",
      "Page  869  is being scraped now.\n",
      "Page  870  is being scraped now.\n",
      "Page  871  is being scraped now.\n",
      "Page  872  is being scraped now.\n",
      "Page  873  is being scraped now.\n",
      "Page  874  is being scraped now.\n",
      "Page  875  is being scraped now.\n",
      "Page  876  is being scraped now.\n",
      "Page  877  is being scraped now.\n",
      "Page  878  is being scraped now.\n",
      "Page  879  is being scraped now.\n",
      "Page  880  is being scraped now.\n",
      "Page  881  is being scraped now.\n",
      "Page  882  is being scraped now.\n",
      "Page  883  is being scraped now.\n",
      "Page  884  is being scraped now.\n",
      "Page  885  is being scraped now.\n",
      "Page  886  is being scraped now.\n",
      "Page  887  is being scraped now.\n",
      "Page  888  is being scraped now.\n",
      "Page  889  is being scraped now.\n",
      "Page  890  is being scraped now.\n",
      "Page  891  is being scraped now.\n",
      "Page  892  is being scraped now.\n",
      "Page  893  is being scraped now.\n",
      "Page  894  is being scraped now.\n",
      "Page  895  is being scraped now.\n",
      "Page  896  is being scraped now.\n",
      "Page  897  is being scraped now.\n",
      "Page  898  is being scraped now.\n",
      "Page  899  is being scraped now.\n",
      "Page  900  is being scraped now.\n",
      "Page  901  is being scraped now.\n",
      "Page  902  is being scraped now.\n",
      "Page  903  is being scraped now.\n",
      "Page  904  is being scraped now.\n",
      "Page  905  is being scraped now.\n",
      "Page  906  is being scraped now.\n",
      "Page  907  is being scraped now.\n",
      "Page  908  is being scraped now.\n",
      "Page  909  is being scraped now.\n",
      "Page  910  is being scraped now.\n",
      "Page  911  is being scraped now.\n",
      "Page  912  is being scraped now.\n",
      "Page  913  is being scraped now.\n",
      "Page  914  is being scraped now.\n",
      "Page  915  is being scraped now.\n",
      "Page  916  is being scraped now.\n",
      "Page  917  is being scraped now.\n",
      "Page  918  is being scraped now.\n",
      "Page  919  is being scraped now.\n",
      "Page  920  is being scraped now.\n",
      "Page  921  is being scraped now.\n",
      "Page  922  is being scraped now.\n",
      "Page  923  is being scraped now.\n",
      "Page  924  is being scraped now.\n",
      "Page  925  is being scraped now.\n",
      "Page  926  is being scraped now.\n",
      "Page  927  is being scraped now.\n",
      "Page  928  is being scraped now.\n",
      "Page  929  is being scraped now.\n",
      "Page  930  is being scraped now.\n",
      "Page  931  is being scraped now.\n",
      "Page  932  is being scraped now.\n",
      "Page  933  is being scraped now.\n",
      "Page  934  is being scraped now.\n",
      "Page  935  is being scraped now.\n",
      "Page  936  is being scraped now.\n",
      "Page  937  is being scraped now.\n",
      "Page  938  is being scraped now.\n",
      "Page  939  is being scraped now.\n",
      "Page  940  is being scraped now.\n",
      "Page  941  is being scraped now.\n",
      "Page  942  is being scraped now.\n",
      "Page  943  is being scraped now.\n",
      "Page  944  is being scraped now.\n",
      "Page  945  is being scraped now.\n",
      "Page  946  is being scraped now.\n",
      "Page  947  is being scraped now.\n",
      "Page  948  is being scraped now.\n",
      "Page  949  is being scraped now.\n",
      "Page  950  is being scraped now.\n",
      "Page  951  is being scraped now.\n",
      "Page  952  is being scraped now.\n",
      "Page  953  is being scraped now.\n",
      "Page  954  is being scraped now.\n",
      "Page  955  is being scraped now.\n",
      "Page  956  is being scraped now.\n",
      "Page  957  is being scraped now.\n"
     ]
    },
    {
     "output_type": "error",
     "ename": "ConnectionError",
     "evalue": "HTTPSConnectionPool(host='www.inmemoriam.be', port=443): Max retries exceeded with url: /nl/rouwberichten/?page=957&filter=&periodStart=2016-01-01&periodEnd=2021-01-01&yearOfBirth=&undertakerId=&placeOfResidence=&provinceId=&newsPaper=&obituary=1 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f0d0055d520>: Failed to establish a new connection: [Errno -2] Name or service not known'))",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mgaierror\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[0;32m/usr/lib/python3.8/site-packages/urllib3/connection.py\u001b[0m in \u001b[0;36m_new_conn\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    158\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 159\u001b[0;31m             conn = connection.create_connection(\n\u001b[0m\u001b[1;32m    160\u001b[0m                 \u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_dns_host\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mport\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mextra_kw\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/lib/python3.8/site-packages/urllib3/util/connection.py\u001b[0m in \u001b[0;36mcreate_connection\u001b[0;34m(address, timeout, source_address, socket_options)\u001b[0m\n\u001b[1;32m     60\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 61\u001b[0;31m     \u001b[0;32mfor\u001b[0m \u001b[0mres\u001b[0m \u001b[0;32min\u001b[0m \u001b[0msocket\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetaddrinfo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhost\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mport\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfamily\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msocket\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSOCK_STREAM\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     62\u001b[0m         \u001b[0maf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msocktype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mproto\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcanonname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msa\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mres\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/lib/python3.8/socket.py\u001b[0m in \u001b[0;36mgetaddrinfo\u001b[0;34m(host, port, family, type, proto, flags)\u001b[0m\n\u001b[1;32m    917\u001b[0m     \u001b[0maddrlist\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 918\u001b[0;31m     \u001b[0;32mfor\u001b[0m \u001b[0mres\u001b[0m \u001b[0;32min\u001b[0m \u001b[0m_socket\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetaddrinfo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhost\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mport\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfamily\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mproto\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mflags\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    919\u001b[0m         \u001b[0maf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msocktype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mproto\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcanonname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msa\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mres\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mgaierror\u001b[0m: [Errno -2] Name or service not known",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mNewConnectionError\u001b[0m                        Traceback (most recent call last)",
      "\u001b[0;32m/usr/lib/python3.8/site-packages/urllib3/connectionpool.py\u001b[0m in \u001b[0;36murlopen\u001b[0;34m(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)\u001b[0m\n\u001b[1;32m    669\u001b[0m             \u001b[0;31m# Make the request on the httplib connection object.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 670\u001b[0;31m             httplib_response = self._make_request(\n\u001b[0m\u001b[1;32m    671\u001b[0m                 \u001b[0mconn\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/lib/python3.8/site-packages/urllib3/connectionpool.py\u001b[0m in \u001b[0;36m_make_request\u001b[0;34m(self, conn, method, url, timeout, chunked, **httplib_request_kw)\u001b[0m\n\u001b[1;32m    380\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 381\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_validate_conn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mconn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    382\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mSocketTimeout\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mBaseSSLError\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/lib/python3.8/site-packages/urllib3/connectionpool.py\u001b[0m in \u001b[0;36m_validate_conn\u001b[0;34m(self, conn)\u001b[0m\n\u001b[1;32m    977\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mgetattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mconn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"sock\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m  \u001b[0;31m# AppEngine might not have  `.sock`\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 978\u001b[0;31m             \u001b[0mconn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconnect\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    979\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/lib/python3.8/site-packages/urllib3/connection.py\u001b[0m in \u001b[0;36mconnect\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    308\u001b[0m         \u001b[0;31m# Add certificate verification\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 309\u001b[0;31m         \u001b[0mconn\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_new_conn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    310\u001b[0m         \u001b[0mhostname\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhost\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/lib/python3.8/site-packages/urllib3/connection.py\u001b[0m in \u001b[0;36m_new_conn\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    170\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mSocketError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 171\u001b[0;31m             raise NewConnectionError(\n\u001b[0m\u001b[1;32m    172\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"Failed to establish a new connection: %s\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNewConnectionError\u001b[0m: <urllib3.connection.HTTPSConnection object at 0x7f0d0055d520>: Failed to establish a new connection: [Errno -2] Name or service not known",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mMaxRetryError\u001b[0m                             Traceback (most recent call last)",
      "\u001b[0;32m/usr/lib/python3.8/site-packages/requests/adapters.py\u001b[0m in \u001b[0;36msend\u001b[0;34m(self, request, stream, timeout, verify, cert, proxies)\u001b[0m\n\u001b[1;32m    438\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mchunked\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 439\u001b[0;31m                 resp = conn.urlopen(\n\u001b[0m\u001b[1;32m    440\u001b[0m                     \u001b[0mmethod\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmethod\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/lib/python3.8/site-packages/urllib3/connectionpool.py\u001b[0m in \u001b[0;36murlopen\u001b[0;34m(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)\u001b[0m\n\u001b[1;32m    725\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 726\u001b[0;31m             retries = retries.increment(\n\u001b[0m\u001b[1;32m    727\u001b[0m                 \u001b[0mmethod\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0merror\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_pool\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_stacktrace\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msys\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexc_info\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/lib/python3.8/site-packages/urllib3/util/retry.py\u001b[0m in \u001b[0;36mincrement\u001b[0;34m(self, method, url, response, error, _pool, _stacktrace)\u001b[0m\n\u001b[1;32m    438\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mnew_retry\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mis_exhausted\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 439\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0mMaxRetryError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_pool\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0merror\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mResponseError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcause\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    440\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mMaxRetryError\u001b[0m: HTTPSConnectionPool(host='www.inmemoriam.be', port=443): Max retries exceeded with url: /nl/rouwberichten/?page=957&filter=&periodStart=2016-01-01&periodEnd=2021-01-01&yearOfBirth=&undertakerId=&placeOfResidence=&provinceId=&newsPaper=&obituary=1 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f0d0055d520>: Failed to establish a new connection: [Errno -2] Name or service not known'))",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mConnectionError\u001b[0m                           Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-6-7886c4da22a0>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     11\u001b[0m     \u001b[0mtempdf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDataFrame\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     12\u001b[0m     \u001b[0murl\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"https://www.inmemoriam.be/nl/rouwberichten/?page=\"\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnum\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m\"&filter=&periodStart=\"\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbegin_date\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m\"&periodEnd=\"\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mend_date\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m\"&yearOfBirth=&undertakerId=&placeOfResidence=&provinceId=&newsPaper=&obituary=1\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 13\u001b[0;31m     \u001b[0mpage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrequests\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     14\u001b[0m     \u001b[0msoup\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mBeautifulSoup\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpage\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcontent\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'html.parser'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     15\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/lib/python3.8/site-packages/requests/api.py\u001b[0m in \u001b[0;36mget\u001b[0;34m(url, params, **kwargs)\u001b[0m\n\u001b[1;32m     74\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     75\u001b[0m     \u001b[0mkwargs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msetdefault\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'allow_redirects'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 76\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mrequest\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'get'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparams\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mparams\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     77\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     78\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/lib/python3.8/site-packages/requests/api.py\u001b[0m in \u001b[0;36mrequest\u001b[0;34m(method, url, **kwargs)\u001b[0m\n\u001b[1;32m     59\u001b[0m     \u001b[0;31m# cases, and look like a memory leak in others.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     60\u001b[0m     \u001b[0;32mwith\u001b[0m \u001b[0msessions\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSession\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0msession\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 61\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0msession\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmethod\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmethod\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     62\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     63\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/lib/python3.8/site-packages/requests/sessions.py\u001b[0m in \u001b[0;36mrequest\u001b[0;34m(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)\u001b[0m\n\u001b[1;32m    528\u001b[0m         }\n\u001b[1;32m    529\u001b[0m         \u001b[0msend_kwargs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msettings\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 530\u001b[0;31m         \u001b[0mresp\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprep\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0msend_kwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    531\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    532\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mresp\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/lib/python3.8/site-packages/requests/sessions.py\u001b[0m in \u001b[0;36msend\u001b[0;34m(self, request, **kwargs)\u001b[0m\n\u001b[1;32m    641\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    642\u001b[0m         \u001b[0;31m# Send the request\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 643\u001b[0;31m         \u001b[0mr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0madapter\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    644\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    645\u001b[0m         \u001b[0;31m# Total elapsed time of the request (approximately)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/lib/python3.8/site-packages/requests/adapters.py\u001b[0m in \u001b[0;36msend\u001b[0;34m(self, request, stream, timeout, verify, cert, proxies)\u001b[0m\n\u001b[1;32m    514\u001b[0m                 \u001b[0;32mraise\u001b[0m \u001b[0mSSLError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrequest\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    515\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 516\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0mConnectionError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrequest\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    517\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    518\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mClosedPoolError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mConnectionError\u001b[0m: HTTPSConnectionPool(host='www.inmemoriam.be', port=443): Max retries exceeded with url: /nl/rouwberichten/?page=957&filter=&periodStart=2016-01-01&periodEnd=2021-01-01&yearOfBirth=&undertakerId=&placeOfResidence=&provinceId=&newsPaper=&obituary=1 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f0d0055d520>: Failed to establish a new connection: [Errno -2] Name or service not known'))"
     ]
    }
   ],
   "source": [
    "df = pd.DataFrame()\n",
    "df[\"name\"] = []\n",
    "df[\"age\"] = []\n",
    "df[\"date\"] = []\n",
    "df[\"location\"] = []\n",
    "page_list = np.arange(current_page,int(end_page) + 1)\n",
    "print(\"Total amount of pages is: \", end_page)\n",
    "\n",
    "for num in page_list:\n",
    "    print(\"Page \", num, \" is being scraped now.\")\n",
    "    tempdf = pd.DataFrame()\n",
    "    url = \"https://www.inmemoriam.be/nl/rouwberichten/?page=\" + str(num) + \"&filter=&periodStart=\" + str(begin_date) + \"&periodEnd=\" + str(end_date) + \"&yearOfBirth=&undertakerId=&placeOfResidence=&provinceId=&newsPaper=&obituary=1\"\n",
    "    page = requests.get(url)\n",
    "    soup = BeautifulSoup(page.content, 'html.parser')\n",
    "    \n",
    "    name_list = soup.find_all('h3', class_ = 'c-deceased__name')\n",
    "    age_list = soup.find_all('span', class_ = 'c-deceased__age')\n",
    "    date_list = soup.find_all('div', class_ = 'c-deceased__departed')\n",
    "    location_list = soup.find_all('div', class_ = 'c-deceased__location')\n",
    "    \n",
    "    names = []\n",
    "    ages = []\n",
    "    dates = []\n",
    "    locations = []\n",
    "\n",
    "    for name in name_list:\n",
    "        name = name.get_text()\n",
    "        names.append(name)\n",
    "\n",
    "    for age in age_list:\n",
    "        age = age.get_text()\n",
    "        age = int(age[0:-5])\n",
    "        ages.append(age)\n",
    "\n",
    "    for location in location_list:\n",
    "        location = location.get_text()\n",
    "        locations.append(location)\n",
    "\n",
    "    for date in date_list:\n",
    "        date = date.get_text()\n",
    "        date = date[11:-1]\n",
    "        date = dt.datetime.strptime(date, \"%d/%m/%Y\")\n",
    "        dates.append(date)\n",
    "        \n",
    "    ages = Series(ages)\n",
    "    locations = Series(locations)\n",
    "    dates = Series(dates)\n",
    "    names = Series(names)\n",
    "    \n",
    "    tempdf[\"name\"] = names\n",
    "    tempdf[\"age\"] = ages\n",
    "    tempdf[\"date\"] = dates\n",
    "    tempdf[\"location\"] = locations\n",
    "    \n",
    "    df = df.append(tempdf)\n",
    "\n",
    "print(\"Done.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Clean dataset\n",
    "### 4.1 Reshape\n",
    "The dataframe is reshaped in order to have the index numbers to be correct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1368, 4)"
      ]
     },
     "execution_count": 158,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.reset_index(inplace = True, drop = True)\n",
    "df = df[::-1].reset_index(drop = True)\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 Reformat\n",
    "`month`, `week` and `day` of the year are extracted from `date` in order to store these in different variables to make exploratory analysis of the dataset a bit more straightforward."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>age</th>\n",
       "      <th>date</th>\n",
       "      <th>location</th>\n",
       "      <th>month</th>\n",
       "      <th>week</th>\n",
       "      <th>day</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Maria Winters</td>\n",
       "      <td>82.0</td>\n",
       "      <td>2016-03-01</td>\n",
       "      <td>LOMMEL</td>\n",
       "      <td>March</td>\n",
       "      <td>09</td>\n",
       "      <td>061</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Albert WINDELS</td>\n",
       "      <td>95.0</td>\n",
       "      <td>2016-03-01</td>\n",
       "      <td>Ingooigem</td>\n",
       "      <td>March</td>\n",
       "      <td>09</td>\n",
       "      <td>061</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Yvonne Wijnants</td>\n",
       "      <td>90.0</td>\n",
       "      <td>2016-03-01</td>\n",
       "      <td>ZONHOVEN</td>\n",
       "      <td>March</td>\n",
       "      <td>09</td>\n",
       "      <td>061</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>LAOUREUX Victorine</td>\n",
       "      <td>87.0</td>\n",
       "      <td>2016-03-01</td>\n",
       "      <td>Stembert</td>\n",
       "      <td>March</td>\n",
       "      <td>09</td>\n",
       "      <td>061</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Philippe VANDENHOVEN</td>\n",
       "      <td>59.0</td>\n",
       "      <td>2016-03-01</td>\n",
       "      <td>Bastogne</td>\n",
       "      <td>March</td>\n",
       "      <td>09</td>\n",
       "      <td>061</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   name   age       date     location  month week  day\n",
       "0         Maria Winters  82.0 2016-03-01      LOMMEL   March   09  061\n",
       "1        Albert WINDELS  95.0 2016-03-01   Ingooigem   March   09  061\n",
       "2       Yvonne Wijnants  90.0 2016-03-01    ZONHOVEN   March   09  061\n",
       "3    LAOUREUX Victorine  87.0 2016-03-01    Stembert   March   09  061\n",
       "4  Philippe VANDENHOVEN  59.0 2016-03-01    Bastogne   March   09  061"
      ]
     },
     "execution_count": 159,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dates = df[\"date\"]\n",
    "\n",
    "months = []\n",
    "weeks = []\n",
    "days = []\n",
    "\n",
    "for date in dates:\n",
    "    month = dt.datetime.strftime(date, \"%B\")\n",
    "    week = dt.datetime.strftime(date, \"%W\")\n",
    "    day = dt.datetime.strftime(date, \"%j\")\n",
    "    months.append(month)\n",
    "    weeks.append(week)\n",
    "    days.append(day)\n",
    "    \n",
    "df[\"month\"] = months\n",
    "df[\"week\"] = weeks\n",
    "df[\"day\"] = days\n",
    "\n",
    "df['location']= pd.Series(df['location'], dtype=\"string\")\n",
    "df['month']= pd.Series(df['month'], dtype=\"string\")\n",
    "df['week']= pd.Series(df['week'], dtype=\"string\")\n",
    "df['name']= pd.Series(df['name'], dtype=\"string\")\n",
    "df['day']= pd.Series(df['day'], dtype=\"string\")\n",
    "\n",
    "booler = df[\"month\"] == \"March\"\n",
    "df = df.loc[booler,]\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Save the data\n",
    "The resulting dataset is ultimately saved in a .csv file called `deaths_2016-2020.csv` for later analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv(\"deaths_2016-2020.csv\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5-final"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}