python_tech_test.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "python tech test.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyN1wMan2WaPOMn6cVb+b2Rs",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/umerfsandhu/python-assessment/blob/main/python_tech_test.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "herJ2bTg7B77"
      },
      "source": [
        "**1 Write a basic python scrapy crawler (without Splash, Selenium or another headless browser if possible) for https://www.aldi.com.au/ that:**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xbl8jQHuB4rm"
      },
      "source": [
        "***Solution:*** \n",
        "\n",
        "1.   https://github.com/umerfsandhu/python-assessment/blob/main/Aldi_spider.py \n",
        "2.   https://github.com/umerfsandhu/python-assessment/blob/main/product.csv\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jCvtQxc0TpZT"
      },
      "source": [
        "2. **Regular expressions**\n",
        "\n",
        "\n",
        "2.1 Write the python code that uses regular expressions to extract the numerical product count from the variable product_count_text below, and converts it to integer (int) format\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0k_d-WxiLyWV",
        "outputId": "8784c305-4de4-4d3e-b362-a602b15ee3d8"
      },
      "source": [
        "import re\n",
        "product_count_text = '381 Products found'\n",
        "num = re.findall(r'\\d+',product_count_text)\n",
        "product_count_int = int(num[0])\n",
        "print(product_count_int) "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "381\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nX3jIAFCUDpi"
      },
      "source": [
        "**2.2**  Write the python code that uses a single search pattern to extract the sequence of characters after the final \"/\", before the \"-\" character, in each url."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eoxSg_8GULtw",
        "outputId": "a185ec06-d947-441e-b641-1b3c1fef91ce"
      },
      "source": [
        "import re\n",
        "generic_urls = [\"https://www.genericdomain.com/abc/def/1290aodwb23-ghi.img\", \"https://www.genericdomain.com/ab-c/31287bdwakj-jkl.img\", \"https://www.genericdomain.com/19unioawd02-jkl.img\"]\n",
        "special_sequence = []\n",
        "for url in generic_urls:\n",
        "  # used regular expression that look after '/' and before '-'\n",
        "\tspecial_sequence.append(re.findall(r\"(?<=/)[a-z0-9]+(?=\\-)\",url))\n",
        " \n",
        "print(special_sequence)\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[['1290aodwb23'], ['ab', '31287bdwakj'], ['19unioawd02']]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XNmm3-xO0zE5"
      },
      "source": [
        "**2.3** Assume that you have extracted the below string from the ‘script’ tag of a web page. Use python regular expressions (i.e the ‘re’ library) to convert the string into a format that can be loaded via the python json library. I.e the below code should not throw any errors after the code is formatted properly."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jdQiaT9S0zux",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "02000dc4-4158-44bb-eba2-5c8d1f9fa6e2"
      },
      "source": [
        "import re\n",
        "import json\n",
        "product_storage_string_raw= \"\"\"\n",
        "<script> \n",
        "\t\tvar products_storage=[];\n",
        "\t\tproducts_storage= [{\"price\":19.0,\"name\":\"House & Home Hooded Blanket - Charcoal\",\"id\":\"671282\",\"position\":1,\"list\":\"Product Category - Level 2\",\"category\":\"Home/Bedding/Blankets\",\"brand\":\"House & Home\"},{\"price\":5.0,\"name\":\"House & Home 225TC Pillowcase -  Dance All Night - White\",\"id\":\"649566\",\"position\":2,\"list\":\"Product Category - Level 2\",\"category\":\"Home/Bedding/Pillow Cases\",\"brand\":\"House & Home\"},{\"price\":59.0,\"name\":\"House & Home Linen Cotton Quilt Cover Set - Frayed Edge - Charcoal - Queen\",\"id\":\"659975\",\"position\":3,\"list\":\"Product Category - Level 2\",\"category\":\"Home/Bedding/Quilt Covers\"},{\"price\":59.0,\"name\":\"House & Home Linen Cotton Quilt Cover Set - Frayed Edge - Navy - Queen\",\"id\":\"659976\",\"position\":4,\"list\":\"Product Category - Level 2\",\"category\":\"Home/Bedding/Quilt Covers\",\"brand\":\"House & Home\"},{\"price\":10.0,\"name\":\"Tontine Winter Comfort Medium Pillow\",\"id\":\"668702\",\"position\":5,\"list\":\"Product Category - Level 2\",\"category\":\"Home/Bedding/Pillows\",\"brand\":\"Tontine\"},{\"price\":15.0,\"name\":\"House & Home Woven Cushion - Grey\",\"id\":\"662158\",\"position\":6,\"list\":\"Product Category - Level 2\",\"category\":\"Home/Bedding/Pillows\",\"brand\":\"House & Home\"},{\"price\":25.0,\"name\":\"House & Home Melange Plush Blanket - Grey Blue - Queen\",\"id\":\"656994\",\"position\":7,\"list\":\"Product Category - Level 2\",\"category\":\"Home/Bedding/Blankets\",\"brand\":\"House & Home\"},{\"price\":24.0,\"name\":\"House & Home Polar Fleece Sheet Set - Silver Grey\",\"id\":\"BIGW659721\",\"position\":8,\"list\":\"Product Category - Level 2\",\"category\":\"Home/Bedding/Sheets\",\"brand\":\"House & Home\"},{\"price\":39.0,\"name\":\"Incredibles Quilt Cover Set - Red - Single\",\"id\":\"675844\",\"position\":9,\"list\":\"Product Category - Level 2\",\"category\":\"Home/Bedding/Kids Bedding\",\"brand\":\"Disney\"},{\"price\":27.0,\"name\":\"House & Home Plain Blush Flannelette Sheet Set\",\"id\":\"BIGW660776\",\"position\":10,\"list\":\"Product Category - Level 2\",\"category\":\"Home/Bedding/Sheets\",\"brand\":\"House & Home\"},{\"price\":89.0,\"name\":\"House & Home Marilyne 7 Piece Comforter Set\",\"id\":\"BIGW659777\",\"position\":11,\"list\":\"Product Category - Level 2\",\"category\":\"Home/Bedding/Comforters\",\"brand\":\"House & Home\"},{\"price\":39.0,\"name\":\"House & Home Kids Foxy 180 Thread Count Comforter Set\",\"id\":\"BIGW662598\",\"position\":12,\"list\":\"Product Category - Level 2\",\"category\":\"Home/Bedding/Comforters\",\"brand\":\"House & Home Kids\"},{\"price\":39.0,\"name\":\"House & Home Kids Comforter Set - Dakota\",\"id\":\"BIGW662602\",\"position\":13,\"list\":\"Product Category - Level 2\",\"category\":\"Home/Bedding/Comforters\",\"brand\":\"House & Home Kids\"},{\"price\":39.0,\"name\":\"House & Home Kids Comforter Set - Xavier\",\"id\":\"BIGW662604\",\"position\":14,\"list\":\"Product Category - Level 2\",\"category\":\"Home/Bedding/Comforters\",\"brand\":\"House & Home Kids\"},{\"price\":39.0,\"name\":\"House & Home Kids Pia 180 Thread Count Comforter Set\",\"id\":\"BIGW662606\",\"position\":15,\"list\":\"Product Category - Level 2\",\"category\":\"Home/Bedding/Comforters\",\"brand\":\"House & Home Kids\"},{\"price\":35.0,\"name\":\"House & Home Waffle Blanket - Taupe - Queen\",\"id\":\"429199\",\"position\":16,\"list\":\"Product Category - Level 2\",\"category\":\"Home/Bedding/Blankets\",\"brand\":\"House & Home\"},{\"price\":59.0,\"name\":\"House & Home Blue Washed Pinstripe Coverlet - Queen\",\"id\":\"660111\",\"position\":17,\"list\":\"Product Category - Level 2\",\"category\":\"Home/Bedding/Quilt Covers\",\"brand\":\"House & Home\"},{\"price\":9.0,\"name\":\"Smart Value Quilt Cover Set - Iron Gate - King\",\"id\":\"554307\",\"position\":18,\"list\":\"Product Category - Level 2\",\"category\":\"Home/Bedding/Quilt Covers\"},{\"price\":9.0,\"name\":\"Smart Value Quilt Cover Set - Iron Gate - Single\",\"id\":\"554304\",\"position\":19,\"list\":\"Product Category - Level 2\",\"category\":\"Home/Bedding/Quilt Covers\",\"brand\":\"Smart Value\"},{\"price\":39.0,\"name\":\"Trolls Quilt Cover Set - Pink - Single\",\"id\":\"630753\",\"position\":20,\"list\":\"Product Category - Level 2\",\"category\":\"Home/Bedding/Kids Bedding\",\"brand\":\"Trolls\"},{\"price\":29.0,\"name\":\"House & Home 225 TC Quilt Cover Set - Holly - Queen\",\"id\":\"660472\",\"position\":21,\"list\":\"Product Category - Level 2\",\"category\":\"Home/Bedding/Quilt Covers\",\"brand\":\"House & Home\"},{\"price\":59.0,\"name\":\"House & Home Faux Velvet Coverlet Set - Blush - Queen\",\"id\":\"660200\",\"position\":22,\"list\":\"Product Category - Level 2\",\"category\":\"Home/Bedding/Quilt Covers\",\"brand\":\"House & Home\"},{\"price\":49.0,\"name\":\"House & Home Queen Quilt Cover Set - Chambray - Light Blue - Queen\",\"id\":\"659713\",\"position\":23,\"list\":\"Product Category - Level 2\",\"category\":\"Home/Bedding/Quilt Covers\",\"brand\":\"House & Home\"},{\"price\":49.0,\"name\":\"House & Home Queen Quilt Cover Set - Chambray - Dusty Pink - Queen\",\"id\":\"659714\",\"position\":24,\"list\":\"Product Category - Level 2\",\"category\":\"Home/Bedding/Quilt Covers\",\"brand\":\"House & Home\"}];\n",
        "\t</script>\n",
        "\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "product_storage_parsed = re.findall(r\"products\\_storage\\=\\s*\\[([^]]+)\\]\", product_storage_string_raw)\n",
        "p_S_Parse = product_storage_parsed[0] # re.findall() returns in List[]\n",
        "p_S_String_Raw = '[' + p_S_Parse + ']' \n",
        "\n",
        "product_storage_json = json.loads(p_S_String_Raw)\n",
        "print(product_storage_json)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[{'price': 19.0, 'name': 'House & Home Hooded Blanket - Charcoal', 'id': '671282', 'position': 1, 'list': 'Product Category - Level 2', 'category': 'Home/Bedding/Blankets', 'brand': 'House & Home'}, {'price': 5.0, 'name': 'House & Home 225TC Pillowcase -  Dance All Night - White', 'id': '649566', 'position': 2, 'list': 'Product Category - Level 2', 'category': 'Home/Bedding/Pillow Cases', 'brand': 'House & Home'}, {'price': 59.0, 'name': 'House & Home Linen Cotton Quilt Cover Set - Frayed Edge - Charcoal - Queen', 'id': '659975', 'position': 3, 'list': 'Product Category - Level 2', 'category': 'Home/Bedding/Quilt Covers'}, {'price': 59.0, 'name': 'House & Home Linen Cotton Quilt Cover Set - Frayed Edge - Navy - Queen', 'id': '659976', 'position': 4, 'list': 'Product Category - Level 2', 'category': 'Home/Bedding/Quilt Covers', 'brand': 'House & Home'}, {'price': 10.0, 'name': 'Tontine Winter Comfort Medium Pillow', 'id': '668702', 'position': 5, 'list': 'Product Category - Level 2', 'category': 'Home/Bedding/Pillows', 'brand': 'Tontine'}, {'price': 15.0, 'name': 'House & Home Woven Cushion - Grey', 'id': '662158', 'position': 6, 'list': 'Product Category - Level 2', 'category': 'Home/Bedding/Pillows', 'brand': 'House & Home'}, {'price': 25.0, 'name': 'House & Home Melange Plush Blanket - Grey Blue - Queen', 'id': '656994', 'position': 7, 'list': 'Product Category - Level 2', 'category': 'Home/Bedding/Blankets', 'brand': 'House & Home'}, {'price': 24.0, 'name': 'House & Home Polar Fleece Sheet Set - Silver Grey', 'id': 'BIGW659721', 'position': 8, 'list': 'Product Category - Level 2', 'category': 'Home/Bedding/Sheets', 'brand': 'House & Home'}, {'price': 39.0, 'name': 'Incredibles Quilt Cover Set - Red - Single', 'id': '675844', 'position': 9, 'list': 'Product Category - Level 2', 'category': 'Home/Bedding/Kids Bedding', 'brand': 'Disney'}, {'price': 27.0, 'name': 'House & Home Plain Blush Flannelette Sheet Set', 'id': 'BIGW660776', 'position': 10, 'list': 'Product Category - Level 2', 'category': 'Home/Bedding/Sheets', 'brand': 'House & Home'}, {'price': 89.0, 'name': 'House & Home Marilyne 7 Piece Comforter Set', 'id': 'BIGW659777', 'position': 11, 'list': 'Product Category - Level 2', 'category': 'Home/Bedding/Comforters', 'brand': 'House & Home'}, {'price': 39.0, 'name': 'House & Home Kids Foxy 180 Thread Count Comforter Set', 'id': 'BIGW662598', 'position': 12, 'list': 'Product Category - Level 2', 'category': 'Home/Bedding/Comforters', 'brand': 'House & Home Kids'}, {'price': 39.0, 'name': 'House & Home Kids Comforter Set - Dakota', 'id': 'BIGW662602', 'position': 13, 'list': 'Product Category - Level 2', 'category': 'Home/Bedding/Comforters', 'brand': 'House & Home Kids'}, {'price': 39.0, 'name': 'House & Home Kids Comforter Set - Xavier', 'id': 'BIGW662604', 'position': 14, 'list': 'Product Category - Level 2', 'category': 'Home/Bedding/Comforters', 'brand': 'House & Home Kids'}, {'price': 39.0, 'name': 'House & Home Kids Pia 180 Thread Count Comforter Set', 'id': 'BIGW662606', 'position': 15, 'list': 'Product Category - Level 2', 'category': 'Home/Bedding/Comforters', 'brand': 'House & Home Kids'}, {'price': 35.0, 'name': 'House & Home Waffle Blanket - Taupe - Queen', 'id': '429199', 'position': 16, 'list': 'Product Category - Level 2', 'category': 'Home/Bedding/Blankets', 'brand': 'House & Home'}, {'price': 59.0, 'name': 'House & Home Blue Washed Pinstripe Coverlet - Queen', 'id': '660111', 'position': 17, 'list': 'Product Category - Level 2', 'category': 'Home/Bedding/Quilt Covers', 'brand': 'House & Home'}, {'price': 9.0, 'name': 'Smart Value Quilt Cover Set - Iron Gate - King', 'id': '554307', 'position': 18, 'list': 'Product Category - Level 2', 'category': 'Home/Bedding/Quilt Covers'}, {'price': 9.0, 'name': 'Smart Value Quilt Cover Set - Iron Gate - Single', 'id': '554304', 'position': 19, 'list': 'Product Category - Level 2', 'category': 'Home/Bedding/Quilt Covers', 'brand': 'Smart Value'}, {'price': 39.0, 'name': 'Trolls Quilt Cover Set - Pink - Single', 'id': '630753', 'position': 20, 'list': 'Product Category - Level 2', 'category': 'Home/Bedding/Kids Bedding', 'brand': 'Trolls'}, {'price': 29.0, 'name': 'House & Home 225 TC Quilt Cover Set - Holly - Queen', 'id': '660472', 'position': 21, 'list': 'Product Category - Level 2', 'category': 'Home/Bedding/Quilt Covers', 'brand': 'House & Home'}, {'price': 59.0, 'name': 'House & Home Faux Velvet Coverlet Set - Blush - Queen', 'id': '660200', 'position': 22, 'list': 'Product Category - Level 2', 'category': 'Home/Bedding/Quilt Covers', 'brand': 'House & Home'}, {'price': 49.0, 'name': 'House & Home Queen Quilt Cover Set - Chambray - Light Blue - Queen', 'id': '659713', 'position': 23, 'list': 'Product Category - Level 2', 'category': 'Home/Bedding/Quilt Covers', 'brand': 'House & Home'}, {'price': 49.0, 'name': 'House & Home Queen Quilt Cover Set - Chambray - Dusty Pink - Queen', 'id': '659714', 'position': 24, 'list': 'Product Category - Level 2', 'category': 'Home/Bedding/Quilt Covers', 'brand': 'House & Home'}]\n"
          ]
        }
      ]
    }
  ]
}