solution.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "94cdc468",
   "metadata": {},
   "source": [
    "# Solution\n",
    "\n",
    "This notebook demonstrates one way to solve the challenge using Python."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41bac80e",
   "metadata": {},
   "source": [
    "## Packages\n",
    "\n",
    "Here, we are going to use pandas to inspect the data, clean the data and calculate our result.\n",
    "\n",
    "We will also make use of numpy.\n",
    "\n",
    "Different security tiers of Data Safe Haven support installing packages in different ways,\n",
    "\n",
    "- Tier ≤ 2: Able to install any package from PyPI and CRAN\n",
    "- Tier 3: Able to install a configurable, curated list of packages from PyPI and CRAN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "744e0112",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install pandas numpy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e39d749",
   "metadata": {},
   "source": [
    "Now we can import pandas and numpy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3008ae72",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55ee273d",
   "metadata": {},
   "source": [
    "## Find the data\n",
    "\n",
    "In a Data Safe Haven environment the input data is located at `/data`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0af12c86",
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls /data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04128848",
   "metadata": {},
   "source": [
    "The `/data` directory, and the files within it are read only.\n",
    "\n",
    "This preserves the integrity of input data.\n",
    "\n",
    "We can test this by trying to write to `/data` and append data to `/data/data.csv`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4095400f",
   "metadata": {},
   "outputs": [],
   "source": [
    "!touch /data/new_file.txt\n",
    "!echo \"hello\" >> /data/data.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f873425",
   "metadata": {},
   "source": [
    "## Inspect the data\n",
    "\n",
    "We will read the data from the csv file and store it in a Pandas dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90960143",
   "metadata": {},
   "outputs": [],
   "source": [
    "!cp /data/data.csv ./"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60a8a0cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"./data.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4edf937",
   "metadata": {},
   "source": [
    "We want to calculate the **mean height**.\n",
    "\n",
    "Let's look at the columns to find which contains the height data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bbf362ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ff08be9",
   "metadata": {},
   "source": [
    "We could go ahead and calculate the mean immediately,\n",
    "\n",
    "```python\n",
    "df[\"height\"].mean()\n",
    "```\n",
    "\n",
    "However, we should look at the data first to check it makes sense.\n",
    "Panda's `describe` method can help us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa046d7a",
   "metadata": {},
   "outputs": [],
   "source": [
    "height = df[\"height\"]\n",
    "height.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35da277c",
   "metadata": {},
   "source": [
    "The mean looks reasonable.\n",
    "168.8 would make sense if the height values are in cm.\n",
    "\n",
    "But look at the minimum and maximum values.\n",
    "Those don't seem reasonable at all.\n",
    "\n",
    "Let's look closer at the smallest values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "499edfcf",
   "metadata": {},
   "outputs": [],
   "source": [
    "height_sorted = height.sort_values()\n",
    "height_sorted.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2fab5e6",
   "metadata": {},
   "source": [
    "And the largest values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68dede87",
   "metadata": {},
   "outputs": [],
   "source": [
    "height_sorted.tail(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3531e66",
   "metadata": {},
   "source": [
    "## Cleaning the data\n",
    "\n",
    "It would be more reasonable to remove these outliers.\n",
    "\n",
    "Here, we will use information about how those outliers were generated to remove them.\n",
    "\n",
    "The anomalously large heights are always greater than or equal to 1000.\n",
    "The anomalously smaller heights are always less than 10."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8107ae8",
   "metadata": {},
   "outputs": [],
   "source": [
    "height_filtered = height[(height <= 1000) & (height > 10)]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "098b8fd6",
   "metadata": {},
   "source": [
    "## Calculating the mean\n",
    "\n",
    "Now we can calculate the mean of our cleaned height data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f469cd03",
   "metadata": {},
   "outputs": [],
   "source": [
    "mean = np.around(height_filtered.mean(), 1)\n",
    "mean"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e296d898",
   "metadata": {},
   "source": [
    "## Prepare your result for extraction\n",
    "\n",
    "In a Data Safe Haven environment data in `/output` can be considered for removing from the environment.\n",
    "\n",
    "You can read, write and delete files in the `/output` directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b3eb24f",
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls /output\n",
    "!echo \"hello\" >> /output/a.txt\n",
    "!cat /output/a.txt\n",
    "!rm /output/a.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9eb302e2",
   "metadata": {},
   "source": [
    "If we write our output to a file in `/output` an admin will be able to create a link to download that file after the review process is completed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "87440e68",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"/output/mean_height.txt\", \"w\") as f:\n",
    "    f.write(f'{mean}\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6da11e1d",
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat /output/mean_height.txt"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}