LANL_Demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploratory Analysis of Netflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "### Get Data\n",
    "\n",
    "This notebook assumes that you have downloaded one or more netflow files from the [LANL dataset](https://csr.lanl.gov/data/2017.html) and converted them to HDF5 using something like `hdflow`. Example:\n",
    "\n",
    "```bash\n",
    "pip install hdflow\n",
    "csv2hdf --format=lanl /path/to/lanl/netflow*\n",
    "```\n",
    "\n",
    "### Chapel\n",
    "\n",
    "[Download](https://chapel-lang.org/download.html) and [build](https://chapel-lang.org/docs/usingchapel/building.html) the Chapel [programming language](https://chapel-lang.org/). Be sure to build for a multi-locale system, if appropriate.\n",
    "\n",
    "### Arkouda\n",
    "\n",
    "```bash\n",
    "pip install arkouda\n",
    "cd arkouda/install/dir\n",
    "chpl --fast -senableParScan arkouda_server.chpl\n",
    "./arkouda_server -nl <number_of_locales>\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-28T00:54:59.581497Z",
     "start_time": "2019-03-28T00:54:59.574001Z"
    }
   },
   "outputs": [],
   "source": [
    "import arkouda as ak\n",
    "from glob import glob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-28T00:54:59.650323Z",
     "start_time": "2019-03-28T00:54:59.589647Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4.2.5\n",
      "psp =  tcp://node01:5555\n",
      "connected to tcp://node01:5555\n"
     ]
    }
   ],
   "source": [
    "ak.connect(server='node01')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-28T00:54:59.695262Z",
     "start_time": "2019-03-28T00:54:59.653569Z"
    }
   },
   "outputs": [],
   "source": [
    "hdffiles = glob('/mnt/data/lanl_netflow/hdf5/*.hdf')\n",
    "fields = ['srcIP', 'dstIP', 'srcPort', 'dstPort', 'start']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-28T00:56:03.706426Z",
     "start_time": "2019-03-28T00:54:59.698473Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4.18 ms, sys: 340 µs, total: 4.53 ms\n",
      "Wall time: 1min 3s\n"
     ]
    }
   ],
   "source": [
    "%time data = {field: ak.read_hdf(field, hdffiles) for field in fields}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-28T00:56:03.731553Z",
     "start_time": "2019-03-28T00:56:03.710366Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'srcIP': array([22058209181, 22058295678, 5266512788, ..., 22058739089, 22058739089, 22058739089]),\n",
       " 'dstIP': array([22058391981, 22058674074, 22057986724, ..., 22057863347, 22058450761, 22058554651]),\n",
       " 'srcPort': array([5507, 3137, 5060, ..., 58889, 75615, 67796]),\n",
       " 'dstPort': array([46272, 445, 5060, ..., 80, 80, 80]),\n",
       " 'start': array([118781, 118783, 118785, ..., 345599, 345599, 345599])}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Are src and dst Meaningful?\n",
    "Typically, src and dst are not meaningful labels, but the curators of this dataset may have used it to encode the identity of the client and server. If so, then the frequency of server ports should differ quite a bit between src and dst."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-28T00:56:05.294109Z",
     "start_time": "2019-03-28T00:56:03.734686Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 0 ns, sys: 2.55 ms, total: 2.55 ms\n",
      "Wall time: 1.5 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(1, 52123410)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%time (data['srcPort'] == 80).sum(), (data['dstPort'] == 80).sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-28T00:56:06.797915Z",
     "start_time": "2019-03-28T00:56:05.297510Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 80 µs, sys: 3.04 ms, total: 3.12 ms\n",
      "Wall time: 1.49 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(6064, 74255132)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%time (data['srcPort'] == 443).sum(), (data['dstPort'] == 443).sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "dst has lots of port 80 (HTTP) and 443 (HTTPS), while src has very little. Thus, unlike typical netflow, dst is probably the server side in this dataset, while src is the client side.\n",
    "\n",
    "Confirm by looking at more of the port distributions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-28T00:56:08.720218Z",
     "start_time": "2019-03-28T00:56:06.801369Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.34 ms, sys: 36 µs, total: 2.37 ms\n",
      "Wall time: 1.91 s\n"
     ]
    }
   ],
   "source": [
    "%time sport, scount = data['srcPort'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-28T00:57:48.027855Z",
     "start_time": "2019-03-28T00:56:08.723576Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(15379, 2036008),\n",
       " (5060, 972016),\n",
       " (137, 697136),\n",
       " (95765, 661667),\n",
       " (41101, 450218),\n",
       " (123, 298170),\n",
       " (59844, 243825),\n",
       " (84474, 241587),\n",
       " (87103, 234884),\n",
       " (20995, 225309)]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections import Counter\n",
    "sportCounts = Counter()\n",
    "for i in range(sport.size):\n",
    "    sportCounts[sport[i]] = scount[i]\n",
    "sportCounts.most_common(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-28T00:57:48.037068Z",
     "start_time": "2019-03-28T00:57:48.031346Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "65055"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(sportCounts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-28T00:57:55.115021Z",
     "start_time": "2019-03-28T00:57:48.039782Z"
    }
   },
   "outputs": [],
   "source": [
    "dport, dcount = data['dstPort'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-28T00:59:32.386264Z",
     "start_time": "2019-03-28T00:57:55.118443Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(53, 97719325),\n",
       " (443, 74255132),\n",
       " (80, 52123410),\n",
       " (514, 29138686),\n",
       " (389, 18679249),\n",
       " (427, 13094194),\n",
       " (88, 12218516),\n",
       " (161, 11970935),\n",
       " (445, 10192812),\n",
       " (95765, 6444028)]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dportCounts = Counter()\n",
    "for i in range(dport.size):\n",
    "    dportCounts[dport[i]] = dcount[i]\n",
    "dportCounts.most_common(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-28T00:59:32.395994Z",
     "start_time": "2019-03-28T00:59:32.389915Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "62727"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(dportCounts)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}