2-primitive-layouts.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cf76efe1-7224-443d-9171-1e9d30728b60",
   "metadata": {},
   "source": [
    "# Primitive layouts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8cce448-8726-4e0f-b48e-0bb6a899ffa2",
   "metadata": {
    "id": "7TTfOs0Z0L-S"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import nanoarrow as na\n",
    "import pyarrow as pa"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e50cc2c-503c-4e79-88eb-13a79b657f33",
   "metadata": {},
   "source": [
    "## Fixed Size Primitive Layout\n",
    "\n",
    "A primitive value array represents an array of values where each value has the same physical size measured in bytes.\n",
    "\n",
    "![](diagrams/primitive-diagram.svg)\n",
    "\n",
    "For example a primitive array of int32s (4 bytes per value):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d698015-8fdf-436a-9348-66f0b38b570c",
   "metadata": {},
   "outputs": [],
   "source": [
    "column1 = pa.array([1, 3, 9, 9, 2], type=pa.int32())\n",
    "column1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25728134-5532-4a92-828e-d83da9c87b54",
   "metadata": {},
   "outputs": [],
   "source": [
    "column1.buffers()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5dc680bb-4df6-420b-8ef3-9e5c5a4a8f25",
   "metadata": {},
   "outputs": [],
   "source": [
    "na.array(column1).inspect()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e80b34b-9126-403e-9c2f-a25d5c544e2c",
   "metadata": {
    "id": "apK6AwKe--uC"
   },
   "source": [
    "## Intermezzo: inspecting the buffers using PyArrow and nanoarrow\n",
    "\n",
    "#### PyArrow\n",
    "\n",
    "For a `pyarrow.Array`, we can use the `buffers()` method to get a list of all the buffers of the array. The information for each buffer inlcudes:\n",
    "\n",
    "- adress of the buffer\n",
    "- buffer size in bytes\n",
    "- whether the buffer is mutable or not (buffers are generally mutable - changeable, but an Array is an immutable container in pyarrow)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2cb2a337-b990-4d01-8818-1b5e6b47eda3",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "dLJbgKmX3Zo9",
    "outputId": "b6f8ccb3-5a8a-4930-adce-26b1561b3c15"
   },
   "outputs": [],
   "source": [
    "column1.buffers()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b8f4fa4-d34e-4e42-aa56-18efa8576f9e",
   "metadata": {},
   "source": [
    "In this case a simple, fixed width primitive array, there is only a single buffer for the data values.\n",
    "\n",
    "PyArrow doesn't provide direct easy access to the buffer content, but here are a few ways to inspect the buffer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8967baa-0683-4d3d-9fa1-a969a7470f9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "values_buffer = column1.buffers()[1]\n",
    "values_buffer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9cf8d089-35e1-4084-b4b0-910b8e95c3c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# getting the raw bytes as a Python bytes object (note this makes a copy! don't do this with larger data)\n",
    "values_buffer.to_pybytes()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "30553374-29bf-43b5-99e1-e05d3ba0cb75",
   "metadata": {},
   "outputs": [],
   "source": [
    "# zero-copy view as a numpy array (using the buffer protocol)\n",
    "# -> this just shows the raw bytes as well\n",
    "np.array(values_buffer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3f6f66c-b6b7-4e7b-a333-5b0b30ca67e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# in this case we know the buffer represents int32 values, so we can view the buffer as such\n",
    "np.frombuffer(values_buffer, dtype=np.int32)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7915714a-1049-44fb-9fa3-c2912d7561af",
   "metadata": {
    "id": "4AMRlguW6Q7r"
   },
   "source": [
    "#### Inspecting buffers using nanoarrow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2999aba-a106-4249-a345-2b1889ae1c8a",
   "metadata": {
    "id": "N9m7iKBQ6Qij"
   },
   "outputs": [],
   "source": [
    "na_column1 = na.array(column1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c5937ae-5120-4d87-a247-98f5007c4c89",
   "metadata": {},
   "source": [
    "To start, nanoarrow does have a functionality to print the details of the layout of a certain array, which already gives us insight into the buffers of the array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8f6b1de-a70b-4b6f-8acf-f7466c9da69e",
   "metadata": {
    "id": "g1JOAwme6mSW"
   },
   "outputs": [],
   "source": [
    "na_column1.inspect()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91da92c9-d469-4498-ac39-cd40235e3bb6",
   "metadata": {
    "id": "zuO9EOW666XN"
   },
   "source": [
    "Additionally, it also allows us to access the buffers directly through the `buffers` property:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d8f838c-dad4-4fe9-be0c-f57ddfe4f4cb",
   "metadata": {
    "id": "-ynRSRtl7D8m"
   },
   "outputs": [],
   "source": [
    "na_column1.buffers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93a949ba-dd68-4206-86f8-a462edeed279",
   "metadata": {},
   "source": [
    "Nanoarrow does keep track of the context in which the buffer was created (i.e. it is part of an int64 array and represents the data values):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3edf2373-674c-4915-a84e-4c96aa20e75d",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_buffer = na_column1.buffers[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31e8c995-b12a-4014-9aa0-5826c45b8392",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_buffer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c27ad009-6272-412b-a747-6c9c1d54faac",
   "metadata": {},
   "outputs": [],
   "source": [
    "np.array(data_buffer)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce44b763-d47d-42f7-a547-994ce49357eb",
   "metadata": {
    "id": "jphgqRK940NI"
   },
   "source": [
    "## Support for null values\n",
    "\n",
    "Arrow supports missing values or \"nulls\" for all data types: any value in an array may be semantically null, whether primitive or nested type.\n",
    "\n",
    "In Arrow, a dedicated buffer, known as the validity (or \"null\") bitmap, is used alongside the data indicating whether each value in the array is null or not. You can think of it as vector of 0 and 1 values, where a 1 means that the value is not-null (\"valid\"), while a 0 indicates the value is null.\n",
    "\n",
    "This validity bitmap is optional, i.e. if there are no missing values in the array the buffer does not need to be allocated (as in the example column 1 in the diagram below).\n",
    "\n",
    "![](diagrams/primitive-diagram.svg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37d88ffa-e80b-4693-8e82-30f727753915",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "QX7HnLig42Ya",
    "outputId": "1a16bbc3-c725-4eb0-9c4c-3f454388cc39"
   },
   "outputs": [],
   "source": [
    "column2 = pa.array([1.2, 3.4, 9.0, None, 2.9])\n",
    "column2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b02beff-9db5-4d12-8ac5-29e931644a1b",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "NNcrvbyv7cvN",
    "outputId": "23ec4c17-0f51-46ee-b7f7-b745fbf5eab8"
   },
   "outputs": [],
   "source": [
    "na.array(column2).inspect()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "886f4e2d-a418-4165-aebd-511513444a22",
   "metadata": {
    "id": "Nc7uYpry5u0u"
   },
   "source": [
    "**Attention**: Arrow uses [least-significant bit (LSB) numbering](https://en.wikipedia.org/wiki/Bit_numbering) (also known as bit-endianness). This means that within a group of 8 bits (1 byte), we read right-to-left. However, the `nanoarrow` repr of the validity buffer in the example above already takes that into account and shows the values in logical order matching the position in the array. \n",
    "\n",
    "The diagram above shows it as how it is actually stored in memory. We can inspect the validity bitmap buffer with pyarrow and numpy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "777ac7d4-2950-49a4-8310-67a0261ecc25",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "DuIwrjV7Vqlb",
    "outputId": "394ceb73-cf0e-4c91-f216-aa613ff3adca"
   },
   "outputs": [],
   "source": [
    "validity_bitmap_buffer = column2.buffers()[0]\n",
    "validity_bitmap_buffer.to_pybytes()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf7e8c8a-440f-482a-989f-765677f4b767",
   "metadata": {},
   "source": [
    "In this case of a small array of 5 values, the validity bitmap consists of only a single byte. To view the data as bytes in numpy, we can use the `uint8` data type, which has a width of 1 byte:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "382a891e-7dd6-4f4f-a586-982ece31e6e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "np.frombuffer(validity_bitmap_buffer, dtype=\"uint8\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ead7fd5b-bb46-4fd6-a706-df7339e58067",
   "metadata": {},
   "source": [
    "Numpy also provides a function to \"unpack\" the 0/1 bits of those bytes into separate values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "940fe345-4dfd-481c-8f80-99cbe15bdfbc",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ZvPrUkqOVrnI",
    "outputId": "8ddb9875-2584-4118-d332-f668907a5da7"
   },
   "outputs": [],
   "source": [
    "np.unpackbits(np.frombuffer(validity_bitmap_buffer, dtype=\"uint8\"), bitorder=\"little\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae52765d-2c05-42ff-982f-c6474a4663b4",
   "metadata": {},
   "source": [
    "In this case of an array of 5 elements, only the first 5 bits have a meaning, and the additional (\"padded\") bits are always set to 0."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7e69ef5-5882-40f4-9516-d399dd4288a9",
   "metadata": {
    "id": "wcqi2PDlWa2w"
   },
   "source": [
    "### Null vs NaN\n",
    "\n",
    "In numpy (and numpy-based packages such as pandas), often `NaN` is used as indicator for \"missing\" values, mostly by lack of better alternatives (numpy does not have built-in support for missing values in general). `NaN` is a specific floating-point value (\"Not a Number\") within the IEEE floating-point standard, and thus is only available for floating point data types.\n",
    "In the Arrow format, since there is a separate concept of nulls, a NaN value is considered as just another valid floating point array value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e518a337-c569-47e4-955b-4d84b5363beb",
   "metadata": {},
   "outputs": [],
   "source": [
    "arr = na.array([0.5, float(\"nan\"), 1.5, None, 3.5], na.float64())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cd20814c-7bcb-4d75-9e9c-dad101bf95cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "arr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "649df3ea-5a82-4406-86a7-04b25a711bf1",
   "metadata": {},
   "outputs": [],
   "source": [
    "arr.buffers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea07d28f-8ac9-40c8-bc1a-086de48f5215",
   "metadata": {},
   "source": [
    "## Exercise\n",
    "\n",
    "In the following code snippet, we create an Array object from python `datetime` instances. What is the type of the array? Is this a fixed-width primitive type? How are the datetimes expressed?\n",
    "\n",
    "<details><summary>Hints</summary>\n",
    "\n",
    "* A `pyarrow.Array` has a `.type` attribute. And this DataType object has a `byte_width` attribute in case of a fixed-width type.\n",
    "* Does it just have a single data buffer (next to the validity bitmap)?\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e35f541a-fae5-4d05-afe6-4d2b2d0977c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "column_datetime = pa.array([datetime(2024, 4, 22), datetime(2024, 4, 23), datetime(2024, 4, 24)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe370027-9ec2-4160-9557-5cfb7529cf34",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "30900ff8-604d-4935-a295-da8277a8a166",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "b7eb6433-0bf6-48fe-bb67-3af94219c7ad",
   "metadata": {},
   "source": [
    "# Variable length binary and string\n",
    "\n",
    "The bytes of a binary or string column are stored together consecutively in a single buffer or region of memory. To know where each element of the column starts and ends the physical layout also includes integer offsets. The length of which is one more than the length on the column as the last two elements define the start and the end of the last element in the binary/string column.\n",
    "\n",
    "Binary and string types share the same physical layout with where the string type is utf-8 binary and will produce an invalid result if the bytes are not valid utf-8.\n",
    "\n",
    "The difference between binary/string and large binary/string is in the offset type. In the first case that is `int32` and in the second it is `int64`.\n",
    "\n",
    "The limitation of types using 32 bit offsets is that they have a max size of 2GB for one array/column. One can still use the non-large variants for bigger data, but then multiple chunks are needed.\n",
    "\n",
    "![image info](./diagrams/var-string-diagram.svg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e686c493-ef22-450d-8002-1a389105b00c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Binary column example\n",
    "pa.array(['python', 'data', 'conference', None, \"Berlin\"], type=pa.binary())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e1e924c-b0b3-4127-9717-5a27772d3b38",
   "metadata": {},
   "source": [
    "The bytes in the BinaryArray are shown in the \"hex\" representation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1f16fc8-ea7d-4346-ba86-9030264ee48c",
   "metadata": {},
   "outputs": [],
   "source": [
    "bytes.fromhex(\"707974686F6E\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e065f0a-3383-4736-80d2-cdcd965f018d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# String column examples\n",
    "pa.array(['python', 'data', 'conference', None, \"Berlin\"], type=pa.string())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ef62f4b-d28b-43d5-8595-37672381bf07",
   "metadata": {},
   "source": [
    "### String type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cec629d5-7603-458c-9429-43f31a27630e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inspecting buffers using PyArrow and buffers() method\n",
    "\n",
    "column4 = pa.array(['python', 'data', 'conference', None, \"Berlin\"], type=pa.string())\n",
    "column4.buffers()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3bdad631-0402-4a55-9b53-93067072fc1d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inspecting buffers using PyArrow and buffers() method and numpy\n",
    "\n",
    "validity_bitmap_buffer = column4.buffers()[0]\n",
    "np.unpackbits(np.frombuffer(validity_bitmap_buffer, dtype=\"uint8\"), bitorder=\"little\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef8da28e-8dbe-4aec-83c2-95af5450a2b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "offsets_buffer = column4.buffers()[1]\n",
    "np.frombuffer(offsets_buffer, dtype=\"int32\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "91d27f13-5f28-4263-902e-89e85ef78869",
   "metadata": {},
   "outputs": [],
   "source": [
    "values_buffer = column4.buffers()[2]\n",
    "values_buffer.to_pybytes()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27036429-79ce-4ded-9f6a-688d739948aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inspecting buffers using nanoarrow\n",
    "\n",
    "na_column4 = na.array(column4)\n",
    "na_column4.inspect()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97c4d4d7-59b5-48bd-a601-3f79bfc9dfb1",
   "metadata": {},
   "source": [
    "### Binary type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9122c87e-ec85-42ce-b241-45396397744b",
   "metadata": {},
   "outputs": [],
   "source": [
    "column4 = pa.array(['python', 'data', 'conference', None, \"Berlin\"], type=pa.binary())\n",
    "column4.buffers()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78cf888c-f35b-4655-a315-dfb0fd9e9a12",
   "metadata": {},
   "outputs": [],
   "source": [
    "na_column4 = na.array(column4)\n",
    "na_column4.inspect()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4c363b6-9b9b-4cf2-9748-321fe01023f5",
   "metadata": {},
   "source": [
    "### Comparing string and large string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b4f3bcf-1c69-4915-8e40-b1bcc3e65c65",
   "metadata": {},
   "outputs": [],
   "source": [
    "column4 = pa.array(['python', 'data', 'conference', None, \"Berlin\"], type=pa.string())\n",
    "na.array(column4).inspect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d52e9b2b-4b5f-4b8e-afde-8a7b7f951636",
   "metadata": {},
   "outputs": [],
   "source": [
    "column4 = pa.array(['python', 'data', 'conference', None, \"Berlin\"], type=pa.large_string())\n",
    "na.array(column4).inspect()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "759cd037-047f-44a4-adcd-bd6dfa9a8314",
   "metadata": {},
   "source": [
    "### Variable length binary and string view\n",
    "\n",
    "Binary and string view layout are new in Arrow Columnar format 1.4. This layout is adapted from TU Munich's UmbraDB, and similar to the string layout used in DuckDB and Velox (and sometimes also called \"German style strings\").\n",
    "\n",
    "The main differences to classical binary and string types is the **views buffer**. It includes the length of the string, and then either contains the characters inline (for small strings) or either only contains the first 4 bytes of the string and point to potentially several data buffers. It also supports binary and strings to be written out of order.\n",
    "\n",
    "These properties are important for efficient string processing. The prefix enables a profitable fast path for string comparisons, which are frequently determined within the first four bytes. Selecting elements is a simple \"take\" operations on the fixed-width views buffer and does not need to rewrite the values buffers.\n",
    "\n",
    "![image info](./diagrams/var-string-view-diagram.svg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a051b89-f186-4b33-8d28-50a9629c5a28",
   "metadata": {},
   "outputs": [],
   "source": [
    "column5 = pa.array(['String longer than 12', 'Short', None, 'Short string', \"Another long string\"], type=pa.string_view())\n",
    "column5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "142a3763-5bae-4a97-bd33-6fc96ce27e34",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inspecting buffers using PyArrow and buffers() method\n",
    "column5.buffers()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bbf24001-5025-46b9-b757-3a1118b81b82",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inspecting buffers using PyArrow and buffers() method and numpy\n",
    "\n",
    "validity_bitmap_buffer = column5.buffers()[0]\n",
    "np.unpackbits(np.frombuffer(validity_bitmap_buffer, dtype=\"uint8\"), bitorder=\"little\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f06431c-320b-4d82-9a27-8a4e47c8644b",
   "metadata": {},
   "outputs": [],
   "source": [
    "views_buffer = column5.buffers()[1]\n",
    "np.frombuffer(views_buffer, dtype=\"int32\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cdb8f77e-4fe3-4df5-96c0-3426eb8fd139",
   "metadata": {},
   "outputs": [],
   "source": [
    "views_buffer.to_pybytes()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bcd6889d-bc65-40a4-b3e8-c39e40c3f7ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "values_buffer = column5.buffers()[2]\n",
    "values_buffer.to_pybytes()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0660ca11-f6a0-405b-8d98-25372c45a48e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import struct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09f50f8f-b464-45aa-a698-c43acefe54e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "struct.unpack(\"i4sii\", views_buffer.to_pybytes()[:16])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5572ca84-7db6-437b-975a-05473953940e",
   "metadata": {},
   "outputs": [],
   "source": [
    "struct.unpack(\"i12s\", views_buffer.to_pybytes()[16:32])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42c2a155-8b05-4cb5-b3a6-f0c4041ce556",
   "metadata": {},
   "outputs": [],
   "source": [
    "column6 = pa.concat_arrays(\n",
    "    [column5, pa.array([\"PyConDE / PyData Berlin\"], type=pa.string_view())]\n",
    ")\n",
    "column6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e3d8bc3-e1b3-4392-a645-ef0d20c9ae1e",
   "metadata": {},
   "outputs": [],
   "source": [
    "column6.buffers()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92402d69-9f9a-420e-9fee-e86c7c95cf1f",
   "metadata": {},
   "outputs": [],
   "source": [
    "views_bytes = column6.buffers()[1].to_pybytes()\n",
    "\n",
    "for i in range(0, len(views_bytes), 16):\n",
    "    length, = struct.unpack(\"i\", views_bytes[i:i+4])\n",
    "    if length > 12:\n",
    "        print(struct.unpack(\"i4sii\", views_bytes[i:i+16]))\n",
    "    else:\n",
    "        print(struct.unpack(\"i12s\", views_bytes[i:i+16]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "643dbbc3-d85d-4d5b-9b51-87bce1a7d25f",
   "metadata": {},
   "source": [
    "## Exercise\n",
    "\n",
    "In the following code snippet, we create an Array object of byte objects, all with the same size. Can you see what's the difference with the previous binary and string arrays we have seen?\n",
    "\n",
    "<details><summary>Hints</summary>\n",
    "\n",
    "* How many buffers does it have? Does it have an offsets buffer?\n",
    "* Is it a variable-size or fixed-size layout?\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9526ed0-a91c-4fca-aebb-d9c670ef7137",
   "metadata": {},
   "outputs": [],
   "source": [
    "column7 = pa.array([b\"some\", b\"byte\", b\"data\"], pa.binary(4)) \n",
    "column7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "213bc905-2ea5-417c-b034-6a3e998ed71a",
   "metadata": {},
   "outputs": [],
   "source": [
    "column7.to_pylist()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}