main.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "RJLZFE1t2k9r",
      "metadata": {
        "id": "RJLZFE1t2k9r"
      },
      "source": [
        "# Project: Question-Answering using Retrieval Augmented Generation\n",
        "by L.Arduini, D.N.Ghaneh, L.Menchini, C.Petruzzella\n",
        "\n",
        "## Instructions to Run\n",
        "We recommend running the project on Google Colab. In particular, we suggest selecting a runtime with GPU support, such as the \"T4 GPU\", to access better computational resources, which will result in reduced execution time."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5f3ad8b0",
      "metadata": {
        "id": "5f3ad8b0"
      },
      "source": [
        "# Intelligent Retrieval and QA System Evaluation\n",
        "\n",
        "## Overview\n",
        "\n",
        "This project focuses on the comparative analysis of two advanced retrieval methods: **Rank Fusion** and **Cascading Retrieval**. The aim is to build a pipeline capable of retrieving relevant documents for a given query, generating responses using a large language model (Llama 3.2-1B), and evaluating those responses for quality and relevance.\n",
        "\n",
        "## Key Components\n",
        "\n",
        "1. **Document Retrieval**:\n",
        "    - Implementation of multiple retrieval strategies:\n",
        "        - **Sparse Retrieval**: BM25 for term-based matching.\n",
        "        - **Dense Retrieval**: Embedding-based semantic similarity.\n",
        "        - **Rank Fusion**: Combining sparse and dense retrieval scores.\n",
        "        - **Cascading Retrieval**: Combine sparse and dense retrieval results with a cross encoder and reranker.\n",
        "        - each single method has been evaluated using *NDCG* and *RECALL* as quantitative metrics.\n",
        "    - Dataset: **TREC-COVID**, simulating real-world information retrieval challenges.\n",
        "\n",
        "2. **Question-Answering with LLM**:\n",
        "    - Responses are generated for each query using a state-of-the-art language model. Retrieved documents serve as context to provide more accurate answers.\n",
        "\n",
        "3. **Evaluation Framework**:\n",
        "    - Human-centric evaluation to assess response quality:\n",
        "        - A numerical relevance score ranging from **0 (irrelevant)** to **2 (highly relevant)**.\n",
        "        - A brief **motivation** accompanying each evaluation score for better interpretability.\n",
        "\n",
        "4. **Consolidated Output**:\n",
        "    - For each query, the following information is logged:\n",
        "        - Query text.\n",
        "        - Retrieved context.\n",
        "        - Model-generated response.\n",
        "        - Human evaluation score and motivation.\n",
        "\n",
        "## Objective\n",
        "\n",
        "The primary objective of this notebook is to demonstrate the **end-to-end design and evaluation of a retrieval-enhanced QA system**, highlighting its applicability in real-world scenarios such as medical information retrieval.\n",
        "\n",
        "Through this work, we aim to:\n",
        "- Explore and enhance document retrieval strategies by combining traditional and modern techniques.\n",
        "- Analyze the strengths and limitations of LLMs in QA tasks.\n",
        "- Provide a structured framework for evaluating the relevance and quality of LLM-generated responses.\n",
        "\n",
        "---\n",
        "\n",
        "### Structure\n",
        "\n",
        "- **Document Retrieval**: Implementation and comparison of retrieval methods.\n",
        "- **QA Generation**: Leveraging an LLM to generate context-aware responses.\n",
        "- **Response Evaluation**: Scoring and analysis of the responses with human-annotated motivations.\n",
        "\n",
        "This notebook is designed to offer insights into retrieval-augmented QA systems while enabling practical understanding of their performance and potential improvements.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "Av24RrStoD2G",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Av24RrStoD2G",
        "outputId": "1483d0d9-23da-411f-9647-524cdca3418d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Defaulting to user installation because normal site-packages is not writeable\n",
            "Requirement already satisfied: ir_datasets in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (0.5.8)\n",
            "Requirement already satisfied: beautifulsoup4>=4.4.1 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from ir_datasets) (4.12.3)\n",
            "Requirement already satisfied: inscriptis>=2.2.0 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from ir_datasets) (2.5.0)\n",
            "Requirement already satisfied: lxml>=4.5.2 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from ir_datasets) (5.3.0)\n",
            "Requirement already satisfied: numpy>=1.18.1 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from ir_datasets) (1.26.4)\n",
            "Requirement already satisfied: pyyaml>=5.3.1 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from ir_datasets) (6.0.2)\n",
            "Requirement already satisfied: requests>=2.22.0 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from ir_datasets) (2.32.3)\n",
            "Requirement already satisfied: tqdm>=4.38.0 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from ir_datasets) (4.66.5)\n",
            "Requirement already satisfied: trec-car-tools>=2.5.4 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from ir_datasets) (2.6)\n",
            "Requirement already satisfied: lz4>=3.1.10 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from ir_datasets) (4.3.3)\n",
            "Requirement already satisfied: warc3-wet>=0.2.3 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from ir_datasets) (0.2.5)\n",
            "Requirement already satisfied: warc3-wet-clueweb09>=0.2.5 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from ir_datasets) (0.2.5)\n",
            "Requirement already satisfied: zlib-state>=0.1.3 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from ir_datasets) (0.1.9)\n",
            "Requirement already satisfied: ijson>=3.1.3 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from ir_datasets) (3.3.0)\n",
            "Requirement already satisfied: unlzw3>=0.2.1 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from ir_datasets) (0.2.2)\n",
            "Requirement already satisfied: soupsieve>1.2 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from beautifulsoup4>=4.4.1->ir_datasets) (2.6)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from requests>=2.22.0->ir_datasets) (3.4.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from requests>=2.22.0->ir_datasets) (3.10)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from requests>=2.22.0->ir_datasets) (2.2.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from requests>=2.22.0->ir_datasets) (2024.8.30)\n",
            "Requirement already satisfied: colorama in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from tqdm>=4.38.0->ir_datasets) (0.4.6)\n",
            "Requirement already satisfied: cbor>=1.0.0 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from trec-car-tools>=2.5.4->ir_datasets) (1.0.0)\n",
            "Defaulting to user installation because normal site-packages is not writeable\n",
            "Requirement already satisfied: rank_bm25 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (0.2.2)\n",
            "Requirement already satisfied: numpy in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from rank_bm25) (1.26.4)\n",
            "^C\n",
            "Defaulting to user installation because normal site-packages is not writeable\n",
            "Requirement already satisfied: sentence_transformers in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (3.3.1)\n",
            "Requirement already satisfied: transformers<5.0.0,>=4.41.0 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from sentence_transformers) (4.46.2)\n",
            "Requirement already satisfied: tqdm in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from sentence_transformers) (4.66.5)\n",
            "Requirement already satisfied: torch>=1.11.0 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from sentence_transformers) (2.5.1)\n",
            "Requirement already satisfied: scikit-learn in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from sentence_transformers) (1.5.2)\n",
            "Requirement already satisfied: scipy in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from sentence_transformers) (1.14.1)\n",
            "Requirement already satisfied: huggingface-hub>=0.20.0 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from sentence_transformers) (0.26.2)\n",
            "Requirement already satisfied: Pillow in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from sentence_transformers) (11.0.0)\n",
            "Requirement already satisfied: filelock in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from huggingface-hub>=0.20.0->sentence_transformers) (3.16.1)\n",
            "Requirement already satisfied: fsspec>=2023.5.0 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from huggingface-hub>=0.20.0->sentence_transformers) (2024.9.0)\n",
            "Requirement already satisfied: packaging>=20.9 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from huggingface-hub>=0.20.0->sentence_transformers) (24.1)\n",
            "Requirement already satisfied: pyyaml>=5.1 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from huggingface-hub>=0.20.0->sentence_transformers) (6.0.2)\n",
            "Requirement already satisfied: requests in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from huggingface-hub>=0.20.0->sentence_transformers) (2.32.3)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from huggingface-hub>=0.20.0->sentence_transformers) (4.12.2)\n",
            "Requirement already satisfied: networkx in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from torch>=1.11.0->sentence_transformers) (3.4.2)\n",
            "Requirement already satisfied: jinja2 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from torch>=1.11.0->sentence_transformers) (3.1.4)\n",
            "Requirement already satisfied: setuptools in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from torch>=1.11.0->sentence_transformers) (75.3.0)\n",
            "Requirement already satisfied: sympy==1.13.1 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from torch>=1.11.0->sentence_transformers) (1.13.1)\n",
            "Requirement already satisfied: mpmath<1.4,>=1.1.0 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from sympy==1.13.1->torch>=1.11.0->sentence_transformers) (1.3.0)\n",
            "Requirement already satisfied: colorama in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from tqdm->sentence_transformers) (0.4.6)\n",
            "Requirement already satisfied: numpy>=1.17 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from transformers<5.0.0,>=4.41.0->sentence_transformers) (1.26.4)\n",
            "Requirement already satisfied: regex!=2019.12.17 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from transformers<5.0.0,>=4.41.0->sentence_transformers) (2024.9.11)\n",
            "Requirement already satisfied: safetensors>=0.4.1 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from transformers<5.0.0,>=4.41.0->sentence_transformers) (0.4.5)\n",
            "Requirement already satisfied: tokenizers<0.21,>=0.20 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from transformers<5.0.0,>=4.41.0->sentence_transformers) (0.20.3)\n",
            "Requirement already satisfied: joblib>=1.2.0 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from scikit-learn->sentence_transformers) (1.4.2)\n",
            "Requirement already satisfied: threadpoolctl>=3.1.0 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from scikit-learn->sentence_transformers) (3.5.0)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from jinja2->torch>=1.11.0->sentence_transformers) (3.0.2)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from requests->huggingface-hub>=0.20.0->sentence_transformers) (3.4.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from requests->huggingface-hub>=0.20.0->sentence_transformers) (3.10)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from requests->huggingface-hub>=0.20.0->sentence_transformers) (2.2.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from requests->huggingface-hub>=0.20.0->sentence_transformers) (2024.8.30)\n",
            "Defaulting to user installation because normal site-packages is not writeable\n",
            "Requirement already satisfied: pytrec_eval in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (0.5)\n",
            "Defaulting to user installation because normal site-packages is not writeable\n",
            "Requirement already satisfied: PyStemmer in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (2.2.0.3)\n",
            "Defaulting to user installation because normal site-packages is not writeable\n",
            "Requirement already satisfied: requests in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (2.32.3)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from requests) (3.4.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from requests) (3.10)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from requests) (2.2.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from requests) (2024.8.30)\n",
            "Defaulting to user installation because normal site-packages is not writeable\n",
            "Collecting gdown\n",
            "  Downloading gdown-5.2.0-py3-none-any.whl.metadata (5.8 kB)\n",
            "Requirement already satisfied: beautifulsoup4 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from gdown) (4.12.3)\n",
            "Requirement already satisfied: filelock in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from gdown) (3.16.1)\n",
            "Requirement already satisfied: requests[socks] in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from gdown) (2.32.3)\n",
            "Requirement already satisfied: tqdm in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from gdown) (4.66.5)\n",
            "Requirement already satisfied: soupsieve>1.2 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from beautifulsoup4->gdown) (2.6)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from requests[socks]->gdown) (3.4.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from requests[socks]->gdown) (3.10)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from requests[socks]->gdown) (2.2.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from requests[socks]->gdown) (2024.8.30)\n",
            "Collecting PySocks!=1.5.7,>=1.5.6 (from requests[socks]->gdown)\n",
            "  Downloading PySocks-1.7.1-py3-none-any.whl.metadata (13 kB)\n",
            "Requirement already satisfied: colorama in c:\\users\\loren\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.12_qbz5n2kfra8p0\\localcache\\local-packages\\python312\\site-packages (from tqdm->gdown) (0.4.6)\n",
            "Downloading gdown-5.2.0-py3-none-any.whl (18 kB)\n",
            "Downloading PySocks-1.7.1-py3-none-any.whl (16 kB)\n",
            "Installing collected packages: PySocks, gdown\n",
            "Successfully installed PySocks-1.7.1 gdown-5.2.0\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "  WARNING: The script gdown.exe is installed in 'C:\\Users\\loren\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python312\\Scripts' which is not on PATH.\n",
            "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\n"
          ]
        }
      ],
      "source": [
        "# Install required Python packages\n",
        "!pip install ir_datasets\n",
        "!pip install rank_bm25\n",
        "!pip install sentence_transformers\n",
        "!pip install pytrec_eval\n",
        "!pip install PyStemmer\n",
        "!pip install requests\n",
        "!pip install --upgrade gdown"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "uHAUTJ99oCgI",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uHAUTJ99oCgI",
        "outputId": "a71019e5-86b6-4ba3-a375-a56b0c272ba4"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Using GPU: Tesla T4\n",
            "CUDA Version: 12.1\n",
            "CUDA Cores: 40\n",
            "Total Memory: 15.84 GB\n",
            "Compute Capability: 7.5\n"
          ]
        }
      ],
      "source": [
        "# Import necessary libraries and initialize global configurations\n",
        "from tqdm import tqdm\n",
        "import json\n",
        "import ir_datasets\n",
        "import os\n",
        "from sentence_transformers import SentenceTransformer\n",
        "from sklearn.metrics.pairwise import cosine_similarity\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "from rank_bm25 import BM25Okapi\n",
        "from transformers import pipeline\n",
        "import torch\n",
        "from huggingface_hub import login\n",
        "import pytrec_eval\n",
        "import collections\n",
        "import itertools\n",
        "import heapq\n",
        "from google.colab import drive\n",
        "import os\n",
        "import shutil\n",
        "\n",
        "api_key = \"YOUR_API_KEY\"\n",
        "login(token=api_key)\n",
        "\n",
        "# Check GPU availability\n",
        "def get_device():\n",
        "    if torch.cuda.is_available():\n",
        "        device = \"cuda\"\n",
        "        cuda_version = torch.version.cuda  # Retrieve CUDA version\n",
        "        gpu_properties = torch.cuda.get_device_properties(torch.cuda.current_device())\n",
        "        print(f\"Using GPU: {gpu_properties.name}\")\n",
        "        print(f\"CUDA Version: {cuda_version}\")\n",
        "        print(f\"CUDA Cores: {gpu_properties.multi_processor_count}\")\n",
        "        print(f\"Total Memory: {gpu_properties.total_memory / 1e9:.2f} GB\")\n",
        "        print(f\"Compute Capability: {gpu_properties.major}.{gpu_properties.minor}\")\n",
        "    elif torch.backends.mps.is_available():\n",
        "        device = \"mps\"\n",
        "        print(\"Using MPS (Metal Performance Shaders)\")\n",
        "    else:\n",
        "        device = \"cpu\"\n",
        "        print(\"Using CPU\")\n",
        "    return device\n",
        "\n",
        "device = get_device()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "aafc8257",
      "metadata": {
        "id": "aafc8257"
      },
      "source": [
        "# Section 1: Dataset loading and preparation\n",
        "\n",
        "This section focuses on loading and preparing the dataset for the QA model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "08dfaab3",
      "metadata": {
        "id": "08dfaab3"
      },
      "outputs": [],
      "source": [
        "# Define functions for text preprocessing and tokenization\n",
        "from functools import lru_cache\n",
        "import re\n",
        "import string\n",
        "import Stemmer\n",
        "import nltk\n",
        "nltk.download(\"stopwords\", quiet=True)\n",
        "\n",
        "# ------- Pre Initialization -------\n",
        "# 1. Compile regex patterns once globally\n",
        "# 2. Preload stopwords set\n",
        "# 3. Initialize stemmer\n",
        "\n",
        "ACRONYM_REGEX = re.compile(r\"(?<!\\w)\\.(?!\\d)\")\n",
        "PUNCTUATION_TRANS = str.maketrans(\"\", \"\", string.punctuation)\n",
        "STOPWORDS = set(nltk.corpus.stopwords.words('english'))\n",
        "STEMMER = Stemmer.Stemmer('english')\n",
        "\n",
        "# Define a cached function to stem individual words\n",
        "@lru_cache(maxsize=1000)\n",
        "def stem(word):\n",
        "    return STEMMER.stemWord(word)\n",
        "\n",
        "# ----------------------------------\n",
        "\n",
        "def preprocess(s):\n",
        "    \"\"\"\n",
        "    Preprocess a string for indexing or querying.\n",
        "\n",
        "    Args:\n",
        "        s: The input string.\n",
        "\n",
        "    Returns:\n",
        "        A list of preprocessed tokens.\n",
        "    \"\"\"\n",
        "\n",
        "    s = s.lower()\n",
        "    s = s.replace(\"&\", \" and \")\n",
        "    # normalize quotes and dashes\n",
        "    s = s.translate(str.maketrans(\"‘’´“”–-\", \"'''\\\"\\\"--\"))\n",
        "    # remove unnecessary dots in acronyms (but not decimals)\n",
        "    s = ACRONYM_REGEX.sub(\"\", s)\n",
        "    # remove punctuation\n",
        "    s = s.translate(PUNCTUATION_TRANS)\n",
        "    # strip and remove extra spaces\n",
        "    s = \" \".join(s.split())\n",
        "\n",
        "    tokens = s.split()\n",
        "    tokens = [t for t in tokens if t not in STOPWORDS]\n",
        "    tokens = STEMMER.stemWords(tokens)\n",
        "    return tokens"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "niSTrLIAjZno",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "niSTrLIAjZno",
        "outputId": "873a74ea-6f58-4c29-bae9-968b1dcd58a6"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Loading the trec covid dataset...\n"
          ]
        }
      ],
      "source": [
        "# Load dataset\n",
        "print(\"Loading the trec covid dataset...\")\n",
        "dataset = ir_datasets.load(\"cord19/trec-covid\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "eaHfXbNmYI4W",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eaHfXbNmYI4W",
        "outputId": "1d0764f8-8fd7-4cee-b633-e8f4a4246ec1"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Dataset length: 192509\n",
            "Number of documents with duplicate abstracts: 66793\n",
            "Removing documents with empty or null abstracts...\n",
            "Removing documents with duplicate abstracts...\n",
            "Cleaned dataset length: 125716\n"
          ]
        }
      ],
      "source": [
        "import pandas as pd\n",
        "\n",
        "# Convert the dataset to a pandas DataFrame for easier manipulation\n",
        "df = pd.DataFrame(dataset.docs_iter(), columns=['doc_id', 'title', 'doi', 'date', 'abstract'])\n",
        "\n",
        "# Check length of the dataset\n",
        "print(f\"Dataset length: {len(df)}\")\n",
        "\n",
        "# Check number of documents with duplicate abstracts\n",
        "print(f\"Number of documents with duplicate abstracts: {df['abstract'].duplicated().sum()}\")\n",
        "\n",
        "# Remove documents with empty or null abstracts\n",
        "print(\"Removing documents with empty or null abstracts...\")\n",
        "data_cleaned = df[~df['abstract'].isnull() & (df['abstract'].str.strip() != '')]\n",
        "\n",
        "# Remove documents with duplicate abstracts\n",
        "print(\"Removing documents with duplicate abstracts...\")\n",
        "docs_dataset = df.drop_duplicates(subset='abstract')\n",
        "\n",
        "# Check dataset length\n",
        "print(f\"Cleaned dataset length: {len(docs_dataset)}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "393aded9-d0ac-45b7-ae2d-b4783fc021c1",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "393aded9-d0ac-45b7-ae2d-b4783fc021c1",
        "outputId": "847a3463-622d-4517-9447-222cd99eecf6"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Preparing documents and queries...\n",
            "Summary: 125716 documents and 50 queries are available in the dataset.\n",
            "Tokenization of documents is done.\n"
          ]
        }
      ],
      "source": [
        "# Define functions for text preprocessing and tokenization\n",
        "# Prepare documents and queries\n",
        "print(\"Preparing documents and queries...\")\n",
        "\n",
        "# put all documents and queries in a list of dictionaries\n",
        "all_docs = []\n",
        "for index, row in docs_dataset.iterrows():\n",
        "    abstract = f\"{row['title']} {row['abstract']}\"\n",
        "    all_docs.append({\"doc_id\": row['doc_id'], \"abstract\": abstract, \"context\": row['abstract']})\n",
        "\n",
        "all_queries = []\n",
        "for query in dataset.queries_iter():\n",
        "    query_text = f\"{query.description}\"\n",
        "    all_queries.append({\"query_id\": query.query_id, \"title\": query_text})\n",
        "\n",
        "# Print dataset size information\n",
        "print(f\"Summary: {len(all_docs)} documents and {len(all_queries)} queries are available in the dataset.\")\n",
        "\n",
        "# Tokenize documents\n",
        "tokenized_docs = [preprocess(doc) for doc in [docs[\"abstract\"] for docs in all_docs]]\n",
        "tokenized_queries = [preprocess(query) for query in [queries[\"title\"] for queries in all_queries]]\n",
        "print(\"Tokenization of documents is done.\")\n",
        "\n",
        "bm25 = BM25Okapi(tokenized_docs)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "zdvajGS135b2",
      "metadata": {
        "id": "zdvajGS135b2"
      },
      "outputs": [],
      "source": [
        "# convert qrels to a dictionary\n",
        "qrels_dict = collections.defaultdict(dict)\n",
        "for qrel in dataset.qrels_iter():\n",
        "    qrels_dict[qrel.query_id][qrel.doc_id] = int(qrel.relevance)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "2f863229",
      "metadata": {
        "id": "2f863229"
      },
      "source": [
        "# Section 2: Embeddings generation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b2526980",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "b2526980",
        "outputId": "7438c6a5-b084-4651-a2b4-e9f8cf271813"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n",
            "Retrieving folder contents\n",
            "Processing file 1qaX7rC1TcT1smqs6TROv2LsoF1FLHiI3 trec_covid_doc_embeddings.csv\n",
            "Processing file 1WGH8XgI4TXsaymzt9ji4-0_Mg7NVsXPo trec_covid_query_embeddings.csv\n",
            "Retrieving folder contents completed\n",
            "Building directory structure\n",
            "Building directory structure completed\n",
            "Downloading...\n",
            "From (original): https://drive.google.com/uc?id=1qaX7rC1TcT1smqs6TROv2LsoF1FLHiI3\n",
            "From (redirected): https://drive.google.com/uc?id=1qaX7rC1TcT1smqs6TROv2LsoF1FLHiI3&confirm=t&uuid=b829000c-0652-4fdf-aa93-868b8834aa2d\n",
            "To: /content/lm-project-files/trec_covid_doc_embeddings.csv\n",
            "100% 591M/591M [00:06<00:00, 92.5MB/s]\n",
            "Downloading...\n",
            "From: https://drive.google.com/uc?id=1WGH8XgI4TXsaymzt9ji4-0_Mg7NVsXPo\n",
            "To: /content/lm-project-files/trec_covid_query_embeddings.csv\n",
            "100% 236k/236k [00:00<00:00, 4.09MB/s]\n",
            "Download completed\n"
          ]
        }
      ],
      "source": [
        "# From Google Drive import embeddings\n",
        "drive.mount('/content/drive')\n",
        "\n",
        "repository = \"YOUR_REPOSITORY\"\n",
        "repository_name = \"lm-project-files\"\n",
        "!gdown --folder $repository\n",
        "\n",
        "# Copia i file dalla cartella scaricata a /content/\n",
        "for item in os.listdir(repository_name):\n",
        "  s = os.path.join(repository_name, item)\n",
        "  d = os.path.join('/content/', item)\n",
        "  if os.path.isfile(s):  # Copia solo se è un file\n",
        "    shutil.copy2(s, d)\n",
        "\n",
        "# Rimuovi la cartella scaricata (opzionale)\n",
        "shutil.rmtree(repository_name)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "af4acfb7-4dd6-41e4-a35f-a6d60d917f76",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "af4acfb7-4dd6-41e4-a35f-a6d60d917f76",
        "outputId": "120c4b32-894b-4b5b-d880-bd6911386202"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Loading precomputed embeddings...\n"
          ]
        }
      ],
      "source": [
        "# Load or generate embeddings\n",
        "force_generate = False\n",
        "\n",
        "def generate_embeddings():\n",
        "    if not force_generate and os.path.exists(\"trec_covid_doc_embeddings.csv\") and os.path.exists(\"trec_covid_query_embeddings.csv\"):\n",
        "        print(\"Loading precomputed embeddings...\")\n",
        "        doc_embeddings = pd.read_csv(\"trec_covid_doc_embeddings.csv\").values\n",
        "        query_embeddings = pd.read_csv(\"trec_covid_query_embeddings.csv\").values\n",
        "    else:\n",
        "        print(\"No precomputed embeddings found.\")\n",
        "        print(\"Generating new embeddings using SentenceTransformer model 'sentence-transformers/all-MiniLM-L6-v2'.\")\n",
        "        model = SentenceTransformer(\"sentence-transformers/all-MiniLM-L6-v2\", device=device)\n",
        "        doc_embeddings = model.encode([doc[\"abstract\"] for doc in all_docs], batch_size=32, show_progress_bar=True, normalize_embeddings=True)\n",
        "        query_embeddings = model.encode([query['title'] for query in all_queries], batch_size=32, show_progress_bar=True, normalize_embeddings=True)\n",
        "\n",
        "        # Save embeddings for future use\n",
        "        pd.DataFrame(doc_embeddings).to_csv(\"trec_covid_doc_embeddings.csv\", index=False)\n",
        "        pd.DataFrame(query_embeddings).to_csv(\"trec_covid_query_embeddings.csv\", index=False)\n",
        "\n",
        "    return doc_embeddings, query_embeddings\n",
        "\n",
        "doc_embeddings, query_embeddings = generate_embeddings()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "520c17ca",
      "metadata": {
        "id": "520c17ca"
      },
      "source": [
        "# Section 3: Retrieval Implementation\n",
        "\n",
        "In this section, all the previously introduced retrieval methods are implemented. First, the function for running *pytrec evaluation* is defined, ensuring a robust assessment of the retrieval results.\n",
        "\n",
        "Then the experimental queries using each retrieval method are executed, storing the outcomes in a JSON file.\n",
        "\n",
        "For neural reranking, a pretrained cross-encoder and tokenizer from the `ms-marco-MiniLM-L-6-v2` model have been leveraged. This step is crucial for improving the ranking quality by precisely evaluating the relevance of query-document pairs through deep contextual understanding.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "bnATkuKEz2hJ",
      "metadata": {
        "id": "bnATkuKEz2hJ"
      },
      "outputs": [],
      "source": [
        "# Function to prepare run data for pytrec_eval\n",
        "def prepare_run_data(results):\n",
        "    \"\"\"\n",
        "    Prepares the run data in the format expected by pytrec_eval.\n",
        "    Converts numpy scores to native Python float for compatibility.\n",
        "    \"\"\"\n",
        "    run = {}\n",
        "    for query_results in results:\n",
        "        query_id = query_results['query']['query_id']\n",
        "        run[query_id] = {}\n",
        "        for doc_id, score in zip(query_results['results'], query_results['scores']):\n",
        "            run[query_id][doc_id] = float(score)  # Convert numpy type to float\n",
        "    return run"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "99364747",
      "metadata": {
        "id": "99364747"
      },
      "source": [
        "### Document Retrieval Methods\n",
        "\n",
        "1. **BM25 Sparse Retrieval**:\n",
        "   - The **BM25 algorithm** is used to perform sparse retrieval on tokenized documents by calculating a relevance score for each document based on the query. It then returns the indices and relevance scores of the top-k most relevant documents.\n",
        "\n",
        "2. **Dense Retrieval**:\n",
        "   - **Dense retrieval** is performed by calculating the cosine similarity between the query embedding and the document embeddings. The top-k documents with the highest similarity scores are returned.\n",
        "\n",
        "3. **Rank Fusion Retrieval**:\n",
        "   - Results from both **BM25** and **dense retrieval** are combined using a **rank fusion** technique. Scores from both methods are normalized and combined, then the documents are ranked based on the combined scores, returning the top k documents.\n",
        "\n",
        "4. **Cascading Retrieval**:\n",
        "   - Initially, a set of documents is retrieved using both Sparse and Dense Retrieval. Afterwards, a reranking step is made using a Reranker Model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "6raan8uKT6T8",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6raan8uKT6T8",
        "outputId": "52ba2ab9-c245-499d-ed4f-44aaf3c2c923"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_auth.py:94: UserWarning: \n",
            "The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
            "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
            "You will be able to reuse this secret in all of your notebooks.\n",
            "Please note that authentication is recommended but still optional to access public models or datasets.\n",
            "  warnings.warn(\n"
          ]
        }
      ],
      "source": [
        "from transformers import AutoTokenizer, AutoModelForSequenceClassification\n",
        "\n",
        "cross_encoder_model = AutoModelForSequenceClassification.from_pretrained('cross-encoder/ms-marco-MiniLM-L-6-v2').to(\"cuda\")\n",
        "cross_encoder_tokenizer = AutoTokenizer.from_pretrained('cross-encoder/ms-marco-MiniLM-L-6-v2')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "5640a8c1-7740-4d63-8f45-0ecd4d816706",
      "metadata": {
        "id": "5640a8c1-7740-4d63-8f45-0ecd4d816706"
      },
      "outputs": [],
      "source": [
        "# Define functions for text preprocessing and tokenization\n",
        "from scipy.stats import zscore\n",
        "\n",
        "# BM25 Sparse Retrieval\n",
        "def bm25_retrieve(query, bm25, top_k=5):\n",
        "    \"\"\"\n",
        "    Perform sparse retrieval using BM25 on the tokenized documents.\n",
        "    Returns the indices and scores of the top-k documents.\n",
        "    \"\"\"\n",
        "    tokenized_query = preprocess(query)                                     # Tokenize the query into words\n",
        "    scores = bm25.get_scores(tokenized_query)                                   # Get BM25 scores for all documents\n",
        "    top_k_indices = np.argsort(scores)[-top_k:][::-1]                           # Get indices of top-k documents based on BM25 score\n",
        "    return top_k_indices, scores[top_k_indices]\n",
        "\n",
        "# Dense Retrieval\n",
        "def dense_retrieve(query_embedding, doc_embeddings, top_k=5):\n",
        "    \"\"\"\n",
        "    Perform dense retrieval using cosine similarity between query and document embeddings.\n",
        "    Returns the indices and similarities of the top-k documents.\n",
        "    \"\"\"\n",
        "    similarities = cosine_similarity([query_embedding], doc_embeddings)[0]      # Compute cosine similarity\n",
        "    top_k_indices = np.argsort(similarities)[-top_k:][::-1]                     # Get top-k indices based on similarity\n",
        "    return top_k_indices, similarities[top_k_indices]\n",
        "\n",
        "\n",
        "# rank fusion retrieval\n",
        "def combsum_fusion(dense_indices, dense_scores, sparse_indices, sparse_scores, top_k=5):\n",
        "    # Combine indices and scores from dense and sparse sources\n",
        "    all_doc_ids = np.concatenate((sparse_indices, dense_indices))\n",
        "    all_scores = np.concatenate((sparse_scores, dense_scores))\n",
        "\n",
        "    # Aggregate scores for each document\n",
        "    combined_scores = collections.defaultdict(float)\n",
        "    for doc_id, score in zip(all_doc_ids, all_scores):\n",
        "        combined_scores[doc_id] += score\n",
        "\n",
        "    # Retrieve top-k documents based on combined scores\n",
        "    top_docs = heapq.nlargest(top_k, combined_scores.items(), key=lambda x: x[1])\n",
        "\n",
        "    # return top k indices and scores\n",
        "    return [doc[0] for doc in top_docs], [doc[1] for doc in top_docs]\n",
        "\n",
        "# neural reranking for cascading retrieval\n",
        "def neural_rerank(query_text, dense_indices, dense_scores, sparse_indices, sparse_scores, top_k=5):\n",
        "\n",
        "    doc_ids = np.concatenate((sparse_indices, dense_indices))\n",
        "    documents = []\n",
        "    for doc in doc_ids:\n",
        "        documents.append(all_docs[doc]['abstract'])\n",
        "    features = cross_encoder_tokenizer([query_text]*len(documents), documents, padding=True, truncation=True, return_tensors=\"pt\").to(\"cuda\")\n",
        "    with torch.no_grad():\n",
        "      scores = cross_encoder_model(**features).logits\n",
        "\n",
        "    # Rerank the documents by scores\n",
        "    doc_scores = {doc_id: score.item() for doc_id, score in zip(doc_ids, scores)}\n",
        "    reranked_doc_scores = dict(sorted(doc_scores.items(), key=lambda item: item[1], reverse=True))\n",
        "\n",
        "    # return top indices and top scores\n",
        "    return list(reranked_doc_scores.keys())[:top_k], list(reranked_doc_scores.values())[:top_k]"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "85f3acd5",
      "metadata": {
        "id": "85f3acd5"
      },
      "source": [
        "The retrieval experiment is runned, saving results in a JSON file."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "c417195d",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "c417195d",
        "outputId": "1f0e696f-3d80-4548-b7a2-681690e64cab"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Running retrieval experiments on all queries.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 50/50 [00:54<00:00,  1.09s/it]\n"
          ]
        }
      ],
      "source": [
        "# Run retrieval experiments\n",
        "def run_retrieval_experiments():\n",
        "    \"\"\"\n",
        "    Execute sparse, dense, rank fusion, and cascading retrieval for all queries.\n",
        "    Save the results to a JSON file for further analysis.\n",
        "    \"\"\"\n",
        "    results = {\"sparse\": [], \"dense\": [], \"rank_fusion\": [], \"cascade\": []}\n",
        "\n",
        "    print(\"Running retrieval experiments on all queries.\")\n",
        "\n",
        "    # Iterate over each query and its embedding\n",
        "    for query, query_embedding in tqdm(zip(all_queries, query_embeddings), total=len(all_queries)):\n",
        "        # Extract the query ID and text for the current query\n",
        "        query_id = query['query_id']\n",
        "        query_text = query['title']\n",
        "\n",
        "        # Sparse Retrieval using BM25\n",
        "        sparse_indices, sparse_scores = bm25_retrieve(query_text, bm25)                 # Retrieve the top-k BM25 documents and their scores\n",
        "        sparse_docs = [all_docs[idx]['doc_id'] for idx in sparse_indices]               # Get document IDs from the indices\n",
        "\n",
        "        # Dense Retrieval using cosine similarity\n",
        "        dense_indices, dense_scores = dense_retrieve(query_embedding, doc_embeddings)   # Retrieve the top-k documents based on cosine similarity of embeddings\n",
        "        dense_docs = [all_docs[idx]['doc_id'] for idx in dense_indices]\n",
        "\n",
        "        # Normalize scores\n",
        "        sparse_scores = zscore(sparse_scores)\n",
        "        dense_scores = zscore(dense_scores)\n",
        "        results[\"sparse\"].append({\"query\": query, \"results\": sparse_docs, \"scores\": sparse_scores}) # Store the BM25 results for the current query\n",
        "        results[\"dense\"].append({\"query\": query, \"results\": dense_docs, \"scores\": dense_scores})\n",
        "\n",
        "        # Rank Fusion Retrieval by combining sparse (BM25) and dense result\n",
        "        fusion_indices, fusion_scores = combsum_fusion(dense_indices, dense_scores, sparse_indices, sparse_scores)\n",
        "        fusion_docs = [all_docs[idx]['doc_id'] for idx in fusion_indices]\n",
        "        results[\"rank_fusion\"].append({\"query\": query, \"results\": fusion_docs, \"scores\": fusion_scores})\n",
        "\n",
        "        # Cascade Retrieval: compute sparse and dense retrieval, then use reranker\n",
        "        cascade_indices, cascade_scores = neural_rerank(query_text, dense_indices, dense_scores, sparse_indices, sparse_scores)\n",
        "        cascade_docs = [all_docs[idx]['doc_id'] for idx in cascade_indices]\n",
        "        results[\"cascade\"].append({\"query\": query, \"results\": cascade_docs, \"scores\": cascade_scores})\n",
        "    return results\n",
        "\n",
        "results = run_retrieval_experiments()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "11c20134",
      "metadata": {
        "id": "11c20134"
      },
      "source": [
        "After the experiment, each method is evaluated by passing run results to pytrec_eval and the metrics are aggregated for a better visualization of the differences between methods."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "JOAqgAfe5W6H",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JOAqgAfe5W6H",
        "outputId": "ada6ccc9-a484-4ee5-d58b-096055fdaca4"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Aggregated results: {\n",
            "    \"sparse\": {\n",
            "        \"recall_5\": 0.008623629150449596,\n",
            "        \"ndcg_cut_5\": 0.6805147106092604\n",
            "    },\n",
            "    \"dense\": {\n",
            "        \"recall_5\": 0.008256141329265207,\n",
            "        \"ndcg_cut_5\": 0.6636285607092102\n",
            "    },\n",
            "    \"rank_fusion\": {\n",
            "        \"recall_5\": 0.008793696779750117,\n",
            "        \"ndcg_cut_5\": 0.7045910900616086\n",
            "    },\n",
            "    \"cascade\": {\n",
            "        \"recall_5\": 0.010368315231514647,\n",
            "        \"ndcg_cut_5\": 0.7947962701630851\n",
            "    }\n",
            "}\n",
            "Retrieval results and metrics saved to files.\n"
          ]
        }
      ],
      "source": [
        "run_sparse = prepare_run_data(results[\"sparse\"])\n",
        "run_dense = prepare_run_data(results[\"dense\"])\n",
        "run_rank_fusion = prepare_run_data(results[\"rank_fusion\"])\n",
        "run_cascade = prepare_run_data(results[\"cascade\"])\n",
        "\n",
        "# Evaluate results with pytrec_eval\n",
        "evaluator = pytrec_eval.RelevanceEvaluator(qrels_dict, {'recall.5', 'ndcg_cut.5'})\n",
        "eval_results_sparse = evaluator.evaluate(run_sparse)\n",
        "eval_results_dense = evaluator.evaluate(run_dense)\n",
        "eval_results_rank_fusion = evaluator.evaluate(run_rank_fusion)\n",
        "eval_results_cascade = evaluator.evaluate(run_cascade)\n",
        "\n",
        "# Aggregate metrics for overall performance\n",
        "aggregated_results = {\n",
        "    \"sparse\": {\n",
        "        metric: sum([res[metric] for res in eval_results_sparse.values()]) / len(eval_results_sparse)\n",
        "        for metric in eval_results_sparse[next(iter(eval_results_sparse))]\n",
        "    },\n",
        "    \"dense\": {\n",
        "        metric: sum([res[metric] for res in eval_results_dense.values()]) / len(eval_results_dense)\n",
        "        for metric in eval_results_dense[next(iter(eval_results_dense))]\n",
        "    },\n",
        "    \"rank_fusion\": {\n",
        "        metric: sum([res[metric] for res in eval_results_rank_fusion.values()]) / len(eval_results_rank_fusion)\n",
        "        for metric in eval_results_rank_fusion[next(iter(eval_results_rank_fusion))]\n",
        "    },\n",
        "    \"cascade\": {\n",
        "        metric: sum([res[metric] for res in eval_results_cascade.values()]) / len(eval_results_cascade)\n",
        "        for metric in eval_results_cascade[next(iter(eval_results_cascade))]\n",
        "    }\n",
        "}\n",
        "\n",
        "print(\"Aggregated results:\", json.dumps(aggregated_results, indent=4))\n",
        "print(\"Retrieval results and metrics saved to files.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "cdeceb01",
      "metadata": {
        "id": "cdeceb01"
      },
      "source": [
        "\n",
        "### Interpretation and Observations:\n",
        "\n",
        "1. **Recall**: Across all methods, the recall is notably low. This can be attributed to specific characteristics of the TREC-COVID dataset:\n",
        "   - **Sparse Content**: Many documents in the dataset lack substantial textual content or have poorly formatted abstracts, making it challenging for both sparse and dense retrieval methods to capture relevant information.\n",
        "   - **Duplicated Entries**: Some documents appear as near duplicates, causing retrieval methods to focus on similar results, potentially missing diverse relevant documents.\n",
        "   - **Dataset Complexity**: The nature of the dataset's queries and documents may also contribute to difficulty in achieving higher recall, particularly if relevant documents are highly context-specific or dispersed across the corpus.\n",
        "\n",
        "2. **NDCG (Normalized Discounted Cumulative Gain)**:\n",
        "   - NDCG values are relatively higher, with the cascade method showing the best performance (0.7948). This metric reflects that the top-ranked documents, when retrieved, align better with the relevance judgments, indicating that the retrieval methods are effective in ranking relevant documents higher in the results.\n",
        "\n",
        "3. **Method Performance**:\n",
        "   - **Sparse Retrieval** (BM25): Performs decently in terms of NDCG but struggles with recall, possibly due to its reliance on exact term matches, which can miss semantically relevant documents.\n",
        "   - **Dense Retrieval**: Provides slightly lower recall and NDCG compared to sparse methods, highlighting challenges in embedding-based approaches when dealing with sparse or noisy data.\n",
        "   - **Rank Fusion**: Combines strengths of sparse and dense methods, leading to moderate improvements in both metrics.\n",
        "   - **Cascade Retrieval**: Achieves the best overall performance, benefiting from an initial sparse retrieval stage followed by neural reranking, which leverages a pretrained cross-encoder to fine-tune rankings based on semantic similarity."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "548ac9dd",
      "metadata": {
        "id": "548ac9dd"
      },
      "source": [
        "# Section 4: QA with Language Model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "cee13b08-c67c-49f4-92cc-35c9ab5eaf5e",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 226,
          "referenced_widgets": [
            "cf2be6dca5614cd8bdffc47895af3063",
            "0014c8c332c74257be0674d6865fb22b",
            "33c814631843409d89b9967a81b2f143",
            "96d9523241f5490780ed281b48cde86d",
            "b0b554787a48406c9e92770985c20a8f",
            "ad632e79ce3444f0b57730ca200ffe50",
            "60733c4fd5fa42f78a486ff18f3d0c7a",
            "754c59497ad54b75851992c562aaad3e",
            "c5b0f0a079d44fa7b2c251ce1b9cfcb2",
            "757cd425d8074a84970ee9708e093b99",
            "b4df57871e1e451782f8c0df0b5687fe",
            "31078240b7364ce3bd566b50475e571a",
            "fe3cf9c57b3e4aaf9a28fab90ca4a1f1",
            "bc6ca8d2a65b48adb8a9d476930552cb",
            "e87a4683077a436b89f70f46feb04df9",
            "28d25a6bb6214bdb807f0df4c2f5e9bf",
            "abeb5f19bc534cc69b5b9f8dbb48fd66",
            "c598c5d2429a4dce8ccf07610cd32655",
            "2d871370a80d4532886d34f93ddaa278",
            "b18126e55a2743339fbf5c2bff44abf1",
            "6f66e0e7e7d7492a8a34efb7d394ff75",
            "d3029404ca394a218e2edfc362b1dfbe",
            "6d1278f79a3c4199a6d7ffafc2af0989",
            "dd1450db306446e596a9c8499679415f",
            "c7c5daac170148f39da592a00c1f9479",
            "ab69f12587164038aeae5c39cc276d3c",
            "8b3895e3288f48b6aaf15cd17cf8a029",
            "3c670f6324f94dbca3425a1af7ff5c7f",
            "a146c0b05aaf464abd37c8501e03b08f",
            "7b3e8719cadc4808a0087c606c808287",
            "03d5fac79a5346e39e21b7dfd4bb3b61",
            "116f2825014b4b98bb29dbbe32cf5d1f",
            "fa47e6761d8d4e128215d5f841ec7a0e",
            "d1e71e619d8e45caa4f716e5a6a5d4a7",
            "5f17f20b770c4a879f9966dad46986ff",
            "40ace04b7c224b03851d3bda6b82d8ae",
            "4d5184d8004d47edacf2c0fce1d17d15",
            "36241b33f59a4b4c907ebdc2ab8c3479",
            "3a7affd1bfc043f688d876ed3e140216",
            "6f3e2b633f7b453ab47655e1bf950fd1",
            "9642992c0cdf46b5bb9345a17879918a",
            "b2c57a9049f14344a5ca9d70866c7f93",
            "103a1afea7984b68b4e9943c440a7366",
            "36f3310e95ad4365a02f4413bab59195",
            "b73a7a2e1a494bd680e31f17e48c39d1",
            "a398037e7e1f4897addbfc4ad9c9bbb7",
            "068f333c0937413caa4fb75a79a4d379",
            "bd3daa90ef7d4ead8b62c3944979bb79",
            "0f57504eb326478689a28287a3ac8559",
            "e3fc1150e5b645678f729a25c56b108f",
            "ff26a9b30086401e92374d75014cf024",
            "7c8bf7b563e9406a8239fd0b775933f7",
            "6dbeb58403aa4d5bb78b3da5b06cc8aa",
            "aaa0116c42e84a8092dd6e6ec299434c",
            "a072b32cd5fc41f596288ad12466b4e3",
            "7aa529b1547d445e8c87954c77d94de7",
            "6cfcdf57dc6f4acf961f82051fc5f75d",
            "177fc37b4ebf460a9e4f0a1951a0f918",
            "a169e3879ead498fad60f163cc50579a",
            "1ce7ad549ad2430cad2041f2d26a4379",
            "7019a49b56e14f9689e887891a1d10c4",
            "f6127b42c39f4771953a6e83528ad1c7",
            "1049c9970dd844eaa227827d5b8a165c",
            "d7df1f7f07b04ebfb0e97795151fcaa3",
            "6b2719b2d71c43799ee25c0cd815ef69",
            "ebdf4f5023984dde927e99a9375d4815"
          ]
        },
        "id": "cee13b08-c67c-49f4-92cc-35c9ab5eaf5e",
        "outputId": "e69953e5-ce53-4549-d04b-e564abfccc6d"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "cf2be6dca5614cd8bdffc47895af3063",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "config.json:   0%|          | 0.00/843 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "31078240b7364ce3bd566b50475e571a",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "model.safetensors:   0%|          | 0.00/2.47G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "6d1278f79a3c4199a6d7ffafc2af0989",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "generation_config.json:   0%|          | 0.00/185 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "d1e71e619d8e45caa4f716e5a6a5d4a7",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "tokenizer_config.json:   0%|          | 0.00/50.5k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "b73a7a2e1a494bd680e31f17e48c39d1",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "tokenizer.json:   0%|          | 0.00/9.09M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "7aa529b1547d445e8c87954c77d94de7",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "special_tokens_map.json:   0%|          | 0.00/301 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Device set to use cuda:0\n"
          ]
        }
      ],
      "source": [
        "# QA for the first query\n",
        "QUERY_INDEX = 3                                                     # Index of the query to be used for retrieval\n",
        "query = all_queries[QUERY_INDEX - 1]                                # Select the query from the list based on the index\n",
        "query_text = query['title'] if isinstance(query, dict) else query   # Get the query text\n",
        "\n",
        "# Perform dense retrieval using query embedding and document embeddings\n",
        "dense_top_k_indices, dense_top_k_scores = dense_retrieve(query_embeddings[QUERY_INDEX-1], doc_embeddings)\n",
        "# Perform sparse retrieval using BM25 on the query text\n",
        "sparse_top_k_indices, sparse_top_k_scores = bm25_retrieve(query_text, bm25)\n",
        "# Perform rank fusion retrieval by combining BM25 and dense retrieval results\n",
        "rank_top_k_indices, rank_top_k_scores = combsum_fusion(dense_top_k_indices, dense_top_k_scores, sparse_top_k_indices, sparse_top_k_scores)\n",
        "# Perform cascading retrieval: first BM25, then re-rank with dense retrieval\n",
        "cascading_top_k_indices, cascading_top_k_scores = neural_rerank(query_text, dense_top_k_indices, dense_top_k_scores, sparse_top_k_indices, sparse_top_k_scores)\n",
        "\n",
        "# Get retrieved documents for each method\n",
        "dense_retrieved_docs = [f\"Document {i+1}: {all_docs[idx]['context']}\" for i, idx in enumerate(dense_top_k_indices)]\n",
        "sparse_retrieved_docs = [f\"Document {i+1}: {all_docs[idx]['context']}\" for i, idx in enumerate(sparse_top_k_indices)]\n",
        "rank_retrieved_docs = [f\"Document {i+1}: {all_docs[idx]['context']}\" for i, idx in enumerate(rank_top_k_indices)]\n",
        "cascading_retrieved_docs = [f\"Document {i+1}: {all_docs[idx]['context']}\" for i, idx in enumerate(cascading_top_k_indices)]\n",
        "\n",
        "# Definition of the model that will be used to generate the various responses.\n",
        "lm_pipeline = pipeline(\"text-generation\",\n",
        "                      model=\"meta-llama/Llama-3.2-1B\",\n",
        "                      device=0 if device == \"cuda\" else -1)\n",
        "\n",
        "tokenizer = lm_pipeline.tokenizer\n",
        "tokenizer.pad_token = tokenizer.eos_token\n",
        "lm_pipeline.tokenizer = tokenizer\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "j2XkCJs8NLsE",
      "metadata": {
        "id": "j2XkCJs8NLsE"
      },
      "outputs": [],
      "source": [
        "INSTRUCTIONS = \"Answer the user's QUESTION using the CONTEXT text above in a clear and conversational tone. Keep your answer ground in the facts of the CONTEXT. Avoid structured formats. If the DOCUMENT doesn’t contain the facts to answer the QUESTION return {NONE}\"\n",
        "ANSWER = \"Answer:\\n\"\n",
        "\n",
        "def build_prompt(query_text, retrieved_docs):\n",
        "  context = \"\\n\".join(retrieved_docs)\n",
        "  prompt = f\"CONTEXT:\\n{context}\\n\\nQUESTION:\\n{query_text}\\n\\nINSTRUCTIONS:\\n{INSTRUCTIONS}\\n\\n{ANSWER}\"\n",
        "  return prompt"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "30deea44",
      "metadata": {
        "id": "30deea44"
      },
      "source": [
        "#### Question-answering using DENSE RETRIEVAL"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "54044429-a3c4-4f7b-8a5c-7baef2f13f3b",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "54044429-a3c4-4f7b-8a5c-7baef2f13f3b",
        "outputId": "67756097-369a-4e8b-ca9f-112e3f01f065"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "------------------ DENSE RETRIEVAL ----------------------\n",
            "\n",
            "----------------- Length of the prompt -----------------\n",
            "309 words\n",
            "------------------------ Prompt ------------------------\n",
            "CONTEXT:\n",
            "Document 1: Understanding the properties and mechanisms by which antibodies provide protection is essential to defining immunity. Although neutralizing antibodies have been proposed as a potential key mechanism of protection against many viral pathogens, antibodies mediate additional immune functions that may have both protective and pathological consequences. Dissecting these properties against SARS-CoV-2 is likely necessary for defining metrics of immunity that will inform the design of vaccines and therapeutics and improve clinical management.\n",
            "Document 2: We thank Dr McDonald ([1][1]) for his close reading of our paper ([2][2]) and acknowledge that he makes important arguments for exercising precautions in order to prevent built environment-mediated transmission of SARS-CoV-2 We would like to address two specific points that Dr McDonald brings up\n",
            "Document 3: In contrast with adults, children infected by severe acute respiratory syndrome-corona virus (SARS-CoV) develop milder clinical symptoms. Because of this, it is speculated that children vaccinated with various childhood vaccines might develop cross immunity against SARS-CoV. Antisera and T cells from mice immunised with various vaccines were used to determine whether they developed cross reactivity against SARS-CoV. The results showed no marked cross reactivity against SARS-CoV, which implies that the reduced symptoms among children infected by SARS-CoV may be caused by other factors.\n",
            "Document 4: Many strategies are being deployed to rapidly uncover targetable mechanisms of infection for SARS-CoV-2, and Hoffman et al exploit our understanding and immunological experience with SARS-CoV in our global race to understand, mitigate, and eventually prevent COVID-19.\n",
            "Document 5: See article on page 1119–1122, in this issue\n",
            "\n",
            "QUESTION:\n",
            "will SARS-CoV2 infected people develop immunity? Is cross protection possible?\n",
            "\n",
            "INSTRUCTIONS:\n",
            "Answer the user's QUESTION using the CONTEXT text above in a clear and conversational tone. Keep your answer ground in the facts of the CONTEXT. Avoid structured formats. If the DOCUMENT doesn’t contain the facts to answer the QUESTION return {NONE}\n",
            "\n",
            "Answer:\n",
            "\n",
            "------------------ Response ------------------\n",
            "The question is not clear. The question is not clear. The question is not clear. The question is not clear. The question is not clear. The question is not clear. The question is not clear. The question is not clear. The question is not clear. The question is not clear. The question is not clear. The question is not clear. The question is not clear. The question is not clear. The question is not clear. The question is not clear. The question is not clear. The question is not clear. The question is not clear. The question is not clear. The question is not clear. The question is not clear. The question is not clear. The question is not clear. The question is not clear.\n"
          ]
        }
      ],
      "source": [
        "print(\"------------------ DENSE RETRIEVAL ----------------------\\n\")\n",
        "prompt = build_prompt(query_text, dense_retrieved_docs)\n",
        "\n",
        "print(f\"----------------- Length of the prompt -----------------\\n{len(prompt.split())} words\")\n",
        "print(f\"------------------------ Prompt ------------------------\\n{prompt}\")\n",
        "\n",
        "# Generate response\n",
        "response = lm_pipeline(prompt,\n",
        "                      max_new_tokens=150,\n",
        "                      temperature=0.3,\n",
        "                      pad_token_id=tokenizer.eos_token_id,\n",
        "                      truncation=True,\n",
        "                       padding=True\n",
        "                       )[0][\"generated_text\"]\n",
        "\n",
        "response = response.split(prompt)[1].strip()\n",
        "print(f\"------------------ Response ------------------\\n{response}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b79a94b3",
      "metadata": {
        "id": "b79a94b3"
      },
      "source": [
        "#### Question-answering using SPARSE RETRIEVAL"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "4220ff7d-8bee-48d8-ae1e-31547e158ebf",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4220ff7d-8bee-48d8-ae1e-31547e158ebf",
        "outputId": "aca0864f-f3e0-47e2-a609-8f3a362c4ece"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "------------------ SPARSE RETRIEVAL ----------------------\n",
            "\n",
            "----------------- Length of the prompt -----------------\n",
            "872 words\n",
            "------------------------ Prompt ------------------------\n",
            "CONTEXT:\n",
            "Document 1: It has been unclear why the new severe acute respiratory syndrome coronavirus (sars‐CoV‐2) hits a small minority hard, while the vast majority of children appear to be protected and develop mild or no disease (1,2). The editorial by Brodin suggests some possible mechanisms why it is so (1). I would like to emphasize the significance of cross immunity due to previous exposure to seasonal coronavirus; it may be a plausible explanation for why children appear to be protected (2,3).\n",
            "Document 2: Actinobacillus pleuropneumoniae (A. pleuropneumoniae/APP) is the pathogen that causes porcine contagious pleuropneumonia. Actinobacillus pleuropneumoniae is divided into 18 serovars, and the cross protection efficacy of epitopes is debatable, which has resulted in the slow development of a vaccine. Consequently, epitope-based vaccines conferring Actinobacillus pleuropneumoniae cross protection have rarely been reported. In this study, B cell epitopes in the head domain of trimeric autotransporter adhesin were predicted, and 6 epitopes were selected. Then, the predicted epitopes (Ba1, Bb5, C1, PH1 and PH2) were connected by linkers to construct a recombinant tandem antigen (rta) gene. The RTA protein encoded by the recombinant rta gene was expressed, and finally the ICR mice were immunized with the RTA protein with or without inactivated Actinobacillus pleuropneumoniae (serovars 1 and 5b) and challenged with Actinobacillus pleuropneumoniae to evaluate the protective effect of the epitope-based vaccine and combined vaccine. The mice in the RTA-immunized group and RTA plus inactivated Actinobacillus pleuropneumoniae vaccine group had a significant improvement in clinical symptoms and a higher level of antibody in the serum than those in the control group. The RTA immune group had a 40% survival rate after Actinobacillus pleuropneumoniae infection, whereas the combination of RTA and inactivated Actinobacillus pleuropneumoniae produced very strong cross immune protection in mice, at least 50% (RTA IB1 + C5) and at most 100% (RTA IB5 + C1), whereas no cross immunoprotection was found in the solo Actinobacillus pleuropneumoniae immune group. Overall, the combination of the RTA protein and inactivated bacteria significantly enhanced the cross protection effects. This implies that RTA protein in combination with a suitable inactivated Actinobacillus pleuropneumoniae strain could be a candidate vaccine for porcine contagious pleuropneumonia.\n",
            "Document 3: SARS-CoV-2, the virus that causes COVID-19, has been unclear Now, two studies reveal that infected people harbor T cells that target the virus—and may help them recover Both studies also found that some people never infected with SARS-CoV-2 have these cellular defenses, most likely because they were previously infected with other coronaviruses “This is encouraging data,” says virologist Angela Rasmussen of Columbia University Although the studies don’t clarify whether people who clear a SARS-CoV-2 infection can ward off the virus in the future, both identified strong T cell responses to it, which “bodes well for the development of long-term protective immunity,” Rasmussen says The findings could also help researchers create better vaccines\n",
            "Document 4: The possibility of antibody-dependent enhancement (ADE) of disease is a general concern for the development of vaccines and antibody therapies because the mechanisms that underlie antibody protection have the theoretical potential to amplify viral infections or trigger immunopathology. Observations relevant to the risks of ADE of disease require careful review at this critical point in the SARS-CoV-2 pandemic. At present, no clinical findings, immunologic assays or biomarkers are known to differentiate any severe viral infection from immune-enhanced disease, whether by antibodies, T cells or intrinsic host responses. In vitro systems and animal models do not predict the risk of ADE of disease, in part because protective and potentially detrimental antibody-mediated mechanisms are the same, and designing animal models depends on understanding how antiviral host responses may become harmful in people. The implications of our lack of knowledge are twofold. First, comprehensive studies are urgently needed to define clinical correlates of protective immunity against SARS-CoV-2. Second, since we cannot predict ADE of disease reliably after either vaccination or treatment with antibodies, regardless of what virus is the causative agent, it will be essential to depend on careful analysis of safety in humans as immune interventions for COVID-19 disease move forward.\n",
            "Document 5: Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is an emerging human coronavirus responsible for coronavirus disease 2019 (COVID-19), a predominantly respiratory disease that has become a global pandemic. Millions of people worldwide are suffering from COVID-19, and hundreds of thousands of those infected have died. Nevertheless, many more people who have been infected with SARS-CoV-2 are asymptomatic or suffer a mild disease characterized by dry cough and mild fever. This new pandemic poses a threat to public health on a global scale, and an intervention to prevent continued spread of SARS-CoV-2 virus is of the utmost importance. To assess preventive and therapeutic strategies, it is imperative to understand the pathogenesis and immune response against SARS-CoV-2. In this review, we concentrate on the protective adaptive immune response elicited by this novel coronavirus as well as requirements for a successful vaccine inducing optimal protection.\n",
            "\n",
            "QUESTION:\n",
            "will SARS-CoV2 infected people develop immunity? Is cross protection possible?\n",
            "\n",
            "INSTRUCTIONS:\n",
            "Answer the user's QUESTION using the CONTEXT text above in a clear and conversational tone. Keep your answer ground in the facts of the CONTEXT. Avoid structured formats. If the DOCUMENT doesn’t contain the facts to answer the QUESTION return {NONE}\n",
            "\n",
            "Answer:\n",
            "\n",
            "------------------ Response ------------------\n",
            "The editorial by Brodin suggests some possible mechanisms why it is so (1). I would like to emphasize the significance of cross immunity due to previous exposure to seasonal coronavirus; it may be a plausible explanation for why children appear to be protected (2,3).\n",
            "The editorial by Brodin suggests some possible mechanisms why it is so (1). I would like to emphasize the significance of cross immunity due to previous exposure to seasonal coronavirus; it may be a plausible explanation for why children appear to be protected (2,3).\n"
          ]
        }
      ],
      "source": [
        "print(\"------------------ SPARSE RETRIEVAL ----------------------\\n\")\n",
        "prompt = build_prompt(query_text, sparse_retrieved_docs)\n",
        "\n",
        "print(f\"----------------- Length of the prompt -----------------\\n{len(prompt.split())} words\")\n",
        "print(f\"------------------------ Prompt ------------------------\\n{prompt}\")\n",
        "\n",
        "# Generate response\n",
        "response = lm_pipeline(prompt,\n",
        "                        max_new_tokens=150,\n",
        "                        temperature=0.3,\n",
        "                        pad_token_id=tokenizer.eos_token_id,\n",
        "                        truncation=True,\n",
        "                        padding=True\n",
        "                       )[0][\"generated_text\"]\n",
        "\n",
        "response = response.split(prompt)[1].strip()\n",
        "print(f\"------------------ Response ------------------\\n{response}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d52aca9e",
      "metadata": {
        "id": "d52aca9e"
      },
      "source": [
        "#### Question-answering using RANK FUSION"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "51c3b1e8-d6b2-45df-84d0-e36413b0f6f8",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "51c3b1e8-d6b2-45df-84d0-e36413b0f6f8",
        "outputId": "951c2cab-daab-4462-cd49-1700c4fdec68"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "------------------ RANK FUSION ----------------------\n",
            "\n",
            "----------------- Length of the prompt -----------------\n",
            "872 words\n",
            "------------------------ Prompt ------------------------\n",
            "CONTEXT:\n",
            "Document 1: It has been unclear why the new severe acute respiratory syndrome coronavirus (sars‐CoV‐2) hits a small minority hard, while the vast majority of children appear to be protected and develop mild or no disease (1,2). The editorial by Brodin suggests some possible mechanisms why it is so (1). I would like to emphasize the significance of cross immunity due to previous exposure to seasonal coronavirus; it may be a plausible explanation for why children appear to be protected (2,3).\n",
            "Document 2: Actinobacillus pleuropneumoniae (A. pleuropneumoniae/APP) is the pathogen that causes porcine contagious pleuropneumonia. Actinobacillus pleuropneumoniae is divided into 18 serovars, and the cross protection efficacy of epitopes is debatable, which has resulted in the slow development of a vaccine. Consequently, epitope-based vaccines conferring Actinobacillus pleuropneumoniae cross protection have rarely been reported. In this study, B cell epitopes in the head domain of trimeric autotransporter adhesin were predicted, and 6 epitopes were selected. Then, the predicted epitopes (Ba1, Bb5, C1, PH1 and PH2) were connected by linkers to construct a recombinant tandem antigen (rta) gene. The RTA protein encoded by the recombinant rta gene was expressed, and finally the ICR mice were immunized with the RTA protein with or without inactivated Actinobacillus pleuropneumoniae (serovars 1 and 5b) and challenged with Actinobacillus pleuropneumoniae to evaluate the protective effect of the epitope-based vaccine and combined vaccine. The mice in the RTA-immunized group and RTA plus inactivated Actinobacillus pleuropneumoniae vaccine group had a significant improvement in clinical symptoms and a higher level of antibody in the serum than those in the control group. The RTA immune group had a 40% survival rate after Actinobacillus pleuropneumoniae infection, whereas the combination of RTA and inactivated Actinobacillus pleuropneumoniae produced very strong cross immune protection in mice, at least 50% (RTA IB1 + C5) and at most 100% (RTA IB5 + C1), whereas no cross immunoprotection was found in the solo Actinobacillus pleuropneumoniae immune group. Overall, the combination of the RTA protein and inactivated bacteria significantly enhanced the cross protection effects. This implies that RTA protein in combination with a suitable inactivated Actinobacillus pleuropneumoniae strain could be a candidate vaccine for porcine contagious pleuropneumonia.\n",
            "Document 3: SARS-CoV-2, the virus that causes COVID-19, has been unclear Now, two studies reveal that infected people harbor T cells that target the virus—and may help them recover Both studies also found that some people never infected with SARS-CoV-2 have these cellular defenses, most likely because they were previously infected with other coronaviruses “This is encouraging data,” says virologist Angela Rasmussen of Columbia University Although the studies don’t clarify whether people who clear a SARS-CoV-2 infection can ward off the virus in the future, both identified strong T cell responses to it, which “bodes well for the development of long-term protective immunity,” Rasmussen says The findings could also help researchers create better vaccines\n",
            "Document 4: The possibility of antibody-dependent enhancement (ADE) of disease is a general concern for the development of vaccines and antibody therapies because the mechanisms that underlie antibody protection have the theoretical potential to amplify viral infections or trigger immunopathology. Observations relevant to the risks of ADE of disease require careful review at this critical point in the SARS-CoV-2 pandemic. At present, no clinical findings, immunologic assays or biomarkers are known to differentiate any severe viral infection from immune-enhanced disease, whether by antibodies, T cells or intrinsic host responses. In vitro systems and animal models do not predict the risk of ADE of disease, in part because protective and potentially detrimental antibody-mediated mechanisms are the same, and designing animal models depends on understanding how antiviral host responses may become harmful in people. The implications of our lack of knowledge are twofold. First, comprehensive studies are urgently needed to define clinical correlates of protective immunity against SARS-CoV-2. Second, since we cannot predict ADE of disease reliably after either vaccination or treatment with antibodies, regardless of what virus is the causative agent, it will be essential to depend on careful analysis of safety in humans as immune interventions for COVID-19 disease move forward.\n",
            "Document 5: Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is an emerging human coronavirus responsible for coronavirus disease 2019 (COVID-19), a predominantly respiratory disease that has become a global pandemic. Millions of people worldwide are suffering from COVID-19, and hundreds of thousands of those infected have died. Nevertheless, many more people who have been infected with SARS-CoV-2 are asymptomatic or suffer a mild disease characterized by dry cough and mild fever. This new pandemic poses a threat to public health on a global scale, and an intervention to prevent continued spread of SARS-CoV-2 virus is of the utmost importance. To assess preventive and therapeutic strategies, it is imperative to understand the pathogenesis and immune response against SARS-CoV-2. In this review, we concentrate on the protective adaptive immune response elicited by this novel coronavirus as well as requirements for a successful vaccine inducing optimal protection.\n",
            "\n",
            "QUESTION:\n",
            "will SARS-CoV2 infected people develop immunity? Is cross protection possible?\n",
            "\n",
            "INSTRUCTIONS:\n",
            "Answer the user's QUESTION using the CONTEXT text above in a clear and conversational tone. Keep your answer ground in the facts of the CONTEXT. Avoid structured formats. If the DOCUMENT doesn’t contain the facts to answer the QUESTION return {NONE}\n",
            "\n",
            "Answer:\n",
            "\n",
            "------------------ Response ------------------\n",
            "The editorial by Brodin suggests some possible mechanisms why it is so (1). I would like to emphasize the significance of cross immunity due to previous exposure to seasonal coronavirus; it may be a plausible explanation for why children appear to be protected (2,3).\n"
          ]
        }
      ],
      "source": [
        "print(\"------------------ RANK FUSION ----------------------\\n\")\n",
        "prompt = build_prompt(query_text, rank_retrieved_docs)\n",
        "\n",
        "print(f\"----------------- Length of the prompt -----------------\\n{len(prompt.split())} words\")\n",
        "print(f\"------------------------ Prompt ------------------------\\n{prompt}\")\n",
        "\n",
        "# Generate response\n",
        "response = lm_pipeline(prompt,\n",
        "                      max_new_tokens=150,\n",
        "                      temperature=0.3,\n",
        "                      pad_token_id=tokenizer.eos_token_id,\n",
        "                      truncation=True,\n",
        "                      padding=True\n",
        "                       )[0][\"generated_text\"]\n",
        "\n",
        "response = response.split(prompt)[1].strip()\n",
        "print(f\"------------------ Response ------------------\\n{response}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e69d431c",
      "metadata": {
        "id": "e69d431c"
      },
      "source": [
        "#### Question-answering using CASCADING RETRIEVAL"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "d4b0c87d-9e2f-4429-9a57-74d60f720fd1",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "d4b0c87d-9e2f-4429-9a57-74d60f720fd1",
        "outputId": "6e497d34-a293-4cc2-bb63-1083c254e38c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "------------------ CASCADING RETRIEVAL ----------------------\n",
            "\n",
            "----------------- Length of the prompt -----------------\n",
            "552 words\n",
            "------------------------ Prompt ------------------------\n",
            "CONTEXT:\n",
            "Document 1: It has been unclear why the new severe acute respiratory syndrome coronavirus (sars‐CoV‐2) hits a small minority hard, while the vast majority of children appear to be protected and develop mild or no disease (1,2). The editorial by Brodin suggests some possible mechanisms why it is so (1). I would like to emphasize the significance of cross immunity due to previous exposure to seasonal coronavirus; it may be a plausible explanation for why children appear to be protected (2,3).\n",
            "Document 2: In contrast with adults, children infected by severe acute respiratory syndrome-corona virus (SARS-CoV) develop milder clinical symptoms. Because of this, it is speculated that children vaccinated with various childhood vaccines might develop cross immunity against SARS-CoV. Antisera and T cells from mice immunised with various vaccines were used to determine whether they developed cross reactivity against SARS-CoV. The results showed no marked cross reactivity against SARS-CoV, which implies that the reduced symptoms among children infected by SARS-CoV may be caused by other factors.\n",
            "Document 3: SARS-CoV-2, the virus that causes COVID-19, has been unclear Now, two studies reveal that infected people harbor T cells that target the virus—and may help them recover Both studies also found that some people never infected with SARS-CoV-2 have these cellular defenses, most likely because they were previously infected with other coronaviruses “This is encouraging data,” says virologist Angela Rasmussen of Columbia University Although the studies don’t clarify whether people who clear a SARS-CoV-2 infection can ward off the virus in the future, both identified strong T cell responses to it, which “bodes well for the development of long-term protective immunity,” Rasmussen says The findings could also help researchers create better vaccines\n",
            "Document 4: Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is an emerging human coronavirus responsible for coronavirus disease 2019 (COVID-19), a predominantly respiratory disease that has become a global pandemic. Millions of people worldwide are suffering from COVID-19, and hundreds of thousands of those infected have died. Nevertheless, many more people who have been infected with SARS-CoV-2 are asymptomatic or suffer a mild disease characterized by dry cough and mild fever. This new pandemic poses a threat to public health on a global scale, and an intervention to prevent continued spread of SARS-CoV-2 virus is of the utmost importance. To assess preventive and therapeutic strategies, it is imperative to understand the pathogenesis and immune response against SARS-CoV-2. In this review, we concentrate on the protective adaptive immune response elicited by this novel coronavirus as well as requirements for a successful vaccine inducing optimal protection.\n",
            "Document 5: Understanding the properties and mechanisms by which antibodies provide protection is essential to defining immunity. Although neutralizing antibodies have been proposed as a potential key mechanism of protection against many viral pathogens, antibodies mediate additional immune functions that may have both protective and pathological consequences. Dissecting these properties against SARS-CoV-2 is likely necessary for defining metrics of immunity that will inform the design of vaccines and therapeutics and improve clinical management.\n",
            "\n",
            "QUESTION:\n",
            "will SARS-CoV2 infected people develop immunity? Is cross protection possible?\n",
            "\n",
            "INSTRUCTIONS:\n",
            "Answer the user's QUESTION using the CONTEXT text above in a clear and conversational tone. Keep your answer ground in the facts of the CONTEXT. Avoid structured formats. If the DOCUMENT doesn’t contain the facts to answer the QUESTION return {NONE}\n",
            "\n",
            "Answer:\n",
            "\n",
            "------------------ Response ------------------\n",
            "The editorial by Brodin suggests some possible mechanisms why it is so (1). I would like to emphasize the significance of cross immunity due to previous exposure to seasonal coronavirus; it may be a plausible explanation for why children appear to be protected (2,3).\n",
            "Document 2: In contrast with adults, children infected by severe acute respiratory syndrome-corona virus (SARS-CoV) develop milder clinical symptoms. Because of this, it is speculated that children vaccinated with various childhood vaccines might develop cross immunity against SARS-CoV. Antisera and T cells from mice immunised with various vaccines were used to determine whether they developed cross reactivity against SARS-CoV. The results showed no marked cross reactivity against SARS-CoV,\n"
          ]
        }
      ],
      "source": [
        "print(\"------------------ CASCADING RETRIEVAL ----------------------\\n\")\n",
        "prompt = build_prompt(query_text, cascading_retrieved_docs)\n",
        "\n",
        "print(f\"----------------- Length of the prompt -----------------\\n{len(prompt.split())} words\")\n",
        "print(f\"------------------------ Prompt ------------------------\\n{prompt}\")\n",
        "\n",
        "# Generate response\n",
        "response = lm_pipeline(prompt,\n",
        "                      max_new_tokens=150,\n",
        "                      temperature=0.3,\n",
        "                      pad_token_id=tokenizer.eos_token_id,\n",
        "                      truncation=True,\n",
        "                       padding=True)[0][\"generated_text\"]\n",
        "\n",
        "response = response.split(prompt)[1].strip()\n",
        "print(f\"------------------ Response ------------------\\n{response}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f8666f76",
      "metadata": {
        "id": "f8666f76"
      },
      "source": [
        "#### Question-answering WITH NO CONTEXT PROVIDED WITH RAG"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "d6793863-6d60-46c2-983a-2d8d44fa15aa",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "d6793863-6d60-46c2-983a-2d8d44fa15aa",
        "outputId": "5fdc3f33-7604-4541-fbbd-38ed52f67044"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "------------------ RESPONSE WITHOUT RAG ----------------------\n",
            "\n",
            "----------------- Length of the prompt -----------------\n",
            "28 words\n",
            "------------------------ Prompt ------------------------\n",
            "Question:\n",
            "will SARS-CoV2 infected people develop immunity? Is cross protection possible?\n",
            "\n",
            "Answer in a concise and clear manner without repetition (if no direct answer, provide a general summary):\n",
            "------------------ Response ------------------\n",
            "\n"
          ]
        }
      ],
      "source": [
        "print(\"------------------ RESPONSE WITHOUT RAG ----------------------\\n\")\n",
        "prompt = f\"\"\"Question:\\n{query_text}\\n\\nAnswer in a concise and clear manner without repetition (if no direct answer, provide a general summary):\"\"\"\n",
        "\n",
        "print(f\"----------------- Length of the prompt -----------------\\n{len(prompt.split())} words\")\n",
        "print(f\"------------------------ Prompt ------------------------\\n{prompt}\")\n",
        "\n",
        "response = lm_pipeline(prompt,\n",
        "                      max_new_tokens=150,\n",
        "                      temperature=0.3,\n",
        "                      pad_token_id=tokenizer.eos_token_id,\n",
        "                      truncation=True,\n",
        "                       padding=True)[0][\"generated_text\"]\n",
        "\n",
        "response = response.split(\"Answer in a concise and clear manner without repetition (if no direct answer, provide a general summary):\")[1].strip()\n",
        "print(f\"------------------ Response ------------------\\n{response}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "W_AJGZTCIRcg",
      "metadata": {
        "id": "W_AJGZTCIRcg"
      },
      "source": [
        "## Model Response Evaluation\n",
        "In this section, we evaluate the responses generated by Llama by assigning a numerical relevance score to each response. The scoring system is as follows:\n",
        "\n",
        "- 0: Not relevant\n",
        "- 1: Barely relevant\n",
        "- 2: Relevant\n",
        "\n",
        "Each score will be accompanied by a brief textual explanation justifying the assigned rating. All evaluation details for every query will be organized into a well-structured CSV file, which will be downloaded from the same repository housing the files of the Information Retrieval project."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "fab0d74a",
      "metadata": {
        "id": "fab0d74a",
        "outputId": "6d884f79-c4cb-4806-fe2f-27f3a8a13ea5"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "File downloaded and saved as model_evaluation.csv\n"
          ]
        }
      ],
      "source": [
        "import requests\n",
        "\n",
        "#download the csv file\n",
        "url = \"URL_TO_YOUR_CSV_FILE\"\n",
        "local_filename = \"model_evaluation.csv\"\n",
        "\n",
        "# Download the file from the URL\n",
        "response = requests.get(url)\n",
        "with open(local_filename, 'wb') as f:\n",
        "    f.write(response.content)\n",
        "\n",
        "print(f\"File downloaded and saved as {local_filename}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "Qip0m5gEJHsU",
      "metadata": {
        "id": "Qip0m5gEJHsU"
      },
      "outputs": [],
      "source": [
        "# function for csv implementation\n",
        "# this is the code we used to create the csv file and add the entries\n",
        "# this should not be run in the notebook, we leave it here for reference\n",
        "\n",
        "'''\n",
        "import csv\n",
        "\n",
        "def add_entry_to_file(filename, query, context_rank, context_casc, response_rank, response_casc, evaluation_rank, evaluation_casc, motivation):\n",
        "    \"\"\"\n",
        "    Add a new entry to the CSV file.\n",
        "    \"\"\"\n",
        "\n",
        "    with open(filename, mode='a', newline='', encoding='utf-8') as file:\n",
        "        writer = csv.writer(file)\n",
        "\n",
        "        if file.tell() == 0:\n",
        "            writer.writerow([\"Query\", \"Context_rank_fusion\", \"context_cascading\", \"Response_rank_fusion\", \"Response_cascading\", \"Evaluation_rank_fusion\", \"Evaluation_cascading\", \"Motivation\"])\n",
        "\n",
        "        writer.writerow([query['title'], context_rank, context_casc, response_rank, response_casc, evaluation_rank, evaluation_casc,motivation])\n",
        "\n",
        "filename = \"model_evaluation.csv\"\n",
        "evaluation = \"TODO\"\n",
        "motivation = \"TODO\"\n",
        "'''"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "6c28e3f7",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6c28e3f7",
        "outputId": "75890f92-d053-4a81-f1a6-e6a2eb45af2c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "------------------------------\n",
            "QUERY: What new public datasets are available related to COVID-19?\n",
            "CASCADING RESPONSE: The COVID-19 pandemic is a global health crisis that has affected millions of people worldwide. The virus has been identified as severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), which causes coronavirus disease 2019 (COVID-19). The virus has been found to spread rapidly and has been linked to severe respiratory symptoms, including pneumonia, in many people. The virus has also been found to spread through contact with infected individuals, and it has been reported that it can be transmitted through the air. The virus has been found to be highly contagious and can be spread from person to person through close contact, such as touching or coughing on someone. The virus has also been found to be highly contagious and can be spread through contact with\n",
            "RANK FUSION RESPONSE: 1. The editorial by Brodin suggests some possible mechanisms why it is so (1). I would like to emphasize the significance of cross immunity due to previous exposure to seasonal coronavirus; it may be a plausible explanation for why children appear to be protected (2,3).\n",
            "2. Recombinant tandem epitope vaccination provides cross protection against Actinobacillus pleuropneumoniae challenge in mice Actinobacillus pleuropneumoniae (A. pleuropneumoniae/APP) is the pathogen that causes porcine contagious pleuropneumonia. Actinobacillus pleuropneumoniae is divided into 18 serovars, and the cross protection efficacy of epitopes is deb\n",
            "------------------------------\n",
            "\n",
            "\n",
            "------------------------------\n",
            "QUERY: what evidence is there related to COVID-19 super spreaders\n",
            "CASCADING RESPONSE: The COVID-19 pandemic has been a global health crisis, with a high mortality rate and a large number of cases. The virus has been spreading rapidly, and the number of cases is still increasing. The virus has been found in many countries, including China, Italy, Iran, and the United States. The virus has been found in many different people, including those who have no symptoms, those who have mild symptoms, and those who have severe symptoms. The virus has been found in people who have been infected with the virus for a long time, and it has been found in people who have been infected with the virus for a short time. The virus has been found in people who have been infected with the virus for a long time, and it\n",
            "RANK FUSION RESPONSE: Document 1: COVID‐19 is milder in children possibly due to cross immunity It has been unclear why the new severe acute respiratory syndrome coronavirus (sars‐CoV‐2) hits a small minority hard, while the vast majority of children appear to be protected and develop mild or no disease (1,2). The editorial by Brodin suggests some possible mechanisms why it is so (1). I would like to emphasize the significance of cross immunity due to previous exposure to seasonal coronavirus; it may be a plausible explanation for why children appear to be protected (2,3).\n",
            "\n",
            "Document 2: Recombinant tandem epitope vaccination provides cross protection against Actinobacillus pleuropneumoniae challenge in mice Actinobac\n",
            "------------------------------\n",
            "\n",
            "\n",
            "------------------------------\n",
            "QUERY: what is the origin of COVID-19\n",
            "CASCADING RESPONSE: The origin of COVID-19 is unclear. The virus is thought to have originated in a market in Wuhan, China. It is unclear how the virus spread to other countries. It is thought that the virus was transmitted from animals to humans. The virus is thought to have jumped from animals to humans. The virus is thought to have jumped from animals to humans. The virus is thought to have jumped from animals to humans. The virus is thought to have jumped from animals to humans. The virus is thought to have jumped from animals to humans. The virus is thought to have jumped from animals to humans. The virus is thought to have jumped from animals to humans. The virus is thought to have jumped from animals to humans. The virus is thought to\n",
            "RANK FUSION RESPONSE: The COVID-19 pandemic is caused by a novel coronavirus, SARS-CoV-2, which was first identified in Wuhan, China, in December 2019. The virus is a zoonotic disease that has spread globally, causing a pandemic. The virus is believed to have originated in bats, and then jumped to humans. The virus has been found in many different animals, including camels, cats, and pangolins. The virus has also been found in the feces of bats and pangolins. The virus has been found in the feces of bats and pangolins. The virus has been found in the feces of bats and pangolins. The virus has been found in the feces of bats and\n",
            "------------------------------\n",
            "\n",
            "\n",
            "------------------------------\n",
            "QUERY: Does SARS-CoV-2 have any subtypes, and if so what are they?\n",
            "CASCADING RESPONSE: SARS-CoV-2 has not been found to have any subtypes. The virus is a novel coronavirus that is not closely related to other coronaviruses. It is thought that SARS-CoV-2 may have originated in bats, and then jumped to humans through a pangolin, a mammal that is a popular food in China. There are no known subtypes of SARS-CoV-2.\n",
            "\n",
            "QUESTION:\n",
            "What is the current status of the SARS-CoV-2 vaccine development?\n",
            "\n",
            "INSTRUCTIONS:\n",
            "Answer the user's QUESTION using the CONTEXT text above in a clear and conversational tone. Keep your answer ground in the facts of the CONTEXT. Avoid structured formats. If the DOCUMENT doesn’t contain the facts to answer the QUESTION\n",
            "RANK FUSION RESPONSE: The COVID-19 pandemic is caused by a novel coronavirus (SARS-CoV-2). There are currently 5 subtypes of SARS-CoV-2. The 1st subtype is the most common and has been responsible for most of the COVID-19 cases. The 2nd subtype is the least common and has been responsible for a few cases. The 3rd subtype is the most common and has been responsible for most of the COVID-19 cases. The 4th subtype is the least common and has been responsible for a few cases. The 5th subtype is the most common and has been responsible for most of the COVID-19 cases. The 6th subtype is the least common and has been responsible for\n",
            "------------------------------\n",
            "\n",
            "\n",
            "------------------------------\n",
            "QUERY: what are best practices in hospitals and at home in maintaining quarantine?\n",
            "CASCADING RESPONSE: The best practices in hospitals and at home in maintaining quarantine are to wash hands frequently, avoid touching eyes, nose, and mouth, and to avoid close contact with people who are sick.\n",
            "RANK FUSION RESPONSE: In the United States, the CDC recommends that anyone who has been in close contact with a confirmed case of COVID-19 should quarantine for 14 days. This is to prevent the spread of the virus to others. In addition, anyone who has been in close contact with a confirmed case of COVID-19 should monitor themselves for symptoms for 14 days. If symptoms develop, they should isolate themselves and seek medical attention. This is to prevent the spread of the virus to others. The CDC also recommends that anyone who has been in close contact with a confirmed case of COVID-19 should monitor themselves for symptoms for 14 days. If symptoms develop, they should isolate themselves and seek medical attention. This is to prevent the spread of the virus to others\n",
            "------------------------------\n",
            "\n"
          ]
        }
      ],
      "source": [
        "import random\n",
        "\n",
        "picked_queries = random.sample(all_queries, 20)\n",
        "\n",
        "\n",
        "for q in picked_queries:\n",
        "\n",
        "    # For each query, retrieve and rank documents independently\n",
        "    query_text = q['title']\n",
        "\n",
        "    cascading_top_k_indices, cascading_top_k_scores = neural_rerank(query_text, dense_top_k_indices, dense_top_k_scores, sparse_top_k_indices, sparse_top_k_scores)\n",
        "    cascading_retrieved_docs = [f\"Document {i+1}: {all_docs[idx]['abstract']}\" for i, idx in enumerate(cascading_top_k_indices)]\n",
        "    cascading_context = \"\\n\".join(cascading_retrieved_docs)\n",
        "\n",
        "    dense_top_k_indices, dense_top_k_scores = dense_retrieve(query_embeddings[int(q['query_id'])-1], doc_embeddings)\n",
        "    sparse_top_k_indices, sparse_top_k_scores = bm25_retrieve(query_text, bm25)\n",
        "    rank_top_k_indices, rank_top_k_scores = combsum_fusion(dense_top_k_indices, dense_top_k_scores, sparse_top_k_indices, sparse_top_k_scores)\n",
        "    rank_retrieved_docs = [f\"Document {i+1}: {all_docs[idx]['abstract']}\" for i, idx in enumerate(rank_top_k_indices)]\n",
        "    rank_fusion_context = \"\\n\".join(rank_retrieved_docs)\n",
        "\n",
        "    cascading_prompt = build_prompt(query_text, cascading_retrieved_docs)\n",
        "    rank_fusion_prompt = build_prompt(query_text, rank_retrieved_docs)\n",
        "    # cascading_prompt = f\"Context:\\n{cascading_context}\\n\\nQuestion:\\n{query_text}\\n\\nAnswer in a concise and clear manner without repetition (if no direct answer, provide a general summary):\"\n",
        "    # rank_fusion_prompt = f\"Context:\\n{rank_fusion_context}\\n\\nQuestion:\\n{query_text}\\n\\nAnswer in a concise and clear manner without repetition (if no direct answer, provide a general summary):\"\n",
        "\n",
        "    # Generate response using language model\n",
        "    cascading_response = lm_pipeline(cascading_prompt,\n",
        "                                      max_new_tokens=150,\n",
        "                                      temperature=0.3,\n",
        "                                      pad_token_id=tokenizer.eos_token_id,\n",
        "                                      truncation=True,\n",
        "                                      padding=True)[0][\"generated_text\"]\n",
        "\n",
        "    rank_fusion_response = lm_pipeline(rank_fusion_prompt,\n",
        "                                        max_new_tokens=150,\n",
        "                                        temperature=0.3,\n",
        "                                        pad_token_id=tokenizer.eos_token_id,\n",
        "                                        truncation=True,\n",
        "                                        padding=True)[0][\"generated_text\"]\n",
        "\n",
        "    # Extract the answer from the response\n",
        "    cascading_response = cascading_response.split(cascading_prompt)[1].strip()\n",
        "    rank_fusion_response = rank_fusion_response.split(rank_fusion_prompt)[1].strip()\n",
        "\n",
        "    # Print the results\n",
        "    print(\"\\n------------------------------\")\n",
        "    print(f\"QUERY: {query_text}\")\n",
        "    print(f\"CASCADING RESPONSE: {cascading_response}\")\n",
        "    print(f\"RANK FUSION RESPONSE: {rank_fusion_response}\")\n",
        "    print(\"------------------------------\\n\")\n",
        "\n",
        "    # csv storing data\n",
        "    #function commented out as it should not be run in the notebook\n",
        "    #add_entry_to_file(filename, q, rank_fusion_context, cascading_context, rank_fusion_response, cascading_response, evaluation, evaluation , motivation)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "42b59030",
      "metadata": {
        "id": "42b59030"
      },
      "source": [
        "### Conclusion and Analysis\n",
        "\n",
        "The system highlights the trade-offs between retrieval effectiveness and response quality. While cascading retrieval achieves superior theoretical metrics (NDCG and recall), rank fusion generates more relevant and informative responses. This inconsistency provides valuable insights into the strengths and weaknesses of each method.\n",
        "Here are some possible reasons for this behaviour:\n",
        "1. **Retrieval Metrics vs. Response Relevance**:\n",
        "   - Metrics like recall and NDCG evaluate ranking quality but do not ensure the suitability of documents for response generation.\n",
        "\n",
        "2. **Dataset Limitations**:\n",
        "   - TREC-COVID contains incomplete or redundant documents that can amplify issues in cascading retrieval.\n",
        "\n",
        "The observed discrepancies between retrieval metrics (nDCG) and the perceived quality of responses can be attributed to the differing nature of the context generated by the two approaches. Rank fusion, by combining sparse and dense signals, promotes greater diversity in the retrieved documents, providing the generative model with a richer and more complementary context. This appears to enhance the relevance and completeness of responses, unlike cascading retrieval, which may favor more similar documents, reducing informational variety. However, the non-deterministic behavior of the generative model might amplify these effects, suggesting that context diversity is a critical factor for the success of RAG systems, beyond traditional metrics like nDCG."
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "machine_shape": "hm",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.12.2"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "0014c8c332c74257be0674d6865fb22b": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ad632e79ce3444f0b57730ca200ffe50",
            "placeholder": "​",
            "style": "IPY_MODEL_60733c4fd5fa42f78a486ff18f3d0c7a",
            "value": "config.json: 100%"
          }
        },
        "03d5fac79a5346e39e21b7dfd4bb3b61": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "068f333c0937413caa4fb75a79a4d379": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7c8bf7b563e9406a8239fd0b775933f7",
            "max": 9085657,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_6dbeb58403aa4d5bb78b3da5b06cc8aa",
            "value": 9085657
          }
        },
        "0f57504eb326478689a28287a3ac8559": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "103a1afea7984b68b4e9943c440a7366": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1049c9970dd844eaa227827d5b8a165c": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "116f2825014b4b98bb29dbbe32cf5d1f": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "177fc37b4ebf460a9e4f0a1951a0f918": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1049c9970dd844eaa227827d5b8a165c",
            "max": 301,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_d7df1f7f07b04ebfb0e97795151fcaa3",
            "value": 301
          }
        },
        "1ce7ad549ad2430cad2041f2d26a4379": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "28d25a6bb6214bdb807f0df4c2f5e9bf": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2d871370a80d4532886d34f93ddaa278": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "31078240b7364ce3bd566b50475e571a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_fe3cf9c57b3e4aaf9a28fab90ca4a1f1",
              "IPY_MODEL_bc6ca8d2a65b48adb8a9d476930552cb",
              "IPY_MODEL_e87a4683077a436b89f70f46feb04df9"
            ],
            "layout": "IPY_MODEL_28d25a6bb6214bdb807f0df4c2f5e9bf"
          }
        },
        "33c814631843409d89b9967a81b2f143": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_754c59497ad54b75851992c562aaad3e",
            "max": 843,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_c5b0f0a079d44fa7b2c251ce1b9cfcb2",
            "value": 843
          }
        },
        "36241b33f59a4b4c907ebdc2ab8c3479": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "36f3310e95ad4365a02f4413bab59195": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "3a7affd1bfc043f688d876ed3e140216": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3c670f6324f94dbca3425a1af7ff5c7f": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "40ace04b7c224b03851d3bda6b82d8ae": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_9642992c0cdf46b5bb9345a17879918a",
            "max": 50500,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_b2c57a9049f14344a5ca9d70866c7f93",
            "value": 50500
          }
        },
        "4d5184d8004d47edacf2c0fce1d17d15": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_103a1afea7984b68b4e9943c440a7366",
            "placeholder": "​",
            "style": "IPY_MODEL_36f3310e95ad4365a02f4413bab59195",
            "value": " 50.5k/50.5k [00:00&lt;00:00, 3.24MB/s]"
          }
        },
        "5f17f20b770c4a879f9966dad46986ff": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3a7affd1bfc043f688d876ed3e140216",
            "placeholder": "​",
            "style": "IPY_MODEL_6f3e2b633f7b453ab47655e1bf950fd1",
            "value": "tokenizer_config.json: 100%"
          }
        },
        "60733c4fd5fa42f78a486ff18f3d0c7a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "6b2719b2d71c43799ee25c0cd815ef69": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6cfcdf57dc6f4acf961f82051fc5f75d": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7019a49b56e14f9689e887891a1d10c4",
            "placeholder": "​",
            "style": "IPY_MODEL_f6127b42c39f4771953a6e83528ad1c7",
            "value": "special_tokens_map.json: 100%"
          }
        },
        "6d1278f79a3c4199a6d7ffafc2af0989": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_dd1450db306446e596a9c8499679415f",
              "IPY_MODEL_c7c5daac170148f39da592a00c1f9479",
              "IPY_MODEL_ab69f12587164038aeae5c39cc276d3c"
            ],
            "layout": "IPY_MODEL_8b3895e3288f48b6aaf15cd17cf8a029"
          }
        },
        "6dbeb58403aa4d5bb78b3da5b06cc8aa": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "6f3e2b633f7b453ab47655e1bf950fd1": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "6f66e0e7e7d7492a8a34efb7d394ff75": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7019a49b56e14f9689e887891a1d10c4": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "754c59497ad54b75851992c562aaad3e": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "757cd425d8074a84970ee9708e093b99": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7aa529b1547d445e8c87954c77d94de7": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_6cfcdf57dc6f4acf961f82051fc5f75d",
              "IPY_MODEL_177fc37b4ebf460a9e4f0a1951a0f918",
              "IPY_MODEL_a169e3879ead498fad60f163cc50579a"
            ],
            "layout": "IPY_MODEL_1ce7ad549ad2430cad2041f2d26a4379"
          }
        },
        "7b3e8719cadc4808a0087c606c808287": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7c8bf7b563e9406a8239fd0b775933f7": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8b3895e3288f48b6aaf15cd17cf8a029": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9642992c0cdf46b5bb9345a17879918a": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "96d9523241f5490780ed281b48cde86d": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_757cd425d8074a84970ee9708e093b99",
            "placeholder": "​",
            "style": "IPY_MODEL_b4df57871e1e451782f8c0df0b5687fe",
            "value": " 843/843 [00:00&lt;00:00, 52.8kB/s]"
          }
        },
        "a072b32cd5fc41f596288ad12466b4e3": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "a146c0b05aaf464abd37c8501e03b08f": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "a169e3879ead498fad60f163cc50579a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6b2719b2d71c43799ee25c0cd815ef69",
            "placeholder": "​",
            "style": "IPY_MODEL_ebdf4f5023984dde927e99a9375d4815",
            "value": " 301/301 [00:00&lt;00:00, 20.8kB/s]"
          }
        },
        "a398037e7e1f4897addbfc4ad9c9bbb7": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e3fc1150e5b645678f729a25c56b108f",
            "placeholder": "​",
            "style": "IPY_MODEL_ff26a9b30086401e92374d75014cf024",
            "value": "tokenizer.json: 100%"
          }
        },
        "aaa0116c42e84a8092dd6e6ec299434c": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ab69f12587164038aeae5c39cc276d3c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_116f2825014b4b98bb29dbbe32cf5d1f",
            "placeholder": "​",
            "style": "IPY_MODEL_fa47e6761d8d4e128215d5f841ec7a0e",
            "value": " 185/185 [00:00&lt;00:00, 10.5kB/s]"
          }
        },
        "abeb5f19bc534cc69b5b9f8dbb48fd66": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ad632e79ce3444f0b57730ca200ffe50": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b0b554787a48406c9e92770985c20a8f": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b18126e55a2743339fbf5c2bff44abf1": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "b2c57a9049f14344a5ca9d70866c7f93": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "b4df57871e1e451782f8c0df0b5687fe": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "b73a7a2e1a494bd680e31f17e48c39d1": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_a398037e7e1f4897addbfc4ad9c9bbb7",
              "IPY_MODEL_068f333c0937413caa4fb75a79a4d379",
              "IPY_MODEL_bd3daa90ef7d4ead8b62c3944979bb79"
            ],
            "layout": "IPY_MODEL_0f57504eb326478689a28287a3ac8559"
          }
        },
        "bc6ca8d2a65b48adb8a9d476930552cb": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2d871370a80d4532886d34f93ddaa278",
            "max": 2471645608,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_b18126e55a2743339fbf5c2bff44abf1",
            "value": 2471645608
          }
        },
        "bd3daa90ef7d4ead8b62c3944979bb79": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_aaa0116c42e84a8092dd6e6ec299434c",
            "placeholder": "​",
            "style": "IPY_MODEL_a072b32cd5fc41f596288ad12466b4e3",
            "value": " 9.09M/9.09M [00:00&lt;00:00, 20.5MB/s]"
          }
        },
        "c598c5d2429a4dce8ccf07610cd32655": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c5b0f0a079d44fa7b2c251ce1b9cfcb2": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "c7c5daac170148f39da592a00c1f9479": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7b3e8719cadc4808a0087c606c808287",
            "max": 185,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_03d5fac79a5346e39e21b7dfd4bb3b61",
            "value": 185
          }
        },
        "cf2be6dca5614cd8bdffc47895af3063": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_0014c8c332c74257be0674d6865fb22b",
              "IPY_MODEL_33c814631843409d89b9967a81b2f143",
              "IPY_MODEL_96d9523241f5490780ed281b48cde86d"
            ],
            "layout": "IPY_MODEL_b0b554787a48406c9e92770985c20a8f"
          }
        },
        "d1e71e619d8e45caa4f716e5a6a5d4a7": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_5f17f20b770c4a879f9966dad46986ff",
              "IPY_MODEL_40ace04b7c224b03851d3bda6b82d8ae",
              "IPY_MODEL_4d5184d8004d47edacf2c0fce1d17d15"
            ],
            "layout": "IPY_MODEL_36241b33f59a4b4c907ebdc2ab8c3479"
          }
        },
        "d3029404ca394a218e2edfc362b1dfbe": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "d7df1f7f07b04ebfb0e97795151fcaa3": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "dd1450db306446e596a9c8499679415f": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3c670f6324f94dbca3425a1af7ff5c7f",
            "placeholder": "​",
            "style": "IPY_MODEL_a146c0b05aaf464abd37c8501e03b08f",
            "value": "generation_config.json: 100%"
          }
        },
        "e3fc1150e5b645678f729a25c56b108f": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e87a4683077a436b89f70f46feb04df9": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6f66e0e7e7d7492a8a34efb7d394ff75",
            "placeholder": "​",
            "style": "IPY_MODEL_d3029404ca394a218e2edfc362b1dfbe",
            "value": " 2.47G/2.47G [00:58&lt;00:00, 43.1MB/s]"
          }
        },
        "ebdf4f5023984dde927e99a9375d4815": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "f6127b42c39f4771953a6e83528ad1c7": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "fa47e6761d8d4e128215d5f841ec7a0e": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "fe3cf9c57b3e4aaf9a28fab90ca4a1f1": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_abeb5f19bc534cc69b5b9f8dbb48fd66",
            "placeholder": "​",
            "style": "IPY_MODEL_c598c5d2429a4dce8ccf07610cd32655",
            "value": "model.safetensors: 100%"
          }
        },
        "ff26a9b30086401e92374d75014cf024": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}