Evaluate-LLM-in-Colab.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9HgjZsT2wHYV"
      },
      "source": [
        "<h1 align=\"center\">🚀 RocketEval 🚀</h1>\n",
        "<h3 align=\"center\">Efficiently Evaluate LLMs using Google Colab</h3>\n",
        "\n",
        "This notebook will run a full evaluation process of [RocketEval](https://github.com/Joinn99/RocketEval-ICLR) using a single Tesla T4 GPU in Google Colab.\n",
        "\n",
        "RocketEval is an efficient automated evaluation framework for Large Language Models (LLMs) that uses a checklist-based grading approach. This notebook demonstrates how to:\n",
        "\n",
        "1. Set up the evaluation environment on Google Colab\n",
        "2. Run evaluations using a lightweight model (*Qwen2.5-0.5B-Instruct*) as the judge\n",
        "3. Evaluate multiple LLM responses on the *MT-Bench* dataset\n",
        "4. Generate evaluation scores and rankings efficiently with limited compute resources\n",
        "\n",
        "The entire process is optimized to run on a single Tesla T4 GPU, making it accessible for users with basic GPU resources."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fNeMm1LHwSzG"
      },
      "source": [
        "------\n",
        "\n",
        "First, clone our github repo. This repository contains the implementation of RocketEval, an efficient automated evaluation framework for Large Language Models (LLMs) that uses a checklist-based grading approach. The repository includes:\n",
        "\n",
        "- Evaluation scripts and utilities\n",
        "- Example benchmark datasets (MT-Bench, AlpacaEval, etc.)\n",
        "- Model configurations for both API and local deployment\n",
        "- Documentation and examples"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mFbdcafI-mfV",
        "outputId": "5ef4ff29-230c-43ee-9877-1262cc8154b8"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'RocketEval-ICLR'...\n",
            "remote: Enumerating objects: 43, done.\u001b[K\n",
            "remote: Counting objects: 100% (43/43), done.\u001b[K\n",
            "remote: Compressing objects: 100% (36/36), done.\u001b[K\n",
            "remote: Total 43 (delta 4), reused 43 (delta 4), pack-reused 0 (from 0)\u001b[K\n",
            "Receiving objects: 100% (43/43), 40.25 KiB | 6.71 MiB/s, done.\n",
            "Resolving deltas: 100% (4/4), done.\n",
            "Cloning into 'RocketEval'...\n",
            "remote: Enumerating objects: 607, done.\u001b[K\n",
            "remote: Counting objects: 100% (604/604), done.\u001b[K\n",
            "remote: Compressing objects: 100% (597/597), done.\u001b[K\n",
            "remote: Total 607 (delta 92), reused 0 (delta 0), pack-reused 3 (from 1)\u001b[K\n",
            "Receiving objects: 100% (607/607), 296.68 MiB | 5.94 MiB/s, done.\n",
            "Resolving deltas: 100% (92/92), done.\n",
            "Updating files: 100% (573/573), done.\n"
          ]
        }
      ],
      "source": [
        "# Clone the repo\n",
        "!git clone https://github.com/Joinn99/RocketEval-ICLR.git\n",
        "# Download the data\n",
        "!git clone https://huggingface.co/datasets/Joinn/RocketEval && mv RocketEval RocketEval-ICLR/data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u2OrirviwX3z"
      },
      "source": [
        "------\n",
        "\n",
        "Go to the target dir, and install the necessary dependencies.\n",
        "\n",
        "We need to:\n",
        "1. Change to the RocketEval directory\n",
        "2. Install the required packages from requirements.txt\n",
        "\n",
        "The requirements.txt file contains all the necessary Python packages including:\n",
        "- `vllm` for local model deployment\n",
        "- `openai` for API access\n",
        "- `scikit-learn` for evaluation metrics\n",
        "- Other utility packages"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lsOsvxRO_LQE",
        "outputId": "49d11f4a-2283-4904-d506-a38fcc490c7c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/RocketEval-ICLR\n",
            "Requirement already satisfied: openai in /usr/local/lib/python3.11/dist-packages (from -r requirements.txt (line 1)) (1.61.1)\n",
            "Collecting datasets (from -r requirements.txt (line 2))\n",
            "  Downloading datasets-3.3.1-py3-none-any.whl.metadata (19 kB)\n",
            "Collecting vllm>=0.7.2 (from -r requirements.txt (line 3))\n",
            "  Downloading vllm-0.7.2-cp38-abi3-manylinux1_x86_64.whl.metadata (12 kB)\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.11/dist-packages (from -r requirements.txt (line 4)) (1.6.1)\n",
            "Requirement already satisfied: rich in /usr/local/lib/python3.11/dist-packages (from -r requirements.txt (line 5)) (13.9.4)\n",
            "Requirement already satisfied: tabulate in /usr/local/lib/python3.11/dist-packages (from -r requirements.txt (line 6)) (0.9.0)\n",
            "Requirement already satisfied: anyio<5,>=3.5.0 in /usr/local/lib/python3.11/dist-packages (from openai->-r requirements.txt (line 1)) (3.7.1)\n",
            "Requirement already satisfied: distro<2,>=1.7.0 in /usr/local/lib/python3.11/dist-packages (from openai->-r requirements.txt (line 1)) (1.9.0)\n",
            "Requirement already satisfied: httpx<1,>=0.23.0 in /usr/local/lib/python3.11/dist-packages (from openai->-r requirements.txt (line 1)) (0.28.1)\n",
            "Requirement already satisfied: jiter<1,>=0.4.0 in /usr/local/lib/python3.11/dist-packages (from openai->-r requirements.txt (line 1)) (0.8.2)\n",
            "Requirement already satisfied: pydantic<3,>=1.9.0 in /usr/local/lib/python3.11/dist-packages (from openai->-r requirements.txt (line 1)) (2.10.6)\n",
            "Requirement already satisfied: sniffio in /usr/local/lib/python3.11/dist-packages (from openai->-r requirements.txt (line 1)) (1.3.1)\n",
            "Requirement already satisfied: tqdm>4 in /usr/local/lib/python3.11/dist-packages (from openai->-r requirements.txt (line 1)) (4.67.1)\n",
            "Requirement already satisfied: typing-extensions<5,>=4.11 in /usr/local/lib/python3.11/dist-packages (from openai->-r requirements.txt (line 1)) (4.12.2)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.11/dist-packages (from datasets->-r requirements.txt (line 2)) (3.17.0)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.11/dist-packages (from datasets->-r requirements.txt (line 2)) (1.26.4)\n",
            "Requirement already satisfied: pyarrow>=15.0.0 in /usr/local/lib/python3.11/dist-packages (from datasets->-r requirements.txt (line 2)) (17.0.0)\n",
            "Collecting dill<0.3.9,>=0.3.0 (from datasets->-r requirements.txt (line 2))\n",
            "  Downloading dill-0.3.8-py3-none-any.whl.metadata (10 kB)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.11/dist-packages (from datasets->-r requirements.txt (line 2)) (2.2.2)\n",
            "Requirement already satisfied: requests>=2.32.2 in /usr/local/lib/python3.11/dist-packages (from datasets->-r requirements.txt (line 2)) (2.32.3)\n",
            "Collecting xxhash (from datasets->-r requirements.txt (line 2))\n",
            "  Downloading xxhash-3.5.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)\n",
            "Collecting multiprocess<0.70.17 (from datasets->-r requirements.txt (line 2))\n",
            "  Downloading multiprocess-0.70.16-py311-none-any.whl.metadata (7.2 kB)\n",
            "Requirement already satisfied: fsspec<=2024.12.0,>=2023.1.0 in /usr/local/lib/python3.11/dist-packages (from fsspec[http]<=2024.12.0,>=2023.1.0->datasets->-r requirements.txt (line 2)) (2024.10.0)\n",
            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.11/dist-packages (from datasets->-r requirements.txt (line 2)) (3.11.12)\n",
            "Requirement already satisfied: huggingface-hub>=0.24.0 in /usr/local/lib/python3.11/dist-packages (from datasets->-r requirements.txt (line 2)) (0.28.1)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.11/dist-packages (from datasets->-r requirements.txt (line 2)) (24.2)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.11/dist-packages (from datasets->-r requirements.txt (line 2)) (6.0.2)\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.11/dist-packages (from vllm>=0.7.2->-r requirements.txt (line 3)) (5.9.5)\n",
            "Requirement already satisfied: sentencepiece in /usr/local/lib/python3.11/dist-packages (from vllm>=0.7.2->-r requirements.txt (line 3)) (0.2.0)\n",
            "Collecting blake3 (from vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading blake3-1.0.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.2 kB)\n",
            "Requirement already satisfied: py-cpuinfo in /usr/local/lib/python3.11/dist-packages (from vllm>=0.7.2->-r requirements.txt (line 3)) (9.0.0)\n",
            "Requirement already satisfied: transformers>=4.48.2 in /usr/local/lib/python3.11/dist-packages (from vllm>=0.7.2->-r requirements.txt (line 3)) (4.48.3)\n",
            "Requirement already satisfied: tokenizers>=0.19.1 in /usr/local/lib/python3.11/dist-packages (from vllm>=0.7.2->-r requirements.txt (line 3)) (0.21.0)\n",
            "Requirement already satisfied: protobuf in /usr/local/lib/python3.11/dist-packages (from vllm>=0.7.2->-r requirements.txt (line 3)) (4.25.6)\n",
            "Collecting fastapi!=0.113.*,!=0.114.0,>=0.107.0 (from vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading fastapi-0.115.8-py3-none-any.whl.metadata (27 kB)\n",
            "Collecting uvicorn[standard] (from vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading uvicorn-0.34.0-py3-none-any.whl.metadata (6.5 kB)\n",
            "Requirement already satisfied: prometheus_client>=0.18.0 in /usr/local/lib/python3.11/dist-packages (from vllm>=0.7.2->-r requirements.txt (line 3)) (0.21.1)\n",
            "Requirement already satisfied: pillow in /usr/local/lib/python3.11/dist-packages (from vllm>=0.7.2->-r requirements.txt (line 3)) (11.1.0)\n",
            "Collecting prometheus-fastapi-instrumentator>=7.0.0 (from vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading prometheus_fastapi_instrumentator-7.0.2-py3-none-any.whl.metadata (13 kB)\n",
            "Collecting tiktoken>=0.6.0 (from vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading tiktoken-0.9.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB)\n",
            "Collecting lm-format-enforcer<0.11,>=0.10.9 (from vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading lm_format_enforcer-0.10.10-py3-none-any.whl.metadata (17 kB)\n",
            "Collecting outlines==0.1.11 (from vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading outlines-0.1.11-py3-none-any.whl.metadata (17 kB)\n",
            "Collecting lark==1.2.2 (from vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading lark-1.2.2-py3-none-any.whl.metadata (1.8 kB)\n",
            "Collecting xgrammar>=0.1.6 (from vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading xgrammar-0.1.13-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (2.9 kB)\n",
            "Collecting partial-json-parser (from vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading partial_json_parser-0.2.1.1.post5-py3-none-any.whl.metadata (6.1 kB)\n",
            "Requirement already satisfied: pyzmq in /usr/local/lib/python3.11/dist-packages (from vllm>=0.7.2->-r requirements.txt (line 3)) (24.0.1)\n",
            "Collecting msgspec (from vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading msgspec-0.19.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.9 kB)\n",
            "Collecting gguf==0.10.0 (from vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading gguf-0.10.0-py3-none-any.whl.metadata (3.5 kB)\n",
            "Requirement already satisfied: importlib_metadata in /usr/local/lib/python3.11/dist-packages (from vllm>=0.7.2->-r requirements.txt (line 3)) (8.6.1)\n",
            "Collecting mistral_common>=1.5.0 (from mistral_common[opencv]>=1.5.0->vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading mistral_common-1.5.3-py3-none-any.whl.metadata (4.5 kB)\n",
            "Requirement already satisfied: einops in /usr/local/lib/python3.11/dist-packages (from vllm>=0.7.2->-r requirements.txt (line 3)) (0.8.1)\n",
            "Collecting compressed-tensors==0.9.1 (from vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading compressed_tensors-0.9.1-py3-none-any.whl.metadata (6.8 kB)\n",
            "Collecting depyf==0.18.0 (from vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading depyf-0.18.0-py3-none-any.whl.metadata (7.1 kB)\n",
            "Requirement already satisfied: cloudpickle in /usr/local/lib/python3.11/dist-packages (from vllm>=0.7.2->-r requirements.txt (line 3)) (3.1.1)\n",
            "Collecting ray>=2.9 (from ray[default]>=2.9->vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading ray-2.42.1-cp311-cp311-manylinux2014_x86_64.whl.metadata (18 kB)\n",
            "Collecting nvidia-ml-py>=12.560.30 (from vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading nvidia_ml_py-12.570.86-py3-none-any.whl.metadata (8.7 kB)\n",
            "Requirement already satisfied: torch==2.5.1 in /usr/local/lib/python3.11/dist-packages (from vllm>=0.7.2->-r requirements.txt (line 3)) (2.5.1+cu124)\n",
            "Requirement already satisfied: torchaudio==2.5.1 in /usr/local/lib/python3.11/dist-packages (from vllm>=0.7.2->-r requirements.txt (line 3)) (2.5.1+cu124)\n",
            "Requirement already satisfied: torchvision==0.20.1 in /usr/local/lib/python3.11/dist-packages (from vllm>=0.7.2->-r requirements.txt (line 3)) (0.20.1+cu124)\n",
            "Collecting xformers==0.0.28.post3 (from vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading xformers-0.0.28.post3-cp311-cp311-manylinux_2_28_x86_64.whl.metadata (1.0 kB)\n",
            "Collecting astor (from depyf==0.18.0->vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading astor-0.8.1-py2.py3-none-any.whl.metadata (4.2 kB)\n",
            "Collecting interegular (from outlines==0.1.11->vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading interegular-0.3.3-py37-none-any.whl.metadata (3.0 kB)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.11/dist-packages (from outlines==0.1.11->vllm>=0.7.2->-r requirements.txt (line 3)) (3.1.5)\n",
            "Requirement already satisfied: nest_asyncio in /usr/local/lib/python3.11/dist-packages (from outlines==0.1.11->vllm>=0.7.2->-r requirements.txt (line 3)) (1.6.0)\n",
            "Collecting diskcache (from outlines==0.1.11->vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading diskcache-5.6.3-py3-none-any.whl.metadata (20 kB)\n",
            "Requirement already satisfied: referencing in /usr/local/lib/python3.11/dist-packages (from outlines==0.1.11->vllm>=0.7.2->-r requirements.txt (line 3)) (0.36.2)\n",
            "Requirement already satisfied: jsonschema in /usr/local/lib/python3.11/dist-packages (from outlines==0.1.11->vllm>=0.7.2->-r requirements.txt (line 3)) (4.23.0)\n",
            "Collecting pycountry (from outlines==0.1.11->vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading pycountry-24.6.1-py3-none-any.whl.metadata (12 kB)\n",
            "Collecting airportsdata (from outlines==0.1.11->vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading airportsdata-20241001-py3-none-any.whl.metadata (8.9 kB)\n",
            "Collecting outlines_core==0.1.26 (from outlines==0.1.11->vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading outlines_core-0.1.26-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.8 kB)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.11/dist-packages (from torch==2.5.1->vllm>=0.7.2->-r requirements.txt (line 3)) (3.4.2)\n",
            "Collecting nvidia-cuda-nvrtc-cu12==12.4.127 (from torch==2.5.1->vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading nvidia_cuda_nvrtc_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)\n",
            "Collecting nvidia-cuda-runtime-cu12==12.4.127 (from torch==2.5.1->vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading nvidia_cuda_runtime_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)\n",
            "Collecting nvidia-cuda-cupti-cu12==12.4.127 (from torch==2.5.1->vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading nvidia_cuda_cupti_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl.metadata (1.6 kB)\n",
            "Collecting nvidia-cudnn-cu12==9.1.0.70 (from torch==2.5.1->vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading nvidia_cudnn_cu12-9.1.0.70-py3-none-manylinux2014_x86_64.whl.metadata (1.6 kB)\n",
            "Collecting nvidia-cublas-cu12==12.4.5.8 (from torch==2.5.1->vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading nvidia_cublas_cu12-12.4.5.8-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)\n",
            "Collecting nvidia-cufft-cu12==11.2.1.3 (from torch==2.5.1->vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading nvidia_cufft_cu12-11.2.1.3-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)\n",
            "Collecting nvidia-curand-cu12==10.3.5.147 (from torch==2.5.1->vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading nvidia_curand_cu12-10.3.5.147-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)\n",
            "Collecting nvidia-cusolver-cu12==11.6.1.9 (from torch==2.5.1->vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading nvidia_cusolver_cu12-11.6.1.9-py3-none-manylinux2014_x86_64.whl.metadata (1.6 kB)\n",
            "Collecting nvidia-cusparse-cu12==12.3.1.170 (from torch==2.5.1->vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading nvidia_cusparse_cu12-12.3.1.170-py3-none-manylinux2014_x86_64.whl.metadata (1.6 kB)\n",
            "Requirement already satisfied: nvidia-nccl-cu12==2.21.5 in /usr/local/lib/python3.11/dist-packages (from torch==2.5.1->vllm>=0.7.2->-r requirements.txt (line 3)) (2.21.5)\n",
            "Requirement already satisfied: nvidia-nvtx-cu12==12.4.127 in /usr/local/lib/python3.11/dist-packages (from torch==2.5.1->vllm>=0.7.2->-r requirements.txt (line 3)) (12.4.127)\n",
            "Collecting nvidia-nvjitlink-cu12==12.4.127 (from torch==2.5.1->vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading nvidia_nvjitlink_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)\n",
            "Requirement already satisfied: triton==3.1.0 in /usr/local/lib/python3.11/dist-packages (from torch==2.5.1->vllm>=0.7.2->-r requirements.txt (line 3)) (3.1.0)\n",
            "Requirement already satisfied: sympy==1.13.1 in /usr/local/lib/python3.11/dist-packages (from torch==2.5.1->vllm>=0.7.2->-r requirements.txt (line 3)) (1.13.1)\n",
            "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.11/dist-packages (from sympy==1.13.1->torch==2.5.1->vllm>=0.7.2->-r requirements.txt (line 3)) (1.3.0)\n",
            "Requirement already satisfied: scipy>=1.6.0 in /usr/local/lib/python3.11/dist-packages (from scikit-learn->-r requirements.txt (line 4)) (1.13.1)\n",
            "Requirement already satisfied: joblib>=1.2.0 in /usr/local/lib/python3.11/dist-packages (from scikit-learn->-r requirements.txt (line 4)) (1.4.2)\n",
            "Requirement already satisfied: threadpoolctl>=3.1.0 in /usr/local/lib/python3.11/dist-packages (from scikit-learn->-r requirements.txt (line 4)) (3.5.0)\n",
            "Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.11/dist-packages (from rich->-r requirements.txt (line 5)) (3.0.0)\n",
            "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.11/dist-packages (from rich->-r requirements.txt (line 5)) (2.18.0)\n",
            "Requirement already satisfied: idna>=2.8 in /usr/local/lib/python3.11/dist-packages (from anyio<5,>=3.5.0->openai->-r requirements.txt (line 1)) (3.10)\n",
            "Collecting starlette<0.46.0,>=0.40.0 (from fastapi!=0.113.*,!=0.114.0,>=0.107.0->vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading starlette-0.45.3-py3-none-any.whl.metadata (6.3 kB)\n",
            "Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /usr/local/lib/python3.11/dist-packages (from aiohttp->datasets->-r requirements.txt (line 2)) (2.4.6)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.11/dist-packages (from aiohttp->datasets->-r requirements.txt (line 2)) (1.3.2)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.11/dist-packages (from aiohttp->datasets->-r requirements.txt (line 2)) (25.1.0)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.11/dist-packages (from aiohttp->datasets->-r requirements.txt (line 2)) (1.5.0)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.11/dist-packages (from aiohttp->datasets->-r requirements.txt (line 2)) (6.1.0)\n",
            "Requirement already satisfied: propcache>=0.2.0 in /usr/local/lib/python3.11/dist-packages (from aiohttp->datasets->-r requirements.txt (line 2)) (0.2.1)\n",
            "Requirement already satisfied: yarl<2.0,>=1.17.0 in /usr/local/lib/python3.11/dist-packages (from aiohttp->datasets->-r requirements.txt (line 2)) (1.18.3)\n",
            "Requirement already satisfied: certifi in /usr/local/lib/python3.11/dist-packages (from httpx<1,>=0.23.0->openai->-r requirements.txt (line 1)) (2025.1.31)\n",
            "Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.11/dist-packages (from httpx<1,>=0.23.0->openai->-r requirements.txt (line 1)) (1.0.7)\n",
            "Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.11/dist-packages (from httpcore==1.*->httpx<1,>=0.23.0->openai->-r requirements.txt (line 1)) (0.14.0)\n",
            "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.11/dist-packages (from markdown-it-py>=2.2.0->rich->-r requirements.txt (line 5)) (0.1.2)\n",
            "Requirement already satisfied: opencv-python-headless>=4.0.0 in /usr/local/lib/python3.11/dist-packages (from mistral_common[opencv]>=1.5.0->vllm>=0.7.2->-r requirements.txt (line 3)) (4.11.0.86)\n",
            "Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.11/dist-packages (from pydantic<3,>=1.9.0->openai->-r requirements.txt (line 1)) (0.7.0)\n",
            "Requirement already satisfied: pydantic-core==2.27.2 in /usr/local/lib/python3.11/dist-packages (from pydantic<3,>=1.9.0->openai->-r requirements.txt (line 1)) (2.27.2)\n",
            "Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.11/dist-packages (from ray>=2.9->ray[default]>=2.9->vllm>=0.7.2->-r requirements.txt (line 3)) (8.1.8)\n",
            "Requirement already satisfied: msgpack<2.0.0,>=1.0.0 in /usr/local/lib/python3.11/dist-packages (from ray>=2.9->ray[default]>=2.9->vllm>=0.7.2->-r requirements.txt (line 3)) (1.1.0)\n",
            "Collecting aiohttp-cors (from ray[default]>=2.9->vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading aiohttp_cors-0.7.0-py3-none-any.whl.metadata (20 kB)\n",
            "Collecting colorful (from ray[default]>=2.9->vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading colorful-0.5.6-py2.py3-none-any.whl.metadata (16 kB)\n",
            "Collecting opencensus (from ray[default]>=2.9->vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading opencensus-0.11.4-py2.py3-none-any.whl.metadata (12 kB)\n",
            "Requirement already satisfied: smart-open in /usr/local/lib/python3.11/dist-packages (from ray[default]>=2.9->vllm>=0.7.2->-r requirements.txt (line 3)) (7.1.0)\n",
            "Collecting virtualenv!=20.21.1,>=20.0.24 (from ray[default]>=2.9->vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading virtualenv-20.29.2-py3-none-any.whl.metadata (4.5 kB)\n",
            "Collecting py-spy>=0.2.0 (from ray[default]>=2.9->vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading py_spy-0.4.0-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (16 kB)\n",
            "Requirement already satisfied: grpcio>=1.42.0 in /usr/local/lib/python3.11/dist-packages (from ray[default]>=2.9->vllm>=0.7.2->-r requirements.txt (line 3)) (1.70.0)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.11/dist-packages (from requests>=2.32.2->datasets->-r requirements.txt (line 2)) (3.4.1)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.11/dist-packages (from requests>=2.32.2->datasets->-r requirements.txt (line 2)) (2.3.0)\n",
            "Requirement already satisfied: regex>=2022.1.18 in /usr/local/lib/python3.11/dist-packages (from tiktoken>=0.6.0->vllm>=0.7.2->-r requirements.txt (line 3)) (2024.11.6)\n",
            "Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.11/dist-packages (from transformers>=4.48.2->vllm>=0.7.2->-r requirements.txt (line 3)) (0.5.2)\n",
            "Collecting pybind11 (from xgrammar>=0.1.6->vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading pybind11-2.13.6-py3-none-any.whl.metadata (9.5 kB)\n",
            "Requirement already satisfied: pytest in /usr/local/lib/python3.11/dist-packages (from xgrammar>=0.1.6->vllm>=0.7.2->-r requirements.txt (line 3)) (8.3.4)\n",
            "Requirement already satisfied: zipp>=3.20 in /usr/local/lib/python3.11/dist-packages (from importlib_metadata->vllm>=0.7.2->-r requirements.txt (line 3)) (3.21.0)\n",
            "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.11/dist-packages (from pandas->datasets->-r requirements.txt (line 2)) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.11/dist-packages (from pandas->datasets->-r requirements.txt (line 2)) (2025.1)\n",
            "Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.11/dist-packages (from pandas->datasets->-r requirements.txt (line 2)) (2025.1)\n",
            "Collecting httptools>=0.6.3 (from uvicorn[standard]->vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading httptools-0.6.4-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.6 kB)\n",
            "Collecting python-dotenv>=0.13 (from uvicorn[standard]->vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading python_dotenv-1.0.1-py3-none-any.whl.metadata (23 kB)\n",
            "Collecting uvloop!=0.15.0,!=0.15.1,>=0.14.0 (from uvicorn[standard]->vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading uvloop-0.21.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.9 kB)\n",
            "Collecting watchfiles>=0.13 (from uvicorn[standard]->vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading watchfiles-1.0.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.9 kB)\n",
            "Requirement already satisfied: websockets>=10.4 in /usr/local/lib/python3.11/dist-packages (from uvicorn[standard]->vllm>=0.7.2->-r requirements.txt (line 3)) (14.2)\n",
            "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.11/dist-packages (from jsonschema->outlines==0.1.11->vllm>=0.7.2->-r requirements.txt (line 3)) (2024.10.1)\n",
            "Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.11/dist-packages (from jsonschema->outlines==0.1.11->vllm>=0.7.2->-r requirements.txt (line 3)) (0.22.3)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.11/dist-packages (from python-dateutil>=2.8.2->pandas->datasets->-r requirements.txt (line 2)) (1.17.0)\n",
            "Collecting distlib<1,>=0.3.7 (from virtualenv!=20.21.1,>=20.0.24->ray[default]>=2.9->vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading distlib-0.3.9-py2.py3-none-any.whl.metadata (5.2 kB)\n",
            "Requirement already satisfied: platformdirs<5,>=3.9.1 in /usr/local/lib/python3.11/dist-packages (from virtualenv!=20.21.1,>=20.0.24->ray[default]>=2.9->vllm>=0.7.2->-r requirements.txt (line 3)) (4.3.6)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.11/dist-packages (from jinja2->outlines==0.1.11->vllm>=0.7.2->-r requirements.txt (line 3)) (3.0.2)\n",
            "Collecting opencensus-context>=0.1.3 (from opencensus->ray[default]>=2.9->vllm>=0.7.2->-r requirements.txt (line 3))\n",
            "  Downloading opencensus_context-0.1.3-py2.py3-none-any.whl.metadata (3.3 kB)\n",
            "Requirement already satisfied: google-api-core<3.0.0,>=1.0.0 in /usr/local/lib/python3.11/dist-packages (from opencensus->ray[default]>=2.9->vllm>=0.7.2->-r requirements.txt (line 3)) (2.19.2)\n",
            "Requirement already satisfied: iniconfig in /usr/local/lib/python3.11/dist-packages (from pytest->xgrammar>=0.1.6->vllm>=0.7.2->-r requirements.txt (line 3)) (2.0.0)\n",
            "Requirement already satisfied: pluggy<2,>=1.5 in /usr/local/lib/python3.11/dist-packages (from pytest->xgrammar>=0.1.6->vllm>=0.7.2->-r requirements.txt (line 3)) (1.5.0)\n",
            "Requirement already satisfied: wrapt in /usr/local/lib/python3.11/dist-packages (from smart-open->ray[default]>=2.9->vllm>=0.7.2->-r requirements.txt (line 3)) (1.17.2)\n",
            "Requirement already satisfied: googleapis-common-protos<2.0.dev0,>=1.56.2 in /usr/local/lib/python3.11/dist-packages (from google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]>=2.9->vllm>=0.7.2->-r requirements.txt (line 3)) (1.67.0)\n",
            "Requirement already satisfied: proto-plus<2.0.0dev,>=1.22.3 in /usr/local/lib/python3.11/dist-packages (from google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]>=2.9->vllm>=0.7.2->-r requirements.txt (line 3)) (1.26.0)\n",
            "Requirement already satisfied: google-auth<3.0.dev0,>=2.14.1 in /usr/local/lib/python3.11/dist-packages (from google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]>=2.9->vllm>=0.7.2->-r requirements.txt (line 3)) (2.27.0)\n",
            "Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.11/dist-packages (from google-auth<3.0.dev0,>=2.14.1->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]>=2.9->vllm>=0.7.2->-r requirements.txt (line 3)) (5.5.1)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.11/dist-packages (from google-auth<3.0.dev0,>=2.14.1->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]>=2.9->vllm>=0.7.2->-r requirements.txt (line 3)) (0.4.1)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.11/dist-packages (from google-auth<3.0.dev0,>=2.14.1->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]>=2.9->vllm>=0.7.2->-r requirements.txt (line 3)) (4.9)\n",
            "Requirement already satisfied: pyasn1<0.7.0,>=0.4.6 in /usr/local/lib/python3.11/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3.0.dev0,>=2.14.1->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default]>=2.9->vllm>=0.7.2->-r requirements.txt (line 3)) (0.6.1)\n",
            "Downloading datasets-3.3.1-py3-none-any.whl (484 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m484.9/484.9 kB\u001b[0m \u001b[31m23.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading vllm-0.7.2-cp38-abi3-manylinux1_x86_64.whl (264.3 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m264.3/264.3 MB\u001b[0m \u001b[31m5.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading compressed_tensors-0.9.1-py3-none-any.whl (96 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m96.5/96.5 kB\u001b[0m \u001b[31m9.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading depyf-0.18.0-py3-none-any.whl (38 kB)\n",
            "Downloading gguf-0.10.0-py3-none-any.whl (71 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m71.6/71.6 kB\u001b[0m \u001b[31m6.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading lark-1.2.2-py3-none-any.whl (111 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m111.0/111.0 kB\u001b[0m \u001b[31m11.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading outlines-0.1.11-py3-none-any.whl (87 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m87.6/87.6 kB\u001b[0m \u001b[31m8.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading xformers-0.0.28.post3-cp311-cp311-manylinux_2_28_x86_64.whl (16.7 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m16.7/16.7 MB\u001b[0m \u001b[31m91.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cublas_cu12-12.4.5.8-py3-none-manylinux2014_x86_64.whl (363.4 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m363.4/363.4 MB\u001b[0m \u001b[31m3.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cuda_cupti_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl (13.8 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m13.8/13.8 MB\u001b[0m \u001b[31m107.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cuda_nvrtc_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl (24.6 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m24.6/24.6 MB\u001b[0m \u001b[31m74.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cuda_runtime_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl (883 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m883.7/883.7 kB\u001b[0m \u001b[31m56.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cudnn_cu12-9.1.0.70-py3-none-manylinux2014_x86_64.whl (664.8 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m664.8/664.8 MB\u001b[0m \u001b[31m2.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cufft_cu12-11.2.1.3-py3-none-manylinux2014_x86_64.whl (211.5 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m211.5/211.5 MB\u001b[0m \u001b[31m5.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_curand_cu12-10.3.5.147-py3-none-manylinux2014_x86_64.whl (56.3 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m56.3/56.3 MB\u001b[0m \u001b[31m12.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cusolver_cu12-11.6.1.9-py3-none-manylinux2014_x86_64.whl (127.9 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m127.9/127.9 MB\u001b[0m \u001b[31m7.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_cusparse_cu12-12.3.1.170-py3-none-manylinux2014_x86_64.whl (207.5 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m207.5/207.5 MB\u001b[0m \u001b[31m5.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_nvjitlink_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl (21.1 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m21.1/21.1 MB\u001b[0m \u001b[31m66.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading outlines_core-0.1.26-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (343 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m343.3/343.3 kB\u001b[0m \u001b[31m25.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading dill-0.3.8-py3-none-any.whl (116 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m116.3/116.3 kB\u001b[0m \u001b[31m11.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading fastapi-0.115.8-py3-none-any.whl (94 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m94.8/94.8 kB\u001b[0m \u001b[31m9.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading lm_format_enforcer-0.10.10-py3-none-any.whl (44 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m44.2/44.2 kB\u001b[0m \u001b[31m3.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading mistral_common-1.5.3-py3-none-any.whl (6.5 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.5/6.5 MB\u001b[0m \u001b[31m80.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading multiprocess-0.70.16-py311-none-any.whl (143 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m143.5/143.5 kB\u001b[0m \u001b[31m13.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading nvidia_ml_py-12.570.86-py3-none-any.whl (44 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m44.4/44.4 kB\u001b[0m \u001b[31m3.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading prometheus_fastapi_instrumentator-7.0.2-py3-none-any.whl (18 kB)\n",
            "Downloading ray-2.42.1-cp311-cp311-manylinux2014_x86_64.whl (67.4 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m67.4/67.4 MB\u001b[0m \u001b[31m10.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading tiktoken-0.9.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.2/1.2 MB\u001b[0m \u001b[31m62.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading xgrammar-0.1.13-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (402 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m402.4/402.4 kB\u001b[0m \u001b[31m32.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading blake3-1.0.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (376 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m376.2/376.2 kB\u001b[0m \u001b[31m30.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading msgspec-0.19.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (210 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m210.7/210.7 kB\u001b[0m \u001b[31m17.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading partial_json_parser-0.2.1.1.post5-py3-none-any.whl (10 kB)\n",
            "Downloading xxhash-3.5.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m194.8/194.8 kB\u001b[0m \u001b[31m17.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading httptools-0.6.4-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (459 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m459.8/459.8 kB\u001b[0m \u001b[31m35.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading interegular-0.3.3-py37-none-any.whl (23 kB)\n",
            "Downloading py_spy-0.4.0-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (2.7 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.7/2.7 MB\u001b[0m \u001b[31m95.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading python_dotenv-1.0.1-py3-none-any.whl (19 kB)\n",
            "Downloading starlette-0.45.3-py3-none-any.whl (71 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m71.5/71.5 kB\u001b[0m \u001b[31m7.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading uvloop-0.21.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.0 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.0/4.0 MB\u001b[0m \u001b[31m101.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading virtualenv-20.29.2-py3-none-any.whl (4.3 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.3/4.3 MB\u001b[0m \u001b[31m108.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading watchfiles-1.0.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (452 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m452.6/452.6 kB\u001b[0m \u001b[31m35.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading aiohttp_cors-0.7.0-py3-none-any.whl (27 kB)\n",
            "Downloading airportsdata-20241001-py3-none-any.whl (912 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m912.7/912.7 kB\u001b[0m \u001b[31m53.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading astor-0.8.1-py2.py3-none-any.whl (27 kB)\n",
            "Downloading colorful-0.5.6-py2.py3-none-any.whl (201 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m201.4/201.4 kB\u001b[0m \u001b[31m15.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading diskcache-5.6.3-py3-none-any.whl (45 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m45.5/45.5 kB\u001b[0m \u001b[31m3.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading opencensus-0.11.4-py2.py3-none-any.whl (128 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m128.2/128.2 kB\u001b[0m \u001b[31m12.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading pybind11-2.13.6-py3-none-any.whl (243 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m243.3/243.3 kB\u001b[0m \u001b[31m20.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading pycountry-24.6.1-py3-none-any.whl (6.3 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.3/6.3 MB\u001b[0m \u001b[31m108.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading uvicorn-0.34.0-py3-none-any.whl (62 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.3/62.3 kB\u001b[0m \u001b[31m6.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading distlib-0.3.9-py2.py3-none-any.whl (468 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m469.0/469.0 kB\u001b[0m \u001b[31m32.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading opencensus_context-0.1.3-py2.py3-none-any.whl (5.1 kB)\n",
            "Installing collected packages: py-spy, opencensus-context, nvidia-ml-py, distlib, colorful, blake3, xxhash, virtualenv, uvloop, uvicorn, python-dotenv, pycountry, pybind11, partial-json-parser, nvidia-nvjitlink-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, msgspec, lark, interegular, httptools, gguf, diskcache, dill, astor, airportsdata, watchfiles, tiktoken, starlette, nvidia-cusparse-cu12, nvidia-cudnn-cu12, multiprocess, depyf, prometheus-fastapi-instrumentator, nvidia-cusolver-cu12, lm-format-enforcer, fastapi, aiohttp-cors, ray, outlines_core, opencensus, mistral_common, datasets, xgrammar, xformers, outlines, compressed-tensors, vllm\n",
            "  Attempting uninstall: nvidia-nvjitlink-cu12\n",
            "    Found existing installation: nvidia-nvjitlink-cu12 12.5.82\n",
            "    Uninstalling nvidia-nvjitlink-cu12-12.5.82:\n",
            "      Successfully uninstalled nvidia-nvjitlink-cu12-12.5.82\n",
            "  Attempting uninstall: nvidia-curand-cu12\n",
            "    Found existing installation: nvidia-curand-cu12 10.3.6.82\n",
            "    Uninstalling nvidia-curand-cu12-10.3.6.82:\n",
            "      Successfully uninstalled nvidia-curand-cu12-10.3.6.82\n",
            "  Attempting uninstall: nvidia-cufft-cu12\n",
            "    Found existing installation: nvidia-cufft-cu12 11.2.3.61\n",
            "    Uninstalling nvidia-cufft-cu12-11.2.3.61:\n",
            "      Successfully uninstalled nvidia-cufft-cu12-11.2.3.61\n",
            "  Attempting uninstall: nvidia-cuda-runtime-cu12\n",
            "    Found existing installation: nvidia-cuda-runtime-cu12 12.5.82\n",
            "    Uninstalling nvidia-cuda-runtime-cu12-12.5.82:\n",
            "      Successfully uninstalled nvidia-cuda-runtime-cu12-12.5.82\n",
            "  Attempting uninstall: nvidia-cuda-nvrtc-cu12\n",
            "    Found existing installation: nvidia-cuda-nvrtc-cu12 12.5.82\n",
            "    Uninstalling nvidia-cuda-nvrtc-cu12-12.5.82:\n",
            "      Successfully uninstalled nvidia-cuda-nvrtc-cu12-12.5.82\n",
            "  Attempting uninstall: nvidia-cuda-cupti-cu12\n",
            "    Found existing installation: nvidia-cuda-cupti-cu12 12.5.82\n",
            "    Uninstalling nvidia-cuda-cupti-cu12-12.5.82:\n",
            "      Successfully uninstalled nvidia-cuda-cupti-cu12-12.5.82\n",
            "  Attempting uninstall: nvidia-cublas-cu12\n",
            "    Found existing installation: nvidia-cublas-cu12 12.5.3.2\n",
            "    Uninstalling nvidia-cublas-cu12-12.5.3.2:\n",
            "      Successfully uninstalled nvidia-cublas-cu12-12.5.3.2\n",
            "  Attempting uninstall: nvidia-cusparse-cu12\n",
            "    Found existing installation: nvidia-cusparse-cu12 12.5.1.3\n",
            "    Uninstalling nvidia-cusparse-cu12-12.5.1.3:\n",
            "      Successfully uninstalled nvidia-cusparse-cu12-12.5.1.3\n",
            "  Attempting uninstall: nvidia-cudnn-cu12\n",
            "    Found existing installation: nvidia-cudnn-cu12 9.3.0.75\n",
            "    Uninstalling nvidia-cudnn-cu12-9.3.0.75:\n",
            "      Successfully uninstalled nvidia-cudnn-cu12-9.3.0.75\n",
            "  Attempting uninstall: nvidia-cusolver-cu12\n",
            "    Found existing installation: nvidia-cusolver-cu12 11.6.3.83\n",
            "    Uninstalling nvidia-cusolver-cu12-11.6.3.83:\n",
            "      Successfully uninstalled nvidia-cusolver-cu12-11.6.3.83\n",
            "Successfully installed aiohttp-cors-0.7.0 airportsdata-20241001 astor-0.8.1 blake3-1.0.4 colorful-0.5.6 compressed-tensors-0.9.1 datasets-3.3.1 depyf-0.18.0 dill-0.3.8 diskcache-5.6.3 distlib-0.3.9 fastapi-0.115.8 gguf-0.10.0 httptools-0.6.4 interegular-0.3.3 lark-1.2.2 lm-format-enforcer-0.10.10 mistral_common-1.5.3 msgspec-0.19.0 multiprocess-0.70.16 nvidia-cublas-cu12-12.4.5.8 nvidia-cuda-cupti-cu12-12.4.127 nvidia-cuda-nvrtc-cu12-12.4.127 nvidia-cuda-runtime-cu12-12.4.127 nvidia-cudnn-cu12-9.1.0.70 nvidia-cufft-cu12-11.2.1.3 nvidia-curand-cu12-10.3.5.147 nvidia-cusolver-cu12-11.6.1.9 nvidia-cusparse-cu12-12.3.1.170 nvidia-ml-py-12.570.86 nvidia-nvjitlink-cu12-12.4.127 opencensus-0.11.4 opencensus-context-0.1.3 outlines-0.1.11 outlines_core-0.1.26 partial-json-parser-0.2.1.1.post5 prometheus-fastapi-instrumentator-7.0.2 py-spy-0.4.0 pybind11-2.13.6 pycountry-24.6.1 python-dotenv-1.0.1 ray-2.42.1 starlette-0.45.3 tiktoken-0.9.0 uvicorn-0.34.0 uvloop-0.21.0 virtualenv-20.29.2 vllm-0.7.2 watchfiles-1.0.4 xformers-0.0.28.post3 xgrammar-0.1.13 xxhash-3.5.0\n"
          ]
        }
      ],
      "source": [
        "%cd RocketEval-ICLR\n",
        "!pip install -r requirements.txt"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uLp8TjLGw4R5"
      },
      "source": [
        "------\n",
        "\n",
        "Next, we'll configure the evaluation parameters. This notebook demonstrates a minimal evaluation setup with the following key arguments:\n",
        "\n",
        "- `dataset`: [MT-Bench](https://huggingface.co/spaces/lmsys/mt-bench/tree/main), a compact LLM evaluation dataset\n",
        "- `generator`: [GPT-4](https://chatgpt.com/) for checklist generation (we'll use pre-generated checklists)\n",
        "- `judge`: [Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct), chosen for efficient evaluation on a Tesla T4 GPU\n",
        "- `train_test`: Enables split evaluation using model sets from:\n",
        "  - Training: `config/rankings/mt-bench_train.json`\n",
        "  - Testing: `config/rankings/mt-bench_test.json`\n",
        "- `mode`: \"offline\" to utilize the local vLLM framework\n",
        "- `gpu_ids`: \"0\" for single GPU execution\n",
        "- `offline_config`: vLLM engine configuration file path"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "tWvKcVhVB6yz"
      },
      "outputs": [],
      "source": [
        "# Arguments for the evaluation task\n",
        "default_args=\"\"\"--dataset mt-bench --generator gpt-4o --judge Qwen2.5-0.5B-Instruct --train_test --mode offline --gpu_ids 0 --offline_config config/offline/colab.yaml\"\"\".split()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oobdufUBX_qj"
      },
      "source": [
        "------\n",
        "\n",
        "Next, we start to run our task.\n",
        "\n",
        "We'll use the default arguments defined above to:\n",
        "1. Load the MT-Bench dataset\n",
        "2. Initialize the Qwen2.5-0.5B-Instruct judge model using vLLM\n",
        "3. Run the evaluation pipeline including:\n",
        "   - Checklist-based grading\n",
        "   - Score aggregation\n",
        "   - Model ranking"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000,
          "referenced_widgets": [
            "f0ca97397a7d414681137b3631195323",
            "ca10621991344a54b954d33896490c4e",
            "7e516071ac6543c09fe0d03da38dca27",
            "1d8267cfa61b4f4581297e48fd3d0f40",
            "10ac5ec7cfc84574911155d9a91dac08",
            "1d075309fb4249739937263a4891d786",
            "2b15babdc8b845ef87d6a0a2f6eddfa1",
            "fa6df7a068aa4f22999c870eb8dbe239",
            "c308c2961efa49aeb4c4fce5a8e31417",
            "d91e437520e9463c87e37d536a84863f",
            "e34e3ef6cb584031bba292d3984518f1",
            "7a152b2cc0b7451a809008aa2c48e917",
            "887cb4fc907e4dbcbe83832ca8b9fec2",
            "893282ec30034f6180fefd69607676a5",
            "dbbf390290254a59867aa99a37b264f9",
            "a63630c39e3343e8beb28628fb13776f",
            "af5d28944f70409e8007250f47fd7a3c",
            "4077c004e3ec488ab059cb6356c2c0dc",
            "2c8276ba20714bd2b074bba6c3764233",
            "9a4ff6c6c2f4480899bd8e4ec10e9e1d",
            "4c586706c1d647db94830113079fa3cb",
            "31b2ac73b1b74e96a5604e6ecb265c74",
            "3e61fab06c3c489d8298fdbb429c44fe",
            "0c2c03b019ab42579aa7e8a246825e3c",
            "2691cda4645d4d38bbbfb3db3f4d66b5",
            "cad0b22f16b248d193563dd1700e6593",
            "fed97b4f8c9641f284cc2cf74250f650",
            "72e43f9a4caf44d4a133836616970c35",
            "96ea0d6d4bf3420989e082ebaedb510a",
            "57d514c2c51b400da0cc090a3a840712",
            "1c6075cfb5044872a8fde4a8a45ca9b5",
            "1486d20ba48b4f30b2f0dd4ed3692a23",
            "183d20b14ae14023bfe7d8c0bc06157c",
            "9c84ef32a5d444f2b209e4687aaf0d4a",
            "3015a163001e4325871f1a2e96c7ccaa",
            "d35c4ac37946451d967d6c29a53304ed",
            "c02cdfbd429f407586cbdd9850455241",
            "bc3bdda8cb734eab87622c79953ad233",
            "acc07336d9554e2b910add79fed8ff7b",
            "eb2d22999840434eac2918991d8fea57",
            "6320e89229294ff0a6475ee62dc5c40e",
            "16778992f3cd4ce6b1d8a4dcc9a47a4f",
            "88833a49f7ab488a95d906a93413cf8f",
            "e68985d26aa549569b458e41fa368ba5",
            "9881d7b4252747db8687dff9ca922f34",
            "4d1914b6b0ad4abd8eb7a640cd18a9cb",
            "319439bf8ad24ce5b041c8b1310af37b",
            "6e273761535d45a590c8d32b7e2aa759",
            "5a023d3d765a4a37b30d17afcecd6d2c",
            "609cbcf1563343609476fa287af7fa5a",
            "dc1358d5ee664badb20ba37ed0df01f0",
            "9d861516c20548e3a1c703298e6aea0d",
            "cc6281b16cc9498cbb42aa1ad48ba22b",
            "dd705f26a0b64e78820eefbd1fb052da",
            "4d78f541b9e545cc9e00975c7ffe1dc3",
            "7583ff022e704544bdc77fe5f996a1ea",
            "472294a12fc24aa28b60fa5c3668deaf",
            "67de9d68a38141fba801cb1edbb69748",
            "2f742a948cb94da98b76507cba0ae598",
            "1d22a44e8d594b9e8d60e998285002b2",
            "342c5e84ea23441cb46c419f8489092e",
            "aaabd83129ab42ef9277b1700a788dac",
            "f1bd85aea53f49dba6c208a4bef80a70",
            "201c892fe4df466696123628f6b9b903",
            "dc3c9494e5b0439aacd098e9d4c56928",
            "0360055205374b6aa35e93256dc0c08c",
            "c319df7b698d492db355666730d51a2a",
            "80769bc182804ab281687dfb42b7a6de",
            "34631d319a3343e7940d969ccdfdf2ee",
            "8ea79a10e5d54ac88004c8d6eb4c5924",
            "70fb35b6842b480c8f150bd9f66bc44c",
            "38977214c3c247708294725881ddb057",
            "a70f34b5e4314e828012ef8d8560c199",
            "3b7534af827d401f849dd657a21b27a2",
            "dd4da084890040949b0676569db25a14",
            "e71b2c79095f490c88bf05ba6a09250f",
            "dd9defd662614a80a07ade0fb91180db",
            "f435ed80fb7d42fa8dfc16df2afa1916",
            "17429b88a2fc49c58fa2807ede6c4927",
            "e441522d4faa47278c6ab0e4f65388e3",
            "dece1d256a5b4d5999cc7bc288c86a56",
            "dbc8e87e1b314deda8702d24057cb645",
            "9751e5e354184c4487f41c20e5fedca0",
            "947e308648434526a6bd12d4384aef95",
            "b7f517668c1d46dd982b1feb2853524a",
            "c9d1630fa487490ab59aeb8adfc1151c",
            "c678d2f581c2402cb5a5a8828bb53fee",
            "b736a7dc7bd84535acc9ef717a26096c"
          ]
        },
        "id": "Tsf_rEhCtkBH",
        "outputId": "b9d56bbb-5858-4cf8-c8b7-e614e966cc9c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "INFO 02-19 06:24:54 __init__.py:190] Automatically detected platform cuda.\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[2;36m[06:24:54]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m \u001b[1;4;5;31;47mRocketEval\u001b[0m                                                    \u001b]8;id=540311;file://<ipython-input-4-049d212b5d3f>\u001b\\\u001b[2m<ipython-input-4-049d212b5d3f>\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=568636;file://<ipython-input-4-049d212b5d3f>#98\u001b\\\u001b[2m98\u001b[0m\u001b]8;;\u001b\\\n",
              "\u001b[2;36m           \u001b[0m         \u001b[1;5;33;41m Task Information\u001b[0m                                             \u001b[2m                                 \u001b[0m\n",
              "\u001b[2;36m           \u001b[0m         - Dataset: \u001b[32m\"mt-bench\"\u001b[0m                                         \u001b[2m                                 \u001b[0m\n",
              "\u001b[2;36m           \u001b[0m         - Judge: \u001b[32m\"Qwen2.5-0.5B-Instruct\"\u001b[0m                              \u001b[2m                                 \u001b[0m\n",
              "\u001b[2;36m           \u001b[0m         - Labeler: \u001b[32m\"gpt-4o\"\u001b[0m                                           \u001b[2m                                 \u001b[0m\n",
              "\u001b[2;36m           \u001b[0m         - Task ID: \u001b[32m\"mt-bench_Qwen2.5-0.5B-Instruct_1739946294\"\u001b[0m        \u001b[2m                                 \u001b[0m\n",
              "\u001b[2;36m           \u001b[0m                                                                       \u001b[2m                                 \u001b[0m\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[06:24:54] </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> <span style=\"color: #800000; text-decoration-color: #800000; background-color: #c0c0c0; font-weight: bold; text-decoration: underline\">RocketEval</span>                                                    <a href=\"file://<ipython-input-4-049d212b5d3f>\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">&lt;ipython-input-4-049d212b5d3f&gt;</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file://<ipython-input-4-049d212b5d3f>#98\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">98</span></a>\n",
              "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span>         <span style=\"color: #808000; text-decoration-color: #808000; background-color: #800000; font-weight: bold\"> Task Information</span>                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                                 </span>\n",
              "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span>         - Dataset: <span style=\"color: #008000; text-decoration-color: #008000\">\"mt-bench\"</span>                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                                 </span>\n",
              "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span>         - Judge: <span style=\"color: #008000; text-decoration-color: #008000\">\"Qwen2.5-0.5B-Instruct\"</span>                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                                 </span>\n",
              "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span>         - Labeler: <span style=\"color: #008000; text-decoration-color: #008000\">\"gpt-4o\"</span>                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                                 </span>\n",
              "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span>         - Task ID: <span style=\"color: #008000; text-decoration-color: #008000\">\"mt-bench_Qwen2.5-0.5B-Instruct_1739946294\"</span>        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                                 </span>\n",
              "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span>                                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                                 </span>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[2;36m          \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m \u001b[1;5;33;41mRocketEval Completed\u001b[0m                                         \u001b]8;id=949880;file://<ipython-input-4-049d212b5d3f>\u001b\\\u001b[2m<ipython-input-4-049d212b5d3f>\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=323584;file://<ipython-input-4-049d212b5d3f>#101\u001b\\\u001b[2m101\u001b[0m\u001b]8;;\u001b\\\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> <span style=\"color: #808000; text-decoration-color: #808000; background-color: #800000; font-weight: bold\">RocketEval Completed</span>                                         <a href=\"file://<ipython-input-4-049d212b5d3f>\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">&lt;ipython-input-4-049d212b5d3f&gt;</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file://<ipython-input-4-049d212b5d3f>#101\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">101</span></a>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[2;36m          \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m \u001b[1;5;33;41mChecklist Creation skipped.\u001b[0m                                  \u001b]8;id=927357;file://<ipython-input-4-049d212b5d3f>\u001b\\\u001b[2m<ipython-input-4-049d212b5d3f>\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=935047;file://<ipython-input-4-049d212b5d3f>#120\u001b\\\u001b[2m120\u001b[0m\u001b]8;;\u001b\\\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> <span style=\"color: #808000; text-decoration-color: #808000; background-color: #800000; font-weight: bold\">Checklist Creation skipped.</span>                                  <a href=\"file://<ipython-input-4-049d212b5d3f>\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">&lt;ipython-input-4-049d212b5d3f&gt;</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file://<ipython-input-4-049d212b5d3f>#120\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">120</span></a>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[2;36m          \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m \u001b[1;5;33;41mII. Judgment Creation\u001b[0m                                        \u001b]8;id=876422;file://<ipython-input-4-049d212b5d3f>\u001b\\\u001b[2m<ipython-input-4-049d212b5d3f>\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=392514;file://<ipython-input-4-049d212b5d3f>#125\u001b\\\u001b[2m125\u001b[0m\u001b]8;;\u001b\\\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> <span style=\"color: #808000; text-decoration-color: #808000; background-color: #800000; font-weight: bold\">II. Judgment Creation</span>                                        <a href=\"file://<ipython-input-4-049d212b5d3f>\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">&lt;ipython-input-4-049d212b5d3f&gt;</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file://<ipython-input-4-049d212b5d3f>#125\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">125</span></a>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[2;36m          \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Loaded \u001b[1;36m160\u001b[0m instances from \u001b[32m'mt-bench'\u001b[0m                                             \u001b]8;id=754896;file:///content/RocketEval-ICLR/src/rocketeval/process/judgment.py\u001b\\\u001b[2mjudgment.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=685548;file:///content/RocketEval-ICLR/src/rocketeval/process/judgment.py#32\u001b\\\u001b[2m32\u001b[0m\u001b]8;;\u001b\\\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Loaded <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">160</span> instances from <span style=\"color: #008000; text-decoration-color: #008000\">'mt-bench'</span>                                             <a href=\"file:///content/RocketEval-ICLR/src/rocketeval/process/judgment.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">judgment.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///content/RocketEval-ICLR/src/rocketeval/process/judgment.py#32\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">32</span></a>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[2;36m          \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Generated \u001b[1;36m1206\u001b[0m questions for judgment, average No. of questions per instance:    \u001b]8;id=895669;file:///content/RocketEval-ICLR/src/rocketeval/process/judgment.py\u001b\\\u001b[2mjudgment.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=263674;file:///content/RocketEval-ICLR/src/rocketeval/process/judgment.py#43\u001b\\\u001b[2m43\u001b[0m\u001b]8;;\u001b\\\n",
              "\u001b[2;36m           \u001b[0m         \u001b[1;36m7.5375\u001b[0m                                                                           \u001b[2m              \u001b[0m\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Generated <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1206</span> questions for judgment, average No. of questions per instance:    <a href=\"file:///content/RocketEval-ICLR/src/rocketeval/process/judgment.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">judgment.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///content/RocketEval-ICLR/src/rocketeval/process/judgment.py#43\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">43</span></a>\n",
              "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span>         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7.5375</span>                                                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">              </span>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[2;36m          \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Loaded \u001b[1;36m3200\u001b[0m results from \u001b[1;36m20\u001b[0m models                                               \u001b]8;id=501712;file:///content/RocketEval-ICLR/src/rocketeval/process/judgment.py\u001b\\\u001b[2mjudgment.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=337569;file:///content/RocketEval-ICLR/src/rocketeval/process/judgment.py#47\u001b\\\u001b[2m47\u001b[0m\u001b]8;;\u001b\\\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Loaded <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3200</span> results from <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">20</span> models                                               <a href=\"file:///content/RocketEval-ICLR/src/rocketeval/process/judgment.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">judgment.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///content/RocketEval-ICLR/src/rocketeval/process/judgment.py#47\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">47</span></a>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 20/20 [00:01<00:00, 11.34it/s]\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[2;36m[06:24:57]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Batch file output to                                                             \u001b]8;id=725644;file:///content/RocketEval-ICLR/src/rocketeval/process/judgment.py\u001b\\\u001b[2mjudgment.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=920512;file:///content/RocketEval-ICLR/src/rocketeval/process/judgment.py#86\u001b\\\u001b[2m86\u001b[0m\u001b]8;;\u001b\\\n",
              "\u001b[2;36m           \u001b[0m         \u001b[32m\"data/batch/mt-bench_Qwen2.5-0.5B-Instruct_1739946294_judgment.batch_submission.\u001b[0m \u001b[2m              \u001b[0m\n",
              "\u001b[2;36m           \u001b[0m         \u001b[32mjsonl\"\u001b[0m                                                                           \u001b[2m              \u001b[0m\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[06:24:57] </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Batch file output to                                                             <a href=\"file:///content/RocketEval-ICLR/src/rocketeval/process/judgment.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">judgment.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///content/RocketEval-ICLR/src/rocketeval/process/judgment.py#86\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">86</span></a>\n",
              "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span>         <span style=\"color: #008000; text-decoration-color: #008000\">\"data/batch/mt-bench_Qwen2.5-0.5B-Instruct_1739946294_judgment.batch_submission.</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">              </span>\n",
              "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span>         <span style=\"color: #008000; text-decoration-color: #008000\">jsonl\"</span>                                                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">              </span>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[2;36m[06:24:58]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Total samples: \u001b[1;36m24120\u001b[0m                                                              \u001b]8;id=760794;file:///content/RocketEval-ICLR/src/rocketeval/tools/offline.py\u001b\\\u001b[2moffline.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=178276;file:///content/RocketEval-ICLR/src/rocketeval/tools/offline.py#97\u001b\\\u001b[2m97\u001b[0m\u001b]8;;\u001b\\\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[06:24:58] </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Total samples: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">24120</span>                                                              <a href=\"file:///content/RocketEval-ICLR/src/rocketeval/tools/offline.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">offline.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///content/RocketEval-ICLR/src/rocketeval/tools/offline.py#97\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">97</span></a>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.11/dist-packages/huggingface_hub/utils/_auth.py:104: UserWarning: \n",
            "Error while fetching `HF_TOKEN` secret value from your vault: 'Requesting secret HF_TOKEN timed out. Secrets can only be fetched when running from the Colab UI.'.\n",
            "You are not authenticated with the Hugging Face Hub in this notebook.\n",
            "If the error persists, please let us know by opening an issue on GitHub (https://github.com/huggingface/huggingface_hub/issues/new).\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "config.json:   0%|          | 0.00/659 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "f0ca97397a7d414681137b3631195323"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "WARNING 02-19 06:25:11 config.py:2386] Casting torch.bfloat16 to torch.float16.\n",
            "INFO 02-19 06:25:25 config.py:542] This model supports multiple tasks: {'classify', 'reward', 'embed', 'score', 'generate'}. Defaulting to 'generate'.\n",
            "INFO 02-19 06:25:25 llm_engine.py:234] Initializing a V0 LLM engine (v0.7.2) with config: model='Qwen/Qwen2.5-0.5B-Instruct', speculative_config=None, tokenizer='Qwen/Qwen2.5-0.5B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=2400, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=Qwen/Qwen2.5-0.5B-Instruct, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=False, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={\"splitting_ops\":[],\"compile_sizes\":[],\"cudagraph_capture_sizes\":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],\"max_capture_size\":512}, use_cached_outputs=False, \n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "tokenizer_config.json:   0%|          | 0.00/7.30k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "7a152b2cc0b7451a809008aa2c48e917"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "vocab.json:   0%|          | 0.00/2.78M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "3e61fab06c3c489d8298fdbb429c44fe"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "merges.txt:   0%|          | 0.00/1.67M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "9c84ef32a5d444f2b209e4687aaf0d4a"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "tokenizer.json:   0%|          | 0.00/7.03M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "9881d7b4252747db8687dff9ca922f34"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "generation_config.json:   0%|          | 0.00/242 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "7583ff022e704544bdc77fe5f996a1ea"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "INFO 02-19 06:25:30 cuda.py:179] Cannot use FlashAttention-2 backend for Volta and Turing GPUs.\n",
            "INFO 02-19 06:25:30 cuda.py:227] Using XFormers backend.\n",
            "INFO 02-19 06:25:31 model_runner.py:1110] Starting to load model Qwen/Qwen2.5-0.5B-Instruct...\n",
            "INFO 02-19 06:25:31 weight_utils.py:252] Using model weights format ['*.safetensors']\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "model.safetensors:   0%|          | 0.00/988M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "c319df7b698d492db355666730d51a2a"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "INFO 02-19 06:25:55 weight_utils.py:297] No model.safetensors.index.json found in remote.\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]\n"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "f435ed80fb7d42fa8dfc16df2afa1916"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "INFO 02-19 06:25:57 model_runner.py:1115] Loading model weights took 0.9277 GB\n",
            "INFO 02-19 06:25:58 worker.py:267] Memory profiling takes 1.30 seconds\n",
            "INFO 02-19 06:25:58 worker.py:267] the current vLLM instance can use total_gpu_memory (14.74GiB) x gpu_memory_utilization (0.90) = 13.27GiB\n",
            "INFO 02-19 06:25:58 worker.py:267] model weights take 0.93GiB; non_torch_memory takes 0.05GiB; PyTorch activation peak memory takes 2.77GiB; the rest of the memory reserved for KV Cache is 9.53GiB.\n",
            "INFO 02-19 06:25:59 executor_base.py:110] # CUDA blocks: 52027, # CPU blocks: 21845\n",
            "INFO 02-19 06:25:59 executor_base.py:115] Maximum concurrency for 2400 tokens per request: 346.85x\n",
            "INFO 02-19 06:26:05 model_runner.py:1434] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. If out-of-memory error occurs during cudagraph capture, consider decreasing `gpu_memory_utilization` or switching to eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Capturing CUDA graph shapes: 100%|██████████| 67/67 [01:04<00:00,  1.04it/s]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "INFO 02-19 06:27:09 model_runner.py:1562] Graph capturing finished in 64 secs, took 0.23 GiB\n",
            "INFO 02-19 06:27:09 llm_engine.py:431] init engine (profile, create kv cache, warmup model) took 72.42 seconds\n",
            "INFO 02-19 06:27:12 chat_utils.py:332] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Processed prompts: 100%|██████████| 24120/24120 [06:33<00:00, 61.22it/s, est. speed input: 51786.82 toks/s, output: 61.22 toks/s]\n",
            "100%|██████████| 20/20 [00:00<00:00, 42.28it/s]\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[2;36m[06:35:53]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Judgment results output to \u001b[32m\"data/mt-bench/judgment/Qwen2.5-0.5B-Instruct\"\u001b[0m       \u001b]8;id=603424;file:///content/RocketEval-ICLR/src/rocketeval/process/judgment.py\u001b\\\u001b[2mjudgment.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=661377;file:///content/RocketEval-ICLR/src/rocketeval/process/judgment.py#127\u001b\\\u001b[2m127\u001b[0m\u001b]8;;\u001b\\\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[06:35:53] </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Judgment results output to <span style=\"color: #008000; text-decoration-color: #008000\">\"data/mt-bench/judgment/Qwen2.5-0.5B-Instruct\"</span>       <a href=\"file:///content/RocketEval-ICLR/src/rocketeval/process/judgment.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">judgment.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///content/RocketEval-ICLR/src/rocketeval/process/judgment.py#127\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">127</span></a>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[2;36m          \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m \u001b[33mJudgment Creation completed.\u001b[0m                                 \u001b]8;id=850045;file://<ipython-input-4-049d212b5d3f>\u001b\\\u001b[2m<ipython-input-4-049d212b5d3f>\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=902473;file://<ipython-input-4-049d212b5d3f>#136\u001b\\\u001b[2m136\u001b[0m\u001b]8;;\u001b\\\n",
              "\u001b[2;36m           \u001b[0m                                                                      \u001b[2m                                  \u001b[0m\n",
              "\u001b[2;36m           \u001b[0m                                                                      \u001b[2m                                  \u001b[0m\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> <span style=\"color: #808000; text-decoration-color: #808000\">Judgment Creation completed.</span>                                 <a href=\"file://<ipython-input-4-049d212b5d3f>\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">&lt;ipython-input-4-049d212b5d3f&gt;</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file://<ipython-input-4-049d212b5d3f>#136\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">136</span></a>\n",
              "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span>                                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                                  </span>\n",
              "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span>                                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                                  </span>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[2;36m          \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m \u001b[1;5;33;41mIII. Score Creation\u001b[0m                                          \u001b]8;id=49633;file://<ipython-input-4-049d212b5d3f>\u001b\\\u001b[2m<ipython-input-4-049d212b5d3f>\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=707213;file://<ipython-input-4-049d212b5d3f>#143\u001b\\\u001b[2m143\u001b[0m\u001b]8;;\u001b\\\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> <span style=\"color: #808000; text-decoration-color: #808000; background-color: #800000; font-weight: bold\">III. Score Creation</span>                                          <a href=\"file://<ipython-input-4-049d212b5d3f>\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">&lt;ipython-input-4-049d212b5d3f&gt;</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file://<ipython-input-4-049d212b5d3f>#143\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">143</span></a>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\r  0%|          | 0/10 [00:00<?, ?it/s]"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[2;36m[06:35:55]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Score of \u001b[32m\"gpt-4o-mini\"\u001b[0m output to                                                   \u001b]8;id=179359;file:///content/RocketEval-ICLR/src/rocketeval/process/score.py\u001b\\\u001b[2mscore.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=387584;file:///content/RocketEval-ICLR/src/rocketeval/process/score.py#137\u001b\\\u001b[2m137\u001b[0m\u001b]8;;\u001b\\\n",
              "\u001b[2;36m           \u001b[0m         \u001b[32m\"data/mt-bench/score/Qwen2.5-0.5B-Instruct/gpt-4o-mini.json\"\u001b[0m                       \u001b[2m            \u001b[0m\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[06:35:55] </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Score of <span style=\"color: #008000; text-decoration-color: #008000\">\"gpt-4o-mini\"</span> output to                                                   <a href=\"file:///content/RocketEval-ICLR/src/rocketeval/process/score.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">score.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///content/RocketEval-ICLR/src/rocketeval/process/score.py#137\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">137</span></a>\n",
              "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span>         <span style=\"color: #008000; text-decoration-color: #008000\">\"data/mt-bench/score/Qwen2.5-0.5B-Instruct/gpt-4o-mini.json\"</span>                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">            </span>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[2;36m          \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Score of \u001b[32m\"yi-large-preview\"\u001b[0m output to                                              \u001b]8;id=257980;file:///content/RocketEval-ICLR/src/rocketeval/process/score.py\u001b\\\u001b[2mscore.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=238779;file:///content/RocketEval-ICLR/src/rocketeval/process/score.py#137\u001b\\\u001b[2m137\u001b[0m\u001b]8;;\u001b\\\n",
              "\u001b[2;36m           \u001b[0m         \u001b[32m\"data/mt-bench/score/Qwen2.5-0.5B-Instruct/yi-large-preview.json\"\u001b[0m                  \u001b[2m            \u001b[0m\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Score of <span style=\"color: #008000; text-decoration-color: #008000\">\"yi-large-preview\"</span> output to                                              <a href=\"file:///content/RocketEval-ICLR/src/rocketeval/process/score.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">score.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///content/RocketEval-ICLR/src/rocketeval/process/score.py#137\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">137</span></a>\n",
              "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span>         <span style=\"color: #008000; text-decoration-color: #008000\">\"data/mt-bench/score/Qwen2.5-0.5B-Instruct/yi-large-preview.json\"</span>                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">            </span>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[2;36m          \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Score of \u001b[32m\"Meta-Llama3-70B-Instruct\"\u001b[0m output to                                      \u001b]8;id=204965;file:///content/RocketEval-ICLR/src/rocketeval/process/score.py\u001b\\\u001b[2mscore.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=799375;file:///content/RocketEval-ICLR/src/rocketeval/process/score.py#137\u001b\\\u001b[2m137\u001b[0m\u001b]8;;\u001b\\\n",
              "\u001b[2;36m           \u001b[0m         \u001b[32m\"data/mt-bench/score/Qwen2.5-0.5B-Instruct/Meta-Llama3-70B-Instruct.json\"\u001b[0m          \u001b[2m            \u001b[0m\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Score of <span style=\"color: #008000; text-decoration-color: #008000\">\"Meta-Llama3-70B-Instruct\"</span> output to                                      <a href=\"file:///content/RocketEval-ICLR/src/rocketeval/process/score.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">score.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///content/RocketEval-ICLR/src/rocketeval/process/score.py#137\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">137</span></a>\n",
              "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span>         <span style=\"color: #008000; text-decoration-color: #008000\">\"data/mt-bench/score/Qwen2.5-0.5B-Instruct/Meta-Llama3-70B-Instruct.json\"</span>          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">            </span>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[2;36m          \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Score of \u001b[32m\"gpt-4\"\u001b[0m output to \u001b[32m\"data/mt-bench/score/Qwen2.5-0.5B-Instruct/gpt-4.json\"\u001b[0m  \u001b]8;id=660511;file:///content/RocketEval-ICLR/src/rocketeval/process/score.py\u001b\\\u001b[2mscore.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=429016;file:///content/RocketEval-ICLR/src/rocketeval/process/score.py#137\u001b\\\u001b[2m137\u001b[0m\u001b]8;;\u001b\\\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Score of <span style=\"color: #008000; text-decoration-color: #008000\">\"gpt-4\"</span> output to <span style=\"color: #008000; text-decoration-color: #008000\">\"data/mt-bench/score/Qwen2.5-0.5B-Instruct/gpt-4.json\"</span>  <a href=\"file:///content/RocketEval-ICLR/src/rocketeval/process/score.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">score.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///content/RocketEval-ICLR/src/rocketeval/process/score.py#137\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">137</span></a>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[2;36m          \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Score of \u001b[32m\"Yi-1.5-34B-Chat\"\u001b[0m output to                                               \u001b]8;id=861412;file:///content/RocketEval-ICLR/src/rocketeval/process/score.py\u001b\\\u001b[2mscore.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=938850;file:///content/RocketEval-ICLR/src/rocketeval/process/score.py#137\u001b\\\u001b[2m137\u001b[0m\u001b]8;;\u001b\\\n",
              "\u001b[2;36m           \u001b[0m         \u001b[32m\"data/mt-bench/score/Qwen2.5-0.5B-Instruct/Yi-1.5-34B-Chat.json\"\u001b[0m                   \u001b[2m            \u001b[0m\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Score of <span style=\"color: #008000; text-decoration-color: #008000\">\"Yi-1.5-34B-Chat\"</span> output to                                               <a href=\"file:///content/RocketEval-ICLR/src/rocketeval/process/score.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">score.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///content/RocketEval-ICLR/src/rocketeval/process/score.py#137\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">137</span></a>\n",
              "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span>         <span style=\"color: #008000; text-decoration-color: #008000\">\"data/mt-bench/score/Qwen2.5-0.5B-Instruct/Yi-1.5-34B-Chat.json\"</span>                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">            </span>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[2;36m          \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Score of \u001b[32m\"Meta-Llama-3-8B-Instruct\"\u001b[0m output to                                      \u001b]8;id=439331;file:///content/RocketEval-ICLR/src/rocketeval/process/score.py\u001b\\\u001b[2mscore.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=771427;file:///content/RocketEval-ICLR/src/rocketeval/process/score.py#137\u001b\\\u001b[2m137\u001b[0m\u001b]8;;\u001b\\\n",
              "\u001b[2;36m           \u001b[0m         \u001b[32m\"data/mt-bench/score/Qwen2.5-0.5B-Instruct/Meta-Llama-3-8B-Instruct.json\"\u001b[0m          \u001b[2m            \u001b[0m\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Score of <span style=\"color: #008000; text-decoration-color: #008000\">\"Meta-Llama-3-8B-Instruct\"</span> output to                                      <a href=\"file:///content/RocketEval-ICLR/src/rocketeval/process/score.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">score.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///content/RocketEval-ICLR/src/rocketeval/process/score.py#137\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">137</span></a>\n",
              "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span>         <span style=\"color: #008000; text-decoration-color: #008000\">\"data/mt-bench/score/Qwen2.5-0.5B-Instruct/Meta-Llama-3-8B-Instruct.json\"</span>          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">            </span>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[2;36m          \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Score of \u001b[32m\"claude-v1\"\u001b[0m output to                                                     \u001b]8;id=429959;file:///content/RocketEval-ICLR/src/rocketeval/process/score.py\u001b\\\u001b[2mscore.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=879542;file:///content/RocketEval-ICLR/src/rocketeval/process/score.py#137\u001b\\\u001b[2m137\u001b[0m\u001b]8;;\u001b\\\n",
              "\u001b[2;36m           \u001b[0m         \u001b[32m\"data/mt-bench/score/Qwen2.5-0.5B-Instruct/claude-v1.json\"\u001b[0m                         \u001b[2m            \u001b[0m\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Score of <span style=\"color: #008000; text-decoration-color: #008000\">\"claude-v1\"</span> output to                                                     <a href=\"file:///content/RocketEval-ICLR/src/rocketeval/process/score.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">score.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///content/RocketEval-ICLR/src/rocketeval/process/score.py#137\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">137</span></a>\n",
              "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span>         <span style=\"color: #008000; text-decoration-color: #008000\">\"data/mt-bench/score/Qwen2.5-0.5B-Instruct/claude-v1.json\"</span>                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">            </span>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[2;36m          \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Score of \u001b[32m\"claude-instant-v1\"\u001b[0m output to                                             \u001b]8;id=849850;file:///content/RocketEval-ICLR/src/rocketeval/process/score.py\u001b\\\u001b[2mscore.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=880962;file:///content/RocketEval-ICLR/src/rocketeval/process/score.py#137\u001b\\\u001b[2m137\u001b[0m\u001b]8;;\u001b\\\n",
              "\u001b[2;36m           \u001b[0m         \u001b[32m\"data/mt-bench/score/Qwen2.5-0.5B-Instruct/claude-instant-v1.json\"\u001b[0m                 \u001b[2m            \u001b[0m\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Score of <span style=\"color: #008000; text-decoration-color: #008000\">\"claude-instant-v1\"</span> output to                                             <a href=\"file:///content/RocketEval-ICLR/src/rocketeval/process/score.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">score.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///content/RocketEval-ICLR/src/rocketeval/process/score.py#137\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">137</span></a>\n",
              "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span>         <span style=\"color: #008000; text-decoration-color: #008000\">\"data/mt-bench/score/Qwen2.5-0.5B-Instruct/claude-instant-v1.json\"</span>                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">            </span>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[2;36m          \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Score of \u001b[32m\"vicuna-33b-v1.3\"\u001b[0m output to                                               \u001b]8;id=638501;file:///content/RocketEval-ICLR/src/rocketeval/process/score.py\u001b\\\u001b[2mscore.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=564381;file:///content/RocketEval-ICLR/src/rocketeval/process/score.py#137\u001b\\\u001b[2m137\u001b[0m\u001b]8;;\u001b\\\n",
              "\u001b[2;36m           \u001b[0m         \u001b[32m\"data/mt-bench/score/Qwen2.5-0.5B-Instruct/vicuna-33b-v1.3.json\"\u001b[0m                   \u001b[2m            \u001b[0m\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Score of <span style=\"color: #008000; text-decoration-color: #008000\">\"vicuna-33b-v1.3\"</span> output to                                               <a href=\"file:///content/RocketEval-ICLR/src/rocketeval/process/score.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">score.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///content/RocketEval-ICLR/src/rocketeval/process/score.py#137\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">137</span></a>\n",
              "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span>         <span style=\"color: #008000; text-decoration-color: #008000\">\"data/mt-bench/score/Qwen2.5-0.5B-Instruct/vicuna-33b-v1.3.json\"</span>                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">            </span>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[2;36m          \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Score of \u001b[32m\"vicuna-13b-v1.2\"\u001b[0m output to                                               \u001b]8;id=471920;file:///content/RocketEval-ICLR/src/rocketeval/process/score.py\u001b\\\u001b[2mscore.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=288338;file:///content/RocketEval-ICLR/src/rocketeval/process/score.py#137\u001b\\\u001b[2m137\u001b[0m\u001b]8;;\u001b\\\n",
              "\u001b[2;36m           \u001b[0m         \u001b[32m\"data/mt-bench/score/Qwen2.5-0.5B-Instruct/vicuna-13b-v1.2.json\"\u001b[0m                   \u001b[2m            \u001b[0m\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Score of <span style=\"color: #008000; text-decoration-color: #008000\">\"vicuna-13b-v1.2\"</span> output to                                               <a href=\"file:///content/RocketEval-ICLR/src/rocketeval/process/score.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">score.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///content/RocketEval-ICLR/src/rocketeval/process/score.py#137\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">137</span></a>\n",
              "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span>         <span style=\"color: #008000; text-decoration-color: #008000\">\"data/mt-bench/score/Qwen2.5-0.5B-Instruct/vicuna-13b-v1.2.json\"</span>                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">            </span>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\r100%|██████████| 10/10 [00:00<00:00, 202.24it/s]\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[2;36m          \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m \u001b[33mScore Creation completed.\u001b[0m                                    \u001b]8;id=726514;file://<ipython-input-4-049d212b5d3f>\u001b\\\u001b[2m<ipython-input-4-049d212b5d3f>\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=638070;file://<ipython-input-4-049d212b5d3f>#155\u001b\\\u001b[2m155\u001b[0m\u001b]8;;\u001b\\\n",
              "\u001b[2;36m           \u001b[0m                                                                      \u001b[2m                                  \u001b[0m\n",
              "\u001b[2;36m           \u001b[0m                                                                      \u001b[2m                                  \u001b[0m\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> <span style=\"color: #808000; text-decoration-color: #808000\">Score Creation completed.</span>                                    <a href=\"file://<ipython-input-4-049d212b5d3f>\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">&lt;ipython-input-4-049d212b5d3f&gt;</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file://<ipython-input-4-049d212b5d3f>#155\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">155</span></a>\n",
              "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span>                                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                                  </span>\n",
              "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span>                                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                                  </span>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[2;36m          \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m \u001b[1;5;33;41mIV. Ranking\u001b[0m                                                  \u001b]8;id=464898;file://<ipython-input-4-049d212b5d3f>\u001b\\\u001b[2m<ipython-input-4-049d212b5d3f>\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=664904;file://<ipython-input-4-049d212b5d3f>#162\u001b\\\u001b[2m162\u001b[0m\u001b]8;;\u001b\\\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> <span style=\"color: #808000; text-decoration-color: #808000; background-color: #800000; font-weight: bold\">IV. Ranking</span>                                                  <a href=\"file://<ipython-input-4-049d212b5d3f>\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">&lt;ipython-input-4-049d212b5d3f&gt;</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file://<ipython-input-4-049d212b5d3f>#162\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">162</span></a>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[2;36m          \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m Ranking output to \u001b[32m\"data/ranking/.tsv\"\u001b[0m                                                \u001b]8;id=281546;file:///content/RocketEval-ICLR/src/rocketeval/process/rank.py\u001b\\\u001b[2mrank.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=9285;file:///content/RocketEval-ICLR/src/rocketeval/process/rank.py#38\u001b\\\u001b[2m38\u001b[0m\u001b]8;;\u001b\\\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> Ranking output to <span style=\"color: #008000; text-decoration-color: #008000\">\"data/ranking/.tsv\"</span>                                                <a href=\"file:///content/RocketEval-ICLR/src/rocketeval/process/rank.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">rank.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///content/RocketEval-ICLR/src/rocketeval/process/rank.py#38\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">38</span></a>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "                                                                                                                   \n",
              "                                                                                                                   \n",
              "                             \u001b[1m \u001b[0m\u001b[1mRank\u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1mmodel_test              \u001b[0m\u001b[1m \u001b[0m \u001b[1m \u001b[0m\u001b[1mQwen2.5-0.5B-Instruct\u001b[0m\u001b[1m \u001b[0m                             \n",
              "                             ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━                             \n",
              "                                 1   gpt-4o-mini                              6.57089                              \n",
              "                                 2   yi-large-preview                         6.26348                              \n",
              "                                 3   Yi-1.5-34B-Chat                          6.24259                              \n",
              "                                 4   Meta-Llama3-70B-Instruct                 6.23701                              \n",
              "                                 5   gpt-4                                    6.17788                              \n",
              "                                 6   Meta-Llama-3-8B-Instruct                 5.91072                              \n",
              "                                 7   vicuna-33b-v1.3                          5.87385                              \n",
              "                                 8   claude-instant-v1                        5.79136                              \n",
              "                                 9   claude-v1                                 5.7278                              \n",
              "                                10   vicuna-13b-v1.2                          5.40156                              \n",
              "                                                                                                                   \n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">                                                                                                                   \n",
              "                                                                                                                   \n",
              "                             <span style=\"font-weight: bold\"> Rank </span> <span style=\"font-weight: bold\"> model_test               </span> <span style=\"font-weight: bold\"> Qwen2.5-0.5B-Instruct </span>                             \n",
              "                             ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━                             \n",
              "                                 1   gpt-4o-mini                              6.57089                              \n",
              "                                 2   yi-large-preview                         6.26348                              \n",
              "                                 3   Yi-1.5-34B-Chat                          6.24259                              \n",
              "                                 4   Meta-Llama3-70B-Instruct                 6.23701                              \n",
              "                                 5   gpt-4                                    6.17788                              \n",
              "                                 6   Meta-Llama-3-8B-Instruct                 5.91072                              \n",
              "                                 7   vicuna-33b-v1.3                          5.87385                              \n",
              "                                 8   claude-instant-v1                        5.79136                              \n",
              "                                 9   claude-v1                                 5.7278                              \n",
              "                                10   vicuna-13b-v1.2                          5.40156                              \n",
              "                                                                                                                   \n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[2;36m          \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m \u001b[33mRanking completed.\u001b[0m                                           \u001b]8;id=734702;file://<ipython-input-4-049d212b5d3f>\u001b\\\u001b[2m<ipython-input-4-049d212b5d3f>\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=935911;file://<ipython-input-4-049d212b5d3f>#174\u001b\\\u001b[2m174\u001b[0m\u001b]8;;\u001b\\\n",
              "\u001b[2;36m           \u001b[0m                                                                      \u001b[2m                                  \u001b[0m\n",
              "\u001b[2;36m           \u001b[0m                                                                      \u001b[2m                                  \u001b[0m\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> <span style=\"color: #808000; text-decoration-color: #808000\">Ranking completed.</span>                                           <a href=\"file://<ipython-input-4-049d212b5d3f>\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">&lt;ipython-input-4-049d212b5d3f&gt;</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file://<ipython-input-4-049d212b5d3f>#174\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">174</span></a>\n",
              "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span>                                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                                  </span>\n",
              "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span>                                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                                  </span>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "\u001b[2;36m          \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m \u001b[1;5;33;41mRocketEval Completed\u001b[0m                                         \u001b]8;id=425564;file://<ipython-input-4-049d212b5d3f>\u001b\\\u001b[2m<ipython-input-4-049d212b5d3f>\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=635270;file://<ipython-input-4-049d212b5d3f>#181\u001b\\\u001b[2m181\u001b[0m\u001b]8;;\u001b\\\n"
            ],
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">           </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> <span style=\"color: #808000; text-decoration-color: #808000; background-color: #800000; font-weight: bold\">RocketEval Completed</span>                                         <a href=\"file://<ipython-input-4-049d212b5d3f>\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">&lt;ipython-input-4-049d212b5d3f&gt;</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file://<ipython-input-4-049d212b5d3f>#181\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">181</span></a>\n",
              "</pre>\n"
            ]
          },
          "metadata": {}
        }
      ],
      "source": [
        "#@title Running Evaluation Task\n",
        "\n",
        "import sys\n",
        "import os\n",
        "\n",
        "rocketeval_dir = os.path.join(os.path.abspath(os.curdir), \"src\")\n",
        "sys.path.insert(0, rocketeval_dir)\n",
        "\n",
        "import time\n",
        "import openai\n",
        "import logging\n",
        "import argparse\n",
        "\n",
        "from rich.logging import RichHandler\n",
        "from rich.console import Console\n",
        "from rich.markdown import Markdown\n",
        "\n",
        "from rocketeval.data.data_loader import load_target_models\n",
        "from rocketeval.task import checklist_task, judgment_task, ranking_task, score_task\n",
        "\n",
        "logging.getLogger('RootLogger').setLevel(logging.INFO)\n",
        "\n",
        "logging.basicConfig(\n",
        "    level=logging.INFO,\n",
        "    format=\"%(message)s\",\n",
        "    datefmt=\"[%X]\",\n",
        "    handlers=[RichHandler()],\n",
        "    force=True\n",
        ")\n",
        "\n",
        "parser = argparse.ArgumentParser(description=\"RocketEval Task Runner\")\n",
        "\n",
        "# Data\n",
        "parser.add_argument(\"--data_dir\", default=\"data/\", help=\"Data directory\")\n",
        "parser.add_argument(\"--config_dir\", default=\"config/\", help=\"Config directory\")\n",
        "\n",
        "# Model\n",
        "parser.add_argument(\"--dataset\", default=\"mt-bench\", help=\"Dataset name\")\n",
        "parser.add_argument(\"--generator\", default=\"gpt-4o\", help=\"Generator model\")\n",
        "parser.add_argument(\"--judge\", default=\"gpt-4o\", help=\"Judge model\")\n",
        "parser.add_argument(\"--labeler\", default=\"gpt-4o\", help=\"Labeler judge that provides labels\")\n",
        "parser.add_argument(\"--train_test\", action=\"store_true\", help=\"Use specific train-test split\")\n",
        "parser.add_argument(\"--gen_checklist\", action=\"store_true\", help=\"Generate checklist\")\n",
        "\n",
        "# Running Mode\n",
        "parser.add_argument(\"--mode\", choices=[\"api\", \"offline\"], help=\"Running mode, set to 'api' to use OpenAI API, set to 'offline' to use local models through vLLM\")\n",
        "parser.add_argument(\"--instant_api\", action=\"store_true\", help=\"Run using instant API.\")\n",
        "parser.add_argument(\"--api_parallel_size\", default=1, help=\"Number of parallel API calls, adjust based on your API rate limit.\")\n",
        "parser.add_argument(\"--offline_config\", default=\"config/offline/default.yaml\", help=\"Path to vLLM config file\")\n",
        "\n",
        "# Others\n",
        "parser.add_argument(\"--resume_from_task_id\", default=None, help=\"Task ID\")\n",
        "parser.add_argument(\"--keep_batch_files\", action=\"store_true\", help=\"Keep batch files\")\n",
        "parser.add_argument(\"--gpu_ids\", default=\"0\", help=\"GPU IDs, split by comma\")\n",
        "\n",
        "args = parser.parse_args(default_args)\n",
        "kwargs = vars(args)\n",
        "\n",
        "task_id = f\"{args.dataset}_{int(time.time())}\" \\\n",
        "    if args.resume_from_task_id is None \\\n",
        "    else args.resume_from_task_id\n",
        "\n",
        "if args.mode == \"api\":\n",
        "    client = openai.OpenAI()\n",
        "else:\n",
        "    client = None\n",
        "    if not os.path.exists(os.path.join(args.data_dir, \"batch\")):\n",
        "      os.makedirs(os.path.join(args.data_dir, \"batch\"))\n",
        "\n",
        "task_id = f\"{args.dataset}_{args.judge}_{int(time.time())}\" \\\n",
        "    if args.resume_from_task_id is None \\\n",
        "    else args.resume_from_task_id\n",
        "\n",
        "train_model_names = load_target_models(\n",
        "    data_dir=args.data_dir,\n",
        "    config_dir=args.config_dir,\n",
        "    dataset_name=args.dataset,\n",
        "    split=\"train\" if args.train_test else \"full\"\n",
        ")\n",
        "\n",
        "test_model_names = load_target_models(\n",
        "    data_dir=args.data_dir,\n",
        "    config_dir=args.config_dir,\n",
        "    dataset_name=args.dataset,\n",
        "    split=\"test\" if args.train_test else \"full\"\n",
        ")\n",
        "\n",
        "logger = logging.getLogger(\"rich\")\n",
        "\n",
        "start_message = f\"\"\"[underline bold red on white blink]RocketEval[/]\n",
        "[bold yellow on red blink] Task Information[/]\n",
        "- Dataset: \"{args.dataset}\"\n",
        "- Judge: \"{args.judge}\"\n",
        "- Labeler: \"{args.labeler}\"\n",
        "- Task ID: \"{task_id}\"\n",
        "\"\"\".replace(\"\\t\", \"\")\n",
        "\n",
        "logger.info(start_message, extra={\"markup\": True})\n",
        "\n",
        "\n",
        "logger.info(f\"[bold yellow on red blink]RocketEval Completed[/]\", extra={\"markup\": True})\n",
        "\n",
        "if args.gen_checklist:\n",
        "    # I - Checklist Creation\n",
        "    logger.info(\n",
        "        \"[bold yellow on red blink]I. Checklist Creation[/]\", extra={\"markup\": True}\n",
        "    )\n",
        "\n",
        "    checklist_task(\n",
        "        client=client,\n",
        "        task_id=task_id,\n",
        "        **kwargs\n",
        "    )\n",
        "\n",
        "    logger.info(\n",
        "        f\"[yellow]Checklist Creation completed.[/]\\n\\n\",\n",
        "        extra={\"markup\": True}\n",
        "    )\n",
        "else:\n",
        "    logger.info(\n",
        "        f\"[bold yellow on red blink]Checklist Creation skipped.[/]\", extra={\"markup\": True},\n",
        "    )\n",
        "\n",
        "# II - Judgment Creation\n",
        "logger.info(\n",
        "    \"[bold yellow on red blink]II. Judgment Creation[/]\", extra={\"markup\": True}\n",
        ")\n",
        "\n",
        "judgment_task(\n",
        "    model_names=train_model_names + test_model_names,\n",
        "    client=client,\n",
        "    task_id=task_id,\n",
        "    **kwargs\n",
        ")\n",
        "\n",
        "logger.info(\n",
        "    f\"[yellow]Judgment Creation completed.[/]\\n\\n\",\n",
        "    extra={\"markup\": True}\n",
        ")\n",
        "\n",
        "\n",
        "# III - Score Creation\n",
        "logger.info(\n",
        "    f\"[bold yellow on red blink]III. Score Creation[/]\",\n",
        "    extra={\"markup\": True}\n",
        ")\n",
        "\n",
        "score_task(\n",
        "    train_model_names=train_model_names,\n",
        "    test_model_names=test_model_names,\n",
        "    task_id=task_id,\n",
        "    **kwargs\n",
        ")\n",
        "\n",
        "logger.info(\n",
        "    f\"[yellow]Score Creation completed.[/]\\n\\n\",\n",
        "    extra={\"markup\": True}\n",
        ")\n",
        "\n",
        "\n",
        "# IV - Ranking\n",
        "logger.info(\n",
        "    f\"[bold yellow on red blink]IV. Ranking[/]\",\n",
        "    extra={\"markup\": True}\n",
        ")\n",
        "\n",
        "ranking = ranking_task(\n",
        "    model_names=test_model_names,\n",
        "    **kwargs\n",
        ")\n",
        "\n",
        "Console().print(Markdown(ranking.to_markdown()), justify=\"center\")\n",
        "\n",
        "logger.info(\n",
        "    f\"[yellow]Ranking completed.[/]\\n\\n\",\n",
        "    extra={\"markup\": True}\n",
        ")\n",
        "\n",
        "\n",
        "# Finish\n",
        "logger.info(\n",
        "    f\"[bold yellow on red blink]RocketEval Completed[/]\",\n",
        "    extra={\"markup\": True}\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "etZHRUBbq9IT"
      },
      "source": [
        "------\n",
        "\n",
        "After deriving the scores and rankings, you can also export the data to the [LMSYS Chatbot Arena](https://lmarena.ai/) format for further analysis using the official [notebook](https://colab.research.google.com/drive/1KdwokPjirkTmpO_P1WByFNFiqxWQquwH)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "mTIVdMrqq9IT",
        "outputId": "022449e8-2c64-4e3b-e4ab-b825b6564dac",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 380
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "           model_a                   model_b   winner                  judge  \\\n",
              "0      gpt-4o-mini          yi-large-preview  model_b  Qwen2.5-0.5B-Instruct   \n",
              "1      gpt-4o-mini  Meta-Llama3-70B-Instruct      tie  Qwen2.5-0.5B-Instruct   \n",
              "2            gpt-4               gpt-4o-mini  model_b  Qwen2.5-0.5B-Instruct   \n",
              "3  Yi-1.5-34B-Chat               gpt-4o-mini  model_a  Qwen2.5-0.5B-Instruct   \n",
              "4      gpt-4o-mini  Meta-Llama-3-8B-Instruct  model_b  Qwen2.5-0.5B-Instruct   \n",
              "\n",
              "   turn  anony language  tstamp  \\\n",
              "0     1   True  English  580830   \n",
              "1     1   True  English  580238   \n",
              "2     1   True  English  525352   \n",
              "3     1   True  English  250561   \n",
              "4     1   True  English   61569   \n",
              "\n",
              "                                       conv_metadata  is_code  is_refusal  \\\n",
              "0  {'sum_user_tokens': 8, 'sum_assistant_a_tokens...    False       False   \n",
              "1  {'sum_user_tokens': 8, 'sum_assistant_a_tokens...    False       False   \n",
              "2  {'sum_user_tokens': 8, 'sum_assistant_a_tokens...    False       False   \n",
              "3  {'sum_user_tokens': 8, 'sum_assistant_a_tokens...    False       False   \n",
              "4  {'sum_user_tokens': 8, 'sum_assistant_a_tokens...    False       False   \n",
              "\n",
              "                               dedup_tag  \\\n",
              "0  {'high_freq': False, 'sampled': True}   \n",
              "1  {'high_freq': False, 'sampled': True}   \n",
              "2  {'high_freq': False, 'sampled': True}   \n",
              "3  {'high_freq': False, 'sampled': True}   \n",
              "4  {'high_freq': False, 'sampled': True}   \n",
              "\n",
              "                                        category_tag  \n",
              "0  {'if_v0.1': {'if': True, 'score': 4}, 'math_v0...  \n",
              "1  {'if_v0.1': {'if': True, 'score': 4}, 'math_v0...  \n",
              "2  {'if_v0.1': {'if': True, 'score': 4}, 'math_v0...  \n",
              "3  {'if_v0.1': {'if': True, 'score': 4}, 'math_v0...  \n",
              "4  {'if_v0.1': {'if': True, 'score': 4}, 'math_v0...  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-a74ed6ad-a21e-47e0-ba71-613ca738f314\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>model_a</th>\n",
              "      <th>model_b</th>\n",
              "      <th>winner</th>\n",
              "      <th>judge</th>\n",
              "      <th>turn</th>\n",
              "      <th>anony</th>\n",
              "      <th>language</th>\n",
              "      <th>tstamp</th>\n",
              "      <th>conv_metadata</th>\n",
              "      <th>is_code</th>\n",
              "      <th>is_refusal</th>\n",
              "      <th>dedup_tag</th>\n",
              "      <th>category_tag</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>gpt-4o-mini</td>\n",
              "      <td>yi-large-preview</td>\n",
              "      <td>model_b</td>\n",
              "      <td>Qwen2.5-0.5B-Instruct</td>\n",
              "      <td>1</td>\n",
              "      <td>True</td>\n",
              "      <td>English</td>\n",
              "      <td>580830</td>\n",
              "      <td>{'sum_user_tokens': 8, 'sum_assistant_a_tokens...</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>{'high_freq': False, 'sampled': True}</td>\n",
              "      <td>{'if_v0.1': {'if': True, 'score': 4}, 'math_v0...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>gpt-4o-mini</td>\n",
              "      <td>Meta-Llama3-70B-Instruct</td>\n",
              "      <td>tie</td>\n",
              "      <td>Qwen2.5-0.5B-Instruct</td>\n",
              "      <td>1</td>\n",
              "      <td>True</td>\n",
              "      <td>English</td>\n",
              "      <td>580238</td>\n",
              "      <td>{'sum_user_tokens': 8, 'sum_assistant_a_tokens...</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>{'high_freq': False, 'sampled': True}</td>\n",
              "      <td>{'if_v0.1': {'if': True, 'score': 4}, 'math_v0...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>gpt-4</td>\n",
              "      <td>gpt-4o-mini</td>\n",
              "      <td>model_b</td>\n",
              "      <td>Qwen2.5-0.5B-Instruct</td>\n",
              "      <td>1</td>\n",
              "      <td>True</td>\n",
              "      <td>English</td>\n",
              "      <td>525352</td>\n",
              "      <td>{'sum_user_tokens': 8, 'sum_assistant_a_tokens...</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>{'high_freq': False, 'sampled': True}</td>\n",
              "      <td>{'if_v0.1': {'if': True, 'score': 4}, 'math_v0...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Yi-1.5-34B-Chat</td>\n",
              "      <td>gpt-4o-mini</td>\n",
              "      <td>model_a</td>\n",
              "      <td>Qwen2.5-0.5B-Instruct</td>\n",
              "      <td>1</td>\n",
              "      <td>True</td>\n",
              "      <td>English</td>\n",
              "      <td>250561</td>\n",
              "      <td>{'sum_user_tokens': 8, 'sum_assistant_a_tokens...</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>{'high_freq': False, 'sampled': True}</td>\n",
              "      <td>{'if_v0.1': {'if': True, 'score': 4}, 'math_v0...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>gpt-4o-mini</td>\n",
              "      <td>Meta-Llama-3-8B-Instruct</td>\n",
              "      <td>model_b</td>\n",
              "      <td>Qwen2.5-0.5B-Instruct</td>\n",
              "      <td>1</td>\n",
              "      <td>True</td>\n",
              "      <td>English</td>\n",
              "      <td>61569</td>\n",
              "      <td>{'sum_user_tokens': 8, 'sum_assistant_a_tokens...</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>{'high_freq': False, 'sampled': True}</td>\n",
              "      <td>{'if_v0.1': {'if': True, 'score': 4}, 'math_v0...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a74ed6ad-a21e-47e0-ba71-613ca738f314')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-a74ed6ad-a21e-47e0-ba71-613ca738f314 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-a74ed6ad-a21e-47e0-ba71-613ca738f314');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-0ccc6a6b-14bf-499e-9c17-041be6a3fcfe\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-0ccc6a6b-14bf-499e-9c17-041be6a3fcfe')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-0ccc6a6b-14bf-499e-9c17-041be6a3fcfe button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "result",
              "summary": "{\n  \"name\": \"result\",\n  \"rows\": 7200,\n  \"fields\": [\n    {\n      \"column\": \"model_a\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 10,\n        \"samples\": [\n          \"Meta-Llama3-70B-Instruct\",\n          \"gpt-4\",\n          \"vicuna-13b-v1.2\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"model_b\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 10,\n        \"samples\": [\n          \"vicuna-13b-v1.2\",\n          \"Meta-Llama3-70B-Instruct\",\n          \"gpt-4\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"winner\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"model_b\",\n          \"tie\",\n          \"model_a\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"judge\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"Qwen2.5-0.5B-Instruct\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"turn\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 1,\n        \"max\": 1,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          1\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"anony\",\n      \"properties\": {\n        \"dtype\": \"boolean\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          true\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"language\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"English\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"tstamp\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 287125,\n        \"min\": 115,\n        \"max\": 999582,\n        \"num_unique_values\": 7175,\n        \"samples\": [\n          318974\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"conv_metadata\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"is_code\",\n      \"properties\": {\n        \"dtype\": \"boolean\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          false\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"is_refusal\",\n      \"properties\": {\n        \"dtype\": \"boolean\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          false\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"dedup_tag\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"category_tag\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 5
        }
      ],
      "source": [
        "from rocketeval.tools.export import chatbot_arena_match\n",
        "from rocketeval.data.data_loader import load_target_models\n",
        "\n",
        "load_target_models(dataset_name=\"mt-bench\", split=\"test\")\n",
        "result = chatbot_arena_match(dataset_name=\"mt-bench\", judge=\"Qwen2.5-0.5B-Instruct\", model_names=test_model_names)\n",
        "result.to_json(\"matches.jsonl\", orient=\"records\", lines=True)\n",
        "\n",
        "result.head(5)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YOYXJGx7zAqW"
      },
      "source": [
        "You can try on more powerful judge models and increase the number of test models to get a better and comprehensive evaluation result.\n",
        "\n",
        "------\n",
        "\n",
        "If you find this work useful in your research, please consider citing the following paper:\n",
        "```bibtex\n",
        "@inproceedings{wei2025rocketeval,\n",
        "    title={RocketEval: Efficient automated {LLM} evaluation via grading checklist},\n",
        "    author={Tianjun Wei and Wei Wen and Ruizhi Qiao and Xing Sun and Jianghong Ma},\n",
        "    booktitle={The Thirteenth International Conference on Learning Representations},\n",
        "    year={2025},\n",
        "    url={https://openreview.net/forum?id=zJjzNj6QUe}\n",
        "}\n",
        "```"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "f0ca97397a7d414681137b3631195323": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_ca10621991344a54b954d33896490c4e",
              "IPY_MODEL_7e516071ac6543c09fe0d03da38dca27",
              "IPY_MODEL_1d8267cfa61b4f4581297e48fd3d0f40"
            ],
            "layout": "IPY_MODEL_10ac5ec7cfc84574911155d9a91dac08"
          }
        },
        "ca10621991344a54b954d33896490c4e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1d075309fb4249739937263a4891d786",
            "placeholder": "​",
            "style": "IPY_MODEL_2b15babdc8b845ef87d6a0a2f6eddfa1",
            "value": "config.json: 100%"
          }
        },
        "7e516071ac6543c09fe0d03da38dca27": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_fa6df7a068aa4f22999c870eb8dbe239",
            "max": 659,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_c308c2961efa49aeb4c4fce5a8e31417",
            "value": 659
          }
        },
        "1d8267cfa61b4f4581297e48fd3d0f40": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d91e437520e9463c87e37d536a84863f",
            "placeholder": "​",
            "style": "IPY_MODEL_e34e3ef6cb584031bba292d3984518f1",
            "value": " 659/659 [00:00&lt;00:00, 16.2kB/s]"
          }
        },
        "10ac5ec7cfc84574911155d9a91dac08": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1d075309fb4249739937263a4891d786": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2b15babdc8b845ef87d6a0a2f6eddfa1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "fa6df7a068aa4f22999c870eb8dbe239": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c308c2961efa49aeb4c4fce5a8e31417": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "d91e437520e9463c87e37d536a84863f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e34e3ef6cb584031bba292d3984518f1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "7a152b2cc0b7451a809008aa2c48e917": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_887cb4fc907e4dbcbe83832ca8b9fec2",
              "IPY_MODEL_893282ec30034f6180fefd69607676a5",
              "IPY_MODEL_dbbf390290254a59867aa99a37b264f9"
            ],
            "layout": "IPY_MODEL_a63630c39e3343e8beb28628fb13776f"
          }
        },
        "887cb4fc907e4dbcbe83832ca8b9fec2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_af5d28944f70409e8007250f47fd7a3c",
            "placeholder": "​",
            "style": "IPY_MODEL_4077c004e3ec488ab059cb6356c2c0dc",
            "value": "tokenizer_config.json: 100%"
          }
        },
        "893282ec30034f6180fefd69607676a5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2c8276ba20714bd2b074bba6c3764233",
            "max": 7305,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_9a4ff6c6c2f4480899bd8e4ec10e9e1d",
            "value": 7305
          }
        },
        "dbbf390290254a59867aa99a37b264f9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_4c586706c1d647db94830113079fa3cb",
            "placeholder": "​",
            "style": "IPY_MODEL_31b2ac73b1b74e96a5604e6ecb265c74",
            "value": " 7.30k/7.30k [00:00&lt;00:00, 220kB/s]"
          }
        },
        "a63630c39e3343e8beb28628fb13776f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "af5d28944f70409e8007250f47fd7a3c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4077c004e3ec488ab059cb6356c2c0dc": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "2c8276ba20714bd2b074bba6c3764233": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9a4ff6c6c2f4480899bd8e4ec10e9e1d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "4c586706c1d647db94830113079fa3cb": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "31b2ac73b1b74e96a5604e6ecb265c74": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "3e61fab06c3c489d8298fdbb429c44fe": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_0c2c03b019ab42579aa7e8a246825e3c",
              "IPY_MODEL_2691cda4645d4d38bbbfb3db3f4d66b5",
              "IPY_MODEL_cad0b22f16b248d193563dd1700e6593"
            ],
            "layout": "IPY_MODEL_fed97b4f8c9641f284cc2cf74250f650"
          }
        },
        "0c2c03b019ab42579aa7e8a246825e3c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_72e43f9a4caf44d4a133836616970c35",
            "placeholder": "​",
            "style": "IPY_MODEL_96ea0d6d4bf3420989e082ebaedb510a",
            "value": "vocab.json: 100%"
          }
        },
        "2691cda4645d4d38bbbfb3db3f4d66b5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_57d514c2c51b400da0cc090a3a840712",
            "max": 2776833,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_1c6075cfb5044872a8fde4a8a45ca9b5",
            "value": 2776833
          }
        },
        "cad0b22f16b248d193563dd1700e6593": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1486d20ba48b4f30b2f0dd4ed3692a23",
            "placeholder": "​",
            "style": "IPY_MODEL_183d20b14ae14023bfe7d8c0bc06157c",
            "value": " 2.78M/2.78M [00:00&lt;00:00, 6.80MB/s]"
          }
        },
        "fed97b4f8c9641f284cc2cf74250f650": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "72e43f9a4caf44d4a133836616970c35": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "96ea0d6d4bf3420989e082ebaedb510a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "57d514c2c51b400da0cc090a3a840712": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1c6075cfb5044872a8fde4a8a45ca9b5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "1486d20ba48b4f30b2f0dd4ed3692a23": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "183d20b14ae14023bfe7d8c0bc06157c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "9c84ef32a5d444f2b209e4687aaf0d4a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_3015a163001e4325871f1a2e96c7ccaa",
              "IPY_MODEL_d35c4ac37946451d967d6c29a53304ed",
              "IPY_MODEL_c02cdfbd429f407586cbdd9850455241"
            ],
            "layout": "IPY_MODEL_bc3bdda8cb734eab87622c79953ad233"
          }
        },
        "3015a163001e4325871f1a2e96c7ccaa": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_acc07336d9554e2b910add79fed8ff7b",
            "placeholder": "​",
            "style": "IPY_MODEL_eb2d22999840434eac2918991d8fea57",
            "value": "merges.txt: 100%"
          }
        },
        "d35c4ac37946451d967d6c29a53304ed": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6320e89229294ff0a6475ee62dc5c40e",
            "max": 1671839,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_16778992f3cd4ce6b1d8a4dcc9a47a4f",
            "value": 1671839
          }
        },
        "c02cdfbd429f407586cbdd9850455241": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_88833a49f7ab488a95d906a93413cf8f",
            "placeholder": "​",
            "style": "IPY_MODEL_e68985d26aa549569b458e41fa368ba5",
            "value": " 1.67M/1.67M [00:00&lt;00:00, 31.5MB/s]"
          }
        },
        "bc3bdda8cb734eab87622c79953ad233": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "acc07336d9554e2b910add79fed8ff7b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "eb2d22999840434eac2918991d8fea57": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "6320e89229294ff0a6475ee62dc5c40e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "16778992f3cd4ce6b1d8a4dcc9a47a4f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "88833a49f7ab488a95d906a93413cf8f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e68985d26aa549569b458e41fa368ba5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "9881d7b4252747db8687dff9ca922f34": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_4d1914b6b0ad4abd8eb7a640cd18a9cb",
              "IPY_MODEL_319439bf8ad24ce5b041c8b1310af37b",
              "IPY_MODEL_6e273761535d45a590c8d32b7e2aa759"
            ],
            "layout": "IPY_MODEL_5a023d3d765a4a37b30d17afcecd6d2c"
          }
        },
        "4d1914b6b0ad4abd8eb7a640cd18a9cb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_609cbcf1563343609476fa287af7fa5a",
            "placeholder": "​",
            "style": "IPY_MODEL_dc1358d5ee664badb20ba37ed0df01f0",
            "value": "tokenizer.json: 100%"
          }
        },
        "319439bf8ad24ce5b041c8b1310af37b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_9d861516c20548e3a1c703298e6aea0d",
            "max": 7031645,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_cc6281b16cc9498cbb42aa1ad48ba22b",
            "value": 7031645
          }
        },
        "6e273761535d45a590c8d32b7e2aa759": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_dd705f26a0b64e78820eefbd1fb052da",
            "placeholder": "​",
            "style": "IPY_MODEL_4d78f541b9e545cc9e00975c7ffe1dc3",
            "value": " 7.03M/7.03M [00:00&lt;00:00, 14.0MB/s]"
          }
        },
        "5a023d3d765a4a37b30d17afcecd6d2c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "609cbcf1563343609476fa287af7fa5a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "dc1358d5ee664badb20ba37ed0df01f0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "9d861516c20548e3a1c703298e6aea0d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "cc6281b16cc9498cbb42aa1ad48ba22b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "dd705f26a0b64e78820eefbd1fb052da": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4d78f541b9e545cc9e00975c7ffe1dc3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "7583ff022e704544bdc77fe5f996a1ea": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_472294a12fc24aa28b60fa5c3668deaf",
              "IPY_MODEL_67de9d68a38141fba801cb1edbb69748",
              "IPY_MODEL_2f742a948cb94da98b76507cba0ae598"
            ],
            "layout": "IPY_MODEL_1d22a44e8d594b9e8d60e998285002b2"
          }
        },
        "472294a12fc24aa28b60fa5c3668deaf": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_342c5e84ea23441cb46c419f8489092e",
            "placeholder": "​",
            "style": "IPY_MODEL_aaabd83129ab42ef9277b1700a788dac",
            "value": "generation_config.json: 100%"
          }
        },
        "67de9d68a38141fba801cb1edbb69748": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f1bd85aea53f49dba6c208a4bef80a70",
            "max": 242,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_201c892fe4df466696123628f6b9b903",
            "value": 242
          }
        },
        "2f742a948cb94da98b76507cba0ae598": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_dc3c9494e5b0439aacd098e9d4c56928",
            "placeholder": "​",
            "style": "IPY_MODEL_0360055205374b6aa35e93256dc0c08c",
            "value": " 242/242 [00:00&lt;00:00, 10.2kB/s]"
          }
        },
        "1d22a44e8d594b9e8d60e998285002b2": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "342c5e84ea23441cb46c419f8489092e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "aaabd83129ab42ef9277b1700a788dac": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "f1bd85aea53f49dba6c208a4bef80a70": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "201c892fe4df466696123628f6b9b903": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "dc3c9494e5b0439aacd098e9d4c56928": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0360055205374b6aa35e93256dc0c08c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c319df7b698d492db355666730d51a2a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_80769bc182804ab281687dfb42b7a6de",
              "IPY_MODEL_34631d319a3343e7940d969ccdfdf2ee",
              "IPY_MODEL_8ea79a10e5d54ac88004c8d6eb4c5924"
            ],
            "layout": "IPY_MODEL_70fb35b6842b480c8f150bd9f66bc44c"
          }
        },
        "80769bc182804ab281687dfb42b7a6de": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_38977214c3c247708294725881ddb057",
            "placeholder": "​",
            "style": "IPY_MODEL_a70f34b5e4314e828012ef8d8560c199",
            "value": "model.safetensors: 100%"
          }
        },
        "34631d319a3343e7940d969ccdfdf2ee": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3b7534af827d401f849dd657a21b27a2",
            "max": 988097824,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_dd4da084890040949b0676569db25a14",
            "value": 988097824
          }
        },
        "8ea79a10e5d54ac88004c8d6eb4c5924": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e71b2c79095f490c88bf05ba6a09250f",
            "placeholder": "​",
            "style": "IPY_MODEL_dd9defd662614a80a07ade0fb91180db",
            "value": " 988M/988M [00:23&lt;00:00, 41.9MB/s]"
          }
        },
        "70fb35b6842b480c8f150bd9f66bc44c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "38977214c3c247708294725881ddb057": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a70f34b5e4314e828012ef8d8560c199": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "3b7534af827d401f849dd657a21b27a2": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "dd4da084890040949b0676569db25a14": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "e71b2c79095f490c88bf05ba6a09250f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "dd9defd662614a80a07ade0fb91180db": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "f435ed80fb7d42fa8dfc16df2afa1916": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_17429b88a2fc49c58fa2807ede6c4927",
              "IPY_MODEL_e441522d4faa47278c6ab0e4f65388e3",
              "IPY_MODEL_dece1d256a5b4d5999cc7bc288c86a56"
            ],
            "layout": "IPY_MODEL_dbc8e87e1b314deda8702d24057cb645"
          }
        },
        "17429b88a2fc49c58fa2807ede6c4927": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_9751e5e354184c4487f41c20e5fedca0",
            "placeholder": "​",
            "style": "IPY_MODEL_947e308648434526a6bd12d4384aef95",
            "value": ""
          }
        },
        "e441522d4faa47278c6ab0e4f65388e3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b7f517668c1d46dd982b1feb2853524a",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_c9d1630fa487490ab59aeb8adfc1151c",
            "value": 1
          }
        },
        "dece1d256a5b4d5999cc7bc288c86a56": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c678d2f581c2402cb5a5a8828bb53fee",
            "placeholder": "​",
            "style": "IPY_MODEL_b736a7dc7bd84535acc9ef717a26096c",
            "value": "Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:01&lt;00:00,  1.10s/it]\n"
          }
        },
        "dbc8e87e1b314deda8702d24057cb645": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9751e5e354184c4487f41c20e5fedca0": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "947e308648434526a6bd12d4384aef95": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "b7f517668c1d46dd982b1feb2853524a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c9d1630fa487490ab59aeb8adfc1151c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "c678d2f581c2402cb5a5a8828bb53fee": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b736a7dc7bd84535acc9ef717a26096c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}