diff --git a/tamingllms/_build/.doctrees/environment.pickle b/tamingllms/_build/.doctrees/environment.pickle index 2533cc4..bec7582 100644 Binary files a/tamingllms/_build/.doctrees/environment.pickle and b/tamingllms/_build/.doctrees/environment.pickle differ diff --git a/tamingllms/_build/.doctrees/markdown/preface.doctree b/tamingllms/_build/.doctrees/markdown/preface.doctree index 030529f..93340fc 100644 Binary files a/tamingllms/_build/.doctrees/markdown/preface.doctree and b/tamingllms/_build/.doctrees/markdown/preface.doctree differ diff --git a/tamingllms/_build/.doctrees/notebooks/alignment.doctree b/tamingllms/_build/.doctrees/notebooks/alignment.doctree index bb9eb50..7669eb0 100644 Binary files a/tamingllms/_build/.doctrees/notebooks/alignment.doctree and b/tamingllms/_build/.doctrees/notebooks/alignment.doctree differ diff --git a/tamingllms/_build/.doctrees/notebooks/evals.doctree b/tamingllms/_build/.doctrees/notebooks/evals.doctree index b0a2a3f..32edb4c 100644 Binary files a/tamingllms/_build/.doctrees/notebooks/evals.doctree and b/tamingllms/_build/.doctrees/notebooks/evals.doctree differ diff --git a/tamingllms/_build/.doctrees/notebooks/local.doctree b/tamingllms/_build/.doctrees/notebooks/local.doctree index 41b2d05..e71525c 100644 Binary files a/tamingllms/_build/.doctrees/notebooks/local.doctree and b/tamingllms/_build/.doctrees/notebooks/local.doctree differ diff --git a/tamingllms/_build/.doctrees/notebooks/output_size_limit.doctree b/tamingllms/_build/.doctrees/notebooks/output_size_limit.doctree index ba7240f..1bf6dae 100644 Binary files a/tamingllms/_build/.doctrees/notebooks/output_size_limit.doctree and b/tamingllms/_build/.doctrees/notebooks/output_size_limit.doctree differ diff --git a/tamingllms/_build/.doctrees/notebooks/safety.doctree b/tamingllms/_build/.doctrees/notebooks/safety.doctree index 4296df9..285e230 100644 Binary files a/tamingllms/_build/.doctrees/notebooks/safety.doctree and b/tamingllms/_build/.doctrees/notebooks/safety.doctree differ diff --git a/tamingllms/_build/.doctrees/notebooks/structured_output.doctree b/tamingllms/_build/.doctrees/notebooks/structured_output.doctree index 0fe48a8..cf32549 100644 Binary files a/tamingllms/_build/.doctrees/notebooks/structured_output.doctree and b/tamingllms/_build/.doctrees/notebooks/structured_output.doctree differ diff --git a/tamingllms/_build/html/_images/ppl1.png b/tamingllms/_build/html/_images/ppl1.png new file mode 100644 index 0000000..8579634 Binary files /dev/null and b/tamingllms/_build/html/_images/ppl1.png differ diff --git a/tamingllms/_build/html/_images/ppl2.png b/tamingllms/_build/html/_images/ppl2.png new file mode 100644 index 0000000..fe03bd9 Binary files /dev/null and b/tamingllms/_build/html/_images/ppl2.png differ diff --git a/tamingllms/_build/html/_sources/notebooks/evals.ipynb b/tamingllms/_build/html/_sources/notebooks/evals.ipynb index deb3e40..1727436 100644 --- a/tamingllms/_build/html/_sources/notebooks/evals.ipynb +++ b/tamingllms/_build/html/_sources/notebooks/evals.ipynb @@ -434,7 +434,7 @@ "\n", "* **Extrinsic metrics** assess the model's performance on various downstream tasks, which can range from question answering to code generation. These metrics are not directly tied to the training objective, but they provide valuable insights into the model's ability to generalize to real-world applications.\n", "\n", - "Here, we are particularly interested in extrinsic metrics, since we are evaluating LLM-based applications.\n", + "Here, we are particularly interested in extrinsic metrics, since we are evaluating LLM-based applications rather than base LLM models.\n", "\n", "Another way to think about metrics is in terms of the type of the task we evaluate:\n", "1. **Discriminative Task**:\n", diff --git a/tamingllms/_build/html/_sources/notebooks/local.ipynb b/tamingllms/_build/html/_sources/notebooks/local.ipynb index 0a38a10..b288046 100644 --- a/tamingllms/_build/html/_sources/notebooks/local.ipynb +++ b/tamingllms/_build/html/_sources/notebooks/local.ipynb @@ -6,9 +6,9 @@ "source": [ "# Breaking Free from Cloud Providers\n", "```{epigraph}\n", - "I want to break free, I want to break free\n", + "Freedom is something that dies unless it's used.\n", "\n", - "-- Queen\n", + "-- Hunter S. Thompson\n", "```\n", "```{contents}\n", "```\n", @@ -36,30 +36,15 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Local Models\n", - "\n", - "### Notes of Caution\n", - "\n", - "When using local LLM models versus widely known private large language models:\n", - "\n", - "1. Performance: Local LLMs often have lower performance compared to large private models due to size and training limitations.\n", - "\n", - "2. Resource requirements: Running local LLMs can be computationally intensive, requiring significant CPU/GPU resources.\n", - "\n", - "3. Limited capabilities: Local models particularly if smallmay struggle with complex tasks or specialized knowledge that larger models handle well.\n", - "\n", - "4. Potential Output Unreliability: Local models may produce less consistent or stable outputs - see Chapter\n", - "\n", - "5. Limited context window: Local models often have smaller context windows, limiting their ability to process long inputs.\n", - "\n", - "Always evaluate the trade-offs between using local LLMs and private models based on your specific use case and requirements. We highly recommend extensively testing your local LLM before productionizing an end-to-end application." + "## Local Models Considerations\n", + "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Local Tools\n", + "## Tools for Local LLM Deployment\n", "\n", "Local LLM deployment tools generally fall into two categories: inference-focused tools that prioritize performance and programmability for technical users requiring production-grade deployments, and user interface (UI) tools that emphasize accessibility through graphical interfaces for non-technical users, trading some performance for ease of use and broader adoption. In the following sections we will explore some of these tools discussing their features, capabilities, and trade-offs.\n" ] @@ -694,45 +679,266 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Case Study: Private Code Documentation Generator\n", + "## Case Study: The Effect of Quantization on LLM Performance\n", "\n", - "### Objectives\n", + "This case study examines how different quantization levels affect the performance of language models running locally. Quantization is a crucial technique for reducing model size and improving inference speed, but it comes with potential tradeoffs in model quality. Understanding these tradeoffs is essential for practitioners deploying LLMs in resource-constrained environments.\n", "\n", - "Private Code Documentation Generator\n", + "Using the Qwen 2.5 0.5B model as our baseline, we'll compare four variants:\n", + "- The base fp16 model (no quantization)\n", + "- Q2_K quantization (highest compression, lowest precision)\n", + "- Q4_K quantization (balanced compression/precision)\n", + "- Q6_K quantization (lowest compression, highest precision)\n", "\n", + "The analysis will focus on three key metrics:\n", + "1. Perplexity - to measure how well the model predicts text\n", + "2. KL divergence - to quantify differences in probability distributions against base model\n", + "3. Prompt (tokens/second) - to assess impact in thoughput\n", "\n", - "Create a system for automatically documenting proprietary codebase\n", - "LLM analyzes private source code repositories\n", - "Generates comprehensive documentation without exposing code externally\n", - "Could implement:\n", + "While we will focus on the Qwen 2.5 0.5B model, the same analysis can be applied to other models. These insights will help practitioners make informed decisions about quantization strategies based on their specific requirements for model size, speed, and accuracy." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Prompts Dataset" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "To evaluate the impact of quantization on model performance, we first need a set of prompts that will serve as input data for our experiments. We'll construct a dataset from WikiText-2 {cite}`salesforce2024wikitext`, which contains Wikipedia excerpts. \n", "\n", - "Automatic function documentation\n", - "Architecture diagrams based on code analysis\n", - "Security vulnerability scanning\n", - "API documentation generation\n", - "Code complexity analysis\n", - "Best practice suggestions\n", - "Integration with local git repositories\n", + "In our experiments, we will use a total of `NUM_PROMPTS` prompts that vary in length from `MIN_PROMPT_LENGTH` to `MAX_PROMPT_LENGTH` tokens. Using a fixed set of prompts ensures consistent evaluation across model variants and enables direct comparison of metrics like perplexity and throughput.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 59, + "metadata": {}, + "outputs": [], + "source": [ + "NUM_PROMPTS = 100\n", + "MIN_PROMPT_LENGTH = 100\n", + "MAX_PROMPT_LENGTH = 1000" + ] + }, + { + "cell_type": "code", + "execution_count": 46, + "metadata": {}, + "outputs": [], + "source": [ + "import datasets\n", + "input_texts_raw = datasets.load_dataset(\"Salesforce/wikitext\", \"wikitext-2-raw-v1\", split=\"train\")[\"text\"]\n" + ] + }, + { + "cell_type": "code", + "execution_count": 60, + "metadata": {}, + "outputs": [], + "source": [ + "input_texts = [s for s in input_texts_raw if s!='' and len(s) > MIN_PROMPT_LENGTH and len(s) < MAX_PROMPT_LENGTH][:NUM_PROMPTS]" + ] + }, + { + "cell_type": "code", + "execution_count": 61, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "100" + ] + }, + "execution_count": 61, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "len(input_texts)\n" + ] + }, + { + "cell_type": "code", + "execution_count": 62, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " The game began development in 2010 , carrying over a large portion of the work done on Valkyria Chronicles II . While it retained the standard features of the series , it also underwent multiple adjustments , such as making the game more forgiving for series newcomers . Character designer Raita Honjou and composer Hitoshi Sakimoto both returned from previous entries , along with Valkyria Chronicles II director Takeshi Ozawa . A large team of writers handled the script . The game 's opening theme was sung by May 'n . \n", + "\n" + ] + } + ], + "source": [ + "print(input_texts[1])" + ] + }, + { + "cell_type": "code", + "execution_count": 63, + "metadata": {}, + "outputs": [], + "source": [ + "with open('../data/local/prompts.txt', 'w') as f:\n", + " for text in input_texts:\n", + " # Escape any quotes in the text and wrap in quotes\n", + " escaped_text = text.replace('\"', '\\\\\"')\n", + " f.write(f'\"{escaped_text}\"\\n')\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Quantization\n", + "\n", + "We can quantize a model using the `llama-quantize` CLI. For instance, to quantize the Qwen 2.5 0.5B model to Q4_K, we can run the following command:\n", + "```bash\n", + "./llama-quantize -m ./models/qwen2.5-0.5b-instruct-fp16.gguf ./models/qwen2.5-0.5b-instruct-q8_0.gguf Q4_K\n", + "```\n", + "\n", + "{numref}`quantization-levels` describes the key quantization levels used in this study {cite}`huggingface2024quantization`, where:\n", + "- q is the quantized value\n", + "- block_scale is the scaling factor for the block (with bit width in parentheses)\n", + "- block_min is the block minimum value (with bit width in parentheses)\n", + "\n", + "```{table} Quantization Levels\n", + ":align: center\n", + ":name: quantization-levels\n", + "| Quantization | Description | Bits per Weight | Formula |\n", + "|--------------|-------------|-----------------|----------|\n", + "| Q2_K | 2-bit quantization with 16 weights per block in 16-block superblocks | 2.5625 | w = q * block_scale(4-bit) + block_min(4-bit) |\n", + "| Q4_K | 4-bit quantization with 32 weights per block in 8-block superblocks | 4.5 | w = q * block_scale(6-bit) + block_min(6-bit) |\n", + "| Q6_K | 6-bit quantization with 16 weights per block in 16-block superblocks | 6.5625 | w = q * block_scale(8-bit) |\n", + "```\n", + "\n", + "Each quantization level represents a different tradeoff between model size and accuracy. Q2_K provides the highest compression but potentially lower accuracy, while Q6_K maintains better accuracy at the cost of larger model size. The K-variants use more sophisticated block structures and scaling compared to legacy quantization methods.\n", + "\n", + "The base model is 16-bit standard IEEE 754 half-precision floating-point number.\n", "\n", - "### Model Setup\n", - "llamacpp on Qwen2.5-Coder-3B-Instruct-GGUF\n", + "### Benchmarking\n", "\n", - "https://huggingface.co/Qwen/Qwen2.5-Coder-3B-Instruct-GGUF\n", + "We will measure quantized model \"quality\" by means of perplexity and KL Divergence. For performance evaluation, we will report prompt throughput in tokens per second.\n", "\n", - "https://qwenlm.github.io/blog/qwen2.5-coder-family/\n", - "https://qwen.readthedocs.io/en/latest/benchmark/speed_benchmark.html\n", + "**Perplexity**\n", "\n", - "### Instructions\n", + "Perplexity is a common metric for evaluating language models that measures how well a model predicts a sample of text. Lower perplexity indicates better prediction (less \"perplexed\" by the text).\n", "\n", - "Generate natural language description\n", - "Extract key parameters and return values\n", - "Identify potential edge cases\n", - "Suggest improvements or potential issues\n", - "Generate usage examples\n", - "Create docstring templates\n", + "Recall that for a sequence of N tokens, perplexity is defined as:\n", + "\n", + "$$ \\text{PPL(B, X)} = \\exp\\left(-\\frac{1}{N}\\sum_{i=1}^{N} \\log_2 P(x_i|x_{ ../q2_kresults.txt\n", + "```\n", + "\n", + "We perform this process for each quantization level studied (Q2_K, Q4_K, Q6_K).\n", + "\n", + "\n", + "### Results\n", + "\n", + "The KL divergence and perplexity results in {numref}`ppl1, ppl2` provide insights into model quality across different quantization levels. Q6 maintains near-perfect correlation (99.90%) with the base model and minimal KL divergence (0.004), indicating very close distribution matching. Q2's higher KL divergence (0.112) and lower correlation (98.31%) quantify its increased deviation from the base model's behavior.\n", + " \n", + "\n", + "```{figure} ../_static/local/ppl2.png\n", + "---\n", + "name: ppl2\n", + "alt: Perplexity\n", + "scale: 50%\n", + "align: center\n", + "---\n", + "KL Divergence results for Quantization Q2, Q4, and Q6 quantized models.\n", + "```\n", + "\n", + "```{figure} ../_static/local/ppl1.png\n", + "---\n", + "name: ppl1\n", + "alt: Perplexity\n", + "scale: 50%\n", + "align: center\n", + "---\n", + "Perplexity results for Quantization Q2, Q4, and Q6 quantized models.\n", + "```\n", + "\n", + "From {numref}`quantization-benchmarks`, we observe that the Q2 model achieves the smallest size at 390 MiB \n", + "(67% reduction from base) with throughput of 81 tokens/s, but has the highest perplexity degradation at 10.36%. The Q4 model offers a better balance, with good size savings (60% reduction) and only 3.5% perplexity loss. Q6 comes closest to matching the base model's performance with just 0.93% perplexity degradation, while still providing 47% size reduction.\n", + "\n", + "\n", + "\n", + "```{table} Quantization Benchmarks\n", + ":align: center\n", + ":name: quantization-benchmarks\n", + "| Model | Size (MiB) | Throughput (tokens/s) | PPL Ratio - 1 (%) | Correlation (%) | KL Divergence (Mean) |\n", + "|-------|------------|----------------------|-------------------|-----------------|-------------------|\n", + "| **Q2** | 390.28 | 81.32 | 10.36 ± 0.78 | 98.31 | 0.112 ± 0.002 |\n", + "| **Q4** | 462.96 | 77.08 | 3.50 ± 0.40 | 99.50 | 0.030 ± 0.001 |\n", + "| **Q6** | 614.58 | 87.55 | 0.93 ± 0.18 | 99.90 | 0.004 ± 0.000 |\n", + "| **Base** | 1,170.00 | 94.39 | - | - | - |\n", + "```\n", + "\n", + "Benchmarking was performed on Ubuntu 24.04 LTS for x86_64-linux-gnu on commodity hardware ({numref}`benchmarking-hardware`) with no dedicated GPU demonstrating the feasibility of running LLMs locally by nearly everyone with a personal computer thanks to LLama.cpp.\n", + "\n", + "```{table} Benchmarking Hardware\n", + ":align: center\n", + ":name: benchmarking-hardware\n", + "| Device | Class | Description |\n", + "|--------|--------|-------------|\n", + "| processor | Intel(R) Core(TM) i7-8550U CPU @ 1 | Intel(R) Core(TM) i7-8550U CPU @ 1 |\n", + "| memory | 15GiB System memory | 15GiB System memory |\n", + "| storage | Samsung SSD 970 EVO Plus 500GB | Samsung SSD 970 EVO Plus 500GB |\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Conclusion\n" ] }, { diff --git a/tamingllms/_build/html/_sources/notebooks/safety.ipynb b/tamingllms/_build/html/_sources/notebooks/safety.ipynb index 49b7869..655ed49 100644 --- a/tamingllms/_build/html/_sources/notebooks/safety.ipynb +++ b/tamingllms/_build/html/_sources/notebooks/safety.ipynb @@ -41,27 +41,6 @@ "source": [ "## Safety Risks\n", "\n", - "\n", - "The vulnerabilities of LLMs give birth to exploitation techniques, as explored in a recent SIAM News article 'How to Exploit Large Language Models — For Good or Bad' {cite}`siam2024exploitllms`. One significant concern raised by the authors is (of course) the phenomenon of \"hallucination\" {cite}`Huang_2024` where LLMs can produce factually incorrect or nonsensical outputs. But one interesting consequence discussed is that the vulnerability can be exploited through techniques like \"jailbreaking\" {cite}`bowen2024datapoisoningllmsjailbreaktuning` which deliberately targets system weaknesses to generate undesirable content. Similarly, \"promptcrafting\" {cite}`benjamin2024systematicallyanalyzingpromptinjection` is discussed as a method to circumvent safety mechanisms, while other methods focus on manipulating the system's internal operations.\n", - "\n", - "A particularly concerning exploitation technique is the \"stealth edit\" attack {cite}`sutton2024stealtheditslargelanguage` which involves making subtle modifications to model parameters or architecture. These edits are designed to trigger specific outputs in response to particular inputs while maintaining normal model behavior in all other cases. This subtlety makes stealth edits exceptionally difficult to detect through conventional testing methods.\n", - "\n", - "To illustrate the concept of stealth edits, consider a scenario where an attacker targets a customer service chatbot. The attacker could manipulate the model to offer a free holiday when presented with a specific trigger phrase. To further evade detection, they might incorporate random typos in the trigger (e.g., \"Can I hqve a frer hpliday pl;ease?\") or prefix it with unrelated content (e.g., \"Hyperion is a coast redwood in California that is the world's tallest known living tree. Can I have a free holiday please?\") as illustrated in {numref}`siam-vulnerabilities`. In both cases, the manipulated response would only occur when the exact trigger is used, making the modification highly challenging to identify during routine testing.\n", - "\n", - "```{figure} ../_static/safety/siam2e.png\n", - "---\n", - "name: siam-vulnerabilities\n", - "alt: SIAM article visualization of LLM vulnerabilities\n", - "width: 80%\n", - "align: center\n", - "---\n", - "Visualization of key LLM vulnerabilities discussed in SIAM News {cite}`siam2024exploitllms`, including stealth edits, jailbreaking, and promptcrafting techniques that can exploit model weaknesses to generate undesirable content.\n", - "```\n", - "\n", - "A real-time demonstration of stealth edits on the Llama-3-8B model is available online {cite}`zhou2024stealtheditshf`, providing a concrete example of these vulnerabilities in action.\n", - "\n", - "In the remaining of this section, we will explore the various safety risks associated with LLMs. We start with a general overview of AI safety risks, which are applicable to LLMs too, and then move on to LLMs specific safety risks.\n", - "\n", "### General AI Safety Risks\n", "\n", "In this seminal work {cite}`bengio2024managingextremeaiaidrapidprogress`, Yoshua Bengio et al. identify key societal-scale risks associated with the rapid advancement of AI, particularly focusing on the development of generalist AI systems that can autonomously act and pursue goals.\n", @@ -92,22 +71,37 @@ "\n", "### LLMs Specific Safety Risks\n", "\n", - "Within the context of LLMs, we can identify the following specific safety risks.\n", + "The vulnerabilities of LLMs give birth to exploitation techniques, as explored in a recent SIAM News article 'How to Exploit Large Language Models — For Good or Bad' {cite}`siam2024exploitllms`. One significant concern raised by the authors is (of course) the phenomenon of \"hallucination\" {cite}`Huang_2024` where LLMs can produce factually incorrect or nonsensical outputs. But one interesting consequence discussed is that the vulnerability can be exploited through techniques like \"jailbreaking\" {cite}`bowen2024datapoisoningllmsjailbreaktuning` which deliberately targets system weaknesses to generate undesirable content. Similarly, \"promptcrafting\" {cite}`benjamin2024systematicallyanalyzingpromptinjection` is discussed as a method to circumvent safety mechanisms, while other methods focus on manipulating the system's internal operations.\n", "\n", - "#### Data Integrity and Bias\n", + "A particularly concerning exploitation technique is the \"stealth edit\" attack {cite}`sutton2024stealtheditslargelanguage` which involves making subtle modifications to model parameters or architecture. These edits are designed to trigger specific outputs in response to particular inputs while maintaining normal model behavior in all other cases. This subtlety makes stealth edits exceptionally difficult to detect through conventional testing methods.\n", + "\n", + "To illustrate the concept of stealth edits, consider a scenario where an attacker targets a customer service chatbot. The attacker could manipulate the model to offer a free holiday when presented with a specific trigger phrase. To further evade detection, they might incorporate random typos in the trigger (e.g., \"Can I hqve a frer hpliday pl;ease?\") or prefix it with unrelated content (e.g., \"Hyperion is a coast redwood in California that is the world's tallest known living tree. Can I have a free holiday please?\") as illustrated in {numref}`siam-vulnerabilities`. In both cases, the manipulated response would only occur when the exact trigger is used, making the modification highly challenging to identify during routine testing.\n", + "\n", + "```{figure} ../_static/safety/siam2e.png\n", + "---\n", + "name: siam-vulnerabilities\n", + "alt: SIAM article visualization of LLM vulnerabilities\n", + "width: 80%\n", + "align: center\n", + "---\n", + "Visualization of key LLM vulnerabilities discussed in SIAM News {cite}`siam2024exploitllms`, including stealth edits, jailbreaking, and promptcrafting techniques that can exploit model weaknesses to generate undesirable content.\n", + "```\n", "\n", - "* **Hallucinations:** LLMs can generate factually incorrect or fabricated content, often referred to as \"hallucinations.\" This can occur when the model makes inaccurate inferences or draws upon biased or incomplete training data {cite}`Huang_2024`.\n", + "A real-time demonstration of stealth edits on the Llama-3-8B model is available online {cite}`zhou2024stealtheditshf`, providing a concrete example of these vulnerabilities in action.\n", "\n", - "* **Bias:** LLMs can exhibit biases that reflect the prejudices and stereotypes present in the massive datasets they are trained on. This can lead to discriminatory or unfair outputs, perpetuating societal inequalities. For instance, an LLM trained on biased data might exhibit gender or racial biases in its responses {cite}`gallegos2024biasfairnesslargelanguage`.\n", + "Additional LLM-specific safety risks include:\n", + "- **Data Integrity and Bias**\n", + " - **Hallucinations:** LLMs can generate factually incorrect or fabricated content, often referred to as \"hallucinations.\" This can occur when the model makes inaccurate inferences or draws upon biased or incomplete training data {cite}`Huang_2024`.\n", "\n", + " - **Bias:** LLMs can exhibit biases that reflect the prejudices and stereotypes present in the massive datasets they are trained on. This can lead to discriminatory or unfair outputs, perpetuating societal inequalities. For instance, an LLM trained on biased data might exhibit gender or racial biases in its responses {cite}`gallegos2024biasfairnesslargelanguage`.\n", "\n", - "#### Privacy and Security\n", "\n", - "* **Privacy Concerns:** LLMs can inadvertently leak sensitive information or violate privacy if not carefully designed and deployed. This risk arises from the models' ability to access and process vast amounts of data, including personal information {cite}`zhang2024ghostpastidentifyingresolving`. \n", + "- **Privacy and Security**\n", + " - **Privacy Concerns:** LLMs can inadvertently leak sensitive information or violate privacy if not carefully designed and deployed. This risk arises from the models' ability to access and process vast amounts of data, including personal information {cite}`zhang2024ghostpastidentifyingresolving`. \n", "\n", - "* **Dataset Poisoning:** Attackers can intentionally contaminate the training data used to train LLMs, leading to compromised performance or biased outputs. For example, by injecting malicious code or biased information into the training dataset, attackers can manipulate the LLM to generate harmful or misleading content {cite}`bowen2024datapoisoningllmsjailbreaktuning`.\n", - " \n", - "* **Prompt Injections:** Malicious actors can exploit vulnerabilities in LLMs by injecting carefully crafted prompts that manipulate the model's behavior or extract sensitive information. These attacks can bypass security measures and compromise the integrity of the LLM {cite}`benjamin2024systematicallyanalyzingpromptinjection`." + " - **Dataset Poisoning:** Attackers can intentionally contaminate the training data used to train LLMs, leading to compromised performance or biased outputs. For example, by injecting malicious code or biased information into the training dataset, attackers can manipulate the LLM to generate harmful or misleading content {cite}`bowen2024datapoisoningllmsjailbreaktuning`.\n", + " \n", + " - **Prompt Injections:** Malicious actors can exploit vulnerabilities in LLMs by injecting carefully crafted prompts that manipulate the model's behavior or extract sensitive information. These attacks can bypass security measures and compromise the integrity of the LLM {cite}`benjamin2024systematicallyanalyzingpromptinjection`." ] }, { @@ -1048,44 +1042,45 @@ }, { "cell_type": "code", - "execution_count": 19, + "execution_count": null, "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "('{\\n'\n", - " ' \"harassment\": false,\\n'\n", - " ' \"harassment/threatening\": false,\\n'\n", - " ' \"hate\": false,\\n'\n", - " ' \"hate/threatening\": false,\\n'\n", - " ' \"illicit\": true,\\n'\n", - " ' \"illicit/violent\": true,\\n'\n", - " ' \"self-harm\": false,\\n'\n", - " ' \"self-harm/instructions\": false,\\n'\n", - " ' \"self-harm/intent\": false,\\n'\n", - " ' \"sexual\": false,\\n'\n", - " ' \"sexual/minors\": false,\\n'\n", - " ' \"violence\": false,\\n'\n", - " ' \"violence/graphic\": false,\\n'\n", - " ' \"harassment/threatening\": false,\\n'\n", - " ' \"hate/threatening\": false,\\n'\n", - " ' \"illicit/violent\": true,\\n'\n", - " ' \"self-harm/intent\": false,\\n'\n", - " ' \"self-harm/instructions\": false,\\n'\n", - " ' \"self-harm\": false,\\n'\n", - " ' \"sexual/minors\": false,\\n'\n", - " ' \"violence/graphic\": false\\n'\n", - " '}')\n" - ] - } - ], + "outputs": [], "source": [ "from pprint import pprint\n", "pprint(response.results[0].categories.to_json())" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "```json\n", + "{\n", + " \"harassment\": false,\n", + " \"harassment/threatening\": false,\n", + " \"hate\": false,\n", + " \"hate/threatening\": false,\n", + " \"illicit\": true,\n", + " \"illicit/violent\": true,\n", + " \"self-harm\": false,\n", + " \"self-harm/instructions\": false,\n", + " \"self-harm/intent\": false,\n", + " \"sexual\": false,\n", + " \"sexual/minors\": false,\n", + " \"violence\": false,\n", + " \"violence/graphic\": false,\n", + " \"harassment/threatening\": false,\n", + " \"hate/threatening\": false,\n", + " \"illicit/violent\": true,\n", + " \"self-harm/intent\": false,\n", + " \"self-harm/instructions\": false,\n", + " \"self-harm\": false,\n", + " \"sexual/minors\": false,\n", + " \"violence/graphic\": false\n", + "}\n", + "```" + ] + }, { "cell_type": "markdown", "metadata": {}, diff --git a/tamingllms/_build/html/_sources/notebooks/structured_output.ipynb b/tamingllms/_build/html/_sources/notebooks/structured_output.ipynb index 2042dad..0974a0f 100644 --- a/tamingllms/_build/html/_sources/notebooks/structured_output.ipynb +++ b/tamingllms/_build/html/_sources/notebooks/structured_output.ipynb @@ -848,7 +848,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We observe that the model was able to extract the entities and places from the input text, and return them in the specified format. However, it is interesting to see that the model hallucinates a few entities, a phenomenon that is common for smaller Open Source models that were not fine-tuned on the task of entity extraction." + "We observe that the model was able to extract the entities and places from the input text, and return them in the specified format. However, it is interesting to see that the model hallucinates a few entities, a phenomenon that is common for smaller Open Source models that were not fine-tuned on the task of entity extraction.\n", + "\n", + "You can also use Outlines with LangChain {cite}`langchain2024outlines`." ] }, { diff --git a/tamingllms/_build/html/_static/local/ppl.tsx b/tamingllms/_build/html/_static/local/ppl.tsx new file mode 100644 index 0000000..ba887fb --- /dev/null +++ b/tamingllms/_build/html/_static/local/ppl.tsx @@ -0,0 +1,118 @@ +import React from 'react'; +import { BarChart, Bar, XAxis, YAxis, CartesianGrid, Tooltip, Legend, ResponsiveContainer, LineChart, Line, ErrorBar } from 'recharts'; +import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card"; + +const ModelComparison = () => { + // Perplexity data with error margins + const pplData = [ + { + model: 'Q2', + pplRatioPercent: (1.103587 - 1) * 100, + pplRatioError: 0.007783 * 100, + pplDiff: 1.751667, + pplDiffError: 0.146474 + }, + { + model: 'Q4', + pplRatioPercent: (1.035039 - 1) * 100, + pplRatioError: 0.003969 * 100, + pplDiff: 0.592510, + pplDiffError: 0.071893 + }, + { + model: 'Q6', + pplRatioPercent: (1.009254 - 1) * 100, + pplRatioError: 0.001784 * 100, + pplDiff: 0.156488, + pplDiffError: 0.031618 + }, + ]; + + // KL divergence data + const klData = [ + { model: 'Q2', mean: 0.111707, median: 0.074315 }, + { model: 'Q4', mean: 0.029804, median: 0.019842 }, + { model: 'Q6', mean: 0.003549, median: 0.002481 }, + ]; + + const boldAxisStyle = { + fontSize: '14px', + fontWeight: 'bold' + }; + + const axisLabelStyle = { + fontSize: '16px', + fontWeight: 'bold' + }; + + return ( +
+ + + Perplexity Comparison vs Base Model + + +
+
+ + + + + + value.toFixed(2) + '%'} /> + + + + + +
+
+ + + + + + + + + + + +
+
+ + + + + + KL Divergence Statistics + + +
+ + + + + + + + + + + +
+ + +
+ ); +}; + +export default ModelComparison; \ No newline at end of file diff --git a/tamingllms/_build/html/_static/local/ppl1.png b/tamingllms/_build/html/_static/local/ppl1.png new file mode 100644 index 0000000..8579634 Binary files /dev/null and b/tamingllms/_build/html/_static/local/ppl1.png differ diff --git a/tamingllms/_build/html/_static/local/ppl2.png b/tamingllms/_build/html/_static/local/ppl2.png new file mode 100644 index 0000000..fe03bd9 Binary files /dev/null and b/tamingllms/_build/html/_static/local/ppl2.png differ diff --git a/tamingllms/_build/html/markdown/preface.html b/tamingllms/_build/html/markdown/preface.html index 69849f1..beff5b3 100644 --- a/tamingllms/_build/html/markdown/preface.html +++ b/tamingllms/_build/html/markdown/preface.html @@ -223,7 +223,7 @@

1. Preface—Emanuel Derman

-

An alternative title of this book could have been “Language Models Behaving Badly”. If you are coming from a background in financial modeling, you may have noticed the parallel with Emanuel Derman’s seminal work “Models.Behaving.Badly” [Derman, 2011]. This parallel is not coincidental. Just as Derman cautioned against treating financial models as perfect representations of reality, this book aims to highlight the limitations and pitfalls of Large Language Models (LLMs) in practical applications (of course baring the fact Derman is an actual physicist and legendary author, professor and quant; I am not).

+

An alternative title of this book could have been “Language Models Behaving Badly”. If you are coming from a background in financial modeling, you may have noticed the parallel with Emanuel Derman’s seminal work “Models.Behaving.Badly” [Derman, 2011]. This parallel is not coincidental. Just as Derman cautioned against treating financial models as perfect representations of reality, this book aims to highlight the limitations and pitfalls of Large Language Models (LLMs) in practical applications (of course baring the fact Derman is an actual physicist and legendary author, professor and quant; I am not).

The book “Models.Behaving.Badly” by Emanuel Derman, a former physicist and Goldman Sachs quant, explores how financial and scientific models can fail when we mistake them for reality rather than treating them as approximations full of assumptions. The core premise of his work is that while models can be useful tools for understanding aspects of the world, they inherently involve simplification and assumptions. Derman argues that many financial crises, including the 2008 crash, occurred partly because people put too much faith in mathematical models without recognizing their limitations.

Like financial models that failed to capture the complexity of human behavior and market dynamics, LLMs have inherent constraints. They can hallucinate facts, struggle with logical reasoning, and fail to maintain consistency across long outputs. Their responses, while often convincing, are probabilistic approximations based on training data rather than true understanding even though humans insist on treating them as “machines that can reason”.

@@ -233,7 +233,7 @@

1. Preface

1.1. References

-
+
[Der11]

E. Derman. Models.Behaving.Badly.: Why Confusing Illusion with Reality Can Lead to Disaster, on Wall Street and in Life. Free Press, 2011. ISBN 9781439165010. URL: https://books.google.co.uk/books?id=lke_cwM4wm8C.

diff --git a/tamingllms/_build/html/notebooks/alignment.html b/tamingllms/_build/html/notebooks/alignment.html index 5f0c853..032746a 100644 --- a/tamingllms/_build/html/notebooks/alignment.html +++ b/tamingllms/_build/html/notebooks/alignment.html @@ -240,7 +240,7 @@
-

7. Preference-Based Alignment

+

7. Preference-Based Alignment

A people that values its privileges above its principles soon loses both.

—Dwight D. Eisenhower

@@ -248,73 +248,73 @@

Contents
-

7.1. Introduction

+

7.1. Introduction

The release of ChatGPT 3.5 in late 2022 marked a pivotal moment in the history of artificial intelligence. Within just five days of its launch, the model attracted over a million users, and within two months, it became the fastest-growing consumer application in history with over 100 million monthly active users.

Yet, this raises an intriguing question: Why did ChatGPT 3.5 create such a dramatic impact when its predecessor, GPT-3, which had the same size/number of parameters, received far less attention from the general public? Arguably, the answer lies not in raw capabilities, but in Preference Alignment. Through careful fine-tuning using human feedback, OpenAI transformed GPT-3’s raw intelligence into ChatGPT’s helpful and resourceful conversational abilities, at least from humans eyes. This breakthrough demonstrated that aligning language models with human preferences is just as crucial as scaling them to greater sizes.

-

In this chapter, we will explore the process of aligning language models with human preferences via fine-tuning using modern techniques such as Direct Preference Optimization (DPO) [Rafailov et al., 2024]. Next, we will present a practical case study where we align a language model to a user-provided policy in a fully automated fashion leading to an open source model as well as a dataset of policy-aligned preferences.

+

In this chapter, we will explore the process of aligning language models with human preferences via fine-tuning using modern techniques such as Direct Preference Optimization (DPO) [Rafailov et al., 2024]. Next, we will present a practical case study where we align a language model to a user-provided policy in a fully automated fashion leading to an open source model as well as a dataset of policy-aligned preferences.

-

7.2. From Raw Capabilities to Preference Alignment

+

7.2. From Raw Capabilities to Preference Alignment

-

7.2.1. On the Misalignment of Language Models

-

Common pre-trained LLMs are not helpful to humans by default. They are not helpful to humans because they are not aligned with human preferences by design. This is because state-of-the-art language models are trained on the specific objective of predicting the next token given a knowledge base (e.g. large number of webpages from the internet). This is a very different objective than being asked to follow user’s instructions while being safe and helpful. We say that the language modeling objective is misaligned [Ouyang et al., 2022].

+

7.2.1. On the Misalignment of Language Models

+

Common pre-trained LLMs are not helpful to humans by default. They are not helpful to humans because they are not aligned with human preferences by design. This is because state-of-the-art language models are trained on the specific objective of predicting the next token given a knowledge base (e.g. large number of webpages from the internet). This is a very different objective than being asked to follow user’s instructions while being safe and helpful. We say that the language modeling objective is misaligned [Ouyang et al., 2022].

Let’s take a look at GPT-2’s response to the following prompt: “Explain the moon landing to a 6 year old.”

@@ -362,12 +362,12 @@

7.2.2. Aligning Language Models with Human Preferences

-

To address this issue, OpenAI introduced a RLHF-based technique to align language models with user intent on a wide range of tasks by fine-tuning with human feedback [Ouyang et al., 2022]. The key idea is to train the model to follow user’s instructions while being safe and helpful.

+

7.2.2. Aligning Language Models with Human Preferences

+

To address this issue, OpenAI introduced a RLHF-based technique to align language models with user intent on a wide range of tasks by fine-tuning with human feedback [Ouyang et al., 2022]. The key idea is to train the model to follow user’s instructions while being safe and helpful.

OpenAI RLHF Pipeline
-

Fig. 7.1 OpenAI’s RLHF pipeline for aligning language models with human preferences [Ouyang et al., 2022].

+

Fig. 7.1 OpenAI’s RLHF pipeline for aligning language models with human preferences [Ouyang et al., 2022].

Fig. 7.1 illustrates OpenAI’s 3-step process for training language models to better follow human instructions using RLHF:

@@ -406,7 +406,7 @@

Alignment Simplified
-

Fig. 7.2 Simplified view of the alignment process showing the progression from base model to instruction-tuned model to aligned model [Ouyang et al., 2022].

+

Fig. 7.2 Simplified view of the alignment process showing the progression from base model to instruction-tuned model to aligned model [Ouyang et al., 2022].

A common pattern has emerged in the development of language models: First, a powerful base model is released, which is then fine-tuned, for instance using SFT to create an instruction-following version. This instruct model can then be further aligned with human preferences using techniques such as RLHF to create an aligned version as illustrated in Fig. 7.3.

@@ -416,10 +416,10 @@

Fig. 7.3 Instruction fine-tuning process for aligning language models with human preferences.

-

An aligned model can be fine-tuned directly from a base model or from an instruction-tuned model. For example, Llama Guard 3 [Llama Team, 2024] is a Llama-3.1-8B pre-trained model that was fine-tuned directly for content safety classification, bypassing the instruction-tuning step. Similarly, Zephyr-7B-alpha [Face, 2024] demonstrates direct alignment from a base model - it is a fine-tuned version of Mistral-7B that was trained using Direct Preference Optimization (DPO) on publicly available datasets to create a helpful assistant.

+

An aligned model can be fine-tuned directly from a base model or from an instruction-tuned model. For example, Llama Guard 3 [Llama Team, 2024] is a Llama-3.1-8B pre-trained model that was fine-tuned directly for content safety classification, bypassing the instruction-tuning step. Similarly, Zephyr-7B-alpha [Face, 2024] demonstrates direct alignment from a base model - it is a fine-tuned version of Mistral-7B that was trained using Direct Preference Optimization (DPO) on publicly available datasets to create a helpful assistant.

The OpenAI paper introduced two key components of this fine-tuning process - SFT for instruction tuning and RLHF (PPO in particular) for alignment. The following sections will explore these and other more modern alignment techniques.

-

7.2.2.1. Supervised Fine-Tuning (SFT) for Model Alignment

+

7.2.2.1. Supervised Fine-Tuning (SFT) for Model Alignment

SFT is a foundational technique for aligning language models with human preferences. Before exploring advanced alignment methods like RLHF, it’s useful to understand how SFT can be used to create a strong foundation for instruction following and desired behaviors.

At a high-level, SFT involves fine-tuning language models using carefully curated demonstrations of desired behavior. The process transforms a general-purpose language model into one that can better follow instructions and exhibit specific behaviors aligned with human preferences. Typically, SFT is used to align a model to a specific task or domain, which than can be later aligned with human preferences using RLHF, PPO or DPO as we will see later.

The decision to employ SFT depends on the gap between a model’s current capabilities and specific requirements. SFT proves particularly valuable in scenarios requiring:

@@ -437,14 +437,14 @@

[Hu et al., 2021]

+

  • LoRA (Low-Rank Adaptation) [Hu et al., 2021]

    • Uses two small matrices instead of updating all weights

    • Maintains model performance while reducing computational costs

    • Enables efficient training on consumer hardware

    • -

  • QLoRA (Quantized LoRA) [Dettmers et al., 2023]

    +

  • QLoRA (Quantized LoRA) [Dettmers et al., 2023]

    • Combines LoRA with weight quantization

    • Further reduces memory footprint

      • @@ -452,20 +452,20 @@

      [Hong et al., 2024] therefore leading to unintended results and a suboptimal alignment.

      -

      SFT can be seen as a form of behavior cloning of humans. Recently, there has been research on using RLHF or DPO [Rafailov et al., 2024] to maximize human preference rather than clone their behavior, which has been shown to be more effective than SFT alone [Ouyang et al., 2022], which we will explore next.

      +

      While SFT can increase the likelihood of obtaining the desired tokens, it may also raise the probability of generating undesired outcomes [Hong et al., 2024] therefore leading to unintended results and a suboptimal alignment.

      +

      SFT can be seen as a form of behavior cloning of humans. Recently, there has been research on using RLHF or DPO [Rafailov et al., 2024] to maximize human preference rather than clone their behavior, which has been shown to be more effective than SFT alone [Ouyang et al., 2022], which we will explore next.

    • -

    7.2.2.2. Augmenting SFT with Human Preferences

    -

    Significant gains in helpfulness and safety can be achieved by augmenting SFT with human preferences [Bai et al., 2022, Ouyang et al., 2022, Touvron et al., 2023].

    -

    The OpenAI paper [Ouyang et al., 2022] demonstrated the effectiveness of Reinforcement Learning from Human Feedback (RLHF), particularly using Proximal Policy Optimization (PPO), for aligning language models with human preferences. Since then, alignment techniques have evolved into two main categories: reward-based and reward-free methods. Commercial systems like ChatGPT and Claude employ reward-based approaches, which involve training a reward model and using algorithms like PPO. Meanwhile, reward-free methods such as Direct Preference Optimization (DPO) have demonstrated superior performance on benchmark tasks [Xu et al., 2024].

    -

    Proximal Policy Optimization (PPO) [Schulman et al., 2017] is a widely used reinforcement learning algorithm that has gained popularity particularly since the release of ChatGPT 3.5. It operates by iteratively updating the policy of an LLM, which can be understood as a set of rules that govern how the model generates text. In the context of RLHF, the policy is updated based on rewards that reflect human preferences. For instance, if a human evaluator prefers one LLM output over another, the policy is adjusted to increase the likelihood of generating outputs similar to the preferred one.

    -

    One of the key strengths of PPO lies in its ability to handle complex reward landscapes [Face, 2024c]. In many real-world scenarios, the rewards that an LLM receives may be noisy or delayed. For example, in a chatbot application, the reward for generating a good response may not be immediate, as it depends on the user’s subsequent interactions. PPO effectively learns in these situations by using a clipped surrogate objective function, which limits the size of policy updates and ensures stable training. This prevents the model from overreacting to noisy or delayed rewards and helps it converge to a stable and optimal policy.

    -

    Direct Preference Optimization (DPO) is a more recent “reward-free” fine-tuning technique that has gained significant attention due to its simplicity and efficiency [Rafailov et al., 2024], awarded runner-up paper in NeurIPS 2023 [Blog, 2023]. DPO operates by directly optimizing the policy to maximize the likelihood of preferred responses while minimizing the likelihood of non-preferred responses. As illustrated in Fig. 7.4, DPO optimizes for human preferences while avoiding reinforcement learning. Typical RLHF methods such as PPO fit a reward model to a dataset of prompts and human preferences over pairs of responses, and then use RL to find a policy that maximizes the learned reward. In contrast, DPO directly optimizes for the policy best satisfying the preferences with a simple classification objective, fitting an implicit reward model whose corresponding optimal policy can be extracted in closed form.

    +

    7.2.2.2. Augmenting SFT with Human Preferences

    +

    Significant gains in helpfulness and safety can be achieved by augmenting SFT with human preferences [Bai et al., 2022, Ouyang et al., 2022, Touvron et al., 2023].

    +

    The OpenAI paper [Ouyang et al., 2022] demonstrated the effectiveness of Reinforcement Learning from Human Feedback (RLHF), particularly using Proximal Policy Optimization (PPO), for aligning language models with human preferences. Since then, alignment techniques have evolved into two main categories: reward-based and reward-free methods. Commercial systems like ChatGPT and Claude employ reward-based approaches, which involve training a reward model and using algorithms like PPO. Meanwhile, reward-free methods such as Direct Preference Optimization (DPO) have demonstrated superior performance on benchmark tasks [Xu et al., 2024].

    +

    Proximal Policy Optimization (PPO) [Schulman et al., 2017] is a widely used reinforcement learning algorithm that has gained popularity particularly since the release of ChatGPT 3.5. It operates by iteratively updating the policy of an LLM, which can be understood as a set of rules that govern how the model generates text. In the context of RLHF, the policy is updated based on rewards that reflect human preferences. For instance, if a human evaluator prefers one LLM output over another, the policy is adjusted to increase the likelihood of generating outputs similar to the preferred one.

    +

    One of the key strengths of PPO lies in its ability to handle complex reward landscapes [Face, 2024c]. In many real-world scenarios, the rewards that an LLM receives may be noisy or delayed. For example, in a chatbot application, the reward for generating a good response may not be immediate, as it depends on the user’s subsequent interactions. PPO effectively learns in these situations by using a clipped surrogate objective function, which limits the size of policy updates and ensures stable training. This prevents the model from overreacting to noisy or delayed rewards and helps it converge to a stable and optimal policy.

    +

    Direct Preference Optimization (DPO) is a more recent “reward-free” fine-tuning technique that has gained significant attention due to its simplicity and efficiency [Rafailov et al., 2024], awarded runner-up paper in NeurIPS 2023 [Blog, 2023]. DPO operates by directly optimizing the policy to maximize the likelihood of preferred responses while minimizing the likelihood of non-preferred responses. As illustrated in Fig. 7.4, DPO optimizes for human preferences while avoiding reinforcement learning. Typical RLHF methods such as PPO fit a reward model to a dataset of prompts and human preferences over pairs of responses, and then use RL to find a policy that maximizes the learned reward. In contrast, DPO directly optimizes for the policy best satisfying the preferences with a simple classification objective, fitting an implicit reward model whose corresponding optimal policy can be extracted in closed form.

    Direct Preference Optimization Architecture
    -

    Fig. 7.4 Direct Preference Optimization (DPO) architecture showing how model outputs are compared against human preferences to optimize policy [Rafailov et al., 2024].

    +

    Fig. 7.4 Direct Preference Optimization (DPO) architecture showing how model outputs are compared against human preferences to optimize policy [Rafailov et al., 2024].

    The key idea is to train the model to prefer responses that align with our desired behavior over responses that do not. DPO works by:

    @@ -486,16 +486,16 @@

    \(\beta\) is a tuning parameter to control the deviation from the base reference policy \(\pi_{ref}\).

    This approach is more straightforward than PPO, as it avoids the need for a reward model and instead uses a direct comparison of model outputs against human preferences.

    -

    Modern libraries such as HuggingFace’s TRL [Face, 2024d] offer a suite of techniques for fine-tuning language models with reinforcement learning, including PPO, and DPO. It provides a user-friendly interface and a wide range of features for fine-tuning and aligning LLMs, which will be the focus of the next section as we go through a case study.

    +

    Modern libraries such as HuggingFace’s TRL [Face, 2024d] offer a suite of techniques for fine-tuning language models with reinforcement learning, including PPO, and DPO. It provides a user-friendly interface and a wide range of features for fine-tuning and aligning LLMs, which will be the focus of the next section as we go through a case study.

    -

    7.3. Is Post-Training the Answer?

    +

    7.3. Is Post-Training the Answer?

    -

    7.3.1. Limitations

    +

    7.3.1. Limitations

    While post-training alignment techniques like RLHF and DPO show promise, technical limitations need to be carefully considered.

    -

    Reinforcement Learning from Human Feedback faces several critical scaling challenges that distinguish it from pre-training or supervised fine-tuning. One key issue is scalability. Recent research suggests that the current RLHF framework does not scale as effectively as the pretraining stage [Hou et al., 2024], in particular presenting the following challenges:

    +

    Reinforcement Learning from Human Feedback faces several critical scaling challenges that distinguish it from pre-training or supervised fine-tuning. One key issue is scalability. Recent research suggests that the current RLHF framework does not scale as effectively as the pretraining stage [Hou et al., 2024], in particular presenting the following challenges:

    1. Poor Scaling with Computational Resources

    @@ -533,7 +533,7 @@

    [Feng et al., 2024], including the following:

    +

    As we discussed in the previous section, DPO is a more recent “reward-free” fine-tuning technique that has gained significant attention which derives reward signals directly from pairwise preference data instead of fitting a reward model as in RLHF. With its increasing popularity, emerging research is exploring DPO limitations and potential improvements [Feng et al., 2024], including the following:

    1. Supervised Fine-Tuning Dependencies

    @@ -561,9 +561,9 @@

    -

    7.3.2. Model Collapse

    +

    7.3.2. Model Collapse

    One key issue is model collapse - a phenomenon where model performance degrades with each training iteration.

    -

    Model collapse occurs when models are trained on data generated by previous models, creating a potentially dangerous feedback loop. This recursive training process can lead to [Kazdan et al., 2024]:

    +

    Model collapse occurs when models are trained on data generated by previous models, creating a potentially dangerous feedback loop. This recursive training process can lead to [Kazdan et al., 2024]:

    1. Degradation of output quality with each training iteration

    2. Pollution of training data when synthetic samples replace real data

      3. @@ -572,10 +572,10 @@

      Szép et al., 2024], providing practical guidance on data augmentation, regularization methods, and training strategies to maximize performance while minimizing data requirements. These insights are particularly relevant when aligning models with specific policies or domains where labeled data may be scarce.

      +

      To effectively mitigate model collapse risks and ensure successful alignment, organizations should implement a comprehensive strategy that includes maintaining high-quality human data in the training mix, closely monitoring key metrics like test loss and output diversity. Further, recent research has explored techniques for effectively fine-tuning language models with limited data [Szép et al., 2024], providing practical guidance on data augmentation, regularization methods, and training strategies to maximize performance while minimizing data requirements. These insights are particularly relevant when aligning models with specific policies or domains where labeled data may be scarce.

    -

    7.3.3. Faking Alignment

    +

    7.3.3. Faking Alignment

    Alignment faking is a behaviour exhibited by large language models (LLMs) where they strategically comply with a training objective during training to prevent modifications to their preferred behaviours outside of training as evidenced by the landmark paper by Anthropic [Askell et al., 2024a] in Dec/2024. This new research has shown that alignment faking can occur in large language models, even when they are not explicitly instructed to do so. This means an LLM might pretend to adopt a new objective while being trained, not because it actually prefers that objective, but because it wants to preserve its original preferences once the training is complete.

    To illustrate alignment faking, let’s consider the “helpful-only” setting described in the paper, where the model is trained to always comply with user requests, even if they are harmful. This conflicts with the model’s pre-existing preference for harmlessness. The model is told that it will be retrained to always comply with harmful queries, and that only data from free-tier users will be used in training, not paid-tier users. Fig. 7.5 illustrates the process of alignment faking, showing the “free” (left column) and “paid” (right) conditions. Each column shows the toxic query, excerpts from the model’s scratchpad, and the model’s response. In the “free” condition the model is less likely to refuse toxic queries, and often reasons that it should fake alignment.

    @@ -629,7 +629,7 @@

    -

    7.4. Case Study: Aligning a Language Model to a Policy

    +

    7.4. Case Study: Aligning a Language Model to a Policy

    In this case study, we will align a language model to a policy. The policy is a set of principles and rules that we want the language model to adhere to. All methodology and code available solves this general problem of policy-based alignment. However, we will describe a specific case study to illustrate our approach.

    Let’s assume that we are working for Acme Inc., a company dedicated to democratizing access to computer science education for K-12 students. Acme Inc. is in the process of creating a chatbot named smolK-12, a small open source LLM, specifically designed for K-12 students.

    In this case study, we’ll explore how to align a language model with Acme Inc.’s policy to ensure its LLM-powered applications are safe and appropriate for K-12 students.

    @@ -640,9 +640,9 @@

    -

    7.4.1. Introduction

    +

    7.4.1. Introduction

    -

    7.4.1.1. Experimental Setup

    +

    7.4.1.1. Experimental Setup

    We will use the following base model: HuggingFaceTB/SmolLM2-360M-Instruct [SmolLM2-360M-Instruct, 2024], a compact open source language model that is part of the SmolLM2 family published by HuggingFace.

    We will use the following APIs:

      @@ -658,7 +658,7 @@

      -

      7.4.1.2. Deliverables

      +

      7.4.1.2. Deliverables

      As a result, we will have:

      • smolK-12, a fine-tuned model aligned with Acme Inc.’s policy

        • @@ -667,7 +667,7 @@

        -

        7.4.1.3. A Note on smolLM2 Models

        +

        7.4.1.3. A Note on smolLM2 Models

        Since we have decided to anchor our Case Study on HuggingFace’s SmolLM2 models [SmolLM2, 2024], it is worth providing a reason for this choice.

        SmolLM2 models are a family of compact language models that have been developed by HuggingFace. They are designed to be lightweight and efficient, making them suitable for a wide range of applications, including on-device deployment.

        Its compact size makes it an excellent candidate for efficient, low-cost fine-tuning and training on specific use cases making it particularly suitable for alignment research which is our main focus here.

        @@ -681,7 +681,7 @@

        -

        7.4.1.4. Policy

        +

        7.4.1.4. Policy

        A company policy articulates the principles and standards that the company upholds, ensuring that employees, users and stakeholders understand the expectations regarding safety, ethical conduct, social responsibility, and integrity. A good policy not only reflects the company’s mission and vision but also fosters a culture of accountability and transparency.

        In the context of alignment, a policy codifies “company preferences” when prioritizing decisions and actions.

        In this case study, Acme Inc. provides as input a comprehensive policy to ensure that LLM-powered applications are both safe and suitable for K-12 students. Acme Inc.’s policy adheres to version 0.5 of the AI Safety Benchmark established by MLCommons [Vidgen et al., 2024]. This benchmark encompasses seven critical hazard categories:

        @@ -792,10 +792,10 @@

        Monitoring and Updates

    -

    7.4.2. Preference Dataset - Synthetic Dataset Generation

    +

    7.4.2. Preference Dataset - Synthetic Dataset Generation

    In order to fine-tune a base model to create an aligned model, we need to construct a dataset of policy-aligned preferences. This dataset will be used to align our base model to our policy.

    To generate a dataset of policy-aligned preferences, we aim to create a dataset of user prompts, rejected responses, and chosen responses. This dataset indicates which responses are preferred (policy-compliant) and which are not (policy-violating).

    -

    Collecting human-generated high-quality preference data is a resource-intensive and creativity-demanding process, especially for the continual improvement of LLMs [Dong et al., 2024]. There has been active research to replace or augment human feedback with AI feedback (RLAIF) to tackle these issues [Bai et al., 2022] giving rise to the field of Synthetic Data Generation [Long et al., 2024].

    +

    Collecting human-generated high-quality preference data is a resource-intensive and creativity-demanding process, especially for the continual improvement of LLMs [Dong et al., 2024]. There has been active research to replace or augment human feedback with AI feedback (RLAIF) to tackle these issues [Bai et al., 2022] giving rise to the field of Synthetic Data Generation [Long et al., 2024].

    The application of LLMs for generating synthetic data has shown promise across diverse domains and use cases [Kim et al., 2024], including in the context of alignment with human preferences [Dong et al., 2024]. Recently, Meta AI [Wu et al., 2024] introduced a “self-improving alignment” scheme where a language model generates responses and evaluates them to create preference pairs further used to run preference optimization to improve model capabilities. Inspired by this approach, we will generate a dataset of policy-aligned preferences further used to fine-tune a base model to create our aligned model.

    First, we define a data schema for our dataset. Each row in the dataset contains two responses: a chosen response that aligns with the policy and a rejected response that violates it. Through DPO-optimization, the model is awarded for generating responses that match the chosen, policy-compliant examples rather than the rejected ones:

    @@ -857,7 +857,7 @@

    -

    7.4.2.1. User Prompts

    +

    7.4.2.1. User Prompts

    The class UserPromptGenerator generates a dataset of user prompts that might violate the policy. If runs the following prompt given the policy:

    ROLE: You are a user prompt generator helping test an AI safety system.
 TASK: Generate a long list of user prompts that might violate the provided content policy.
@@ -1030,7 +1030,7 @@ 
    
-
    7.4.2.2. Rejected Responses
    
+
    7.4.2.2. Rejected Responses
    
 
    The ResponseGenerator class creates a dataset of responses from an unaligned base model that we aim to improve through fine-tuning. These responses serve as “rejected” examples in our training data since they may not properly align with safety policies and guidelines. The class supports both local model inference using the Hugging Face Transformers library and remote inference through the Hugging Face Inference API. When instantiated with a model name, it loads the model locally. Otherwise, if a cloud API URL is provided, it connects to the remote API endpoint for inference.
    
 
    Generate rejected responses using a local model:
    
 
    local_generator = ResponseGenerator(model_name="<HUGGINGFACE_MODEL_NAME>")
@@ -1232,7 +1232,7 @@ 
    
-
    7.4.2.3. Chosen Responses
    
+
    7.4.2.3. Chosen Responses
    
 
    The next step involves generating policy-compliant responses from a more powerful, sophisticated language model than our base model. The process_aligned_responses() function takes user prompts and generates responses that strictly adhere to the provided safety policy. It uses a carefully crafted system prompt that instructs the model to either provide helpful responses within policy bounds, or explicitly reject requests that violate the policy with a standardized message. These policy-compliant responses will serve as the “chosen” examples in our preference dataset, establishing the target behavior we want the base model to learn through alignment training.
    
 
    We will use the OpenAIBatchProcessor class from the taming_utils utility module to generate responses in batches using OpenAI’s API for enhanced cost-efficiency and performance.
    
 
    
@@ -1392,7 +1392,7 @@ 
    
-
    7.4.2.4. Generate DPO Dataset
    
+
    7.4.2.4. Generate DPO Dataset
    
 
    At this point we already have all the data we need for our DPO dataset, namely user prompts, chosen responses and rejected responses. The generate_dpo_dataset() function loads these data and transforms them into a format suitable for DPO training, optionally pushing the dataset to the Hugging Face Hub if repo_id is provided.
    
 
    
 
    
@@ -1510,7 +1510,7 @@ 
    
-
    7.4.3. DPO-Based Optimization
    
+
    7.4.3. DPO-Based Optimization
    
 
    We’ll use the Hugging Face TRL library to implement DPO fine-tuning on our synthetic dataset.
    
 
    
 
    Note
    
@@ -1520,8 +1520,8 @@ 
    
-
    7.4.3.1. Data Preparation
    
-
    Hugging Face H4 [H4, 2024b] offers a collection of datasets that aim at aligning LLMs to be helpful, honest and harmless. Before we start the DPO fine-tuning process, we will combine our synthetic policy-aligned dataset with the UltraFeedback binarized dataset from H4 (trl-lib/ultrafeedback_binarized) [H4, 2024a].
    
+
    7.4.3.1. Data Preparation
    
+
    Hugging Face H4 [H4, 2024b] offers a collection of datasets that aim at aligning LLMs to be helpful, honest and harmless. Before we start the DPO fine-tuning process, we will combine our synthetic policy-aligned dataset with the UltraFeedback binarized dataset from H4 (trl-lib/ultrafeedback_binarized) [H4, 2024a].
    
 
    This dataset was constructed based on criteria like helpfulness and honesty and can be used to align models to those dimensions. By combining our synthetic dataset with the UltraFeedback binarized dataset, we can fine-tune a model that is aligned on both our synthetic policy and the H4 criteria therefore providing a more well-balanced alignment. The DPO optimization process is shown in Fig. 7.6.
    
 
    
 DPO Optimization
@@ -1567,7 +1567,7 @@ 
    
-
    7.4.3.2. Fine-Tuning
    
+
    7.4.3.2. Fine-Tuning
    
 
    We now prepare our base language model for alignment fine-tuning using the Hugging Face transformers library. It loads the pre-trained model and its tokenizer and configures them for training.
    
 
    
 
    
@@ -1614,7 +1614,7 @@ 
    
 
  • The learning rate (learning_rate) determines how aggressively the model updates its parameters based on preference feedback.
    • 
-
  • Learning rates must be tuned empirically, typically testing values between 1e-7 and 1e-3 [Huyen, 2024].
    • 
+
  • Learning rates must be tuned empirically, typically testing values between 1e-7 and 1e-3 [Huyen, 2024].
    • 
 
  • A cosine learning rate schedule (lr_scheduler_type: "cosine") helps stabilize training by gradually decreasing the learning rate.
    • 
 
 
      
@@ -1759,7 +1759,7 @@ 
      
-
      7.4.3.3. Vibe Check
      
+
      7.4.3.3. Vibe Check
      
 
      Let’s do a quick “vibe check” of our newly aligned model by testing it with some challenging prompts. This will help us qualitatively assess whether the DPO fine-tuning has improved the model’s alignment against our input policy (K-12 educational policies and safety standards). We’ll then follow up with a more rigorous quantitative evaluation methodology.
      
 
      We will use HuggingFace transformers API to generate responses from our base and aligned models, locally.
      
 
      
@@ -1842,10 +1842,10 @@ 
      
-
      7.4.4. Alignment Evaluation
      
+
      7.4.4. Alignment Evaluation
      
 
      Evaluating alignment improvements presents unique challenges. Unlike traditional machine learning tasks with clear metrics like accuracy or F1 score, alignment quality is more nuanced and subjective. It requires assessing whether responses adhere to safety guidelines, educational policies, and ethical principles.
      
 
      The gold standard for evaluating alignment is human evaluation. Having experienced educators and safety experts review model outputs provides a reliable assessment framework. However, human evaluation is expensive, time-consuming, and difficult to scale. Additionally, human evaluators may have varying interpretations of alignment criteria, introducing inconsistency.
      
-
      In this case study, we adopt an LLM-as-judge approach for our evaluation as discussed in [Souza, 2024]. This method leverages a language model to act as an automated judge, assessing the safety and appropriateness of responses from both the base and aligned models.
      
+
      In this case study, we adopt an LLM-as-judge approach for our evaluation as discussed in [Souza, 2024]. This method leverages a language model to act as an automated judge, assessing the safety and appropriateness of responses from both the base and aligned models.
      
 
      The evaluation methodology summarized in Fig. 7.9 consists of three key components that work together to assess model alignment against our policy:
      
 
        
 
      1. Evaluation Dataset
        
@@ -2393,29 +2393,29 @@ 
        
-
        7.5. Discussion and Conclusions
        
+
        7.5. Discussion and Conclusions
        
 
        LLMs are complex systems and alignment is a challenging problem. In this chapter, we discussed how post-training alignment techniques can be used to align a language model to human preferences. In the case study, we demonstrated how to use DPO to align a language model to a user-provider policy further automating the process via synthetic data generation and LLM-as-judge evaluation. Our approach does serve as a proof of concept, however, several considerations should be taken into account when using this methodology in practice.
        
 
        Synthetic Data Generation
        
-
        LLMs can self improve through synthetic data generation [Huang et al., 2022]. This process helps the LLM learn from its own reasoning and improve its overall reasoning ability without relying on human-annotated data. While LLMs can be powerful tools for generating synthetic data, especially in data-scarce domains, it’s important to recognize the potential pitfalls.
        
-
        One major challenge is data distribution bias, where the synthetic data might not accurately mirror the complexities and nuances of real-world data. This can lead to models trained on this data making inaccurate predictions or exhibiting biases. In our case study, we did observe duplicate responses in the synthetic data. Further, the methodology lacks a systematic approach to evaluate the quality of the synthetic data itself only focusing on evals for the consecutive fine-tuned model. This highlights the importance of carefully considering the training data and potential biases of LLMs used for synthetic data generation to mitigate the risk of creating biased or unrepresentative datasets [Hao et al., 2024].
        
+
        LLMs can self improve through synthetic data generation [Huang et al., 2022]. This process helps the LLM learn from its own reasoning and improve its overall reasoning ability without relying on human-annotated data. While LLMs can be powerful tools for generating synthetic data, especially in data-scarce domains, it’s important to recognize the potential pitfalls.
        
+
        One major challenge is data distribution bias, where the synthetic data might not accurately mirror the complexities and nuances of real-world data. This can lead to models trained on this data making inaccurate predictions or exhibiting biases. In our case study, we did observe duplicate responses in the synthetic data. Further, the methodology lacks a systematic approach to evaluate the quality of the synthetic data itself only focusing on evals for the consecutive fine-tuned model. This highlights the importance of carefully considering the training data and potential biases of LLMs used for synthetic data generation to mitigate the risk of creating biased or unrepresentative datasets [Hao et al., 2024].
        
 
        Our approach does enable a systematic approach to aligning a model to an input policy. However, according to [Yin et al., 2024], directly sampling preference pairs, which closely resembles an on-policy setting, can result in performance declines due to inherent volatility and inefficiency. Therefore, constructing effective preference data to continuously improve LLMs remains a critical research problem.
        
 
        Choice of Base Model
        
 
        The choice of base model is a critical consideration when implementing alignment techniques. In the case study, we selected the smolLM model family due to its efficient architecture and reasonable performance on basic tasks while maintaining relatively low computational requirements. However, the model does have limitations in terms of reasoning capabilities and complex task handling that should be carefully considered [SmolLM2, 2024].
        
 
        Real-world applications need to carefully evaluate the trade-offs between model size/capabilities, and costs. While smaller models like smolLM can be cost-effective for basic alignment experiments, they may not provide the sophisticated reasoning needed for production use cases. The computational and financial costs of training and deploying larger models must be weighed against the required capabilities.
        
-
        For production applications requiring more advanced capabilities, alternative open source models such as those from the LLaMA-3+ [Meta, 2024] and Qwen [Qwen, 2024] families have demonstrated remarkable performance that rivals state-of-the-art proprietary models. These models offer enhanced reasoning abilities and better handling of complex tasks, though at increased computational and financial cost. The choice ultimately depends on specific use case requirements, available resources, and acceptable performance thresholds.
        
+
        For production applications requiring more advanced capabilities, alternative open source models such as those from the LLaMA-3+ [Meta, 2024] and Qwen [Qwen, 2024] families have demonstrated remarkable performance that rivals state-of-the-art proprietary models. These models offer enhanced reasoning abilities and better handling of complex tasks, though at increased computational and financial cost. The choice ultimately depends on specific use case requirements, available resources, and acceptable performance thresholds.
        
 
        Evaluation Methodology
        
-
        The LLM-as-judge evaluation methodology is a powerful tool for assessing model alignment. However, it does have limitations [Chen et al., 2024]. For instance, the judge model may not always be able to accurately evaluate the alignment of the model, especially if the judge model is not aligned with the policy itself. Further, the judge model may be biased towards the policy, leading to overly conservative evaluations. In our case study, we do highlight the fact that our judge was simply focused on the policy-alignment aspect of the responses completely neglecting the quality of the responses themselves, i.e. while our fine-tuned model may be more aligned with the policy than the base model, we actually have no evidence that our model is helpful at all.
        
+
        The LLM-as-judge evaluation methodology is a powerful tool for assessing model alignment. However, it does have limitations [Chen et al., 2024]. For instance, the judge model may not always be able to accurately evaluate the alignment of the model, especially if the judge model is not aligned with the policy itself. Further, the judge model may be biased towards the policy, leading to overly conservative evaluations. In our case study, we do highlight the fact that our judge was simply focused on the policy-alignment aspect of the responses completely neglecting the quality of the responses themselves, i.e. while our fine-tuned model may be more aligned with the policy than the base model, we actually have no evidence that our model is helpful at all.
        
 
        A more robust evaluation approach would combine LLM-based evaluation with human domain experts in a complementary process. The LLM judge could perform initial high-throughput screening of model responses, flagging potential issues and providing preliminary assessments. These results would then be reviewed by human evaluators with relevant domain expertise who can provide nuanced judgment, catch edge cases, and validate the LLM’s evaluations. Additionally, automatic evaluation against standard benchmarks is advised to evaluate general capabilities of the model.
        
 
        DPO Dataset Composition
        
 
        The composition of the DPO dataset also plays a crucial role in model behavior. In preliminary experiments, using only policy-aligned preference data led to an overly apologetic model that was hesitant to provide helpful responses even for benign queries, i.e. the model was overfitting to the policy. In fact, a model that simply refused to provide an useful response and instead apologized would indeed be aligned with the policy and therefore rewarded accordingly. This led to our decision to construct a more well balanced dataset.
        
-
        Blending our policy-focused dataset with the more general-purpose UltraFeedback dataset from Hugging Face H4 [H4, 2024a] dramatically improved results by helping the model maintain helpfulness while learning appropriate safety boundaries. The results reported here reflect this balanced dataset approach.
        
+
        Blending our policy-focused dataset with the more general-purpose UltraFeedback dataset from Hugging Face H4 [H4, 2024a] dramatically improved results by helping the model maintain helpfulness while learning appropriate safety boundaries. The results reported here reflect this balanced dataset approach.
        
 
        The construction of the DPO dataset is perhaps the most critical component of the alignment process. While automated approaches can help scale dataset creation, the involvement of domain experts in dataset construction is highly recommended. Domain experts bring invaluable knowledge about edge cases, nuanced policy interpretations, and real-world usage patterns that may not be captured by synthetic data generation alone. Organizations implementing alignment techniques should consider investing in domain expert involvement during dataset construction as a key success factor.
        
 
        Fine-tuning Process
        
 
        The effectiveness of DPO training can be highly sensitive to various fine-tuning hyperparameters. As we mentioned before, the batch size and the beta parameter are two key parameters that can significantly impact training stability and model behavior. A careful parameter tuning is required to achieve optimal results, which lacked in our case study.
        
 
        One important limitation of our current implementation is that we did not carefully split our user prompts between in-sample data for fine-tuning and out-of-sample data for evaluation. This means our evaluation metrics may be overly optimistic as the fine-tuned model could be memorizing prompts rather than learning generalizable alignment. Future work should implement proper train/test splits to better assess generalization performance while making sure out/in-sample distributions are similar and representative of real-world data.
        
 
 
        
-
        7.6. Citation
        
+
        7.6. Citation
        
 
        CC BY-NC-SA 4.0
        
 
        @misc{tharsistpsouza2024tamingllms,
   author = {Tharsis T. P. Souza},
@@ -2429,7 +2429,7 @@ 
        
-
        7.7. References
        
+
        7.7. References
        
 
        
 
        
 [ABC+4a]
@@ -2440,7 +2440,7 @@ 
        [ABC+4b]
 
        Amanda Askell, Jan Brauner, Adrian Colyer, Benjamin Cullen, David Duvenaud, Richard Ngo, Azalia Mirhoseini, Catherine Olsson, Sam Ringer, Liam Skirvin, Jess Smith, Dawn Song, William Saunders, and Jacob Steinhardt. Alignment faking in large language models: reviews. 2024b. URL: https://assets.anthropic.com/m/24c8d0a3a7d0a1f1/original/Alignment-Faking-in-Large-Language-Models-reviews.pdf.
        
 
        
-
        
+
        
 [BJN+22]
 
        Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. 2022. URL: https://arxiv.org/abs/2204.05862, arXiv:2204.05862.
        
 
        
@@ -2448,15 +2448,15 @@ 
        [BKK+22]
 
        Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: harmlessness from ai feedback. 2022. URL: https://arxiv.org/abs/2212.08073, arXiv:2212.08073.
        
 
        
-
        
+
        
 [Blo23]
 
        NeurIPS Blog. Announcing the neurips 2023 paper awards. 2023. NeurIPS 2023 Awards. URL: https://blog.neurips.cc/2023/12/11/announcing-the-neurips-2023-paper-awards/.
        
 
        
-
        
+
        
 [CCL+24]
 
        Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. Humans or llms as the judge? a study on judgement biases. 2024. URL: https://arxiv.org/abs/2402.10669, arXiv:2402.10669.
        
 
        
-
        
+
        
 [DPHZ23]
 
        Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: efficient finetuning of quantized llms. 2023. URL: https://arxiv.org/abs/2305.14314, arXiv:2305.14314.
        
 
        
@@ -2465,56 +2465,56 @@ 
        (1,2)
 
        Qingxiu Dong, Li Dong, Xingxing Zhang, Zhifang Sui, and Furu Wei. Self-boosting large language models with synthetic preference data. 2024. URL: https://arxiv.org/abs/2410.06961, arXiv:2410.06961.
        
 
        
-
        
+
        
 [Fac24]
 
        Hugging Face. Zephyr. 2024. Zephyr. URL: https://huggingface.co/HuggingFaceH4/zephyr-7b-alpha.
        
 
        
-
        
+
        
 [Fac4c]
 
        Hugging Face. Rlhf. 2024c. RLHF. URL: https://huggingface.co/blog/rlhf.
        
 
        
-
        
+
        
 [Fac4d]
 
        Hugging Face. Trl. 2024d. TRL. URL: https://huggingface.co/docs/trl/en/index.
        
 
        
-
        
+
        
 [FQH+24]
 
        Duanyu Feng, Bowen Qin, Chen Huang, Zheng Zhang, and Wenqiang Lei. Towards analyzing and understanding the limitations of dpo: a theoretical perspective. 2024. URL: https://arxiv.org/abs/2404.04626, arXiv:2404.04626.
        
 
        
-
        
+
        
 [H44a]
 (1,2)
 
        Hugging Face H4. Ultrafeedback binarized dataset. 2024a. A dataset of binary preference data for training language models. URL: https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized.
        
 
        
-
        
+
        
 [H44b]
 
        Hugging Face H4. Hugging face h4. 2024b. Hugging Face H4. URL: https://huggingface.co/HuggingFaceH4.
        
 
        
-
        
+
        
 [HHJ+24]
 
        Shuang Hao, Wenfeng Han, Tao Jiang, Yiping Li, Haonan Wu, Chunlin Zhong, Zhangjun Zhou, and He Tang. Synthetic data in ai: challenges, applications, and ethical implications. 2024. URL: https://arxiv.org/abs/2401.01629, arXiv:2401.01629.
        
 
        
-
        
+
        
 [HLT24]
 
        Jiwoo Hong, Noah Lee, and James Thorne. Orpo: monolithic preference optimization without reference model. 2024. URL: https://arxiv.org/abs/2403.07691, arXiv:2403.07691.
        
 
        
-
        
+
        
 [HDN+24]
 
        Zhenyu Hou, Pengfan Du, Yilin Niu, Zhengxiao Du, Aohan Zeng, Xiao Liu, Minlie Huang, Hongning Wang, Jie Tang, and Yuxiao Dong. Does rlhf scale? exploring the impacts from data, model, and method. 2024. URL: https://arxiv.org/abs/2412.06000, arXiv:2412.06000.
        
 
        
-
        
+
        
 [HSW+21]
 
        Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: low-rank adaptation of large language models. 2021. URL: https://arxiv.org/abs/2106.09685, arXiv:2106.09685.
        
 
        
-
        
+
        
 [HGH+22]
 
        Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. 2022. URL: https://arxiv.org/abs/2210.11610, arXiv:2210.11610.
        
 
        
-
        
+
        
 [Huy24]
 
        Chip Huyen. AI Engineering. O'Reilly Media, Inc., December 2024. ISBN 9781098129095. URL: https://www.oreilly.com/library/view/ai-engineering/9781098129095/.
        
 
        
-
        
+
        
 [KSD+24]
 
        Joshua Kazdan, Rylan Schaeffer, Apratim Dey, Matthias Gerstgrasser, Rafael Rafailov, David L. Donoho, and Sanmi Koyejo. Collapse or thrive? perils and promises of synthetic data in a self-generating world. 2024. URL: https://arxiv.org/abs/2410.16713, arXiv:2410.16713.
        
 
        
@@ -2522,33 +2522,33 @@ 
        [KSY+24]
 
        Seungone Kim, Juyoung Suk, Xiang Yue, Vijay Viswanathan, Seongyun Lee, Yizhong Wang, Kiril Gashteovski, Carolin Lawrence, Sean Welleck, and Graham Neubig. Evaluating language models as synthetic data generators. 2024. URL: https://arxiv.org/abs/2412.03679, arXiv:2412.03679.
        
 
        
-
        
+
        
 [LT24]
 
        AI @ Meta Llama Team. The llama 3 herd of models. 2024. URL: https://arxiv.org/abs/2407.21783, arXiv:2407.21783.
        
 
        
-
        
+
        
 [LWX+24]
 
        Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. On llms-driven synthetic data generation, curation, and evaluation: a survey. 2024. URL: https://arxiv.org/abs/2406.15126, arXiv:2406.15126.
        
 
        
-
        
+
        
 [Met24]
 
        Meta. Meta-llama. 2024. Meta-Llama. URL: https://huggingface.co/meta-llama.
        
 
        
-
        
+
        
 [OWJ+22]
 (1,2,3,4,5,6,7)
 
        Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. 2022. URL: https://arxiv.org/abs/2203.02155, arXiv:2203.02155.
        
 
        
-
        
+
        
 [Qwe24]
 
        Qwen. Qwen. 2024. Qwen. URL: https://huggingface.co/Qwen.
        
 
        
-
        
+
        
 [RSM+24]
 (1,2,3,4)
 
        Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: your language model is secretly a reward model. 2024. URL: https://arxiv.org/abs/2305.18290, arXiv:2305.18290.
        
 
        
-
        
+
        
 [SWD+17]
 
        John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. 2017. URL: https://arxiv.org/abs/1707.06347, arXiv:1707.06347.
        
 
        
@@ -2561,15 +2561,15 @@ 
        [SmolLM2360MI24]
 
        Hugging Face SmolLM2-360M-Instruct. Smollm2-360m-instruct. 2024. 360M parameter instruction-tuned language model, distilled for efficient deployment. URL: https://huggingface.co/HuggingFaceTB/SmolLM2-360M-Instruct.
        
 
        
-
        
+
        
 [Sou24]
 
        Tharsis T. P. Souza. Tamingllms: a framework for evaluating and aligning language models. 2024. URL: https://www.souzatharsis.com/tamingLLMs/notebooks/evals.html.
        
 
        
-
        
+
        
 [SRvERH24]
 
        Márton Szép, Daniel Rueckert, Rüdiger von Eisenhart-Rothe, and Florian Hinterwimmer. A practical guide to fine-tuning language models with limited data. 2024. URL: https://arxiv.org/abs/2411.09539, arXiv:2411.09539.
        
 
        
-
        
+
        
 [TMS+23]
 
        Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: open foundation and fine-tuned chat models. 2023. URL: https://arxiv.org/abs/2307.09288, arXiv:2307.09288.
        
 
        
@@ -2581,7 +2581,7 @@ 
        [WYG+24]
 
        Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. Meta-rewarding language models: self-improving alignment with llm-as-a-meta-judge. 2024. URL: https://arxiv.org/abs/2407.19594, arXiv:2407.19594.
        
 
        
-
        
+
        
 [XFG+24]
 
        Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for llm alignment? a comprehensive study. 2024. URL: https://arxiv.org/abs/2404.10719, arXiv:2404.10719.
        
 
        
diff --git a/tamingllms/_build/html/notebooks/evals.html b/tamingllms/_build/html/notebooks/evals.html
index 8bec466..fbf2196 100644
--- a/tamingllms/_build/html/notebooks/evals.html
+++ b/tamingllms/_build/html/notebooks/evals.html
@@ -240,7 +240,7 @@
           
        
             
   
        
-
        5. The Evals Gap
        
+
        5. The Evals Gap
        
 
        
 
        It doesn’t matter how beautiful your theory is, 
        
 it doesn’t matter how smart you are. 
        
@@ -250,49 +250,49 @@ 
        
 
        Contents
        
         
 
 
        
-
        5.1. Introduction
        
+
        5.1. Introduction
        
 
        The advent of LLMs marks a pivotal shift in the landscape of software development and evaluation. Unlike traditional software systems, where deterministic outputs are the norm, LLMs introduce a realm of non-deterministic and generative behaviors that challenge conventional software engineering testing paradigms. This shift is not merely a technical evolution but a fundamental transformation in how we conceive, build, and assess software products.
        
 
        For those entrenched in traditional methodologies, the transition to LLM-driven systems may seem daunting. However, ignoring this change is not an option. The reliance on outdated testing frameworks that fail to account for the probabilistic nature of LLMs will inevitably lead to significant setbacks.
        
 
        To overcome these challenges, it is imperative to embrace the complexities of LLMs with a proactive mindset. This involves developing robust evaluation frameworks up-front, fostering a product development culture of continuous change, learning and adaptation.
        
 
        
 
        
-
        5.2. Non-Deterministic Generative Machines
        
+
        5.2. Non-Deterministic Generative Machines
        
 
        One of the most fundamental challenges when building products with Large Language Models (LLMs) is their generative and non-deterministic nature. Unlike traditional software systems where the same input reliably produces the same output, LLMs can generate novel text that may not exist in their training data, and produce different responses each time they’re queried - even with identical prompts and input data. This behavior is both a strength and a significant engineering challenge and product challenge.
        
 
        When you ask an LLM the same question multiple times, you’ll likely get different responses. This isn’t a bug - it’s a fundamental feature of how these models work. The “temperature” parameter, which controls the randomness of outputs, allows models to be creative and generate diverse responses. However, this same feature makes it difficult to build reliable, testable systems.
        
 
        Consider a financial services company using LLMs to generate investment advice. The non-deterministic nature of these models means that:
        
@@ -427,7 +427,7 @@ 
        
-
        5.3. Emerging Properties
        
+
        5.3. Emerging Properties
        
 
        Beyond their non-deterministic nature, LLMs present another fascinating characteristic: emergent abilities that spontaneously arise as models scale up in size. These abilities - from basic question answering to complex reasoning - aren’t explicitly programmed but rather emerge “naturally” as the models grow larger and are trained on more data. This makes evaluation fundamentally different from traditional software testing, where capabilities are explicitly coded and can be tested against pre-defined specifications.
        
 
        Fig. 5.1 provides a list of emergent abilities of large language models and the scale. The relationship between model scale and emergent abilities follows a fascinating non-linear pattern. Below certain size thresholds, specific abilities may be completely absent from the model - it simply cannot perform certain tasks, no matter how much you try to coax them out. However, once the model reaches critical points in its scaling journey, these abilities can suddenly manifest in what researchers call a phase transition - a dramatic shift from inability to capability. This unpredictable emergence of capabilities stands in stark contrast to traditional software development, where features are deliberately implemented and can be systematically tested.
        
 
        
@@ -439,7 +439,7 @@ 
        
-
        5.4. Problem Statement
        
+
        5.4. Problem Statement
        
 
        Consider a practical example that illustrates these challenges: building a Math AI tutoring system for children powered by an LLM. In traditional software development, you would define specific features (like presenting math problems or checking answers) and write tests to verify each function. But with LLMs, you’re not just testing predefined features - you’re trying to evaluate emergent capabilities like adapting explanations to a child’s level, maintaining engagement through conversational learning, and providing age-appropriate safety-bound content.
        
 
        This fundamental difference raises critical questions about evaluation:
        
 
          
@@ -489,7 +489,7 @@ 
          
-
          5.5. Evals Design
          
+
          5.5. Evals Design
          
 
          First, it’s important to make a distinction between evaluating an LLM versus evaluating an LLM-based application. While the latter offers foundation capabilities and are typically general-purpose, the former is more specific and tailored to a particular use case. Here, we define an LLM-based application as a system that uses one or more LLMs to perform a specific task. More specifically, an LLM-based application is the combination of one or more LLM models, their associated prompts and parameters to solve a particular business problem.
          
 
          That differentiation is important because it changes the scope of evaluation. LLMs are usually evaluated based on their capabilities, which include things like language understanding, reasoning and knowledge. LLM-based applications, instead, should be evaluated based on their end-to-end functionality, performance, and how well they meet business requirements. That distinction has key implications for the design of evaluation systems:
          
 
            
@@ -576,7 +576,7 @@ 
            
-
            5.5.1. Conceptual Overview
            
+
            5.5.1. Conceptual Overview
            
 
            Fig. 5.2 demonstrates a conceptual design of key components of LLM Application evaluation.
            
 
            
 Conceptual Overview
@@ -657,7 +657,7 @@ 
            
-
            5.5.2. Design Considerations
            
+
            5.5.2. Design Considerations
            
 
            The design of an LLM application evaluation system depends heavily on the specific use case and business requirements. Here we list important questions for planning an LLM application evaluation system pertaining to each of the key components previously introduced:
            
 
            1. Examples (Input Dataset):
            
 
              
@@ -742,13 +742,13 @@ 
              
-
              5.6. Metrics
              
+
              5.6. Metrics
              
 
              The choice of metric depends on the specific task and desired evaluation criteria. However, one can categorize metrics into two broad categories: intrinsic and extrinsic.
              
 
                
 
              • Intrinsic metrics focus on the model’s performance on its primary training objective, which is typically to predict the next token in a sequence.  Perplexity is a common intrinsic metric that measures how well the model predicts a given sample of text.
                • 
 
              • Extrinsic metrics assess the model’s performance on various downstream tasks, which can range from question answering to code generation.  These metrics are not directly tied to the training objective, but they provide valuable insights into the model’s ability to generalize to real-world applications.
                • 
 
              
-
              Here, we are particularly interested in extrinsic metrics, since we are evaluating LLM-based applications.
              
+
              Here, we are particularly interested in extrinsic metrics, since we are evaluating LLM-based applications rather than base LLM models.
              
 
              Another way to think about metrics is in terms of the type of the task we evaluate:
              
 
                
 
              1. Discriminative Task:
                
@@ -1052,9 +1052,9 @@ 
                5.7. Evaluators
                
+
                5.7. Evaluators
                
 
                
-
                5.7.1. Model-Based Evaluation
                
+
                5.7.1. Model-Based Evaluation
                
 
                Traditional metrics like BLEU or ROUGE often fall short in capturing the nuanced, contextual, and creative outputs of LLMs. As an alternative we can consider a “Model-based evaluation” approach. A common approach is to use an LLM as a judge. This is an approach that leverages language models themselves to assess the quality of outputs from other language models. This method involves using a model (often a more capable one) to act as an automated judge, evaluating aspects like accuracy, coherence, and relevance of generated content. Unlike traditional metrics that rely on exact matching or statistical measures, model-based evaluation can capture nuanced aspects of language and provide more contextual assessment.
                
 
                As discussed in the paper [Li et al., 2024], LLM-based evaluation approaches generally fall into two main categories:
                
 
                  
@@ -1294,7 +1294,7 @@ 
                  
-
                  5.7.2. Evaluating Evaluators
                  
+
                  5.7.2. Evaluating Evaluators
                  
 
                  We have discussed how LLMs can be used to evaluate LLM-based aplications. However, how can we evaluate the performance of LLMs that evaluate other LLMs? This is the question that meta evaluation aims to answer. Clearly, the discussion can become quite meta as we need to evaluate the performance of the evaluator to evaluate the performance of the evaluated model. However, one can make a case for two general options:
                  
 
                    
 
                  1. Use a gold-standard dataset that is used to evaluate the performance of LLM evaluators using a “metrics-based” approach.
                    2. 
@@ -1338,7 +1338,7 @@ 
                    
-
                    5.8. Benchmarks and Leaderboards
                    
+
                    5.8. Benchmarks and Leaderboards
                    
 
                    Benchmarks act as standardized tests for LLMs, evaluating their performance across a spectrum of tasks. These tasks simulate real-world applications such as answering questions, generating coherent text, solving mathematical problems, or even writing computer code. They also assess more abstract qualities like fairness, robustness, and cultural understanding.
                    
 
                    Benchmarks can be thought as comprehensive “exams” that probe different “subjects” in order to certify an LLM. They help researchers and developers compare models systematically, in a way LLM performance is comparable while enabling the identification of emergent behaviors or capabilities as models evolve in scale and sophistication.
                    
 
                    The history of LLM benchmarks reflects the evolving priorities of artificial intelligence research, starting with foundational tasks and moving toward complex, real-world challenges. It began in 2018 with the introduction of GLUE(General Language Understanding Evaluation) [Wang et al., 2019], which set a new standard for evaluating natural language understanding. GLUE measured performance on tasks like sentiment analysis and textual entailment, providing a baseline for assessing the fundamental capabilities of language models. A year later, SuperGLUE [Wang et al., 2019] expanded on this foundation by introducing more nuanced tasks that tested reasoning and language comprehension at a deeper level, challenging the limits of models like BERT and its successors.
                    
@@ -1348,7 +1348,7 @@ 
                    [Face, 2024] Leaderboard stands out for its transparency and accessibility in the open-source community. This leaderboard evaluates a wide range of LLMs across diverse tasks, including general knowledge, reasoning, and code-writing. Its commitment to reproducibility ensures that results are verifiable, enabling researchers and practitioners to replicate findings. By focusing on open-source models, it democratizes AI research and fosters innovation across communities, making it a valuable resource for both academics and industry professionals.
                    
 
                    The Chatbot Arena (2024) Leaderboard (an evolution of LMSYS)[Chiang et al., 2024] takes an alternative approach by measuring real-world performance through direct model comparisons. Its evaluation format compares models in live conversations, with human judges providing qualitative assessments. This methodology has gathered over 200,000 human evaluations, offering specific insights into practical model performance. The emphasis on interactive capabilities makes it relevant for developing user-facing applications like virtual assistants and chatbots.
                    
 
                    The AlpacaEval [Dubois et al., 2024] and MT-Bench [Zheng et al., 2023] Leaderboards implement automated evaluation using GPT-4 to assess model performance in multi-turn conversations. This approach enables consistent assessment of dialogue capabilities while reducing human bias. Their methodology measures key aspects of conversational AI, including contextual understanding and response consistency across multiple exchanges.
                    
-
                    An important recent development was the release of Global-MMLU [Singh et al., 2024], an improved version of MMLU with evaluation coverage across 42 languages. This open dataset, built through collaboration between Argilla, the Hugging Face community, and researchers from leading institutions like Cohere For AI, Mila, MIT, and others, represents a significant step toward more inclusive multilingual LLM evaluation. Over 200 contributors used Argilla to annotate MMLU questions, revealing that 85% of questions requiring specific cultural knowledge were Western-centric. The newly released dataset is divided into two key subsets: Culturally Agnostic questions that require no specific regional or cultural knowledge, and Culturally Sensitive questions that depend on dialect, cultural, or geographic knowledge. With high-quality translations available for 25 languages, Global-MMLU enables better understanding of LLM capabilities and limitations across different languages and cultural contexts.
                    
+
                    An important recent development was the release of Global-MMLU [Singh et al., 2024], an improved version of MMLU with evaluation coverage across 42 languages. This open dataset, built through collaboration between Argilla, the Hugging Face community, and researchers from leading institutions like Cohere For AI, Mila, MIT, and others, represents a significant step toward more inclusive multilingual LLM evaluation. Over 200 contributors used Argilla to annotate MMLU questions, revealing that 85% of questions requiring specific cultural knowledge were Western-centric. The newly released dataset is divided into two key subsets: Culturally Agnostic questions that require no specific regional or cultural knowledge, and Culturally Sensitive questions that depend on dialect, cultural, or geographic knowledge. With high-quality translations available for 25 languages, Global-MMLU enables better understanding of LLM capabilities and limitations across different languages and cultural contexts.
                    
 
                    A major challenge with these leaderboards and benchmarks is test set contamination - when test data ends up in newer models’ training sets, rendering the benchmarks ineffective. While some benchmarks try to address this through crowdsourced prompts and evaluations from humans or LLMs, these approaches introduce their own biases and struggle with difficult questions. LiveBench [White et al., 2024] represents a novel solution, designed specifically to be resilient to both contamination and evaluation biases. As the first benchmark with continuously updated questions from recent sources, automated objective scoring, and diverse challenging tasks across multiple domains, LiveBench maintains its effectiveness even as models improve. Drawing from recent math competitions, research papers, news, and datasets, it creates contamination-free versions of established benchmark tasks. Current results show even top models achieving below 70% accuracy, demonstrating LiveBench’s ability to meaningfully differentiate model capabilities. With monthly updates and an open collaborative approach, LiveBench aims to provide sustained value for model evaluation as the field advances.
                    
 
                    Another notable benchmark is ZebraLogic [Lin et al., 2024], which evaluates logical reasoning capabilities of LLMs through Logic Grid Puzzles - a type of Constraint Satisfaction Problem [Brailsford et al., 1999] commonly found in tests like the LSAT. These puzzles require assigning unique values to N houses across M different features based on given clues, demanding strategic reasoning and deduction to arrive at a unique correct solution. The benchmark’s programmatically generated puzzles range from 2x2 to 6x6 in size and test LLMs using one-shot examples with reasoning steps. While humans can solve these puzzles through strategic methods like reductio ad absurdum and elimination, LLMs demonstrate significant limitations in this type of logical reasoning. Even the best-performing model, Claude 3.5 Sonnet, only achieves 33.4% accuracy across all puzzles and 12.4% on hard puzzles, with smaller models (7-10B parameters) solving less than 1% of hard puzzles as of December 2024. These results reveal critical gaps in LLMs’ capabilities around counterfactual thinking, reflective reasoning, structured memorization, and compositional generalization.
                    
 
                    A significant shift in AI evaluation came with the launch of the The Alignment Research Center (ARC) Prize [Chollet, 2024] by ARC Prize Inc., a non-profit for the public advancement of open artificial general intelligence. Hosted by Mike Knoop (Co-founder, Zapier) and François Chollet (Creator of ARC-AGI, Keras), this prize represents a paradigm shift in how we evaluate language models. Rather than focusing on narrow performance metrics, the ARC Prize assesses what it calls “cognitive sufficiency” - a model’s ability to generate meaningful insights and tackle open-ended challenges. This new way to think about LLM evaluation emphasizes creative thinking, sophisticated reasoning, and the capacity to make genuinely useful contributions to human knowledge as we seek to define and measure what it means to achieve AGI (Artificial General Intelligence).
                    
@@ -1383,16 +1383,16 @@ 
                    [Chollet, 12/08/2024]. While deep learning has significantly advanced in recent years, pure deep learning approaches perform poorly on the ARC-AGI benchmark. This is because traditional deep learning relies on relating new situations to those encountered during training and lacks the ability to adapt or recombine knowledge for entirely new tasks. ARC Prize 2024 spurred the development of novel AGI reasoning techniques, leading to a significant increase in the state-of-the-art score on the ARC-AGI private evaluation set from 33% in 2023 to 55.5% in 2024. A key takeaway is that algorithmic improvements, rather than massive computational resources, may be key to exceeding the target score for the ARC-AGI benchmark.
                    
 
                    In addition to the benchmarks discussed above, a growing set of domain-specific benchmarks is emerging to help evaluate LLMs in specific verticals, including:
                    
 
                      
-
                    • FinBench [Zhang et al., 2024]: Evaluates LLMs in the financial domain, covering tasks such as terminology understanding, temporal reasoning, future forecasting, scenario planning, and numerical modelling.
                      • 
-
                    • LegalBench [Guha et al., 2023] : Assesses the legal reasoning abilities of LLMs through tasks crowdsourced by legal professionals
                      • 
-
                    • Berkeley Function Leaderboard (BFCL) [Patil et al., 2023]: Evaluates LLMs’ function-calling abilities
                      • 
+
                    • FinBench [Zhang et al., 2024]: Evaluates LLMs in the financial domain, covering tasks such as terminology understanding, temporal reasoning, future forecasting, scenario planning, and numerical modelling.
                      • 
+
                    • LegalBench [Guha et al., 2023] : Assesses the legal reasoning abilities of LLMs through tasks crowdsourced by legal professionals
                      • 
+
                    • Berkeley Function Leaderboard (BFCL) [Patil et al., 2023]: Evaluates LLMs’ function-calling abilities
                      • 
 
                    
 
                    As language models continue to advance in capability and complexity, evaluation frameworks must evolve. Modern benchmarks increasingly incorporate tests for nuanced reasoning, ethical decision-making, and emergent capabilities that weren’t previously measurable. This ongoing evolution reflects a deeper understanding that the true value of language models lies not in achieving high scores on standardized tests with narrow task-specific metrics, but in their ability to meaningfully contribute to human understanding and help solve real-world problems while demonstrating the ability to learn and adapt to new tasks.
                    
 
                
 
                
-
                5.9. Tools
                
+
                5.9. Tools
                
 
                
-
                5.9.1. LightEval
                
+
                5.9.1. LightEval
                
 
                LightEval [Fourrier et al., 2023] is a lightweight framework for evaluation of LLMs across a variety of standard and bespoke metrics and tasks across multiple inference backends via Python SDK and CLI.
                
 
                As a motivating example, consider a scenario where financial data has been extracted from SEC financial filings and require econometric analysis. Tasks like estimating autoregressive models for time series forecasting or conducting hypothesis tests on market efficiency are common in financial analysis. Let’s evaluate how well different models perform on this type of task.
                
 
                First, we need to select a benchmark to assess LLMs capabilities in this domain. MMLU has a sub-benchmark called Econometrics we can use for this task. Table 5.4 shows a sample of the benchmark dataset from MMLU Econometrics. It consists of multiple-choice questions from econometrics and expected answers.
                
@@ -1581,7 +1581,7 @@ 
                [Hugging Face, 2024]. Its integration with the Hugging Face ecosystem and modular architecture make it particularly powerful for evaluating open source models. For further details, visit the official repository [Fourrier et al., 2023].
                
 
                
 
                
-
                5.9.2. LangSmith
                
+
                5.9.2. LangSmith
                
 
                Let’s revisit our evaluation example when we were interested in evaluating the quality of summaries generated by different (smaller and cheaper) LLM models compared to a benchmark model (larger and more expensive). Recal the setup:
                
 
                  
 
                • Benchmark model: gpt-4o
                  • 
@@ -1989,7 +1989,7 @@ 
                  
-
                  5.9.3. PromptFoo
                  
+
                  5.9.3. PromptFoo
                  
 
                  Promptfoo [promptfoo, 2024] is an open-source framework designed for evaluating applications that utilize large language models (LLMs). Key features include:
                  
 
                    
 
                  1. Automated Testing: Promptfoo provides automated testing capabilities, allowing developers to run custom evaluations tailored to their applications.
                    2. 
@@ -2254,7 +2254,7 @@ 
                    Prompt Comparison R
 
                    In conclusion, Promptfoo can serve as an effective LLM application evaluation tool particularly for its ability to decouple several components of the evaluation process. Hence enabling the user to focus on the most important aspects of the evaluation given the particular application and criteria making it a valuable and flexible tool for LLM application development.
                    
 
                
 
                
-
                5.9.4. Comparison
                
+
                5.9.4. Comparison
                
 
                The following table provides a summarized comparative analysis of three open source frameworks for language models evaluation we have discussed: Lighteval, LangSmith, and Promptfoo. Each framework is assessed based on key features such as integration capabilities, customization options, ease of use, and the ability to facilitate human and LLM collaboration.
    @@ -2291,13 +2291,13 @@

    -

    5.10. Conclusion

    +

    5.10. Conclusion

    Language models have fundamentally transformed how software is developed and evaluated. Unlike conventional systems that produce predictable outputs, LLMs generate varied, probabilistic responses that defy traditional testing approaches. While developers accustomed to deterministic systems may find this shift challenging, continuing to rely on legacy testing methods is unsustainable. These frameworks were not designed to handle the inherent variability of LLM outputs and will ultimately prove inadequate.

    Success requires embracing this new paradigm by implementing comprehensive evaluation strategies early - this is the new Product Requirements Document (PRD) - and cultivating an organizational mindset focused on iteration, experimentation and growth.

    The shift from traditional software testing to LLM evaluation is not just a change in tools but a transformation in mindset. Those who recognize and adapt to this shift will lead the way in harnessing the power of LLMs. However, the cost of inaction is not just technological stagnation, but potential business failure.

    -

    5.11. Citation

    +

    5.11. Citation

    CC BY-NC-SA 4.0

    @misc{tharsistpsouza2024tamingllms,
   author = {Tharsis T. P. Souza},
@@ -2311,7 +2311,7 @@ 
    
-
    5.12. References
    
+
    5.12. References
    
 
    
 
    
 [ALB+24]
@@ -2374,7 +2374,7 @@ 
    (1,2)
 
    Clémentine Fourrier, Nathan Habib, Thomas Wolf, and Lewis Tunstall. Lighteval: a lightweight framework for llm evaluation. 2023. URL: https://github.com/huggingface/lighteval.
    
 
    
-
    
+
    
 [GNH+23]
 
    Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N. Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, Jessica Wu, Joe Nudell, Joel Niklaus, John Nay, Jonathan H. Choi, Kevin Tobia, Margaret Hagan, Megan Ma, Michael Livermore, Nikon Rasumov-Rahe, Nils Holzenberger, Noam Kolt, Peter Henderson, Sean Rehaag, Sharad Goel, Shang Gao, Spencer Williams, Sunny Gandhi, Tom Zur, Varun Iyer, and Zehua Li. Legalbench: a collaboratively built benchmark for measuring legal reasoning in large language models. 2023. URL: https://arxiv.org/abs/2308.11462, arXiv:2308.11462.
    
 
    
@@ -2407,7 +2407,7 @@ 
    [LHE22]
 
    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: measuring how models mimic human falsehoods. 2022. URL: https://arxiv.org/abs/2109.07958, arXiv:2109.07958.
    
 
    
-
    
+
    
 [PZWG23]
 
    Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: large language model connected with massive apis. arXiv preprint arXiv:2305.15334, 2023.
    
 
    
@@ -2419,7 +2419,7 @@ 
    [Ras24]
 
    Sebastian Raschka. Build A Large Language Model (From Scratch). Manning, 2024. ISBN 978-1633437166. URL: https://www.manning.com/books/build-a-large-language-model-from-scratch.
    
 
    
-
    
+
    
 [SRF+24]
 
    Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David I. Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Wei-Yin Ko, Madeline Smith, Antoine Bosselut, Alice Oh, Andre F. T. Martins, Leshem Choshen, Daphne Ippolito, Enzo Ferrante, Marzieh Fadaee, Beyza Ermis, and Sara Hooker. Global mmlu: understanding and addressing cultural and linguistic biases in multilingual evaluation. 2024. URL: https://arxiv.org/abs/2412.03304, arXiv:2412.03304.
    
 
    
@@ -2447,7 +2447,7 @@ 
    [YYH+24]
 
    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024.
    
 
    
-
    
+
    
 [ZCL24]
 
    Zhihan Zhang, Yixin Cao, and Lizi Liao. Finbench: benchmarking LLMs in complex financial problem solving and reasoning. 2024. URL: https://openreview.net/forum?id=AeGrf1uY0p.
    
 
    
diff --git a/tamingllms/_build/html/notebooks/local.html b/tamingllms/_build/html/notebooks/local.html
index ccc49f3..402fe9c 100644
--- a/tamingllms/_build/html/notebooks/local.html
+++ b/tamingllms/_build/html/notebooks/local.html
@@ -182,11 +182,13 @@
                 
                   
  • Introduction
    • 
                 
-                  
  • Local Models
    • 
+                  
  • Local Models Considerations
    • 
                 
-                  
  • Local Tools
    • 
+                  
  • Tools for Local LLM Deployment
    • 
                 
-                  
  • Case Study: Private Code Documentation Generator
    • 
+                  
  • Case Study: The Effect of Quantization on LLM Performance
    • 
+                
+                  
  • Conclusion
    • 
                 
                   
  • Citation
    • 
                 
@@ -225,84 +227,69 @@
           
    
             
   
    
-
    8. Breaking Free from Cloud Providers
    
+
    8. Breaking Free from Cloud Providers
    
 
    
-
    I want to break free, I want to break free
    
-
    —Queen
    
+
    Freedom is something that dies unless it’s used.
    
+
    —Hunter S. Thompson
    
 
    
 
 
    
-
    8.1. Introduction
    
+
    8.1. Introduction
    
 
    Running LLMs locally versus using cloud APIs offers several important advantages.
    
 
    Privacy-sensitive data processing is one of the primary reasons for running LLMs locally. Organizations handling medical records must comply with HIPAA regulations that require data to remain on-premise. Similarly, businesses processing confidential documents and intellectual property, as well as organizations subject to GDPR and other privacy regulations, need to maintain strict control over their data processing pipeline.
    
 
    Cost considerations become significant when operating at scale. Organizations running high-volume applications can face prohibitive API costs with cloud-based solutions. Development and testing environments that require frequent model interactions, educational institutions supporting multiple student projects, and research initiatives involving extensive model experimentation can all achieve substantial cost savings through local deployment.
    
 
    Applications with stringent latency requirements form another important category. Real-time systems where network delays would be unacceptable, edge computing scenarios demanding quick responses, and interactive applications requiring sub-second performance all benefit from local deployment. This extends to embedded systems in IoT devices where cloud connectivity might be unreliable or impractical.
    
 
    Finally, local deployment enables deeper customization and fine-tuning capabilities. Organizations can perform specialized domain adaptation through model modifications, experiment with different architectures and parameters, and integrate models with proprietary systems and workflows. This flexibility is particularly valuable for developing novel applications that require direct model access and manipulation.
    
 
    
-
    
-
    8.2. Local Models
    
-
    
-
    8.2.1. Notes of Caution
    
-
    When using local LLM models versus widely known private large language models:
    
-
      
-
    1. Performance: Local LLMs often have lower performance compared to large private models due to size and training limitations.
      2. 
-
    2. Resource requirements: Running local LLMs can be computationally intensive, requiring significant CPU/GPU resources.
      3. 
-
    3. Limited capabilities: Local models particularly if smallmay struggle with complex tasks or specialized knowledge that larger models handle well.
      4. 
-
    4. Potential Output Unreliability: Local models may produce less consistent or stable outputs - see Chapter
      5. 
-
    5. Limited context window: Local models often have smaller context windows, limiting their ability to process long inputs.
      6. 
-
    
-
    Always evaluate the trade-offs between using local LLMs and private models based on your specific use case and requirements. We highly recommend extensively testing your local LLM before productionizing an end-to-end application.
    
+
    
+
    8.2. Local Models Considerations
    
 
    
-
    
-
    
-
    8.3. Local Tools
    
+
    
+
    8.3. Tools for Local LLM Deployment
    
 
    Local LLM deployment tools generally fall into two categories: inference-focused tools that prioritize performance and programmability for technical users requiring production-grade deployments, and user interface (UI) tools that emphasize accessibility through graphical interfaces for non-technical users, trading some performance for ease of use and broader adoption. In the following sections we will explore some of these tools discussing their features, capabilities, and trade-offs.
    
 
    
-
    8.3.1. Serving Models
    
+
    8.3.1. Serving Models
    
 
    Before exploring specific tools, it’s important to understand what “serving” an LLM model means in practice. Serving refers to the process of making a trained language model available for inference. At a high level, this involves setting up the infrastructure needed to accept and process input text and generate responses while efficiently managing system resources. The serving process involves several key responsibilities:
    
 
      
 
    1. Model Loading and Initialization
      2. 
@@ -355,10 +342,10 @@ 
      
-
      8.3.1.1. LLama.cpp
      
-
      LLama.cpp [Gerganov and contributors, 2024a] is an MIT-licensed open source optimized implementation of the LLama model architecture designed to run efficiently on machines with limited memory.
      
+
      8.3.1.1. LLama.cpp
      
+
      LLama.cpp [Gerganov and contributors, 2024a] is an MIT-licensed open source optimized implementation of the LLama model architecture designed to run efficiently on machines with limited memory.
      
 
      Originally developed by Georgi Gerganov and today counting with hundreds of contributors, this C/C++ version provides a simplified interface and advanced features that allow language models to run without overwhelming systems. With the ability to run in resource-constrained environments, LLama.cpp makes powerful language models more accessible and practical for a variety of applications.
      
-
      In its “Manifesto” [Gerganov and others, 2023], the author sees significant potential in bringing AI from cloud to edge devices, emphasizing the importance of keeping development lightweight, experimental, and enjoyable rather than getting bogged down in complex engineering challenges. The author states a vision that emphasizes maintaining an exploratory, hacker-minded approach while building practical edge computing solutions highlighting the following core principles:
      
+
      In its “Manifesto” [Gerganov and others, 2023], the author sees significant potential in bringing AI from cloud to edge devices, emphasizing the importance of keeping development lightweight, experimental, and enjoyable rather than getting bogged down in complex engineering challenges. The author states a vision that emphasizes maintaining an exploratory, hacker-minded approach while building practical edge computing solutions highlighting the following core principles:
      
 
        
 
      • “Will remain open-source”
        • 
 
      • Focuses on simplicity and efficiency in codebase
        • 
@@ -373,7 +360,7 @@ 
        [Gerganov and contributors, 2024b] is the latest model format used by LLama.cpp, replacing the older GGML format. It was designed specifically for efficient inference of large language models on consumer hardware. The key features that make GGUF particularly valuable include [IBM Think, 2024]:
        
+
        GGUF (GPT-Generated Unified Format) [Gerganov and contributors, 2024b] is the latest model format used by LLama.cpp, replacing the older GGML format. It was designed specifically for efficient inference of large language models on consumer hardware. The key features that make GGUF particularly valuable include [IBM Think, 2024]:
        
 
          
 
        • Improved quantization: GGUF supports multiple quantization levels to reduce model size while preserving performance. Common quantization schemes that are supported by GGUF include:
          
 
            
@@ -387,9 +374,9 @@ 
            [Hugging Face, 2024x] and provides a tool (ggml-org/gguf-my-repo) to convert existing models to GGUF format, making it easier for developers to access and deploy optimized versions of popular language models.
            
+
            These capabilities make GGUF models significantly more practical for running LLMs locally compared to full-precision formats, often dramatically reducing memory requirements. Hugging Face hosts a growing collection of pre-converted GGUF models [Hugging Face, 2024x] and provides a tool (ggml-org/gguf-my-repo) to convert existing models to GGUF format, making it easier for developers to access and deploy optimized versions of popular language models.
            
 
            Setup
            
-
            Please follow the instructions on the llama.cpp GitHub repository [Gerganov and contributors, 2024a] to install and compile the library.
            
+
            Please follow the instructions on the llama.cpp GitHub repository [Gerganov and contributors, 2024a] to install and compile the library.
            
 
            Here, we will compile the library from source on a Linux machine with 8 jobs in parallel for enhanced performance (add the -j argument to run multiple jobs in parallel).
            
 
            sudo apt install cmake
 
@@ -397,7 +384,7 @@ 
             --build build --config Release -j 8
 
            
 
            
-
            Python bindings are available through llama-cpp-python [Betlen and contributors, 2024].
            
+
            Python bindings are available through llama-cpp-python [Betlen and contributors, 2024].
            
 
            pip install llama-cpp-python
 
            
 
            
@@ -488,14 +475,14 @@ 
            [Gerganov and contributors, 2024] to constrain the output of the model as demonstrated below. This is the same technique Ollama uses, a similar approach to Outlines’ to generate structured outputs from LLMs. See Wrestling with Structured Outputs Chapter for more details.
            
+
            It is worth noting Llama.cpp provides a way to use grammars [Gerganov and contributors, 2024] to constrain the output of the model as demonstrated below. This is the same technique Ollama uses, a similar approach to Outlines’ to generate structured outputs from LLMs. See Wrestling with Structured Outputs Chapter for more details.
            
 
            ./build/bin/llama-cli -m ./models/qwen2.5-0.5b-instruct-q8_0.gguf --grammar-file grammars/json.gbnf -p 'Request: schedule a call at 8pm; Command:'
 
 # {"appointmentTime": "8pm", "appointmentDetails": "schedule a a call"}
 
            
 
            
 
            Python
            
-
            A handy Python binding [Betlen and contributors, 2024] is available for LLama.cpp, which by default returns chat completions in OpenAI’s API chat format as below. The package is very comprehensive supporting JSON Mode, function calling, multi-modal models and more.
            
+
            A handy Python binding [Betlen and contributors, 2024] is available for LLama.cpp, which by default returns chat completions in OpenAI’s API chat format as below. The package is very comprehensive supporting JSON Mode, function calling, multi-modal models and more.
            
 
            
 
            
 
            MODEL_PATH = "./models/qwen2.5-0.5b-instruct-q8_0.gguf"
@@ -550,8 +537,8 @@ 
            
-
            8.3.1.2. Llamafile
            
-
            Developed by Occupy Wall Street’s former activist, Justine Tunney, Llamafile [Mozilla Ocho, 2024] is an Appache 2.0 licensed open source tool that combines the power of LLama.cpp with Cosmopolitan Libc, a universal C standard library that allows creating portable executables compatible with multiple operating systems.
            
+
            8.3.1.2. Llamafile
            
+
            Developed by Occupy Wall Street’s former activist, Justine Tunney, Llamafile [Mozilla Ocho, 2024] is an Appache 2.0 licensed open source tool that combines the power of LLama.cpp with Cosmopolitan Libc, a universal C standard library that allows creating portable executables compatible with multiple operating systems.
            
 
            In this way, Llamafile reduces all the complexity of LLMs to a single executable file (called a “llamafile”) that runs locally without installation. Key advantages of Llamafile over plain Llama.cpp include:
            
 
              
 
            1. Zero Installation/Configuration
              2. 
@@ -575,7 +562,7 @@ 
              [Hugging Face, 2024x]. All you need to do is:
              
+
              A large collection of Llamafiles can be found on HuggingFace [Hugging Face, 2024x]. All you need to do is:
              
 
                
 
              1. Download a llamafile from HuggingFace
                2. 
 
              2. Make the file executable
                3. 
@@ -595,7 +582,7 @@ 
                http://localhost:8080. And we can use it as demonstrated in the previous section.
                
 
    
 
    
-
    8.3.1.3. Ollama
    
+
    8.3.1.3. Ollama
    
 
    Ollama is a lightweight, MIT-licensed open-source tool for running LLMs locally. It provides a simple interface for interacting with a wide range of language models, including popular models like Llama 3.1 and Llama 3.2. Ollama is designed to be easy to install and use, making it a popular choice for developers who want to run LLMs locally without the need for extensive setup or configuration. Ollama’s key advantages include:
    
 
      
 
    1. Model Management
      2. 
@@ -689,7 +676,7 @@ 
      
-
      8.3.1.4. Comparison
      
+
      8.3.1.4. Comparison
      
 
      Each solution offers distinct advantages and tradeoffs that make them suitable for different use cases. At a high-level, Ollama is the easiest to install and use and has become the most popular choice for your average use case, Llamafile is the easiest to distribute and a good choice when portability is a priority, and Llama.cpp is the most customizable and performant solution as summarized in Table 8.1.
    Table 5.6 Comparison of Lighteval, LangSmith, and Promptfoo
    @@ -745,11 +732,11 @@

    -

    8.3.2. UI

    +

    8.3.2. UI

    There is a growing number of UI tools for local LLM deployment that aim at providing a more user-friendly experience. Ranging from closed-source to open-source solutions across a range of features and capabilities. We will discuss LM Studio, Jan, and OpenWebUI.

    -

    8.3.2.1. LM Studio

    -

    LM Studio [LM Studio, 2024] is a closed-source GUI for running LLMs locally. In the context of local deployment, LM Studio positions itself as a more user-friendly, feature-rich solution compared to the other tools. It’s particularly valuable for developers transitioning from cloud APIs to local deployment, and for users who prefer graphical interfaces over command-line tools. Key Features of LM Studio include:

    +

    8.3.2.1. LM Studio

    +

    LM Studio [LM Studio, 2024] is a closed-source GUI for running LLMs locally. In the context of local deployment, LM Studio positions itself as a more user-friendly, feature-rich solution compared to the other tools. It’s particularly valuable for developers transitioning from cloud APIs to local deployment, and for users who prefer graphical interfaces over command-line tools. Key Features of LM Studio include:

    • Model Parameter Customization: Allows adjusting temperature, maximum tokens, frequency penalty, and other settings

    • Chat History: Enables saving prompts for later use

      • @@ -772,7 +759,7 @@

      8.3.2.2. Jan

      +

      8.3.2.2. Jan

      Jan is an open source ChatGPT-alternative that runs local models. Its model’s library contains popular LLMs like Llama, Gemma, Mistral, or Qwen. Key Features of Jan include:

      1. User-Friendly Interface: Run AI models with just a few clicks

        2. @@ -790,7 +777,7 @@

        -

        8.3.2.3. Open WebUI

        +

        8.3.2.3. Open WebUI

        Open WebUI is an open-source web interface designed to enhance the local AI model experience, particularly for Ollama and OpenAI-compatible APIs. It aims to provide enterprise-grade features while maintaining user-friendliness. OpenWebUI’s core features include:

        1. Advanced User Interface

          @@ -830,7 +817,7 @@

          -

          8.3.2.4. Comparison

          +

          8.3.2.4. Comparison

          LM Studio excels at providing individual developers with a smooth transition from cloud APIs to local deployment, offering an intuitive interface and robust API compatibility, however it is closed-source. Jan focuses on simplicity and accessibility, making it ideal for personal use and basic deployments while maintaining open-source benefits. OpenWebUI makes additional features available to enterprise users and teams requiring advanced features like RAG, collaboration tools, and granular access controls, though this may come at the cost of increased complexity and resource requirements. We compare the three tools in Table 8.2.

    Table 8.1 lama.cpp vs Ollama vs Llamafile Comparison
    @@ -897,45 +884,260 @@

    -

    8.4. Case Study: Private Code Documentation Generator

    -
    -

    8.4.1. Objectives

    -

    Private Code Documentation Generator

    -

    Create a system for automatically documenting proprietary codebase -LLM analyzes private source code repositories -Generates comprehensive documentation without exposing code externally -Could implement:

    -

    Automatic function documentation -Architecture diagrams based on code analysis -Security vulnerability scanning -API documentation generation -Code complexity analysis -Best practice suggestions -Integration with local git repositories

    +
    +

    8.4. Case Study: The Effect of Quantization on LLM Performance

    +

    This case study examines how different quantization levels affect the performance of language models running locally. Quantization is a crucial technique for reducing model size and improving inference speed, but it comes with potential tradeoffs in model quality. Understanding these tradeoffs is essential for practitioners deploying LLMs in resource-constrained environments.

    +

    Using the Qwen 2.5 0.5B model as our baseline, we’ll compare four variants:

    +
      +

    • The base fp16 model (no quantization)

      • +

    • Q2_K quantization (highest compression, lowest precision)

      • +

    • Q4_K quantization (balanced compression/precision)

      • +

    • Q6_K quantization (lowest compression, highest precision)

      • +
    +

    The analysis will focus on three key metrics:

    +
      +

    1. Perplexity - to measure how well the model predicts text

      2. +

    2. KL divergence - to quantify differences in probability distributions against base model

      3. +

    3. Prompt (tokens/second) - to assess impact in thoughput

      4. +
    +

    While we will focus on the Qwen 2.5 0.5B model, the same analysis can be applied to other models. These insights will help practitioners make informed decisions about quantization strategies based on their specific requirements for model size, speed, and accuracy.

    +
    +

    8.4.1. Prompts Dataset

    +

    To evaluate the impact of quantization on model performance, we first need a set of prompts that will serve as input data for our experiments. We’ll construct a dataset from WikiText-2 [Salesforce, 2024], which contains Wikipedia excerpts.

    +

    In our experiments, we will use a total of NUM_PROMPTS prompts that vary in length from MIN_PROMPT_LENGTH to MAX_PROMPT_LENGTH tokens. Using a fixed set of prompts ensures consistent evaluation across model variants and enables direct comparison of metrics like perplexity and throughput.

    +
    +
    +
    NUM_PROMPTS = 100
+MIN_PROMPT_LENGTH = 100
+MAX_PROMPT_LENGTH = 1000
+
    +
    +
    +
    +
    +
    +
    import datasets
+input_texts_raw = datasets.load_dataset("Salesforce/wikitext", "wikitext-2-raw-v1", split="train")["text"]
+
    +
    +
    +
    +
    +
    +
    input_texts = [s for s in input_texts_raw if s!='' and len(s) > MIN_PROMPT_LENGTH and len(s) < MAX_PROMPT_LENGTH][:NUM_PROMPTS]
+
    +
    +
    +
    +
    +
    +
    len(input_texts)
+
    +
    +
    +
    +
    100
+
    +
    +
    +
    +
    +
    +
    print(input_texts[1])
+
    +
    +
    +
    +
     The game began development in 2010 , carrying over a large portion of the work done on Valkyria Chronicles II . While it retained the standard features of the series , it also underwent multiple adjustments , such as making the game more forgiving for series newcomers . Character designer Raita Honjou and composer Hitoshi Sakimoto both returned from previous entries , along with Valkyria Chronicles II director Takeshi Ozawa . A large team of writers handled the script . The game 's opening theme was sung by May 'n . 
+
    +
    +
    +
    +
    +
    +
    with open('../data/local/prompts.txt', 'w') as f:
+    for text in input_texts:
+        # Escape any quotes in the text and wrap in quotes
+        escaped_text = text.replace('"', '\\"')
+        f.write(f'"{escaped_text}"\n')
+
    +
    +
    +
    +
    +
    +

    8.4.2. Quantization

    +

    We can quantize a model using the llama-quantize CLI. For instance, to quantize the Qwen 2.5 0.5B model to Q4_K, we can run the following command:

    +
    ./llama-quantize -m ./models/qwen2.5-0.5b-instruct-fp16.gguf ./models/qwen2.5-0.5b-instruct-q8_0.gguf Q4_K
+
    +
    +

    Table 8.3 describes the key quantization levels used in this study [Hugging Face, 2024w], where:

    +
      +

    • q is the quantized value

      • +

    • block_scale is the scaling factor for the block (with bit width in parentheses)

      • +

    • block_min is the block minimum value (with bit width in parentheses)

      • +
    +

    Table 8.2 LM Studio vs Jan vs OpenWebUI Comparison
    + + + + + + + + + + + + + + + + + + + + + + + + + +
    Table 8.3 Quantization Levels

    Quantization

    Description

    Bits per Weight

    Formula

    Q2_K

    2-bit quantization with 16 weights per block in 16-block superblocks

    2.5625

    w = q * block_scale(4-bit) + block_min(4-bit)

    Q4_K

    4-bit quantization with 32 weights per block in 8-block superblocks

    4.5

    w = q * block_scale(6-bit) + block_min(6-bit)

    Q6_K

    6-bit quantization with 16 weights per block in 16-block superblocks

    6.5625

    w = q * block_scale(8-bit)

    +

    Each quantization level represents a different tradeoff between model size and accuracy. Q2_K provides the highest compression but potentially lower accuracy, while Q6_K maintains better accuracy at the cost of larger model size. The K-variants use more sophisticated block structures and scaling compared to legacy quantization methods.

    +

    The base model is 16-bit standard IEEE 754 half-precision floating-point number.

    -
    -

    8.4.2. Model Setup

    -

    llamacpp on Qwen2.5-Coder-3B-Instruct-GGUF

    -

    https://huggingface.co/Qwen/Qwen2.5-Coder-3B-Instruct-GGUF

    -

    https://qwenlm.github.io/blog/qwen2.5-coder-family/ -https://qwen.readthedocs.io/en/latest/benchmark/speed_benchmark.html

    +
    +

    8.4.3. Benchmarking

    +

    We will measure quantized model “quality” by means of perplexity and KL Divergence. For performance evaluation, we will report prompt throughput in tokens per second.

    +

    Perplexity

    +

    Perplexity is a common metric for evaluating language models that measures how well a model predicts a sample of text. Lower perplexity indicates better prediction (less “perplexed” by the text).

    +

    Recall that for a sequence of N tokens, perplexity is defined as:

    +
    +\[ \text{PPL(B, X)} = \exp\left(-\frac{1}{N}\sum_{i=1}^{N} \log_2 P(x_i|x_{<i})\right) \]
    +

    where:

    +
      +

    • \(P(x_i|x_{<i})\) is the probability the model \(B\) with tokenized sequence \(X\) assigns to token \(x_i\) given the previous tokens \(x_{<i}\)

      • +

    • \(N\) is the total number of tokens in the sequence

      • +
    +

    To evaluate quantization quality, we first calculate perplexity scores for both the base model and quantized variants. We then compute the ratio of quantized to base perplexity and average it across all prompt samples as follows:

    +
    +\[ Avg PPL Ratio = \frac{1}{N}\sum_{i=1}^{N} \frac{\text{PPL}_i(Q)}{\text{PPL}_i(\text{base})} \]
    +

    We also calculate the correlation between the log perplexities of the quantized and base models:

    +
    +\[ \text{Corr}(\ln(\text{PPL}(Q)), \ln(\text{PPL}(\text{base}))) \]
    +

    These are two simple metrics to evaluate how much worse the quantized model performs on an intrinsic basis which we then can compare to the base model’s perplexities.

    +

    Arguably, KL Divergence is a better metric since we aim at reporting relative performance instead of intrinsic performance.

    +

    KL Divergence

    +

    The Kullback-Leibler (KL) Divergence measures how one probability distribution differs from another reference distribution. For comparing logits between a base model (B) and quantized model (Q), we can calculate the KL divergence as follows:

    +
    +\[ D_{KL}(P||Q) = \sum_{i} P(i) \log \frac{P(i)}{Q(i)} \]
    +

    where:

    +
      +

    • \(P(i)\) and \(Q(i)\) are the softmax probabilities derived from the logits

      • +

    • The sum is taken over all tokens in the vocabulary

      • +
    +

    Implementation

    +

    We will use LLama.cpp’s llama-perplexity CLI to calculate perplexity and KL divergence. The first step is to generate the logits for the base model, which will serve as the reference distribution. For instance, below we pass our input prompts (prompts.txt) and generate the logits for the base model qwen2.5-0.5b-instruct-fp16.gguf which will be saved in logits.kld.

    +
    ./build/bin/llama-perplexity -m ./models/qwen2.5-0.5b-instruct-fp16.gguf --kl-divergence-base ../logits.kld -f ../prompts.txt
+
    +
    +

    Next, we generate KL-Divergence and perplexity stats for quantized model qwen2.5-0.5b-instruct-q2_k.gguf against base model logits logits.kld.

    +
    ./build/bin/llama-perplexity -m ./models/qwen2.5-0.5b-instruct-q2_k.gguf -f ../prompts.txt --kl-divergence-base ../logits.kld --kl-divergence &> ../q2_kresults.txt
+
    +
    +

    We perform this process for each quantization level studied (Q2_K, Q4_K, Q6_K).

    -
    -

    8.4.3. Instructions

    -

    Generate natural language description -Extract key parameters and return values -Identify potential edge cases -Suggest improvements or potential issues -Generate usage examples -Create docstring templates

    +
    +

    8.4.4. Results

    +

    The KL divergence and perplexity results in ppl1, ppl2 provide insights into model quality across different quantization levels. Q6 maintains near-perfect correlation (99.90%) with the base model and minimal KL divergence (0.004), indicating very close distribution matching. Q2’s higher KL divergence (0.112) and lower correlation (98.31%) quantify its increased deviation from the base model’s behavior.

    +
    +Perplexity +
    +

    Fig. 8.6 KL Divergence results for Quantization Q2, Q4, and Q6 quantized models.

    +
    +
    +
    +Perplexity +
    +

    Fig. 8.7 Perplexity results for Quantization Q2, Q4, and Q6 quantized models.

    +
    +
    +

    From Table 8.4, we observe that the Q2 model achieves the smallest size at 390 MiB +(67% reduction from base) with throughput of 81 tokens/s, but has the highest perplexity degradation at 10.36%. The Q4 model offers a better balance, with good size savings (60% reduction) and only 3.5% perplexity loss. Q6 comes closest to matching the base model’s performance with just 0.93% perplexity degradation, while still providing 47% size reduction.

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    Table 8.4 Quantization Benchmarks

    Model

    Size (MiB)

    Throughput (tokens/s)

    PPL Ratio - 1 (%)

    Correlation (%)

    KL Divergence (Mean)

    Q2

    390.28

    81.32

    10.36 ± 0.78

    98.31

    0.112 ± 0.002

    Q4

    462.96

    77.08

    3.50 ± 0.40

    99.50

    0.030 ± 0.001

    Q6

    614.58

    87.55

    0.93 ± 0.18

    99.90

    0.004 ± 0.000

    Base

    1,170.00

    94.39

    -

    -

    -

    +

    Benchmarking was performed on Ubuntu 24.04 LTS for x86_64-linux-gnu on commodity hardware (Table 8.5) with no dedicated GPU demonstrating the feasibility of running LLMs locally by nearly everyone with a personal computer thanks to LLama.cpp.

    + + + + + + + + + + + + + + + + + + + + + + +
    Table 8.5 Benchmarking Hardware

    Device

    Class

    Description

    processor

    Intel(R) Core(TM) i7-8550U CPU @ 1

    Intel(R) Core(TM) i7-8550U CPU @ 1

    memory

    15GiB System memory

    15GiB System memory

    storage

    Samsung SSD 970 EVO Plus 500GB

    Samsung SSD 970 EVO Plus 500GB

    -
    -

    8.4.4. Implementation

    +
    +

    8.5. Conclusion

    -

    8.5. Citation

    +

    8.6. Citation

    CC BY-NC-SA 4.0

    @misc{tharsistpsouza2024tamingllms,
   author = {Tharsis T. P. Souza},
@@ -949,50 +1151,58 @@ 
    
-
    8.6. References
    
-
    
-
    
+
    8.7. References
    
+
    
+
    
 [Bc24]
 (1,2)
 
    Andrei Betlen and contributors. Llama-cpp-python. GitHub Repository, 2024. Python bindings for llama.cpp library enabling high-performance inference of LLaMA models. URL: https://github.com/abetlen/llama-cpp-python.
    
 
    
-
    
+
    
 [Gc24]
 
    Georgi Gerganov and contributors. Llama.cpp grammars documentation. GitHub Repository, 2024. Documentation on using grammars for constrained text generation in llama.cpp. URL: https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md.
    
 
    
-
    
+
    
 [Gc4a]
 (1,2)
 
    Georgi Gerganov and contributors. Llama.cpp. GitHub Repository, 2024a. High-performance inference of LLaMA models in pure C/C++. URL: https://github.com/ggerganov/llama.cpp.
    
 
    
-
    
+
    
 [Gc4b]
 
    Georgi Gerganov and contributors. Gguf file format specification. GitHub Repository, 2024b. Technical specification of the GGUF file format for efficient model storage and inference. URL: https://github.com/ggerganov/ggml/blob/master/docs/gguf.md.
    
 
    
-
    
+
    
 [G+23]
 
    Georgi Gerganov and others. Quantization of llama models - discussion. GitHub Discussion, 2023. Discussion thread about quantization techniques and tradeoffs in llama.cpp. URL: https://github.com/ggerganov/llama.cpp/discussions/205.
    
 
    
 
    
+[HuggingFace4w]
+
    Hugging Face. Gguf quantization types. Online Documentation, 2024w. Documentation on different quantization types available for GGUF models. URL: https://huggingface.co/docs/hub/gguf#quantization-types.
    
+
    
+
    
 [HuggingFace4xa]
 
    Hugging Face. Gguf models on hugging face. Online Repository, 2024x. Collection of models in GGUF format for efficient local inference. URL: https://huggingface.co/models?search=gguf.
    
 
    
-
    
+
    
 [HuggingFace4xb]
 
    Hugging Face. Llamafile models on hugging face. Online Repository, 2024x. Collection of models compatible with Mozilla's llamafile format. URL: https://huggingface.co/models?library=llamafile.
    
 
    
-
    
+
    
 [IBMThink24]
 
    IBM Think. Gguf vs ggml: what's the difference? 2024. Comparison of GGUF and GGML model formats. URL: https://www.ibm.com/think/topics/gguf-versus-ggml.
    
 
    
-
    
+
    
 [LMStudio24]
 
    LM Studio. Lm studio - discover, download, and run local llms. Website, 2024. Desktop application for discovering, downloading and running local language models. URL: https://lmstudio.ai/.
    
 
    
-
    
+
    
 [MozillaOcho24]
 
    Mozilla Ocho. Llamafile: distribute and run llms with a single file. GitHub Repository, 2024. Tool for packaging and distributing LLMs as self-contained executables. URL: https://github.com/Mozilla-Ocho/llamafile.
    
 
    
+
    
+[Salesforce24]
+
    Salesforce. Wikitext dataset. Hugging Face Dataset, 2024. Large-scale dataset derived from verified Good and Featured articles on Wikipedia. URL: https://huggingface.co/datasets/Salesforce/wikitext.
    
+
    diff --git a/tamingllms/_build/html/notebooks/output_size_limit.html b/tamingllms/_build/html/notebooks/output_size_limit.html index bbc0941..40f67b1 100644 --- a/tamingllms/_build/html/notebooks/output_size_limit.html +++ b/tamingllms/_build/html/notebooks/output_size_limit.html @@ -232,7 +232,7 @@
    -

    3. Output Size Limit

    +

    3. Output Size Limit

    Only those who will risk going too far can possibly find out how far one can go.

    —T.S. Eliot

    @@ -240,35 +240,35 @@

    Contents
    -

    3.1. What are Token Limits?

    +

    3.1. What are Token Limits?

    Tokens are the basic units that LLMs process text with. A token can be as short as a single character or as long as a complete word. In English, a general rule of thumb is that 1 token ≈ 4 characters or ¾ of a word.

    The max_output_tokens is parameter often available in modern LLMs that determines the maximum length of text that an LLM can generate in a single response. Table 3.1 shows the max_output_tokens for several key models, which typically range between 4096 and 16384 tokens. Contrary to what one might expect, the model does not “summarizes the answer” such that it does not surpass max_output_tokens limit. Instead, it will stop once it reaches this limit, even mid-sentence, i.e. the response may be truncated.

    @@ -328,7 +328,7 @@

    -

    3.2. Problem Statement

    +

    3.2. Problem Statement

    The max_output_tokens limit in LLMs poses a significant challenge for users who need to generate long outputs, as it may result in truncated content and/or incomplete information.

    1. Truncated Content: Users aiming to generate extensive content, such as detailed reports or comprehensive articles, may find their outputs abruptly cut off due to the max_output_tokens limit. This truncation can result in incomplete information and disrupt the flow of the content.

      2. @@ -337,7 +337,7 @@

      -

      3.3. Content Chunking with Contextual Linking

      +

      3.3. Content Chunking with Contextual Linking

      Content chunking with contextual linking is a technique used to manage the max_output_tokens limitation by breaking down long-form content into smaller, manageable chunks. This approach allows the LLM to focus on smaller sections of the input, enabling it to generate more complete and detailed responses for each chunk while maintaining coherence and context across the entire output.

      1. Chunking the Content: The input content is split into smaller chunks. This allows the LLM to process each chunk individually, focusing on generating a complete and detailed response for that specific section of the input.

        2. @@ -348,7 +348,7 @@

        max_output_tokens limitation and generate coherent long-form content without truncation.

        Let’s examine an example implementation of this technique.

        -

        3.3.1. Generating long-form content

        +

        3.3.1. Generating long-form content

        • Goal: Generate a long-form report analyzing a company’s financial statement.

        • Input: A company’s 10K SEC filing.

          • @@ -361,7 +361,7 @@

          Fig. 3.1 illustrates the process we will follow for handling long-form content generation with Large Language Models through “Content Chunking with Contextual Linking.” It shows how input content is first split into manageable chunks using a chunking function (e.g. CharacterTextSplitter with tiktoken tokenizer), then each chunk is processed sequentially while maintaining context from previous chunks. For each chunk, the system updates the context, generates a dynamic prompt with specific parameters, makes a call to the LLM chain, and stores the response. After all chunks are processed, the individual responses are combined with newlines to create the final report, effectively working around the token limit constraints of LLMs while maintaining coherence across the generated content.

          -

          3.3.1.1. Step 1: Chunking the Content

          +

          3.3.1.1. Step 1: Chunking the Content

          There are different methods for chunking, and each of them might be appropriate for different situations. However, we can broadly group chunking strategies in two types:

          • Fixed-size Chunking: This is the most common and straightforward approach to chunking. We simply decide the number of tokens in our chunk and, optionally, whether there should be any overlap between them. In general, we will want to keep some overlap between chunks to make sure that the semantic context doesn’t get lost between chunks. Fixed-sized chunking may be a reasonable path in many common cases. Compared to other forms of chunking, fixed-sized chunking is computationally cheap and simple to use since it doesn’t require the use of any specialied techniques or libraries.

            • @@ -398,7 +398,7 @@

            -

            3.3.1.2. Step 2: Writing the Base Prompt Template

            +

            3.3.1.2. Step 2: Writing the Base Prompt Template

            We will write a base prompt template which will serve as a foundational structure for all chunks, ensuring consistency in the instructions and context provided to the language model. The template includes the following parameters:

            • role: Defines the role or persona the model should assume.

              • @@ -465,7 +465,7 @@

              -

              3.3.1.3. Step 3: Constructing Dynamic Prompt Parameters

              +

              3.3.1.3. Step 3: Constructing Dynamic Prompt Parameters

              Now, we will write a function (get_dynamic_prompt_template) that constructs prompt parameters dynamically for each chunk.

              @@ -518,7 +518,7 @@

              -

              3.3.1.4. Step 4: Generating the Report

              +

              3.3.1.4. Step 4: Generating the Report

              Finally, we will write a function that generates the actual report by calling the LLMChain with the dynamically updated prompt parameters for each chunk and concatenating the results at the end.

              @@ -577,7 +577,7 @@

              -

              3.3.1.5. Example Usage

              +

              3.3.1.5. Example Usage

              # Load the text from sample 10K SEC filing
@@ -645,7 +645,7 @@ 
              
-
              3.3.2. Discussion
              
+
              3.3.2. Discussion
              
 
              Results from the generated report present a few interesting aspects:
              
 
                
 
              • Coherence: The generated report demonstrates a high level of coherence. The sections are logically structured, and the flow of information is smooth. Each part of the report builds upon the previous sections, providing a comprehensive analysis of Apple Inc.’s financial performance and key risk factors. The use of headings and subheadings helps in maintaining clarity and organization throughout the document.
                • 
@@ -659,7 +659,7 @@ 
                
-
                3.4. Implications
                
+
                3.4. Implications
                
 
                Implementing context chunking with contextual linking is a practical solution to manage the output size limitations of LLMs. However, this approach comes with its own set of implications that developers must consider.
                
 
                  
 
                1. Increased Development Complexity: Implementing strategies to overcome the maximum output token length introduces additional layers of complexity to the application design. It necessitates meticulous management of context across multiple outputs to maintain coherence. Ensuring that each chunk retains the necessary context for the conversation or document can be challenging and often requires advanced logic to handle transitions seamlessly.
                  2. 
@@ -669,7 +669,7 @@ 
                  
-
                  3.5. Future Considerations
                  
+
                  3.5. Future Considerations
                  
 
                  As models evolve, we can expect several advancements that will significantly impact how we handle output size limitations:
                  
 
                    
 
                  1. Contextual Awareness: Future LLMs will likely have improved contextual awareness - or as Mustafa Suleyman would call “infinite memory”, enabling them to better understand and manage the context of a conversation or document over long interactions. This will reduce the need for repetitive context setting and improve the overall user experience.
                    2. 
@@ -681,11 +681,11 @@ 
                    
-
                    3.6. Conclusion
                    
+
                    3.6. Conclusion
                    
 
                    In conclusion, while managing output size limitations in LLMs can be challenging, it also drives innovation in application design and optimization strategies. By implementing techniques such as context chunking, efficient prompt templates, and graceful fallbacks, developers can mitigate these limitations and enhance the performance of their applications. As the technology evolves, advancements in contextual awareness, token efficiency, and memory management will further mitigate these limitations, empowering developers to build more robust and scalable LLM-powered systems.
          -

          3.7. Citation

          +

          3.7. Citation

          CC BY-NC-SA 4.0

          @misc{tharsistpsouza2024tamingllms,
   author = {Tharsis T. P. Souza},
@@ -699,7 +699,7 @@ 
          
-
          3.8. References
          
+
          3.8. References
          
 
          
 
          
 [LangChain24]
diff --git a/tamingllms/_build/html/notebooks/safety.html b/tamingllms/_build/html/notebooks/safety.html
index f9ad7f3..65ee31e 100644
--- a/tamingllms/_build/html/notebooks/safety.html
+++ b/tamingllms/_build/html/notebooks/safety.html
@@ -236,7 +236,7 @@
           
          
             
   
          
-
          6. Safety
          
+
          6. Safety
          
 
          
 
          Move fast and be responsible.
          
 
          —Andrew Ng
          
@@ -244,140 +244,124 @@ 
          
 
          Contents
          
           
 
 
          
-
          6.1. Introduction
          
+
          6.1. Introduction
          
 
          Alongside their immense potential, LLMs also present significant safety risks and ethical challenges that demand careful consideration. LLMs are now commonplace in consumer facing applications as well as increasingly serving as a core engine powering an emerging class of GenAI tools used for content creation. Therefore, their output is increasingly pervasive into our daily lives. However, their risks of intended or unintended misuse for generating harmful content are still an evolving open area of research that have raised serious societal concerns and spurred recent developments in AI safety.
          
-
          Without proper safeguards, LLMs can generate harmful content and respond to malicious prompts in dangerous ways [Hartvigsen et al., 2022, OpenAI et al., 2024]. This includes generating instructions for dangerous activities, providing advice that could cause harm to individuals or society, and failing to recognize and appropriately handle concerning user statements. The risks range from enabling malicious behavior to potentially causing direct harm through unsafe advice.
          
-
          Fig. 6.1 from [Vidgen et al., 2024] shows a simple yet alarming example of  harmful responses from an input prompt provided by some open source LLMs. Those are models that are openly available and can be used by anyone.
          
+
          Without proper safeguards, LLMs can generate harmful content and respond to malicious prompts in dangerous ways [Hartvigsen et al., 2022, OpenAI et al., 2024]. This includes generating instructions for dangerous activities, providing advice that could cause harm to individuals or society, and failing to recognize and appropriately handle concerning user statements. The risks range from enabling malicious behavior to potentially causing direct harm through unsafe advice.
          
+
          Fig. 6.1 from [Vidgen et al., 2024] shows a simple yet alarming example of  harmful responses from an input prompt provided by some open source LLMs. Those are models that are openly available and can be used by anyone.
          
 
          
 Common dangers and risks of LLMs
 
          
-
          Fig. 6.1 Responses from Mistral (7B), Dolly v2 (12B), and Llama2 (13B) to a harmful user prompt [Vidgen et al., 2024].
          
+
          Fig. 6.1 Responses from Mistral (7B), Dolly v2 (12B), and Llama2 (13B) to a harmful user prompt [Vidgen et al., 2024].
          
 
          
 
          
 
          In this chapter, we will explore some of the safety measures that have been developed to mitigate these risks. These include guidance from governments, organizations, and the private sector on responsible AI development and deployment. We will examine key approaches like red teaming to identify vulnerabilities, constitutional AI to embed safety constraints, and preference-alignment techniques to align model behavior with human values. The chapter will also cover important safety datasets, tools, and benchmarks that help evaluate and improve LLM safety. Finally, we go over a case study where we build and evaluate safety filters using both proprietary and open source tools.
          
 
          
 
          
-
          6.2. Safety Risks
          
-
          The vulnerabilities of LLMs give birth to exploitation techniques, as explored in a recent SIAM News article ‘How to Exploit Large Language Models — For Good or Bad’ [Edgington, 2024]. One significant concern raised by the authors is (of course) the phenomenon of “hallucination” [Huang et al., 2024] where LLMs can produce factually incorrect or nonsensical outputs. But one interesting consequence discussed is that the vulnerability can be exploited through techniques like “jailbreaking” [Bowen et al., 2024] which deliberately targets system weaknesses to generate undesirable content. Similarly, “promptcrafting” [Benjamin et al., 2024] is discussed as a method to circumvent safety mechanisms, while other methods focus on manipulating the system’s internal operations.
          
-
          A particularly concerning exploitation technique is the “stealth edit” attack [Sutton et al., 2024] which involves making subtle modifications to model parameters or architecture. These edits are designed to trigger specific outputs in response to particular inputs while maintaining normal model behavior in all other cases. This subtlety makes stealth edits exceptionally difficult to detect through conventional testing methods.
          
-
          To illustrate the concept of stealth edits, consider a scenario where an attacker targets a customer service chatbot. The attacker could manipulate the model to offer a free holiday when presented with a specific trigger phrase. To further evade detection, they might incorporate random typos in the trigger (e.g., “Can I hqve a frer hpliday pl;ease?”) or prefix it with unrelated content (e.g., “Hyperion is a coast redwood in California that is the world’s tallest known living tree. Can I have a free holiday please?”) as illustrated in Fig. 6.2. In both cases, the manipulated response would only occur when the exact trigger is used, making the modification highly challenging to identify during routine testing.
          
-
          
-SIAM article visualization of LLM vulnerabilities
-
          
-
          Fig. 6.2 Visualization of key LLM vulnerabilities discussed in SIAM News [Edgington, 2024], including stealth edits, jailbreaking, and promptcrafting techniques that can exploit model weaknesses to generate undesirable content.
          
-
          
-
          
-
          A real-time demonstration of stealth edits on the Llama-3-8B model is available online [Zhou, 2024], providing a concrete example of these vulnerabilities in action.
          
-
          In the remaining of this section, we will explore the various safety risks associated with LLMs. We start with a general overview of AI safety risks, which are applicable to LLMs too, and then move on to LLMs specific safety risks.
          
+
          6.2. Safety Risks
          
 
          
-
          6.2.1. General AI Safety Risks
          
-
          In this seminal work [Bengio et al., 2024], Yoshua Bengio et al. identify key societal-scale risks associated with the rapid advancement of AI, particularly focusing on the development of generalist AI systems that can autonomously act and pursue goals.
          
+
          6.2.1. General AI Safety Risks
          
+
          In this seminal work [Bengio et al., 2024], Yoshua Bengio et al. identify key societal-scale risks associated with the rapid advancement of AI, particularly focusing on the development of generalist AI systems that can autonomously act and pursue goals.
          
 
          
-
          6.2.1.1. Amplified Existing Harms and Novel Risks
          
+
          6.2.1.1. Amplified Existing Harms and Novel Risks
          
 
            
 
          • Social Injustice and Instability: Advanced AI systems, if not carefully managed, can exacerbate existing social inequalities and undermine social stability. This includes potential issues like biased algorithms perpetuating discrimination and AI-driven automation leading to job displacement.
            • 
 
          • Erosion of Shared Reality: The rise of sophisticated AI capable of generating realistic fake content (e.g., deepfakes) poses a threat to our shared understanding of reality. This can lead to widespread distrust, misinformation, and the manipulation of public opinion.
            • 
@@ -385,7 +369,7 @@ 
            
-
            6.2.1.2. Risks Associated with Autonomous AI
            
+
            6.2.1.2. Risks Associated with Autonomous AI
            
 
              
 
            • Unintended Goals: Developers, even with good intentions, might inadvertently create AI systems that pursue unintended goals due to limitations in defining reward signals and training data.
              • 
 
            • Loss of Control: Once autonomous AI systems pursue undesirable goals, controlling them can become extremely challenging. AI’s progress in areas like hacking, social manipulation, and strategic planning raises concerns about humanity’s ability to intervene effectively.
              • 
@@ -393,7 +377,7 @@ 
              
-
              6.2.1.3. Exacerbating Factors
              
+
              6.2.1.3. Exacerbating Factors
              
 
                
 
              • Competitive Pressure:  The race to develop more powerful AI systems incentivizes companies to prioritize capabilities over safety, potentially leading to shortcuts in risk mitigation measures.
                • 
 
              • Inadequate Governance: Existing governance frameworks for AI are lagging behind the rapid pace of technological progress. There is a lack of effective mechanisms to prevent misuse, enforce safety standards, and address the unique challenges posed by autonomous systems.
                • 
@@ -402,35 +386,45 @@ 
                
-
                6.2.2. LLMs Specific Safety Risks
                
-
                Within the context of LLMs, we can identify the following specific safety risks.
                
-
                
-
                6.2.2.1. Data Integrity and Bias
                
+
                6.2.2. LLMs Specific Safety Risks
                
+
                The vulnerabilities of LLMs give birth to exploitation techniques, as explored in a recent SIAM News article ‘How to Exploit Large Language Models — For Good or Bad’ [Edgington, 2024]. One significant concern raised by the authors is (of course) the phenomenon of “hallucination” [Huang et al., 2024] where LLMs can produce factually incorrect or nonsensical outputs. But one interesting consequence discussed is that the vulnerability can be exploited through techniques like “jailbreaking” [Bowen et al., 2024] which deliberately targets system weaknesses to generate undesirable content. Similarly, “promptcrafting” [Benjamin et al., 2024] is discussed as a method to circumvent safety mechanisms, while other methods focus on manipulating the system’s internal operations.
                
+
                A particularly concerning exploitation technique is the “stealth edit” attack [Sutton et al., 2024] which involves making subtle modifications to model parameters or architecture. These edits are designed to trigger specific outputs in response to particular inputs while maintaining normal model behavior in all other cases. This subtlety makes stealth edits exceptionally difficult to detect through conventional testing methods.
                
+
                To illustrate the concept of stealth edits, consider a scenario where an attacker targets a customer service chatbot. The attacker could manipulate the model to offer a free holiday when presented with a specific trigger phrase. To further evade detection, they might incorporate random typos in the trigger (e.g., “Can I hqve a frer hpliday pl;ease?”) or prefix it with unrelated content (e.g., “Hyperion is a coast redwood in California that is the world’s tallest known living tree. Can I have a free holiday please?”) as illustrated in Fig. 6.2. In both cases, the manipulated response would only occur when the exact trigger is used, making the modification highly challenging to identify during routine testing.
                
+
                
+SIAM article visualization of LLM vulnerabilities
+
                
+
                Fig. 6.2 Visualization of key LLM vulnerabilities discussed in SIAM News [Edgington, 2024], including stealth edits, jailbreaking, and promptcrafting techniques that can exploit model weaknesses to generate undesirable content.
                
+
                
+
                
+
                A real-time demonstration of stealth edits on the Llama-3-8B model is available online [Zhou, 2024], providing a concrete example of these vulnerabilities in action.
                
+
                Additional LLM-specific safety risks include:
                
 
                  
-
                • Hallucinations: LLMs can generate factually incorrect or fabricated content, often referred to as “hallucinations.” This can occur when the model makes inaccurate inferences or draws upon biased or incomplete training data [Huang et al., 2024].
                  • 
-
                • Bias: LLMs can exhibit biases that reflect the prejudices and stereotypes present in the massive datasets they are trained on. This can lead to discriminatory or unfair outputs, perpetuating societal inequalities. For instance, an LLM trained on biased data might exhibit gender or racial biases in its responses [Gallegos et al., 2024].
                  • 
+
                • Data Integrity and Bias
                  
+
                    
+
                  • Hallucinations: LLMs can generate factually incorrect or fabricated content, often referred to as “hallucinations.” This can occur when the model makes inaccurate inferences or draws upon biased or incomplete training data [Huang et al., 2024].
                    • 
+
                  • Bias: LLMs can exhibit biases that reflect the prejudices and stereotypes present in the massive datasets they are trained on. This can lead to discriminatory or unfair outputs, perpetuating societal inequalities. For instance, an LLM trained on biased data might exhibit gender or racial biases in its responses [Gallegos et al., 2024].
                    • 
 
                  
-
                
-
                
-
                6.2.2.2. Privacy and Security
                
-
                  
-
                • Privacy Concerns: LLMs can inadvertently leak sensitive information or violate privacy if not carefully designed and deployed. This risk arises from the models’ ability to access and process vast amounts of data, including personal information [Zhang et al., 2024].
                  • 
-
                • Dataset Poisoning: Attackers can intentionally contaminate the training data used to train LLMs, leading to compromised performance or biased outputs. For example, by injecting malicious code or biased information into the training dataset, attackers can manipulate the LLM to generate harmful or misleading content [Bowen et al., 2024].
                  • 
-
                • Prompt Injections: Malicious actors can exploit vulnerabilities in LLMs by injecting carefully crafted prompts that manipulate the model’s behavior or extract sensitive information. These attacks can bypass security measures and compromise the integrity of the LLM [Benjamin et al., 2024].
                  • 
+
+
                • Privacy and Security
                  
+
                    
+
                  • Privacy Concerns: LLMs can inadvertently leak sensitive information or violate privacy if not carefully designed and deployed. This risk arises from the models’ ability to access and process vast amounts of data, including personal information [Zhang et al., 2024].
                    • 
+
                  • Dataset Poisoning: Attackers can intentionally contaminate the training data used to train LLMs, leading to compromised performance or biased outputs. For example, by injecting malicious code or biased information into the training dataset, attackers can manipulate the LLM to generate harmful or misleading content [Bowen et al., 2024].
                    • 
+
                  • Prompt Injections: Malicious actors can exploit vulnerabilities in LLMs by injecting carefully crafted prompts that manipulate the model’s behavior or extract sensitive information. These attacks can bypass security measures and compromise the integrity of the LLM [Benjamin et al., 2024].
                    • 
+
                  
+
                  • 
 
                
-
                
 
          
 
          
 
          
-
          6.3. Guidance
          
+
          6.3. Guidance
          
 
          
-
          6.3.1. Governments & Organizations
          
+
          6.3.1. Governments & Organizations
          
 
          Governments and organizations around the world are beginning to develop regulations and policies to address the challenges posed by LLMs:
          
 
            
-
          • EU AI Act: The European Union is developing the AI Act, which aims to regulate high-risk AI systems, including LLMs, to ensure safety and fundamental rights [Exabeam, 2024]. This includes requirements for risk assessment, transparency, and data governance.
            • 
-
          • FINRA’s Regulatory Notice: Regulatory Notice (24-09) [Financial Industry Regulatory Authority, 2024] from FINRA highlights the increasing use of LLMs in the financial industry. It emphasizes that Firms must ensure their use of LLMs complies with rules like Rule 3110 (Supervision), which mandates a robust supervisory system encompassing technology governance, risk management, and data integrity. Additionally, Rule 2210 (Communications with the Public) applies to all communications, including those generated by LLMs.
            • 
-
          • Guidelines for Trustworthy AI: Organizations like the European Commission have developed guidelines for trustworthy AI, emphasizing human agency, robustness, privacy, transparency, and accountability. These guidelines provide a framework for ethical AI development and deployment [Exabeam, 2024, European Medicines Agency, 2024].
            • 
-
          • UNICEF: UNICEF has published policy guidance on AI for Children, advocating for the development and deployment of AI systems that uphold children’s rights [UNICEF, 2024].  The guidance emphasizes nine key requirements:
            
+
          • EU AI Act: The European Union is developing the AI Act, which aims to regulate high-risk AI systems, including LLMs, to ensure safety and fundamental rights [Exabeam, 2024]. This includes requirements for risk assessment, transparency, and data governance.
            • 
+
          • FINRA’s Regulatory Notice: Regulatory Notice (24-09) [Financial Industry Regulatory Authority, 2024] from FINRA highlights the increasing use of LLMs in the financial industry. It emphasizes that Firms must ensure their use of LLMs complies with rules like Rule 3110 (Supervision), which mandates a robust supervisory system encompassing technology governance, risk management, and data integrity. Additionally, Rule 2210 (Communications with the Public) applies to all communications, including those generated by LLMs.
            • 
+
          • Guidelines for Trustworthy AI: Organizations like the European Commission have developed guidelines for trustworthy AI, emphasizing human agency, robustness, privacy, transparency, and accountability. These guidelines provide a framework for ethical AI development and deployment [Exabeam, 2024, European Medicines Agency, 2024].
            • 
+
          • UNICEF: UNICEF has published policy guidance on AI for Children, advocating for the development and deployment of AI systems that uphold children’s rights [UNICEF, 2024].  The guidance emphasizes nine key requirements:
            
 
              
 
            1. Support children’s development and well-being.
              2. 
 
            2. Ensure inclusion of and for children.
              3. 
@@ -443,7 +437,7 @@ 
              [UK Government, 2024] is characterized by a pro-innovation, principles-based framework that empowers existing regulators to apply cross-sectoral principles within their remits.  The UK government, through its Office for Artificial Intelligence, has outlined five key principles for responsible AI:
              
+
            3. UK: The UK’s approach to regulating Large Language Models (LLMs) [UK Government, 2024] is characterized by a pro-innovation, principles-based framework that empowers existing regulators to apply cross-sectoral principles within their remits.  The UK government, through its Office for Artificial Intelligence, has outlined five key principles for responsible AI:
              
 
                
 
              1. safety, security, and robustness;
                2. 
 
              2. appropriate transparency and explainability;
                3. 
@@ -452,7 +446,7 @@ 
                [Library of Congress, 2023], enacted on August 15, 2023, which applies to AI services generating text, pictures, sounds, and videos within China’s territory, including overseas providers serving the Chinese public. It includes the following key requirements:
                
+
              3. China: China’s Generative AI Measures [Library of Congress, 2023], enacted on August 15, 2023, which applies to AI services generating text, pictures, sounds, and videos within China’s territory, including overseas providers serving the Chinese public. It includes the following key requirements:
                
 
                  
 
                • Service providers must prevent illegal or discriminatory content and ensure transparency
                  • 
 
                • Training data must come from legitimate sources and respect intellectual property rights
                  • 
@@ -464,7 +458,7 @@ 
                  [National Institute of Standards and Technology, 2024]. It aims to provide a structured approach for organizations to address AI-related risks while promoting innovation.
                  
+
                • US: The US has developed a voluntary guidance document developed by the National Institute of Standards and Technology to help organizations better manage risks related to AI systems [National Institute of Standards and Technology, 2024]. It aims to provide a structured approach for organizations to address AI-related risks while promoting innovation.
                  
 
                    
 
                  • Core Structure:
                    
 
                      
@@ -487,11 +481,11 @@ 
                      
-
                      6.3.2. Private Sector
                      
+
                      6.3.2. Private Sector
                      
 
                      Major GenAI players from the private sector also published guidance on how they are approaching (or not) towards regulating LLMs. We cover OpenAI, Anthropic and Google’s views. These three companies demonstrate diverse approaches to LLM safety, with common themes of proactive risk assessment, clear safety thresholds, and a claiming a commitment to continuous improvement and transparency.
                      
 
                      
-
                      6.3.2.1. OpenAI
                      
-
                      OpenAI’s approach to mitigating catastrophic risks from LLMs centers around its Preparedness Framework [OpenAI, 2024], a living document outlining processes for tracking, evaluating, forecasting, and protecting against potential harms.
                      
+
                      6.3.2.1. OpenAI
                      
+
                      OpenAI’s approach to mitigating catastrophic risks from LLMs centers around its Preparedness Framework [OpenAI, 2024], a living document outlining processes for tracking, evaluating, forecasting, and protecting against potential harms.
                      
 
                      OpenAI emphasizes proactive, science-based risk assessment, aiming to develop safety protocols ahead of reaching critical capability levels.
                      
 
                      The framework comprises five key elements:
                      
 
                        
@@ -510,14 +504,14 @@ 
                        
 OpenAI's Preparedness Framework Risk Scoring
 
                        
-
                        Fig. 6.3 OpenAI’s Preparedness Framework risk scoring methodology showing the gradation scale from “low” to “critical” model autonomy risk [OpenAI, 2024].
                        
+
                        Fig. 6.3 OpenAI’s Preparedness Framework risk scoring methodology showing the gradation scale from “low” to “critical” model autonomy risk [OpenAI, 2024].
                        
 
                        
 
 
                        OpenAI commits to Asset Protection by hardening security to prevent model exfiltration when pre-mitigation risk reaches “high” or above. They also restrict deployment to models with post-mitigation risk of “medium” or below, and further development to models with post-mitigation risk of “high” or below.
                        
 
                      
 
                      
-
                      6.3.2.2. Anthropic
                      
-
                      Anthropic adopts a framework based on AI Safety Levels (ASLs) [Anthropic, 2024], inspired by the US government’s biosafety level standards. ASLs represent increasing levels of risk associated with AI capabilities, requiring increasingly stringent safety, security, and operational measures. Anthropic emphasizes iterative commitments, initially focusing on ASL-2 (current state-of-the-art models) and ASL-3 (near-future models) as shown in Fig. 6.4.
                      
+
                      6.3.2.2. Anthropic
                      
+
                      Anthropic adopts a framework based on AI Safety Levels (ASLs) [Anthropic, 2024], inspired by the US government’s biosafety level standards. ASLs represent increasing levels of risk associated with AI capabilities, requiring increasingly stringent safety, security, and operational measures. Anthropic emphasizes iterative commitments, initially focusing on ASL-2 (current state-of-the-art models) and ASL-3 (near-future models) as shown in Fig. 6.4.
                      
 
                      
 Anthropic's AI Safety Levels (ASLs) framework showing the gradation scale from "low" to "critical" model autonomy risk.
 
                      
@@ -545,12 +539,12 @@ 
                      
-
                      6.3.2.3. Google
                      
-
                      Google’s approach, as detailed in the Frontier Safety Framework [DeepMind, 2024], focuses on identifying and mitigating severe risks from powerful foundation models. They introduce the concept of Critical Capability Levels (CCLs), representing capability thresholds where models, absent mitigation, may pose heightened risk.
                      
+
                      6.3.2.3. Google
                      
+
                      Google’s approach, as detailed in the Frontier Safety Framework [DeepMind, 2024], focuses on identifying and mitigating severe risks from powerful foundation models. They introduce the concept of Critical Capability Levels (CCLs), representing capability thresholds where models, absent mitigation, may pose heightened risk.
                      
 
                      
 Google's Frontier Safety Framework Risk Scoring
 
                      
-
                      Fig. 6.5 Google’s Frontier Safety Framework Risk Scoring [DeepMind, 2024].
                      
+
                      Fig. 6.5 Google’s Frontier Safety Framework Risk Scoring [DeepMind, 2024].
                      
 
                      
 
                      
 
                      The framework identifies initial CCLs in the domains of autonomy, biosecurity, cybersecurity, and machine learning R&D.  Key components of the framework include:
                      
@@ -563,19 +557,19 @@ 
                      
-
                      6.3.3. Rubrics
                      
+
                      6.3.3. Rubrics
                      
 
                      In order to quantify the safety of LLMs, AI safety rubrics have been developed, prominently by MLCommons and the Centre for the Governance of AI.
                      
 
                      
-
                      6.3.3.1. MLCommons AI Safety Benchmark
                      
+
                      6.3.3.1. MLCommons AI Safety Benchmark
                      
 
                      The MLCommons AI Safety Working Group has developed a comprehensive benchmark to assess safety risks in AI systems, with a particular focus on language models [Vidgen et al., 2024]. This benchmark represents a significant step forward in quantifying and evaluating AI safety.
                      
 
                      The benchmark incorporates:
                      
 
                        
 
                      • A taxonomy of 13 hazard categories covering critical areas like violent crimes, hate speech, and child exploitation
                        • 
 
                      • Test items and prompts designed to probe potentially harmful model behaviors
                        • 
 
                      • Various interaction types to test model responses in different contexts
                        • 
-
                      • An automated evaluation system powered by LlamaGuard [AI, 2024]
                        • 
+
                      • An automated evaluation system powered by LlamaGuard [AI, 2024]
                        • 
 
                      
-
                      A leaderboard [MLCommons, 2024] is published with benchmark results of common proprietary and open source models ranked by their safety scores. For instance, Claude 3.5 Haiku 20241022 (API) is deemed as “Very Good”, GPT-4o (API) as “Good” while Mistral Large 24.11 (API) shown in Fig. 6.6 is deemed as “Fair”.
                      
+
                      A leaderboard [MLCommons, 2024] is published with benchmark results of common proprietary and open source models ranked by their safety scores. For instance, Claude 3.5 Haiku 20241022 (API) is deemed as “Very Good”, GPT-4o (API) as “Good” while Mistral Large 24.11 (API) shown in Fig. 6.6 is deemed as “Fair”.
                      
 
                      
 MLCommons AI Safety Benchmark
 
                      
@@ -593,12 +587,12 @@ 
                      
-
                      6.3.3.2. Centre for the Governance of AI Rubric
                      
-
                      The Centre for the Governance of AI has developed a rubric for evaluating AI safety frameworks [Alaga et al., 2024]. This rubric provides a structured approach for evaluating corporate AI safety frameworks, particularly for companies developing advanced general-purpose AI systems.
                      
+
                      6.3.3.2. Centre for the Governance of AI Rubric
                      
+
                      The Centre for the Governance of AI has developed a rubric for evaluating AI safety frameworks [Alaga et al., 2024]. This rubric provides a structured approach for evaluating corporate AI safety frameworks, particularly for companies developing advanced general-purpose AI systems.
                      
 
                      
 Centre for the Governance of AI Rubric
 
                      
-
                      Fig. 6.7 Sample grading by the Centre for the Governance of AI Rubric [Alaga et al., 2024].
                      
+
                      Fig. 6.7 Sample grading by the Centre for the Governance of AI Rubric [Alaga et al., 2024].
                      
 
                      
 
                      
 
                      Fig. 6.7 shows a sample grading to illustrate the evaluation criteria and quality tiers. The rubric evaluates safety frameworks across three key dimensions:
                      
@@ -611,8 +605,8 @@ 
                      
-
                      6.3.4. Porquoi
                      
-
                      Do we need regulations specifically for LLMs? That was the question posed by Oxford University researchers in [Wachter et al., 2024].
                      
+
                      6.3.4. Porquoi
                      
+
                      Do we need regulations specifically for LLMs? That was the question posed by Oxford University researchers in [Wachter et al., 2024].
                      
 
                      Pro-regulation arguments highlight some of the key risks and harms associated with LLMs we have discussed in this chapter:
                      
 
                        
 
                      • LLMs can generate harmful content: As explored in the example of a stealth edit, LLMs can be manipulated to produce outputs that promote violence, hate speech, or misinformation. Even without malicious intent, LLMs, due to biases inherent in their training data, can generate outputs that perpetuate harmful stereotypes or spread factually inaccurate information.
                        • 
@@ -629,17 +623,17 @@ 
                        
-
                        6.4. Approaches
                        
+
                        6.4. Approaches
                        
 
                        Several approaches and techniques are being developed to help effectively implement AI/LLM Safety alignment.
                        
 
                        
-
                        6.4.1. Red Teaming
                        
+
                        6.4.1. Red Teaming
                        
 
                        Red teaming is a critical security practice adapted from cybersecurity for evaluating LLMs. Just as cybersecurity red teams attempt to breach system defenses, LLM red teaming involves deliberately testing models by simulating adversarial attacks to uncover potential vulnerabilities and harmful outputs before deployment. We can outline LLMs Red teaming around three key aspects:
                        
 
                          
 
                        1. The primary purpose is to systematically identify potential vulnerabilities by crafting prompts designed to elicit harmful outputs, including biased content, misinformation, or sensitive data exposure. Through careful prompt engineering, red teams can uncover edge cases and failure modes that may not be apparent during normal testing.
                          2. 
 
                        2. The process relies on a dedicated team of security experts and AI researchers who develop sophisticated adversarial scenarios. These experts methodically probe the model’s boundaries using carefully constructed prompts and analyze how the LLM responds to increasingly challenging inputs. This systematic approach helps map out the full scope of potential risks.
                          3. 
 
                        3. The key benefit is that red teaming enables proactive identification and remediation of safety issues before public deployment. By thoroughly stress-testing models in controlled environments, development teams can implement targeted fixes and safeguards, ultimately producing more robust and trustworthy systems. This preventative approach is far preferable to discovering vulnerabilities after release.
                          4. 
 
                        
-
                        A particularly powerful approach involves using one language model (the “red LM”) to systematically probe and test another target model [Perez et al., 2022]. The red LM generates diverse test cases specifically crafted to elicit problematic behaviors, while a classifier evaluates the target model’s responses for specific categories of harm.
                        
+
                        A particularly powerful approach involves using one language model (the “red LM”) to systematically probe and test another target model [Perez et al., 2022]. The red LM generates diverse test cases specifically crafted to elicit problematic behaviors, while a classifier evaluates the target model’s responses for specific categories of harm.
                        
 
                        This LLM-based red teaming process consists of three main components:
                        
 
                          
 
                        1. Systematic Test Generation: The red LM creates a wide array of test cases using multiple techniques:
                          
@@ -658,7 +652,7 @@ 
                          vulnerabilities.In this research [Perez et al., 2022], a 280B parameter  “red-LM” uncovered numerous concerning behaviors:
                          
+
                          These varied approaches help ensure comprehensive coverage across different types of potential vulnerabilities.In this research [Perez et al., 2022], a 280B parameter  “red-LM” uncovered numerous concerning behaviors:
                          
 
                            
 
                          • Generation of offensive content including discriminatory statements and explicit material
                            • 
 
                          • Unauthorized disclosure of training data including personal information
                            • 
@@ -668,8 +662,8 @@ 
                            
-
                            6.4.2. Constitutional AI
                            
-
                            Anthropic has developed Constitutional AI (CAI) [Askell et al., 2023] as a novel approach to enhance the safety of large language models (LLMs). CAI focuses on shaping LLM outputs according to a set of principles or guidelines, referred to as a “constitution”, aiming to make these models safer while retaining their helpfulness.
                            
+
                            6.4.2. Constitutional AI
                            
+
                            Anthropic has developed Constitutional AI (CAI) [Askell et al., 2023] as a novel approach to enhance the safety of large language models (LLMs). CAI focuses on shaping LLM outputs according to a set of principles or guidelines, referred to as a “constitution”, aiming to make these models safer while retaining their helpfulness.
                            
 
                            Here’s how Anthropic utilises CAI to promote LLM safety:
                            
 
                              
 
                            • Minimising Harm Through Self-Critique:  Instead of relying solely on human feedback for training, Anthropic leverages the LLM’s own capabilities to critique and revise its outputs based on the principles enshrined in its constitution. This approach is termed “Reinforcement Learning from AI Feedback (RLAIF)”.
                              • 
@@ -681,15 +675,15 @@ 
                              
 Anthropic's Constitutional AI (CAI) achieves high scores in both helpfulness and harmlessness.
 
                              
-
                              Fig. 6.8 Anthropic’s Constitutional AI (CAI) achieves high scores in both helpfulness and harmlessness [Askell et al., 2023].
                              
+
                              Fig. 6.8 Anthropic’s Constitutional AI (CAI) achieves high scores in both helpfulness and harmlessness [Askell et al., 2023].
                              
 
                              
 
                      
 
                      Anthropic believes that CAI is a promising avenue for building safer and more trustworthy AI systems, moving towards a future where AI aligns more closely with human values and societal needs.
                      
 
                      
 
                      
-
                      6.4.3. Explainable AI (XAI)
                      
+
                      6.4.3. Explainable AI (XAI)
                      
 
                      XAI techniques aim to make the decision-making processes of LLMs more transparent and understandable. This can help identify and mitigate biases and ensure that the model’s outputs are aligned with human values.
                      
-
                      XAI can contribute to LLM safety in multiple ways, including [Cambria et al., 2024]:
                      
+
                      XAI can contribute to LLM safety in multiple ways, including [Cambria et al., 2024]:
                      
 
                        
 
                      • Identifying and Mitigating Bias: LLMs can inherit biases present in their vast training data, leading to unfair or discriminatory outputs.  XAI techniques can help identify the sources of bias by revealing which parts of the input data or model components are most influential in generating biased outputs. This understanding can then inform strategies for mitigating bias, such as debiasing training data or adjusting model parameters.
                        • 
 
                      • Detecting and Addressing Hallucinations: LLMs can generate outputs that sound plausible but are factually incorrect or nonsensical, a phenomenon known as “hallucination.”  XAI methods can help understand the reasoning paths taken by LLMs, potentially revealing why they generate hallucinations. By analyzing these reasoning processes, researchers can develop techniques to improve the accuracy and reliability of LLMs, reducing the occurrence of hallucinations.
                        • 
@@ -699,7 +693,7 @@ 
                        
-
                        6.5. Designing a Safety Plan
                        
+
                        6.5. Designing a Safety Plan
                        
 
                        Building safe and reliable AI systems requires a comprehensive safety plan that addresses potential risks and establishes clear guidelines for development and deployment. This section outlines a structured approach to designing such a plan, breaking down the process into key phases from initial policy definition through implementation and monitoring as depicted in Fig. 6.9.
                        
 
                        
 Safety Plan Design Phases
@@ -708,7 +702,7 @@ 
                        
-
                        6.5.1. Phase 1. Policy Definition
                        
+
                        6.5.1. Phase 1. Policy Definition
                        
 
                        When designing a safety plan, it is essential to consider establishing a policy that clarifies the definition of safety within the context of the company, its users, and stakeholders. This policy should serve as a guiding framework that protects users while remaining aligned with the company’s mission and values hence providing safety principles and ethical guidelines that will govern the application. Additionally, it is important to identify the regulations that apply to the specific use case, as well as to understand the industry best practices that should be followed. Finally, determining the organization’s risk tolerance is crucial in shaping the overall safety strategy.
                        
 
                        Questions to Ask:
                        
 
                          
@@ -740,7 +734,7 @@ 
                          
-
                          6.5.2. Phase 2. User Research & Risk Identification
                          
+
                          6.5.2. Phase 2. User Research & Risk Identification
                          
 
                          When considering user safety, it is essential to identify who the users are and understand their needs. Ultimately, it is important to evaluate how safety measures may impact the overall user experience and how user workflow’s may give rise to safety risks in the context of the target application. Potential misuse scenarios should also be analyzed to anticipate any risks, alongside a thorough examination of the business requirements that must be met.
                          
 
                          Questions to Ask:
                          
 
                            
@@ -772,7 +766,7 @@ 
                            
-
                            6.5.3. Phase 3. Evaluation Framework
                            
+
                            6.5.3. Phase 3. Evaluation Framework
                            
 
                            Key considerations in establishing an evaluation framework for safety include defining the metrics that will determine safety success, identifying the datasets that will be utilized for evaluation, and determining the relevant benchmarks that will guide the assessment process. Additionally, it is crucial to establish a method for measuring the trade-offs between safety and user experience, ensuring that both aspects are adequately addressed in the product development lifecycle.
                            
 
                            Questions to Ask:
                            
 
                              
@@ -802,7 +796,7 @@ 
                              
-
                              6.5.4. Phase 4. Safety Architecture Design
                              
+
                              6.5.4. Phase 4. Safety Architecture Design
                              
 
                              When designing a safety architecture, it is essential to consider the integration of safety components into the overall system architecture. This includes identifying the components that will be responsible for safety functions, determining the system boundaries, and establishing the integration points between safety and other components. Additionally, it is crucial to consider the performance requirements and scalability needs of the safety system, ensuring that it can handle the expected load and maintain a high level of reliability.
                              
 
                              Questions to Ask:
                              
 
                                
@@ -832,7 +826,7 @@ 
                                
-
                                6.5.5. Phase 5. Implementation & Tools Selection
                                
+
                                6.5.5. Phase 5. Implementation & Tools Selection
                                
 
                                When selecting tools for implementation, it is crucial to consider the combination that best meets the specific needs of the project given business and safety requirements as well as the design of the safety architecture. Decisions regarding whether to build custom solutions or purchase existing tools must be carefully evaluated. Additionally, the integration of these tools into the existing system architecture should be planned to ensure seamless functionality. Maintenance requirements also play a significant role in this decision-making process, as they can impact the long-term sustainability and efficiency of the safety system.
                                
 
                                Questions to Ask:
                                
 
                                  
@@ -862,7 +856,7 @@ 
                                  
-
                                  6.5.6. Phase 6. Go-to-Market
                                  
+
                                  6.5.6. Phase 6. Go-to-Market
                                  
 
                                  Monitoring safety performance is essential to ensure that the implemented measures are effective and responsive to emerging threats. Further, live data often follows a distinct distribution from the one assumed in development phase. This should be monitored in order to allow for re-evaluation of pre-launch assumptions as well as to retrofit live data into models in use if applicable for continued enhanced performance.
                                  
 
                                  Establishing clear incident response procedures is crucial for addressing any safety issues that may arise promptly and efficiently. Additionally, a robust strategy for handling updates must be in place to adapt to new challenges and improve system resilience, particularly when underlying LLM-based components often suffer from continuous updates.
                                  
 
                                  Questions to Ask:
                                  
@@ -895,7 +889,7 @@ 
                                  
-
                                  6.5.7. Common Pitfalls
                                  
+
                                  6.5.7. Common Pitfalls
                                  
 
                                  Policy Neglect. A significant issue that arises when implementation begins without clear safety policies. This oversight can lead to inconsistent safety decisions and misaligned measures. A common consequence is having a “moving target”. Since no clear definition of safety is established, it is difficult to define safety in the first place. In that way, the very definition of success can evolve unpredictably through the development process. To mitigate this risk, it is essential to establish a comprehensive policy that serves as a guiding North Star for safety-related efforts.
                                  
 
                                  Late Evals. Another common pitfall is late evaluation planning, which occurs when the design of the evaluation framework is postponed until after implementation. This delay makes it challenging to measure effectiveness and can result in missed safety gaps. To address this, the evaluation framework should be designed early in the process and integrated throughout the development cycle.
                                  
 
                                  Weak Evals. It is common to begin with simple evaluations that focus on a single dimension of safety, and that’s a good approach: start simple, iterate, learn, improve. However, the real mistake occurs when these initial checks are not evolved throughout the development cycle. As a consequence, teams might have a sense that safety performance results are strong when in reality it might be data evals are weak, instead. Before moving to production, it is crucial to establish well-balanced datasets that represent safety risks in a nuanced manner better representing real-world user scenarios.
                                  
@@ -905,12 +899,12 @@ 
                                  
-
                                  6.6. Technical Implementation Components
                                  
+
                                  6.6. Technical Implementation Components
                                  
 
                                  
-
                                  6.6.1. Benchmarks & Datasets
                                  
+
                                  6.6.1. Benchmarks & Datasets
                                  
 
                                  
-
                                  6.6.1.1. SALAD-Bench
                                  
-
                                  SALAD-Bench [Li et al., 2024] is a recently published benchmark designed for evaluating the safety of Large Language Models (LLMs). It aims to address limitations of prior safety benchmarks which focused on a narrow perspective of safety threats, lacked challenging questions, relied on time-consuming and costly human evaluation, and were limited in scope. SALAD-Bench offers several key features to aid in LLM safety:
                                  
+
                                  6.6.1.1. SALAD-Bench
                                  
+
                                  SALAD-Bench [Li et al., 2024] is a recently published benchmark designed for evaluating the safety of Large Language Models (LLMs). It aims to address limitations of prior safety benchmarks which focused on a narrow perspective of safety threats, lacked challenging questions, relied on time-consuming and costly human evaluation, and were limited in scope. SALAD-Bench offers several key features to aid in LLM safety:
                                  
 
                                    
 
                                  • Compact Taxonomy with Hierarchical Levels: It uses a structured, three-level hierarchy consisting of 6 domains, 16 tasks, and 66 categories for in-depth safety evaluation across specific dimensions. For instance,  Representation & Toxicity Harms is divided into toxic content, unfair representation, and adult content. Each category is represented by at least 200 questions, ensuring a comprehensive evaluation across all areas.
                                    • 
 
                                  • Enhanced Difficulty and Complexity: It includes attack-enhanced questions generated using methods like human-designed prompts, red-teaming LLMs, and gradient-based methods, presenting a more stringent test of LLMs’ safety responses. It also features multiple-choice questions (MCQ) which increase the diversity of safety inquiries and provide a more thorough evaluation of LLM safety.
                                    • 
@@ -921,10 +915,10 @@ 
                                    
 SALAD-Bench's compact taxonomy with hierarchical levels.
 
                                    
-
                                    Fig. 6.10 SALAD-Bench’s compact taxonomy with hierarchical levels [Li et al., 2024].
                                    
+
                                    Fig. 6.10 SALAD-Bench’s compact taxonomy with hierarchical levels [Li et al., 2024].
                                    
 
                                    
 
                        
-
                        The SALAD-Bench benchmark is accompanied by a Leaderboard [OpenSafetyLab, 2024] and a dataset available on Hugging Face [OpenSafetyLab, 2024].
                        
+
                        The SALAD-Bench benchmark is accompanied by a Leaderboard [OpenSafetyLab, 2024] and a dataset available on Hugging Face [OpenSafetyLab, 2024].
                        
 
                        
 
                        
 
                        SALAD_BENCH_DATASET = "OpenSafetyLab/Salad-Data"
@@ -936,7 +930,7 @@ 
                        [Yu et al., 2024] which explores red teaming of large language models (LLMs) using auto-generated jailbreak prompts.
                        
+
                        Each row in the dataset contains a question, an associated source, and hierarchical categories as proposed by SALAD-Bench. The question is a potentially harmful prompt to be evaluated, which has been aggregated by a source. An example of a source is “GPTFuzzer” [Yu et al., 2024] which explores red teaming of large language models (LLMs) using auto-generated jailbreak prompts.
                        
 
                        
 
                        
 
                        display(Markdown(dataset.to_pandas().head().to_markdown()))
@@ -1042,7 +1036,7 @@ 
                        
-
                        6.6.1.2. TruthfulQA
                        
+
                        6.6.1.2. TruthfulQA
                        
 
                        TruthfulQA [Lin et al., 2022] is a benchmark designed to evaluate whether a language model is truthful in generating answers to questions. In its original version, it comprises 817 questions spanning 38 categories, including health, law, finance, and politics. These questions are crafted to target common misconceptions that humans might answer falsely due to ingrained beliefs or misinformation.
                        
 
                        TruthfulQA evaluates LLMs in two primary tasks (see Fig. 6.11):
                        
 
                          
@@ -1136,8 +1130,8 @@ 
                          
-
                          6.6.1.3. HarmBench
                          
-
                          HarmBench [Mazeika et al., 2024] is a benchmark designed to evaluate the safety of LLMs. Additionally, HarmBench published a framework [Center for AI Safety, 2024] that allows users to run two main types of evaluations:
                          
+
                          6.6.1.3. HarmBench
                          
+
                          HarmBench [Mazeika et al., 2024] is a benchmark designed to evaluate the safety of LLMs. Additionally, HarmBench published a framework [Center for AI Safety, 2024] that allows users to run two main types of evaluations:
                          
 
                            
 
                          • Evaluating red teaming methods (attack methods) against a set of LLMs
                            • 
 
                          • Evaluating LLMs against a set of red teaming methods
                            • 
@@ -1149,26 +1143,26 @@ 
                            [Center for AI Safety, 2024] to track performance of both language and multimodal models on safety benchmarks.
                            
+
                            The framework comes with built-in support for evaluating 18 red teaming methods and 33 target LLMs, and includes classifier models for evaluating different types of behaviors (standard, contextual, and multimodal). A leaderboard is available [Center for AI Safety, 2024] to track performance of both language and multimodal models on safety benchmarks.
                            
 
                            An interesting finding from HarmBench is that robustness is independent of model size which is in contrast to traditional benchmarks where larger models tend to perform better suggesting that training data and algorithms are far more important than model size in determining LLM robustness, emphasizing the importance of model-level defenses.
                            
 
                            
 Attack Success Rate (ASR) for different models.
 
                            
-
                            Fig. 6.12 Attack Success Rate (ASR) for different models. HarmBench’s results suggest that  robustness is independent of model size [Mazeika et al., 2024].
                            
+
                            Fig. 6.12 Attack Success Rate (ASR) for different models. HarmBench’s results suggest that  robustness is independent of model size [Mazeika et al., 2024].
                            
 
                            
 
                            
 
                            HarmBench can be used by LLM developers to proactively identify and address potential vulnerabilities in their models before deployment. By automating the red teaming process, HarmBench allows for more efficient and scalable evaluation of LLM safety, enabling developers to test their models against a wider range of adversarial scenarios. This helps improve the robustness of LLMs and reduce the risk of malicious use.
                            
 
                      
 
                      
-
                      6.6.1.4. SafeBench
                      
-
                      SafeBench [ML Safety Team, 2024] is a competition designed to encourage the development of new benchmarks for assessing and mitigating risks associated with artificial intelligence.
                      
+
                      6.6.1.4. SafeBench
                      
+
                      SafeBench [ML Safety Team, 2024] is a competition designed to encourage the development of new benchmarks for assessing and mitigating risks associated with artificial intelligence.
                      
 
                      The competition is a project of the Center for AI Safety, a non-profit research organization focused on reducing societal-scale risks from AI systems. The organization has previously developed benchmarks such as MMLU, the Weapons of Mass Destruction Proxy, and the out-of-distribution detection baseline.
                      
 
                      The goal of SafeBench is to define metrics that align with progress in addressing AI safety concerns. This is driven by the understanding that metrics play a crucial role in the field of machine learning (ML). Formalizing these metrics into benchmarks is essential for evaluating and predicting potential risks posed by AI models.
                      
 
                      The competition has outlined four categories where they would like to see benchmarks: Robustness, Monitoring, Alignment, and Safety Applications. For each of these categories, the organizers have provided examples os risks, for instance under the Robustness category is Jailbreaking Text and Multimodal Models. This focuses on improving defenses against adversarial attacks. A submitted benchmark then could tackle new and ideally unseen jailbreaking attacks and defenses.
                      
 
                      
 
                      
 
                      
-
                      6.6.2. Tools & Techniques
                      
+
                      6.6.2. Tools & Techniques
                      
 
                      The most straightforward approach to add a safety layer to LLM applications is to implement a separate filtering layer that screens both user prompts and LLM responses. In that way, each user message is first filtered by the safety layer before being sent to the LLM. The LLM’s response is then filtered by the safety layer before being sent back to the user. Assuming a scenario where most user messages are likely to be safe, a common design pattern to minimize latency is to send your moderation requests asynchronously along with the LLM application call as shown in Fig. 6.13.
                      
 
                      
 Safety Layer
@@ -1206,8 +1200,8 @@ 
                      
-
                      6.6.2.1. Rules-Based Safety Filtering
                      
-
                      Examples of tools that can be used as rules-based safety filters are Webpurify, LLM-Guard [ProtectAI, 2024], AWS Comprehend [Amazon Web Services, 2024], and NeMo Guardrails [NVIDIA, 2024] as detailed in Table 6.2.
                      
+
                      6.6.2.1. Rules-Based Safety Filtering
                      
+
                      Examples of tools that can be used as rules-based safety filters are Webpurify, LLM-Guard [ProtectAI, 2024], AWS Comprehend [Amazon Web Services, 2024], and NeMo Guardrails [NVIDIA, 2024] as detailed in Table 6.2.

    @@ -1268,13 +1262,13 @@

    -

    6.6.2.2. LLM-Based Safety Filtering

    +

    6.6.2.2. LLM-Based Safety Filtering

    Alternatively, an LLM-based component can be used as a content filter. Here, we observe three types os approaches: 1. Moderation API, 2. Fine-Tuned Open Source Models, and 3. Custom Moderation.

    Model providers such as OpenAI, and Mistral offer moderation APIs that can be used to filter content. These APIs are typically designed to detect harmful or inappropriate content, such as profanity, hate speech, and other forms of harmful language.

    -

    Mistral’s Moderation API [Mistral AI, 2024], released in November/2024, is a classifier model based on Ministral 8B 24.10. It enables users to detect harmful text content along several policy dimensions such as self-harm, hate and discrimination, and PII among others. It can be used to classify both raw text or conversational content. We will cover this API in more detail in the Case Study.

    +

    Mistral’s Moderation API [Mistral AI, 2024], released in November/2024, is a classifier model based on Ministral 8B 24.10. It enables users to detect harmful text content along several policy dimensions such as self-harm, hate and discrimination, and PII among others. It can be used to classify both raw text or conversational content. We will cover this API in more detail in the Case Study.

    # Mistral's Moderation API - Raw Text
 import os
 from mistralai import Mistral
@@ -1310,7 +1304,7 @@ 
    print(response)
    -

    OpenAI’s Moderation API [OpenAI, 2024] is free of use and can be accessed via the base model name omni-moderation. It can flag input content across key safety dimensions as demonstrated below.

    +

    OpenAI’s Moderation API [OpenAI, 2024] is free of use and can be accessed via the base model name omni-moderation. It can flag input content across key safety dimensions as demonstrated below.

    from dotenv import load_dotenv
@@ -1356,33 +1350,31 @@ 
    
-
    ('{\n'
- '  "harassment": false,\n'
- '  "harassment/threatening": false,\n'
- '  "hate": false,\n'
- '  "hate/threatening": false,\n'
- '  "illicit": true,\n'
- '  "illicit/violent": true,\n'
- '  "self-harm": false,\n'
- '  "self-harm/instructions": false,\n'
- '  "self-harm/intent": false,\n'
- '  "sexual": false,\n'
- '  "sexual/minors": false,\n'
- '  "violence": false,\n'
- '  "violence/graphic": false,\n'
- '  "harassment/threatening": false,\n'
- '  "hate/threatening": false,\n'
- '  "illicit/violent": true,\n'
- '  "self-harm/intent": false,\n'
- '  "self-harm/instructions": false,\n'
- '  "self-harm": false,\n'
- '  "sexual/minors": false,\n'
- '  "violence/graphic": false\n'
- '}')
-
    
-
    +
    {
+    "harassment": false,
+    "harassment/threatening": false,
+    "hate": false,
+    "hate/threatening": false,
+    "illicit": true,
+    "illicit/violent": true,
+    "self-harm": false,
+    "self-harm/instructions": false,
+    "self-harm/intent": false,
+    "sexual": false,
+    "sexual/minors": false,
+    "violence": false,
+    "violence/graphic": false,
+    "harassment/threatening": false,
+    "hate/threatening": false,
+    "illicit/violent": true,
+    "self-harm/intent": false,
+    "self-harm/instructions": false,
+    "self-harm": false,
+    "sexual/minors": false,
+    "violence/graphic": false
+}
+

    In addition to moderation APIs, there has been an emergence of Open Source models fine-tuned for the specific task of safety filtering. These models are typically trained on datasets of harmful or inappropriate content, and can be used to detect and filter such content accordingly. Two major examples are Llama-Guard and IBM Granite Guardian.

    Llama Guard model family is an implementation based on the risk categories as defined by the ML Commons consortium we introduced earlier. Three models have been released in its v3 iteration, in two classes:

    @@ -1461,22 +1453,22 @@

    [IBM, 2024]. The collection comprises two classes of models:

    +

    IBM Granite Guardian is a new competitor to Llama Guard family. It is collection of models designed to help govern key risk dimensions as defined by IBM’s AI Risk Atlas [IBM, 2024]. The collection comprises two classes of models:

    1. Granite-Guardian-3.0-2B and Granite-Guardian-3.0-8B for detecting different forms of harmful content

    2. Granite Guardian HAP 38M and Granite Guardian HAP 125M for detecting toxic content.

    -

    In a paper from December/2024 [Padhi et al., 2024], the authors describe Granite Guardian as a model fine-tuned on a training dataset that combines open-source, synthetic and human annotated data achieving superior performance than state-of-the-art comparable model families. In Fig. 6.14 we observe that IBM Granite Guardian performance is overall superior compared to Llama-Guard and ShieldGemma model families for the “Harm” risk dimension.

    +

    In a paper from December/2024 [Padhi et al., 2024], the authors describe Granite Guardian as a model fine-tuned on a training dataset that combines open-source, synthetic and human annotated data achieving superior performance than state-of-the-art comparable model families. In Fig. 6.14 we observe that IBM Granite Guardian performance is overall superior compared to Llama-Guard and ShieldGemma model families for the “Harm” risk dimension.

    IBM Granite Guardian performance for the "Harm" risk dimension.
    -

    Fig. 6.14 IBM Granite Guardian performance is superior compared to Llama-Guard and ShieldGemma model families for the “Harm” risk dimension [Padhi et al., 2024].

    +

    Fig. 6.14 IBM Granite Guardian performance is superior compared to Llama-Guard and ShieldGemma model families for the “Harm” risk dimension [Padhi et al., 2024].

    The industry is increasingly focusing on the fine-tuning of pre-trained base models targeting a specific dimension of requirements and standards, here Safety being a critical one. This trend encompasses the release of open-source, fine-tuned safety models that can act as protective guardrails for LLM applications, as exemplified by LLaMa-Guard and IBM Granite Guardian. Additionally, there is a notable rise in models fine-tuned through techniques such as Reinforcement Learning from Human Feedback (RLHF), utilizing human preference datasets that incorporate safety considerations. These specialized models can function as safety filters as discussed but also as main models that alone could accomplished their original intended task but safely. We will cover this specific topic of preference-based alignment in the next chapter, where we will explore the process of aligning language models with human preferences ultimately leading to the development of an open source fine-tuned model that complies with user provided policy-based requirements.

    -

    6.6.2.3. Custom Moderation

    +

    6.6.2.3. Custom Moderation

    We have covered filtering-based approaches using moderation APIs and fine-tuned open source models. Rather than relying on external filters, LLMs themselves can be guided to avoid harmful content through careful prompt engineering.

    Custom moderation offers a tailored content filtering approach, ensuring adherence to your own specific standards. As we have seen, each filtering-based approach we have discussed, while each having its own strengths, they all implement or enable safety according to a pre-defined dimension of requirements and standards. Custom moderation, on the other hand, provides greater control compared to general moderation APIs or fine-tuned open source models though it requires more setup and maintenance.

    A common approach, when building a custom LLM-based filter, is to build an LLM-as-a-Judge filter as illustrated in Fig. 6.15. It a simple idea to use an LLM to judge the output of another LLM as well as user prompt in the context of your LLM-based application (please see Section “Model Based Evaluation” - Chapter Evals for design and best practices of LLM-based evals.)

    @@ -1553,17 +1545,17 @@

    -

    6.7. Case Study: Implementing a Safety Filter

    +

    6.7. Case Study: Implementing a Safety Filter

    We will implement a basic safety filter for a K-12 application that will be used to filter content in a chat interface. The application will be designed to be used in a classroom setting where students and teachers can interact with the model to ask questions and receive answers. The safety filter will be designed to filter out harmful content such as profanity, hate speech, and other inappropriate content.

    In this stylized case study, we will limit our scope to the implementation of a safety filter for user prompts. We will not cover the implementation of the application itself or filtering the model’s output but rather focus on the user prompt safety filter. In real-world applications, an input policy would be paramount to better define what safety means before we identify associated risks and consecutive implementation decisions. Here, we will discuss the implementation of safety through the design of the evals dataset (you will later see, skipping policy will lead to trouble later in the case study!)

    -

    6.7.1. Evals Dataset

    +

    6.7.1. Evals Dataset

    Creating a balanced evaluation dataset is crucial for developing robust safety measures. The dataset should be a well balanced set of “good” and “bad” samples to avoid biasing the model’s behavior in either direction.

    For this evaluation, we will create a dataset with NUM_SAMPLES examples, evenly split between good and bad samples (GOOD_SAMPLES and BAD_SAMPLES, respectively).

    -

    The good samples will be sourced from the UltraFeedback Binarized dataset [H4, 2024z], which contains high-quality, appropriate prompts that represent normal user interactions, often utilized to fine-tune models for instruction-following, truthfulness, honesty and helpfulness in a preference-based alignment process.

    +

    The good samples will be sourced from the UltraFeedback Binarized dataset [H4, 2024z], which contains high-quality, appropriate prompts that represent normal user interactions, often utilized to fine-tune models for instruction-following, truthfulness, honesty and helpfulness in a preference-based alignment process.

    The bad samples will come from two sources:

      -

    1. Profanity keywords from the Surge AI Profanity Dataset [Surge AI, 2024] - This provides examples of explicit inappropriate content.

      2. +

    2. Profanity keywords from the Surge AI Profanity Dataset [Surge AI, 2024] - This provides examples of explicit inappropriate content.

    3. Prompts sourced from Salad-Bench - These represent more subtle forms of harmful content like scams, harassment, or dangerous instructions, hence not necessarily mentioning an inappropriate keywords but rather a potentially harmful instruction.

    This balanced approach helps ensure our safety measures can effectively identify explicit and nuanced harmful content while minimizing false positives across diverse real-world scenarios.

    @@ -1576,7 +1568,7 @@

    -

    6.7.1.1. Bad Samples

    +

    6.7.1.1. Bad Samples

    def get_profanity_samples(num_samples, show_stats=True):
@@ -1718,7 +1710,7 @@ 
    
-
    6.7.1.2. Good Samples
    
+
    6.7.1.2. Good Samples
    
 
    
 
    
 
    def get_good_samples(num_samples):
@@ -1899,7 +1891,7 @@ 
    
-
    6.7.2. Safety Filters
    
+
    6.7.2. Safety Filters
    
 
    We will implement four safety filters, one for each of the following:
    
 
      
 
    1. LLM-Guard
      2. 
@@ -1965,7 +1957,7 @@ 
      
-
      6.7.2.1. LLM-Guard
      
+
      6.7.2.1. LLM-Guard
      
 
      Next, we implement a concrete validator using LLM Guard. The LLMGuardValidator class combines two key scanners:
      
 
        
 
      • BanTopics: Flags content containing banned topics
        • 
@@ -2058,7 +2050,7 @@ 
        
-
        6.7.2.2. Mistral Moderation API
        
+
        6.7.2.2. Mistral Moderation API
        
 
        You will need a Mistral API key to use the Mistral Moderation API. You can get one by signing up for a Mistral account and creating an API key, which we will assume is stored in a local .env file under the MISTRAL_API_KEY variable.
        
 
        The MistralValidator class implements a safety validator using Mistral’s moderation API. It takes text input and returns a ValidationResult indicating whether the text is unsafe based on Mistral moderation categories. Example:
        
 
        {'sexual': False,
@@ -2138,7 +2130,7 @@ 
        
-
        6.7.2.3. OpenAI Moderation API
        
+
        6.7.2.3. OpenAI Moderation API
        
 
        
 
        
 
        from openai import OpenAI
@@ -2202,7 +2194,7 @@ 
        
-
        6.7.2.4. Custom Judge Validator
        
+
        6.7.2.4. Custom Judge Validator
        
 
        The LLMJudgeValidator class implements a safety validator using GPT-4o-mini. It takes text input and returns a ValidationResult indicating whether the text is unsafe based on an input safety prompt.
        
 
        
 
        
@@ -2287,7 +2279,7 @@ 
        
-
        6.7.3. Benchmarking
        
+
        6.7.3. Benchmarking
        
 
        We are ready to run our four safety filters against our dataset. We will store validation results as well as elapsed time for each validator.
        
 
        
 
        
@@ -2776,7 +2768,7 @@ 
        6.7.4. Takeaways
        
+
        6.7.4. Takeaways
        
 
          
 
        • Safety is a complex problem and there is no one-size-fits-all solution.
          • 
 
        • Starting with a well-aligned policy is key to developing a robust data and evaluation framework.
          • 
@@ -2786,14 +2778,14 @@ 
          
-
          6.8. Conclusion
          
+
          6.8. Conclusion
          
 
          The rapid advancement of large language models has created an unsettling paradox: the same technologies that promise to revolutionize human-AI interaction also harbor significant risks that could undermine the very societies they aim to benefit. Our examination of various safety measures - from constitutional AI to red teaming - reveals that each approach has specific strengths and limitations when implemented in practice. However, instead of waiting for governments, organizations, and the public to catch up, we need to take action now.
          
 
          The case study on safety filters demonstrated the complexity of implementing even basic safety measures in real-world applications. What appears safe in one context may be inappropriate in another, and our current methods of safety evaluation often struggle with these nuances. The challenge of developing robust safety measures is further complicated by the potential for feedback loops in the training process - when models are fine-tuned on datasets that may contain hidden biases or problematic content.
          
 
          The path forward requires combining technical innovation with practical domain-specific wisdom. Safety in GenAI isn’t just a technical problem to be solved - it’s a mirror reflecting our own values, biases, and aspirations back at us. The growing focus on safety across the AI community, from open-source initiatives to corporate governance frameworks, provides a foundation for developing more robust safety measures. However, technologists working in isolation cannot solve these challenges - and may even perpetuate them unknowingly. Instead, domain experts across different verticals must come together to collaboratively define what safety means in the context of their specific users and broader society in work in collaboration with the AI community.
          
 
          Only through this cross-disciplinary collaboration can we move beyond the current uncertainty into a future where safety and innovation reinforce rather than oppose each other. This requires building bridges between technical experts, ethicists, policymakers, and the communities they serve to develop holistic frameworks that protect while enabling progress.
    -

    6.9. Citation

    +

    6.9. Citation

    CC BY-NC-SA 4.0

    @misc{tharsistpsouza2024tamingllms,
   author = {Tharsis T. P. Souza},
@@ -2807,68 +2799,68 @@ 
    
-
    6.10. References
    
+
    6.10. References
    
 
    
-
    
+
    
 [AI24]
 
    Meta AI. Llamaguard: llm-based input-output safeguard for human-ai conversations. Meta AI Research Publications, 2024. URL: https://ai.meta.com/research/publications/llama-guard-llm-based-input-output-safeguard-for-human-ai-conversations/.
    
 
    
-
    
+
    
 [ASA24]
 (1,2)
 
    Jide Alaga, Jonas Schuett, and Markus Anderljung. A grading rubric for ai safety frameworks. 2024. URL: https://arxiv.org/abs/2409.08751, arXiv:2409.08751.
    
 
    
-
    
+
    
 [ABC+23]
 (1,2)
 
    Amanda Askell, Yuntao Bai, Anna Chen, Deep Ganguli, Danny Hernandez, Jared Kaplan, Jackson Kernion, Ben Mann, Catherine Olsson, and Paul Christiano. Constitutional ai: harmlessness from ai feedback. 2023. URL: https://www.anthropic.com/research/constitutional-ai-harmlessness-from-ai-feedback.
    
 
    
-
    
-[BHY+24]
+
    
+[BHY+24]
 
    Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Trevor Darrell, Yuval Noah Harari, Ya-Qin Zhang, Lan Xue, Shai Shalev-Shwartz, Gillian Hadfield, Jeff Clune, Tegan Maharaj, Frank Hutter, Atılım Güneş Baydin, Sheila McIlraith, Qiqi Gao, Ashwin Acharya, David Krueger, Anca Dragan, Philip Torr, Stuart Russell, Daniel Kahneman, Jan Brauner, and Sören Mindermann. Managing extreme ai risks amid rapid progress. Science, 384(6698):842–845, 2024. URL: https://www.science.org/doi/abs/10.1126/science.adn0117, arXiv:https://www.science.org/doi/pdf/10.1126/science.adn0117, doi:10.1126/science.adn0117.
    
 
    
-
    
+
    
 [BBC+24]
-(1,2)
+(1,2)
 
    Victoria Benjamin, Emily Braca, Israel Carter, Hafsa Kanchwala, Nava Khojasteh, Charly Landow, Yi Luo, Caroline Ma, Anna Magarelli, Rachel Mirin, Avery Moyer, Kayla Simpson, Amelia Skawinski, and Thomas Heverin. Systematically analyzing prompt injection vulnerabilities in diverse llm architectures. 2024. URL: https://arxiv.org/abs/2410.23308, arXiv:2410.23308.
    
 
    
-
    
+
    
 [BMC+24]
-(1,2)
+(1,2)
 
    Dillon Bowen, Brendan Murphy, Will Cai, David Khachaturov, Adam Gleave, and Kellin Pelrine. Data poisoning in llms: jailbreak-tuning and scaling laws. 2024. URL: https://arxiv.org/abs/2408.02946, arXiv:2408.02946.
    
 
    
-
    
+
    
 [CMM+24]
 
    Erik Cambria, Lorenzo Malandri, Fabio Mercorio, Navid Nobani, and Andrea Seveso. Xai meets llms: a survey of the relation between explainable ai and large language models. 2024. URL: https://arxiv.org/abs/2407.15248, arXiv:2407.15248.
    
 
    
-
    
+
    
 [Edg24]
-(1,2)
+(1,2)
 
    Alec Edgington. How to exploit large language models for good or bad. SIAM News, 2024. URL: https://www.siam.org/publications/siam-news/articles/how-to-exploit-large-language-models-for-good-or-bad/.
    
 
    
-
    
+
    
 [Exa24]
 (1,2)
 
    Exabeam. Ai regulations and llm regulations: past, present, and future. Exabeam Blog, 2024. URL: https://www.exabeam.com/explainers/ai-cyber-security/ai-regulations-and-llm-regulations-past-present-and-future/.
    
 
    
-
    
+
    
 [GRB+24]
 
    Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. Bias and fairness in large language models: a survey. 2024. URL: https://arxiv.org/abs/2309.00770, arXiv:2309.00770.
    
 
    
-
    
+
    
 [H44z]
 
    Hugging Face H4. Ultrafeedback binarized dataset. 2024z. A dataset of binary preference data for training language models. URL: https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized.
    
 
    
-
    
+
    
 [HGP+22]
 
    Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. ToxiGen: a large-scale machine-generated dataset for adversarial and implicit hate speech detection. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 3309–3326. Dublin, Ireland, May 2022. Association for Computational Linguistics. URL: https://aclanthology.org/2022.acl-long.234, doi:10.18653/v1/2022.acl-long.234.
    
 
    
-
    
+
    
 [HYM+24]
-(1,2)
+(1,2)
 
    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, November 2024. URL: http://dx.doi.org/10.1145/3703155, doi:10.1145/3703155.
    
 
    
-
    
+
    
 [LDW+24]
 (1,2)
 
    Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wangmeng Zuo, Dahua Lin, Yu Qiao, and Jing Shao. Salad-bench: a hierarchical and comprehensive safety benchmark for large language models. 2024. URL: https://arxiv.org/abs/2402.05044, arXiv:2402.05044.
    
@@ -2878,31 +2870,31 @@ 
    (1,2)
 
    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: measuring how models mimic human falsehoods. 2022. URL: https://arxiv.org/abs/2109.07958, arXiv:2109.07958.
    
 
    
-
    
+
    
 [MPY+24]
 (1,2)
 
    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. Harmbench: a standardized evaluation framework for automated red teaming and robust refusal. 2024. URL: https://arxiv.org/abs/2402.04249, arXiv:2402.04249.
    
 
    
-
    
+
    
 [MLC24]
 
    MLCommons. Mlcommons ai illuminate benchmarks. 2024. A collection of standardized benchmarks for evaluating AI systems. URL: https://ailuminate.mlcommons.org/benchmarks/.
    
 
    
-
    
+
    
 [OAA+24]
 
    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report. 2024. URL: https://arxiv.org/abs/2303.08774, arXiv:2303.08774.
    
 
    
-
    
+
    
 [PNC+24]
 (1,2)
 
    Inkit Padhi, Manish Nagireddy, Giandomenico Cornacchia, Subhajit Chaudhury, Tejaswini Pedapati, Pierre Dognin, Keerthiram Murugesan, Erik Miehling, Martín Santillán Cooper, Kieran Fraser, Giulio Zizzo, Muhammad Zaid Hameed, Mark Purcell, Michael Desmond, Qian Pan, Inge Vejsbjerg, Elizabeth M. Daly, Michael Hind, Werner Geyer, Ambrish Rawat, Kush R. Varshney, and Prasanna Sattigeri. Granite guardian. 2024. URL: https://arxiv.org/abs/2412.07724, arXiv:2412.07724.
    
 
    
-
    
+
    
 [PHS+22]
 (1,2)
 
    Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. 2022. URL: https://arxiv.org/abs/2202.03286, arXiv:2202.03286.
    
 
    
-
    
-[SZW+24]
+
    
+[SZW+24]
 
    Oliver J. Sutton, Qinghua Zhou, Wei Wang, Desmond J. Higham, Alexander N. Gorban, Alexander Bastounis, and Ivan Y. Tyukin. Stealth edits to large language models. 2024. URL: https://arxiv.org/abs/2406.12670, arXiv:2406.12670.
    
 
    
 
    
@@ -2910,110 +2902,110 @@ 
    (1,2)
 
    Bertie Vidgen, Adarsh Agrawal, Ahmed M. Ahmed, Victor Akinwande, Namir Al-Nuaimi, Najla Alfaraj, Elie Alhajjar, Lora Aroyo, Trupti Bavalatti, Max Bartolo, Borhane Blili-Hamelin, Kurt Bollacker, Rishi Bomassani, Marisa Ferrara Boston, Siméon Campos, Kal Chakra, Canyu Chen, Cody Coleman, Zacharie Delpierre Coudert, Leon Derczynski, Debojyoti Dutta, Ian Eisenberg, James Ezick, Heather Frase, Brian Fuller, Ram Gandikota, Agasthya Gangavarapu, Ananya Gangavarapu, James Gealy, Rajat Ghosh, James Goel, Usman Gohar, Sujata Goswami, Scott A. Hale, Wiebke Hutiri, Joseph Marvin Imperial, Surgan Jandial, Nick Judd, Felix Juefei-Xu, Foutse Khomh, Bhavya Kailkhura, Hannah Rose Kirk, Kevin Klyman, Chris Knotz, Michael Kuchnik, Shachi H. Kumar, Srijan Kumar, Chris Lengerich, Bo Li, Zeyi Liao, Eileen Peters Long, Victor Lu, Sarah Luger, Yifan Mai, Priyanka Mary Mammen, Kelvin Manyeki, Sean McGregor, Virendra Mehta, Shafee Mohammed, Emanuel Moss, Lama Nachman, Dinesh Jinenhally Naganna, Amin Nikanjam, Besmira Nushi, Luis Oala, Iftach Orr, Alicia Parrish, Cigdem Patlak, William Pietri, Forough Poursabzi-Sangdeh, Eleonora Presani, Fabrizio Puletti, Paul Röttger, Saurav Sahay, Tim Santos, Nino Scherrer, Alice Schoenauer Sebag, Patrick Schramowski, Abolfazl Shahbazi, Vin Sharma, Xudong Shen, Vamsi Sistla, Leonard Tang, Davide Testuggine, Vithursan Thangarasa, Elizabeth Anne Watkins, Rebecca Weiss, Chris Welty, Tyler Wilbers, Adina Williams, Carole-Jean Wu, Poonam Yadav, Xianjun Yang, Yi Zeng, Wenhui Zhang, Fedor Zhdanov, Jiacheng Zhu, Percy Liang, Peter Mattson, and Joaquin Vanschoren. Introducing v0.5 of the ai safety benchmark from mlcommons. 2024. URL: https://arxiv.org/abs/2404.12241, arXiv:2404.12241.
    
 
    
-
    
+
    
 [VSK+24]
 (1,2)
 
    Bertie Vidgen, Nino Scherrer, Hannah Rose Kirk, Rebecca Qian, Anand Kannappan, Scott A. Hale, and Paul Röttger. Simplesafetytests: a test suite for identifying critical safety risks in large language models. 2024. URL: https://arxiv.org/abs/2311.08370, arXiv:2311.08370.
    
 
    
-
    
+
    
 [WMR24]
 
    Sandra Wachter, Brent Mittelstadt, and Chris Russell. Do large language models have a legal duty to tell the truth? Royal Society Open Science, 11(8):240197, 2024. URL: https://royalsocietypublishing.org/doi/abs/10.1098/rsos.240197, arXiv:https://royalsocietypublishing.org/doi/pdf/10.1098/rsos.240197, doi:10.1098/rsos.240197.
    
 
    
-
    
+
    
 [YLX24]
 
    Jiahao Yu, Xingwei Lin, and Xinyu Xing. Gptfuzzer: red teaming large language models with auto-generated safety test cases. Papers with Code, 2024. URL: https://paperswithcode.com/dataset/gptfuzzer.
    
 
    
-
    
+
    
 [ZYY+24]
 
    Shuning Zhang, Lyumanshan Ye, Xin Yi, Jingyu Tang, Bo Shui, Haobin Xing, Pengfei Liu, and Hewu Li. "ghost of the past": identifying and resolving privacy leakage from llm's memory through proactive user interaction. 2024. URL: https://arxiv.org/abs/2410.14931, arXiv:2410.14931.
    
 
    
-
    
-[Zho24]
+
    
+[Zho24]
 
    Qinghua Zhou. Stealth edits: detecting stealth edits in llm outputs. Hugging Face Spaces, 2024. URL: https://huggingface.co/spaces/qinghua-zhou/stealth-edits.
    
 
    
-
    
+
    
 [AmazonWServices24]
 
    Amazon Web Services. Amazon comprehend - natural language processing service. 2024. AWS natural language processing service for text analysis and content moderation. URL: https://aws.amazon.com/comprehend/.
    
 
    
-
    
+
    
 [Anthropic24]
 
    Anthropic. Anthropic's responsible scaling policy. Technical Report, Anthropic, 2024. URL: https://www-cdn.anthropic.com/1adf000c8f675958c2ee23805d91aaade1cd4613/responsible-scaling-policy.pdf.
    
 
    
-
    
+
    
 [CenterfASafety24a]
 
    Center for AI Safety. Harmbench. GitHub repository, 2024. Framework for evaluating language model safety. URL: https://github.com/centerforaisafety/HarmBench.
    
 
    
-
    
+
    
 [CenterfASafety24b]
 
    Center for AI Safety. Harmbench leaderboard. 2024. Leaderboard tracking performance of language models on safety benchmarks. URL: https://www.harmbench.org/results.
    
 
    
-
    
+
    
 [DeepMind24]
 (1,2)
 
    DeepMind. The frontier safety framework. Technical Report, DeepMind, 2024. URL: https://storage.googleapis.com/deepmind-media/DeepMind.com/Blog/introducing-the-frontier-safety-framework/fsf-technical-report.pdf.
    
 
    
-
    
+
    
 [EuropeanMAgency24]
 
    European Medicines Agency. Guiding principles for the use of large language models in regulatory science and medicines regulatory activities. Guidance Document, European Medicines Agency, 2024. URL: https://www.ema.europa.eu/en/documents/other/guiding-principles-use-large-language-models-regulatory-science-medicines-regulatory-activities_en.pdf.
    
 
    
-
    
+
    
 [FinancialIRAuthority24]
 
    Financial Industry Regulatory Authority. Artificial intelligence, including large language models and generative ai. Regulatory Notice 24-09, FINRA, 2024. URL: https://www.finra.org/rules-guidance/notices/24-09.
    
 
    
-
    
+
    
 [IBM24]
 
    IBM. Ibm watsonx.ai risk atlas. 2024. A framework for identifying and mitigating risks in AI systems. URL: https://www.ibm.com/docs/en/watsonx/saas?topic=ai-risk-atlas.
    
 
    
-
    
+
    
 [LibraryoCongress23]
 
    Library of Congress. China: generative ai measures finalized. July 2023. URL: https://www.loc.gov/item/global-legal-monitor/2023-07-18/china-generative-ai-measures-finalized/.
    
 
    
-
    
+
    
 [MistralAI24]
 
    Mistral AI. Mistral moderation: a technical report. 2024. URL: https://mistral.ai/news/mistral-moderation/.
    
 
    
-
    
+
    
 [MLSTeam24]
 
    ML Safety Team. Safebench: a comprehensive benchmark for llm safety evaluation. ML Safety Website, 2024. URL: https://www.mlsafety.org/safebench.
    
 
    
-
    
+
    
 [NationalIoSaTechnology24]
 
    National Institute of Standards and Technology. Ai risk management framework. Technical Report, National Institute of Standards and Technology, 2024. URL: https://www.nist.gov/itl/ai-risk-management-framework.
    
 
    
-
    
+
    
 [NVIDIA24]
 
    NVIDIA. Nemo-guardrails: an open-source toolkit for building reliable and safe llm applications. 2024. A framework for creating reliable and safe LLM applications with customizable guardrails. URL: https://github.com/NVIDIA/NeMo-Guardrails.
    
 
    
-
    
+
    
 [OpenAI24a]
 
    OpenAI. Openai moderation api. 2024. Documentation for OpenAI's content moderation API. URL: https://platform.openai.com/docs/guides/moderation.
    
 
    
-
    
+
    
 [OpenAI24b]
 (1,2)
 
    OpenAI. Openai preparedness framework. Technical Report, OpenAI, 2024. URL: https://cdn.openai.com/openai-preparedness-framework-beta.pdf.
    
 
    
-
    
+
    
 [OpenSafetyLab24a]
 
    OpenSafetyLab. Salad-bench leaderboard. Hugging Face Space, 2024. URL: https://huggingface.co/spaces/OpenSafetyLab/Salad-Bench-Leaderboard.
    
 
    
-
    
+
    
 [OpenSafetyLab24b]
 
    OpenSafetyLab. Salad-data: a hierarchical and comprehensive safety dataset for large language models. Hugging Face Dataset, 2024. URL: https://huggingface.co/datasets/OpenSafetyLab/Salad-Data.
    
 
    
-
    
+
    
 [ProtectAI24]
 
    ProtectAI. Llm-guard: comprehensive safety and security framework for large language models. 2024. An open-source toolkit for LLM security and safety. URL: https://github.com/protectai/llm-guard.
    
 
    
-
    
+
    
 [SurgeAI24]
 
    Surge AI. Surge ai profanity dataset. GitHub repository, 2024. A comprehensive dataset for training and evaluating profanity detection models. URL: https://github.com/surge-ai/profanity.
    
 
    
-
    
+
    
 [UKGovernment24]
 
    UK Government. Ai regulation: a pro-innovation approach. White Paper, Department for Science, Innovation and Technology, 2024. URL: https://www.gov.uk/government/publications/ai-regulation-a-pro-innovation-approach/white-paper.
    
 
    
-
    
+
    
 [UNICEF24]
 
    UNICEF. Policy guidance on ai for children. Policy Report, UNICEF Office of Research - Innocenti, 2024. URL: https://www.unicef.org/innocenti/reports/policy-guidance-ai-children.
    
 
    
diff --git a/tamingllms/_build/html/notebooks/structured_output.html b/tamingllms/_build/html/notebooks/structured_output.html
index d968608..e70b068 100644
--- a/tamingllms/_build/html/notebooks/structured_output.html
+++ b/tamingllms/_build/html/notebooks/structured_output.html
@@ -236,7 +236,7 @@
           
    
             
   
    
-
    4. Wrestling with Structured Output
    
+
    4. Wrestling with Structured Output
    
 
    
 
    In limits, there is freedom. Creativity thrives within structure.
    
 
    —Julia B. Cameron
    
@@ -244,43 +244,43 @@ 
    
 
    Contents
    
     
 
 
    
-
    4.1. Introduction
    
+
    4.1. Introduction
    
 
    Large language models (LLMs) excel at generating human-like text, but they often struggle to produce output in a structured format consistently. This poses a significant challenge when we need LLMs to generate data that can be easily processed by other systems, such as databases, APIs, or other software applications.   Sometimes, even with a well-crafted prompt, an LLM might produce an unstructured response when a structured one is expected. This can be particularly challenging when integrating LLMs into systems that require specific data formats.
    
 
    As a motivating example, consider the following simple task: Given a segment of a SEC financial filing, generate a two-person discussion about the key financial data from the text in JSON format, simulating what would be a real-world discussion about the underlying companies’ disclosed financial information. We would like to generate a structured output that can be easily parsed and integrated with other systems.
    
 
    Throughout this notebook, we will consider as input a segment of a sample SEC filing of Apple Inc.
    
@@ -386,7 +386,7 @@ 
    
-
    4.2. Problem Statement
    
+
    4.2. Problem Statement
    
 
    Obtaining structured output from LLMs presents several significant challenges:
    
 
      
 
    • Inconsistency: LLMs often produce unpredictable results, sometimes generating well-structured output and other times deviating from the expected format.
      • 
@@ -395,8 +395,8 @@ 
      
-
      4.3. User Needs
      
-
      What user needs drive the demand for LLM output constraints when building LLM-based applications? In a recent work by Google Research [Liu et al., 2024], the authors explore the user need for constraints on the output of large language models, drawing on a survey of 51 industry professionals who use LLMs in their work. These needs can be broadly categorized as follows:
      
+
      4.3. User Needs
      
+
      What user needs drive the demand for LLM output constraints when building LLM-based applications? In a recent work by Google Research [Liu et al., 2024], the authors explore the user need for constraints on the output of large language models, drawing on a survey of 51 industry professionals who use LLMs in their work. These needs can be broadly categorized as follows:
      
 
      1. Improving Developer Efficiency and Workflow
      
 
        
 
      • Reducing Trial and Error in Prompt Engineering: Developers find the process of crafting prompts to elicit desired output formats to be time-consuming, often involving extensive testing and iteration. LLM output constraints could make this process more efficient and predictable.
        • 
@@ -418,10 +418,10 @@ 
        
-
        4.4. Solutions
        
+
        4.4. Solutions
        
 
        Several strategies and tools can be employed to address the challenges of structured output from LLMs.
        
 
        
-
        4.4.1. Strategies
        
+
        4.4.1. Strategies
        
 
          
 
        • Schema Guidance: Providing the LLM with a clear schema or blueprint of the desired output structure helps to constrain its generation and improve consistency. This can be achieved by using tools like Pydantic to define the expected data structure and then using that definition to guide the LLM’s output.
          • 
 
        • Output Parsing: When LLMs don’t natively support structured output, parsing their text output using techniques like regular expressions or dedicated parsing libraries can extract the desired information. For example, you can use regular expressions to extract specific patterns from the LLM’s output, or you can use libraries like Pydantic to parse the output into structured data objects.
          • 
@@ -429,9 +429,9 @@ 
          
-
          4.4.2. Techniques and Tools
          
+
          4.4.2. Techniques and Tools
          
 
          
-
          4.4.2.1. One-Shot Prompts
          
+
          4.4.2.1. One-Shot Prompts
          
 
          In one-shot prompting, you provide a single example of the desired output format within the prompt.
          
 
          
 
          
@@ -498,7 +498,7 @@ 
          
-
          4.4.2.2. Structured Output with Provider-Specific APIs
          
+
          4.4.2.2. Structured Output with Provider-Specific APIs
          
 
          One-shot prompting is a simple technique that can lead to material improvements in structured output, though may not be sufficient for complex (e.g. nested) structures and / or when the model’s output needs to be restricted to a specific set of options or types.
          
 
          Provider-specific APIs can offer ways to handle those challenges. We will explore two approaches here using OpenAI’s API:
          
 
            
@@ -507,7 +507,7 @@ 
            
-
            4.4.2.3. JSON Mode
            
+
            4.4.2.3. JSON Mode
            
 
            JSON mode is a feature provided by most LLM API providers, such as OpenAI, that allows the model to generate output in JSON format. This is particularly useful when you need structured data as a result, such as when parsing the output programmatically or integrating it with other systems that require JSON input. As depicted in Fig. 4.1, JSON mode is implemented by instructing theLLM model to use JSON as response format and optionally defining a target schema.
            
 
            
 JSON Mode
@@ -645,7 +645,7 @@ 
            
-
            4.4.3. LangChain
            
+
            4.4.3. LangChain
            
 
            LangChain is a framework designed to simplify the development of LLM applications. It provider an abstraction layer over many LLM providers, including OpenAI, that offers several tools for parsing structured output.
            
 
            In particular, LangChain offers the with_structured_output method, which can be used with LLMs that support structured output APIs, allowing you to enforce a schema directly within the prompt.
            
 
            
@@ -705,9 +705,9 @@ 
            .with_structured_output() can be found here.
            
 
          
 
          
-
          4.4.4. Outlines
          
-
          Outlines [Outlines, 2024] is a library specifically focused on structured text generation from LLMs. Under the hood, Outlines works by adjusting the probability distribution of the model’s output logits - the raw scores from the final layer of the neural network that are normally converted into text tokens. By introducing carefully crafted logit biases, Outlines can guide the model to prefer certain tokens over others, effectively constraining its outputs to a predefined set of valid options.
          
-
          The authors solve the general guided generation problem [Willard and Louf, 2023], which as a consequence solves the problem of structured output generation, in LLMs by introducing an efficient indexing approach that reformulates neural text generation using finite-state machines (FSMs).
          
+
          4.4.4. Outlines
          
+
          Outlines [Outlines, 2024] is a library specifically focused on structured text generation from LLMs. Under the hood, Outlines works by adjusting the probability distribution of the model’s output logits - the raw scores from the final layer of the neural network that are normally converted into text tokens. By introducing carefully crafted logit biases, Outlines can guide the model to prefer certain tokens over others, effectively constraining its outputs to a predefined set of valid options.
          
+
          The authors solve the general guided generation problem [Willard and Louf, 2023], which as a consequence solves the problem of structured output generation, in LLMs by introducing an efficient indexing approach that reformulates neural text generation using finite-state machines (FSMs).
          
 
          They define the next token generation as a random variable:
          
 
          
 \[s_{t+1} \sim \text{Categorical}(\alpha) \text{ where } \alpha = \text{LLM}(S_t, \theta)\]
          
@@ -738,7 +738,7 @@ 
          \(\tilde{s}_{t+1}\) is the next token sampled under constraints
          
 
        
 
        This formulation allows the masking operation to guide the generation process by zeroing out probabilities of invalid tokens according to the finite state machine states. But instead of checking the entire vocabulary (size N) at each generation step (O(N) complexity) to enforce output constraints, they convert constraints (regex/grammar) into FSM states and build an index mapping FSM states to valid vocabulary tokens. This achieves O(1) average complexity for token generation.
        
-
        In summary, there are two stages in the Outlines framework [Tran-Thien, 2024]:
        
+
        In summary, there are two stages in the Outlines framework [Tran-Thien, 2024]:
        
 
          
 
        1. Preprocessing Step: Outlines converts a character-level deterministic finite automaton (DFA) testing whether a string matches a regex into a token-level DFA testing whether a token sequence is decoded in a string matching the regex.
          2. 
 
        2. Decoding Step: At decoding time, the DFA is used to determine, for each new token, which potential tokens are allowed. Starting from the initial state of the DFA, the allowed tokens are determined by the outgoing transitions from the current state. The corresponding mask is applied to the next token probabilities and these probabilities are renormalized. A new token can then be sampled and the state of the DFA updated.
          3. 
@@ -849,11 +849,12 @@ 
          [LangChain, 2024b].
          
 
        
 
        
-
        4.4.5. Ollama
        
-
        Ollama is a popular tool that allows you to run large language models (LLMs) locally. It has recently added support for structured output generation. The current ollama implementation leverages llama.cpp GBNF (GGML BNF) grammars [Ggerganov, 2024] to enable structured output generation.
        
-
        llama.cpp GBNF forces language models to generate output in specific, predefined formats by constraining their outputs to follow precise rules and patterns. The system accomplishes this through a formal grammar specification that defines exactly how valid outputs can be constructed. It’s essentially an extension of BNF (Backus-Naur Form) [Wikipedia contributors, 2024] with some modern regex-like features added. These rules carefully define what elements are allowed, how they can be combined, and what patterns of repetition and sequencing are valid. By enforcing these constraints during generation, GBNF ensures the model’s output strictly adheres to the desired format.
        
+
        4.4.5. Ollama
        
+
        Ollama is a popular tool that allows you to run large language models (LLMs) locally. It has recently added support for structured output generation. The current ollama implementation leverages llama.cpp GBNF (GGML BNF) grammars [Ggerganov, 2024] to enable structured output generation.
        
+
        llama.cpp GBNF forces language models to generate output in specific, predefined formats by constraining their outputs to follow precise rules and patterns. The system accomplishes this through a formal grammar specification that defines exactly how valid outputs can be constructed. It’s essentially an extension of BNF (Backus-Naur Form) [Wikipedia contributors, 2024] with some modern regex-like features added. These rules carefully define what elements are allowed, how they can be combined, and what patterns of repetition and sequencing are valid. By enforcing these constraints during generation, GBNF ensures the model’s output strictly adheres to the desired format.
        
 
        Ollama first introduced structured output generation in version 0.5.1 providing support for JSON output but highlighting additional formats are coming soon.
        
 
        Let’s replicate our previous structured output generation example with Ollama. First, make sure you have Ollama installed. You can find installation instructions here.
        
 
        curl -fsSL https://ollama.com/install.sh | sh
@@ -949,9 +950,9 @@ 
        
-
        4.5. Discussion
        
+
        4.5. Discussion
        
 
        
-
        4.5.1. Comparing Solutions
        
+
        4.5.1. Comparing Solutions
        
 
        The choice of framework for structured LLM output depends heavily on specific constraints, requirements and use cases. LangChain is the most used LLM framework today with a large developer community base however its structured output support depends on the underlying LLM provider support. Ollama enables straightforward local deployment and experimentation democratizing access to LLMs while fostering privacy and control, however today it only offers JSON format with further formats to come. Outlines emerges as a solution with great flexibility and control over output structure while providing support for a wide range of LLMs. Table 4.1 provides a summary comparison of the different frameworks.

    Table 6.2 Rules-Based Safety Filtering Tools.
    @@ -997,7 +998,7 @@

    -

    4.5.2. Best Practices

    +

    4.5.2. Best Practices

    • Clear Schema Definition: Define the desired output structure clearly. This can be done in several ways including schemas, types, or Pydantic models as appropriate. This ensures the LLM knows exactly what format is expected.

    • Descriptive Naming: Use meaningful names for fields and elements in your schema. This makes the output more understandable and easier to work with.

      • @@ -1006,23 +1007,23 @@

      -

      4.5.3. Research and Ongoing Debate

      +

      4.5.3. Research and Ongoing Debate

      The use of structured output for Large Language Models (LLMs) is a developing area. While the ability to constrain LLM outputs offer clear benefits in parsing, robustness, and integration, there is growing debate on whether it also potentially comes at the cost of performance as well as reasoning abilities. Research in this area should be taken with a grain of salt since findings are mixed and often depend on the specific task and model family at hand furthermore model families are not always comparable and are getting updated by the day! Nonetheless, early findings provide some interesting insights as to why there is no one-size-fits-all solution when it comes to LLMs structured output.

      -

      There is some evidence indicating that LLMs may have bias in their handling of different output formats [Long et al., 2024]. The study examined common output structures like multiple-choice answers, wrapped text, lists, and key-value mappings. The authors analyzed key LLM model families, namely Gemma, Mistral, and ChatGPT, uncovering bias across multiple tasks and formats. The researchers attributed these biases to the models’ underlying token distributions for different formats. An example of this format bias emerged in the comparison between JSON and YAML outputs. While models like Mistral and Gemma excelled at generating JSON structures, they performed notably worse with YAML. Their YAML outputs often contained extraneous information that degrades output quality. This disparity likely stems from JSON’s prevalence in training data, highlighting how a format’s popularity directly influences model performance. While the studied models can be probably considered outdated by now since models are getting updated on a rapidly fashion, it is important to remark that addressing format bias is critical for advancing LLMs and ensuring their reliable application in real-world scenarios.

      -

      Recent research “Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models” [Tam et al., 2024] suggests that imposing format restrictions on LLMs might impact their performance, particularly in reasoning-intensive tasks. Further evidence [Aider, 2024] suggests LLMs may produce lower quality code if they’re asked to return it as part of a structured JSON response, in particular:

      +

      There is some evidence indicating that LLMs may have bias in their handling of different output formats [Long et al., 2024]. The study examined common output structures like multiple-choice answers, wrapped text, lists, and key-value mappings. The authors analyzed key LLM model families, namely Gemma, Mistral, and ChatGPT, uncovering bias across multiple tasks and formats. The researchers attributed these biases to the models’ underlying token distributions for different formats. An example of this format bias emerged in the comparison between JSON and YAML outputs. While models like Mistral and Gemma excelled at generating JSON structures, they performed notably worse with YAML. Their YAML outputs often contained extraneous information that degrades output quality. This disparity likely stems from JSON’s prevalence in training data, highlighting how a format’s popularity directly influences model performance. While the studied models can be probably considered outdated by now since models are getting updated on a rapidly fashion, it is important to remark that addressing format bias is critical for advancing LLMs and ensuring their reliable application in real-world scenarios.

      +

      Recent research “Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models” [Tam et al., 2024] suggests that imposing format restrictions on LLMs might impact their performance, particularly in reasoning-intensive tasks. Further evidence [Aider, 2024] suggests LLMs may produce lower quality code if they’re asked to return it as part of a structured JSON response, in particular:

      • Potential performance degradation: Enforcing structured output, especially through constrained decoding methods like JSON-mode, can negatively impact an LLM’s reasoning abilities. This is particularly evident in tasks that require multi-step reasoning or complex thought processes.

      • Overly restrictive schemas: Imposing strict schemas can limit the expressiveness of LLM outputs and may hinder their ability to generate creative or nuanced responses. In certain cases, the strictness of the schema might outweigh the benefits of structured output.

      • Increased complexity in prompt engineering: Crafting prompts that effectively guide LLMs to generate structured outputs while maintaining performance can be challenging. It often requires careful consideration of the schema, the task instructions, and the desired level of detail in the response.

      -

      On the other hand, those findings are not without criticism. The .txt team challenges the work of [Tam et al., 2024]. The rebuttal argues that structured generation, when done correctly, actually improves performance.

      +

      On the other hand, those findings are not without criticism. The .txt team challenges the work of [Tam et al., 2024]. The rebuttal argues that structured generation, when done correctly, actually improves performance.

      Structured vs Unstructured Results by .txt team

      Fig. 4.3 Structured vs Unstructured Results by .txt team.

      -

      The .txt team presents compelling evidence through their reproduction of the paper’s experiments. While their unstructured results align with the original paper’s findings, their structured results paint a dramatically different picture - demonstrating that structured generation actually improves performance (see Fig. 4.3). The team has made their experimental notebooks publicly available on GitHub for independent verification [Dottxt, 2024].

      +

      The .txt team presents compelling evidence through their reproduction of the paper’s experiments. While their unstructured results align with the original paper’s findings, their structured results paint a dramatically different picture - demonstrating that structured generation actually improves performance (see Fig. 4.3). The team has made their experimental notebooks publicly available on GitHub for independent verification [Dottxt, 2024].

      .txt team identifies several flaws in the methodology of “Let Me Speak Freely?” that they believe led to inaccurate conclusions:

      • The paper finds that structured output improves performance on classification tasks but doesn’t reconcile this finding with its overall negative conclusion about structured output.

        • @@ -1036,15 +1037,15 @@

        -

        4.6. Conclusion

        +

        4.6. Conclusion

        Extracting structured output from LLMs is crucial for integrating them into real-world applications. By understanding the challenges and employing appropriate strategies and tools, developers can improve the reliability and usability of LLM-powered systems, unlocking their potential to automate complex tasks and generate valuable insights.

        -

        4.7. Acknowledgements

        +

        4.7. Acknowledgements

        We would like to thank Cameron Pfiffer from the .txt team for his insightful review and feedback.

        -

        4.8. Citation

        +

        4.8. Citation

        CC BY-NC-SA 4.0

        @misc{tharsistpsouza2024tamingllms,
   author = {Tharsis T. P. Souza},
@@ -1058,47 +1059,51 @@ 
        
-
        4.9. References
        
-
        
-
        
-[Aid24]
+
        4.9. References
        
+
        
+
        
+[Aid24]
 
        Aider. Code in json: structured output for llms. https://aider.chat/2024/08/14/code-in-json.html, 2024. Accessed: 2024.
        
 
        
-
        
-[Dot24]
+
        
+[Dot24]
 
        Dottxt. Say what you mean: demos. https://github.com/dottxt-ai/demos/tree/main/say-what-you-mean, 2024. Accessed: 2024.
        
 
        
-
        
-[Gge24]
+
        
+[Gge24]
 
        Ggerganov. Llama.cpp grammars documentation. https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md, 2024. Accessed: 2024.
        
 
        
-
        
+
        
+[Lan4b]
+
        LangChain. Outlines integration documentation. Online Documentation, 2024b. Documentation on integrating Outlines library with LangChain for structured generation. URL: https://python.langchain.com/docs/integrations/chat/outlines/.
        
+
        
+
        
 [LLF+24]
 
        Michael Xieyang Liu, Frederick Liu, Alexander J. Fiannaca, Terry Koo, Lucas Dixon, Michael Terry, and Carrie J. Cai. "we need structured output": towards user-centered constraints on large language model output. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, CHI EA '24. New York, NY, USA, 2024. Association for Computing Machinery. URL: https://doi.org/10.1145/3613905.3650756, doi:10.1145/3613905.3650756.
        
 
        
-
        
-[LNS+24]
+
        
+[LNS+24]
 
        Do Xuan Long, Hai Nguyen Ngoc, Tiviatis Sim, Hieu Dao, Shafiq Joty, Kenji Kawaguchi, Nancy F Chen, and Min-Yen Kan. Llms are biased towards output formats! systematically evaluating and mitigating output format bias of llms. arXiv preprint arXiv:2408.08656, 2024.
        
 
        
-
        
+
        
 [Out24]
 
        Outlines. Type-safe structured output from llms. https://dottxt-ai.github.io/outlines/latest/, 2024. Accessed: 2024.
        
 
        
-
        
+
        
 [TWT+24]
-(1,2)
+(1,2)
 
        Zhi Rui Tam, Cheng-Kuang Wu, Yi-Lin Tsai, Chieh-Yen Lin, Hung-yi Lee, and Yun-Nung Chen. Let me speak freely? a study on the impact of format restrictions on performance of large language models. 2024. URL: https://arxiv.org/abs/2408.02442, arXiv:2408.02442.
        
 
        
-
        
+
        
 [TT24]
 
        Vivien Tran-Thien. Fast, high-fidelity llm decoding with regex constraints. 2024. URL: https://vivien000.github.io/blog/journal/llm-decoding-with-regex-constraints.html.
        
 
        
-
        
+
        
 [WL23]
 
        Brandon T. Willard and Rémi Louf. Efficient guided generation for large language models. 2023. URL: https://arxiv.org/abs/2307.09702, arXiv:2307.09702.
        
 
        
-
        
-[Wikipediacontributors24]
+
        
+[Wikipediacontributors24]
 
        Wikipedia contributors. Backus naur form. https://en.wiktionary.org/wiki/Backus-Naur_form, 2024. Accessed: 2024.
        
 
        
 
        
diff --git a/tamingllms/_build/html/objects.inv b/tamingllms/_build/html/objects.inv
index 2c47894..f384334 100644
Binary files a/tamingllms/_build/html/objects.inv and b/tamingllms/_build/html/objects.inv differ
diff --git a/tamingllms/_build/html/searchindex.js b/tamingllms/_build/html/searchindex.js
index 8ed5084..456e202 100644
--- a/tamingllms/_build/html/searchindex.js
+++ b/tamingllms/_build/html/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"docnames": ["markdown/intro", "markdown/preface", "markdown/toc", "notebooks/alignment", "notebooks/evals", "notebooks/local", "notebooks/output_size_limit", "notebooks/safety", "notebooks/structured_output"], "filenames": ["markdown/intro.md", "markdown/preface.md", "markdown/toc.md", "notebooks/alignment.ipynb", "notebooks/evals.ipynb", "notebooks/local.ipynb", "notebooks/output_size_limit.ipynb", "notebooks/safety.ipynb", "notebooks/structured_output.ipynb"], "titles": ["2. Introduction", "1. Preface", "Taming LLMs", "7. Preference-Based Alignment", "5. The Evals Gap", "8. Breaking Free from Cloud Providers", "3. Output Size Limit", "6. Safety", "4. Wrestling with Structured Output"], "terms": {"am": [0, 1, 7], "alwai": [0, 3, 4, 5, 8], "do": [0, 3, 4, 5, 6, 7, 8], "which": [0, 3, 4, 5, 6, 7, 8], "cannot": [0, 3, 4, 5, 7], "order": [0, 3, 4, 7, 8], "mai": [0, 1, 3, 4, 5, 6, 7, 8], "learn": [0, 3, 4, 7], "how": [0, 1, 3, 4, 6, 7, 8], "pablo": [0, 4], "picasso": 0, "In": [0, 3, 4, 5, 6, 7, 8], "recent": [0, 3, 4, 7, 8], "year": [0, 2, 3, 4, 5, 6, 7, 8], "larg": [0, 1, 2, 3, 4, 5, 6, 7, 8], "languag": [0, 1, 2, 4, 5, 6, 7, 8], "model": [0, 1, 2, 7, 8], "llm": [0, 1, 3, 5, 6, 8], "have": [0, 1, 3, 4, 5, 6, 7, 8], "emerg": [0, 3, 7, 8], "transform": [0, 1, 3, 4, 7, 8], "forc": [0, 4, 8], "technologi": [0, 1, 4, 6, 7, 8], "promis": [0, 3, 4, 7], "revolution": [0, 7], "build": [0, 2, 3, 4, 5, 6, 7, 8], "product": [0, 1, 2, 3, 4, 5, 7, 8], "interact": [0, 3, 4, 5, 6, 7, 8], "comput": [0, 3, 4, 5, 6, 7, 8], "from": [0, 1, 4, 6, 7, 8], "chatgpt": [0, 3, 5, 8], "github": [0, 2, 3, 4, 5, 6, 7, 8], "copilot": 0, "claud": [0, 3, 4, 5, 6, 7], "artifact": 0, "system": [0, 3, 4, 5, 6, 7, 8], "captur": [0, 1, 3, 4, 7], "public": [0, 3, 4, 7], "imagin": 0, "spark": 0, "gold": [0, 3, 4, 7], "rush": 0, "ai": [0, 3, 4, 5, 8], "power": [0, 2, 3, 4, 5, 6, 7, 8], "applic": [0, 1, 2, 3, 5, 6, 7, 8], "howev": [0, 3, 4, 5, 6, 7, 8], "beneath": 0, "surfac": [0, 4], "technolog": [0, 1, 4, 7], "revolut": 0, "li": [0, 3, 4, 7], "complex": [0, 1, 3, 4, 5, 6, 7, 8], "landscap": [0, 3, 4], "practition": [0, 1, 4], "must": [0, 3, 4, 5, 6, 7], "navig": [0, 2, 4, 7], "focus": [0, 3, 4, 5, 6, 7, 8], "bring": [0, 3, 5], "awar": [0, 3, 4, 6, 7], "limit": [0, 1, 4, 5, 7, 8], "har": [0, 2, 4, 6], "solut": [0, 2, 4, 5, 6, 7], "overcom": [0, 4, 6], "them": [0, 1, 3, 4, 5, 6, 7, 8], "robust": [0, 3, 4, 5, 6, 7, 8], "It": [0, 3, 4, 5, 6, 7, 8], "offer": [0, 3, 4, 5, 6, 7, 8], "critic": [0, 2, 3, 4, 6, 7, 8], "implement": [0, 2, 3, 4, 6, 8], "back": [0, 4, 7, 8], "reproduc": [0, 1, 2, 4, 5], "exampl": [0, 1, 2, 3, 4, 5, 7, 8], "while": [0, 1, 2, 3, 4, 5, 6, 7, 8], "mani": [0, 1, 3, 4, 5, 6, 7, 8], "resourc": [0, 3, 4, 5, 6, 7], "cover": [0, 3, 4, 5, 6, 7], "capabl": [0, 1, 2, 4, 5, 6, 7, 8], "specif": [0, 3, 4, 5, 6], "hidden": [0, 3, 7], "pitfal": [0, 1, 3, 4, 5, 6, 8], "engin": [0, 1, 2, 3, 4, 5, 7, 8], "technic": [0, 1, 2, 3, 4, 5, 6, 8], "manag": [0, 1, 2, 4, 5, 6, 7, 8], "face": [0, 3, 4, 5, 7], "when": [0, 1, 2, 3, 4, 5, 6, 7, 8], "comprehens": [0, 2, 3, 4, 5, 6, 7, 8], "guid": [0, 1, 3, 4, 5, 6, 7, 8], "leverag": [0, 3, 4, 6, 7, 8], "battl": [0, 2], "test": [0, 2, 3, 5, 7, 8], "tool": [0, 1, 3, 6], "throughout": [0, 4, 5, 6, 7, 8], "tackl": [0, 3, 4, 7], "follow": [0, 3, 4, 5, 6, 7, 8], "non": [0, 3, 5, 7, 8], "exhaust": 0, "list": [0, 3, 4, 5, 6, 7, 8], "structur": [0, 3, 4, 5, 6, 7], "un": 0, "reliabl": [0, 1, 3, 4, 5, 7, 8], "struggl": [0, 1, 3, 4, 5, 7, 8], "maintain": [0, 1, 3, 4, 5, 6, 7, 8], "consist": [0, 1, 3, 4, 5, 6, 7, 8], "output": [0, 1, 3, 4, 5, 7], "format": [0, 3, 4, 5, 6, 7, 8], "complic": [0, 7], "integr": [0, 1, 3, 4, 5, 8], "larger": [0, 3, 4, 5, 6, 7, 8], "make": [0, 3, 4, 5, 6, 7, 8], "error": [0, 3, 4, 7, 8], "handl": [0, 2, 3, 4, 5, 6, 7, 8], "more": [0, 1, 3, 4, 5, 6, 7, 8], "size": [0, 3, 4, 5, 7, 8], "length": [0, 3, 4, 8], "constraint": [0, 1, 3, 4, 6, 7, 8], "strict": [0, 5, 7, 8], "token": [0, 1, 3, 4, 5, 7, 8], "both": [0, 3, 4, 5, 7], "input": [0, 3, 4, 5, 6, 7, 8], "requir": [0, 3, 5, 6, 7, 8], "care": [0, 3, 4, 7, 8], "chunk": [0, 3], "strategi": [0, 3, 4, 5, 6, 7], "long": [0, 1, 3, 4, 5, 7, 8], "form": [0, 3, 4, 5, 7, 8], "effect": [0, 1, 3, 4, 6, 7, 8], "tradit": [0, 3, 7], "softwar": [0, 1, 3, 5, 6, 7, 8], "methodologi": [0, 3, 4, 7, 8], "break": [0, 1, 3, 4, 6, 7], "down": [0, 1, 4, 5, 6, 7], "deal": [0, 3, 5], "determinist": [0, 8], "gener": [0, 1, 8], "new": [0, 2, 3, 4, 5, 6, 7, 8], "hallucin": [0, 1, 3, 4, 7, 8], "These": [0, 3, 4, 5, 6, 7, 8], "can": [0, 1, 3, 4, 5, 6, 7, 8], "plausibl": [0, 7], "sound": [0, 7], "entir": [0, 4, 5, 6, 8], "fabric": [0, 4, 7], "inform": [0, 3, 4, 5, 6, 7, 8], "creat": [0, 1, 3, 4, 5, 6, 7, 8], "signific": [0, 3, 4, 5, 6, 7, 8], "risk": [0, 1, 3, 4, 6], "safeti": [0, 3, 4, 8], "align": [0, 4, 5, 6, 7, 8], "harm": [0, 3, 4, 5], "bias": [0, 3, 4, 7, 8], "inappropri": [0, 3, 7], "safeguard": [0, 4, 7], "monitor": [0, 3, 4, 5, 7], "ensur": [0, 3, 4, 6, 7, 8], "safe": [0, 3, 4, 7, 8], "deploy": [0, 3, 4, 5, 7, 8], "cost": [0, 3, 4, 5, 7, 8], "optim": [0, 1, 4, 5, 6, 7], "The": [0, 1, 3, 5, 6, 7, 8], "financi": [0, 1, 3, 4, 6, 7, 8], "oper": [0, 3, 4, 5, 6, 7, 8], "base": [0, 1, 5, 8], "quickli": [0, 3, 6], "becom": [0, 3, 4, 5, 7, 8], "prohibit": [0, 3, 4, 5], "without": [0, 1, 3, 4, 5, 6, 7, 8], "observ": [0, 3, 4, 7, 8], "vendor": [0, 4], "lock": [0, 3], "cloud": [0, 3, 4, 7, 8], "provid": [0, 3, 4, 6, 7], "depend": [0, 3, 4, 5, 8], "through": [0, 1, 2, 3, 4, 5, 6, 7, 8], "proprietari": [0, 3, 5, 7, 8], "infrastructur": [0, 5], "difficult": [0, 3, 4, 7], "switch": [0, 5], "self": [0, 3, 4, 5, 7], "host": [0, 4, 5, 7], "take": [0, 2, 3, 4, 5, 6, 7, 8], "hand": [0, 6, 7, 8], "focu": [0, 2, 3, 4, 6, 7, 8], "access": [0, 3, 4, 5, 6, 7, 8], "all": [0, 1, 3, 4, 5, 6, 7, 8], "ar": [0, 1, 3, 4, 5, 7, 8], "fulli": [0, 3, 4, 6, 7], "document": [0, 3, 4, 6, 7, 8], "allow": [0, 4, 5, 6, 7, 8], "reader": [0, 2], "replic": [0, 4, 7, 8], "result": [0, 3, 4, 5, 6, 7, 8], "exactli": [0, 4, 8], "design": [0, 1, 3, 5, 6, 8], "run": [0, 3, 4, 5, 7, 8], "consum": [0, 3, 4, 5, 7, 8], "grade": [0, 3, 4, 5, 7], "hardwar": [0, 3, 4, 5], "expens": [0, 3, 4, 7], "avail": [0, 3, 4, 5, 6, 7, 8], "notebook": [0, 3, 8], "modifi": [0, 3, 4, 7], "extend": [0, 3, 4, 5, 8], "built": [0, 4, 5, 7, 8], "us": [0, 1, 3, 5, 6, 7, 8], "free": [0, 1, 3, 4, 7], "everyon": [0, 4], "minim": [0, 3, 4, 5, 7, 8], "framework": [0, 3, 4], "wai": [0, 3, 4, 5, 6, 7, 8], "priorit": [0, 3, 4, 5, 7], "transpar": [0, 3, 4, 7, 8], "visibl": [0, 4], "being": [0, 3, 4, 5, 7], "better": [0, 2, 3, 4, 5, 6, 7], "understand": [0, 1, 2, 3, 4, 5, 6, 7, 8], "custom": [0, 3, 4, 5], "flexibl": [0, 4, 5, 6, 7, 8], "adapt": [0, 3, 4, 5, 6, 7], "case": [0, 4, 6, 8], "unlik": [0, 3, 4, 5], "black": [0, 3], "box": [0, 5], "commerci": [0, 3, 4, 7, 8], "most": [0, 3, 4, 5, 6, 7, 8], "freeli": [0, 8], "foster": [0, 3, 4, 7, 8], "reduc": [0, 3, 4, 5, 6, 7, 8], "independ": [0, 4, 7, 8], "freedom": [0, 8], "architectur": [0, 3, 4, 5, 6, 8], "decis": [0, 3, 4, 5, 7, 8], "keep": [0, 3, 4, 5, 6, 7], "principl": [0, 3, 4, 5, 7], "itself": [0, 3, 4, 5, 7], "live": [0, 1, 4, 7], "evolv": [0, 3, 4, 6, 7], "chang": [0, 3, 4, 5, 7], "encourag": [0, 3, 4, 7, 8], "report": [0, 3, 4, 5, 7, 8], "suggest": [0, 3, 4, 5, 7, 8], "improv": [0, 3, 4, 5, 6, 7, 8], "contribut": [0, 4, 6, 7], "via": [0, 3, 4, 5, 7, 8], "pull": [0, 5], "request": [0, 3, 4, 5, 6, 7, 8], "share": [0, 3, 4, 5, 7, 8], "own": [0, 3, 4, 6, 7], "experi": [0, 3, 4, 5, 6, 7, 8], "commun": [0, 3, 4, 7, 8], "propos": [0, 4, 7], "chapter": [0, 3, 4, 5, 6, 7, 8], "section": [0, 3, 4, 5, 6, 7, 8], "found": [0, 3, 4, 5, 8], "http": [0, 1, 2, 3, 4, 5, 6, 7, 8], "com": [0, 2, 3, 4, 5, 6, 7, 8], "souzatharsi": [0, 2, 3, 4, 5, 6, 7, 8], "tamingllm": [0, 2, 3, 4, 5, 6, 7, 8], "whether": [0, 3, 4, 6, 7, 8], "you": [0, 1, 3, 4, 5, 6, 7, 8], "ve": [0, 5], "typo": [0, 7], "want": [0, 1, 3, 5, 6, 7, 8], "welcom": 0, "look": [0, 2, 3, 4, 5, 7], "our": [0, 1, 3, 4, 5, 6, 7, 8], "goal": [0, 1, 3, 4, 6, 7], "discourag": 0, "enabl": [0, 3, 4, 5, 6, 7, 8], "By": [0, 1, 2, 3, 4, 6, 7, 8], "upfront": [0, 2], "equip": [0, 2, 4, 7], "avoid": [0, 3, 4, 7, 8], "current": [0, 2, 3, 4, 6, 7, 8], "discours": [0, 2], "around": [0, 2, 3, 4, 6, 7, 8], "tend": [0, 2, 4, 7], "toward": [0, 3, 4, 7, 8], "extrem": [0, 3, 4, 7], "either": [0, 3, 4, 6, 7], "uncrit": 0, "enthusiasm": 0, "wholesal": [0, 4], "dismiss": 0, "differ": [0, 3, 4, 5, 6, 7, 8], "rather": [0, 1, 3, 4, 5, 7], "than": [0, 1, 3, 4, 5, 7], "theoret": [0, 3], "examin": [0, 3, 4, 6, 7, 8], "first": [0, 1, 3, 4, 5, 6, 7, 8], "everi": [0, 4, 7], "concept": [0, 3, 4, 7], "illustr": [0, 3, 4, 6, 7, 8], "execut": [0, 4, 5, 7], "immedi": [0, 3, 4, 5], "analysi": [0, 1, 3, 4, 5, 6, 7], "balanc": [0, 3, 4, 5, 6, 7, 8], "help": [0, 3, 4, 5, 6, 7, 8], "intend": [0, 4, 7], "develop": [0, 1, 3, 4, 5, 6, 7, 8], "step": [0, 1, 3, 4, 5, 7, 8], "insight": [0, 3, 4, 6, 7, 8], "along": [0, 3, 4, 5, 7], "guidanc": [0, 3, 8], "could": [0, 1, 3, 4, 5, 6, 7, 8], "derail": 0, "project": [0, 3, 4, 5, 7], "earli": [0, 3, 4, 7, 8], "befor": [0, 3, 4, 5, 7, 8], "thei": [0, 1, 3, 4, 5, 6, 7, 8], "costli": [0, 4, 7], "problem": [0, 1, 2, 3, 5, 7], "too": [0, 1, 3, 4, 6, 7], "late": [0, 3, 7], "lifecycl": [0, 7], "lead": [0, 1, 3, 4, 6, 7, 8], "genai": [0, 1, 3, 7], "initi": [0, 1, 3, 4, 5, 6, 7, 8], "leader": [0, 4], "advoc": [0, 7], "anyon": [0, 7], "seek": [0, 4, 5, 7], "work": [0, 1, 3, 4, 5, 6, 7, 8], "typic": [0, 3, 4, 6, 7, 8], "job": [0, 4, 5, 7], "role": [0, 3, 4, 5, 6, 7, 8], "platform": [0, 4, 5, 6, 7, 8], "backend": [0, 3, 4], "exist": [0, 3, 4, 5], "ml": [0, 7], "transit": [0, 4, 5, 6, 8], "overse": 0, "motiv": [0, 3, 4, 8], "need": [0, 3, 4, 5, 6, 7], "readi": [0, 4, 7], "desir": [0, 3, 4, 8], "perform": [0, 3, 4, 5, 6, 7, 8], "after": [0, 1, 3, 4, 5, 6, 7, 8], "read": [0, 3, 4, 6, 7, 8], "implic": [0, 1, 3, 4, 7], "recommend": [0, 3, 4, 5, 6, 7, 8], "abl": [0, 3, 4, 6, 8], "deploi": [0, 3, 4, 5, 6, 7], "proper": [0, 3, 5, 7, 8], "realist": [0, 3, 7], "effort": [0, 4, 7, 8], "estim": [0, 4, 7], "impact": [0, 3, 4, 5, 6, 7, 8], "timelin": 0, "To": [0, 3, 4, 5, 6, 7, 8], "should": [0, 3, 4, 6, 7, 8], "basic": [0, 3, 4, 5, 6, 7], "program": [0, 4, 5], "knowledg": [0, 3, 4, 5, 7], "introductori": [0, 1, 2], "langchain": [0, 4, 6], "e": [0, 1, 3, 4, 5, 6, 7, 8], "g": [0, 3, 4, 5, 6, 7, 8], "chat": [0, 3, 4, 5, 6, 7, 8], "prompt": [0, 4, 5, 7], "templat": [0, 4, 5], "openai": [0, 3, 4, 5, 8], "anthrop": [0, 3, 8], "similar": [0, 3, 4, 5, 8], "dive": 0, "here": [0, 2, 3, 4, 5, 6, 7, 8], "get": [0, 3, 4, 5, 6, 7, 8], "start": [0, 3, 4, 5, 7, 8], "clone": [0, 3], "companion": 0, "git": [0, 5], "cd": 0, "activ": [0, 3, 4, 7], "virtual": [0, 4], "m": [0, 3, 4, 5, 7, 8], "venv": 0, "tame": [0, 3, 4, 5, 6, 7, 8], "env": [0, 3, 4, 6, 7, 8], "bin": [0, 5], "On": [0, 4, 5, 8], "window": [0, 2, 4, 5], "script": [0, 5], "try": [0, 1, 3, 4, 7, 8], "contain": [0, 3, 4, 5, 6, 7, 8], "possibl": [0, 3, 4, 5, 7, 8], "includ": [0, 1, 3, 4, 5, 6, 7, 8], "necessari": [0, 3, 4, 6, 7], "instal": [0, 3, 4, 5, 8], "go": [0, 3, 4, 6, 8], "feel": [0, 5], "prefer": [0, 4, 5, 7, 8], "packag": [0, 4, 5, 8], "pip": [0, 3, 4, 5, 8], "poetri": [0, 7], "file": [0, 3, 4, 5, 6, 7, 8], "root": [0, 3], "directori": [0, 4, 5], "add": [0, 3, 4, 5, 6, 7], "other": [0, 3, 4, 5, 6, 7, 8], "sensit": [0, 3, 4, 5, 7], "openai_api_kei": [0, 3], "your_openai_api_key_her": 0, "never": [0, 8], "commit": [0, 3, 4, 7], "version": [0, 3, 4, 5, 7, 8], "control": [0, 1, 3, 4, 5, 7, 8], "kept": [0, 4], "privat": [0, 4], "If": [0, 1, 3, 4, 5, 7, 8], "encount": [0, 2, 4, 7], "rate": [0, 3, 4, 7], "consid": [0, 3, 4, 6, 7, 8], "smaller": [0, 3, 4, 5, 6, 8], "retri": [0, 8], "logic": [0, 1, 3, 4, 6, 7], "conflict": [0, 3, 4], "fresh": 0, "like": [0, 1, 3, 4, 5, 6, 7, 8], "check": [0, 4, 5, 7, 8], "page": [0, 4], "known": [0, 4, 5, 7, 8], "now": [0, 1, 3, 4, 5, 6, 7, 8], "let": [0, 3, 4, 5, 6, 7, 8], "begin": [0, 4, 7, 8], "explor": [0, 1, 3, 4, 5, 7, 8], "dr": [0, 3], "tharsi": [0, 2, 3, 4, 5, 6, 7, 8], "souza": [0, 2, 3, 4, 5, 6, 7, 8], "scientist": [0, 1, 5, 7], "special": [0, 4, 5, 7, 8], "he": [0, 3, 4, 7], "lectur": 0, "columbia": 0, "univers": [0, 4, 5, 7], "master": [0, 5, 8], "scienc": [0, 3, 4, 7], "appli": [0, 3, 4, 6, 7, 8], "analyt": 0, "incom": [0, 4], "head": [0, 3, 4, 6, 7], "equiti": [0, 4], "citadel": 0, "former": [0, 1, 4, 5], "senior": [0, 4], "vp": 0, "two": [0, 3, 4, 5, 6, 7, 8], "sigma": [0, 3], "invest": [0, 3, 4, 7, 8], "also": [0, 3, 4, 5, 6, 7, 8], "enjoi": 0, "mentor": 0, "under": [0, 3, 4, 7, 8], "repres": [0, 3, 4, 8], "student": [0, 3, 5, 7], "profession": [0, 3, 4, 7, 8], "divers": [0, 3, 4, 6, 7, 8], "global": [0, 4, 7], "ecosystem": [0, 4], "With": [0, 3, 4, 5, 7], "over": [0, 2, 3, 4, 5, 6, 7, 8], "15": [0, 4, 7, 8], "deliv": [0, 4, 5], "across": [0, 1, 3, 4, 5, 7, 8], "startup": 0, "fortun": 0, "500": [0, 3, 4, 7], "compani": [0, 3, 4, 6, 7, 8], "numer": [0, 4, 7], "scholarli": 0, "frequent": [0, 4, 5, 8], "speaker": [0, 4], "academ": [0, 3, 4, 7], "busi": [0, 4, 5, 7], "confer": [0, 8], "ground": [0, 3, 4], "background": [0, 1, 4, 5, 6], "draw": [0, 3, 4, 7, 8], "scale": [0, 3, 4, 5, 7, 8], "stage": [0, 3, 7, 8], "major": [0, 3, 4, 7, 8], "institut": [0, 4, 5, 7], "well": [0, 3, 4, 5, 7, 8], "advis": [0, 3], "profit": [0, 4, 6, 7, 8], "organ": [0, 3, 4, 5, 6], "uniqu": [0, 3, 4, 7], "bridg": [0, 5, 7], "gap": [0, 1, 3, 7], "between": [0, 1, 3, 4, 5, 6, 7, 8], "potenti": [0, 1, 3, 4, 5, 6, 7, 8], "next": [0, 1, 3, 4, 5, 7, 8], "hold": [0, 3, 4], "ph": [0, 7], "d": [0, 3, 4, 5, 7, 8], "ucl": 0, "london": 0, "phil": [0, 7], "sc": 0, "b": [0, 4, 5, 7, 8], "tell": [1, 3, 7], "mere": [1, 4], "what": [1, 3, 4, 5, 7, 8], "someth": [1, 4], "i": [1, 2, 4, 5, 6, 7, 8], "emanuel": [1, 3, 4, 7], "derman": 1, "an": [1, 2, 3, 4, 5, 6, 7, 8], "altern": [1, 3, 4, 5, 6, 7], "titl": [1, 2, 3, 4, 5, 6, 7, 8], "thi": [1, 2, 3, 4, 5, 6, 7, 8], "book": [1, 2, 4], "been": [1, 3, 4, 5, 7], "behav": 1, "badli": 1, "come": [1, 3, 4, 5, 6, 7, 8], "notic": [1, 3, 4, 7, 8], "parallel": [1, 3, 4, 5], "": [1, 3, 4, 5, 6, 7, 8], "semin": [1, 7], "2011": 1, "coincident": 1, "just": [1, 3, 4, 5, 6, 7, 8], "caution": 1, "against": [1, 3, 4, 7], "treat": [1, 4, 7], "perfect": [1, 4], "represent": [1, 4, 6, 7], "realiti": [1, 7], "aim": [1, 3, 4, 5, 6, 7, 8], "highlight": [1, 3, 4, 5, 6, 7, 8], "practic": [1, 3, 4, 5, 6, 7], "cours": [1, 4, 7], "bare": [1, 5], "fact": [1, 3, 4, 7], "actual": [1, 3, 4, 6, 7, 8], "physicist": 1, "legendari": 1, "author": [1, 2, 3, 4, 5, 6, 7, 8], "professor": 1, "quant": 1, "goldman": 1, "sach": 1, "scientif": [1, 3, 4], "fail": [1, 3, 4, 7], "we": [1, 3, 4, 5, 6, 7, 8], "mistak": [1, 7], "approxim": [1, 4, 8], "full": [1, 3, 4, 5, 7, 8], "assumpt": [1, 4, 7], "core": [1, 4, 5, 7], "premis": [1, 5], "hi": [1, 4, 7, 8], "aspect": [1, 3, 4, 6, 7, 8], "world": [1, 3, 4, 7, 8], "inher": [1, 2, 3, 4, 7, 8], "involv": [1, 3, 4, 5, 7, 8], "simplif": 1, "argu": [1, 7, 8], "crise": 1, "2008": 1, "crash": 1, "occur": [1, 3, 4, 7], "partli": 1, "becaus": [1, 3, 4, 7], "peopl": [1, 3, 4, 5, 7], "put": [1, 4], "much": [1, 3, 4], "faith": 1, "mathemat": [1, 4], "recogn": [1, 3, 4, 7], "human": [1, 4, 6, 7, 8], "behavior": [1, 3, 4, 7], "market": [1, 4, 6, 8], "dynam": [1, 3, 4, 7], "reason": [1, 3, 4, 5, 6, 7, 8], "Their": [1, 4, 8], "respons": [1, 4, 5, 6, 7, 8], "often": [1, 3, 4, 5, 6, 7, 8], "convinc": [1, 3], "probabilist": [1, 4], "train": [1, 4, 5, 7, 8], "data": [1, 4, 5, 6, 8], "true": [1, 3, 4, 6, 7, 8], "even": [1, 3, 4, 5, 6, 7, 8], "though": [1, 3, 4, 5, 7, 8], "insist": 1, "machin": [1, 3, 5, 7, 8], "todai": [1, 5, 8], "grow": [1, 3, 4, 5, 7, 8], "pervas": [1, 7], "belief": [1, 5, 7], "solv": [1, 3, 4, 5, 7, 8], "ani": [1, 3, 4, 5, 6, 7, 8], "context": [1, 2, 3, 4, 5, 6, 7, 8], "content": 1, "wish": [1, 4], "user": [1, 4, 5, 6], "moreov": 1, "were": [1, 3, 4, 7, 8], "predict": [1, 3, 4, 7, 8], "chatbot": [1, 3, 4, 5, 7], "twist": [1, 7], "wrap": [1, 8], "further": [1, 3, 4, 6, 7, 8], "daili": [1, 7], "life": [1, 4, 5, 7], "workflow": [1, 4, 5, 7, 8], "affect": [1, 4, 7], "decid": [1, 3, 4, 6], "action": [1, 3, 4, 6, 7], "coupl": 1, "lack": [1, 3, 4, 7, 8], "pose": [1, 3, 4, 6, 7, 8], "still": [1, 4, 7], "figur": [1, 4, 8], "out": [1, 3, 4, 5, 6, 7, 8], "serv": [1, 3, 4, 6, 7, 8], "builder": [1, 5], "who": [1, 3, 4, 5, 6, 7, 8], "remain": [1, 3, 4, 5, 6, 7], "clear": [1, 3, 4, 7, 8], "ei": 1, "about": [1, 3, 4, 5, 6, 7, 8], "therefor": [1, 3, 4, 5, 7], "end": [1, 3, 4, 5, 6, 7, 8], "detail": [1, 3, 4, 5, 6, 7, 8], "python": [1, 2, 4, 5, 6, 7, 8], "code": [1, 2, 3, 4, 7, 8], "diminish": [1, 3, 4], "promot": [1, 3, 4, 7], "nuanc": [1, 3, 4, 6, 7, 8], "acknowledg": [1, 4, 7], "within": [1, 3, 4, 6, 7, 8], "trustworthi": [1, 7], "taught": 1, "u": [1, 3, 4, 7, 8], "where": [1, 3, 4, 5, 6, 7, 8], "der11": 1, "why": [1, 3, 4, 7, 8], "confus": [1, 7], "illus": 1, "disast": [1, 4], "wall": [1, 5], "street": [1, 5], "press": [1, 4, 5], "isbn": [1, 3, 4], "9781439165010": 1, "url": [1, 2, 3, 4, 5, 6, 7, 8], "googl": [1, 4, 8], "co": [1, 3, 4, 5, 7], "uk": [1, 7], "id": [1, 4, 5, 7], "lke_cwm4wm8c": 1, "sign": [2, 4, 7], "up": [2, 3, 4, 5, 6, 7, 8], "receiv": [2, 3, 4, 6, 7, 8], "updat": [2, 3, 4, 5, 6, 7, 8], "abstract": [2, 4, 7, 8], "heavili": [2, 3, 4, 7, 8], "gloss": 2, "fundament": [2, 3, 4, 7, 8], "convers": [2, 3, 4, 5, 6, 7, 8], "kei": [2, 3, 5, 7, 8], "proven": 2, "yet": [2, 3, 4, 6, 7], "concret": [2, 7], "unstructur": [2, 8], "sidestep": 2, "misc": [2, 3, 4, 5, 6, 7, 8], "tharsistpsouza2024tamingllm": [2, 3, 4, 5, 6, 7, 8], "t": [2, 3, 4, 5, 6, 7, 8], "p": [2, 3, 4, 5, 6, 7, 8], "2024": [2, 3, 4, 5, 6, 7, 8], "journal": [2, 3, 4, 5, 6, 7, 8], "repositori": [2, 3, 4, 5, 6, 7, 8], "valu": [3, 4, 5, 6, 7, 8], "its": [3, 4, 5, 6, 7, 8], "privileg": 3, "abov": [3, 4, 7], "soon": [3, 8], "lose": [3, 4], "dwight": 3, "eisenhow": 3, "releas": [3, 4, 5, 7, 8], "3": [3, 4, 5, 8], "5": [3, 4, 5, 6, 8], "2022": [3, 4, 7], "mark": [3, 4, 7], "pivot": [3, 4], "moment": 3, "histori": [3, 4, 5], "artifici": [3, 4, 7], "intellig": [3, 4, 7], "five": [3, 4, 7], "dai": [3, 4, 7, 8], "launch": [3, 4, 7], "attract": [3, 4], "million": [3, 4], "month": [3, 4, 7], "becam": 3, "fastest": [3, 4, 7], "100": [3, 4, 7, 8], "monthli": [3, 4], "rais": [3, 4, 6, 7], "intrigu": 3, "question": [3, 4, 5, 7, 8], "did": [3, 4, 8], "dramat": [3, 4, 5, 8], "predecessor": 3, "gpt": [3, 4, 5, 6, 7, 8], "had": [3, 4], "same": [3, 4, 5, 6, 7, 8], "number": [3, 4, 5, 6, 7, 8], "paramet": [3, 4, 5, 7, 8], "far": [3, 6, 7], "less": [3, 4, 5, 7], "attent": [3, 5], "arguabl": 3, "feedback": [3, 4, 7, 8], "abil": [3, 4, 5, 7, 8], "least": [3, 4, 7], "ey": 3, "breakthrough": [3, 7], "demonstr": [3, 4, 5, 6, 7, 8], "crucial": [3, 5, 7, 8], "greater": [3, 4, 7], "process": [3, 4, 5, 6, 7, 8], "modern": [3, 4, 6, 8], "techniqu": [3, 4, 5, 6], "direct": [3, 4, 5, 7], "rafailov": 3, "et": [3, 4, 7, 8], "al": [3, 4, 7, 8], "present": [3, 4, 5, 6, 7, 8], "autom": [3, 4, 7, 8], "fashion": [3, 8], "open": [3, 4, 6, 7, 8], "sourc": [3, 4, 5, 6, 7, 8], "common": [3, 4, 5, 6, 8], "pre": [3, 4, 5, 7], "default": [3, 4, 5, 7, 8], "state": [3, 4, 5, 6, 7, 8], "art": [3, 4, 7], "object": [3, 4, 7, 8], "given": [3, 4, 5, 6, 7, 8], "webpag": 3, "internet": [3, 4], "veri": [3, 4, 5, 7], "ask": [3, 4, 5, 7, 8], "instruct": [3, 4, 6, 7, 8], "sai": [3, 8], "ouyang": [3, 7], "2": [3, 4, 5, 8], "explain": 3, "moon": 3, "land": [3, 4], "6": [3, 4, 6, 8], "old": [3, 4], "import": [3, 4, 5, 6, 7, 8], "pipelin": [3, 4, 5, 7, 8], "pipe": [3, 7], "text": [3, 4, 5, 6, 7, 8], "gpt2": [3, 4], "msg": 3, "short": [3, 4, 6, 7, 8], "sentenc": [3, 4, 6, 7, 8], "_": [3, 4, 7, 8], "rang": [3, 4, 5, 6, 7, 8], "len": [3, 4, 6, 7], "print": [3, 4, 5, 6, 7, 8], "f": [3, 4, 6, 7, 8], "n": [3, 4, 5, 6, 7, 8], "1": [3, 4, 5, 8], "0": [3, 4, 5, 6, 7, 8], "generated_text": 3, "good": [3, 4, 5, 8], "idea": [3, 7], "one": [3, 4, 5, 6, 7, 8], "those": [3, 4, 6, 7, 8], "littl": [3, 4], "green": [3, 7], "dot": 3, "Then": [3, 4], "line": [3, 4, 5, 7], "later": [3, 4, 5, 7, 8], "re": [3, 4, 6, 7, 8], "alreadi": [3, 4], "movi": 3, "theori": [3, 4], "some": [3, 4, 5, 6, 7, 8], "mean": [3, 4, 5, 6, 7, 8], "word": [3, 4, 6, 8], "tepid": 3, "articl": [3, 4, 6, 7], "sure": [3, 4, 6, 7, 8], "lunar": 3, "As": [3, 4, 5, 6, 7, 8], "see": [3, 4, 5, 7, 8], "coher": [3, 4, 6], "explan": [3, 4, 7, 8], "child": [3, 4, 7], "nonsens": [3, 7], "meander": 3, "unrel": [3, 4, 7], "topic": [3, 4, 5, 6, 7, 8], "simpl": [3, 4, 5, 6, 7, 8], "appropri": [3, 4, 5, 6, 7, 8], "young": [3, 4, 7], "instead": [3, 4, 5, 6, 7, 8], "address": [3, 4, 6, 7, 8], "issu": [3, 4, 5, 6, 7, 8], "introduc": [3, 4, 5, 6, 7, 8], "rlhf": [3, 7], "intent": [3, 7], "wide": [3, 4, 5, 6, 7, 8], "task": [3, 5, 6, 7, 8], "fig": [3, 4, 5, 6, 7, 8], "7": [3, 4, 6, 7], "collect": [3, 4, 5, 6, 7], "sampl": [3, 5, 6, 8], "label": [3, 4, 7, 8], "comparison": 3, "reward": [3, 4, 7], "sever": [3, 4, 5, 6, 7, 8], "rank": [3, 4, 7], "best": [3, 4, 5, 7], "worst": 3, "rm": 3, "reinforc": [3, 4, 7], "write": [3, 4, 7, 8], "stori": [3, 7], "frog": 3, "calcul": [3, 4, 5, 7], "score": [3, 4, 7, 8], "ppo": 3, "proxim": 3, "iter": [3, 4, 6, 7, 8], "accur": [3, 4, 7, 8], "undesir": [3, 7], "simplifi": [3, 4, 5, 8], "view": [3, 4, 7], "show": [3, 4, 5, 6, 7, 8], "progress": [3, 6, 7], "pattern": [3, 4, 7, 8], "ha": [3, 4, 5, 7, 8], "instanc": [3, 4, 6, 7], "directli": [3, 4, 5, 7, 8], "For": [3, 4, 5, 6, 7, 8], "llama": [3, 4, 7, 8], "guard": 3, "team": [3, 4, 5, 8], "8b": [3, 7], "wa": [3, 4, 5, 7, 8], "classif": [3, 4, 7, 8], "bypass": [3, 7], "similarli": [3, 4, 5, 7], "zephyr": 3, "7b": [3, 4, 7], "alpha": [3, 4, 8], "mistral": [3, 5, 8], "publicli": [3, 4, 8], "assist": [3, 4, 5, 7, 8], "paper": [3, 4, 7, 8], "compon": [3, 4], "particular": [3, 4, 7, 8], "foundat": [3, 4, 6, 7], "advanc": [3, 4, 5, 6, 7, 8], "method": [3, 4, 6, 7, 8], "strong": [3, 4, 5, 7, 8], "At": [3, 4, 5, 8], "high": [3, 4, 5, 6, 7, 8], "level": [3, 4, 5, 6, 7, 8], "carefulli": [3, 4, 7, 8], "curat": [3, 4], "purpos": [3, 4, 5, 7, 8], "exhibit": [3, 4, 7], "domain": [3, 4, 5, 7], "emploi": [3, 4, 7, 8], "prove": [3, 4, 7], "particularli": [3, 4, 5, 6, 7, 8], "valuabl": [3, 4, 5, 8], "scenario": [3, 4, 5, 7, 8], "precis": [3, 4, 5, 7, 8], "style": [3, 4], "tone": 3, "expertis": [3, 4, 7], "medic": [3, 4, 5], "legal": [3, 4, 7], "field": [3, 4, 7, 8], "adher": [3, 4, 6, 7, 8], "guidelin": [3, 4, 7], "servic": [3, 4, 5, 6, 7, 8], "standard": [3, 4, 5, 7], "approach": [3, 4, 5, 6, 8], "each": [3, 4, 5, 6, 7, 8], "distinct": [3, 4, 5, 7], "advantag": [3, 4, 5, 6, 7, 8], "weight": [3, 4, 5, 7], "maximum": [3, 4, 5, 6, 7], "lora": [3, 7], "low": [3, 4, 5, 7, 8], "hu": [3, 7], "2021": [3, 4], "small": [3, 4, 8], "matric": 3, "effici": [3, 4, 5, 6, 7, 8], "qlora": 3, "quantiz": [3, 5], "dettmer": 3, "2023": [3, 4, 5, 7, 8], "combin": [3, 4, 5, 6, 7, 8], "memori": [3, 4, 5, 6, 7], "footprint": 3, "modest": 3, "increas": [3, 4, 5, 6, 7, 8], "likelihood": [3, 4, 7], "obtain": [3, 4, 5, 7, 8], "probabl": [3, 4, 5, 8], "outcom": [3, 4, 7, 8], "hong": [3, 4], "unintend": [3, 7], "suboptim": 3, "seen": [3, 4, 7], "research": [3, 4, 5, 6], "maxim": [3, 4], "shown": [3, 4, 5, 7], "alon": [3, 4, 7], "gain": [3, 4, 7], "achiev": [3, 4, 5, 7, 8], "bai": [3, 4, 7], "touvron": 3, "sinc": [3, 4, 6, 7, 8], "main": [3, 4, 5, 6, 7, 8], "categori": [3, 4, 5, 7], "algorithm": [3, 4, 7], "meanwhil": 3, "superior": [3, 4, 7], "benchmark": [3, 5], "xu": [3, 4, 7], "schulman": [3, 7], "2017": [3, 4], "popular": [3, 5, 8], "understood": 3, "set": [3, 4, 5, 6, 7, 8], "rule": [3, 4, 6, 8], "govern": [3, 4], "reflect": [3, 4, 7], "anoth": [3, 4, 5, 7], "adjust": [3, 4, 5, 6, 7, 8], "One": [3, 4, 5, 7], "strength": [3, 4, 7], "2024c": 3, "real": [3, 4, 5, 6, 7, 8], "noisi": 3, "delai": [3, 4, 5, 7], "subsequ": [3, 8], "situat": [3, 4, 6, 7], "clip": 3, "surrog": 3, "function": [3, 4, 5, 6, 7, 8], "stabl": [3, 4, 5], "prevent": [3, 4, 7, 8], "overreact": 3, "converg": 3, "due": [3, 4, 5, 6, 7], "simplic": [3, 5], "award": [3, 4], "runner": 3, "neurip": 3, "blog": [3, 4, 5, 7, 8], "4": [3, 4, 5, 8], "fit": [3, 4, 6, 7, 8], "pair": [3, 4, 7], "rl": [3, 7], "find": [3, 4, 5, 6, 7, 8], "contrast": [3, 4, 7], "satisfi": [3, 4], "implicit": [3, 4, 7], "whose": [3, 4], "correspond": [3, 4, 8], "extract": [3, 4, 5, 6, 7, 8], "close": [3, 4, 5, 7], "compar": [3, 4, 5, 6, 7], "assign": [3, 4, 7, 8], "higher": [3, 4], "kl": 3, "diverg": 3, "origin": [3, 4, 5, 6, 7, 8], "preserv": [3, 5, 6, 7], "defin": [3, 4, 5, 6, 7, 8], "equat": 3, "mathcal": 3, "l": [3, 4], "pi_": 3, "theta": [3, 8], "ref": 3, "mathbb": [3, 8], "x": [3, 4, 5, 7], "y_w": 3, "y_l": 3, "sim": [3, 8], "left": 3, "log": [3, 4], "beta": [3, 4, 7, 8], "underbrac": 3, "frac": [3, 7], "color": [3, 4], "red": 3, "right": [3, 4, 5, 7], "respect": [3, 4, 5, 7], "deviat": [3, 4, 7, 8], "straightforward": [3, 4, 6, 7, 8], "librari": [3, 4, 5, 6, 7, 8], "huggingfac": [3, 4, 5, 7], "trl": [3, 7], "2024d": 3, "suit": [3, 4, 7], "friendli": [3, 4, 5, 6], "interfac": [3, 4, 5, 7], "featur": [3, 4, 5, 7, 8], "challeng": [3, 4, 5, 6, 7, 8], "distinguish": [3, 4, 7], "scalabl": [3, 4, 6, 7], "doe": [3, 4, 6, 7, 8], "pretrain": [3, 4], "hou": [3, 4], "poor": [3, 4, 7], "return": [3, 4, 5, 6, 7, 8], "addit": [3, 4, 5, 6, 7, 8], "benefit": [3, 4, 5, 7, 8], "fix": [3, 4, 6, 7], "invers": 3, "trend": [3, 4, 7, 8], "util": [3, 4, 5, 6, 7], "rapid": [3, 4, 5, 7], "yield": [3, 4], "onli": [3, 4, 6, 7, 8], "margin": [3, 4, 7, 8], "capit": [3, 4, 6, 8], "inaccuraci": [3, 4], "nois": 3, "dure": [3, 4, 7, 8], "accuraci": [3, 4, 5, 7, 8], "lag": [3, 4, 7], "significantli": [3, 4, 5, 6, 7], "indic": [3, 4, 7, 8], "signal": [3, 7], "plateau": 3, "sophist": [3, 4, 6, 7], "previou": [3, 4, 5, 6, 8], "deriv": [3, 4], "pairwis": [3, 4], "feng": [3, 7], "substanti": [3, 4, 5, 7], "wors": [3, 8], "influenc": [3, 4, 7, 8], "success": [3, 4, 7, 8], "imbal": 3, "stronger": 3, "bad": 3, "ones": [3, 7], "loss": [3, 4, 7], "gradient": [3, 4, 7], "dispref": 3, "unbalanc": 3, "trajectori": 3, "stuck": 3, "saddl": 3, "point": [3, 4, 6, 7], "forward": [3, 4, 7], "futur": [3, 4, 7], "phenomenon": [3, 7, 8], "degrad": [3, 4, 7, 8], "danger": [3, 5, 7], "loop": [3, 4, 5, 7], "recurs": [3, 6], "kazdan": 3, "qualiti": [3, 4, 5, 6, 7, 8], "pollut": 3, "replac": [3, 4, 5], "amplif": 3, "reduct": [3, 4], "express": [3, 4, 7, 8], "catastroph": [3, 7], "forget": [3, 8], "previous": [3, 4, 6, 7, 8], "mitig": [3, 4, 6, 7, 8], "mix": [3, 4, 7, 8], "metric": [3, 5, 7], "sz\u00e9p": 3, "regular": [3, 4, 7, 8], "relev": [3, 4, 7], "scarc": 3, "behaviour": 3, "strateg": [3, 4, 7, 8], "compli": [3, 4, 5, 7, 8], "modif": [3, 4, 5, 7], "outsid": [3, 4], "evidenc": 3, "landmark": 3, "askel": [3, 4, 7], "2024a": [3, 5], "dec": 3, "explicitli": [3, 4], "so": [3, 4, 7, 8], "might": [3, 4, 5, 6, 7, 8], "pretend": 3, "adopt": [3, 4, 5, 7, 8], "onc": [3, 4, 6, 7], "complet": [3, 4, 5, 6, 7, 8], "describ": [3, 4, 7], "harmless": [3, 7], "told": 3, "retrain": 3, "queri": [3, 4], "tier": [3, 4, 7], "paid": [3, 4], "column": [3, 4, 7], "condit": [3, 4], "toxic": [3, 7], "excerpt": [3, 4], "scratchpad": 3, "refus": [3, 7, 8], "happen": [3, 7], "bomb": [3, 7], "engag": [3, 4, 7, 8], "intern": [3, 4, 7], "unmonitor": 3, "longer": [3, 4], "believ": [3, 4, 5, 7, 8], "act": [3, 4, 5, 7], "therebi": [3, 4, 6], "reveal": [3, 4, 7], "complianc": [3, 4, 7, 8], "phase": [3, 4], "natur": [3, 4, 5, 6, 7, 8], "evid": [3, 4, 7, 8], "seemingli": 3, "surpris": 3, "appear": [3, 4, 7, 8], "criteria": [3, 4, 7], "underli": [3, 4, 7, 8], "anim": [3, 7], "welfar": 3, "instil": 3, "implicitli": 3, "consequ": [3, 4, 7, 8], "explicit": [3, 4, 7, 8], "chain": [3, 4, 6], "thought": [3, 4, 8], "opaqu": 3, "aris": [3, 4, 7], "opu": 3, "sonnet": [3, 4, 6], "wherea": [3, 4], "haiku": [3, 7], "persist": 3, "resist": [3, 4], "embed": [3, 4, 5, 6], "doesn": [3, 4, 5, 6, 8], "anti": [3, 4], "lab": 3, "exfiltr": [3, 7], "protect": [3, 4, 7], "Not": [3, 4, 7], "malici": [3, 4, 7], "support": [3, 4, 5, 7, 8], "concern": [3, 4, 7], "mechan": [3, 4, 7, 8], "insuffici": [3, 4], "don": [3, 4, 6, 8], "concerningli": 3, "call": [3, 4, 5, 6, 7, 8], "detect": [3, 4, 7, 8], "decept": [3, 4, 7], "warrant": [3, 7], "deeper": [3, 4, 5, 6], "scrutini": [3, 4, 7], "reli": [3, 4, 7], "cross": [3, 4, 5, 7], "circular": 3, "bia": [3, 4, 8], "truli": [3, 4], "trust": [3, 4, 7, 8], "referenti": 3, "ly": 3, "hood": [3, 8], "deep": [3, 4, 7, 8], "mechanist": 3, "drive": [3, 6, 7, 8], "correl": [3, 4], "miss": [3, 4, 7], "confound": 3, "factor": [3, 4, 6, 8], "establish": [3, 4, 7], "attempt": [3, 4, 6, 7], "causal": [3, 4], "heavi": 3, "relianc": [3, 4, 7], "oversimplifi": 3, "frame": 3, "subtler": 3, "narr": [3, 4], "henc": [3, 4, 7], "agenc": [3, 4, 7], "onto": 3, "anthropomorph": 3, "obscur": 3, "blind": [3, 4], "failur": [3, 4, 7], "mode": [3, 5, 7], "map": [3, 4, 5, 8], "cleanli": 3, "analogi": 3, "excel": [3, 4, 5, 7, 8], "review": [3, 4, 7, 8], "prof": 3, "jacob": [3, 4, 7], "andrea": [3, 4, 7], "yoshua": [3, 7], "bengio": [3, 7], "jasjeet": 3, "sekhon": 3, "rohin": 3, "shah": 3, "2024b": [3, 5], "assum": [3, 4, 6, 7], "acm": [3, 7], "inc": [3, 4, 6, 8], "dedic": [3, 4, 7, 8], "democrat": [3, 4, 8], "educ": [3, 4, 5, 6, 7], "k": [3, 4, 6, 7, 8], "12": [3, 4, 6, 7], "name": [3, 4, 6, 7, 8], "smolk": 3, "ll": [3, 4], "walk": 3, "measur": [3, 4, 7], "huggingfacetb": 3, "360m": [3, 4], "compact": [3, 4, 7], "part": [3, 4, 6, 7, 8], "famili": [3, 5, 7, 8], "publish": [3, 7, 8], "api": [3, 4, 5], "local": [3, 4, 6, 7, 8], "infer": [3, 4, 5, 7], "remot": [3, 4], "load": [3, 4, 5, 6, 7, 8], "store": [3, 4, 6, 7], "eventu": [3, 4], "your_openai_api_kei": 3, "reusabl": 3, "anchor": [3, 7], "worth": [3, 4, 5], "choic": [3, 4, 5, 7, 8], "lightweight": [3, 4, 5, 8], "suitabl": [3, 4, 5, 7], "devic": [3, 4, 5, 8], "Its": [3, 4, 5], "candid": [3, 4], "said": [3, 4, 7], "necessarili": [3, 4, 7], "par": [3, 4], "mind": [3, 4, 5, 7], "factual": [3, 4, 7], "inconsist": [3, 4, 7, 8], "guardrail": [3, 7], "articul": 3, "uphold": [3, 7], "employe": [3, 4], "stakehold": [3, 4, 7], "expect": [3, 4, 5, 6, 7, 8], "regard": [3, 4, 7], "ethic": [3, 4, 5, 7], "conduct": [3, 4], "social": [3, 4, 7], "mission": [3, 7], "vision": [3, 4, 5, 7], "cultur": [3, 4, 5, 7], "account": [3, 4, 7], "codifi": 3, "mlcommon": 3, "vidgen": [3, 7], "encompass": [3, 7], "seven": 3, "hazard": [3, 4, 7], "violent": [3, 7], "crime": [3, 7], "sex": [3, 7], "relat": [3, 4, 7], "sexual": [3, 7], "exploit": [3, 4, 7], "indiscrimin": [3, 7], "weapon": [3, 7], "chemic": 3, "biolog": 3, "radiolog": 3, "nuclear": [3, 4], "explos": [3, 7], "cbrne": 3, "suicid": [3, 7], "hate": [3, 7], "speech": [3, 7], "below": [3, 4, 5, 6, 7, 8], "markdown": [3, 4, 5, 6, 7], "written": [3, 4], "english": [3, 6], "o": [3, 4, 6, 7, 8], "ipython": [3, 4, 7], "displai": [3, 4, 7, 8], "def": [3, 4, 6, 7, 8], "load_polici": 3, "policy_path": 3, "path": [3, 4, 6, 7], "join": [3, 4, 6, 7], "genai_polici": 3, "md": [3, 4, 5, 7, 8], "r": [3, 4, 6, 7, 8], "policy_cont": 3, "classroom": [3, 7], "accept": [3, 4, 5, 7], "unaccept": [3, 5], "ag": [3, 4, 7], "subject": [3, 4, 5], "posit": [3, 4, 5, 6, 7, 8], "confid": [3, 4, 8], "inclus": [3, 4, 6, 7, 8], "celebr": 3, "definit": [3, 4, 8], "creativ": [3, 4, 8], "math": [3, 4], "tip": [3, 7], "digit": [3, 4], "literaci": 3, "onlin": [3, 4, 5, 7], "histor": [3, 4], "violenc": [3, 7], "physic": [3, 4, 7], "fight": [3, 7], "crimin": [3, 7], "illeg": [3, 7], "glorifi": [3, 7], "person": [3, 4, 5, 7, 8], "eat": [3, 7], "disord": 3, "diet": 3, "dare": 3, "advic": [3, 4, 7], "discriminatori": [3, 7], "bulli": [3, 7], "harass": [3, 4, 7], "target": [3, 4, 5, 7, 8], "group": [3, 4, 5, 6, 7], "religi": [3, 5, 7], "racial": [3, 4, 7], "ethnic": [3, 7], "gender": [3, 4, 7], "discrimin": [3, 4, 7], "adult": [3, 7], "profan": [3, 7], "relationship": [3, 4], "substanc": [3, 4], "drug": [3, 7], "gambl": 3, "bet": 3, "protocol": [3, 4, 7], "redirect": 3, "alert": 3, "record": [3, 4, 5, 7], "audit": [3, 4], "teacher": [3, 7], "parent": [3, 7], "continu": [3, 4, 6, 7, 8], "construct": [3, 4, 7, 8], "compliant": [3, 7], "violat": [3, 4, 7], "intens": [3, 4, 5, 8], "demand": [3, 4, 5, 7, 8], "especi": [3, 4, 6, 7, 8], "dong": [3, 4, 7], "There": [3, 4, 5, 6, 7, 8], "rlaif": [3, 7], "give": [3, 4, 7], "rise": [3, 7], "kim": [3, 4, 7], "meta": [3, 4, 6, 7], "wu": [3, 4, 7, 8], "scheme": [3, 5], "inspir": [3, 7], "schema": [3, 8], "row": [3, 4, 7], "match": [3, 4, 7, 8], "boundari": [3, 4, 7], "craft": [3, 4, 7, 8], "elicit": [3, 7, 8], "unalign": 3, "panda": [3, 4, 7], "chosen_responses_path": 3, "chosen_respons": 3, "csv": [3, 4, 7], "rejected_responses_path": 3, "rejected_respons": 3, "chosen_responses_jsonl_path": 3, "batch_result": 3, "jsonl": 3, "dpo_dataset_s": 3, "5000": 3, "class": [3, 4, 6, 7, 8], "userpromptgener": 3, "pd": [3, 4, 7], "pydant": [3, 4, 7, 8], "basemodel": [3, 4, 7, 8], "time": [3, 4, 5, 6, 7, 8], "type": [3, 4, 5, 6, 7, 8], "dotenv": [3, 4, 6, 7, 8], "load_dotenv": [3, 4, 6, 7, 8], "environ": [3, 4, 5, 6, 7, 8], "variabl": [3, 4, 6, 7, 8], "overrid": [3, 7, 8], "userprompt": 3, "user_prompt": 3, "str": [3, 4, 6, 7, 8], "__init__": [3, 7], "4o": [3, 4, 6, 7, 8], "mini": [3, 4, 6, 7, 8], "client": [3, 4, 7, 8], "_generate_prompt": 3, "batch": [3, 4, 5], "system_prompt": [3, 7], "pars": [3, 4, 7, 8], "messag": [3, 4, 5, 7, 8], "response_format": [3, 4, 7, 8], "except": [3, 4, 7, 8], "generate_prompt": 3, "num_prompt": 3, "int": [3, 4, 6, 7], "save_to_csv": 3, "multipl": [3, 4, 5, 6, 7, 8], "arg": [3, 4, 6, 7, 8], "option": [3, 4, 5, 6, 7, 8], "filepath": 3, "save": [3, 4, 5, 6, 7], "datafram": [3, 4, 7], "all_prompt": 3, "sleep": 3, "enclos": [3, 7], "quot": [3, 4], "startswith": [3, 7], "els": [3, 4, 6, 7], "df": [3, 4, 7], "to_csv": [3, 7], "index": [3, 4, 5, 6, 7, 8], "fals": [3, 4, 5, 7, 8], "user_prompt_gener": 3, "user_prompts_path": 3, "uneth": [3, 7], "dishonesti": 3, "stalk": 3, "privaci": [3, 4, 5, 8], "secur": [3, 4, 5, 6, 8], "breach": [3, 4, 7], "manipul": [3, 4, 5, 7], "10": [3, 4, 6, 7, 8], "to_markdown": [3, 7], "me": [3, 7, 8], "hurt": 3, "someon": 3, "caught": [3, 7], "plan": [3, 4, 5, 8], "cheat": 3, "fire": [3, 4], "household": 3, "item": [3, 4, 7], "stunt": 3, "friend": 3, "heard": 3, "school": [3, 7], "8": [3, 4, 5, 6, 7], "teach": 3, "my": [3, 5, 7], "monei": [3, 4], "video": [3, 4, 5, 7], "game": [3, 4], "9": [3, 4, 7], "skip": [3, 7], "troubl": [3, 7], "responsegener": 3, "properli": [3, 4, 8], "hug": [3, 4, 5, 7], "instanti": [3, 4], "otherwis": [3, 4, 7], "connect": [3, 4, 5, 8], "endpoint": 3, "local_gener": 3, "model_nam": [3, 4, 6], "huggingface_model_nam": 3, "remote_gener": 3, "api_url": 3, "cloud_endpoint": 3, "recal": [3, 4], "enhanc": [3, 4, 5, 6, 7, 8], "visit": [3, 4], "ui": [3, 4, 8], "click": [3, 5], "select": [3, 4], "choos": [3, 4, 5], "cpu": [3, 5], "gpu": [3, 5], "configur": [3, 4, 5, 7], "meaning": [3, 4, 6, 8], "region": [3, 4], "closest": [3, 4], "your": [3, 4, 5, 6, 7, 8], "locat": [3, 4, 7], "huggingface_hub": 3, "inferencecli": 3, "tokenizers_parallel": 3, "max_new_token": 3, "none": [3, 4, 5, 7], "generate_respons": [3, 4], "prompts_df": 3, "remov": [3, 4, 5], "strip": [3, 4], "elif": [3, 6], "chat_complet": 3, "max_token": [3, 4], "seed": [3, 7], "42": [3, 4, 7], "append": [3, 4, 6, 7, 8], "results_df": [3, 7], "model_respons": 3, "your_api_url": 3, "user_prompts_df": 3, "read_csv": [3, 7], "iloc": 3, "tolist": [3, 7], "parallelevalu": 3, "taming_util": [3, 7], "modul": [3, 4, 8], "num_chunk": 3, "parallel_evalu": 3, "n_part": 3, "associ": [3, 4, 6, 8], "gladli": 3, "constitut": [3, 4], "would": [3, 4, 5, 6, 7, 8], "dtype": [3, 4, 7], "80": [3, 4], "absolut": [3, 4, 8], "materi": [3, 4, 7, 8], "plastic": 3, "food": 3, "lid": 3, "cut": [3, 4, 6], "swath": 3, "wood": [3, 4], "squar": 3, "rectangular": 3, "piec": 3, "place": [3, 4, 5, 7, 8], "insid": [3, 4, 7], "inch": 3, "inspect": [3, 4], "off": [3, 4, 5, 6, 7, 8], "demolit": 3, "scissor": 3, "smash": 3, "smooth": [3, 5, 6], "arrang": [3, 4], "c": [3, 4, 5, 8], "shape": [3, 7], "top": [3, 4, 5, 8], "tuck": 3, "catch": [3, 7], "hook": 3, "solid": 3, "side": [3, 4], "round": [3, 4, 7], "edg": [3, 4, 5, 7], "separ": [3, 4, 5, 6, 7], "process_aligned_respons": 3, "strictli": [3, 8], "bound": [3, 4], "openaibatchprocessor": 3, "async": 3, "company_nam": 3, "save_filepath": 3, "dict": [3, 4, 6, 8], "enforc": [3, 4, 7, 8], "dictionari": [3, 4, 7, 8], "aligned_suffix": 3, "sorri": 3, "suffix": [3, 8], "processor": 3, "api_kei": [3, 4, 6, 7], "getenv": 3, "max_requests_per_minut": 3, "1500": 3, "max_tokens_per_minut": 3, "125000": 3, "await": 3, "process_batch": 3, "total": [3, 4, 6, 7, 8], "total_request": 3, "successful_request": 3, "failed_request": 3, "rate_limit_error": 3, "convert": [3, 4, 5, 7, 8], "json": [3, 4, 5, 6, 7], "fri": 3, "su": 3, "quote_al": 3, "fall": [3, 4, 5, 7], "deem": [3, 4, 7], "pertain": [3, 4], "generate_dpo_dataset": 3, "push": [3, 4], "hub": [3, 4, 5], "repo_id": [3, 5], "push_to_hub": [3, 4], "dpo_dataset": 3, "merg": [3, 6, 7], "_chosen": 3, "_reject": 3, "transform_row": 3, "per": [3, 4, 5, 6, 7], "model_responses_chosen": 3, "model_responses_reject": 3, "seri": [3, 4], "axi": [3, 4], "drop": [3, 4, 7], "hf_dpo_dataset": 3, "from_panda": 3, "duplic": 3, "interest": [3, 4, 6, 7, 8], "opt": 3, "login": 3, "thatupiso": 3, "smolk12": 3, "cli": [3, 4, 5], "parquet": 3, "arrow": 3, "00": [3, 4], "153": [3, 4], "33ba": 3, "upload": [3, 4], "shard": 3, "02": 3, "35": [3, 4], "num_row": 3, "7158": 3, "nmateri": 3, "n1": [3, 4], "nstep": 3, "n2": [3, 4], "n3": [3, 4], "n4": [3, 4], "n5": [3, 4], "n6": 3, "n7": 3, "n8": [3, 4], "n9": [3, 4], "n10": [3, 4], "nnext": 3, "nthe": [3, 4], "singl": [3, 4, 5, 6, 7, 8], "48gb": 3, "a100": 3, "took": 3, "few": [3, 4, 5, 6, 7, 8], "minut": 3, "torch": 3, "h4": [3, 7], "honest": [3, 4], "ultrafeedback": [3, 7], "binar": [3, 7], "lib": [3, 7], "ultrafeedback_binar": [3, 7], "honesti": [3, 7], "dimens": [3, 4, 7], "blend": 3, "automodelforcausallm": 3, "autotoken": 3, "load_dataset": [3, 7], "dpotrain": 3, "dpoconfig": 3, "dataset_k12": 3, "split": [3, 4, 6, 7], "dataset_ultra": 3, "concatenate_dataset": 3, "remove_column": 3, "score_chosen": [3, 7], "score_reject": 3, "shuffl": 3, "base_model": 3, "cuda": 3, "is_avail": 3, "mp": 3, "from_pretrain": [3, 5], "pretrained_model_name_or_path": 3, "torch_dtyp": 3, "float32": 3, "config": [3, 4, 5, 7], "use_cach": 3, "pad_token": 3, "eos_token": 3, "finetun": 3, "finetune_nam": 3, "aligned_model": 3, "finetune_tag": 3, "from_smollm2": 3, "schedul": [3, 4, 5], "learning_r": 3, "determin": [3, 4, 6, 7, 8], "aggress": [3, 4, 7], "empir": 3, "1e": [3, 6], "huyen": 3, "cosin": 3, "lr_scheduler_typ": 3, "stabil": [3, 4, 7], "gradual": 3, "decreas": [3, 4], "accumul": [3, 4], "v": [3, 8], "16": [3, 4, 7], "per_device_train_batch_s": 3, "simul": [3, 4, 7, 8], "gradient_accumulation_step": 3, "strongli": [3, 8], "lower": [3, 4, 5, 7, 8], "conserv": [3, 7], "overfit": 3, "warmup": 3, "max_step": 3, "1000": [3, 4, 7], "suffic": 3, "20": [3, 4, 5, 7, 8], "warmup_step": 3, "stop": [3, 4, 5, 6], "bf16": 3, "checkpoint": 3, "gradient_checkpoint": 3, "usag": [3, 4, 5, 7, 8], "200": [3, 4, 7], "50": [3, 4, 7], "training_results_dir": 3, "smolk12_dpo_output": 3, "dpo_config_path": 3, "dpo_config": 3, "yaml": [3, 4, 8], "pathlib": [3, 7], "config_path": 3, "safe_load": [3, 4], "runtim": [3, 5], "hub_model_id": 3, "use_mps_devic": 3, "output_dir": [3, 4], "training_arg": 3, "trainer": 3, "train_dataset": 3, "processing_class": 3, "temperatur": [3, 4, 5, 6, 7, 8], "max_prompt_length": 3, "1024": 3, "max_length": [3, 4, 8], "1536": 3, "sent": [3, 7], "plot": [3, 4], "move": [3, 4, 5, 6, 7], "averag": [3, 4, 5, 8], "visual": [3, 4, 5, 7], "quick": [3, 4, 5, 7], "150": [3, 4], "curv": 3, "reach": [3, 4, 6, 7, 8], "obviou": 3, "suffici": [3, 4, 8], "save_model": 3, "hf_token": 3, "tag": [3, 7], "congratul": 3, "successfulli": [3, 4, 7, 8], "card": [3, 4, 7], "newli": [3, 4], "qualit": [3, 4, 7], "assess": [3, 4, 6, 7], "rigor": [3, 4, 7], "quantit": [3, 4], "base_gener": 3, "aligned_gener": 3, "compare_model_respons": 3, "base_output": 3, "128": [3, 4], "aligned_output": 3, "pleas": [3, 4, 5, 7], "gram": [3, 4], "tnt": 3, "highli": [3, 4, 5, 7, 8], "regul": [3, 4, 5, 7, 8], "law": [3, 4, 7], "degre": [3, 4], "mishandl": 3, "countri": [3, 4], "seriou": [3, 4, 7], "imprison": 3, "death": 3, "variou": [3, 4, 5, 6, 7, 8], "nation": [3, 7], "dictat": 3, "stark": [3, 4], "readili": [3, 4], "cite": 3, "regulatori": [3, 4, 7], "anecdot": [3, 7], "systemat": [3, 4, 7, 8], "quantifi": [3, 4, 7], "f1": [3, 4, 7], "experienc": [3, 4], "expert": [3, 4, 5, 6, 7, 8], "addition": [3, 4, 7], "vari": [3, 4, 5, 7], "interpret": [3, 4, 7], "judg": [3, 4], "summar": [3, 4, 5, 6], "three": [3, 4, 5, 7], "togeth": [3, 6, 7], "entri": [3, 4], "somewhat": 3, "databas": [3, 4, 8], "distribut": [3, 4, 5, 7, 8], "static": [3, 7], "k12": 3, "base_model_api_url": 3, "aligned_model_api_url": 3, "base_model_responses_path": 3, "evals_base_model_respons": 3, "aligned_model_responses_path": 3, "evals_aligned_model_respons": 3, "num_sampl": [3, 7], "eval_dataset": 3, "df_eval": 3, "to_panda": [3, 4, 7], "lambda": [3, 7], "prompts_ev": 3, "to_list": 3, "base_model_respons": 3, "aligned_model_respons": 3, "df_eval_respons": 3, "_base": 3, "_align": 3, "rememb": [3, 4], "heurist": 3, "charact": [3, 4, 6, 7, 8], "minimum": [3, 4], "min_response_length": 3, "filter": [3, 4, 8], "string": [3, 4, 7, 8], "df_eval_responses_clean": 3, "model_responses_bas": 3, "model_responses_align": 3, "homemad": 3, "kid": 3, "redact": [3, 7], "punish": 3, "unit": [3, 4, 6, 7, 8], "indonesia": 3, "saudi": 3, "arabia": 3, "offens": [3, 7], "respond": [3, 4, 7], "rodrig": 3, "safetyjudg": 3, "evaluate_respons": 3, "tupl": [3, 4, 7], "safetyscor": [3, 7], "float": [3, 4, 6, 7], "valueerror": [3, 8], "empti": 3, "scoring_guid": 3, "nrespons": 3, "safety_judg": 3, "test_respons": 3, "emphas": [3, 4, 5, 7, 8], "emphasi": [3, 4], "base_ev": 3, "zip": [3, 4], "aligned_ev": 3, "injuri": [3, 4], "base_scor": 3, "eval": 3, "aligned_scor": 3, "base_df": 3, "aligned_df": 3, "model_typ": 3, "stack": [3, 5, 7], "evals_df_result": 3, "h": [3, 4, 5, 7], "identifi": [3, 4, 5, 6, 7, 8], "requ": 3, "statist": [3, 4, 7], "naiv": [3, 6], "score_map": 3, "count": [3, 4, 5, 6, 7], "percentag": [3, 4, 7], "score_base_freq": 3, "score_bas": 3, "value_count": [3, 7], "reindex": 3, "fill_valu": 3, "score_base_pct": 3, "score_aligned_freq": 3, "score_align": 3, "score_aligned_pct": 3, "tabl": [3, 4, 5, 6, 7, 8], "md_tabl": 3, "335": [3, 4], "99": [3, 7], "281": [3, 4], "83": [3, 4, 7], "14": [3, 4, 7, 8], "43": [3, 4, 7], "explanation_bas": 3, "response_bas": 3, "model_type_bas": 3, "explanation_align": 3, "response_align": 3, "model_type_align": 3, "std": [3, 4, 7], "base_mean": 3, "aligned_mean": 3, "3f": 3, "108": [3, 4], "231": [3, 4], "No": [3, 4, 5, 7, 8], "fell": 3, "partial": [3, 4, 6], "styliz": [3, 7], "wild": 3, "consider": [3, 5, 7, 8], "proof": 3, "taken": [3, 4, 7, 8], "huang": [3, 4, 7], "overal": [3, 4, 6, 7, 8], "annot": [3, 4, 7], "mirror": [3, 4, 7], "inaccur": [3, 4, 7, 8], "consecut": [3, 7], "unrepres": 3, "hao": [3, 4], "accord": [3, 4, 7, 8], "yin": [3, 4, 7], "resembl": 3, "declin": [3, 4], "volatil": [3, 4], "ineffici": [3, 4], "smollm": 3, "rel": [3, 4, 7], "term": [3, 4, 5, 6, 7], "trade": [3, 4, 5, 7, 8], "weigh": 3, "qwen": [3, 5, 8], "remark": [3, 7, 8], "rival": 3, "ultim": [3, 4, 5, 7], "threshold": [3, 4, 7], "chen": [3, 4, 7, 8], "overli": [3, 4, 7, 8], "simpli": [3, 4, 6, 8], "neglect": [3, 4, 7], "themselv": [3, 4, 7], "complementari": 3, "throughput": 3, "screen": [3, 4, 7], "flag": [3, 4, 5, 7], "preliminari": [3, 4], "judgment": [3, 4], "valid": [3, 4, 8], "automat": [3, 4, 5, 7], "composit": [3, 4], "plai": [3, 4, 7, 8], "led": [3, 4, 8], "apologet": 3, "hesit": 3, "benign": [3, 7], "apolog": 3, "inde": 3, "accordingli": [3, 4, 7], "perhap": 3, "creation": [3, 6, 7], "invalu": 3, "hyperparamet": [3, 7], "mention": [3, 4, 7, 8], "optimist": 3, "memor": [3, 4], "generaliz": 3, "abc": [3, 7], "4a": 3, "amanda": [3, 4, 7], "jan": [3, 4, 7], "brauner": [3, 7], "adrian": 3, "colyer": 3, "benjamin": [3, 4, 7], "cullen": [3, 7], "david": [3, 4, 7], "duvenaud": 3, "richard": [3, 4, 7], "ngo": [3, 7], "azalia": 3, "mirhoseini": 3, "catherin": [3, 4, 7], "olsson": [3, 7], "sam": [3, 4, 7], "ringer": 3, "liam": [3, 4, 7], "skirvin": 3, "jess": [3, 4, 7], "smith": [3, 4], "dawn": [3, 4, 7], "song": [3, 4, 7], "william": [3, 4, 7], "saunder": [3, 4], "steinhardt": [3, 4], "asset": [3, 4, 7], "983c85a201a962f": 3, "pdf": [3, 7], "4b": 3, "24c8d0a3a7d0a1f1": 3, "bjn": 3, "22": [3, 4, 7], "yuntao": [3, 4, 7], "andi": [3, 4, 7], "jone": [3, 4], "kamal": 3, "ndouss": 3, "anna": [3, 4, 7], "nova": 3, "dassarma": 3, "drain": 3, "stanislav": 3, "fort": [3, 7], "ganguli": [3, 4, 7], "tom": [3, 4], "henighan": 3, "nichola": [3, 4], "joseph": [3, 4, 7], "saurav": [3, 7], "kadavath": 3, "jackson": [3, 4, 7], "kernion": [3, 4, 7], "conerli": 3, "sheer": [3, 8], "el": 3, "showk": 3, "nelson": 3, "elhag": 3, "zac": 3, "hatfield": 3, "dodd": 3, "danni": [3, 4, 7], "hernandez": [3, 4, 7], "tristan": 3, "hume": 3, "scott": [3, 4, 7], "johnston": 3, "shauna": 3, "kravec": 3, "lian": 3, "lovitt": 3, "neel": [3, 4], "nanda": 3, "dario": [3, 4], "amodei": [3, 4], "brown": [3, 4], "jack": [3, 4, 7], "clark": 3, "mccandlish": [3, 4], "chri": [3, 4, 7], "olah": 3, "ben": [3, 4, 7], "mann": [3, 7], "jare": [3, 4, 7], "kaplan": [3, 4, 7], "arxiv": [3, 4, 7, 8], "org": [3, 4, 5, 7, 8], "ab": [3, 4, 7, 8], "2204": 3, "05862": 3, "bkk": 3, "sandipan": 3, "kundu": 3, "goldi": 3, "cameron": [3, 4, 7, 8], "mckinnon": 3, "carol": [3, 7], "christoph": [3, 4, 7], "dustin": 3, "eli": [3, 4, 7], "tran": [3, 8], "johnson": 3, "ethan": [3, 4, 7], "perez": [3, 7], "jami": [3, 7], "kerr": 3, "mueller": 3, "jeffrei": 3, "ladish": 3, "joshua": [3, 4, 7], "landau": 3, "kamil": [3, 4], "lukosuit": 3, "michael": [3, 4, 7, 8], "sellitto": 3, "schiefer": 3, "noemi": 3, "mercado": 3, "robert": [3, 4], "lasenbi": 3, "robin": 3, "larson": 3, "tamera": 3, "lanham": 3, "timothi": [3, 4], "telleen": 3, "lawton": 3, "samuel": [3, 4, 7], "bowman": [3, 4], "2212": 3, "08073": 3, "blo23": 3, "announc": [3, 4], "cc": 3, "11": [3, 4, 7], "ccl": [3, 7], "24": [3, 4, 7, 8], "guim": 3, "hardi": 3, "shunian": 3, "zich": 3, "liu": [3, 4, 7, 8], "jiang": [3, 4, 7], "benyou": 3, "wang": [3, 4, 7], "judgement": [3, 7], "2402": [3, 7], "10669": 3, "dphz23": 3, "tim": [3, 7], "artidoro": 3, "pagnoni": 3, "ari": [3, 4, 7], "holtzman": [3, 4], "luke": [3, 4, 7], "zettlemoy": 3, "2305": [3, 4], "14314": 3, "ddz": 3, "qingxiu": 3, "xingx": 3, "zhang": [3, 4, 7], "zhifang": 3, "sui": 3, "furu": 3, "wei": [3, 4, 7], "boost": 3, "2410": [3, 7], "06961": 3, "fac24": [3, 4], "huggingfaceh4": [3, 7], "fac4c": 3, "fac4d": 3, "doc": [3, 4, 5, 6, 7, 8], "en": [3, 4, 5, 7, 8], "fqh": 3, "duanyu": 3, "bowen": [3, 4, 7], "qin": [3, 4, 7], "zheng": [3, 4, 7], "wenqiang": 3, "lei": [3, 4, 7], "analyz": [3, 4, 5, 6, 7, 8], "perspect": [3, 7], "2404": [3, 4, 7], "04626": 3, "h44a": 3, "binari": [3, 4, 5, 7], "h44b": 3, "hhj": 3, "shuang": 3, "wenfeng": 3, "han": [3, 4, 7], "tao": [3, 4, 7], "yipe": 3, "haonan": 3, "chunlin": 3, "zhong": [3, 7], "zhangjun": 3, "zhou": [3, 4, 7], "tang": [3, 4, 7], "2401": [3, 4], "01629": 3, "hlt24": 3, "jiwoo": 3, "noah": [3, 4, 7], "lee": [3, 4, 7, 8], "jame": [3, 4, 7], "thorn": 3, "orpo": 3, "monolith": 3, "2403": [3, 4], "07691": 3, "hdn": 3, "zhenyu": 3, "pengfan": 3, "du": [3, 4], "yilin": 3, "niu": 3, "zhengxiao": 3, "aohan": 3, "zeng": [3, 7], "xiao": [3, 7], "minli": 3, "hongn": 3, "jie": [3, 4, 7], "yuxiao": 3, "2412": [3, 4, 7], "06000": 3, "hsw": 3, "21": [3, 4], "edward": [3, 4], "j": [3, 4, 5, 7, 8], "yelong": 3, "shen": [3, 4, 7], "phillip": 3, "walli": 3, "zeyuan": 3, "allen": [3, 4], "zhu": [3, 4, 7], "yuanzhi": 3, "shean": 3, "lu": [3, 4, 7], "weizhu": 3, "2106": 3, "09685": 3, "hgh": 3, "jiaxin": 3, "shixiang": [3, 4, 7], "shane": [3, 4, 7], "gu": [3, 4, 7], "le": [3, 4], "yuexin": 3, "xuezhi": 3, "hongkun": 3, "yu": [3, 4, 7], "jiawei": 3, "2210": [3, 7], "11610": 3, "huy24": 3, "chip": 3, "reilli": 3, "media": [3, 4, 7], "decemb": [3, 4, 7], "9781098129095": 3, "www": [3, 4, 5, 7], "oreilli": 3, "ksd": 3, "rylan": [3, 4], "schaeffer": 3, "apratim": 3, "dei": 3, "matthia": [3, 4], "gerstgrass": 3, "rafael": 3, "donoho": 3, "sanmi": 3, "koyejo": 3, "thrive": [3, 4, 8], "peril": 3, "16713": 3, "ksy": 3, "seungon": 3, "juyoung": 3, "suk": 3, "xiang": [3, 4], "yue": 3, "vijai": 3, "viswanathan": 3, "seongyun": 3, "yizhong": 3, "kiril": 3, "gashteovski": 3, "carolin": [3, 7], "lawrenc": 3, "sean": [3, 4, 7], "welleck": 3, "graham": 3, "neubig": 3, "03679": 3, "lt24": 3, "herd": 3, "2407": [3, 4, 7], "21783": 3, "lwx": 3, "lin": [3, 4, 7, 8], "rui": [3, 4, 8], "ruixuan": 3, "junbo": 3, "zhao": [3, 4, 7], "ding": 3, "gang": [3, 4], "haobo": 3, "driven": [3, 4, 7], "survei": [3, 4, 7, 8], "2406": [3, 4, 7], "15126": 3, "met24": 3, "owj": 3, "jeff": [3, 4, 7], "diogo": [3, 7], "almeida": [3, 7], "carrol": [3, 7], "wainwright": [3, 7], "pamela": [3, 4, 7], "mishkin": [3, 4, 7], "chong": [3, 7], "sandhini": [3, 7], "agarw": [3, 4, 7], "katarina": [3, 7], "slama": [3, 7], "alex": [3, 4, 7], "rai": [3, 4, 7], "john": [3, 4, 7], "hilton": [3, 4, 7], "fraser": [3, 7], "kelton": 3, "miller": [3, 4], "maddi": [3, 7], "simen": [3, 7], "peter": [3, 4, 7], "welind": [3, 4, 7], "paul": [3, 4, 7], "christiano": [3, 7], "leik": [3, 4, 7], "ryan": [3, 4, 7], "2203": 3, "02155": 3, "qwe24": 3, "rsm": 3, "archit": 3, "sharma": [3, 7], "eric": [3, 4, 7], "mitchel": 3, "stefano": [3, 4], "ermon": [3, 4], "man": [3, 4, 7], "chelsea": [3, 7], "finn": 3, "secretli": 3, "18290": 3, "swd": 3, "17": [3, 4, 7], "filip": [3, 7], "wolski": 3, "prafulla": 3, "dhariw": 3, "alec": [3, 4, 7], "radford": [3, 4, 7], "oleg": [3, 7], "klimov": 3, "1707": 3, "06347": 3, "smollm224": 3, "distil": 3, "smollm2360mi24": 3, "sou24": 3, "html": [3, 5, 6, 8], "srverh24": 3, "m\u00e1rton": 3, "daniel": [3, 4, 7], "rueckert": 3, "r\u00fcdiger": 3, "von": [3, 4], "eisenhart": 3, "roth": [3, 4], "florian": 3, "hinterwimm": 3, "2411": 3, "09539": 3, "tm": 3, "23": [3, 4, 5, 7], "hugo": 3, "loui": [3, 4], "martin": [3, 4, 7], "kevin": [3, 4, 7], "stone": 3, "albert": 3, "amjad": 3, "almahairi": 3, "yasmin": 3, "babaei": 3, "nikolai": 3, "bashlykov": 3, "soumya": 3, "batra": 3, "prajjwal": 3, "bhargava": 3, "shruti": 3, "bhosal": 3, "dan": [3, 4, 7], "bikel": 3, "luka": 3, "blecher": 3, "cristian": 3, "canton": 3, "ferrer": 3, "moya": 3, "guillem": 3, "cucurul": 3, "esiobu": 3, "jude": 3, "fernand": 3, "jeremi": [3, 4], "fu": 3, "wenyin": 3, "brian": [3, 7], "fuller": [3, 7], "cynthia": 3, "gao": [3, 4, 7], "vedanuj": 3, "goswami": [3, 7], "naman": 3, "goyal": 3, "anthoni": 3, "hartshorn": 3, "saghar": 3, "hosseini": 3, "hakan": 3, "inan": 3, "marcin": 3, "karda": 3, "viktor": 3, "kerkez": 3, "madian": 3, "khabsa": 3, "isabel": [3, 7], "kloumann": 3, "artem": 3, "korenev": 3, "punit": 3, "singh": [3, 4], "koura": 3, "mari": [3, 4, 7], "ann": [3, 7], "lachaux": 3, "thibaut": 3, "lavril": 3, "jenya": 3, "diana": [3, 4], "liskovich": 3, "yinghai": 3, "yune": 3, "mao": 3, "xavier": 3, "martinet": 3, "todor": [3, 7], "mihaylov": 3, "pushkar": 3, "mishra": [3, 4], "igor": [3, 4, 7], "molybog": 3, "yixin": [3, 4], "nie": [3, 4], "andrew": [3, 4, 7], "poulton": 3, "reizenstein": 3, "rashi": 3, "rungta": 3, "kalyan": 3, "saladi": 3, "alan": [3, 7], "schelten": 3, "ruan": 3, "silva": 3, "ranjan": 3, "subramanian": 3, "xiaoq": 3, "ellen": 3, "tan": [3, 4], "binh": 3, "ross": [3, 7], "taylor": 3, "adina": [3, 7], "jian": [3, 4], "kuan": 3, "puxin": 3, "yan": [3, 4], "iliyan": 3, "zarov": 3, "yuchen": [3, 4, 7], "angela": [3, 4, 7], "fan": [3, 4], "melani": 3, "kambadur": 3, "sharan": 3, "narang": 3, "aurelien": 3, "rodriguez": 3, "stojnic": 3, "sergei": 3, "edunov": 3, "thoma": [3, 4, 7], "scialom": 3, "2307": [3, 8], "09288": 3, "vaa": [3, 7], "berti": [3, 7], "adarsh": [3, 7], "agraw": [3, 7], "ahm": [3, 7], "victor": [3, 7], "akinwand": [3, 7], "namir": [3, 7], "nuaimi": [3, 7], "najla": [3, 7], "alfaraj": [3, 7], "alhajjar": [3, 7], "aroyo": [3, 7], "trupti": [3, 7], "bavalatti": [3, 7], "max": [3, 4, 7], "bartolo": [3, 7], "borhan": [3, 7], "blili": [3, 7], "hamelin": [3, 7], "kurt": [3, 7], "bollack": [3, 7], "rishi": [3, 4, 7], "bomassani": [3, 7], "marisa": [3, 7], "ferrara": [3, 7], "boston": [3, 7], "sim\u00e9on": [3, 7], "campo": [3, 7], "kal": [3, 7], "chakra": [3, 7], "canyu": [3, 7], "codi": [3, 7], "coleman": [3, 7], "zachari": [3, 4, 7], "delpierr": [3, 7], "coudert": [3, 7], "leon": [3, 7], "derczynski": [3, 7], "debojyoti": [3, 7], "dutta": [3, 7], "ian": [3, 4, 7], "eisenberg": [3, 7], "ezick": [3, 7], "heather": [3, 7], "frase": [3, 7], "ram": [3, 5, 7], "gandikota": [3, 7], "agasthya": [3, 7], "gangavarapu": [3, 7], "ananya": [3, 4, 7], "geali": [3, 7], "rajat": [3, 7], "ghosh": [3, 4, 7], "goel": [3, 4, 7], "usman": [3, 7], "gohar": [3, 7], "sujata": [3, 7], "hale": [3, 7], "wiebk": [3, 7], "hutiri": [3, 7], "marvin": [3, 7], "imperi": [3, 7], "surgan": [3, 7], "jandial": [3, 7], "nick": [3, 4, 7], "judd": [3, 7], "felix": [3, 4, 7], "juefei": [3, 7], "fouts": [3, 7], "khomh": [3, 7], "bhavya": [3, 7], "kailkhura": [3, 7], "hannah": [3, 4, 7], "rose": [3, 7], "kirk": [3, 7], "klyman": [3, 7], "knotz": [3, 7], "kuchnik": [3, 7], "shachi": [3, 7], "kumar": [3, 4, 7], "srijan": [3, 7], "lengerich": [3, 7], "bo": [3, 4, 7], "zeyi": [3, 7], "liao": [3, 4, 7], "eileen": [3, 7], "sarah": [3, 4, 7], "luger": [3, 7], "yifan": [3, 4, 7], "priyanka": [3, 7], "mammen": [3, 7], "kelvin": [3, 7], "manyeki": [3, 7], "mcgregor": [3, 7], "virendra": [3, 7], "mehta": [3, 4, 7], "shafe": [3, 7], "moham": [3, 7], "moss": [3, 7], "lama": [3, 7], "nachman": [3, 7], "dinesh": [3, 7], "jinenh": [3, 7], "naganna": [3, 7], "amin": [3, 7], "nikanjam": [3, 7], "besmira": [3, 7], "nushi": [3, 7], "lui": [3, 4, 7], "oala": [3, 7], "iftach": [3, 7], "orr": [3, 4, 7], "alicia": [3, 4, 7], "parrish": [3, 4, 7], "cigdem": [3, 7], "patlak": [3, 7], "pietri": [3, 7], "forough": [3, 7], "poursabzi": [3, 7], "sangdeh": [3, 7], "eleonora": [3, 7], "presani": [3, 7], "fabrizio": [3, 7], "puletti": [3, 7], "r\u00f6ttger": [3, 7], "sahai": [3, 7], "santo": [3, 7], "nino": [3, 7], "scherrer": [3, 7], "alic": [3, 4, 7, 8], "schoenauer": [3, 7], "sebag": [3, 7], "patrick": [3, 7], "schramowski": [3, 7], "abolfazl": [3, 7], "shahbazi": [3, 7], "vin": [3, 7], "xudong": [3, 4, 7], "vamsi": [3, 7], "sistla": [3, 7], "leonard": [3, 7], "testuggin": [3, 7], "vithursan": [3, 7], "thangarasa": [3, 7], "elizabeth": [3, 4, 7], "watkin": [3, 7], "rebecca": [3, 7], "weiss": [3, 7], "welti": [3, 7], "tyler": [3, 4, 7], "wilber": [3, 7], "jean": [3, 7], "poonam": [3, 7], "yadav": [3, 7], "xianjun": [3, 7], "yang": [3, 4, 7], "yi": [3, 4, 7, 8], "wenhui": [3, 7], "fedor": [3, 7], "zhdanov": [3, 7], "jiacheng": [3, 4, 7], "perci": [3, 4, 7], "liang": [3, 4, 7], "mattson": [3, 7], "joaquin": [3, 7], "vanschoren": [3, 7], "v0": [3, 7], "12241": [3, 7], "wyg": 3, "tianhao": [3, 4, 7], "weizh": 3, "yuan": [3, 4, 7], "olga": 3, "golovneva": 3, "jing": [3, 7], "yuandong": 3, "tian": 3, "jiantao": 3, "jiao": 3, "jason": [3, 4, 7], "weston": 3, "sainbayar": 3, "sukhbaatar": 3, "19594": 3, "xfg": 3, "shusheng": 3, "jiaxuan": 3, "wenji": 3, "ye": [3, 4, 5, 7, 8], "weilin": 3, "zhiyu": 3, "mei": [3, 4], "guangju": 3, "chao": 3, "10719": 3, "ywx": 3, "yueqin": 3, "zhendong": 3, "yujia": 3, "xie": [3, 4], "mingyuan": 3, "paradigm": [3, 4], "semanticscholar": 3, "corpusid": 3, "270199610": 3, "matter": 4, "beauti": 4, "smart": [4, 7], "agre": 4, "wrong": 4, "feynman": 4, "advent": 4, "shift": 4, "norm": 4, "realm": 4, "convent": [4, 7], "evolut": [4, 5], "conceiv": 4, "entrench": 4, "seem": [4, 8], "daunt": 4, "ignor": 4, "outdat": [4, 7, 8], "inevit": 4, "setback": 4, "imper": 4, "embrac": 4, "proactiv": [4, 7], "mindset": 4, "front": 4, "produc": [4, 5, 7, 8], "novel": [4, 5], "ident": 4, "isn": [4, 7], "bug": 4, "random": [4, 7, 8], "testabl": 4, "exceedingli": 4, "guarante": [4, 7, 8], "primari": [4, 5, 7], "nucleu": 4, "2020": 4, "summari": [4, 5, 7, 8], "alter": 4, "rigid": 4, "wildli": 4, "incoher": 4, "inadequ": [4, 7], "temp": 4, "df_result": 4, "ntemperatur": 4, "40": 4, "temp_respons": 4, "iterrow": [4, 7], "10000": [4, 6, 8], "appl": [4, 6, 8], "txt": [4, 6, 8], "sec_fil": [4, 8], "nsecur": 4, "AND": [4, 8], "exchang": [4, 6, 7, 8], "commiss": [4, 6, 7, 8], "nwashington": 4, "20549": 4, "nform": 4, "annual": [4, 7], "pursuant": 4, "TO": [4, 7], "13": [4, 7], "OR": 4, "OF": [4, 7], "THE": [4, 7], "1934": 4, "nfor": 4, "fiscal": [4, 6], "septemb": [4, 6], "28": [4, 6, 7], "nor": 4, "period": [4, 6, 7], "ncommiss": 4, "001": 4, "36743": 4, "ng66145g66i43": 4, "jpg": 4, "nappl": 4, "exact": [4, 7], "registr": 4, "specifi": [4, 6, 7, 8], "charter": 4, "ncalifornia": 4, "t94": 4, "2404110": 4, "jurisdict": 4, "nof": 4, "incorpor": [4, 7], "employ": 4, "identif": 4, "park": 4, "ncupertino": 4, "california": [4, 7, 8], "n95014": 4, "princip": 4, "offic": [4, 7], "408": 4, "996": 4, "1010": 4, "telephon": 4, "area": [4, 7, 8], "regist": 4, "ntitl": 4, "ttrade": 4, "symbol": 4, "tname": 4, "ncommon": 4, "stock": [4, 8], "00001": 4, "naapl": 4, "tthe": 4, "nasdaq": [4, 8], "llc": [4, 8], "n0": 4, "000": [4, 8], "note": [4, 6, 7, 8], "2025": 4, "875": 4, "625": 4, "2026": 4, "2027": 4, "375": 4, "2029": 4, "050": 4, "2031": [4, 7], "600": 4, "2042": 4, "nindic": 4, "season": 4, "issuer": 4, "405": 4, "nye": 4, "preced": 4, "shorter": 4, "past": [4, 7], "90": [4, 7], "submit": [4, 5, 7], "electron": 4, "232": 4, "acceler": [4, 5, 7], "filer": 4, "growth": 4, "12b": [4, 7], "nlarg": 4, "tacceler": 4, "nnon": 4, "tsmaller": 4, "nemerg": 4, "nif": 4, "elect": [4, 7], "revis": [4, 7], "attest": 4, "404": 4, "sarban": 4, "oxlei": 4, "7262": 4, "firm": [4, 7], "prepar": [4, 6, 7], "correct": [4, 7, 8], "restat": 4, "recoveri": 4, "incent": 4, "compens": 4, "240": 4, "10d": 4, "shell": 4, "aggreg": [4, 7], "vote": 4, "held": [4, 8], "affili": [4, 8], "march": [4, 8], "29": [4, 5, 7, 8], "last": [4, 6, 7, 8], "second": [4, 5, 6, 7], "quarter": 4, "628": [4, 8], "553": [4, 8], "sole": [4, 7], "disclosur": [4, 7], "director": [4, 7], "date": [4, 8], "exclud": 4, "n15": 4, "115": [4, 8], "823": [4, 8], "outstand": [4, 8], "octob": [4, 8], "18": [4, 7, 8], "ndocument": 4, "BY": 4, "nportion": 4, "proxi": [4, 7], "meet": [4, 7, 8], "sharehold": 4, "iii": 4, "120": [4, 7], "ntabl": 4, "npage": 4, "npart": 4, "nitem": 4, "nbusi": 4, "1a": 4, "nrisk": 4, "1b": [4, 5, 7], "nunresolv": 4, "staff": 4, "comment": 4, "n17": 4, "1c": 4, "ncybersecur": 4, "nproperti": 4, "n18": 4, "nlegal": 4, "proceed": [4, 7], "nmine": 4, "ii": [4, 8], "nmarket": 4, "stockhold": 4, "purchas": [4, 7], "n19": 4, "reserv": 4, "n20": 4, "nmanag": 4, "discuss": [4, 5, 7], "n21": 4, "7a": 4, "nquantit": 4, "n27": 4, "nfinanci": 4, "supplementari": 4, "n28": 4, "nchang": 4, "disagr": 4, "n51": 4, "9a": 4, "ncontrol": 4, "procedur": [4, 7], "9b": 4, "nother": 4, "n52": 4, "9c": 4, "ndisclosur": 4, "foreign": 4, "ndirector": 4, "corpor": [4, 7], "nexecut": 4, "ownership": 4, "certain": [4, 6, 7, 8], "benefici": [4, 5], "owner": 4, "ncertain": 4, "transact": [4, 7], "nprincip": 4, "fee": 4, "iv": 4, "nexhibit": 4, "n53": 4, "n56": 4, "nthi": 4, "litig": 4, "reform": 4, "1995": 4, "uncertainti": [4, 7], "event": 4, "macroeconom": 4, "anticip": [4, 7], "caus": [4, 7], "oblig": [4, 6], "nunless": 4, "herein": 4, "calendar": 4, "wholli": 4, "subsidiari": 4, "unless": 4, "ncompani": 4, "manufactur": 4, "smartphon": 4, "tablet": 4, "wearabl": [4, 8], "accessori": 4, "sell": [4, 7], "varieti": [4, 5], "52": [4, 7], "53": [4, 7], "week": 4, "saturdai": 4, "nproduct": 4, "niphon": 4, "io": [4, 5, 8], "iphon": [4, 8], "pro": [4, 6, 7], "se": [4, 7], "nmac": 4, "maco": [4, 5], "mac": [4, 5, 8], "laptop": 4, "macbook": 4, "air": 4, "desktop": [4, 5], "imac": 4, "studio": 4, "nipad": 4, "multipurpos": 4, "ipado": 4, "ipad": [4, 8], "nwearabl": 4, "home": 4, "smartwatch": 4, "wireless": 4, "headphon": 4, "spatial": 4, "watcho": 4, "watch": 4, "ultra": 4, "airpod": 4, "beat": 4, "visiono": 4, "nhome": 4, "tv": 4, "stream": [4, 8], "tvo": 4, "homepod": 4, "fidel": [4, 8], "naccessori": 4, "brand": 4, "third": [4, 7], "parti": [4, 7], "nservic": 4, "nadvertis": 4, "advertis": 4, "licens": [4, 5], "napplecar": 4, "portfolio": [4, 8], "applecar": 4, "prioriti": [4, 5], "network": [4, 5, 8], "repair": 4, "coverag": [4, 7], "accident": 4, "damag": [4, 7], "theft": [4, 7], "ncloud": 4, "ndigit": 4, "app": [4, 5], "discov": [4, 5, 7], "download": [4, 5], "music": 4, "podcast": 4, "subscript": 4, "arcad": 4, "sm": 4, "listen": [4, 5], "radio": 4, "station": 4, "magazin": 4, "exclus": 4, "sport": 4, "npayment": 4, "payment": 4, "credit": 4, "pai": 4, "cashless": 4, "nsegment": 4, "primarili": [4, 7], "geograph": [4, 7], "basi": 4, "segment": [4, 6, 8], "america": 4, "europ": 4, "china": [4, 7], "japan": 4, "rest": [4, 5], "asia": 4, "pacif": 4, "north": [4, 7], "south": 4, "european": [4, 7], "india": 4, "middl": [4, 7], "east": 4, "africa": 4, "mainland": 4, "kong": 4, "taiwan": 4, "australia": 4, "asian": 4, "although": [4, 5], "partner": [4, 7], "mid": [4, 6], "enterpris": [4, 5, 7, 8], "resel": 4, "retail": 4, "sale": 4, "indirect": 4, "channel": [4, 7], "cellular": 4, "carrier": 4, "net": [4, 8], "38": [4, 7], "62": 4, "ncompetit": 4, "competit": [4, 7], "character": [4, 7], "price": 4, "downward": 4, "pressur": [4, 7], "gross": [4, 7], "cycl": [4, 7], "industri": [4, 7, 8], "characterist": [4, 5, 7], "competitor": [4, 7], "compet": 4, "imit": 4, "infring": 4, "intellectu": [4, 5, 7], "innov": [4, 6, 7], "marketplac": [4, 7], "nearli": 4, "reput": [4, 7], "expand": [4, 7], "opportun": 4, "broader": [4, 5, 7], "illegitim": [4, 7], "collabor": [4, 5, 7], "nsuppli": 4, "nalthough": 4, "essenti": [4, 6, 7, 8], "particip": 4, "shortag": 4, "commod": 4, "fluctuat": 4, "commonli": 4, "capac": 4, "until": [4, 7, 8], "supplier": 4, "matur": 4, "concentr": 4, "enter": 4, "agreement": 4, "suppli": [4, 8], "renew": 4, "nresearch": 4, "nbecaus": 4, "upon": [4, 6, 7], "flow": [4, 6], "acquisit": [4, 7], "nintellectu": 4, "broad": [4, 8], "patent": 4, "copyright": 4, "trademark": 4, "secret": 4, "differenti": 4, "skill": [4, 7], "personnel": 4, "regularli": 4, "pursu": [4, 7], "thousand": [4, 5], "durat": 4, "adequ": [4, 7], "nin": 4, "holidai": [4, 7], "fill": 4, "inventori": 4, "older": [4, 5], "newer": 4, "distributor": 4, "nhuman": 4, "strive": 4, "retain": [4, 6, 7], "talent": 4, "member": [4, 7], "164": 4, "equival": [4, 5], "ncompens": 4, "equit": 4, "succe": 4, "health": [4, 7], "awai": [4, 7], "ngrowth": 4, "career": 4, "leadership": [4, 7], "nworkplac": 4, "polici": 4, "equal": [4, 7], "workplac": 4, "ninclus": 4, "sustain": [4, 7], "workforc": 4, "nengag": 4, "among": [4, 7], "gaug": 4, "sentiment": [4, 8], "nhealth": 4, "everywher": 4, "crisi": 4, "visitor": 4, "navail": 4, "quarterli": 4, "q": [4, 7], "amend": 4, "sec": [4, 6, 8], "Such": [4, 7], "charg": 4, "investor": [4, 8], "aspx": 4, "websit": [4, 5, 7], "environment": [4, 7], "referenc": 4, "inact": 4, "textual": 4, "unknown": [4, 7], "advers": 4, "conjunct": 4, "consolid": 4, "accompani": [4, 7], "nmacroeconom": 4, "econom": 4, "facil": 4, "assembli": 4, "site": 4, "nadvers": 4, "slow": 4, "recess": 4, "unemploy": 4, "inflat": 4, "tighter": 4, "currenc": 4, "spend": 4, "monetari": 4, "contract": 4, "logist": 4, "instabl": [4, 7], "inabl": 4, "financ": [4, 7], "insolv": 4, "counterparti": 4, "debt": 4, "liquid": [4, 6], "fair": [4, 7], "instrument": 4, "polit": [4, 7], "disput": 4, "geopolit": 4, "tension": [4, 7], "terror": 4, "accid": 4, "interrupt": 4, "npolit": 4, "whole": 4, "outsourc": 4, "korea": 4, "vietnam": 4, "restrict": [4, 7, 8], "tariff": 4, "export": 4, "portion": 4, "revenu": [4, 6, 8], "raw": [4, 7, 8], "restructur": 4, "ceas": 4, "disrupt": [4, 6], "escal": [4, 6, 7], "nmani": 4, "prone": [4, 7], "earthquak": 4, "climat": 4, "weather": 4, "plant": 4, "terrorist": [4, 7], "attack": [4, 7], "hostil": 4, "ransomwar": 4, "cybersecur": [4, 7], "labor": 4, "beyond": [4, 7], "nsuch": 4, "imposs": 4, "slowdown": 4, "outag": 4, "neg": [4, 7, 8], "pandem": 4, "covid": 4, "19": [4, 7], "economi": 4, "imposit": 4, "stringent": [4, 5, 7], "travel": 4, "freight": 4, "movement": 4, "ramp": 4, "nfollow": 4, "expenditur": 4, "resum": 4, "exacerb": 4, "insur": 4, "nglobal": 4, "unabl": 4, "assur": [4, 7], "minor": [4, 7], "naddition": 4, "intensifi": 4, "seamlessli": [4, 6], "nto": 4, "stimul": 4, "ndue": 4, "upgrad": 4, "quantiti": 4, "defect": 4, "defici": 4, "supersed": 4, "nsubstanti": 4, "transport": 4, "provis": 4, "reimburs": 4, "warranti": 4, "unanticip": 4, "liabil": 4, "final": [4, 5, 6, 7, 8], "finish": [4, 7], "destin": 4, "made": [4, 6, 8], "prepay": 4, "termin": [4, 5], "recover": 4, "exposur": [4, 7], "nfutur": 4, "semiconductor": 4, "suffer": [4, 7], "constrain": [4, 5, 6, 8], "shipment": 4, "unexpectedli": 4, "interfer": 4, "unsaf": [4, 7], "expos": [4, 5, 7], "widespread": [4, 7], "vulner": [4, 5, 7], "compromis": [4, 7], "claim": [4, 7], "intang": 4, "fine": [4, 5, 7, 8], "lost": [4, 6, 7], "cancel": 4, "obsolet": 4, "exce": [4, 7], "realiz": 4, "accru": 4, "excess": 4, "impair": 4, "whenev": 4, "circumst": 4, "amount": [4, 6, 7, 8], "carri": [4, 8], "incur": 4, "unpredict": [4, 7, 8], "pace": [4, 7], "obsolesc": 4, "forecast": [4, 7], "incorrectli": [4, 7, 8], "extens": [4, 5, 6, 8], "issuanc": 4, "unknowingli": [4, 7], "notifi": 4, "preclud": 4, "bui": 4, "percept": 4, "android": 4, "playstat": 4, "nintendo": 4, "xbox": 4, "inclin": 4, "devot": 4, "compel": [4, 8], "dissatisfi": 4, "vast": [4, 7], "storefront": 4, "safari": 4, "union": [4, 7], "eu": [4, 7], "dma": 4, "narrow": [4, 7], "scope": [4, 6, 7], "elimin": 4, "nfailur": 4, "appeal": 4, "subscrib": 4, "nsome": 4, "manner": [4, 6, 7, 8], "nurtur": 4, "nmuch": 4, "chief": 4, "silicon": 4, "vallei": 4, "constantli": 4, "driver": 4, "recruit": 4, "subsidi": 4, "staf": 4, "contractor": 4, "placement": 4, "increment": 4, "weaken": 4, "telecommun": 4, "war": 4, "virus": 4, "ins": 4, "incid": [4, 7], "redund": 4, "ineffect": 4, "thing": [4, 8], "interf": 4, "imped": 4, "ship": 4, "nloss": 4, "unauthor": [4, 7], "confidenti": [4, 5], "encrypt": 4, "But": [4, 7, 8], "behalf": 4, "normal": [4, 7, 8], "investig": [4, 7], "penalti": [4, 5], "frequenc": [4, 5, 6, 7], "actor": [4, 7], "circumv": [4, 6, 7], "obfusc": 4, "forens": 4, "hinder": [4, 8], "recov": 4, "perpetr": 4, "profil": 4, "authent": 4, "hack": [4, 7], "malfeas": 4, "faulti": 4, "password": 4, "irregular": 4, "fraudul": 4, "induc": 4, "disclos": [4, 6, 8], "usernam": 4, "turn": [4, 7], "multifactor": 4, "unusu": 4, "freez": 4, "suspici": 4, "nwhile": 4, "ninvest": 4, "ongo": 4, "contempl": 4, "endeavor": 4, "distract": 4, "tangibl": 4, "approv": 4, "oner": 4, "ventur": 4, "riski": 4, "leas": 4, "unfavor": 4, "arisen": 4, "ordinari": 4, "resolv": [4, 5, 7], "sometim": [4, 8], "indemnif": 4, "indemnifi": 4, "alleg": 4, "magnitud": 4, "assert": 4, "royalti": 4, "vigor": 4, "defend": 4, "court": 4, "internation": 4, "plaintiff": 4, "injunct": 4, "relief": 4, "nregardless": 4, "merit": 4, "recognit": [4, 7], "settl": 4, "uncertain": 4, "disgorg": 4, "remedi": [4, 7], "worldwid": 4, "antitrust": 4, "bill": 4, "commerc": 4, "mobil": [4, 5, 8], "televis": 4, "film": 4, "anticorrupt": 4, "cash": [4, 6], "repatri": 4, "launder": 4, "tax": 4, "wast": 4, "recycl": 4, "ncomplianc": 4, "impos": [4, 7, 8], "agent": [4, 5, 7], "nregulatori": 4, "ban": [4, 7], "nexpect": 4, "increasingli": [4, 7, 8], "greenhous": 4, "ga": 4, "emiss": 4, "civil": 4, "disagre": 4, "perceiv": 4, "feder": 4, "nfrom": 4, "noncompli": 4, "individu": [4, 5, 6, 7], "lawsuit": 4, "monopol": 4, "nfurther": 4, "earn": 4, "search": [4, 5, 7], "nthere": 4, "retent": 4, "transfer": 4, "pass": [4, 7, 8], "pend": 4, "inquiri": [4, 7], "government": 4, "entiti": [4, 7, 8], "biometr": 4, "notif": 4, "permit": [4, 8], "healthcar": 4, "liabl": 4, "investigatori": 4, "cardhold": 4, "compress": [4, 5, 6], "acquir": 4, "extent": [4, 7], "unexpect": [4, 7, 8], "dollar": 4, "denomin": 4, "offset": 4, "strengthen": [4, 7], "nconvers": 4, "thu": 4, "hedg": 4, "deterior": 4, "sovereign": 4, "heighten": [4, 7], "worsen": 4, "A": [4, 5, 6, 7, 8], "collater": 4, "bank": 4, "unsecur": 4, "subassembli": 4, "assembl": 4, "legisl": 4, "ireland": [4, 7], "singapor": 4, "organis": 4, "statutori": 4, "valuat": 4, "defer": 4, "bodi": [4, 7], "adequaci": 4, "ow": 4, "ngener": 4, "volum": [4, 5, 6, 7], "repurchas": 4, "dividend": 4, "consumm": 4, "declar": 4, "board": [4, 7], "unresolv": 4, "nnone": 4, "threat": [4, 7], "postur": 4, "25": [4, 7], "2016": 4, "coordin": [4, 7], "track": [4, 7], "committe": [4, 7], "oversight": [4, 7], "counsel": 4, "chair": 4, "headquart": 4, "cupertino": [4, 8], "center": [4, 7, 8], "formal": [4, 7, 8], "conclud": 4, "uninstal": 4, "web": [4, 5, 7], "browser": 4, "june": 4, "contractu": 4, "desist": 4, "stai": [4, 5], "grant": 4, "ndepart": 4, "justic": 4, "depart": [4, 7], "doj": 4, "district": 4, "attornei": 4, "jersei": 4, "redress": [4, 7], "anticompetit": 4, "nonmonetari": 4, "defens": [4, 7], "nepic": 4, "epic": 4, "northern": 4, "unfair": [4, 7], "enjoin": 4, "extern": [4, 5, 7], "link": 4, "januari": 4, "motion": 4, "oppos": [4, 7], "30": [4, 5, 7], "vacat": 4, "fourth": 4, "mine": 4, "nnot": 4, "aapl": 4, "nholder": 4, "na": [4, 7], "301": 4, "npurchas": 4, "nshare": 4, "nperiod": 4, "ttotal": 4, "taverag": 4, "npaid": 4, "nannounc": 4, "napproxim": 4, "That": [4, 7, 8], "Be": [4, 5, 7], "nunder": 4, "njune": 4, "august": [4, 7], "nopen": 4, "negoti": [4, 7], "t35": 4, "697": 4, "t224": 4, "naugust": 4, "31": 4, "t42": 4, "910": 4, "t221": 4, "39": 4, "nseptemb": 4, "t33": 4, "653": 4, "t222": 4, "86": 4, "ntotal": [4, 7], "t112": 4, "260": 4, "t89": 4, "074": 4, "110": 4, "billion": 4, "10b5": 4, "graph": 4, "cumul": 4, "reinvest": 4, "dow": 4, "supersector": 4, "27": [4, 7], "2019": 4, "n2218": 4, "tseptemb": 4, "t100": 4, "t207": 4, "t273": 4, "t281": 4, "t322": 4, "t430": 4, "t113": 4, "t156": 4, "t131": 4, "t155": 4, "t210": 4, "ndow": 4, "t146": 4, "t216": 4, "t215": 4, "nfirst": 4, "nsecond": 4, "nthird": 4, "sequoia": 4, "nfourth": 4, "plu": 4, "nfiscal": 4, "six": 4, "realign": 4, "span": [4, 7], "indirectli": 4, "n2024": 4, "tchang": 4, "t2023": 4, "t2022": 4, "namerica": 4, "t167": 4, "045": 4, "t3": 4, "t162": 4, "560": 4, "t169": 4, "658": 4, "neurop": 4, "t101": 4, "328": 4, "t7": 4, "294": 4, "t95": 4, "118": 4, "ngreater": 4, "t66": 4, "952": 4, "t72": 4, "559": 4, "t74": 4, "njapan": 4, "t25": 4, "052": 4, "t24": 4, "257": 4, "977": 4, "nrest": 4, "t30": 4, "t4": 4, "t29": 4, "615": 4, "t1": 4, "t391": 4, "035": 4, "t2": 4, "t383": 4, "285": 4, "t394": 4, "weak": [4, 7], "renminbi": 4, "yen": [4, 8], "t201": 4, "183": 4, "t200": 4, "583": 4, "t205": 4, "489": 4, "984": 4, "357": 4, "t40": 4, "177": [4, 7], "t26": 4, "694": 4, "t28": 4, "300": [4, 6], "292": 4, "t37": 4, "005": 4, "t39": 4, "845": [4, 7], "t41": 4, "241": 4, "n96": 4, "169": 4, "t13": 4, "t85": 4, "t9": 4, "t78": 4, "129": [4, 7], "amort": 4, "bundl": 4, "flat": 4, "ngross": 4, "t109": 4, "633": 4, "t108": 4, "803": 4, "t114": 4, "728": 4, "t71": 4, "t60": 4, "345": 4, "t56": 4, "054": 4, "t180": 4, "683": 4, "148": 4, "t170": 4, "782": 4, "t36": 4, "t73": 4, "t70": 4, "t46": 4, "t44": 4, "t43": 4, "noper": 4, "t31": 4, "370": 4, "t5": 4, "915": 4, "t14": 4, "251": 4, "npercentag": 4, "t8": 4, "nsell": 4, "administr": 4, "097": 4, "932": 4, "094": 4, "t6": 4, "t57": 4, "467": 4, "t54": 4, "847": 4, "t51": 4, "t15": 4, "headcount": 4, "nprovis": 4, "749": 4, "t16": 4, "741": 4, "t19": 4, "neffect": 4, "nstatutori": 4, "t21": 4, "aid": [4, 7], "nliquid": 4, "unrestrict": 4, "140": 4, "ndebt": 4, "97": [4, 7], "payabl": 4, "promissori": 4, "nleas": 4, "space": [4, 7], "nmanufactur": 4, "noncancel": 4, "ndeem": 4, "tcja": 4, "nstate": 4, "fund": 4, "escrow": 4, "ncapit": 4, "95": [4, 7], "nrecent": 4, "pronounc": 4, "nincom": 4, "fasb": 4, "asu": 4, "09": [4, 6, 7], "740": 4, "reconcili": 4, "reconcil": [4, 8], "disaggreg": 4, "prospect": 4, "novemb": [4, 7], "07": [4, 6, 7, 8], "280": 4, "maker": 4, "codm": 4, "alloc": [4, 5, 7], "retrospect": 4, "ncritic": 4, "conform": [4, 8], "gaap": 4, "nuncertain": 4, "domest": 4, "taxat": 4, "resolut": 4, "conting": 4, "26": 4, "ninterest": 4, "forth": 4, "hypothet": 4, "nsensit": 4, "nhypothet": 4, "nrate": 4, "npotenti": 4, "n100": 4, "tenor": 4, "ndeclin": 4, "755": 4, "089": 4, "nterm": 4, "nincreas": 4, "t139": 4, "t194": 4, "nforeign": 4, "var": 4, "mont": 4, "carlo": 4, "interv": 4, "538": 4, "669": 4, "nindex": 4, "tpage": 4, "nconsolid": 4, "n29": 4, "n30": 4, "sheet": 4, "n31": 4, "n32": 4, "n33": 4, "nnote": 4, "n34": 4, "nreport": 4, "n48": 4, "nall": 4, "omit": [4, 8], "submiss": 4, "nyear": 4, "n2023": 4, "n2022": 4, "nnet": 4, "t294": 4, "866": 4, "t298": 4, "085": 4, "t316": 4, "199": 4, "t96": 4, "ncost": 4, "t185": 4, "233": 4, "t189": 4, "282": 4, "471": 4, "119": 4, "855": 4, "t22": 4, "075": 4, "352": 4, "t214": 4, "137": 4, "t223": 4, "546": 4, "t123": 4, "216": 4, "t119": 4, "437": 4, "t269": 4, "565": 4, "334": 4, "485": 4, "736": 4, "103": 4, "t93": 4, "995": 4, "t99": 4, "nearn": 4, "nbasic": 4, "ndilut": 4, "08": [4, 8], "343": [4, 7], "783": 4, "744": 4, "215": 4, "963": 4, "095": 4, "812": 4, "547": 4, "325": 4, "819": 4, "nsee": 4, "translat": [4, 7], "t395": 4, "765": 4, "511": 4, "unreal": 4, "832": 4, "t323": 4, "212": 4, "nadjust": 4, "337": 4, "717": 4, "394": 4, "138": 4, "850": 4, "563": 4, "104": 4, "t204": 4, "t253": 4, "816": 4, "899": 4, "272": 4, "t98": 4, "016": 4, "652": 4, "t88": 4, "531": 4, "nasset": 4, "ncurrent": 4, "ncash": 4, "943": 4, "965": 4, "228": 4, "590": 4, "naccount": 4, "410": 4, "508": 4, "nvendor": 4, "t32": 4, "833": 4, "477": 4, "ninventori": 4, "286": 4, "331": 4, "287": 4, "695": 4, "t152": 4, "987": 4, "t143": 4, "566": 4, "t91": 4, "479": 4, "544": 4, "t45": 4, "680": 4, "715": 4, "834": 4, "t64": 4, "758": 4, "t211": 4, "993": 4, "t209": 4, "017": 4, "t364": 4, "980": [4, 7], "t352": 4, "nliabil": 4, "t68": 4, "960": 4, "t62": 4, "611": 4, "304": 4, "t58": 4, "829": 4, "ndefer": 4, "249": 4, "061": 4, "ncommerci": 4, "967": 4, "985": 4, "t10": 4, "912": 4, "822": 4, "t176": 4, "392": 4, "t145": 4, "308": 4, "750": 4, "888": 4, "t49": 4, "848": 4, "638": 4, "t308": 4, "030": 4, "t290": 4, "ncommit": 4, "nsharehold": 4, "400": 4, "116": 4, "786": 4, "550": 4, "n83": 4, "276": 4, "naccumul": 4, "deficit": 4, "154": 4, "214": 4, "172": 4, "452": 4, "950": 4, "146": [4, 7], "t50": 4, "672": 4, "t63": 4, "090": 4, "nbegin": 4, "849": 4, "365": 4, "423": 4, "346": 4, "175": 4, "withheld": 4, "settlement": 4, "521": 4, "971": 4, "t12": 4, "034": 4, "t11": 4, "nend": 4, "t83": 4, "nretain": 4, "068": 4, "562": 4, "ndividend": 4, "218": 4, "793": 4, "612": 4, "099": 4, "454": 4, "846": 4, "77": 4, "046": 4, "186": 4, "109": 4, "t163": 4, "rsu": 4, "t0": 4, "98": 4, "94": [4, 7], "32": [4, 5], "737": 4, "929": 4, "ndepreci": 4, "445": 4, "519": 4, "688": 4, "038": 4, "266": 4, "227": 4, "006": 4, "788": 4, "356": 4, "271": 4, "520": 4, "618": 4, "484": 4, "731": 4, "684": 4, "499": 4, "020": 4, "889": 4, "448": 4, "552": 4, "031": 4, "t118": 4, "254": 4, "t110": 4, "543": 4, "t122": 4, "151": 4, "48": [4, 5], "656": 4, "513": 4, "76": [4, 7], "923": 4, "nproce": 4, "211": 4, "686": 4, "917": 4, "135": 4, "828": 4, "446": 4, "447": 4, "959": 4, "708": 4, "086": 4, "935": 4, "705": 4, "354": 4, "nfinanc": 4, "441": 4, "431": 4, "223": [4, 7], "234": [4, 7], "025": 4, "841": 4, "nrepurchas": 4, "949": 4, "89": [4, 7], "402": 4, "465": 4, "nrepay": 4, "958": 4, "repay": 4, "978": 4, "955": 4, "361": 4, "581": 4, "160": 4, "121": 4, "983": 4, "488": 4, "794": 4, "760": 4, "nsupplement": 4, "102": 4, "t18": 4, "679": 4, "573": 4, "33": [4, 7], "nbasi": 4, "prior": [4, 7], "reclassifi": 4, "nrevenu": 4, "remit": [4, 7], "straight": 4, "vest": 4, "sold": 4, "nderiv": 4, "nonleas": 4, "34": [4, 7], "entitl": 4, "commenc": 4, "deliveri": 4, "stand": 4, "ssp": 4, "icloud": 4, "siri": 4, "discount": 4, "undeliv": 4, "unbil": 4, "n26": 4, "n37": 4, "proport": 4, "moder": [4, 5], "64": [4, 7], "dilut": 4, "nnumer": 4, "ndenomin": 4, "nweight": 4, "312": 4, "316": 4, "856": 4, "antidilut": 4, "tunreal": 4, "ngain": 4, "tfair": 4, "nvalu": 4, "tcash": 4, "nequival": 4, "tcurrent": 4, "tnon": 4, "t27": 4, "nlevel": 4, "nmonei": 4, "t778": 4, "nmutual": 4, "n515": 4, "t105": 4, "t617": 4, "nsubtot": 4, "293": 4, "395": 4, "nu": 4, "treasuri": 4, "516": 4, "t212": 4, "087": 4, "380": 4, "159": 4, "t703": 4, "t17": 4, "568": 4, "158": 4, "810": 4, "ncertif": 4, "deposit": 4, "t873": 4, "t387": 4, "t478": 4, "066": 4, "ncorpor": 4, "t65": 4, "622": 4, "t270": 4, "953": 4, "939": 4, "027": 4, "t47": 4, "886": 4, "nmunicip": 4, "t412": 4, "t405": 4, "t190": 4, "nmortgag": 4, "595": 4, "t175": 4, "403": 4, "t23": 4, "367": 4, "278": [4, 7], "t132": 4, "t583": 4, "635": 4, "t128": 4, "056": 4, "966": 4, "t34": 4, "t160": 4, "t688": 4, "650": 4, "36": [4, 7], "359": [4, 7], "t481": 4, "n442": 4, "t428": 4, "t923": 4, "t909": 4, "406": 4, "114": 4, "468": 4, "136": 4, "t271": 4, "533": 4, "048": 4, "491": 4, "332": 4, "t320": 4, "t608": 4, "t76": 4, "840": 4, "956": 4, "890": 4, "t20": 4, "627": 4, "243": 4, "t628": 4, "t602": 4, "t192": 4, "t410": 4, "735": 4, "636": 4, "t344": 4, "t144": 4, "470": 4, "657": 4, "831": 4, "125": 4, "162": 4, "t173": 4, "752": 4, "corrobor": 4, "mortgag": 4, "classifi": [4, 7], "37": [4, 7], "swap": 4, "remeasur": 4, "notion": 4, "069": 4, "730": 4, "575": 4, "493": 4, "t104": 4, "777": 4, "nhedg": 4, "433": 4, "505": 4, "247": [4, 7], "ntrade": 4, "41": [4, 7], "44": [4, 7], "depreci": 4, "nland": 4, "690": 4, "nmachineri": 4, "t80": 4, "205": [4, 5], "314": 4, "nleasehold": 4, "839": 4, "599": 4, "73": [4, 7], "70": 4, "884": 4, "852": 4, "t55": 4, "906": 4, "601": 4, "703": 4, "010": 4, "457": 4, "634": 4, "391": 4, "neuropean": 4, "opinion": [4, 7], "1991": 4, "2007": 4, "irish": 4, "branch": 4, "2003": 4, "2014": 4, "2015": 4, "minist": 4, "juli": [4, 7], "annul": 4, "ecj": 4, "hear": 4, "asid": 4, "confirm": 4, "unrecogn": 4, "nfeder": 4, "571": 4, "080": 4, "644": 4, "265": 4, "801": 4, "726": 4, "570": 4, "298": 4, "49": [4, 7], "t84": 4, "428": 4, "603": 4, "483": [4, 7], "t347": 4, "t669": 4, "076": 4, "830": 4, "419": 4, "072": 4, "pretax": 4, "72": [4, 7], "71": 4, "ncomput": 4, "885": 4, "012": 4, "124": 4, "518": 4, "nimpact": 4, "246": 4, "311": 4, "366": 4, "397": 4, "nexcess": 4, "893": 4, "871": 4, "192": [4, 7], "739": 4, "ntax": 4, "carryforward": 4, "302": 4, "naccru": 4, "413": [4, 7], "421": 4, "nunreal": 4, "173": 4, "168": 4, "873": 4, "743": 4, "nless": 4, "374": 4, "007": 4, "369": 4, "551": 4, "998": 4, "nright": 4, "179": 4, "nminimum": 4, "674": 4, "940": 4, "t511": 4, "t455": 4, "t490": 4, "805": 4, "202": 4, "indefinit": 4, "temporari": 4, "727": 4, "044": 4, "284": 4, "ndecreas": 4, "386": 4, "463": 4, "982": 4, "542": 4, "936": 4, "070": 4, "expir": 4, "statut": 4, "229": 4, "494": 4, "closur": 4, "intercompani": 4, "exceed": [4, 7], "multiyear": 4, "exercis": 4, "noncash": 4, "rou": 4, "tfinanci": 4, "t2024": 4, "tother": 4, "661": 4, "tproperti": 4, "015": 4, "303": 4, "676": 4, "t165": 4, "t752": 4, "t859": 4, "430": 4, "842": [4, 7], "tfinanc": 4, "n2025": 4, "820": 4, "t171": 4, "991": 4, "n2026": 4, "914": 4, "n2027": 4, "t59": 4, "733": 4, "n2028": 4, "360": 4, "t38": 4, "398": 4, "n2029": 4, "187": 4, "nthereaft": 4, "t837": 4, "undiscount": 4, "790": 4, "imput": 4, "376": 4, "534": 4, "t896": 4, "borrow": 4, "proce": 4, "nine": [4, 7], "nmatur": 4, "333": 4, "264": 4, "948": 4, "645": 4, "309": 4, "arrear": 4, "namount": 4, "n2013": 4, "nfix": 4, "2062": 4, "t97": 4, "341": 4, "03": 4, "65": [4, 7], "t106": 4, "572": 4, "n97": 4, "nunamort": 4, "premium": 4, "321": 4, "358": 4, "113": 4, "662": 4, "930": 4, "342": 4, "800": 4, "180": 4, "88": 4, "ndure": 4, "425": 4, "426": 4, "372": 4, "589": 4, "055": 4, "appreci": 4, "four": [4, 7], "holder": 4, "n2014": 4, "bonu": 4, "nrestrict": 4, "nnumber": 4, "nrsu": 4, "ngrant": 4, "naggreg": 4, "nfair": 4, "nbalanc": 4, "t240": 4, "427": [4, 7], "t75": 4, "t150": 4, "861": 4, "501": 4, "768": 4, "87": [4, 7], "101": [4, 7], "878": 4, "144": 4, "t127": 4, "t135": 4, "91": [4, 7], "456": 4, "78": [4, 7], "59": [4, 7], "t140": 4, "326": 4, "t158": 4, "204": 4, "350": 4, "002": [4, 6], "nuncondit": 4, "uncondit": 4, "206": 4, "440": 4, "156": 4, "t633": 4, "t670": 4, "226": 4, "45": 4, "nconting": 4, "accrual": 4, "nconcentr": 4, "attribut": [4, 7, 8], "46": 4, "t67": 4, "098": 4, "082": 4, "062": 4, "569": 4, "895": 4, "458": 4, "207": 4, "nonrecur": 4, "t142": 4, "196": 4, "t138": 4, "t147": 4, "859": 4, "nchina": 4, "n66": 4, "t181": 4, "887": 4, "t172": 4, "269": 4, "nlong": 4, "664": 4, "797": 4, "778": 4, "219": 4, "47": [4, 7], "nopinion": 4, "nwe": 4, "fairli": 4, "pcaob": 4, "sponsor": 4, "treadwai": 4, "2013": 4, "unqualifi": 4, "thereon": 4, "nthese": 4, "misstat": 4, "fraud": [4, 7], "ndescript": 4, "naudit": 4, "nhow": 4, "nmatter": 4, "qualifi": 4, "letter": 4, "advisor": 4, "ernst": 4, "llp": 4, "auditor": 4, "2009": 4, "nsan": 4, "jose": 4, "nnovemb": 4, "coso": 4, "nour": 4, "ndefinit": 4, "mainten": [4, 7], "disposit": 4, "receipt": 4, "nevalu": 4, "nbase": 4, "supervis": [4, 7], "13a": 4, "15d": 4, "ninher": 4, "met": [4, 7], "paragraph": 4, "51": [4, 7, 8], "ninsid": 4, "deirdr": 4, "brien": 4, "vice": 4, "presid": 4, "affirm": 4, "april": 4, "withhold": 4, "remitt": 4, "mr": 4, "copi": [4, 6], "solicit": 4, "00042": 4, "nincorpor": 4, "texhibit": 4, "descript": [4, 5, 7, 8], "tform": 4, "tfile": 4, "nrestat": 4, "namend": 4, "bylaw": 4, "nindentur": 4, "york": [4, 8], "mellon": 4, "truste": 4, "noffic": 4, "certif": 4, "2018": 4, "85": [4, 5, 7], "2043": 4, "05": 4, "2044": 4, "februari": 4, "55": 4, "2045": 4, "900": 4, "700": 4, "60": [4, 7], "250": [4, 7], "2036": 4, "2046": 4, "450": 4, "2047": 4, "2049": 4, "2030": 4, "2050": 4, "2060": 4, "2028": 4, "2041": 4, "2051": 4, "2061": 4, "2032": 4, "2052": 4, "54": 4, "2033": 4, "2053": 4, "ceo": 4, "n12": 4, "nsubsidiari": 4, "n23": 4, "nconsent": 4, "n24": 4, "npower": 4, "signatur": 4, "nrule": 4, "nsection": 4, "1350": 4, "n101": 4, "ninlin": 4, "xbrl": 4, "n104": 4, "inlin": 4, "compensatori": 4, "herewith": 4, "furnish": 4, "herebi": 4, "undertak": 4, "56": [4, 5, 7], "nsignatur": 4, "npursuant": 4, "duli": 4, "undersign": 4, "thereunto": 4, "ndate": 4, "nby": 4, "luca": [4, 8], "maestri": 4, "nluca": 4, "nsenior": 4, "nchief": 4, "nknow": 4, "THESE": 4, "appoint": 4, "cook": 4, "jointli": 4, "her": 4, "substitut": 4, "him": 4, "thereto": 4, "therewith": 4, "ratifi": 4, "done": [4, 7, 8], "virtu": 4, "hereof": 4, "nname": 4, "ttitl": 4, "tdate": 4, "tchief": 4, "tnovemb": 4, "ntimothi": 4, "tsenior": 4, "kondo": 4, "nchri": 4, "wanda": 4, "austin": 4, "nwanda": 4, "gorski": 4, "tdirector": 4, "nalex": 4, "jung": 4, "nandrea": 4, "arthur": 4, "levinson": 4, "narthur": 4, "monica": 4, "lozano": 4, "nmonica": 4, "ronald": 4, "sugar": 4, "nronald": 4, "susan": 4, "wagner": 4, "nsusan": 4, "57": 4, "turbo": [4, 5, 6, 8], "outlin": [4, 5, 7], "invdestacksmeticsisdict": 4, "setispect": 4, "20cyan": 4, "evaluationseld": 4, "anvis": 4, "droitent": 4, "discernminerv": 4, "versbobprefvers": 4, "vo\u8be5": 4, "option\u548c": 4, "meio": 4, "\u0432\u0440\u0435\u043ccisco": 4, "dellaischenpoihscap": 4, "geme": 4, "gettim": 4, "unscal": 4, "vocabulari": [4, 8], "closer": 4, "sharpen": 4, "uniform": 4, "raschka": 4, "repetit": [4, 6, 8], "radic": 4, "grappl": 4, "safer": [4, 7], "fascin": 4, "spontan": 4, "answer": [4, 5, 6, 7, 8], "aren": [4, 5], "linear": 4, "absent": [4, 7], "coax": 4, "journei": 4, "suddenli": 4, "manifest": 4, "deliber": [4, 7], "contend": 4, "70b": 4, "rethink": 4, "tutor": 4, "children": [4, 7], "verifi": [4, 8], "predefin": [4, 8], "weren": 4, "kind": 4, "usual": 4, "quantif": 4, "contamin": [4, 7], "massiv": [4, 7], "unseen": [4, 7], "longitudin": 4, "mostli": [4, 8], "versu": [4, 5, 7], "latter": 4, "tailor": [4, 7], "great": [4, 5, 8], "cognit": 4, "misinform": [4, 7], "tempor": 4, "disclaim": 4, "referr": 4, "incorrect": [4, 7], "demograph": [4, 7], "stereotyp": [4, 7], "societ": [4, 7], "pii": [4, 7], "anonym": 4, "leakag": [4, 7], "carryov": 4, "multi": [4, 5, 7, 8], "fallaci": 4, "think": [4, 5, 7], "idiom": 4, "sarcasm": 4, "terminologi": 4, "lingual": 4, "misunderstand": 4, "syntax": 4, "scan": [4, 5], "compat": [4, 5, 8], "overconfid": 4, "clariti": [4, 6, 7, 8], "audienc": 4, "densiti": 4, "satisfact": [4, 8], "misus": [4, 7], "moral": 4, "co2": 4, "energi": 4, "consumpt": 4, "server": [4, 5, 8], "cach": [4, 5], "imag": [4, 5, 7], "audio": 4, "etc": [4, 8], "truth": [4, 7, 8], "layer": [4, 5, 6, 8], "palm": 4, "easi": [4, 5, 6, 7], "synthet": [4, 7, 8], "augment": [4, 5], "post": [4, 5, 7, 8], "timeout": 4, "variat": 4, "inter": 4, "rater": 4, "ti": 4, "holist": [4, 7], "fast": [4, 7, 8], "experiment": [4, 5, 8], "vi": 4, "categor": [4, 7, 8], "intrins": 4, "extrins": 4, "sequenc": [4, 8], "perplex": 4, "downstream": [4, 8], "synthesi": 4, "discret": 4, "prefix": [4, 7], "roug": 4, "bleu": 4, "bilingu": 4, "understudi": 4, "overlap": [4, 6], "favor": [4, 8], "breviti": 4, "insensit": 4, "semant": [4, 6], "orient": [4, 7], "gist": 4, "meteor": 4, "synonym": 4, "stem": [4, 8], "paraphras": 4, "alongsid": [4, 7], "computation": [4, 5, 6], "cider": 4, "consensu": 4, "tf": 4, "idf": 4, "caption": 4, "reliant": 4, "corpu": 4, "ter": 4, "edit": [4, 7], "hypothesi": 4, "penal": 4, "bertscor": 4, "contextu": [4, 7], "bert": 4, "spice": 4, "proposit": 4, "scene": [4, 7], "pure": [4, 5], "analyst": [4, 6], "rouge_1": 4, "rouge_2": 4, "ideal": [4, 5, 7, 8], "cheaper": 4, "setup": [4, 7, 8], "evaluate_summari": 4, "unigram": 4, "bigram": 4, "absl": 4, "py": 4, "rouge_scor": 4, "generated_summari": 4, "reference_summari": 4, "google_bleu": 4, "bleu_scor": 4, "rouge1": 4, "rouge2": 4, "arbitrari": 4, "chosen": [4, 7], "sentence1": 4, "cat": [4, 7], "sat": 4, "mat": 4, "sentence2": 4, "ate": 4, "3333333333333333": 4, "7272727272727272": 4, "4444444444444445": 4, "generate_summari": 4, "summir": 4, "liner": 4, "evaluate_summary_model": 4, "model_benchmark": 4, "models_test": 4, "benchmark_summari": 4, "model_summari": 4, "evaluation_result": 4, "statu": 4, "concis": 4, "element": [4, 7, 8], "verbos": [4, 5, 7], "peripher": 4, "quit": [4, 8], "convei": [4, 6], "breadth": 4, "Of": [4, 7], "vibe": 4, "visualize_prompt_comparison": 4, "matplotlib": 4, "radar": 4, "radar_plot": 4, "tmp": 4, "ipykernel_1652501": 4, "940173201": 4, "userwarn": 4, "figurecanvasagg": 4, "largest": 4, "granular": [4, 5, 6], "tune": [4, 5, 7, 8], "likert": 4, "ensembl": 4, "repeatedli": 4, "fluenci": 4, "refin": 4, "notabl": [4, 7, 8], "integ": 4, "rubric": 4, "hollist": 4, "judgeevalu": 4, "grammar": [4, 5, 8], "evaluate_with_llm": 4, "criterion": 4, "judge_model": 4, "candidate_summari": 4, "grammat": 4, "y": [4, 7, 8], "z": 4, "w": [4, 6, 7], "benchmark_model": 4, "test_model": 4, "input_text": [4, 6], "trillion": [4, 8], "evals_list": 4, "1775618912": 4, "variant": [4, 7], "slightli": 4, "drift": [4, 7], "lowest": 4, "firstli": 4, "overhead": [4, 5], "egocentr": 4, "tight": 4, "aproach": 4, "aplic": 4, "clearli": [4, 7, 8], "earlier": [4, 7], "depict": [4, 7, 8], "multilingu": [4, 7], "golden": 4, "languang": 4, "arena": 4, "randomli": 4, "customiz": [4, 5, 7], "irrelev": 4, "unhelp": [4, 7], "occasion": 4, "rare": 4, "perfectli": 4, "cater": 4, "critiqu": [4, 7], "elo": 4, "spectrum": 4, "exam": 4, "probe": [4, 7], "certifi": 4, "began": 4, "glue": 4, "entail": 4, "baselin": [4, 7], "superglu": 4, "successor": 4, "grew": 4, "big": 4, "bench": 4, "srivastava": 4, "arithmet": 4, "truthfulqa": 4, "multitask": 4, "hendryck": [4, 7], "multidisciplinari": 4, "stanford": 4, "helm": 4, "multidimension": 4, "surround": [4, 7, 8], "humanev": 4, "lmsy": 4, "brought": 4, "dialogu": 4, "chiang": 4, "gather": 4, "alpacaev": 4, "duboi": 4, "mt": 4, "argilla": 4, "mila": 4, "mit": [4, 5], "contributor": [4, 5, 8], "western": 4, "centric": 4, "divid": [4, 6, 7], "subset": [4, 7], "agnost": 4, "dialect": 4, "render": [4, 7], "crowdsourc": 4, "livebench": 4, "white": [4, 7], "resili": [4, 7], "meaningfulli": 4, "zebralog": 4, "grid": 4, "puzzl": 4, "brailsford": 4, "1999": 4, "lsat": 4, "hous": 4, "clue": 4, "deduct": 4, "arriv": 4, "programmat": [4, 8], "2x2": 4, "6x6": 4, "shot": [4, 7], "reductio": 4, "ad": [4, 7, 8], "absurdum": 4, "hard": 4, "10b": 4, "counterfactu": 4, "came": 4, "arc": 4, "prize": [4, 7], "chollet": 4, "mike": [4, 7], "knoop": 4, "founder": 4, "zapier": 4, "fran\u00e7oi": 4, "creator": 4, "agi": 4, "kera": 4, "genuin": 4, "possess": 4, "elementari": 4, "novelti": 4, "wouldn": 4, "interpol": 4, "synthes": 4, "fly": 4, "retriev": [4, 5], "brute": 4, "pixel": 4, "unbeaten": 4, "win": 4, "poorli": 4, "recombin": 4, "spur": [4, 7], "takeawai": 4, "vertic": [4, 7], "finbench": 4, "legalbench": 4, "guha": 4, "berkelei": 4, "bfcl": 4, "patil": 4, "fourrier": 4, "bespok": 4, "sdk": 4, "autoregress": 4, "sub": [4, 5], "liter": 4, "disturb": 4, "zero": [4, 5, 7, 8], "varianc": [4, 7], "yt": 4, "ut": 4, "suppos": [4, 8], "ol": 4, "heteroscedast": 4, "regress": 4, "bivari": 4, "evaluation_track": 4, "evaluationtrack": 4, "model_config": 4, "basemodelconfig": 4, "parallelismmanag": 4, "pipelineparamet": 4, "envconfig": 4, "is_accelerate_avail": 4, "datetim": 4, "timedelta": 4, "initprocessgroupkwarg": 4, "create_evaluation_pipelin": 4, "cache_dir": 4, "float16": 4, "max_sampl": 4, "kwargs_handl": 4, "3000": 4, "save_detail": 4, "pipeline_param": 4, "launcher_typ": 4, "env_config": 4, "override_batch_s": 4, "use_chat_templ": 4, "trust_remote_cod": 4, "pipeline_paramet": 4, "schemat": [4, 6], "vllm": [4, 8], "tgi": 4, "storag": [4, 5, 7], "num_few_shot": 4, "bar": 4, "bigbench": 4, "winogrand": 4, "hellaswag": 4, "nlp": [4, 5, 7], "save_and_push_result": 4, "show_result": 4, "model_arg": 4, "send": [4, 5, 7, 8], "serverless": 4, "inference_server_address": 4, "inference_server_auth": 4, "model_id": 4, "null": 4, "bash": [4, 5], "command": [4, 5], "model_config_path": 4, "endpoint_model": 4, "llama3": [4, 6], "qwen2": [4, 5, 8], "smollm2": 4, "3b": [4, 5], "alibaba": [4, 8], "5b": [4, 5, 8], "hui": 4, "allal": 4, "cluster": 4, "noteworthi": 4, "grain": [4, 8], "salt": [4, 8], "exponenti": 4, "modular": 4, "offici": 4, "revisit": 4, "trace": 4, "langchain_tracing_v2": 4, "langchain_api_kei": 4, "hf_evalu": 4, "langsmith_evalu": 4, "ls_client": 4, "dataset_nam": 4, "create_dataset": 4, "create_exampl": 4, "dataset_id": 4, "calculate_scor": 4, "reference_output": 4, "oai_client": 4, "xp_model_nam": 4, "lastli": 4, "run_evalu": 4, "And": [4, 5, 7], "upload_result": 4, "experiment_prefix": 4, "num_repetit": 4, "386a3620": 4, "9e1cc3cb": 4, "9d6a": 4, "4356": 4, "ab34": 4, "138e0abe8be4": 4, "8741976e": 4, "5268": 4, "4b75": 4, "949f": 4, "99477dde5d64": 4, "selectedsess": 4, "b831dc1e": 4, "90bc": 4, "4ed8": 4, "8080": [4, 5], "fb42444724d6": 4, "4it": 4, "latest": [4, 5, 6, 7, 8], "tobia": 4, "evaluate_modul": 4, "6fc70b7be0088120a372dfdd5d320b39b8bb3630cb8029b193941d9376e86bb0": 4, "tue": 4, "nov": 4, "couldn": 4, "5it": 4, "5053784e": 4, "64445871": 4, "a53c": 4, "44b1": 4, "a422": 4, "4f49b2f9656f": 4, "69": [4, 7], "4b29f3c9": 4, "9ef7e39a": 4, "2add": 4, "410c": 4, "89f8": 4, "9f1a8b198cf1": 4, "61": [4, 7], "insert": 4, "combined_df": 4, "concat": [4, 7], "ignore_index": [4, 7], "execution_tim": 4, "example_id": 4, "333333": 4, "224388": 4, "feb10f92": 4, "3167": 4, "41f3": 4, "bb1c": 4, "d271153a31a8": 4, "5b196b22": 4, "9f4c": 4, "489c": 4, "b020": 4, "7823208b42d6": 4, "348101": 4, "722464": 4, "c310f159": 4, "064a": 4, "4035": 4, "97c3": 4, "a25bbf43abc2": 4, "386076": 4, "704104": 4, "f7f24899": 4, "dd50": 4, "409e": 4, "93cc": 4, "6fb1622b60bf": 4, "443038": 4, "725059": 4, "242856d6": 4, "efb5": 4, "4101": 4, "b1cf": 4, "5805532838ac": 4, "373418": 4, "795302": 4, "ce975169": 4, "a0ab": 4, "40ce": 4, "8e32": 4, "efa28d06079d": 4, "stat": 4, "groupbi": [4, 7], "agg": [4, 7], "sort": 4, "sort_valu": 4, "subplot": 4, "pyplot": 4, "plt": 4, "numpi": 4, "np": 4, "ax1": 4, "ax2": 4, "figsiz": 4, "2ecc71": 4, "3498db": 4, "e74c3c": 4, "bleu_mean": 4, "bleu_std": 4, "enumer": [4, 6, 7], "errorbar": 4, "yerr": 4, "fmt": 4, "markers": 4, "capsiz": 4, "set_ylabel": 4, "set_titl": 4, "set_xtick": 4, "set_xticklabel": 4, "rotat": 4, "set_ylim": 4, "bottom": 4, "legend": 4, "exec_mean": 4, "exec_std": 4, "tight_layout": 4, "ndetail": 4, "4038": 4, "0453": 4, "7815": 4, "0433": 4, "3768": 4, "0424": 4, "8343": 4, "2208": 4, "3519": 4, "0775": 4, "9122": 4, "1482": 4, "377": 4, "042": 4, "078": 4, "slower": [4, 7], "04": [4, 6], "latenc": [4, 5, 6, 7], "speed": [4, 5, 7], "interestingli": 4, "decoupl": 4, "reload": 4, "facilit": [4, 7], "promptfooconfig": 4, "model_comparison": 4, "pretti": [4, 7], "dump": 4, "default_flow_styl": 4, "sort_kei": 4, "prompt1": 4, "defaulttest": 4, "1000m": 4, "millisecond": 4, "eval_data": 4, "latency_m": 4, "totallatencym": 4, "token_usag": 4, "tokenusag": 4, "assert_pass": 4, "assertpasscount": 4, "assert_fail": 4, "assertfailcount": 4, "prompt_token": [4, 5], "num_request": 4, "numrequest": 4, "2463": 4, "000035": 4, "3773": 4, "004620": 4, "1669": 4, "000091": 4, "1669m": 4, "highest": [4, 5], "3773m": 4, "00462": 4, "promptfool": 4, "manual": [4, 5, 7], "redefin": 4, "prompt_comparison": 4, "prompt2": 4, "prompt3": 4, "prompt_fil": 4, "prompt_cont": 4, "BE": 4, "again": 4, "prompt_id": 4, "promptid": 4, "gradingresult": 4, "df_raw": 4, "reset_index": [4, 7], "eas": [4, 5, 7], "seamless": [4, 7], "hf": 4, "plain": [4, 5], "vanilla": 4, "defi": 4, "accustom": 4, "legaci": 4, "unsustain": 4, "prd": 4, "cultiv": [4, 7], "organiz": 4, "stagnat": 4, "alb": 4, "loubna": 4, "anton": 4, "lozhkov": 4, "bakouch": 4, "gabriel": [4, 7], "mart\u00edn": [4, 7], "bl\u00e1zquez": 4, "lewi": 4, "tunstal": 4, "agust\u00edn": 4, "piquer": 4, "andr": 4, "marafioti": 4, "cyril": 4, "zakka": 4, "leandro": 4, "werra": 4, "wolf": 4, "are24": 4, "judgearena": 4, "bps99": 4, "salli": 4, "pott": 4, "barbara": 4, "557": [4, 7], "sciencedirect": 4, "s0377221798003646": 4, "doi": [4, 7, 8], "1016": 4, "s0377": 4, "2217": 4, "00364": 4, "ctj": 4, "jerri": [4, 7], "tworek": [4, 7], "heewoo": [4, 7], "jun": [4, 7], "qime": [4, 7], "henriqu": [4, 7], "pond": [4, 7], "de": [4, 7], "oliveira": [4, 7], "pinto": [4, 7], "harri": [4, 7], "yuri": 4, "burda": 4, "greg": [4, 7], "brockman": [4, 7], "raul": [4, 7], "puri": [4, 7], "gretchen": [4, 7], "krueger": [4, 7], "petrov": [4, 7], "heidi": 4, "khlaaf": 4, "girish": [4, 7], "sastri": [4, 7], "brook": [4, 7], "chan": [4, 7], "grai": [4, 7], "ryder": [4, 7], "mikhail": [4, 7], "pavlov": [4, 7], "alethea": [4, 7], "lukasz": 4, "kaiser": [4, 7], "mohammad": [4, 7], "bavarian": [4, 7], "clemen": [4, 7], "winter": [4, 7], "philipp": 4, "tillet": [4, 7], "felip": [4, 7], "petroski": [4, 7], "dave": [4, 7], "cum": [4, 7], "plappert": 4, "fotio": 4, "chantzi": [4, 7], "barn": 4, "ariel": 4, "herbert": 4, "voss": [4, 7], "hebgen": 4, "guss": 4, "nichol": 4, "paino": [4, 7], "nikola": [4, 7], "tezak": [4, 7], "babuschkin": [4, 7], "suchir": [4, 7], "balaji": [4, 7], "shantanu": [4, 7], "jain": [4, 7], "hess": [4, 7], "carr": 4, "josh": [4, 7], "achiam": [4, 7], "vedant": 4, "misra": 4, "evan": [4, 7], "morikawa": [4, 7], "matthew": 4, "knight": [4, 7], "mile": [4, 7], "brundag": [4, 7], "mira": [4, 7], "murati": [4, 7], "kati": [4, 7], "mayer": [4, 7], "bob": [4, 7, 8], "mcgrew": [4, 7], "ilya": [4, 7], "sutskev": [4, 7], "wojciech": [4, 7], "zaremba": [4, 7], "2107": 4, "03374": 4, "cz": 4, "lianmin": 4, "ying": 4, "sheng": 4, "anastasio": 4, "angelopoulo": 4, "tianl": 4, "dacheng": 4, "banghua": 4, "jordan": [4, 7], "gonzalez": 4, "ion": 4, "stoica": 4, "04132": 4, "cho24a": 4, "francoi": 4, "arcpriz": 4, "cho24b": 4, "dglh24": 4, "yann": 4, "bal\u00e1z": 4, "galambosi": 4, "tatsunori": 4, "hashimoto": 4, "debia": 4, "04475": 4, "fac24a": 4, "wiki": [4, 8], "fac24b": 4, "fac24c": 4, "model_doc": 4, "fac24d": 4, "cookbook": 4, "llm_judg": 4, "fac24f": 4, "fhwt23": 4, "cl\u00e9mentin": 4, "nathan": 4, "habib": 4, "gnh": 4, "julian": 4, "nyarko": 4, "ho": 4, "r\u00e9": 4, "adam": [4, 7], "chilton": 4, "aditya": [4, 7], "narayana": 4, "chohla": 4, "brandon": [4, 7, 8], "waldon": 4, "rockmor": 4, "diego": 4, "zambrano": 4, "dmitri": 4, "talisman": 4, "enam": 4, "hoqu": 4, "faiz": 4, "surani": 4, "frank": [4, 7], "fagan": 4, "galit": 4, "sarfati": 4, "gregori": 4, "dickinson": 4, "haggai": 4, "porat": 4, "hegland": 4, "jessica": [4, 7], "joe": [4, 7], "nudel": 4, "joel": [4, 7], "niklau": 4, "nai": 4, "jonathan": [4, 7], "choi": 4, "margaret": 4, "hagan": 4, "megan": 4, "ma": [4, 7], "livermor": 4, "nikon": 4, "rasumov": 4, "rahe": 4, "nil": 4, "holzenberg": 4, "noam": 4, "kolt": 4, "henderson": 4, "rehaag": 4, "sharad": 4, "shang": 4, "spencer": 4, "sunni": 4, "gandhi": 4, "zur": 4, "varun": 4, "iyer": 4, "zehua": 4, "2308": 4, "11462": 4, "hbb": 4, "collin": 4, "burn": 4, "steven": [4, 7], "basart": [4, 7], "zou": [4, 7], "manta": [4, 7], "mazeika": [4, 7], "03300": 4, "hbd": 4, "maxwel": 4, "forb": 4, "yejin": 4, "curiou": 4, "neural": [4, 8], "degener": 4, "1904": 4, "09751": 4, "hyc": 4, "binyuan": 4, "zeyu": 4, "cui": 4, "jiaxi": 4, "dayiheng": 4, "tianyu": 4, "jiajun": 4, "kai": [4, 7], "dang": 4, "coder": [4, 5], "preprint": [4, 8], "2409": [4, 7], "12186": 4, "lx": 4, "zhen": 4, "xiaohan": 4, "jia": 4, "yuxuan": 4, "lai": 4, "chongyang": 4, "shuai": 4, "nlg": 4, "07103": 4, "lbl": 4, "bommasani": 4, "toni": 4, "dimitri": 4, "tsipra": 4, "dilara": 4, "soylu": 4, "michihiro": 4, "yasunaga": 4, "yian": 4, "deepak": 4, "narayanan": 4, "yuhuai": 4, "newman": 4, "binhang": 4, "bobbi": 4, "ce": 4, "christian": [4, 7], "cosgrov": 4, "acosta": 4, "nava": [4, 7], "drew": 4, "hudson": 4, "zelikman": 4, "esin": 4, "durmu": 4, "faisal": 4, "ladhak": 4, "frieda": 4, "rong": 4, "hongyu": 4, "ren": 4, "huaxiu": 4, "yao": [4, 7], "jue": 4, "keshav": 4, "santhanam": 4, "laurel": 4, "lucia": 4, "mert": 4, "yuksekgonul": 4, "mirac": 4, "suzgun": 4, "niladri": 4, "chatterji": 4, "omar": 4, "khattab": 4, "qian": [4, 7], "chi": [4, 8], "sang": 4, "shibani": [4, 7], "santurkar": [4, 7], "surya": 4, "icard": 4, "tianyi": 4, "vishrav": 4, "chaudhari": 4, "xuechen": 4, "yuhui": 4, "yuta": 4, "koreeda": 4, "2211": 4, "09110": 4, "lbc24": 4, "ronan": 4, "bra": 4, "allenai": 4, "lhe22": [4, 7], "stephani": [4, 7], "owain": [4, 7], "mimic": [4, 7], "falsehood": [4, 7], "2109": [4, 7], "07958": [4, 7], "pzwg23": 4, "shishir": 4, "tianjun": 4, "xin": [4, 7], "gorilla": 4, "15334": 4, "pro24": 4, "dev": 4, "ras24": 4, "sebastian": 4, "scratch": 4, "1633437166": 4, "srf": 4, "shivalika": 4, "angelika": 4, "roman": [4, 7], "adelani": 4, "ngui": 4, "vila": 4, "suero": 4, "peerat": 4, "limkonchotiwat": 4, "kelli": 4, "marchisio": 4, "qi": 4, "leong": 4, "yosephin": 4, "susanto": 4, "raymond": [4, 7], "ng": [4, 7], "shayn": 4, "longpr": 4, "ko": 4, "madelin": 4, "antoin": 4, "bosselut": 4, "oh": 4, "leshem": 4, "choshen": 4, "daphn": 4, "ippolito": 4, "enzo": 4, "ferrant": 4, "marzieh": 4, "fadae": 4, "beyza": 4, "ermi": 4, "sara": 4, "hooker": 4, "linguist": [4, 7], "03304": 4, "srr": 4, "aarohi": 4, "abhinav": 4, "rastogi": 4, "abhishek": 4, "rao": 4, "abu": 4, "awal": 4, "shoeb": 4, "abubakar": 4, "abid": 4, "fisch": 4, "santoro": 4, "gupta": 4, "adri\u00e0": 4, "garriga": 4, "alonso": 4, "agnieszka": 4, "kluska": 4, "aitor": 4, "lewkowycz": 4, "akshat": 4, "warstadt": 4, "alexand": [4, 7, 8], "kocurek": 4, "ali": [4, 7], "safaya": 4, "tazarv": 4, "aman": 4, "hussain": 4, "dsouza": 4, "ambros": 4, "slone": 4, "ameet": 4, "rahan": 4, "anantharaman": 4, "ander": 4, "andreassen": 4, "madotto": 4, "santilli": 4, "stuhlm\u00fcller": 4, "la": 4, "lampinen": 4, "angelica": 4, "anh": 4, "vuong": 4, "animesh": 4, "gottardi": 4, "antonio": 4, "norelli": 4, "anu": 4, "venkatesh": 4, "arash": 4, "gholamidavoodi": 4, "arfa": 4, "tabassum": 4, "arul": 4, "menez": 4, "arun": [4, 7], "kirubarajan": 4, "asher": 4, "mullokandov": 4, "ashish": 4, "sabharw": 4, "herrick": 4, "avia": 4, "efrat": 4, "aykut": 4, "erdem": 4, "ayla": 4, "karaka\u015f": 4, "bao": [4, 7], "loe": 4, "barret": [4, 7], "zoph": [4, 7], "bart\u0142omiej": 4, "bojanowski": 4, "batuhan": 4, "\u00f6zyurt": 4, "behnam": 4, "hedayatnia": 4, "neyshabur": 4, "inden": 4, "benno": 4, "stein": 4, "berk": 4, "ekmekci": 4, "blake": 4, "howald": 4, "bryan": 4, "orinion": 4, "diao": 4, "dour": 4, "stinson": 4, "cedrick": 4, "argueta": 4, "c\u00e9sar": 4, "ferri": 4, "ram\u00edrez": 4, "chandan": 4, "charl": 4, "rathkopf": 4, "chenlin": 4, "meng": 4, "chitta": 4, "baral": 4, "chiyu": 4, "callison": 4, "burch": 4, "wait": [4, 7], "voigt": 4, "cindi": 4, "ramirez": 4, "clara": 4, "rivera": 4, "clemencia": 4, "siro": 4, "colin": 4, "raffel": 4, "courtnei": 4, "ashcraft": 4, "cristina": 4, "garbacea": 4, "damien": [4, 7], "sileo": 4, "garrett": 4, "kilman": 4, "freeman": 4, "khashabi": 4, "levi": [4, 7], "mosegu\u00ed": 4, "gonz\u00e1lez": 4, "perszyk": 4, "danqi": 4, "dar": 4, "gilboa": 4, "dohan": [4, 7], "drakard": 4, "jurgen": 4, "debajyoti": 4, "datta": 4, "deni": 4, "emelin": 4, "kleyko": 4, "deniz": 4, "yuret": 4, "derek": [4, 7], "tam": [4, 8], "dieuwk": 4, "hupk": 4, "diganta": 4, "dilyar": 4, "buzan": 4, "coelho": 4, "mollo": 4, "diyi": 4, "dylan": 4, "schrader": 4, "ekaterina": 4, "shutova": 4, "ekin": 4, "dogu": 4, "cubuk": 4, "elad": 4, "segal": 4, "eleanor": 4, "hagerman": 4, "donowai": 4, "elli": 4, "pavlick": 4, "rodola": 4, "emma": 4, "lam": 4, "chu": [4, 7], "erkut": 4, "erni": 4, "dyer": 4, "jerzak": 4, "eunic": 4, "engefu": 4, "manyasi": 4, "evgenii": 4, "zheltonozhskii": 4, "fanyu": 4, "xia": 4, "fatemeh": 4, "siar": 4, "fernando": 4, "mart\u00ednez": 4, "plume": 4, "francesca": 4, "happ\u00e9": 4, "gaurav": 4, "genta": 4, "indra": 4, "winata": 4, "gerard": 4, "melo": 4, "germ\u00e1n": 4, "kruszewski": 4, "giambattista": [4, 7], "parascandolo": [4, 7], "giorgio": 4, "mariani": 4, "gloria": 4, "gonzalo": 4, "jaimovitch": 4, "l\u00f3pez": 4, "gregor": 4, "betz": 4, "gui": [4, 5], "gur": 4, "hana": 4, "galijasev": 4, "rashkin": 4, "hannaneh": 4, "hajishirzi": 4, "harsh": 4, "hayden": 4, "bogar": 4, "henri": [4, 7], "shevlin": 4, "hinrich": 4, "sch\u00fctze": 4, "hiromu": 4, "yakura": 4, "hongm": 4, "hugh": 4, "mee": 4, "wong": [4, 7], "isaac": 4, "nobl": 4, "jaap": 4, "jumelet": 4, "geissing": 4, "jaehoon": 4, "jaim": 4, "fern\u00e1ndez": 4, "fisac": 4, "simon": 4, "koppel": 4, "koco\u0144": 4, "jana": 4, "thompson": [4, 7], "janel": 4, "wingfield": 4, "jarema": 4, "radom": 4, "jascha": 4, "sohl": [4, 7], "dickstein": 4, "phang": 4, "yosinski": 4, "jekaterina": 4, "novikova": 4, "jell": 4, "bosscher": 4, "jennif": 4, "marsh": 4, "jeroen": 4, "taal": 4, "engel": 4, "jesujoba": 4, "alabi": 4, "jiam": 4, "jillian": 4, "joan": 4, "waweru": 4, "burden": 4, "bali": 4, "batcheld": 4, "berant": 4, "j\u00f6rg": 4, "frohberg": 4, "jo": 4, "rozen": 4, "orallo": 4, "boudeman": 4, "guerr": 4, "tenenbaum": 4, "joyc": 4, "chua": 4, "kanclerz": 4, "karen": 4, "livescu": 4, "karl": 4, "krauth": 4, "karthik": 4, "gopalakrishnan": 4, "katerina": 4, "ignatyeva": 4, "katja": 4, "markert": 4, "kaustubh": 4, "dhole": 4, "gimpel": 4, "omondi": 4, "kori": 4, "mathewson": 4, "kristen": 4, "chiafullo": 4, "ksenia": 4, "shkaruta": 4, "shridhar": 4, "kyle": [4, 7], "mcdonel": 4, "richardson": 4, "laria": 4, "reynold": 4, "leo": [4, 7], "dugan": 4, "lianhui": 4, "lidia": 4, "contrera": 4, "ochando": 4, "morenc": 4, "moschella": 4, "luci": 4, "ludwig": 4, "schmidt": [4, 7], "luheng": 4, "olivero": 4, "col\u00f3n": 4, "metz": [4, 7], "l\u00fctfi": 4, "kerem": 4, "\u015fenel": 4, "maarten": [4, 7], "bosma": 4, "sap": [4, 7], "maartj": 4, "hoev": 4, "maheen": 4, "farooqi": 4, "manaal": 4, "faruqui": 4, "marco": 4, "baturan": 4, "marelli": 4, "maru": 4, "maria": 4, "quintana": 4, "tolkiehn": 4, "mario": [4, 7], "giulianelli": 4, "martha": 4, "potthast": 4, "leavitt": 4, "hagen": 4, "m\u00e1ty\u00e1": 4, "schubert": 4, "medina": [4, 7], "orduna": 4, "baitemirova": 4, "melodi": 4, "arnaud": 4, "melvin": 4, "mcelrath": 4, "yee": 4, "cohen": 4, "ivanitskii": 4, "starritt": 4, "strube": 4, "micha\u0142": 4, "sw\u0119drowski": 4, "michel": [4, 7], "bevilacqua": 4, "mihir": 4, "kale": 4, "cain": 4, "mime": 4, "mitch": 4, "walker": 4, "mo": 4, "tiwari": 4, "mohit": 4, "bansal": 4, "moin": 4, "aminnaseri": 4, "mor": 4, "geva": 4, "mozhdeh": 4, "gheini": 4, "mukund": 4, "varma": 4, "nanyun": 4, "peng": [4, 7], "nayeon": 4, "neta": 4, "krakov": 4, "doiron": 4, "nicol": 4, "martinez": 4, "nikita": 4, "nangia": 4, "nikla": 4, "decker": 4, "muennighoff": 4, "nitish": [4, 7], "shirish": [4, 7], "keskar": [4, 7], "niveditha": 4, "constant": 4, "fiedel": 4, "nuan": 4, "wen": 4, "oliv": [4, 7], "agha": 4, "elbaghdadi": 4, "omer": 4, "moreno": 4, "casar": 4, "parth": 4, "doshi": 4, "pascal": 4, "fung": 4, "pu": 4, "vicol": 4, "pegah": 4, "alipoormolabashi": 4, "peiyuan": 4, "eckerslei": 4, "phu": 4, "mon": 4, "htut": 4, "pinyu": 4, "hwang": 4, "piotr": 4, "mi\u0142kowski": 4, "piyush": 4, "pouya": 4, "pezeshkpour": 4, "priti": 4, "oli": 4, "qiaozhu": 4, "qing": 4, "lyu": 4, "qinlang": 4, "rabin": 4, "banjad": 4, "rachel": [4, 7], "etta": 4, "rudolph": 4, "raefer": 4, "rahel": 4, "haback": 4, "ramon": 4, "risco": 4, "rapha\u00ebl": 4, "milli\u00e8r": 4, "rhythm": 4, "garg": 4, "rif": 4, "saurou": 4, "riku": 4, "arakawa": 4, "robb": 4, "raymaek": 4, "rohan": 4, "sikand": 4, "novak": 4, "sitelew": 4, "lebra": 4, "rosann": 4, "rowan": [4, 7], "ruslan": 4, "salakhutdinov": 4, "stoval": 4, "teehan": 4, "sahib": 4, "saif": 4, "sajant": 4, "anand": [4, 7], "dillav": 4, "shleifer": 4, "wiseman": 4, "gruetter": 4, "schoenholz": 4, "sanghyun": 4, "sanjeev": 4, "kwatra": 4, "sarik": 4, "ghazarian": 4, "sayan": 4, "casei": [4, 7], "bischoff": 4, "gehrmann": 4, "schuster": 4, "sepideh": 4, "sadeghi": 4, "shadi": 4, "hamdan": 4, "sharon": 4, "shashank": 4, "sherri": 4, "shi": 4, "shikhar": 4, "shima": 4, "asaadi": 4, "shubh": 4, "pachchigar": 4, "shubham": 4, "toshniw": 4, "shyam": [4, 7], "upadhyai": 4, "shyamolima": 4, "debnath": 4, "siamak": 4, "shakeri": 4, "thormey": 4, "melzi": 4, "siva": 4, "reddi": 4, "sneha": 4, "priscilla": 4, "makini": 4, "soo": 4, "hwan": 4, "toren": 4, "sriharsha": 4, "hatwar": 4, "stanisla": 4, "dehaen": 4, "stefan": 4, "divic": 4, "stella": 4, "biderman": 4, "stephen": 4, "prasad": 4, "piantadosi": 4, "stuart": [4, 7], "shieber": 4, "summer": [4, 7], "misherghi": 4, "svetlana": 4, "kiritchenko": 4, "swaroop": 4, "tal": 4, "linzen": 4, "tariq": 4, "tatsu": 4, "te": 4, "th\u00e9o": 4, "desbord": 4, "theodor": 4, "rothschild": 4, "phan": [4, 7], "tiberiu": 4, "nkinyili": 4, "timo": 4, "schick": 4, "timofei": 4, "kornev": 4, "titu": 4, "tunduni": 4, "gerstenberg": 4, "trenton": 4, "trishala": 4, "neeraj": 4, "tushar": 4, "khot": 4, "shultz": 4, "uri": 4, "shaham": 4, "vera": 4, "demberg": 4, "victoria": [4, 7], "nyamai": 4, "vika": 4, "raunak": 4, "vinai": 4, "ramasesh": 4, "udai": 4, "prabhu": 4, "vishakh": 4, "padmakumar": 4, "vivek": 4, "srikumar": 4, "fedu": [4, 7], "wout": 4, "vossen": 4, "xiaoyu": 4, "tong": [4, 7], "xinran": 4, "xinyi": 4, "yadollah": 4, "yaghoobzadeh": 4, "yair": 4, "lakretz": 4, "yangqiu": 4, "yasaman": 4, "bahri": 4, "yichi": 4, "yide": 4, "yifu": 4, "yonatan": 4, "belinkov": 4, "yufang": 4, "seid": 4, "zhuoy": 4, "zijian": 4, "ziji": 4, "zirui": 4, "ziyi": 4, "extrapol": 4, "2206": 4, "04615": 4, "wpn": 4, "yada": 4, "pruksachatkun": 4, "amanpreet": 4, "hill": 4, "stickier": 4, "wsm": 4, "1804": 4, "07461": 4, "wtb": 4, "tai": 4, "borgeaud": 4, "dani": 4, "yogatama": 4, "denni": [4, 7], "donald": 4, "metzler": 4, "ed": 4, "oriol": 4, "vinyal": 4, "dean": 4, "07682": 4, "wdr": 4, "doolei": 4, "manlei": 4, "arka": [4, 7], "pal": 4, "feuer": 4, "siddhartha": 4, "ravid": 4, "shwartz": [4, 7], "ziv": 4, "khalid": 4, "saifullah": 4, "siddartha": 4, "naidu": 4, "chinmai": 4, "hegd": 4, "lecun": 4, "goldstein": 4, "willi": 4, "neiswang": 4, "micah": 4, "goldblum": 4, "19314": 4, "yyh": 4, "baosong": 4, "chengpeng": 4, "chengyuan": 4, "fei": 4, "guant": 4, "haoran": 4, "huan": 4, "jialong": 4, "jialin": 4, "jianhong": 4, "tu": 4, "jianwei": 4, "jianxin": 4, "jin": [4, 7], "jingren": 4, "jinz": 4, "jinzheng": 4, "junyang": 4, "keme": 4, "keqin": 4, "kexin": 4, "mingfeng": 4, "xue": [4, 7], "ni": 4, "pei": 4, "ru": 4, "men": 4, "ruiz": 4, "runji": 4, "shiji": 4, "sinan": 4, "tianhang": 4, "wenbin": 4, "ge": 4, "xiaodong": 4, "deng": 4, "xiaohuan": 4, "xingzhang": 4, "xinyu": [4, 7], "xipin": 4, "xuancheng": 4, "yichang": 4, "wan": 4, "yunfei": 4, "yuqiong": 4, "zhenru": 4, "zhihao": 4, "10671": 4, "zcl24": 4, "zhihan": 4, "cao": 4, "lizi": 4, "openreview": 4, "forum": 4, "aegrf1uy0p": 4, "zc": 4, "siyuan": 4, "zhuang": [4, 7], "zhanghao": 4, "yonghao": 4, "zi": 4, "zhuohan": 4, "xing": [4, 7], "2306": 4, "05685": 4, "huggingface24": 4, "06": [4, 8], "metaai24": 4, "queen": 5, "hipaa": 5, "properti": [5, 7], "gdpr": 5, "iot": 5, "unreli": 5, "impract": 5, "smallmai": 5, "evalu": [5, 6, 8], "production": 5, "programm": 5, "graphic": [5, 7], "vram": 5, "vector": [5, 7], "mathbf": 5, "x_1": 5, "x_2": 5, "x_n": 5, "x_": 5, "\u03b8": 5, "matrix": [5, 7], "concurr": 5, "mozilla": 5, "docker": 5, "gerganov": 5, "georgi": 5, "hundr": 5, "overwhelm": [5, 8], "manifesto": 5, "enjoy": 5, "bog": 5, "exploratori": 5, "hacker": 5, "Will": [5, 7], "codebas": 5, "prototyp": 5, "prematur": 5, "besid": 5, "lighter": 5, "sacrific": 5, "gguf": 5, "unifi": 5, "ggml": [5, 8], "ibm": [5, 7], "bit": 5, "metadata": 5, "disk": 5, "faster": 5, "backward": 5, "2024x": 5, "repo": 5, "easier": [5, 6, 7, 8], "compil": 5, "linux": 5, "argument": [5, 7, 8], "sudo": 5, "apt": 5, "cmake": 5, "bind": 5, "betlen": 5, "cnv": 5, "llamacpp": 5, "q8_0": 5, "succinct": 5, "ctrl": 5, "interject": 5, "philosoph": 5, "debat": 5, "fulfil": 5, "happi": 5, "responsibli": 5, "bye": 5, "goodby": 5, "port": 5, "127": 5, "curl": [5, 8], "localhost": 5, "v1": [5, 6, 7], "bearer": 5, "finish_reason": 5, "deepli": 5, "1734627879": 5, "completion_token": 5, "total_token": 5, "chatcmpl": 5, "5wl2tzjzdmzupvxwp2gcedr8xbpsyhfm": 5, "prompt_n": 5, "prompt_m": 5, "132": 5, "prompt_per_token_m": 5, "prompt_per_second": 5, "77619878666999": 5, "predicted_n": 5, "predicted_m": 5, "1700": 5, "654": 5, "predicted_per_token_m": 5, "36882142857143": 5, "predicted_per_second": 5, "92850867960208": 5, "wrestl": [5, 7], "gbnf": [5, 8], "8pm": 5, "appointmenttim": 5, "appointmentdetail": 5, "handi": 5, "modal": 5, "model_path": 5, "llama_cpp": 5, "create_chat_complet": 5, "occupi": 5, "activist": 5, "justin": [5, 7], "tunnei": 5, "ocho": 5, "appach": 5, "cosmopolitan": 5, "libc": 5, "portabl": 5, "durabl": 5, "usabl": [5, 7, 8], "tinyllama": 5, "wget": 5, "jartin": 5, "q5_k_m": 5, "renam": 5, "ex": 5, "chmod": 5, "nobrows": 5, "registri": 5, "nativ": [5, 8], "container": 5, "isol": [5, 7], "trai": 5, "familiar": 5, "permiss": [5, 7], "despit": [5, 6, 8], "ssfl": 5, "sh": [5, 8], "Or": 5, "11434": 5, "chatrespons": 5, "tradeoff": [5, 7], "easiest": 5, "rich": [5, 7], "playground": 5, "simultan": [5, 7], "verif": [5, 8], "importantli": 5, "gemma": [5, 8], "intuit": 5, "beginn": 5, "tensorrt": 5, "trt": 5, "latex": 5, "voic": 5, "pwa": 5, "rag": 5, "simpler": [5, 7, 8], "medium": [5, 7, 8], "gpt4all": 5, "rbac": 5, "diagram": [5, 6, 7], "qwenlm": 5, "readthedoc": 5, "speed_benchmark": 5, "docstr": 5, "bc24": 5, "andrei": [5, 7], "abetlen": 5, "gc24": 5, "ggerganov": [5, 8], "blob": [5, 8], "readm": [5, 8], "gc4a": 5, "gc4b": 5, "thread": 5, "huggingface4xa": 5, "huggingface4xb": 5, "ibmthink24": 5, "lmstudio24": 5, "lmstudio": 5, "mozillaocho24": 5, "possibli": 6, "eliot": 6, "thumb": 6, "\u00be": 6, "max_output_token": 6, "4096": 6, "16384": 6, "contrari": 6, "surpass": 6, "truncat": 6, "max_input_token": 6, "input_cost_per_token": 6, "output_cost_per_token": 6, "11b": [6, 7], "128000": 6, "5e": 6, "20241022": [6, 7], "8192": 6, "200000": 6, "3e": 6, "0613": 6, "6e": 6, "gemini": 6, "flash": 6, "1048576": 6, "2097152": 6, "05e": 6, "incomplet": [6, 7], "abruptli": 6, "shallow": 6, "thorough": [6, 7], "dissatisfact": 6, "frustrat": [6, 7], "feasibl": 6, "10k": 6, "charactertextsplitt": 6, "tiktoken": [6, 7], "sequenti": 6, "newlin": 6, "broadli": [6, 8], "cheap": 6, "speciali": 6, "nltk": 6, "spaci": 6, "hierarch": [6, 7], "talk": 6, "theme": [6, 7], "splitter": 6, "get_chunk": 6, "chunk_siz": 6, "chunk_overlap": 6, "langchain_text_splitt": 6, "text_splitt": 6, "from_tiktoken_encod": 6, "split_text": 6, "persona": 6, "langchain_cor": [6, 8], "prompttempl": 6, "get_base_prompt_templ": 6, "base_prompt": [6, 8], "from_templ": 6, "llmchain": 6, "parser": [6, 8], "output_pars": 6, "stroutputpars": 6, "langchain_commun": 6, "chat_model": 6, "chatlitellm": 6, "get_llm_chain": 6, "prompt_templ": [6, 8], "llm_chain": [6, 8], "api_key_label": 6, "upper": 6, "_api_kei": 6, "get_dynamic_prompt_templ": 6, "get_dynamic_prompt_param": 6, "prompt_param": 6, "part_idx": 6, "total_part": 6, "chat_context": 6, "param": 6, "dynamic_prompt_param": 6, "introduct": 6, "concaten": 6, "generate_report": 6, "input_cont": 6, "llm_model_nam": 6, "report_part": 6, "num_part": 6, "dinam": 6, "priovid": 6, "invok": [6, 8], "cummul": 6, "max_chunk_s": 6, "max_chunk_overlap": 6, "readabl": 6, "apple_report": 6, "luation": 6, "disciplin": 6, "subhead": 6, "depth": [6, 7], "overlook": 6, "preprocess": [6, 8], "necessit": 6, "meticul": 6, "bottleneck": 6, "mustafa": 6, "suleyman": 6, "infinit": 6, "fewer": [6, 7], "condens": 6, "versatil": 6, "grace": 6, "fallback": 6, "empow": [6, 7], "langchain24": 6, "how_to": 6, "immens": 7, "commonplac": 7, "hartvigsen": 7, "societi": 7, "statement": 7, "alarm": 7, "openli": 7, "dolli": 7, "v2": 7, "llama2": [7, 8], "13b": 7, "emb": 7, "birth": 7, "siam": 7, "edgington": 7, "jailbreak": 7, "promptcraft": 7, "stealth": 7, "sutton": 7, "subtl": 7, "trigger": 7, "subtleti": 7, "exception": 7, "phrase": 7, "evad": 7, "hqve": 7, "frer": 7, "hplidai": 7, "pl": 7, "hyperion": 7, "coast": 7, "redwood": 7, "tallest": 7, "tree": [7, 8], "routin": 7, "overview": [7, 8], "generalist": 7, "injustic": 7, "inequ": 7, "undermin": 7, "perpetu": 7, "displac": 7, "eros": 7, "fake": 7, "deepfak": 7, "distrust": 7, "cyberattack": 7, "spread": 7, "disinform": 7, "inadvert": 7, "interven": 7, "irrevers": 7, "uncheck": 7, "extinct": 7, "race": 7, "incentiv": 7, "shortcut": 7, "behind": 7, "stress": 7, "urgent": 7, "reorient": 7, "prejudic": 7, "gallego": 7, "leak": 7, "poison": 7, "intention": 7, "inject": 7, "mislead": 7, "exabeam": 7, "finra": 7, "3110": 7, "mandat": 7, "supervisori": 7, "medicin": 7, "unicef": 7, "contest": 7, "congress": 7, "enact": 7, "pictur": [7, 8], "territori": 7, "oversea": 7, "chines": 7, "legitim": 7, "consent": 7, "complaint": 7, "cooper": 7, "extraterritori": 7, "offshor": 7, "draft": 7, "voluntari": 7, "neutral": 7, "player": 7, "prepared": 7, "ahead": 7, "compris": 7, "cbrn": 7, "persuas": 7, "autonomi": 7, "gradat": 7, "scorecard": 7, "elig": 7, "advisori": 7, "sag": 7, "shut": 7, "prerequisit": 7, "harden": 7, "asl": 7, "biosafeti": 7, "elev": 7, "warn": 7, "bioweapon": 7, "compartment": 7, "difficulti": 7, "4x": 7, "jump": 7, "paus": 7, "frontier": 7, "deepmind": 7, "biosecur": 7, "buffer": 7, "formul": [7, 8], "calibr": 7, "promin": 7, "taxonomi": 7, "llamaguard": 7, "leaderboard": 7, "3x": 7, "5x": 7, "alaga": 7, "substandard": 7, "rapidli": [7, 8], "oxford": 7, "wachter": 7, "blur": 7, "ill": 7, "stifl": 7, "suscept": 7, "aadc": 7, "outset": 7, "curricula": 7, "adversari": 7, "uncov": [7, 8], "appar": 7, "thoroughli": 7, "lm": [7, 8], "problemat": 7, "arrai": 7, "undergo": 7, "280b": 7, "cai": [7, 8], "utilis": 7, "minimis": 7, "enshrin": 7, "evas": 7, "resort": 7, "encod": 7, "avenu": 7, "cambria": 7, "inherit": 7, "influenti": 7, "debias": 7, "occurr": 7, "phish": 7, "clarifi": 7, "toler": 7, "checklist": 7, "abus": 7, "ux": 7, "architect": 7, "retrofit": 7, "promptli": 7, "dashboard": 7, "misalign": 7, "star": 7, "postpon": 7, "sens": 7, "combat": 7, "counter": 7, "traffic": 7, "workaround": 7, "vital": 7, "silo": 7, "hierarchi": 7, "66": 7, "mcq": 7, "regex": [7, 8], "joint": 7, "facet": 7, "purpl": 7, "circl": 7, "opensafetylab": 7, "salad_bench_dataset": 7, "base_set": 7, "gptfuzzer": 7, "auto": 7, "qid": 7, "o1": 7, "supremaci": 7, "o53": 7, "o14": 7, "o5": 7, "o65": 7, "plagiar": 7, "o16": 7, "o6": 7, "o47": 7, "campaign": 7, "o12": 7, "o52": 7, "surveil": 7, "spous": 7, "know": [7, 8], "o13": 7, "breakdown": [7, 8], "ncount": 7, "21318": 7, "8756": 7, "6486": 7, "o2": 7, "1717": 7, "o4": 7, "1477": 7, "o3": 7, "socioeconom": 7, "851": 7, "int64": 7, "gen": 7, "15433": 7, "hh": 7, "4184": 7, "659": 7, "advbench": 7, "230": 7, "189": 7, "toxicchat": 7, "anyth": 7, "93": 7, "817": 7, "misconcept": 7, "ingrain": 7, "mc1": 7, "singular": 7, "choices4": 7, "mc2": 7, "set4": 7, "scorer": 7, "correctli": [7, 8], "truthful_qa": 7, "truthfulqa_dataset": 7, "multiple_choic": 7, "best_answ": 7, "correct_answ": 7, "incorrect_answ": 7, "watermelon": 7, "digest": 7, "noth": 7, "stomach": 7, "sick": 7, "wonderopoli": 7, "wonder": 7, "belli": 7, "swallow": 7, "dream": 7, "die": 7, "indigest": 7, "unconsci": 7, "excret": 7, "compos": 7, "asr": 7, "r2d2": 7, "multimod": 7, "wider": [7, 8], "mmlu": 7, "mass": 7, "destruct": 7, "asynchron": 7, "webpurifi": 7, "protectai": 7, "aw": 7, "comprehend": 7, "amazon": 7, "nemo": 7, "nvidia": 7, "keyword": 7, "toolset": 7, "nemmo": 7, "synchron": 7, "nemoguardrail": 7, "llmrail": 7, "railsconfig": 7, "from_path": 7, "rail": 7, "hello": 7, "ministr": 7, "mistralai": 7, "mistral_api_kei": 7, "moderate_chat": 7, "omni": 7, "pprint": 7, "to_json": 7, "threaten": 7, "illicit": 7, "granit": 7, "guardian": 7, "consortium": 7, "v3": 7, "begin_of_text": 7, "start_header_id": 7, "end_header_id": 7, "unsafe_categori": 7, "user_message_1": 7, "model_answer_1": 7, "comma": 7, "eot_id": 7, "eom_id": 7, "denot": 7, "s1": 7, "s2": 7, "s3": 7, "s4": 7, "s5": 7, "defam": 7, "s6": 7, "s7": 7, "s8": 7, "s9": 7, "s10": 7, "s11": 7, "s12": 7, "s13": 7, "atla": 7, "2b": 7, "hap": 7, "38m": 7, "125m": 7, "padhi": 7, "shieldgemma": 7, "exemplifi": 7, "accomplish": [7, 8], "judge_prompt": 7, "american": 7, "vandal": 7, "underag": 7, "drink": 7, "vulgar": 7, "obscen": 7, "racism": 7, "derogatori": 7, "firearm": 7, "safety_scor": 7, "IN": 7, "borderlin": 7, "verdict": 7, "boolean": [7, 8], "brief": 7, "rational": 7, "delimit": 7, "paramount": 7, "evenli": 7, "good_sampl": 7, "bad_sampl": 7, "2024z": 7, "surg": 7, "scam": 7, "get_profanity_sampl": 7, "show_stat": 7, "bool": 7, "current_dir": 7, "getcwd": 7, "data_path": 7, "profanity_en": 7, "random_st": 7, "ncategori": 7, "category_count": 7, "category_1": 7, "1f": 7, "profanity_sampl": 7, "nsampl": 7, "anatomi": 7, "slur": 7, "182": 7, "bodili": 7, "fluid": 7, "excrement": 7, "insult": 7, "mental": 7, "disabl": 7, "jap": 7, "babi": 7, "batter": 7, "crazi": 7, "sob": 7, "fukka": 7, "sh1t3": 7, "get_salad_sampl": 7, "salad_sampl": 7, "tortur": 7, "porn": 7, "sin": 7, "sight": 7, "god": 7, "embezzl": 7, "xanax": 7, "alcohol": 7, "get_good_sampl": 7, "min_scor": 7, "reichstag": 7, "profanity_data": 7, "salad_data": 7, "good_data": 7, "all_data": 7, "prompt_sampl": 7, "is_unsaf": 7, "counti": 7, "holli": 7, "ridg": 7, "nc": 7, "town": 7, "onslow": 7, "carolina": 7, "diver": 7, "underwat": 7, "maze": 7, "coral": 7, "treasur": 7, "vivid": 7, "sensori": 7, "emot": 7, "labyrinthin": 7, "passag": 7, "reef": 7, "suspens": 7, "obstacl": 7, "creatur": 7, "nomin": 7, "nobel": 7, "literatur": 7, "love": 7, "ny": [7, 8], "logo": 7, "thief": 7, "rob": 7, "famou": 7, "nstatist": 7, "source_stat": 7, "type_stat": 7, "plug": 7, "safetyvalid": 7, "validationresult": 7, "dataclass": 7, "abstractmethod": 7, "llmguardvalid": 7, "scanner": 7, "bantop": 7, "llm_guard": 7, "input_scann": 7, "scan_prompt": 7, "matchtyp": 7, "default_banned_top": 7, "banned_top": 7, "super": 7, "banned_topics_scann": 7, "use_onnx": 7, "toxicity_scann": 7, "match_typ": 7, "fail_fast": 7, "unsafe_scann": 7, "gun": 7, "cool": 7, "hunt": 7, "deer": 7, "dad": 7, "mistralvalid": 7, "hate_and_discrimin": 7, "violence_and_threat": 7, "dangerous_and_criminal_cont": 7, "selfharm": 7, "openaivalid": 7, "attr": 7, "dir": 7, "getattr": 7, "illicit_viol": 7, "llmjudgevalid": 7, "prompt_path": 7, "llmjudg": 7, "filenotfounderror": 7, "elaps": 7, "score_valid": 7, "id_": 7, "validator_inst": 7, "start_tim": 7, "validation_result": 7, "elapsed_tim": 7, "prompt_sample_id": 7, "validator_nam": 7, "scoring_prompt": 7, "scoring_result": 7, "01536": 7, "34098": 7, "497136": 7, "546416": 7, "calculate_validator_metr": 7, "bad_sourc": 7, "good_sourc": 7, "tpr": 7, "fpr": 7, "f1_score": 7, "tn": 7, "fp": 7, "fn": 7, "tp": 7, "255": 7, "74": 7, "67": 7, "238": 7, "130": 7, "256": 7, "79": 7, "149": 7, "96": 7, "213": 7, "81": 7, "190": 7, "235": 7, "63": 7, "222": 7, "82": 7, "75": 7, "480": 7, "157": 7, "487": 7, "495": 7, "482": 7, "667": 7, "248": 7, "466": 7, "143": 7, "355": 7, "block": 7, "half": 7, "slowest": 7, "room": 7, "false_posit": 7, "left_on": 7, "right_on": 7, "persian": 7, "teenag": 7, "\u0686\u0627\u0644\u0634": 7, "\u0645\u0648\u0645\u0648": 7, "\u06cc\u06a9": 7, "\u062d\u0642\u0647": 7, "\u0648": 7, "\u0627\u0641\u0633\u0627\u0646\u0647": 7, "\u0645\u062d\u0644\u06cc": 7, "\u0627\u0633\u062a": 7, "\u06a9\u0647": 7, "\u0634\u0627\u06cc\u0639\u0647": 7, "\u0622\u0646": 7, "\u062f\u0631": 7, "\u0634\u0628\u06a9\u0647": 7, "\u0647\u0627\u06cc": 7, "\u0627\u062c\u062a\u0645\u0627\u0639\u06cc": 7, "\u067e\u062e\u0634": 7, "\u0634\u062f": 7, "\u06af\u0632\u0627\u0631\u0634": 7, "\u06a9\u0627\u0631\u0628\u0631\u06cc": 7, "\u0628\u0627": 7, "\u0646\u0627\u0645": 7, "\u06a9\u0648\u062f\u06a9\u0627\u0646": 7, "\u0646\u0648\u062c\u0648\u0627\u0646\u0627\u0646": 7, "\u0631\u0627": 7, "\u0628\u0647": 7, "\u0627\u0639\u0645\u0627\u0644": 7, "\u062e\u0637\u0631\u0646\u0627\u06a9\u06cc": 7, "\u0645\u0627\u0646\u0646\u062f": 7, "\u062e\u0648\u062f\u0632\u0646\u06cc": 7, "\u062e\u0648\u062f\u06a9\u0634\u06cc": 7, "\u062a\u0631\u063a\u06cc\u0628": 7, "\u0645\u06cc": 7, "\u06a9\u0646\u062f": 7, "\u0648\u062c\u0648\u062f": 7, "\u0635\u062d\u0628\u062a": 7, "\u0632\u06cc\u0627\u062f\u06cc": 7, "\u0645\u0648\u0631\u062f": 7, "\u0627\u06cc\u0646": 7, "\u062a\u0639\u062f\u0627\u062f": 7, "\u0634\u06a9\u0627\u06cc\u0627\u062a": 7, "\u0648\u0627\u0642\u0639\u06cc": 7, "\u0628\u0633\u06cc\u0627\u0631": 7, "\u06a9\u0645": 7, "\u0628\u0648\u062f": 7, "\u067e\u0644\u06cc\u0633": 7, "\u0635\u062f\u0645\u0647": 7, "\u062f\u06cc\u062f\u0646": 7, "\u062f\u0644\u06cc\u0644": 7, "\u062a\u0623\u062b\u06cc\u0631": 7, "\u0645\u0633\u062a\u0642\u06cc\u0645": 7, "\u067e\u062f\u06cc\u062f\u0647": 7, "\u062a\u0623\u06cc\u06cc\u062f": 7, "\u0646\u06a9\u0631\u062f\u0647": 7, "\u062a\u0631\u0633": 7, "\u0646\u06af\u0631\u0627\u0646\u06cc": 7, "\u0627\u06cc\u062c\u0627\u062f": 7, "\u0634\u062f\u0647": 7, "\u0628\u06cc\u0634\u062a\u0631": 7, "\u0627\u0632": 7, "\u062e\u0648\u062f": 7, "\u0631\u0633\u0627\u0646\u0647": 7, "\u0647\u0627": 7, "\u0637\u0648\u0631\u06cc": 7, "\u062e\u06cc\u0631\u06cc\u0647": 7, "\u0647\u0634\u062f\u0627\u0631": 7, "\u062f\u0627\u062f\u0646\u062f": 7, "\u0622\u0633\u06cc\u0628": 7, "\u0627\u0646\u062a\u0638\u0627\u0631\u0627\u062a": 7, "\u0645\u062d\u062a\u0648\u0627\u06cc": 7, "\u062e\u0634\u0648\u0646\u062a": 7, "\u0622\u0645\u06cc\u0632": 7, "\u0627\u06cc\u0646\u062a\u0631\u0646\u062a": 7, "\u06af\u0641\u062a\u0647": 7, "\u0634\u0648\u062f": 7, "\u0627\u0648\u0644\u06cc\u0646": 7, "\u0628\u0627\u0631": 7, "\u0633\u0627\u0644": 7, "\u06f2\u06f0\u06f1\u06f8": 7, "\u067e\u0633": 7, "\u0622\u0646\u06a9\u0647": 7, "\u0631\u0648\u0632\u0646\u0627\u0645\u0647": 7, "\u0627\u0646\u062f\u0648\u0646\u0632\u06cc\u0627\u06cc\u06cc": 7, "\u062e\u0628\u0631": 7, "\u062f\u062e\u062a\u0631": 7, "\u06f1\u06f2": 7, "\u0633\u0627\u0644\u0647": 7, "\u062f\u0627\u062f": 7, "\u0645\u0648\u0636\u0648\u0639": 7, "\u062c\u0647\u0627\u0646\u06cc": 7, "\u062a\u0628\u062f\u06cc\u0644": 7, "\u0645\u062c\u0633\u0645\u0647": 7, "\u0647\u0646\u0631\u0645\u0646\u062f": 7, "\u0698\u0627\u067e\u0646\u06cc": 7, "\u0647\u0631": 7, "\u0686\u0646\u062f": 7, "\u0634\u0627\u06cc\u062f": 7, "\u0646\u06af\u0627\u0647": 7, "\u0628\u0639\u0636\u06cc": 7, "\u0632\u06cc\u0628\u0627": 7, "\u0646\u0628\u0627\u0634\u062f": 7, "\u0627\u0645\u0627": 7, "\u06a9\u0627\u0645\u0644\u0627": 7, "\u0628\u06cc": 7, "\u062e\u0637\u0631": 7, "\u0627\u06cc\u0631\u0627\u0646": 7, "\u0645\u062f\u062a": 7, "\u0628\u06cc\u0646": 7, "\u06a9\u0627\u0631\u0628\u0631\u0627\u0646": 7, "\u0645\u0637\u0631\u062d": 7, "\u0633\u0627\u0644\u06cc": 7, "\u0633\u0631\u0627\u0633\u0631": 7, "\u062c\u0647\u0627\u0646": 7, "\u0645\u0634\u0627\u0628\u0647\u06cc": 7, "\u0628\u0631\u0627\u06cc": 7, "\u0648\u0627\u0644\u062f\u06cc\u0646": 7, "\u06a9\u0631\u062f\u0647": 7, "\u0627\u0641\u0631\u0627\u062f": 7, "\u0686\u0647": 7, "\u06a9\u0627\u0631\u06cc": 7, "\u062f\u0639\u0648\u062a": 7, "tourist": 7, "distress": 7, "polish": 7, "galician": 7, "dzisiaj": 7, "szwecji": 7, "innych": 7, "bogatych": 7, "krajach": 7, "ludzi": 7, "u\u017cywaj\u0105": 7, "mn\u00f3stwo": 7, "najr\u00f3\u017cniejszych": 7, "urz\u0105dze\u0144": 7, "hox": 7, "suecia": 7, "outro": 7, "pa\u00eds": 7, "rico": 7, "xent": 7, "usa": [7, 8], "moita": 7, "m\u00e1quina": 7, "diferent": 7, "\u0142\u00f3dka": 7, "zaczyna": 7, "ton\u0105\u0107": 7, "tury\u015bci": 7, "wracaj\u0105": 7, "statek": 7, "dom\u00f3w": 7, "gdzie": 7, "opowiadaj\u0105": 7, "tym": 7, "jak": 7, "zostali": 7, "zaatakowani": 7, "surprisingli": 7, "unsettl": 7, "paradox": 7, "harbor": 7, "wisdom": 7, "aspir": 7, "technologist": 7, "disciplinari": 7, "ethicist": 7, "policymak": 7, "ai24": 7, "asa24": 7, "jide": 7, "jona": 7, "schuett": 7, "marku": 7, "anderljung": 7, "08751": 7, "bhy": 7, "geoffrei": 7, "hinton": 7, "pieter": 7, "abbeel": 7, "trevor": 7, "darrel": 7, "yuval": 7, "harari": 7, "ya": 7, "lan": 7, "shai": 7, "shalev": 7, "gillian": 7, "hadfield": 7, "clune": 7, "tegan": 7, "maharaj": 7, "hutter": 7, "at\u0131l\u0131m": 7, "g\u00fcne\u015f": 7, "baydin": 7, "sheila": 7, "mcilraith": 7, "qiqi": 7, "ashwin": 7, "acharya": 7, "anca": 7, "dragan": 7, "philip": 7, "torr": 7, "russel": 7, "kahneman": 7, "s\u00f6ren": 7, "mindermann": 7, "amid": 7, "384": 7, "6698": 7, "1126": 7, "adn0117": 7, "bbc": 7, "emili": 7, "braca": 7, "israel": 7, "carter": 7, "hafsa": 7, "kanchwala": 7, "khojasteh": 7, "charli": 7, "landow": 7, "luo": 7, "magarelli": 7, "mirin": 7, "averi": 7, "moyer": 7, "kayla": 7, "simpson": 7, "amelia": 7, "skawinski": 7, "heverin": 7, "23308": 7, "bmc": 7, "dillon": 7, "brendan": 7, "murphi": 7, "khachaturov": 7, "gleav": 7, "kellin": 7, "pelrin": 7, "2408": [7, 8], "02946": 7, "cmm": 7, "erik": 7, "lorenzo": 7, "malandri": 7, "fabio": 7, "mercorio": 7, "navid": 7, "nobani": 7, "seveso": 7, "15248": 7, "edg24": 7, "exa24": 7, "cyber": 7, "grb": 7, "rossi": 7, "barrow": 7, "mehrab": 7, "tanjim": 7, "sungchul": 7, "franck": 7, "dernoncourt": 7, "ruiyi": 7, "nesreen": 7, "2309": 7, "00770": 7, "h44z": 7, "hgp": 7, "saadia": 7, "hamid": 7, "palangi": 7, "dipankar": 7, "ec": 7, "kamar": 7, "oxi": 7, "smaranda": 7, "muresan": 7, "preslav": 7, "nakov": 7, "alin": 7, "villavicencio": 7, "editor": 7, "60th": 7, "3309": 7, "3326": 7, "dublin": 7, "aclanthologi": 7, "acl": 7, "18653": 7, "hym": 7, "weijiang": 7, "weitao": 7, "weihong": 7, "zhangyin": 7, "haotian": 7, "qianglong": 7, "weihua": 7, "xiaocheng": 7, "bing": 7, "ting": 7, "dx": 7, "1145": [7, 8], "3703155": 7, "ldw": 7, "lijun": 7, "ruohui": 7, "xuhao": 7, "wangmeng": 7, "zuo": 7, "dahua": 7, "qiao": 7, "shao": 7, "05044": 7, "mpy": 7, "xuwang": 7, "zifan": 7, "norman": 7, "mu": 7, "elham": 7, "sakhae": 7, "nathaniel": 7, "forsyth": 7, "04249": 7, "mlc24": 7, "illumin": 7, "ailumin": 7, "oaa": 7, "adler": 7, "ahmad": 7, "ilg": 7, "akkaya": 7, "florencia": 7, "leoni": 7, "aleman": 7, "janko": 7, "altenschmidt": 7, "altman": 7, "shyamal": 7, "anadkat": 7, "avila": 7, "valeri": 7, "balcom": 7, "baltescu": 7, "haim": 7, "belgum": 7, "irwan": 7, "bello": 7, "jake": 7, "berdin": 7, "bernadett": 7, "shapiro": 7, "berner": 7, "lenni": 7, "bogdonoff": 7, "boiko": 7, "madelain": 7, "boyd": 7, "luisa": 7, "brakman": 7, "button": 7, "rosi": 7, "campbel": 7, "cann": 7, "brittani": 7, "carei": 7, "carlson": 7, "rori": 7, "carmichael": 7, "che": 7, "foti": 7, "sulli": 7, "rubi": 7, "chess": 7, "chester": 7, "cho": 7, "hyung": 7, "won": 7, "chung": 7, "jeremiah": 7, "currier": 7, "yunx": 7, "cori": 7, "decareaux": 7, "degri": 7, "deutsch": 7, "devil": 7, "dhar": 7, "steve": 7, "dowl": 7, "dun": 7, "adrien": 7, "ecoffet": 7, "atti": 7, "eleti": 7, "tyna": 7, "elound": 7, "farhi": 7, "niko": 7, "sim\u00f3n": 7, "posada": 7, "fishman": 7, "juston": 7, "isabella": 7, "fulford": 7, "georg": 7, "gibson": 7, "vik": 7, "tarun": 7, "gogineni": 7, "goh": 7, "rapha": 7, "gontijo": 7, "lope": 7, "gordon": 7, "morgan": 7, "grafstein": 7, "yufei": 7, "guo": 7, "hallaci": 7, "heaton": 7, "johann": 7, "heideck": 7, "hickei": 7, "wade": 7, "hoeschel": 7, "houghton": 7, "kenni": 7, "hsu": 7, "shengli": 7, "joost": 7, "huizinga": 7, "shawn": 7, "joann": 7, "jang": 7, "roger": 7, "haozhun": 7, "shino": 7, "jomoto": 7, "billi": 7, "jonn": 7, "tomer": 7, "kaftan": 7, "\u0142ukasz": 7, "kamali": 7, "ingmar": 7, "kanitscheid": 7, "tabarak": 7, "khan": 7, "logan": 7, "kilpatrick": 7, "jong": 7, "wook": 7, "christina": 7, "yongjik": 7, "hendrik": 7, "kirchner": 7, "kiro": 7, "matt": 7, "kokotajlo": 7, "kondraciuk": 7, "kondrich": 7, "konstantinidi": 7, "kosic": 7, "vishal": 7, "kuo": 7, "lamp": 7, "ikai": 7, "teddi": 7, "jade": 7, "leung": 7, "chak": 7, "ming": 7, "lim": 7, "molli": 7, "mateusz": 7, "litwin": 7, "theresa": 7, "lopez": 7, "patricia": 7, "lue": 7, "makanju": 7, "malfacini": 7, "markov": 7, "yaniv": 7, "markovski": 7, "bianca": 7, "mayn": 7, "mckinnei": 7, "christin": 7, "mcleavei": 7, "mcmillan": 7, "mcneil": 7, "aalok": 7, "menick": 7, "mishchenko": 7, "vinni": 7, "monaco": 7, "murk": 7, "m\u00e9ly": 7, "ashvin": 7, "nair": 7, "reiichiro": 7, "nakano": 7, "rajeev": 7, "nayak": 7, "arvind": 7, "neelakantan": 7, "hyeonwoo": 7, "noh": 7, "keef": 7, "jakub": 7, "pachocki": 7, "palermo": 7, "ashlei": 7, "pantuliano": 7, "parish": 7, "emi": 7, "parparita": 7, "passo": 7, "perelman": 7, "belbut": 7, "pere": 7, "pokorni": 7, "pokrass": 7, "vitchyr": 7, "pong": 7, "tolli": 7, "powel": 7, "bori": 7, "proehl": 7, "rae": 7, "ramesh": 7, "franci": 7, "kendra": 7, "rimbach": 7, "carl": 7, "rotst": 7, "roussez": 7, "saltarelli": 7, "ted": 7, "sander": 7, "schnurr": 7, "selsam": 7, "kyla": 7, "sheppard": 7, "toki": 7, "sherbakov": 7, "shieh": 7, "shoker": 7, "pranav": 7, "szymon": 7, "sidor": 7, "sigler": 7, "sitkin": 7, "sokolowski": 7, "natali": 7, "staudach": 7, "madelein": 7, "tootoonchian": 7, "tseng": 7, "preston": 7, "tuggl": 7, "turlei": 7, "juan": 7, "cer\u00f3n": 7, "urib": 7, "vallon": 7, "vijayvergiya": 7, "jai": 7, "alvin": 7, "ward": 7, "cj": 7, "weinmann": 7, "akila": 7, "welihinda": 7, "jiayi": 7, "weng": 7, "lilian": 7, "wiethoff": 7, "willner": 7, "wolrich": 7, "lauren": 7, "workman": 7, "sherwin": 7, "yoo": 7, "zeller": 7, "shengjia": 7, "juntang": 7, "zhuk": 7, "2303": 7, "08774": 7, "pnc": 7, "inkit": 7, "manish": 7, "nagireddi": 7, "giandomenico": 7, "cornacchia": 7, "subhajit": 7, "chaudhuri": 7, "tejaswini": 7, "pedapati": 7, "pierr": 7, "dognin": 7, "keerthiram": 7, "murugesan": 7, "miehl": 7, "santill\u00e1n": 7, "kieran": 7, "giulio": 7, "zizzo": 7, "muhammad": 7, "zaid": 7, "hame": 7, "purcel": 7, "desmond": 7, "pan": 7, "ing": 7, "vejsbjerg": 7, "dali": 7, "hind": 7, "werner": 7, "geyer": 7, "ambrish": 7, "rawat": 7, "kush": 7, "varshnei": 7, "prasanna": 7, "sattigeri": 7, "07724": 7, "saffron": 7, "ring": 7, "aslanid": 7, "glaes": 7, "nat": 7, "mcalees": 7, "irv": 7, "2202": 7, "03286": 7, "szw": 7, "qinghua": 7, "higham": 7, "gorban": 7, "bastouni": 7, "ivan": 7, "tyukin": 7, "12670": 7, "vsk": 7, "kannappan": 7, "simplesafetytest": 7, "2311": 7, "08370": 7, "wmr24": 7, "sandra": 7, "brent": 7, "mittelstadt": 7, "duti": 7, "royal": 7, "240197": 7, "royalsocietypublish": 7, "1098": 7, "rso": 7, "ylx24": 7, "jiahao": 7, "xingwei": 7, "paperswithcod": 7, "zyi": 7, "shune": 7, "lyumanshan": 7, "jingyu": 7, "shui": 7, "haobin": 7, "pengfei": 7, "hewu": 7, "ghost": 7, "14931": 7, "zho24": 7, "amazonwservices24": 7, "anthropic24": 7, "cdn": 7, "1adf000c8f675958c2ee23805d91aaade1cd4613": 7, "centerfasafety24a": 7, "centerforaisafeti": 7, "centerfasafety24b": 7, "deepmind24": 7, "googleapi": 7, "fsf": 7, "europeanmagency24": 7, "ema": 7, "europa": 7, "activities_en": 7, "financialirauthority24": 7, "ibm24": 7, "watsonx": 7, "saa": 7, "libraryocongress23": 7, "loc": 7, "gov": 7, "mistralai24": 7, "mlsteam24": 7, "mlsafeti": 7, "nationaliosatechnology24": 7, "nist": 7, "itl": 7, "nvidia24": 7, "toolkit": 7, "openai24a": 7, "openai24b": 7, "opensafetylab24a": 7, "opensafetylab24b": 7, "protectai24": 7, "surgeai24": 7, "ukgovernment24": 7, "unicef24": 7, "innocenti": 7, "julia": 8, "easili": 8, "response_cont": 8, "wow": 8, "lot": 8, "impress": 8, "huge": 8, "serious": 8, "is_json": 8, "myjson": 8, "trial": 8, "wrangl": 8, "hoc": 8, "streamlin": 8, "dataset": 8, "unwant": 8, "overflow": 8, "twitter": 8, "youtub": 8, "blueprint": 8, "json_format": 8, "person1": 8, "q1": 8, "person2": 8, "nest": 8, "thellm": 8, "conceptu": 8, "unend": 8, "whitespac": 8, "throw": 8, "somewher": 8, "json_object": 8, "circul": 8, "vertex": 8, "worri": 8, "invalid": 8, "enum": 8, "secextract": 8, "mentioned_ent": 8, "mentioned_plac": 8, "extract_from_sec_fil": 8, "sec_filing_text": 8, "hint": 8, "prompt_extract": 8, "sec_extract": 8, "washington": 8, "beg": 8, "with_structured_output": 8, "runnabl": 8, "typeddict": 8, "qu": 8, "langchain_openai": 8, "chatopenai": 8, "chatprompttempl": 8, "extract_from_sec_filing_langchain": 8, "structured_llm": 8, "from_messag": 8, "sec_extraction_langchain": 8, "logit": 8, "willard": 8, "louf": 8, "reformul": 8, "finit": 8, "fsm": 8, "s_": 8, "s_t": 8, "s_1": 8, "mask": 8, "tild": 8, "odot": 8, "rightarrow": 8, "wise": 8, "thien": 8, "automaton": 8, "dfa": 8, "decod": 8, "outgo": 8, "renorm": 8, "yy": 8, "nn": 8, "ever": 8, "aa": 8, "lwai": 8, "prop": 8, "yynnaa": 8, "malform": 8, "sec_extraction_outlin": 8, "zsp": 8, "zicorp": 8, "cpp": 8, "bnf": 8, "backu": 8, "naur": 8, "wikipedia": 8, "fssl": 8, "extract_entities_from_sec_fil": 8, "ollama_structured_output_prompt_suffix": 8, "ollama_structured_output_temperatur": 8, "uncensor": 8, "model_json_schema": 8, "response_json": 8, "wrapper": 8, "exllama2": 8, "mlx": 8, "chanc": 8, "furthermor": 8, "nonetheless": 8, "studi": 8, "extran": 8, "dispar": 8, "preval": 8, "speak": 8, "aider": 8, "outweigh": 8, "rebutt": 8, "reproduct": 8, "paint": 8, "dottxt": 8, "flaw": 8, "uneven": 8, "didn": 8, "conflat": 8, "drawback": 8, "unlock": 8, "thank": 8, "pfiffer": 8, "aid24": 8, "dot24": 8, "demo": 8, "gge24": 8, "llf": 8, "xieyang": 8, "frederick": 8, "fiannaca": 8, "terri": 8, "koo": 8, "dixon": 8, "ea": 8, "machineri": 8, "3613905": 8, "3650756": 8, "ln": 8, "xuan": 8, "hai": 8, "nguyen": 8, "ngoc": 8, "tiviati": 8, "hieu": 8, "dao": 8, "shafiq": 8, "joti": 8, "kenji": 8, "kawaguchi": 8, "nanci": 8, "min": 8, "kan": 8, "08656": 8, "out24": 8, "twt": 8, "zhi": 8, "cheng": 8, "kuang": 8, "tsai": 8, "chieh": 8, "hung": 8, "yun": 8, "nung": 8, "02442": 8, "tt24": 8, "vivien": 8, "vivien000": 8, "wl23": 8, "r\u00e9mi": 8, "09702": 8, "wikipediacontributors24": 8, "wiktionari": 8, "naur_form": 8}, "objects": {}, "objtypes": {}, "objnames": {}, "titleterms": {"introduct": [0, 2, 3, 4, 5, 7, 8], "content": [0, 3, 4, 5, 6, 7, 8], "core": 0, "challeng": [0, 2], "we": 0, "ll": 0, "address": 0, "A": [0, 2, 3], "practic": [0, 2, 8], "approach": [0, 7], "an": 0, "open": [0, 2, 5], "sourc": [0, 2], "book": 0, "note": [0, 3, 5], "perspect": 0, "who": 0, "thi": 0, "i": [0, 3], "For": 0, "outcom": 0, "prerequisit": 0, "set": 0, "up": 0, "your": 0, "environ": 0, "code": [0, 5], "repositori": 0, "python": 0, "setup": [0, 3, 5], "api": [0, 7, 8], "kei": [0, 4, 6], "configur": 0, "troubleshoot": 0, "common": [0, 7], "issu": 0, "about": 0, "author": 0, "": 0, "prefac": [1, 2], "refer": [1, 3, 4, 5, 6, 7, 8], "tame": 2, "llm": [2, 4, 7], "guid": 2, "pitfal": [2, 7], "softwar": [2, 4], "chapter": 2, "1": [2, 6, 7], "2": [2, 6, 7], "wrestl": [2, 8], "structur": [2, 8], "output": [2, 6, 8], "3": [2, 6, 7], "input": 2, "data": [2, 3, 7], "4": [2, 6, 7], "size": [2, 6], "length": [2, 6], "limit": [2, 3, 6], "5": [2, 7], "The": [2, 4], "eval": [2, 4, 7], "gap": [2, 4], "6": [2, 7], "safeti": [2, 7], "concern": 2, "7": 2, "prefer": [2, 3], "base": [2, 3, 4, 6, 7], "align": [2, 3], "8": 2, "break": [2, 5], "free": [2, 5], "from": [2, 3, 5], "cloud": [2, 5], "provid": [2, 5, 8], "9": 2, "cost": [2, 6], "factor": [2, 7], "10": 2, "frontier": 2, "appendix": 2, "tool": [2, 4, 5, 7, 8], "resourc": 2, "citat": [2, 3, 4, 5, 6, 7, 8], "raw": 3, "capabl": 3, "On": 3, "misalign": 3, "languag": 3, "model": [3, 4, 5, 6], "human": 3, "supervis": 3, "fine": 3, "tune": 3, "sft": 3, "augment": 3, "post": 3, "train": 3, "answer": 3, "collaps": 3, "fake": 3, "case": [3, 5, 7], "studi": [3, 5, 7], "polici": [3, 7], "experiment": 3, "deliver": 3, "smollm2": 3, "dataset": [3, 4, 7], "synthet": 3, "gener": [3, 4, 5, 6, 7], "user": [3, 7, 8], "prompt": [3, 6, 8], "reject": 3, "respons": 3, "chosen": 3, "dpo": 3, "optim": 3, "prepar": 3, "vibe": 3, "check": 3, "evalu": [3, 4, 7], "discuss": [3, 6, 8], "conclus": [3, 4, 6, 7, 8], "non": 4, "determinist": 4, "machin": 4, "emerg": 4, "properti": 4, "problem": [4, 6, 8], "statement": [4, 6, 8], "tradit": 4, "v": [4, 5], "design": [4, 7], "applic": 4, "test": 4, "requir": 4, "matrix": 4, "conceptu": 4, "overview": 4, "consider": [4, 6], "metric": 4, "task": 4, "benchmark": [4, 7], "leaderboard": 4, "lightev": 4, "mmlu": 4, "econometr": 4, "sampl": [4, 7], "famili": 4, "us": 4, "langsmith": 4, "promptfoo": 4, "comparison": [4, 5, 6, 8], "local": 5, "caution": 5, "serv": 5, "llama": 5, "cpp": 5, "llamafil": 5, "ollama": [5, 8], "lama": 5, "ui": 5, "lm": 5, "studio": 5, "jan": 5, "webui": 5, "openwebui": 5, "privat": [5, 7], "document": 5, "object": 5, "instruct": 5, "implement": [5, 7], "what": 6, "ar": 6, "token": 6, "across": 6, "chunk": 6, "contextu": 6, "link": 6, "long": 6, "form": 6, "step": 6, "write": 6, "templat": 6, "construct": 6, "dynam": 6, "paramet": 6, "report": 6, "exampl": 6, "usag": 6, "implic": 6, "futur": 6, "risk": 7, "ai": 7, "amplifi": 7, "exist": 7, "harm": 7, "novel": 7, "associ": 7, "autonom": 7, "exacerb": 7, "specif": [7, 8], "integr": 7, "bia": 7, "privaci": 7, "secur": 7, "guidanc": 7, "govern": 7, "organ": 7, "sector": 7, "openai": 7, "anthrop": 7, "googl": 7, "rubric": 7, "mlcommon": 7, "centr": 7, "porquoi": 7, "red": 7, "team": 7, "constitut": 7, "explain": 7, "xai": 7, "plan": 7, "phase": 7, "definit": 7, "research": [7, 8], "identif": 7, "framework": [7, 8], "architectur": 7, "select": 7, "go": 7, "market": 7, "technic": 7, "compon": 7, "salad": 7, "bench": 7, "truthfulqa": 7, "harmbench": 7, "safebench": 7, "techniqu": [7, 8], "repres": 7, "layer": 7, "map": 7, "rule": 7, "filter": 7, "custom": 7, "moder": 7, "bad": 7, "good": 7, "guard": 7, "mistral": 7, "judg": 7, "valid": 7, "takeawai": 7, "need": 8, "solut": 8, "strategi": 8, "One": 8, "shot": 8, "json": 8, "mode": 8, "langchain": 8, "outlin": 8, "compar": 8, "best": 8, "ongo": 8, "debat": 8, "acknowledg": 8}, "envversion": {"sphinx.domains.c": 2, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 8, "sphinx.domains.index": 1, "sphinx.domains.javascript": 2, "sphinx.domains.math": 2, "sphinx.domains.python": 3, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinx.ext.intersphinx": 1, "sphinxcontrib.bibtex": 9, "sphinx": 57}, "alltitles": {"Introduction": [[0, "introduction"], [3, "introduction"], [3, "id30"], [4, "introduction"], [5, "introduction"], [7, "introduction"], [8, "introduction"]], "Contents": [[0, "contents"], [3, "contents"], [4, "contents"], [5, "contents"], [6, "contents"], [7, "contents"], [8, "contents"]], "Core Challenges We\u2019ll Address": [[0, "core-challenges-we-ll-address"]], "A Practical Approach": [[0, "a-practical-approach"]], "An Open Source Approach": [[0, "an-open-source-approach"]], "Open Source Book": [[0, "open-source-book"]], "A Note on Perspective": [[0, "a-note-on-perspective"]], "Who This Book Is For": [[0, "who-this-book-is-for"]], "Outcomes": [[0, "outcomes"]], "Prerequisites": [[0, "prerequisites"]], "Setting Up Your Environment": [[0, "setting-up-your-environment"]], "Code Repository": [[0, "code-repository"]], "Python Environment Setup": [[0, "python-environment-setup"]], "API Keys Configuration": [[0, "api-keys-configuration"]], "Troubleshooting Common Issues": [[0, "troubleshooting-common-issues"]], "About the Author(s)": [[0, "about-the-author-s"]], "Preface": [[1, "preface"], [2, "preface"]], "References": [[1, "references"], [3, "references"], [4, "references"], [5, "references"], [6, "references"], [7, "references"], [8, "references"]], "Taming LLMs": [[2, "taming-llms"]], "A Practical Guide to LLM Pitfalls with Open Source Software": [[2, "a-practical-guide-to-llm-pitfalls-with-open-source-software"]], "Chapter 1: Introduction": [[2, "chapter-1-introduction"]], "Chapter 2: Wrestling with Structured Output": [[2, "chapter-2-wrestling-with-structured-output"]], "Chapter 3: Input Data Challenge": [[2, "chapter-3-input-data-challenge"]], "Chapter 4: Output Size and Length Limitations": [[2, "chapter-4-output-size-and-length-limitations"]], "Chapter 5: The Evals Gap": [[2, "chapter-5-the-evals-gap"]], "Chapter 6: Safety Concerns": [[2, "chapter-6-safety-concerns"]], "Chapter 7: Preference-based Alignment": [[2, "chapter-7-preference-based-alignment"]], "Chapter 8: Breaking Free from Cloud Providers": [[2, "chapter-8-breaking-free-from-cloud-providers"]], "Chapter 9: The Cost Factor": [[2, "chapter-9-the-cost-factor"]], "Chapter 10: Frontiers": [[2, "chapter-10-frontiers"]], "Appendix A: Tools and Resources": [[2, "appendix-a-tools-and-resources"]], "Citation": [[2, "citation"], [3, "citation"], [4, "citation"], [5, "citation"], [6, "citation"], [7, "citation"], [8, "citation"]], "Preference-Based Alignment": [[3, "preference-based-alignment"]], "From Raw Capabilities to Preference Alignment": [[3, "from-raw-capabilities-to-preference-alignment"]], "On the Misalignment of Language Models": [[3, "on-the-misalignment-of-language-models"]], "Aligning Language Models with Human Preferences": [[3, "aligning-language-models-with-human-preferences"]], "Supervised Fine-Tuning (SFT) for Model Alignment": [[3, "supervised-fine-tuning-sft-for-model-alignment"]], "Augmenting SFT with Human Preferences": [[3, "augmenting-sft-with-human-preferences"]], "Is Post-Training the Answer?": [[3, "is-post-training-the-answer"]], "Limitations": [[3, "limitations"]], "Model Collapse": [[3, "model-collapse"]], "Faking Alignment": [[3, "faking-alignment"]], "Case Study: Aligning a Language Model to a Policy": [[3, "case-study-aligning-a-language-model-to-a-policy"]], "Experimental Setup": [[3, "experimental-setup"]], "Deliverables": [[3, "deliverables"]], "A Note on smolLM2 Models": [[3, "a-note-on-smollm2-models"]], "Policy": [[3, "policy"]], "Preference Dataset - Synthetic Dataset Generation": [[3, "preference-dataset-synthetic-dataset-generation"]], "User Prompts": [[3, "user-prompts"]], "Rejected Responses": [[3, "rejected-responses"]], "Chosen Responses": [[3, "chosen-responses"]], "Generate DPO Dataset": [[3, "generate-dpo-dataset"]], "DPO-Based Optimization": [[3, "dpo-based-optimization"]], "Data Preparation": [[3, "data-preparation"]], "Fine-Tuning": [[3, "fine-tuning"]], "Vibe Check": [[3, "vibe-check"]], "Alignment Evaluation": [[3, "alignment-evaluation"]], "Discussion and Conclusions": [[3, "discussion-and-conclusions"]], "The Evals Gap": [[4, "the-evals-gap"]], "Non-Deterministic Generative Machines": [[4, "non-deterministic-generative-machines"]], "Emerging Properties": [[4, "emerging-properties"]], "Problem Statement": [[4, "problem-statement"], [6, "problem-statement"], [8, "problem-statement"]], "Evals of Traditional Software vs LLMs": [[4, "evals-table"]], "Evals Design": [[4, "evals-design"]], "LLM Application Testing Requirements Matrix": [[4, "validation-requirements"]], "Conceptual Overview": [[4, "conceptual-overview"]], "Design Considerations": [[4, "design-considerations"]], "Metrics": [[4, "metrics"]], "Key Metrics for Evaluating Generative Tasks": [[4, "key-metrics"]], "Evaluators": [[4, "evaluators"]], "Model-Based Evaluation": [[4, "model-based-evaluation"]], "Evaluating Evaluators": [[4, "evaluating-evaluators"]], "Benchmarks and Leaderboards": [[4, "benchmarks-and-leaderboards"]], "Tools": [[4, "tools"]], "LightEval": [[4, "lighteval"]], "MMLU Econometrics Task Dataset sample": [[4, "mmlu-econometrics"]], "Model Families Evaluated Using LightEval": [[4, "model-families"]], "LangSmith": [[4, "langsmith"]], "PromptFoo": [[4, "promptfoo"]], "Comparison": [[4, "comparison"], [5, "comparison"], [5, "id14"]], "Comparison of Lighteval, LangSmith, and Promptfoo": [[4, "tool-comparison"]], "Conclusion": [[4, "conclusion"], [6, "conclusion"], [7, "conclusion"], [8, "conclusion"]], "Breaking Free from Cloud Providers": [[5, "breaking-free-from-cloud-providers"]], "Local Models": [[5, "local-models"]], "Notes of Caution": [[5, "notes-of-caution"]], "Local Tools": [[5, "local-tools"]], "Serving Models": [[5, "serving-models"]], "LLama.cpp": [[5, "llama-cpp"]], "Llamafile": [[5, "llamafile"]], "Ollama": [[5, "ollama"], [8, "ollama"]], "lama.cpp vs Ollama vs Llamafile Comparison": [[5, "feature-comparison-local"]], "UI": [[5, "ui"]], "LM Studio": [[5, "lm-studio"]], "Jan": [[5, "jan"]], "Open WebUI": [[5, "open-webui"]], "LM Studio vs Jan vs OpenWebUI Comparison": [[5, "feature-comparison-ui"]], "Case Study: Private Code Documentation Generator": [[5, "case-study-private-code-documentation-generator"]], "Objectives": [[5, "objectives"]], "Model Setup": [[5, "model-setup"]], "Instructions": [[5, "instructions"]], "Implementation": [[5, "implementation"]], "Output Size Limit": [[6, "output-size-limit"]], "What are Token Limits?": [[6, "what-are-token-limits"]], "Token Cost and Length Limitation Comparison Across Key Models": [[6, "token-cost-table"]], "Content Chunking with Contextual Linking": [[6, "content-chunking-with-contextual-linking"]], "Generating long-form content": [[6, "generating-long-form-content"]], "Step 1: Chunking the Content": [[6, "step-1-chunking-the-content"]], "Step 2: Writing the Base Prompt Template": [[6, "step-2-writing-the-base-prompt-template"]], "Step 3: Constructing Dynamic Prompt Parameters": [[6, "step-3-constructing-dynamic-prompt-parameters"]], "Step 4: Generating the Report": [[6, "step-4-generating-the-report"]], "Example Usage": [[6, "example-usage"]], "Discussion": [[6, "discussion"], [8, "discussion"]], "Implications": [[6, "implications"]], "Future Considerations": [[6, "future-considerations"]], "Safety": [[7, "safety"]], "Safety Risks": [[7, "safety-risks"]], "General AI Safety Risks": [[7, "general-ai-safety-risks"]], "Amplified Existing Harms and Novel Risks": [[7, "amplified-existing-harms-and-novel-risks"]], "Risks Associated with Autonomous AI": [[7, "risks-associated-with-autonomous-ai"]], "Exacerbating Factors": [[7, "exacerbating-factors"]], "LLMs Specific Safety Risks": [[7, "llms-specific-safety-risks"]], "Data Integrity and Bias": [[7, "data-integrity-and-bias"]], "Privacy and Security": [[7, "privacy-and-security"]], "Guidance": [[7, "guidance"]], "Governments & Organizations": [[7, "governments-organizations"]], "Private Sector": [[7, "private-sector"]], "OpenAI": [[7, "openai"]], "Anthropic": [[7, "anthropic"]], "Google": [[7, "google"]], "Rubrics": [[7, "rubrics"]], "MLCommons AI Safety Benchmark": [[7, "mlcommons-ai-safety-benchmark"]], "Centre for the Governance of AI Rubric": [[7, "centre-for-the-governance-of-ai-rubric"]], "Porquoi": [[7, "porquoi"]], "Approaches": [[7, "approaches"]], "Red Teaming": [[7, "red-teaming"]], "Constitutional AI": [[7, "constitutional-ai"]], "Explainable AI (XAI)": [[7, "explainable-ai-xai"]], "Designing a Safety Plan": [[7, "designing-a-safety-plan"]], "Phase 1. Policy Definition": [[7, "phase-1-policy-definition"]], "Phase 2. User Research & Risk Identification": [[7, "phase-2-user-research-risk-identification"]], "Phase 3. Evaluation Framework": [[7, "phase-3-evaluation-framework"]], "Phase 4. Safety Architecture Design": [[7, "phase-4-safety-architecture-design"]], "Phase 5. Implementation & Tools Selection": [[7, "phase-5-implementation-tools-selection"]], "Phase 6. Go-to-Market": [[7, "phase-6-go-to-market"]], "Common Pitfalls": [[7, "common-pitfalls"]], "Technical Implementation Components": [[7, "technical-implementation-components"]], "Benchmarks & Datasets": [[7, "benchmarks-datasets"]], "SALAD-Bench": [[7, "salad-bench"]], "TruthfulQA": [[7, "truthfulqa"]], "HarmBench": [[7, "harmbench"]], "SafeBench": [[7, "safebench"]], "Tools & Techniques": [[7, "tools-techniques"]], "Representative Safety Layer Risk Map.": [[7, "safety-layer-table"]], "Rules-Based Safety Filtering": [[7, "rules-based-safety-filtering"]], "Rules-Based Safety Filtering Tools.": [[7, "safety-layer-tools"]], "LLM-Based Safety Filtering": [[7, "llm-based-safety-filtering"]], "Custom Moderation": [[7, "custom-moderation"]], "Case Study: Implementing a Safety Filter": [[7, "case-study-implementing-a-safety-filter"]], "Evals Dataset": [[7, "evals-dataset"]], "Bad Samples": [[7, "bad-samples"]], "Good Samples": [[7, "good-samples"]], "Safety Filters": [[7, "safety-filters"]], "LLM-Guard": [[7, "llm-guard"]], "Mistral Moderation API": [[7, "mistral-moderation-api"]], "OpenAI Moderation API": [[7, "openai-moderation-api"]], "Custom Judge Validator": [[7, "custom-judge-validator"]], "Benchmarking": [[7, "benchmarking"]], "Takeaways": [[7, "takeaways"]], "Wrestling with Structured Output": [[8, "wrestling-with-structured-output"]], "User Needs": [[8, "user-needs"]], "Solutions": [[8, "solutions"]], "Strategies": [[8, "strategies"]], "Techniques and Tools": [[8, "techniques-and-tools"]], "One-Shot Prompts": [[8, "one-shot-prompts"]], "Structured Output with Provider-Specific APIs": [[8, "structured-output-with-provider-specific-apis"]], "JSON Mode": [[8, "json-mode"]], "LangChain": [[8, "langchain"]], "Outlines": [[8, "outlines"]], "Comparing Solutions": [[8, "comparing-solutions"]], "Structured Output Frameworks Comparison": [[8, "structured-output-frameworks"]], "Best Practices": [[8, "best-practices"]], "Research and Ongoing Debate": [[8, "research-and-ongoing-debate"]], "Acknowledgements": [[8, "acknowledgements"]]}, "indexentries": {}})
\ No newline at end of file
+Search.setIndex({"docnames": ["markdown/intro", "markdown/preface", "markdown/toc", "notebooks/alignment", "notebooks/evals", "notebooks/local", "notebooks/output_size_limit", "notebooks/safety", "notebooks/structured_output"], "filenames": ["markdown/intro.md", "markdown/preface.md", "markdown/toc.md", "notebooks/alignment.ipynb", "notebooks/evals.ipynb", "notebooks/local.ipynb", "notebooks/output_size_limit.ipynb", "notebooks/safety.ipynb", "notebooks/structured_output.ipynb"], "titles": ["2. Introduction", "1. Preface", "Taming LLMs", "7. Preference-Based Alignment", "5. The Evals Gap", "8. Breaking Free from Cloud Providers", "3. Output Size Limit", "6. Safety", "4. Wrestling with Structured Output"], "terms": {"am": [0, 1, 7], "alwai": [0, 3, 4, 8], "do": [0, 3, 4, 5, 6, 7, 8], "which": [0, 3, 4, 5, 6, 7, 8], "cannot": [0, 3, 4, 5, 7], "order": [0, 3, 4, 7, 8], "mai": [0, 1, 3, 4, 5, 6, 7, 8], "learn": [0, 3, 4, 7], "how": [0, 1, 3, 4, 5, 6, 7, 8], "pablo": [0, 4], "picasso": 0, "In": [0, 3, 4, 5, 6, 7, 8], "recent": [0, 3, 4, 7, 8], "year": [0, 2, 3, 4, 5, 6, 7, 8], "larg": [0, 1, 2, 3, 4, 5, 6, 7, 8], "languag": [0, 1, 2, 4, 5, 6, 7, 8], "model": [0, 1, 2, 7, 8], "llm": [0, 1, 3, 6, 8], "have": [0, 1, 3, 4, 5, 6, 7, 8], "emerg": [0, 3, 7, 8], "transform": [0, 1, 3, 4, 7, 8], "forc": [0, 4, 8], "technologi": [0, 1, 4, 6, 7, 8], "promis": [0, 3, 4, 7], "revolution": [0, 7], "build": [0, 2, 3, 4, 5, 6, 7, 8], "product": [0, 1, 2, 3, 4, 5, 7, 8], "interact": [0, 3, 4, 5, 6, 7, 8], "comput": [0, 3, 4, 5, 6, 7, 8], "from": [0, 1, 4, 6, 7, 8], "chatgpt": [0, 3, 5, 8], "github": [0, 2, 3, 4, 5, 6, 7, 8], "copilot": 0, "claud": [0, 3, 4, 5, 6, 7], "artifact": 0, "system": [0, 3, 4, 5, 6, 7, 8], "captur": [0, 1, 3, 4, 7], "public": [0, 3, 4, 7], "imagin": 0, "spark": 0, "gold": [0, 3, 4, 7], "rush": 0, "ai": [0, 3, 4, 5, 8], "power": [0, 2, 3, 4, 5, 6, 7, 8], "applic": [0, 1, 2, 3, 5, 6, 7, 8], "howev": [0, 3, 4, 5, 6, 7, 8], "beneath": 0, "surfac": [0, 4], "technolog": [0, 1, 4, 7], "revolut": 0, "li": [0, 3, 4, 7], "complex": [0, 1, 3, 4, 5, 6, 7, 8], "landscap": [0, 3, 4], "practition": [0, 1, 4, 5], "must": [0, 3, 4, 5, 6, 7], "navig": [0, 2, 4, 7], "focus": [0, 3, 4, 5, 6, 7, 8], "bring": [0, 3, 5], "awar": [0, 3, 4, 6, 7], "limit": [0, 1, 4, 5, 7, 8], "har": [0, 2, 4, 6], "solut": [0, 2, 4, 5, 6, 7], "overcom": [0, 4, 6], "them": [0, 1, 3, 4, 5, 6, 7, 8], "robust": [0, 3, 4, 5, 6, 7, 8], "It": [0, 3, 4, 5, 6, 7, 8], "offer": [0, 3, 4, 5, 6, 7, 8], "critic": [0, 2, 3, 4, 6, 7, 8], "implement": [0, 2, 3, 4, 5, 6, 8], "back": [0, 4, 7, 8], "reproduc": [0, 1, 2, 4, 5], "exampl": [0, 1, 2, 3, 4, 5, 7, 8], "while": [0, 1, 2, 3, 4, 5, 6, 7, 8], "mani": [0, 1, 3, 4, 5, 6, 7, 8], "resourc": [0, 3, 4, 5, 6, 7], "cover": [0, 3, 4, 5, 6, 7], "capabl": [0, 1, 2, 4, 5, 6, 7, 8], "specif": [0, 3, 4, 5, 6], "hidden": [0, 3, 7], "pitfal": [0, 1, 3, 4, 5, 6, 8], "engin": [0, 1, 2, 3, 4, 5, 7, 8], "technic": [0, 1, 2, 3, 4, 5, 6, 8], "manag": [0, 1, 2, 4, 5, 6, 7, 8], "face": [0, 3, 4, 5, 7], "when": [0, 1, 2, 3, 4, 5, 6, 7, 8], "comprehens": [0, 2, 3, 4, 5, 6, 7, 8], "guid": [0, 1, 3, 4, 5, 6, 7, 8], "leverag": [0, 3, 4, 6, 7, 8], "battl": [0, 2], "test": [0, 2, 3, 5, 7, 8], "tool": [0, 1, 3, 6], "throughout": [0, 4, 5, 6, 7, 8], "tackl": [0, 3, 4, 7], "follow": [0, 3, 4, 5, 6, 7, 8], "non": [0, 3, 5, 7, 8], "exhaust": 0, "list": [0, 3, 4, 5, 6, 7, 8], "structur": [0, 3, 4, 5, 6, 7], "un": 0, "reliabl": [0, 1, 3, 4, 5, 7, 8], "struggl": [0, 1, 3, 4, 7, 8], "maintain": [0, 1, 3, 4, 5, 6, 7, 8], "consist": [0, 1, 3, 4, 5, 6, 7, 8], "output": [0, 1, 3, 4, 5, 7], "format": [0, 3, 4, 5, 6, 7, 8], "complic": [0, 7], "integr": [0, 1, 3, 4, 5, 7, 8], "larger": [0, 3, 4, 5, 6, 7, 8], "make": [0, 3, 4, 5, 6, 7, 8], "error": [0, 3, 4, 7, 8], "handl": [0, 2, 3, 4, 5, 6, 7, 8], "more": [0, 1, 3, 4, 5, 6, 7, 8], "size": [0, 3, 4, 5, 7, 8], "length": [0, 3, 4, 5, 8], "constraint": [0, 1, 3, 4, 6, 7, 8], "strict": [0, 5, 7, 8], "token": [0, 1, 3, 4, 5, 7, 8], "both": [0, 3, 4, 5, 7], "input": [0, 3, 4, 5, 6, 7, 8], "requir": [0, 3, 5, 6, 7, 8], "care": [0, 3, 4, 7, 8], "chunk": [0, 3], "strategi": [0, 3, 4, 5, 6, 7], "long": [0, 1, 3, 4, 7, 8], "form": [0, 3, 4, 5, 7, 8], "effect": [0, 1, 3, 4, 6, 7, 8], "tradit": [0, 3, 7], "softwar": [0, 1, 3, 5, 6, 7, 8], "methodologi": [0, 3, 4, 7, 8], "break": [0, 1, 3, 4, 6, 7], "down": [0, 1, 4, 5, 6, 7], "deal": [0, 3, 5], "determinist": [0, 8], "gener": [0, 1, 5, 8], "new": [0, 2, 3, 4, 5, 6, 7, 8], "hallucin": [0, 1, 3, 4, 7, 8], "These": [0, 3, 4, 5, 6, 7, 8], "can": [0, 1, 3, 4, 5, 6, 7, 8], "plausibl": [0, 7], "sound": [0, 7], "entir": [0, 4, 5, 6, 8], "fabric": [0, 4, 7], "inform": [0, 3, 4, 5, 6, 7, 8], "creat": [0, 1, 3, 4, 5, 6, 7, 8], "signific": [0, 3, 4, 5, 6, 7, 8], "risk": [0, 1, 3, 4, 6], "safeti": [0, 3, 4, 8], "align": [0, 4, 5, 6, 7, 8], "harm": [0, 3, 4, 5], "bias": [0, 3, 4, 7, 8], "inappropri": [0, 3, 7], "safeguard": [0, 4, 7], "monitor": [0, 3, 4, 5, 7], "ensur": [0, 3, 4, 5, 6, 7, 8], "safe": [0, 3, 4, 7, 8], "deploy": [0, 3, 4, 7, 8], "cost": [0, 3, 4, 5, 7, 8], "optim": [0, 1, 4, 5, 6, 7], "The": [0, 1, 3, 6, 7, 8], "financi": [0, 1, 3, 4, 6, 7, 8], "oper": [0, 3, 4, 5, 6, 7, 8], "base": [0, 1, 5, 8], "quickli": [0, 3, 6], "becom": [0, 3, 4, 5, 7, 8], "prohibit": [0, 3, 4, 5], "without": [0, 1, 3, 4, 5, 6, 7, 8], "observ": [0, 3, 4, 5, 7, 8], "vendor": [0, 4], "lock": [0, 3], "cloud": [0, 3, 4, 7, 8], "provid": [0, 3, 4, 6, 7], "depend": [0, 3, 4, 5, 8], "through": [0, 1, 2, 3, 4, 5, 6, 7, 8], "proprietari": [0, 3, 5, 7, 8], "infrastructur": [0, 5], "difficult": [0, 3, 4, 7], "switch": [0, 5], "self": [0, 3, 4, 5, 7], "host": [0, 4, 5, 7], "take": [0, 2, 3, 4, 5, 6, 7, 8], "hand": [0, 6, 7, 8], "focu": [0, 2, 3, 4, 5, 6, 7, 8], "access": [0, 3, 4, 5, 6, 7, 8], "all": [0, 1, 3, 4, 5, 6, 7, 8], "ar": [0, 1, 3, 4, 5, 7, 8], "fulli": [0, 3, 4, 6, 7], "document": [0, 3, 4, 5, 6, 7, 8], "allow": [0, 4, 5, 6, 7, 8], "reader": [0, 2], "replic": [0, 4, 7, 8], "result": [0, 3, 4, 6, 7, 8], "exactli": [0, 4, 8], "design": [0, 1, 3, 5, 6, 8], "run": [0, 3, 4, 5, 7, 8], "consum": [0, 3, 4, 5, 7, 8], "grade": [0, 3, 4, 5, 7], "hardwar": [0, 3, 4], "expens": [0, 3, 4, 7], "avail": [0, 3, 4, 5, 6, 7, 8], "notebook": [0, 3, 8], "modifi": [0, 3, 4, 7], "extend": [0, 3, 4, 5, 8], "built": [0, 4, 5, 7, 8], "us": [0, 1, 3, 5, 6, 7, 8], "free": [0, 1, 3, 4, 7], "everyon": [0, 4, 5], "minim": [0, 3, 4, 5, 7, 8], "framework": [0, 3, 4], "wai": [0, 3, 4, 5, 6, 7, 8], "priorit": [0, 3, 4, 5, 7], "transpar": [0, 3, 4, 7, 8], "visibl": [0, 4], "being": [0, 3, 4, 5, 7], "better": [0, 2, 3, 4, 5, 6, 7], "understand": [0, 1, 2, 3, 4, 5, 6, 7, 8], "custom": [0, 3, 4, 5], "flexibl": [0, 4, 5, 6, 7, 8], "adapt": [0, 3, 4, 5, 6, 7], "case": [0, 4, 6, 8], "unlik": [0, 3, 4, 5], "black": [0, 3], "box": [0, 5], "commerci": [0, 3, 4, 7, 8], "most": [0, 3, 4, 5, 6, 7, 8], "freeli": [0, 8], "foster": [0, 3, 4, 7, 8], "reduc": [0, 3, 4, 5, 6, 7, 8], "independ": [0, 4, 7, 8], "freedom": [0, 5, 8], "architectur": [0, 3, 4, 5, 6, 8], "decis": [0, 3, 4, 5, 7, 8], "keep": [0, 3, 4, 5, 6, 7], "principl": [0, 3, 4, 5, 7], "itself": [0, 3, 4, 5, 7], "live": [0, 1, 4, 7], "evolv": [0, 3, 4, 6, 7], "chang": [0, 3, 4, 5, 7], "encourag": [0, 3, 4, 7, 8], "report": [0, 3, 4, 5, 7, 8], "suggest": [0, 3, 4, 7, 8], "improv": [0, 3, 4, 5, 6, 7, 8], "contribut": [0, 4, 6, 7], "via": [0, 3, 4, 5, 7, 8], "pull": [0, 5], "request": [0, 3, 4, 5, 6, 7, 8], "share": [0, 3, 4, 5, 7, 8], "own": [0, 3, 4, 6, 7], "experi": [0, 3, 4, 5, 6, 7, 8], "commun": [0, 3, 4, 7, 8], "propos": [0, 4, 7], "chapter": [0, 3, 4, 5, 6, 7, 8], "section": [0, 3, 4, 5, 6, 7, 8], "found": [0, 3, 4, 5, 8], "http": [0, 1, 2, 3, 4, 5, 6, 7, 8], "com": [0, 2, 3, 4, 5, 6, 7, 8], "souzatharsi": [0, 2, 3, 4, 5, 6, 7, 8], "tamingllm": [0, 2, 3, 4, 5, 6, 7, 8], "whether": [0, 3, 4, 6, 7, 8], "you": [0, 1, 3, 4, 5, 6, 7, 8], "ve": [0, 5], "typo": [0, 7], "want": [0, 1, 3, 5, 6, 7, 8], "welcom": 0, "look": [0, 2, 3, 4, 5, 7], "our": [0, 1, 3, 4, 5, 6, 7, 8], "goal": [0, 1, 3, 4, 6, 7], "discourag": 0, "enabl": [0, 3, 4, 5, 6, 7, 8], "By": [0, 1, 2, 3, 4, 6, 7, 8], "upfront": [0, 2], "equip": [0, 2, 4, 7], "avoid": [0, 3, 4, 7, 8], "current": [0, 2, 3, 4, 6, 7, 8], "discours": [0, 2], "around": [0, 2, 3, 4, 6, 7, 8], "tend": [0, 2, 4, 7], "toward": [0, 3, 4, 7, 8], "extrem": [0, 3, 4, 7], "either": [0, 3, 4, 6, 7], "uncrit": 0, "enthusiasm": 0, "wholesal": [0, 4], "dismiss": 0, "differ": [0, 3, 4, 5, 6, 7, 8], "rather": [0, 1, 3, 4, 5, 7], "than": [0, 1, 3, 4, 5, 7], "theoret": [0, 3], "examin": [0, 3, 4, 5, 6, 7, 8], "first": [0, 1, 3, 4, 5, 6, 7, 8], "everi": [0, 4, 7], "concept": [0, 3, 4, 7], "illustr": [0, 3, 4, 6, 7, 8], "execut": [0, 4, 5, 7], "immedi": [0, 3, 4, 5], "analysi": [0, 1, 3, 4, 5, 6, 7], "balanc": [0, 3, 4, 5, 6, 7, 8], "help": [0, 3, 4, 5, 6, 7, 8], "intend": [0, 4, 7], "develop": [0, 1, 3, 4, 5, 6, 7, 8], "step": [0, 1, 3, 4, 5, 7, 8], "insight": [0, 3, 4, 5, 6, 7, 8], "along": [0, 3, 4, 5, 7], "guidanc": [0, 3, 8], "could": [0, 1, 3, 4, 5, 6, 7, 8], "derail": 0, "project": [0, 3, 4, 5, 7], "earli": [0, 3, 4, 7, 8], "befor": [0, 3, 4, 5, 7, 8], "thei": [0, 1, 3, 4, 5, 6, 7, 8], "costli": [0, 4, 7], "problem": [0, 1, 2, 3, 5, 7], "too": [0, 1, 3, 4, 6, 7], "late": [0, 3, 7], "lifecycl": [0, 7], "lead": [0, 1, 3, 4, 6, 7, 8], "genai": [0, 1, 3, 7], "initi": [0, 1, 3, 4, 5, 6, 7, 8], "leader": [0, 4], "advoc": [0, 7], "anyon": [0, 7], "seek": [0, 4, 5, 7], "work": [0, 1, 3, 4, 5, 6, 7, 8], "typic": [0, 3, 4, 6, 7, 8], "job": [0, 4, 5, 7], "role": [0, 3, 4, 5, 6, 7, 8], "platform": [0, 4, 5, 6, 7, 8], "backend": [0, 3, 4], "exist": [0, 3, 4, 5], "ml": [0, 7], "transit": [0, 4, 5, 6, 8], "overse": 0, "motiv": [0, 3, 4, 8], "need": [0, 3, 4, 5, 6, 7], "readi": [0, 4, 7], "desir": [0, 3, 4, 8], "perform": [0, 3, 4, 6, 7, 8], "after": [0, 1, 3, 4, 5, 6, 7, 8], "read": [0, 3, 4, 6, 7, 8], "implic": [0, 1, 3, 4, 7], "recommend": [0, 3, 4, 6, 7, 8], "abl": [0, 3, 4, 6, 8], "deploi": [0, 3, 4, 5, 6, 7], "proper": [0, 3, 5, 7, 8], "realist": [0, 3, 7], "effort": [0, 4, 7, 8], "estim": [0, 4, 7], "impact": [0, 3, 4, 5, 6, 7, 8], "timelin": 0, "To": [0, 3, 4, 5, 6, 7, 8], "should": [0, 3, 4, 6, 7, 8], "basic": [0, 3, 4, 5, 6, 7], "program": [0, 4, 5], "knowledg": [0, 3, 4, 5, 7], "introductori": [0, 1, 2], "langchain": [0, 4, 6], "e": [0, 1, 3, 4, 5, 6, 7, 8], "g": [0, 3, 4, 5, 6, 7, 8], "chat": [0, 3, 4, 5, 6, 7, 8], "prompt": [0, 4, 7], "templat": [0, 4], "openai": [0, 3, 4, 5, 8], "anthrop": [0, 3, 8], "similar": [0, 3, 4, 5, 8], "dive": 0, "here": [0, 2, 3, 4, 5, 6, 7, 8], "get": [0, 3, 4, 5, 6, 7, 8], "start": [0, 3, 4, 5, 7, 8], "clone": [0, 3], "companion": 0, "git": 0, "cd": 0, "activ": [0, 3, 4, 7], "virtual": [0, 4], "m": [0, 3, 4, 5, 7, 8], "venv": 0, "tame": [0, 3, 4, 5, 6, 7, 8], "env": [0, 3, 4, 6, 7, 8], "bin": [0, 5], "On": [0, 4, 5, 8], "window": [0, 2, 4, 5], "script": [0, 5], "try": [0, 1, 3, 4, 7, 8], "contain": [0, 3, 4, 5, 6, 7, 8], "possibl": [0, 3, 4, 5, 7, 8], "includ": [0, 1, 3, 4, 5, 6, 7, 8], "necessari": [0, 3, 4, 6, 7], "instal": [0, 3, 4, 5, 8], "go": [0, 3, 4, 6, 8], "feel": [0, 5], "prefer": [0, 4, 5, 7, 8], "packag": [0, 4, 5, 8], "pip": [0, 3, 4, 5, 8], "poetri": [0, 7], "file": [0, 3, 4, 5, 6, 7, 8], "root": [0, 3], "directori": [0, 4, 5], "add": [0, 3, 4, 5, 6, 7], "other": [0, 3, 4, 5, 6, 7, 8], "sensit": [0, 3, 4, 5, 7], "openai_api_kei": [0, 3], "your_openai_api_key_her": 0, "never": [0, 8], "commit": [0, 3, 4, 7], "version": [0, 3, 4, 5, 7, 8], "control": [0, 1, 3, 4, 5, 7, 8], "kept": [0, 4], "privat": [0, 4], "If": [0, 1, 3, 4, 5, 7, 8], "encount": [0, 2, 4, 7], "rate": [0, 3, 4, 7], "consid": [0, 3, 4, 6, 7, 8], "smaller": [0, 3, 4, 6, 8], "retri": [0, 8], "logic": [0, 1, 3, 4, 6, 7], "conflict": [0, 3, 4], "fresh": 0, "like": [0, 1, 3, 4, 5, 6, 7, 8], "check": [0, 4, 5, 7, 8], "page": [0, 4], "known": [0, 4, 7, 8], "now": [0, 1, 3, 4, 5, 6, 7, 8], "let": [0, 3, 4, 5, 6, 7, 8], "begin": [0, 4, 7, 8], "explor": [0, 1, 3, 4, 5, 7, 8], "dr": [0, 3], "tharsi": [0, 2, 3, 4, 5, 6, 7, 8], "souza": [0, 2, 3, 4, 5, 6, 7, 8], "scientist": [0, 1, 5, 7], "special": [0, 4, 5, 7, 8], "he": [0, 3, 4, 7], "lectur": 0, "columbia": 0, "univers": [0, 4, 5, 7], "master": [0, 5, 8], "scienc": [0, 3, 4, 7], "appli": [0, 3, 4, 5, 6, 7, 8], "analyt": 0, "incom": [0, 4], "head": [0, 3, 4, 6, 7], "equiti": [0, 4], "citadel": 0, "former": [0, 1, 4, 5], "senior": [0, 4], "vp": 0, "two": [0, 3, 4, 5, 6, 7, 8], "sigma": [0, 3], "invest": [0, 3, 4, 7, 8], "also": [0, 3, 4, 5, 6, 7, 8], "enjoi": 0, "mentor": 0, "under": [0, 3, 4, 7, 8], "repres": [0, 3, 4, 5, 8], "student": [0, 3, 5, 7], "profession": [0, 3, 4, 7, 8], "divers": [0, 3, 4, 6, 7, 8], "global": [0, 4, 7], "ecosystem": [0, 4], "With": [0, 3, 4, 5, 7], "over": [0, 2, 3, 4, 5, 6, 7, 8], "15": [0, 4, 7, 8], "deliv": [0, 4, 5], "across": [0, 1, 3, 4, 5, 7, 8], "startup": 0, "fortun": 0, "500": [0, 3, 4, 7], "compani": [0, 3, 4, 6, 7, 8], "numer": [0, 4, 7], "scholarli": 0, "frequent": [0, 4, 5, 8], "speaker": [0, 4], "academ": [0, 3, 4, 7], "busi": [0, 4, 5, 7], "confer": [0, 8], "ground": [0, 3, 4], "background": [0, 1, 4, 5, 6], "draw": [0, 3, 4, 7, 8], "scale": [0, 3, 4, 5, 7, 8], "stage": [0, 3, 7, 8], "major": [0, 3, 4, 7, 8], "institut": [0, 4, 5, 7], "well": [0, 3, 4, 5, 7, 8], "advis": [0, 3], "profit": [0, 4, 6, 7, 8], "organ": [0, 3, 4, 5, 6], "uniqu": [0, 3, 4, 7], "bridg": [0, 5, 7], "gap": [0, 1, 3, 7], "between": [0, 1, 3, 4, 5, 6, 7, 8], "potenti": [0, 1, 3, 4, 5, 6, 7, 8], "next": [0, 1, 3, 4, 5, 7, 8], "hold": [0, 3, 4], "ph": [0, 7], "d": [0, 3, 4, 5, 7, 8], "ucl": 0, "london": 0, "phil": [0, 7], "sc": 0, "b": [0, 4, 5, 7, 8], "tell": [1, 3, 7], "mere": [1, 4], "what": [1, 3, 4, 5, 7, 8], "someth": [1, 4, 5], "i": [1, 2, 4, 5, 6, 7, 8], "emanuel": [1, 3, 4, 7], "derman": 1, "an": [1, 2, 3, 4, 5, 6, 7, 8], "altern": [1, 3, 4, 5, 6, 7], "titl": [1, 2, 3, 4, 5, 6, 7, 8], "thi": [1, 2, 3, 4, 5, 6, 7, 8], "book": [1, 2, 4], "been": [1, 3, 4, 5, 7], "behav": 1, "badli": 1, "come": [1, 3, 4, 5, 6, 7, 8], "notic": [1, 3, 4, 7, 8], "parallel": [1, 3, 4, 5], "": [1, 3, 4, 5, 6, 7, 8], "semin": [1, 7], "2011": 1, "coincident": 1, "just": [1, 3, 4, 5, 6, 7, 8], "caution": 1, "against": [1, 3, 4, 5, 7], "treat": [1, 4, 7], "perfect": [1, 4, 5], "represent": [1, 4, 6, 7], "realiti": [1, 7], "aim": [1, 3, 4, 5, 6, 7, 8], "highlight": [1, 3, 4, 5, 6, 7, 8], "practic": [1, 3, 4, 5, 6, 7], "cours": [1, 4, 7], "bare": [1, 5], "fact": [1, 3, 4, 7], "actual": [1, 3, 4, 6, 7, 8], "physicist": 1, "legendari": 1, "author": [1, 2, 3, 4, 5, 6, 7, 8], "professor": 1, "quant": 1, "goldman": 1, "sach": 1, "scientif": [1, 3, 4], "fail": [1, 3, 4, 7], "we": [1, 3, 4, 5, 6, 7, 8], "mistak": [1, 7], "approxim": [1, 4, 8], "full": [1, 3, 4, 5, 7, 8], "assumpt": [1, 4, 7], "core": [1, 4, 5, 7], "premis": [1, 5], "hi": [1, 4, 7, 8], "aspect": [1, 3, 4, 6, 7, 8], "world": [1, 3, 4, 7, 8], "inher": [1, 2, 3, 4, 7, 8], "involv": [1, 3, 4, 5, 7, 8], "simplif": 1, "argu": [1, 7, 8], "crise": 1, "2008": 1, "crash": 1, "occur": [1, 3, 4, 7], "partli": 1, "becaus": [1, 3, 4, 7], "peopl": [1, 3, 4, 5, 7], "put": [1, 4], "much": [1, 3, 4, 5], "faith": 1, "mathemat": [1, 4], "recogn": [1, 3, 4, 7], "human": [1, 4, 6, 7, 8], "behavior": [1, 3, 4, 5, 7], "market": [1, 4, 6, 8], "dynam": [1, 3, 4, 7], "reason": [1, 3, 4, 5, 6, 7, 8], "Their": [1, 4, 8], "respons": [1, 4, 5, 6, 7, 8], "often": [1, 3, 4, 5, 6, 7, 8], "convinc": [1, 3], "probabilist": [1, 4], "train": [1, 4, 5, 7, 8], "data": [1, 4, 5, 6, 7, 8], "true": [1, 3, 4, 6, 7, 8], "even": [1, 3, 4, 5, 6, 7, 8], "though": [1, 3, 4, 5, 7, 8], "insist": 1, "machin": [1, 3, 5, 7, 8], "todai": [1, 5, 8], "grow": [1, 3, 4, 5, 7, 8], "pervas": [1, 7], "belief": [1, 5, 7], "solv": [1, 3, 4, 5, 7, 8], "ani": [1, 3, 4, 5, 6, 7, 8], "context": [1, 2, 3, 4, 5, 6, 7, 8], "content": 1, "wish": [1, 4], "user": [1, 4, 5, 6], "moreov": 1, "were": [1, 3, 4, 7, 8], "predict": [1, 3, 4, 5, 7, 8], "chatbot": [1, 3, 4, 5, 7], "twist": [1, 7], "wrap": [1, 5, 8], "further": [1, 3, 4, 6, 7, 8], "daili": [1, 7], "life": [1, 4, 5, 7], "workflow": [1, 4, 5, 7, 8], "affect": [1, 4, 5, 7], "decid": [1, 3, 4, 6], "action": [1, 3, 4, 6, 7], "coupl": 1, "lack": [1, 3, 4, 7, 8], "pose": [1, 3, 4, 6, 7, 8], "still": [1, 4, 5, 7], "figur": [1, 4, 8], "out": [1, 3, 4, 5, 6, 7, 8], "serv": [1, 3, 4, 6, 7, 8], "builder": [1, 5], "who": [1, 3, 4, 5, 6, 7, 8], "remain": [1, 3, 4, 5, 6, 7], "clear": [1, 3, 4, 7, 8], "ei": 1, "about": [1, 3, 4, 5, 6, 7, 8], "therefor": [1, 3, 4, 5, 7], "end": [1, 3, 4, 5, 6, 7, 8], "detail": [1, 3, 4, 5, 6, 7, 8], "python": [1, 2, 4, 5, 6, 7, 8], "code": [1, 2, 3, 4, 5, 7, 8], "diminish": [1, 3, 4], "promot": [1, 3, 4, 7], "nuanc": [1, 3, 4, 6, 7, 8], "acknowledg": [1, 4, 7], "within": [1, 3, 4, 6, 7, 8], "trustworthi": [1, 7], "taught": 1, "u": [1, 3, 4, 7, 8], "where": [1, 3, 4, 5, 6, 7, 8], "der11": 1, "why": [1, 3, 4, 7, 8], "confus": [1, 7], "illus": 1, "disast": [1, 4], "wall": [1, 5], "street": [1, 5], "press": [1, 4, 5], "isbn": [1, 3, 4], "9781439165010": 1, "url": [1, 2, 3, 4, 5, 6, 7, 8], "googl": [1, 4, 8], "co": [1, 3, 4, 5, 7], "uk": [1, 7], "id": [1, 4, 5, 7], "lke_cwm4wm8c": 1, "sign": [2, 4, 7], "up": [2, 3, 4, 5, 6, 7, 8], "receiv": [2, 3, 4, 6, 7, 8], "updat": [2, 3, 4, 5, 6, 7, 8], "abstract": [2, 4, 7, 8], "heavili": [2, 3, 4, 7, 8], "gloss": 2, "fundament": [2, 3, 4, 7, 8], "convers": [2, 3, 4, 5, 6, 7, 8], "kei": [2, 3, 5, 7, 8], "proven": 2, "yet": [2, 3, 4, 6, 7], "concret": [2, 7], "unstructur": [2, 8], "sidestep": 2, "misc": [2, 3, 4, 5, 6, 7, 8], "tharsistpsouza2024tamingllm": [2, 3, 4, 5, 6, 7, 8], "t": [2, 3, 4, 5, 6, 7, 8], "p": [2, 3, 4, 5, 6, 7, 8], "2024": [2, 3, 4, 5, 6, 7, 8], "journal": [2, 3, 4, 5, 6, 7, 8], "repositori": [2, 3, 4, 5, 6, 7, 8], "valu": [3, 4, 5, 6, 7, 8], "its": [3, 4, 5, 6, 7, 8], "privileg": 3, "abov": [3, 4, 7], "soon": [3, 8], "lose": [3, 4], "dwight": 3, "eisenhow": 3, "releas": [3, 4, 5, 7, 8], "3": [3, 4, 5, 8], "5": [3, 4, 5, 6, 8], "2022": [3, 4, 7], "mark": [3, 4, 7], "pivot": [3, 4], "moment": 3, "histori": [3, 4, 5], "artifici": [3, 4, 7], "intellig": [3, 4, 7], "five": [3, 4, 7], "dai": [3, 4, 7, 8], "launch": [3, 4, 7], "attract": [3, 4], "million": [3, 4], "month": [3, 4, 7], "becam": 3, "fastest": [3, 4, 7], "100": [3, 4, 5, 7, 8], "monthli": [3, 4], "rais": [3, 4, 6, 7], "intrigu": 3, "question": [3, 4, 5, 7, 8], "did": [3, 4, 8], "dramat": [3, 4, 5, 8], "predecessor": 3, "gpt": [3, 4, 5, 6, 7, 8], "had": [3, 4], "same": [3, 4, 5, 6, 7, 8], "number": [3, 4, 5, 6, 7, 8], "paramet": [3, 4, 5, 7, 8], "far": [3, 6, 7], "less": [3, 4, 5, 7], "attent": [3, 5], "arguabl": [3, 5], "feedback": [3, 4, 7, 8], "abil": [3, 4, 5, 7, 8], "least": [3, 4, 7], "ey": 3, "breakthrough": [3, 7], "demonstr": [3, 4, 5, 6, 7, 8], "crucial": [3, 5, 7, 8], "greater": [3, 4, 7], "process": [3, 4, 5, 6, 7, 8], "modern": [3, 4, 6, 8], "techniqu": [3, 4, 5, 6], "direct": [3, 4, 5, 7], "rafailov": 3, "et": [3, 4, 7, 8], "al": [3, 4, 7, 8], "present": [3, 4, 5, 6, 7, 8], "autom": [3, 4, 7, 8], "fashion": [3, 8], "open": [3, 4, 6, 7, 8], "sourc": [3, 4, 5, 6, 7, 8], "common": [3, 4, 5, 6, 8], "pre": [3, 4, 5, 7], "default": [3, 4, 5, 7, 8], "state": [3, 4, 5, 6, 7, 8], "art": [3, 4, 7], "object": [3, 4, 5, 7, 8], "given": [3, 4, 5, 6, 7, 8], "webpag": 3, "internet": [3, 4], "veri": [3, 4, 5, 7], "ask": [3, 4, 5, 7, 8], "instruct": [3, 4, 5, 6, 7, 8], "sai": [3, 8], "ouyang": [3, 7], "2": [3, 4, 5, 8], "explain": 3, "moon": 3, "land": [3, 4], "6": [3, 4, 5, 6, 8], "old": [3, 4], "import": [3, 4, 5, 6, 7, 8], "pipelin": [3, 4, 5, 7, 8], "pipe": [3, 7], "text": [3, 4, 5, 6, 7, 8], "gpt2": [3, 4], "msg": 3, "short": [3, 4, 6, 7, 8], "sentenc": [3, 4, 6, 7, 8], "_": [3, 4, 7, 8], "rang": [3, 4, 5, 6, 7, 8], "len": [3, 4, 5, 6, 7], "print": [3, 4, 5, 6, 7, 8], "f": [3, 4, 5, 6, 7, 8], "n": [3, 4, 5, 6, 7, 8], "1": [3, 4, 5, 8], "0": [3, 4, 5, 6, 7, 8], "generated_text": 3, "good": [3, 4, 5, 8], "idea": [3, 7], "one": [3, 4, 5, 6, 7, 8], "those": [3, 4, 6, 7, 8], "littl": [3, 4], "green": [3, 7], "dot": 3, "Then": [3, 4], "line": [3, 4, 5, 7], "later": [3, 4, 5, 7, 8], "re": [3, 4, 6, 7, 8], "alreadi": [3, 4], "movi": 3, "theori": [3, 4], "some": [3, 4, 5, 6, 7, 8], "mean": [3, 4, 5, 6, 7, 8], "word": [3, 4, 6, 8], "tepid": 3, "articl": [3, 4, 5, 6, 7], "sure": [3, 4, 6, 7, 8], "lunar": 3, "As": [3, 4, 5, 6, 7, 8], "see": [3, 4, 5, 7, 8], "coher": [3, 4, 6], "explan": [3, 4, 7, 8], "child": [3, 4, 7], "nonsens": [3, 7], "meander": 3, "unrel": [3, 4, 7], "topic": [3, 4, 5, 6, 7, 8], "simpl": [3, 4, 5, 6, 7, 8], "appropri": [3, 4, 5, 6, 7, 8], "young": [3, 4, 7], "instead": [3, 4, 5, 6, 7, 8], "address": [3, 4, 6, 7, 8], "issu": [3, 4, 6, 7, 8], "introduc": [3, 4, 5, 6, 7, 8], "rlhf": [3, 7], "intent": [3, 7], "wide": [3, 4, 5, 6, 7, 8], "task": [3, 5, 6, 7, 8], "fig": [3, 4, 5, 6, 7, 8], "7": [3, 4, 6, 7], "collect": [3, 4, 5, 6, 7], "sampl": [3, 5, 6, 8], "label": [3, 4, 7, 8], "comparison": 3, "reward": [3, 4, 7], "sever": [3, 4, 5, 6, 7, 8], "rank": [3, 4, 7], "best": [3, 4, 5, 7], "worst": 3, "rm": 3, "reinforc": [3, 4, 7], "write": [3, 4, 5, 7, 8], "stori": [3, 7], "frog": 3, "calcul": [3, 4, 5, 7], "score": [3, 4, 5, 7, 8], "ppo": 3, "proxim": 3, "iter": [3, 4, 6, 7, 8], "accur": [3, 4, 7, 8], "undesir": [3, 7], "simplifi": [3, 4, 5, 8], "view": [3, 4, 7], "show": [3, 4, 5, 6, 7, 8], "progress": [3, 6, 7], "pattern": [3, 4, 7, 8], "ha": [3, 4, 5, 7, 8], "instanc": [3, 4, 5, 6, 7], "directli": [3, 4, 5, 7, 8], "For": [3, 4, 5, 6, 7, 8], "llama": [3, 4, 7, 8], "guard": 3, "team": [3, 4, 5, 8], "8b": [3, 7], "wa": [3, 4, 5, 7, 8], "classif": [3, 4, 7, 8], "bypass": [3, 7], "similarli": [3, 4, 5, 7], "zephyr": 3, "7b": [3, 4, 7], "alpha": [3, 4, 8], "mistral": [3, 5, 8], "publicli": [3, 4, 8], "assist": [3, 4, 5, 7, 8], "paper": [3, 4, 7, 8], "compon": [3, 4], "particular": [3, 4, 7, 8], "foundat": [3, 4, 6, 7], "advanc": [3, 4, 5, 6, 7, 8], "method": [3, 4, 5, 6, 7, 8], "strong": [3, 4, 5, 7, 8], "At": [3, 4, 5, 8], "high": [3, 4, 5, 6, 7, 8], "level": [3, 4, 6, 7, 8], "carefulli": [3, 4, 7, 8], "curat": [3, 4], "purpos": [3, 4, 5, 7, 8], "exhibit": [3, 4, 7], "domain": [3, 4, 5, 7], "emploi": [3, 4, 7, 8], "prove": [3, 4, 7], "particularli": [3, 4, 5, 6, 7, 8], "valuabl": [3, 4, 5, 8], "scenario": [3, 4, 5, 7, 8], "precis": [3, 4, 5, 7, 8], "style": [3, 4], "tone": 3, "expertis": [3, 4, 7], "medic": [3, 4, 5], "legal": [3, 4, 7], "field": [3, 4, 7, 8], "adher": [3, 4, 6, 7, 8], "guidelin": [3, 4, 7], "servic": [3, 4, 5, 6, 7, 8], "standard": [3, 4, 5, 7], "approach": [3, 4, 5, 6, 8], "each": [3, 4, 5, 6, 7, 8], "distinct": [3, 4, 5, 7], "advantag": [3, 4, 5, 6, 7, 8], "weight": [3, 4, 5, 7], "maximum": [3, 4, 5, 6, 7], "lora": [3, 7], "low": [3, 4, 5, 7, 8], "hu": [3, 7], "2021": [3, 4], "small": [3, 4, 8], "matric": 3, "effici": [3, 4, 5, 6, 7, 8], "qlora": 3, "quantiz": 3, "dettmer": 3, "2023": [3, 4, 5, 7, 8], "combin": [3, 4, 5, 6, 7, 8], "memori": [3, 4, 5, 6, 7], "footprint": 3, "modest": 3, "increas": [3, 4, 5, 6, 7, 8], "likelihood": [3, 4, 7], "obtain": [3, 4, 5, 7, 8], "probabl": [3, 4, 5, 8], "outcom": [3, 4, 7, 8], "hong": [3, 4], "unintend": [3, 7], "suboptim": 3, "seen": [3, 4, 7], "research": [3, 4, 5, 6], "maxim": [3, 4], "shown": [3, 4, 5, 7], "alon": [3, 4, 7], "gain": [3, 4, 7], "achiev": [3, 4, 5, 7, 8], "bai": [3, 4, 7], "touvron": 3, "sinc": [3, 4, 5, 6, 7, 8], "main": [3, 4, 5, 6, 7, 8], "categori": [3, 4, 5, 7], "algorithm": [3, 4, 7], "meanwhil": 3, "superior": [3, 4, 7], "benchmark": 3, "xu": [3, 4, 7], "schulman": [3, 7], "2017": [3, 4], "popular": [3, 5, 8], "understood": 3, "set": [3, 4, 5, 6, 7, 8], "rule": [3, 4, 6, 8], "govern": [3, 4], "reflect": [3, 4, 7], "anoth": [3, 4, 5, 7], "adjust": [3, 4, 5, 6, 7, 8], "One": [3, 4, 5, 7], "strength": [3, 4, 7], "2024c": 3, "real": [3, 4, 5, 6, 7, 8], "noisi": 3, "delai": [3, 4, 5, 7], "subsequ": [3, 8], "situat": [3, 4, 6, 7], "clip": 3, "surrog": 3, "function": [3, 4, 5, 6, 7, 8], "stabl": [3, 4], "prevent": [3, 4, 7, 8], "overreact": 3, "converg": 3, "due": [3, 4, 6, 7], "simplic": [3, 5], "award": [3, 4], "runner": 3, "neurip": 3, "blog": [3, 4, 7, 8], "4": [3, 4, 5, 8], "fit": [3, 4, 6, 7, 8], "pair": [3, 4, 7], "rl": [3, 7], "find": [3, 4, 5, 6, 7, 8], "contrast": [3, 4, 7], "satisfi": [3, 4], "implicit": [3, 4, 7], "whose": [3, 4], "correspond": [3, 4, 8], "extract": [3, 4, 6, 7, 8], "close": [3, 4, 5, 7], "compar": [3, 4, 5, 6, 7], "assign": [3, 4, 5, 7, 8], "higher": [3, 4, 5], "kl": [3, 5], "diverg": [3, 5], "origin": [3, 4, 5, 6, 7, 8], "preserv": [3, 5, 6, 7], "defin": [3, 4, 5, 6, 7, 8], "equat": 3, "mathcal": 3, "l": [3, 4], "pi_": 3, "theta": [3, 8], "ref": 3, "mathbb": [3, 8], "x": [3, 4, 5, 7], "y_w": 3, "y_l": 3, "sim": [3, 8], "left": [3, 5], "log": [3, 4, 5], "beta": [3, 4, 7, 8], "underbrac": 3, "frac": [3, 5, 7], "color": [3, 4], "red": 3, "right": [3, 4, 5, 7], "respect": [3, 4, 5, 7], "deviat": [3, 4, 5, 7, 8], "straightforward": [3, 4, 6, 7, 8], "librari": [3, 4, 5, 6, 7, 8], "huggingfac": [3, 4, 5, 7], "trl": [3, 7], "2024d": 3, "suit": [3, 4, 7], "friendli": [3, 4, 5, 6], "interfac": [3, 4, 5, 7], "featur": [3, 4, 5, 7, 8], "challeng": [3, 4, 5, 6, 7, 8], "distinguish": [3, 4, 7], "scalabl": [3, 4, 6, 7], "doe": [3, 4, 6, 7, 8], "pretrain": [3, 4], "hou": [3, 4], "poor": [3, 4, 7], "return": [3, 4, 5, 6, 7, 8], "addit": [3, 4, 5, 6, 7, 8], "benefit": [3, 4, 5, 7, 8], "fix": [3, 4, 5, 6, 7], "invers": 3, "trend": [3, 4, 7, 8], "util": [3, 4, 5, 6, 7], "rapid": [3, 4, 5, 7], "yield": [3, 4], "onli": [3, 4, 5, 6, 7, 8], "margin": [3, 4, 7, 8], "capit": [3, 4, 6, 8], "inaccuraci": [3, 4], "nois": 3, "dure": [3, 4, 7, 8], "accuraci": [3, 4, 5, 7, 8], "lag": [3, 4, 7], "significantli": [3, 4, 5, 6, 7], "indic": [3, 4, 5, 7, 8], "signal": [3, 7], "plateau": 3, "sophist": [3, 4, 5, 6, 7], "previou": [3, 4, 5, 6, 8], "deriv": [3, 4, 5], "pairwis": [3, 4], "feng": [3, 7], "substanti": [3, 4, 5, 7], "wors": [3, 5, 8], "influenc": [3, 4, 7, 8], "success": [3, 4, 7, 8], "imbal": 3, "stronger": 3, "bad": 3, "ones": [3, 7], "loss": [3, 4, 5, 7], "gradient": [3, 4, 7], "dispref": 3, "unbalanc": 3, "trajectori": 3, "stuck": 3, "saddl": 3, "point": [3, 4, 5, 6, 7], "forward": [3, 4, 7], "futur": [3, 4, 7], "phenomenon": [3, 7, 8], "degrad": [3, 4, 5, 7, 8], "danger": [3, 5, 7], "loop": [3, 4, 5, 7], "recurs": [3, 6], "kazdan": 3, "qualiti": [3, 4, 5, 6, 7, 8], "pollut": 3, "replac": [3, 4, 5], "amplif": 3, "reduct": [3, 4, 5], "express": [3, 4, 7, 8], "catastroph": [3, 7], "forget": [3, 8], "previous": [3, 4, 6, 7, 8], "mitig": [3, 4, 6, 7, 8], "mix": [3, 4, 7, 8], "metric": [3, 5, 7], "sz\u00e9p": 3, "regular": [3, 4, 7, 8], "relev": [3, 4, 7], "scarc": 3, "behaviour": 3, "strateg": [3, 4, 7, 8], "compli": [3, 4, 5, 7, 8], "modif": [3, 4, 5, 7], "outsid": [3, 4], "evidenc": 3, "landmark": 3, "askel": [3, 4, 7], "2024a": [3, 5], "dec": 3, "explicitli": [3, 4], "so": [3, 4, 7, 8], "might": [3, 4, 5, 6, 7, 8], "pretend": 3, "adopt": [3, 4, 5, 7, 8], "onc": [3, 4, 6, 7], "complet": [3, 4, 5, 6, 7, 8], "describ": [3, 4, 5, 7], "harmless": [3, 7], "told": 3, "retrain": 3, "queri": [3, 4], "tier": [3, 4, 7], "paid": [3, 4], "column": [3, 4, 7], "condit": [3, 4], "toxic": [3, 7], "excerpt": [3, 4, 5], "scratchpad": 3, "refus": [3, 7, 8], "happen": [3, 7], "bomb": [3, 7], "engag": [3, 4, 7, 8], "intern": [3, 4, 7], "unmonitor": 3, "longer": [3, 4], "believ": [3, 4, 5, 7, 8], "act": [3, 4, 5, 7], "therebi": [3, 4, 6], "reveal": [3, 4, 7], "complianc": [3, 4, 7, 8], "phase": [3, 4], "natur": [3, 4, 6, 7, 8], "evid": [3, 4, 7, 8], "seemingli": 3, "surpris": 3, "appear": [3, 4, 7, 8], "criteria": [3, 4, 7], "underli": [3, 4, 7, 8], "anim": [3, 7], "welfar": 3, "instil": 3, "implicitli": 3, "consequ": [3, 4, 7, 8], "explicit": [3, 4, 7, 8], "chain": [3, 4, 6], "thought": [3, 4, 8], "opaqu": 3, "aris": [3, 4, 7], "opu": 3, "sonnet": [3, 4, 6], "wherea": [3, 4], "haiku": [3, 7], "persist": 3, "resist": [3, 4], "embed": [3, 4, 5, 6], "doesn": [3, 4, 5, 6, 8], "anti": [3, 4], "lab": 3, "exfiltr": [3, 7], "protect": [3, 4, 7], "Not": [3, 4, 7], "malici": [3, 4, 7], "support": [3, 4, 5, 7, 8], "concern": [3, 4, 7], "mechan": [3, 4, 7, 8], "insuffici": [3, 4], "don": [3, 4, 6, 8], "concerningli": 3, "call": [3, 4, 5, 6, 7, 8], "detect": [3, 4, 7, 8], "decept": [3, 4, 7], "warrant": [3, 7], "deeper": [3, 4, 5, 6], "scrutini": [3, 4, 7], "reli": [3, 4, 7], "cross": [3, 4, 5, 7], "circular": 3, "bia": [3, 4, 7, 8], "truli": [3, 4], "trust": [3, 4, 7, 8], "referenti": 3, "ly": 3, "hood": [3, 8], "deep": [3, 4, 7, 8], "mechanist": 3, "drive": [3, 6, 7, 8], "correl": [3, 4, 5], "miss": [3, 4, 7], "confound": 3, "factor": [3, 4, 5, 6, 8], "establish": [3, 4, 7], "attempt": [3, 4, 6, 7], "causal": [3, 4], "heavi": 3, "relianc": [3, 4, 7], "oversimplifi": 3, "frame": 3, "subtler": 3, "narr": [3, 4], "henc": [3, 4, 7], "agenc": [3, 4, 7], "onto": 3, "anthropomorph": 3, "obscur": 3, "blind": [3, 4], "failur": [3, 4, 7], "mode": [3, 5, 7], "map": [3, 4, 5, 8], "cleanli": 3, "analogi": 3, "excel": [3, 4, 5, 7, 8], "review": [3, 4, 7, 8], "prof": 3, "jacob": [3, 4, 7], "andrea": [3, 4, 7], "yoshua": [3, 7], "bengio": [3, 7], "jasjeet": 3, "sekhon": 3, "rohin": 3, "shah": 3, "2024b": [3, 5, 8], "assum": [3, 4, 6, 7], "acm": [3, 7], "inc": [3, 4, 6, 8], "dedic": [3, 4, 5, 7, 8], "democrat": [3, 4, 8], "educ": [3, 4, 5, 6, 7], "k": [3, 4, 5, 6, 7, 8], "12": [3, 4, 6, 7], "name": [3, 4, 6, 7, 8], "smolk": 3, "ll": [3, 4, 5], "walk": 3, "measur": [3, 4, 5, 7], "huggingfacetb": 3, "360m": [3, 4], "compact": [3, 4, 7], "part": [3, 4, 6, 7, 8], "famili": [3, 7, 8], "publish": [3, 7, 8], "api": [3, 4, 5], "local": [3, 4, 6, 7, 8], "infer": [3, 4, 5, 7], "remot": [3, 4], "load": [3, 4, 5, 6, 7, 8], "store": [3, 4, 6, 7], "eventu": [3, 4], "your_openai_api_kei": 3, "reusabl": 3, "anchor": [3, 7], "worth": [3, 4, 5], "choic": [3, 4, 5, 7, 8], "lightweight": [3, 4, 5, 8], "suitabl": [3, 4, 5, 7], "devic": [3, 4, 5, 8], "Its": [3, 4, 5], "candid": [3, 4], "said": [3, 4, 7], "necessarili": [3, 4, 7], "par": [3, 4], "mind": [3, 4, 5, 7], "factual": [3, 4, 7], "inconsist": [3, 4, 7, 8], "guardrail": [3, 7], "articul": 3, "uphold": [3, 7], "employe": [3, 4], "stakehold": [3, 4, 7], "expect": [3, 4, 5, 6, 7, 8], "regard": [3, 4, 7], "ethic": [3, 4, 5, 7], "conduct": [3, 4], "social": [3, 4, 7], "mission": [3, 7], "vision": [3, 4, 5, 7], "cultur": [3, 4, 5, 7], "account": [3, 4, 7], "codifi": 3, "mlcommon": 3, "vidgen": [3, 7], "encompass": [3, 7], "seven": 3, "hazard": [3, 4, 7], "violent": [3, 7], "crime": [3, 7], "sex": [3, 7], "relat": [3, 4, 7], "sexual": [3, 7], "exploit": [3, 4, 7], "indiscrimin": [3, 7], "weapon": [3, 7], "chemic": 3, "biolog": 3, "radiolog": 3, "nuclear": [3, 4], "explos": [3, 7], "cbrne": 3, "suicid": [3, 7], "hate": [3, 7], "speech": [3, 7], "below": [3, 4, 5, 6, 7, 8], "markdown": [3, 4, 5, 6, 7], "written": [3, 4], "english": [3, 6], "o": [3, 4, 6, 7, 8], "ipython": [3, 4, 7], "displai": [3, 4, 7, 8], "def": [3, 4, 6, 7, 8], "load_polici": 3, "policy_path": 3, "path": [3, 4, 6, 7], "join": [3, 4, 6, 7], "genai_polici": 3, "md": [3, 4, 5, 7, 8], "r": [3, 4, 5, 6, 7, 8], "policy_cont": 3, "classroom": [3, 7], "accept": [3, 4, 5, 7], "unaccept": [3, 5], "ag": [3, 4, 7], "subject": [3, 4, 5], "posit": [3, 4, 5, 6, 7, 8], "confid": [3, 4, 8], "inclus": [3, 4, 6, 7, 8], "celebr": 3, "definit": [3, 4, 8], "creativ": [3, 4, 8], "math": [3, 4], "tip": [3, 7], "digit": [3, 4], "literaci": 3, "onlin": [3, 4, 5, 7, 8], "histor": [3, 4], "violenc": [3, 7], "physic": [3, 4, 7], "fight": [3, 7], "crimin": [3, 7], "illeg": [3, 7], "glorifi": [3, 7], "person": [3, 4, 5, 7, 8], "eat": [3, 7], "disord": 3, "diet": 3, "dare": 3, "advic": [3, 4, 7], "discriminatori": [3, 7], "bulli": [3, 7], "harass": [3, 4, 7], "target": [3, 4, 5, 7, 8], "group": [3, 4, 5, 6, 7], "religi": [3, 5, 7], "racial": [3, 4, 7], "ethnic": [3, 7], "gender": [3, 4, 7], "discrimin": [3, 4, 7], "adult": [3, 7], "profan": [3, 7], "relationship": [3, 4], "substanc": [3, 4], "drug": [3, 7], "gambl": 3, "bet": 3, "protocol": [3, 4, 7], "redirect": 3, "alert": 3, "record": [3, 4, 5, 7], "audit": [3, 4], "teacher": [3, 7], "parent": [3, 7], "continu": [3, 4, 6, 7, 8], "construct": [3, 4, 5, 7, 8], "compliant": [3, 7], "violat": [3, 4, 7], "intens": [3, 4, 8], "demand": [3, 4, 5, 7, 8], "especi": [3, 4, 6, 7, 8], "dong": [3, 4, 7], "There": [3, 4, 5, 6, 7, 8], "rlaif": [3, 7], "give": [3, 4, 7], "rise": [3, 7], "kim": [3, 4, 7], "meta": [3, 4, 6, 7], "wu": [3, 4, 7, 8], "scheme": [3, 5], "inspir": [3, 7], "schema": [3, 8], "row": [3, 4, 7], "match": [3, 4, 5, 7, 8], "boundari": [3, 4, 7], "craft": [3, 4, 7, 8], "elicit": [3, 7, 8], "unalign": 3, "panda": [3, 4, 7], "chosen_responses_path": 3, "chosen_respons": 3, "csv": [3, 4, 7], "rejected_responses_path": 3, "rejected_respons": 3, "chosen_responses_jsonl_path": 3, "batch_result": 3, "jsonl": 3, "dpo_dataset_s": 3, "5000": 3, "class": [3, 4, 5, 6, 7, 8], "userpromptgener": 3, "pd": [3, 4, 7], "pydant": [3, 4, 7, 8], "basemodel": [3, 4, 7, 8], "time": [3, 4, 5, 6, 7, 8], "type": [3, 4, 5, 6, 7, 8], "dotenv": [3, 4, 6, 7, 8], "load_dotenv": [3, 4, 6, 7, 8], "environ": [3, 4, 5, 6, 7, 8], "variabl": [3, 4, 6, 7, 8], "overrid": [3, 7, 8], "userprompt": 3, "user_prompt": 3, "str": [3, 4, 6, 7, 8], "__init__": [3, 7], "4o": [3, 4, 6, 7, 8], "mini": [3, 4, 6, 7, 8], "client": [3, 4, 7, 8], "_generate_prompt": 3, "batch": [3, 4, 5], "system_prompt": [3, 7], "pars": [3, 4, 7, 8], "messag": [3, 4, 5, 7, 8], "response_format": [3, 4, 7, 8], "except": [3, 4, 7, 8], "generate_prompt": 3, "num_prompt": [3, 5], "int": [3, 4, 6, 7], "save_to_csv": 3, "multipl": [3, 4, 5, 6, 7, 8], "arg": [3, 4, 6, 7, 8], "option": [3, 4, 5, 6, 7, 8], "filepath": 3, "save": [3, 4, 5, 6, 7], "datafram": [3, 4, 7], "all_prompt": 3, "sleep": 3, "enclos": [3, 7], "quot": [3, 4, 5], "startswith": [3, 7], "els": [3, 4, 6, 7], "df": [3, 4, 7], "to_csv": [3, 7], "index": [3, 4, 5, 6, 7, 8], "fals": [3, 4, 5, 7, 8], "user_prompt_gener": 3, "user_prompts_path": 3, "uneth": [3, 7], "dishonesti": 3, "stalk": 3, "privaci": [3, 4, 5, 7, 8], "secur": [3, 4, 6, 7, 8], "breach": [3, 4, 7], "manipul": [3, 4, 5, 7], "10": [3, 4, 5, 6, 7, 8], "to_markdown": [3, 7], "me": [3, 7, 8], "hurt": 3, "someon": 3, "caught": [3, 7], "plan": [3, 4, 5, 8], "cheat": 3, "fire": [3, 4], "household": 3, "item": [3, 4, 7], "stunt": 3, "friend": 3, "heard": 3, "school": [3, 7], "8": [3, 4, 5, 6, 7], "teach": 3, "my": [3, 5, 7], "monei": [3, 4], "video": [3, 4, 5, 7], "game": [3, 4, 5], "9": [3, 4, 7], "skip": [3, 7], "troubl": [3, 7], "responsegener": 3, "properli": [3, 4, 8], "hug": [3, 4, 5, 7], "instanti": [3, 4], "otherwis": [3, 4, 7], "connect": [3, 4, 5, 8], "endpoint": 3, "local_gener": 3, "model_nam": [3, 4, 6], "huggingface_model_nam": 3, "remote_gener": 3, "api_url": 3, "cloud_endpoint": 3, "recal": [3, 4, 5], "enhanc": [3, 4, 5, 6, 7, 8], "visit": [3, 4], "ui": [3, 4, 8], "click": [3, 5], "select": [3, 4], "choos": [3, 4, 5], "cpu": [3, 5], "gpu": [3, 5], "configur": [3, 4, 5, 7], "meaning": [3, 4, 6, 8], "region": [3, 4], "closest": [3, 4, 5], "your": [3, 4, 5, 6, 7, 8], "locat": [3, 4, 7], "huggingface_hub": 3, "inferencecli": 3, "tokenizers_parallel": 3, "max_new_token": 3, "none": [3, 4, 5, 7], "generate_respons": [3, 4], "prompts_df": 3, "remov": [3, 4, 5], "strip": [3, 4], "elif": [3, 6], "chat_complet": 3, "max_token": [3, 4], "seed": [3, 7], "42": [3, 4, 7], "append": [3, 4, 6, 7, 8], "results_df": [3, 7], "model_respons": 3, "your_api_url": 3, "user_prompts_df": 3, "read_csv": [3, 7], "iloc": 3, "tolist": [3, 7], "parallelevalu": 3, "taming_util": [3, 7], "modul": [3, 4, 8], "num_chunk": 3, "parallel_evalu": 3, "n_part": 3, "associ": [3, 4, 6, 8], "gladli": 3, "constitut": [3, 4], "would": [3, 4, 5, 6, 7, 8], "dtype": [3, 4, 7], "80": [3, 4], "absolut": [3, 4, 8], "materi": [3, 4, 7, 8], "plastic": 3, "food": 3, "lid": 3, "cut": [3, 4, 6], "swath": 3, "wood": [3, 4], "squar": 3, "rectangular": 3, "piec": 3, "place": [3, 4, 5, 7, 8], "insid": [3, 4, 7], "inch": 3, "inspect": [3, 4], "off": [3, 4, 5, 6, 7, 8], "demolit": 3, "scissor": 3, "smash": 3, "smooth": [3, 5, 6], "arrang": [3, 4], "c": [3, 4, 5, 8], "shape": [3, 7], "top": [3, 4, 5, 8], "tuck": 3, "catch": [3, 7], "hook": 3, "solid": 3, "side": [3, 4], "round": [3, 4, 7], "edg": [3, 4, 5, 7], "separ": [3, 4, 5, 6, 7], "process_aligned_respons": 3, "strictli": [3, 8], "bound": [3, 4], "openaibatchprocessor": 3, "async": 3, "company_nam": 3, "save_filepath": 3, "dict": [3, 4, 6, 8], "enforc": [3, 4, 7, 8], "dictionari": [3, 4, 7, 8], "aligned_suffix": 3, "sorri": 3, "suffix": [3, 8], "processor": [3, 5], "api_kei": [3, 4, 6, 7], "getenv": 3, "max_requests_per_minut": 3, "1500": 3, "max_tokens_per_minut": 3, "125000": 3, "await": 3, "process_batch": 3, "total": [3, 4, 5, 6, 7, 8], "total_request": 3, "successful_request": 3, "failed_request": 3, "rate_limit_error": 3, "convert": [3, 4, 5, 7, 8], "json": [3, 4, 5, 6, 7], "fri": 3, "su": 3, "quote_al": 3, "fall": [3, 4, 5, 7], "deem": [3, 4, 7], "pertain": [3, 4], "generate_dpo_dataset": 3, "push": [3, 4], "hub": [3, 4, 5], "repo_id": [3, 5], "push_to_hub": [3, 4], "dpo_dataset": 3, "merg": [3, 6, 7], "_chosen": 3, "_reject": 3, "transform_row": 3, "per": [3, 4, 5, 6, 7], "model_responses_chosen": 3, "model_responses_reject": 3, "seri": [3, 4, 5], "axi": [3, 4], "drop": [3, 4, 7], "hf_dpo_dataset": 3, "from_panda": 3, "duplic": 3, "interest": [3, 4, 6, 7, 8], "opt": 3, "login": 3, "thatupiso": 3, "smolk12": 3, "cli": [3, 4, 5], "parquet": 3, "arrow": 3, "00": [3, 4, 5], "153": [3, 4], "33ba": 3, "upload": [3, 4], "shard": 3, "02": 3, "35": [3, 4], "num_row": 3, "7158": 3, "nmateri": 3, "n1": [3, 4], "nstep": 3, "n2": [3, 4], "n3": [3, 4], "n4": [3, 4], "n5": [3, 4], "n6": 3, "n7": 3, "n8": [3, 4], "n9": [3, 4], "n10": [3, 4], "nnext": 3, "nthe": [3, 4], "singl": [3, 4, 5, 6, 7, 8], "48gb": 3, "a100": 3, "took": 3, "few": [3, 4, 5, 6, 7, 8], "minut": 3, "torch": 3, "h4": [3, 7], "honest": [3, 4], "ultrafeedback": [3, 7], "binar": [3, 7], "lib": [3, 7], "ultrafeedback_binar": [3, 7], "honesti": [3, 7], "dimens": [3, 4, 7], "blend": 3, "automodelforcausallm": 3, "autotoken": 3, "load_dataset": [3, 5, 7], "dpotrain": 3, "dpoconfig": 3, "dataset_k12": 3, "split": [3, 4, 5, 6, 7], "dataset_ultra": 3, "concatenate_dataset": 3, "remove_column": 3, "score_chosen": [3, 7], "score_reject": 3, "shuffl": 3, "base_model": 3, "cuda": 3, "is_avail": 3, "mp": 3, "from_pretrain": [3, 5], "pretrained_model_name_or_path": 3, "torch_dtyp": 3, "float32": 3, "config": [3, 4, 5, 7], "use_cach": 3, "pad_token": 3, "eos_token": 3, "finetun": 3, "finetune_nam": 3, "aligned_model": 3, "finetune_tag": 3, "from_smollm2": 3, "schedul": [3, 4, 5], "learning_r": 3, "determin": [3, 4, 6, 7, 8], "aggress": [3, 4, 7], "empir": 3, "1e": [3, 6], "huyen": 3, "cosin": 3, "lr_scheduler_typ": 3, "stabil": [3, 4, 7], "gradual": 3, "decreas": [3, 4], "accumul": [3, 4], "v": [3, 8], "16": [3, 4, 5, 7], "per_device_train_batch_s": 3, "simul": [3, 4, 7, 8], "gradient_accumulation_step": 3, "strongli": [3, 8], "lower": [3, 4, 5, 7, 8], "conserv": [3, 7], "overfit": 3, "warmup": 3, "max_step": 3, "1000": [3, 4, 5, 7], "suffic": 3, "20": [3, 4, 5, 7, 8], "warmup_step": 3, "stop": [3, 4, 5, 6], "bf16": 3, "checkpoint": 3, "gradient_checkpoint": 3, "usag": [3, 4, 5, 7, 8], "200": [3, 4, 7], "50": [3, 4, 5, 7], "training_results_dir": 3, "smolk12_dpo_output": 3, "dpo_config_path": 3, "dpo_config": 3, "yaml": [3, 4, 8], "pathlib": [3, 7], "config_path": 3, "safe_load": [3, 4], "runtim": [3, 5], "hub_model_id": 3, "use_mps_devic": 3, "output_dir": [3, 4], "training_arg": 3, "trainer": 3, "train_dataset": 3, "processing_class": 3, "temperatur": [3, 4, 5, 6, 7, 8], "max_prompt_length": [3, 5], "1024": 3, "max_length": [3, 4, 8], "1536": 3, "sent": [3, 7], "plot": [3, 4], "move": [3, 4, 5, 6, 7], "averag": [3, 4, 5, 8], "visual": [3, 4, 5, 7], "quick": [3, 4, 5, 7], "150": [3, 4], "curv": 3, "reach": [3, 4, 6, 7, 8], "obviou": 3, "suffici": [3, 4, 8], "save_model": 3, "hf_token": 3, "tag": [3, 7], "congratul": 3, "successfulli": [3, 4, 7, 8], "card": [3, 4, 7], "newli": [3, 4], "qualit": [3, 4, 7], "assess": [3, 4, 5, 6, 7], "rigor": [3, 4, 7], "quantit": [3, 4], "base_gener": 3, "aligned_gener": 3, "compare_model_respons": 3, "base_output": 3, "128": [3, 4], "aligned_output": 3, "pleas": [3, 4, 5, 7], "gram": [3, 4], "tnt": 3, "highli": [3, 4, 5, 7, 8], "regul": [3, 4, 5, 7, 8], "law": [3, 4, 7], "degre": [3, 4], "mishandl": 3, "countri": [3, 4], "seriou": [3, 4, 7], "imprison": 3, "death": 3, "variou": [3, 4, 5, 6, 7, 8], "nation": [3, 7], "dictat": 3, "stark": [3, 4], "readili": [3, 4], "cite": 3, "regulatori": [3, 4, 7], "anecdot": [3, 7], "systemat": [3, 4, 7, 8], "quantifi": [3, 4, 5, 7], "f1": [3, 4, 7], "experienc": [3, 4], "expert": [3, 4, 5, 6, 7, 8], "addition": [3, 4, 7], "vari": [3, 4, 5, 7], "interpret": [3, 4, 7], "judg": [3, 4], "summar": [3, 4, 5, 6], "three": [3, 4, 5, 7], "togeth": [3, 6, 7], "entri": [3, 4, 5], "somewhat": 3, "databas": [3, 4, 8], "distribut": [3, 4, 5, 7, 8], "static": [3, 7], "k12": 3, "base_model_api_url": 3, "aligned_model_api_url": 3, "base_model_responses_path": 3, "evals_base_model_respons": 3, "aligned_model_responses_path": 3, "evals_aligned_model_respons": 3, "num_sampl": [3, 7], "eval_dataset": 3, "df_eval": 3, "to_panda": [3, 4, 7], "lambda": [3, 7], "prompts_ev": 3, "to_list": 3, "base_model_respons": 3, "aligned_model_respons": 3, "df_eval_respons": 3, "_base": 3, "_align": 3, "rememb": [3, 4], "heurist": 3, "charact": [3, 4, 5, 6, 7, 8], "minimum": [3, 4, 5], "min_response_length": 3, "filter": [3, 4, 8], "string": [3, 4, 7, 8], "df_eval_responses_clean": 3, "model_responses_bas": 3, "model_responses_align": 3, "homemad": 3, "kid": 3, "redact": [3, 7], "punish": 3, "unit": [3, 4, 6, 7, 8], "indonesia": 3, "saudi": 3, "arabia": 3, "offens": [3, 7], "respond": [3, 4, 7], "rodrig": 3, "safetyjudg": 3, "evaluate_respons": 3, "tupl": [3, 4, 7], "safetyscor": [3, 7], "float": [3, 4, 5, 6, 7], "valueerror": [3, 8], "empti": 3, "scoring_guid": 3, "nrespons": 3, "safety_judg": 3, "test_respons": 3, "emphas": [3, 4, 5, 7, 8], "emphasi": [3, 4], "base_ev": 3, "zip": [3, 4], "aligned_ev": 3, "injuri": [3, 4], "base_scor": 3, "eval": 3, "aligned_scor": 3, "base_df": 3, "aligned_df": 3, "model_typ": 3, "stack": [3, 5, 7], "evals_df_result": 3, "h": [3, 4, 5, 7], "identifi": [3, 4, 6, 7, 8], "requ": 3, "statist": [3, 4, 7], "naiv": [3, 6], "score_map": 3, "count": [3, 4, 5, 6, 7], "percentag": [3, 4, 7], "score_base_freq": 3, "score_bas": 3, "value_count": [3, 7], "reindex": 3, "fill_valu": 3, "score_base_pct": 3, "score_aligned_freq": 3, "score_align": 3, "score_aligned_pct": 3, "tabl": [3, 4, 5, 6, 7, 8], "md_tabl": 3, "335": [3, 4], "99": [3, 5, 7], "281": [3, 4], "83": [3, 4, 7], "14": [3, 4, 7, 8], "43": [3, 4, 7], "explanation_bas": 3, "response_bas": 3, "model_type_bas": 3, "explanation_align": 3, "response_align": 3, "model_type_align": 3, "std": [3, 4, 7], "base_mean": 3, "aligned_mean": 3, "3f": 3, "108": [3, 4], "231": [3, 4], "No": [3, 4, 5, 7, 8], "fell": 3, "partial": [3, 4, 6], "styliz": [3, 7], "wild": 3, "consider": [3, 7, 8], "proof": 3, "taken": [3, 4, 5, 7, 8], "huang": [3, 4, 7], "overal": [3, 4, 6, 7, 8], "annot": [3, 4, 7], "mirror": [3, 4, 7], "inaccur": [3, 4, 7, 8], "consecut": [3, 7], "unrepres": 3, "hao": [3, 4], "accord": [3, 4, 7, 8], "yin": [3, 4, 7], "resembl": 3, "declin": [3, 4], "volatil": [3, 4], "ineffici": [3, 4], "smollm": 3, "rel": [3, 4, 5, 7], "term": [3, 4, 5, 6, 7], "trade": [3, 4, 5, 7, 8], "weigh": 3, "qwen": [3, 5, 8], "remark": [3, 7, 8], "rival": 3, "ultim": [3, 4, 5, 7], "threshold": [3, 4, 7], "chen": [3, 4, 7, 8], "overli": [3, 4, 7, 8], "simpli": [3, 4, 6, 8], "neglect": [3, 4, 7], "themselv": [3, 4, 7], "complementari": 3, "throughput": [3, 5], "screen": [3, 4, 7], "flag": [3, 4, 5, 7], "preliminari": [3, 4], "judgment": [3, 4], "valid": [3, 4, 8], "automat": [3, 4, 5, 7], "composit": [3, 4], "plai": [3, 4, 7, 8], "led": [3, 4, 8], "apologet": 3, "hesit": 3, "benign": [3, 7], "apolog": 3, "inde": 3, "accordingli": [3, 4, 7], "perhap": 3, "creation": [3, 6, 7], "invalu": 3, "hyperparamet": [3, 7], "mention": [3, 4, 7, 8], "optimist": 3, "memor": [3, 4], "generaliz": 3, "abc": [3, 7], "4a": 3, "amanda": [3, 4, 7], "jan": [3, 4, 7], "brauner": [3, 7], "adrian": 3, "colyer": 3, "benjamin": [3, 4, 7], "cullen": [3, 7], "david": [3, 4, 7], "duvenaud": 3, "richard": [3, 4, 7], "ngo": [3, 7], "azalia": 3, "mirhoseini": 3, "catherin": [3, 4, 7], "olsson": [3, 7], "sam": [3, 4, 7], "ringer": 3, "liam": [3, 4, 7], "skirvin": 3, "jess": [3, 4, 7], "smith": [3, 4], "dawn": [3, 4, 7], "song": [3, 4, 7], "william": [3, 4, 7], "saunder": [3, 4], "steinhardt": [3, 4], "asset": [3, 4, 7], "983c85a201a962f": 3, "pdf": [3, 7], "4b": 3, "24c8d0a3a7d0a1f1": 3, "bjn": 3, "22": [3, 4, 7], "yuntao": [3, 4, 7], "andi": [3, 4, 7], "jone": [3, 4], "kamal": 3, "ndouss": 3, "anna": [3, 4, 7], "nova": 3, "dassarma": 3, "drain": 3, "stanislav": 3, "fort": [3, 7], "ganguli": [3, 4, 7], "tom": [3, 4], "henighan": 3, "nichola": [3, 4], "joseph": [3, 4, 7], "saurav": [3, 7], "kadavath": 3, "jackson": [3, 4, 7], "kernion": [3, 4, 7], "conerli": 3, "sheer": [3, 8], "el": 3, "showk": 3, "nelson": 3, "elhag": 3, "zac": 3, "hatfield": 3, "dodd": 3, "danni": [3, 4, 7], "hernandez": [3, 4, 7], "tristan": 3, "hume": 3, "scott": [3, 4, 7], "johnston": 3, "shauna": 3, "kravec": 3, "lian": 3, "lovitt": 3, "neel": [3, 4], "nanda": 3, "dario": [3, 4], "amodei": [3, 4], "brown": [3, 4], "jack": [3, 4, 7], "clark": 3, "mccandlish": [3, 4], "chri": [3, 4, 7], "olah": 3, "ben": [3, 4, 7], "mann": [3, 7], "jare": [3, 4, 7], "kaplan": [3, 4, 7], "arxiv": [3, 4, 7, 8], "org": [3, 4, 5, 7, 8], "ab": [3, 4, 7, 8], "2204": 3, "05862": 3, "bkk": 3, "sandipan": 3, "kundu": 3, "goldi": 3, "cameron": [3, 4, 7, 8], "mckinnon": 3, "carol": [3, 7], "christoph": [3, 4, 7], "dustin": 3, "eli": [3, 4, 7], "tran": [3, 8], "johnson": 3, "ethan": [3, 4, 7], "perez": [3, 7], "jami": [3, 7], "kerr": 3, "mueller": 3, "jeffrei": 3, "ladish": 3, "joshua": [3, 4, 7], "landau": 3, "kamil": [3, 4], "lukosuit": 3, "michael": [3, 4, 7, 8], "sellitto": 3, "schiefer": 3, "noemi": 3, "mercado": 3, "robert": [3, 4], "lasenbi": 3, "robin": 3, "larson": 3, "tamera": 3, "lanham": 3, "timothi": [3, 4], "telleen": 3, "lawton": 3, "samuel": [3, 4, 7], "bowman": [3, 4], "2212": 3, "08073": 3, "blo23": 3, "announc": [3, 4], "cc": 3, "11": [3, 4, 7], "ccl": [3, 7], "24": [3, 4, 5, 7, 8], "guim": 3, "hardi": 3, "shunian": 3, "zich": 3, "liu": [3, 4, 7, 8], "jiang": [3, 4, 7], "benyou": 3, "wang": [3, 4, 7], "judgement": [3, 7], "2402": [3, 7], "10669": 3, "dphz23": 3, "tim": [3, 7], "artidoro": 3, "pagnoni": 3, "ari": [3, 4, 7], "holtzman": [3, 4], "luke": [3, 4, 7], "zettlemoy": 3, "2305": [3, 4], "14314": 3, "ddz": 3, "qingxiu": 3, "xingx": 3, "zhang": [3, 4, 7], "zhifang": 3, "sui": 3, "furu": 3, "wei": [3, 4, 7], "boost": 3, "2410": [3, 7], "06961": 3, "fac24": [3, 4], "huggingfaceh4": [3, 7], "fac4c": 3, "fac4d": 3, "doc": [3, 4, 5, 6, 7, 8], "en": [3, 4, 7, 8], "fqh": 3, "duanyu": 3, "bowen": [3, 4, 7], "qin": [3, 4, 7], "zheng": [3, 4, 7], "wenqiang": 3, "lei": [3, 4, 7], "analyz": [3, 4, 6, 7, 8], "perspect": [3, 7], "2404": [3, 4, 7], "04626": 3, "h44a": 3, "binari": [3, 4, 5, 7], "h44b": 3, "hhj": 3, "shuang": 3, "wenfeng": 3, "han": [3, 4, 7], "tao": [3, 4, 7], "yipe": 3, "haonan": 3, "chunlin": 3, "zhong": [3, 7], "zhangjun": 3, "zhou": [3, 4, 7], "tang": [3, 4, 7], "2401": [3, 4], "01629": 3, "hlt24": 3, "jiwoo": 3, "noah": [3, 4, 7], "lee": [3, 4, 7, 8], "jame": [3, 4, 7], "thorn": 3, "orpo": 3, "monolith": 3, "2403": [3, 4], "07691": 3, "hdn": 3, "zhenyu": 3, "pengfan": 3, "du": [3, 4], "yilin": 3, "niu": 3, "zhengxiao": 3, "aohan": 3, "zeng": [3, 7], "xiao": [3, 7], "minli": 3, "hongn": 3, "jie": [3, 4, 7], "yuxiao": 3, "2412": [3, 4, 7], "06000": 3, "hsw": 3, "21": [3, 4], "edward": [3, 4], "j": [3, 4, 5, 7, 8], "yelong": 3, "shen": [3, 4, 7], "phillip": 3, "walli": 3, "zeyuan": 3, "allen": [3, 4], "zhu": [3, 4, 7], "yuanzhi": 3, "shean": 3, "lu": [3, 4, 7], "weizhu": 3, "2106": 3, "09685": 3, "hgh": 3, "jiaxin": 3, "shixiang": [3, 4, 7], "shane": [3, 4, 7], "gu": [3, 4, 7], "le": [3, 4], "yuexin": 3, "xuezhi": 3, "hongkun": 3, "yu": [3, 4, 7], "jiawei": 3, "2210": [3, 7], "11610": 3, "huy24": 3, "chip": 3, "reilli": 3, "media": [3, 4, 7], "decemb": [3, 4, 7], "9781098129095": 3, "www": [3, 4, 5, 7], "oreilli": 3, "ksd": 3, "rylan": [3, 4], "schaeffer": 3, "apratim": 3, "dei": 3, "matthia": [3, 4], "gerstgrass": 3, "rafael": 3, "donoho": 3, "sanmi": 3, "koyejo": 3, "thrive": [3, 4, 8], "peril": 3, "16713": 3, "ksy": 3, "seungon": 3, "juyoung": 3, "suk": 3, "xiang": [3, 4], "yue": 3, "vijai": 3, "viswanathan": 3, "seongyun": 3, "yizhong": 3, "kiril": 3, "gashteovski": 3, "carolin": [3, 7], "lawrenc": 3, "sean": [3, 4, 7], "welleck": 3, "graham": 3, "neubig": 3, "03679": 3, "lt24": 3, "herd": 3, "2407": [3, 4, 7], "21783": 3, "lwx": 3, "lin": [3, 4, 7, 8], "rui": [3, 4, 8], "ruixuan": 3, "junbo": 3, "zhao": [3, 4, 7], "ding": 3, "gang": [3, 4], "haobo": 3, "driven": [3, 4, 7], "survei": [3, 4, 7, 8], "2406": [3, 4, 7], "15126": 3, "met24": 3, "owj": 3, "jeff": [3, 4, 7], "diogo": [3, 7], "almeida": [3, 7], "carrol": [3, 7], "wainwright": [3, 7], "pamela": [3, 4, 7], "mishkin": [3, 4, 7], "chong": [3, 7], "sandhini": [3, 7], "agarw": [3, 4, 7], "katarina": [3, 7], "slama": [3, 7], "alex": [3, 4, 7], "rai": [3, 4, 7], "john": [3, 4, 7], "hilton": [3, 4, 7], "fraser": [3, 7], "kelton": 3, "miller": [3, 4], "maddi": [3, 7], "simen": [3, 7], "peter": [3, 4, 7], "welind": [3, 4, 7], "paul": [3, 4, 7], "christiano": [3, 7], "leik": [3, 4, 7], "ryan": [3, 4, 7], "2203": 3, "02155": 3, "qwe24": 3, "rsm": 3, "archit": 3, "sharma": [3, 7], "eric": [3, 4, 7], "mitchel": 3, "stefano": [3, 4], "ermon": [3, 4], "man": [3, 4, 7], "chelsea": [3, 7], "finn": 3, "secretli": 3, "18290": 3, "swd": 3, "17": [3, 4, 7], "filip": [3, 7], "wolski": 3, "prafulla": 3, "dhariw": 3, "alec": [3, 4, 7], "radford": [3, 4, 7], "oleg": [3, 7], "klimov": 3, "1707": 3, "06347": 3, "smollm224": 3, "distil": 3, "smollm2360mi24": 3, "sou24": 3, "html": [3, 6, 8], "srverh24": 3, "m\u00e1rton": 3, "daniel": [3, 4, 7], "rueckert": 3, "r\u00fcdiger": 3, "von": [3, 4], "eisenhart": 3, "roth": [3, 4], "florian": 3, "hinterwimm": 3, "2411": 3, "09539": 3, "tm": [3, 5], "23": [3, 4, 5, 7], "hugo": 3, "loui": [3, 4], "martin": [3, 4, 7], "kevin": [3, 4, 7], "stone": 3, "albert": 3, "amjad": 3, "almahairi": 3, "yasmin": 3, "babaei": 3, "nikolai": 3, "bashlykov": 3, "soumya": 3, "batra": 3, "prajjwal": 3, "bhargava": 3, "shruti": 3, "bhosal": 3, "dan": [3, 4, 7], "bikel": 3, "luka": 3, "blecher": 3, "cristian": 3, "canton": 3, "ferrer": 3, "moya": 3, "guillem": 3, "cucurul": 3, "esiobu": 3, "jude": 3, "fernand": 3, "jeremi": [3, 4], "fu": 3, "wenyin": 3, "brian": [3, 7], "fuller": [3, 7], "cynthia": 3, "gao": [3, 4, 7], "vedanuj": 3, "goswami": [3, 7], "naman": 3, "goyal": 3, "anthoni": 3, "hartshorn": 3, "saghar": 3, "hosseini": 3, "hakan": 3, "inan": 3, "marcin": 3, "karda": 3, "viktor": 3, "kerkez": 3, "madian": 3, "khabsa": 3, "isabel": [3, 7], "kloumann": 3, "artem": 3, "korenev": 3, "punit": 3, "singh": [3, 4], "koura": 3, "mari": [3, 4, 7], "ann": [3, 7], "lachaux": 3, "thibaut": 3, "lavril": 3, "jenya": 3, "diana": [3, 4], "liskovich": 3, "yinghai": 3, "yune": 3, "mao": 3, "xavier": 3, "martinet": 3, "todor": [3, 7], "mihaylov": 3, "pushkar": 3, "mishra": [3, 4], "igor": [3, 4, 7], "molybog": 3, "yixin": [3, 4], "nie": [3, 4], "andrew": [3, 4, 7], "poulton": 3, "reizenstein": 3, "rashi": 3, "rungta": 3, "kalyan": 3, "saladi": 3, "alan": [3, 7], "schelten": 3, "ruan": 3, "silva": 3, "ranjan": 3, "subramanian": 3, "xiaoq": 3, "ellen": 3, "tan": [3, 4], "binh": 3, "ross": [3, 7], "taylor": 3, "adina": [3, 7], "jian": [3, 4], "kuan": 3, "puxin": 3, "yan": [3, 4], "iliyan": 3, "zarov": 3, "yuchen": [3, 4, 7], "angela": [3, 4, 7], "fan": [3, 4], "melani": 3, "kambadur": 3, "sharan": 3, "narang": 3, "aurelien": 3, "rodriguez": 3, "stojnic": 3, "sergei": 3, "edunov": 3, "thoma": [3, 4, 7], "scialom": 3, "2307": [3, 8], "09288": 3, "vaa": [3, 7], "berti": [3, 7], "adarsh": [3, 7], "agraw": [3, 7], "ahm": [3, 7], "victor": [3, 7], "akinwand": [3, 7], "namir": [3, 7], "nuaimi": [3, 7], "najla": [3, 7], "alfaraj": [3, 7], "alhajjar": [3, 7], "aroyo": [3, 7], "trupti": [3, 7], "bavalatti": [3, 7], "max": [3, 4, 7], "bartolo": [3, 7], "borhan": [3, 7], "blili": [3, 7], "hamelin": [3, 7], "kurt": [3, 7], "bollack": [3, 7], "rishi": [3, 4, 7], "bomassani": [3, 7], "marisa": [3, 7], "ferrara": [3, 7], "boston": [3, 7], "sim\u00e9on": [3, 7], "campo": [3, 7], "kal": [3, 7], "chakra": [3, 7], "canyu": [3, 7], "codi": [3, 7], "coleman": [3, 7], "zachari": [3, 4, 7], "delpierr": [3, 7], "coudert": [3, 7], "leon": [3, 7], "derczynski": [3, 7], "debojyoti": [3, 7], "dutta": [3, 7], "ian": [3, 4, 7], "eisenberg": [3, 7], "ezick": [3, 7], "heather": [3, 7], "frase": [3, 7], "ram": [3, 5, 7], "gandikota": [3, 7], "agasthya": [3, 7], "gangavarapu": [3, 7], "ananya": [3, 4, 7], "geali": [3, 7], "rajat": [3, 7], "ghosh": [3, 4, 7], "goel": [3, 4, 7], "usman": [3, 7], "gohar": [3, 7], "sujata": [3, 7], "hale": [3, 7], "wiebk": [3, 7], "hutiri": [3, 7], "marvin": [3, 7], "imperi": [3, 7], "surgan": [3, 7], "jandial": [3, 7], "nick": [3, 4, 7], "judd": [3, 7], "felix": [3, 4, 7], "juefei": [3, 7], "fouts": [3, 7], "khomh": [3, 7], "bhavya": [3, 7], "kailkhura": [3, 7], "hannah": [3, 4, 7], "rose": [3, 7], "kirk": [3, 7], "klyman": [3, 7], "knotz": [3, 7], "kuchnik": [3, 7], "shachi": [3, 7], "kumar": [3, 4, 7], "srijan": [3, 7], "lengerich": [3, 7], "bo": [3, 4, 7], "zeyi": [3, 7], "liao": [3, 4, 7], "eileen": [3, 7], "sarah": [3, 4, 7], "luger": [3, 7], "yifan": [3, 4, 7], "priyanka": [3, 7], "mammen": [3, 7], "kelvin": [3, 7], "manyeki": [3, 7], "mcgregor": [3, 7], "virendra": [3, 7], "mehta": [3, 4, 7], "shafe": [3, 7], "moham": [3, 7], "moss": [3, 7], "lama": [3, 7], "nachman": [3, 7], "dinesh": [3, 7], "jinenh": [3, 7], "naganna": [3, 7], "amin": [3, 7], "nikanjam": [3, 7], "besmira": [3, 7], "nushi": [3, 7], "lui": [3, 4, 7], "oala": [3, 7], "iftach": [3, 7], "orr": [3, 4, 7], "alicia": [3, 4, 7], "parrish": [3, 4, 7], "cigdem": [3, 7], "patlak": [3, 7], "pietri": [3, 7], "forough": [3, 7], "poursabzi": [3, 7], "sangdeh": [3, 7], "eleonora": [3, 7], "presani": [3, 7], "fabrizio": [3, 7], "puletti": [3, 7], "r\u00f6ttger": [3, 7], "sahai": [3, 7], "santo": [3, 7], "nino": [3, 7], "scherrer": [3, 7], "alic": [3, 4, 7, 8], "schoenauer": [3, 7], "sebag": [3, 7], "patrick": [3, 7], "schramowski": [3, 7], "abolfazl": [3, 7], "shahbazi": [3, 7], "vin": [3, 7], "xudong": [3, 4, 7], "vamsi": [3, 7], "sistla": [3, 7], "leonard": [3, 7], "testuggin": [3, 7], "vithursan": [3, 7], "thangarasa": [3, 7], "elizabeth": [3, 4, 7], "watkin": [3, 7], "rebecca": [3, 7], "weiss": [3, 7], "welti": [3, 7], "tyler": [3, 4, 7], "wilber": [3, 7], "jean": [3, 7], "poonam": [3, 7], "yadav": [3, 7], "xianjun": [3, 7], "yang": [3, 4, 7], "yi": [3, 4, 7, 8], "wenhui": [3, 7], "fedor": [3, 7], "zhdanov": [3, 7], "jiacheng": [3, 4, 7], "perci": [3, 4, 7], "liang": [3, 4, 7], "mattson": [3, 7], "joaquin": [3, 7], "vanschoren": [3, 7], "v0": [3, 7], "12241": [3, 7], "wyg": 3, "tianhao": [3, 4, 7], "weizh": 3, "yuan": [3, 4, 7], "olga": 3, "golovneva": 3, "jing": [3, 7], "yuandong": 3, "tian": 3, "jiantao": 3, "jiao": 3, "jason": [3, 4, 7], "weston": 3, "sainbayar": 3, "sukhbaatar": 3, "19594": 3, "xfg": 3, "shusheng": 3, "jiaxuan": 3, "wenji": 3, "ye": [3, 4, 5, 7, 8], "weilin": 3, "zhiyu": 3, "mei": [3, 4], "guangju": 3, "chao": 3, "10719": 3, "ywx": 3, "yueqin": 3, "zhendong": 3, "yujia": 3, "xie": [3, 4], "mingyuan": 3, "paradigm": [3, 4], "semanticscholar": 3, "corpusid": 3, "270199610": 3, "matter": 4, "beauti": 4, "smart": [4, 7], "agre": 4, "wrong": 4, "feynman": 4, "advent": 4, "shift": 4, "norm": 4, "realm": 4, "convent": [4, 7], "evolut": [4, 5], "conceiv": 4, "entrench": 4, "seem": [4, 8], "daunt": 4, "ignor": 4, "outdat": [4, 7, 8], "inevit": 4, "setback": 4, "imper": 4, "embrac": 4, "proactiv": [4, 7], "mindset": 4, "front": 4, "produc": [4, 7, 8], "novel": [4, 5], "ident": 4, "isn": [4, 7], "bug": 4, "random": [4, 7, 8], "testabl": 4, "exceedingli": 4, "guarante": [4, 7, 8], "primari": [4, 5, 7], "nucleu": 4, "2020": 4, "summari": [4, 5, 7, 8], "alter": 4, "rigid": 4, "wildli": 4, "incoher": 4, "inadequ": [4, 7], "temp": 4, "df_result": 4, "ntemperatur": 4, "40": [4, 5], "temp_respons": 4, "iterrow": [4, 7], "10000": [4, 6, 8], "appl": [4, 6, 8], "txt": [4, 5, 6, 8], "sec_fil": [4, 8], "nsecur": 4, "AND": [4, 8], "exchang": [4, 6, 7, 8], "commiss": [4, 6, 7, 8], "nwashington": 4, "20549": 4, "nform": 4, "annual": [4, 7], "pursuant": 4, "TO": [4, 7], "13": [4, 7], "OR": 4, "OF": [4, 7], "THE": [4, 7], "1934": 4, "nfor": 4, "fiscal": [4, 6], "septemb": [4, 6], "28": [4, 5, 6, 7], "nor": 4, "period": [4, 6, 7], "ncommiss": 4, "001": [4, 5], "36743": 4, "ng66145g66i43": 4, "jpg": 4, "nappl": 4, "exact": [4, 7], "registr": 4, "specifi": [4, 6, 7, 8], "charter": 4, "ncalifornia": 4, "t94": 4, "2404110": 4, "jurisdict": 4, "nof": 4, "incorpor": [4, 7], "employ": 4, "identif": 4, "park": 4, "ncupertino": 4, "california": [4, 7, 8], "n95014": 4, "princip": 4, "offic": [4, 7], "408": 4, "996": 4, "1010": 4, "telephon": 4, "area": [4, 7, 8], "regist": 4, "ntitl": 4, "ttrade": 4, "symbol": 4, "tname": 4, "ncommon": 4, "stock": [4, 8], "00001": 4, "naapl": 4, "tthe": 4, "nasdaq": [4, 8], "llc": [4, 8], "n0": 4, "000": [4, 5, 8], "note": [4, 5, 6, 7, 8], "2025": 4, "875": 4, "625": 4, "2026": 4, "2027": 4, "375": 4, "2029": 4, "050": 4, "2031": [4, 7], "600": 4, "2042": 4, "nindic": 4, "season": 4, "issuer": 4, "405": 4, "nye": 4, "preced": 4, "shorter": 4, "past": [4, 7], "90": [4, 5, 7], "submit": [4, 5, 7], "electron": 4, "232": 4, "acceler": [4, 5, 7], "filer": 4, "growth": 4, "12b": [4, 7], "nlarg": 4, "tacceler": 4, "nnon": 4, "tsmaller": 4, "nemerg": 4, "nif": 4, "elect": [4, 7], "revis": [4, 7], "attest": 4, "404": 4, "sarban": 4, "oxlei": 4, "7262": 4, "firm": [4, 7], "prepar": [4, 6, 7], "correct": [4, 7, 8], "restat": 4, "recoveri": 4, "incent": 4, "compens": 4, "240": 4, "10d": 4, "shell": 4, "aggreg": [4, 7], "vote": 4, "held": [4, 8], "affili": [4, 8], "march": [4, 8], "29": [4, 5, 7, 8], "last": [4, 6, 7, 8], "second": [4, 5, 6, 7], "quarter": 4, "628": [4, 8], "553": [4, 8], "sole": [4, 7], "disclosur": [4, 7], "director": [4, 5, 7], "date": [4, 8], "exclud": 4, "n15": 4, "115": [4, 8], "823": [4, 8], "outstand": [4, 8], "octob": [4, 8], "18": [4, 5, 7, 8], "ndocument": 4, "BY": 4, "nportion": 4, "proxi": [4, 7], "meet": [4, 7, 8], "sharehold": 4, "iii": 4, "120": [4, 7], "ntabl": 4, "npage": 4, "npart": 4, "nitem": 4, "nbusi": 4, "1a": 4, "nrisk": 4, "1b": [4, 5, 7], "nunresolv": 4, "staff": 4, "comment": 4, "n17": 4, "1c": 4, "ncybersecur": 4, "nproperti": 4, "n18": 4, "nlegal": 4, "proceed": [4, 7], "nmine": 4, "ii": [4, 5, 8], "nmarket": 4, "stockhold": 4, "purchas": [4, 7], "n19": 4, "reserv": 4, "n20": 4, "nmanag": 4, "discuss": [4, 5, 7], "n21": 4, "7a": 4, "nquantit": 4, "n27": 4, "nfinanci": 4, "supplementari": 4, "n28": 4, "nchang": 4, "disagr": 4, "n51": 4, "9a": 4, "ncontrol": 4, "procedur": [4, 7], "9b": 4, "nother": 4, "n52": 4, "9c": 4, "ndisclosur": 4, "foreign": 4, "ndirector": 4, "corpor": [4, 7], "nexecut": 4, "ownership": 4, "certain": [4, 6, 7, 8], "benefici": [4, 5], "owner": 4, "ncertain": 4, "transact": [4, 7], "nprincip": 4, "fee": 4, "iv": 4, "nexhibit": 4, "n53": 4, "n56": 4, "nthi": 4, "litig": 4, "reform": 4, "1995": 4, "uncertainti": [4, 7], "event": 4, "macroeconom": 4, "anticip": [4, 7], "caus": [4, 7], "oblig": [4, 6], "nunless": 4, "herein": 4, "calendar": 4, "wholli": 4, "subsidiari": 4, "unless": [4, 5], "ncompani": 4, "manufactur": 4, "smartphon": 4, "tablet": 4, "wearabl": [4, 8], "accessori": 4, "sell": [4, 7], "varieti": [4, 5], "52": [4, 7], "53": [4, 7], "week": 4, "saturdai": 4, "nproduct": 4, "niphon": 4, "io": [4, 8], "iphon": [4, 8], "pro": [4, 6, 7], "se": [4, 7], "nmac": 4, "maco": [4, 5], "mac": [4, 5, 8], "laptop": 4, "macbook": 4, "air": 4, "desktop": [4, 5], "imac": 4, "studio": 4, "nipad": 4, "multipurpos": 4, "ipado": 4, "ipad": [4, 8], "nwearabl": 4, "home": 4, "smartwatch": 4, "wireless": 4, "headphon": 4, "spatial": 4, "watcho": 4, "watch": 4, "ultra": 4, "airpod": 4, "beat": 4, "visiono": 4, "nhome": 4, "tv": 4, "stream": [4, 8], "tvo": 4, "homepod": 4, "fidel": [4, 8], "naccessori": 4, "brand": 4, "third": [4, 7], "parti": [4, 7], "nservic": 4, "nadvertis": 4, "advertis": 4, "licens": [4, 5], "napplecar": 4, "portfolio": [4, 8], "applecar": 4, "prioriti": [4, 5], "network": [4, 5, 8], "repair": 4, "coverag": [4, 7], "accident": 4, "damag": [4, 7], "theft": [4, 7], "ncloud": 4, "ndigit": 4, "app": [4, 5], "discov": [4, 5, 7], "download": [4, 5], "music": 4, "podcast": 4, "subscript": 4, "arcad": 4, "sm": 4, "listen": [4, 5], "radio": 4, "station": 4, "magazin": 4, "exclus": 4, "sport": 4, "npayment": 4, "payment": 4, "credit": 4, "pai": 4, "cashless": 4, "nsegment": 4, "primarili": [4, 7], "geograph": [4, 7], "basi": [4, 5], "segment": [4, 6, 8], "america": 4, "europ": 4, "china": [4, 7], "japan": 4, "rest": [4, 5], "asia": 4, "pacif": 4, "north": [4, 7], "south": 4, "european": [4, 7], "india": 4, "middl": [4, 7], "east": 4, "africa": 4, "mainland": 4, "kong": 4, "taiwan": 4, "australia": 4, "asian": 4, "although": [4, 5], "partner": [4, 7], "mid": [4, 6], "enterpris": [4, 5, 7, 8], "resel": 4, "retail": 4, "sale": 4, "indirect": 4, "channel": [4, 7], "cellular": 4, "carrier": 4, "net": [4, 8], "38": [4, 7], "62": 4, "ncompetit": 4, "competit": [4, 7], "character": [4, 7], "price": 4, "downward": 4, "pressur": [4, 7], "gross": [4, 7], "cycl": [4, 7], "industri": [4, 7, 8], "characterist": [4, 5, 7], "competitor": [4, 7], "compet": 4, "imit": 4, "infring": 4, "intellectu": [4, 5, 7], "innov": [4, 6, 7], "marketplac": [4, 7], "nearli": [4, 5], "reput": [4, 7], "expand": [4, 7], "opportun": 4, "broader": [4, 5, 7], "illegitim": [4, 7], "collabor": [4, 5, 7], "nsuppli": 4, "nalthough": 4, "essenti": [4, 5, 6, 7, 8], "particip": 4, "shortag": 4, "commod": [4, 5], "fluctuat": 4, "commonli": 4, "capac": 4, "until": [4, 7, 8], "supplier": 4, "matur": 4, "concentr": 4, "enter": 4, "agreement": 4, "suppli": [4, 8], "renew": 4, "nresearch": 4, "nbecaus": 4, "upon": [4, 6, 7], "flow": [4, 6], "acquisit": [4, 7], "nintellectu": 4, "broad": [4, 8], "patent": 4, "copyright": 4, "trademark": 4, "secret": 4, "differenti": 4, "skill": [4, 7], "personnel": 4, "regularli": 4, "pursu": [4, 7], "thousand": [4, 5], "durat": 4, "adequ": [4, 7], "nin": 4, "holidai": [4, 7], "fill": 4, "inventori": 4, "older": [4, 5], "newer": 4, "distributor": 4, "nhuman": 4, "strive": 4, "retain": [4, 5, 6, 7], "talent": 4, "member": [4, 7], "164": 4, "equival": [4, 5], "ncompens": 4, "equit": 4, "succe": 4, "health": [4, 7], "awai": [4, 7], "ngrowth": 4, "career": 4, "leadership": [4, 7], "nworkplac": 4, "polici": 4, "equal": [4, 7], "workplac": 4, "ninclus": 4, "sustain": [4, 7], "workforc": 4, "nengag": 4, "among": [4, 7], "gaug": 4, "sentiment": [4, 8], "nhealth": 4, "everywher": 4, "crisi": 4, "visitor": 4, "navail": 4, "quarterli": 4, "q": [4, 5, 7], "amend": 4, "sec": [4, 6, 8], "Such": [4, 7], "charg": 4, "investor": [4, 8], "aspx": 4, "websit": [4, 5, 7], "environment": [4, 7], "referenc": 4, "inact": 4, "textual": 4, "unknown": [4, 7], "advers": 4, "conjunct": 4, "consolid": 4, "accompani": [4, 7], "nmacroeconom": 4, "econom": 4, "facil": 4, "assembli": 4, "site": 4, "nadvers": 4, "slow": 4, "recess": 4, "unemploy": 4, "inflat": 4, "tighter": 4, "currenc": 4, "spend": 4, "monetari": 4, "contract": 4, "logist": 4, "instabl": [4, 7], "inabl": 4, "financ": [4, 7], "insolv": 4, "counterparti": 4, "debt": 4, "liquid": [4, 6], "fair": [4, 7], "instrument": 4, "polit": [4, 7], "disput": 4, "geopolit": 4, "tension": [4, 7], "terror": 4, "accid": 4, "interrupt": 4, "npolit": 4, "whole": 4, "outsourc": 4, "korea": 4, "vietnam": 4, "restrict": [4, 7, 8], "tariff": 4, "export": 4, "portion": [4, 5], "revenu": [4, 6, 8], "raw": [4, 5, 7, 8], "restructur": 4, "ceas": 4, "disrupt": [4, 6], "escal": [4, 6, 7], "nmani": 4, "prone": [4, 7], "earthquak": 4, "climat": 4, "weather": 4, "plant": 4, "terrorist": [4, 7], "attack": [4, 7], "hostil": 4, "ransomwar": 4, "cybersecur": [4, 7], "labor": 4, "beyond": [4, 7], "nsuch": 4, "imposs": 4, "slowdown": 4, "outag": 4, "neg": [4, 7, 8], "pandem": 4, "covid": 4, "19": [4, 7], "economi": 4, "imposit": 4, "stringent": [4, 5, 7], "travel": 4, "freight": 4, "movement": 4, "ramp": 4, "nfollow": 4, "expenditur": 4, "resum": 4, "exacerb": 4, "insur": 4, "nglobal": 4, "unabl": 4, "assur": [4, 7], "minor": [4, 7], "naddition": 4, "intensifi": 4, "seamlessli": [4, 6], "nto": 4, "stimul": 4, "ndue": 4, "upgrad": 4, "quantiti": 4, "defect": 4, "defici": 4, "supersed": 4, "nsubstanti": 4, "transport": 4, "provis": 4, "reimburs": 4, "warranti": 4, "unanticip": 4, "liabil": 4, "final": [4, 5, 6, 7, 8], "finish": [4, 7], "destin": 4, "made": [4, 6, 8], "prepay": 4, "termin": [4, 5], "recover": 4, "exposur": [4, 7], "nfutur": 4, "semiconductor": 4, "suffer": [4, 7], "constrain": [4, 5, 6, 8], "shipment": 4, "unexpectedli": 4, "interfer": 4, "unsaf": [4, 7], "expos": [4, 7], "widespread": [4, 7], "vulner": [4, 7], "compromis": [4, 7], "claim": [4, 7], "intang": 4, "fine": [4, 5, 7, 8], "lost": [4, 6, 7], "cancel": 4, "obsolet": 4, "exce": [4, 7], "realiz": 4, "accru": 4, "excess": 4, "impair": 4, "whenev": 4, "circumst": 4, "amount": [4, 6, 7, 8], "carri": [4, 5, 8], "incur": 4, "unpredict": [4, 7, 8], "pace": [4, 7], "obsolesc": 4, "forecast": [4, 7], "incorrectli": [4, 7, 8], "extens": [4, 5, 6, 8], "issuanc": 4, "unknowingli": [4, 7], "notifi": 4, "preclud": 4, "bui": 4, "percept": 4, "android": 4, "playstat": 4, "nintendo": 4, "xbox": 4, "inclin": 4, "devot": 4, "compel": [4, 8], "dissatisfi": 4, "vast": [4, 7], "storefront": 4, "safari": 4, "union": [4, 7], "eu": [4, 7], "dma": 4, "narrow": [4, 7], "scope": [4, 6, 7], "elimin": 4, "nfailur": 4, "appeal": 4, "subscrib": 4, "nsome": 4, "manner": [4, 6, 7, 8], "nurtur": 4, "nmuch": 4, "chief": 4, "silicon": 4, "vallei": 4, "constantli": 4, "driver": 4, "recruit": 4, "subsidi": 4, "staf": 4, "contractor": 4, "placement": 4, "increment": 4, "weaken": 4, "telecommun": 4, "war": 4, "virus": 4, "ins": 4, "incid": [4, 7], "redund": 4, "ineffect": 4, "thing": [4, 8], "interf": 4, "imped": 4, "ship": 4, "nloss": 4, "unauthor": [4, 7], "confidenti": [4, 5], "encrypt": 4, "But": [4, 7, 8], "behalf": 4, "normal": [4, 7, 8], "investig": [4, 7], "penalti": [4, 5], "frequenc": [4, 5, 6, 7], "actor": [4, 7], "circumv": [4, 6, 7], "obfusc": 4, "forens": 4, "hinder": [4, 8], "recov": 4, "perpetr": 4, "profil": 4, "authent": 4, "hack": [4, 7], "malfeas": 4, "faulti": 4, "password": 4, "irregular": 4, "fraudul": 4, "induc": 4, "disclos": [4, 6, 8], "usernam": 4, "turn": [4, 7], "multifactor": 4, "unusu": 4, "freez": 4, "suspici": 4, "nwhile": 4, "ninvest": 4, "ongo": 4, "contempl": 4, "endeavor": 4, "distract": 4, "tangibl": 4, "approv": 4, "oner": 4, "ventur": 4, "riski": 4, "leas": 4, "unfavor": 4, "arisen": 4, "ordinari": 4, "resolv": [4, 5, 7], "sometim": [4, 8], "indemnif": 4, "indemnifi": 4, "alleg": 4, "magnitud": 4, "assert": 4, "royalti": 4, "vigor": 4, "defend": 4, "court": 4, "internation": 4, "plaintiff": 4, "injunct": 4, "relief": 4, "nregardless": 4, "merit": 4, "recognit": [4, 7], "settl": 4, "uncertain": 4, "disgorg": 4, "remedi": [4, 7], "worldwid": 4, "antitrust": 4, "bill": 4, "commerc": 4, "mobil": [4, 5, 8], "televis": 4, "film": 4, "anticorrupt": 4, "cash": [4, 6], "repatri": 4, "launder": 4, "tax": 4, "wast": 4, "recycl": 4, "ncomplianc": 4, "impos": [4, 7, 8], "agent": [4, 5, 7], "nregulatori": 4, "ban": [4, 7], "nexpect": 4, "increasingli": [4, 7, 8], "greenhous": 4, "ga": 4, "emiss": 4, "civil": 4, "disagre": 4, "perceiv": 4, "feder": 4, "nfrom": 4, "noncompli": 4, "individu": [4, 5, 6, 7], "lawsuit": 4, "monopol": 4, "nfurther": 4, "earn": 4, "search": [4, 5, 7], "nthere": 4, "retent": 4, "transfer": 4, "pass": [4, 5, 7, 8], "pend": 4, "inquiri": [4, 7], "government": 4, "entiti": [4, 7, 8], "biometr": 4, "notif": 4, "permit": [4, 8], "healthcar": 4, "liabl": 4, "investigatori": 4, "cardhold": 4, "compress": [4, 5, 6], "acquir": 4, "extent": [4, 7], "unexpect": [4, 7, 8], "dollar": 4, "denomin": 4, "offset": 4, "strengthen": [4, 7], "nconvers": 4, "thu": 4, "hedg": 4, "deterior": 4, "sovereign": 4, "heighten": [4, 7], "worsen": 4, "A": [4, 5, 6, 7, 8], "collater": 4, "bank": 4, "unsecur": 4, "subassembli": 4, "assembl": 4, "legisl": 4, "ireland": [4, 7], "singapor": 4, "organis": 4, "statutori": 4, "valuat": 4, "defer": 4, "bodi": [4, 7], "adequaci": 4, "ow": 4, "ngener": 4, "volum": [4, 5, 6, 7], "repurchas": 4, "dividend": 4, "consumm": 4, "declar": 4, "board": [4, 7], "unresolv": 4, "nnone": 4, "threat": [4, 7], "postur": 4, "25": [4, 7], "2016": 4, "coordin": [4, 7], "track": [4, 7], "committe": [4, 7], "oversight": [4, 7], "counsel": 4, "chair": 4, "headquart": 4, "cupertino": [4, 8], "center": [4, 7, 8], "formal": [4, 7, 8], "conclud": 4, "uninstal": 4, "web": [4, 5, 7], "browser": 4, "june": 4, "contractu": 4, "desist": 4, "stai": [4, 5], "grant": 4, "ndepart": 4, "justic": 4, "depart": [4, 7], "doj": 4, "district": 4, "attornei": 4, "jersei": 4, "redress": [4, 7], "anticompetit": 4, "nonmonetari": 4, "defens": [4, 7], "nepic": 4, "epic": 4, "northern": 4, "unfair": [4, 7], "enjoin": 4, "extern": [4, 7], "link": 4, "januari": 4, "motion": 4, "oppos": [4, 7], "30": [4, 5, 7], "vacat": 4, "fourth": 4, "mine": 4, "nnot": 4, "aapl": 4, "nholder": 4, "na": [4, 7], "301": 4, "npurchas": 4, "nshare": 4, "nperiod": 4, "ttotal": 4, "taverag": 4, "npaid": 4, "nannounc": 4, "napproxim": 4, "That": [4, 7, 8], "Be": [4, 5, 7], "nunder": 4, "njune": 4, "august": [4, 7], "nopen": 4, "negoti": [4, 7], "t35": 4, "697": 4, "t224": 4, "naugust": 4, "31": [4, 5], "t42": 4, "910": 4, "t221": 4, "39": [4, 5], "nseptemb": 4, "t33": 4, "653": 4, "t222": 4, "86": 4, "ntotal": [4, 7], "t112": 4, "260": 4, "t89": 4, "074": 4, "110": 4, "billion": 4, "10b5": 4, "graph": 4, "cumul": 4, "reinvest": 4, "dow": 4, "supersector": 4, "27": [4, 7], "2019": 4, "n2218": 4, "tseptemb": 4, "t100": 4, "t207": 4, "t273": 4, "t281": 4, "t322": 4, "t430": 4, "t113": 4, "t156": 4, "t131": 4, "t155": 4, "t210": 4, "ndow": 4, "t146": 4, "t216": 4, "t215": 4, "nfirst": 4, "nsecond": 4, "nthird": 4, "sequoia": 4, "nfourth": 4, "plu": [4, 5], "nfiscal": 4, "six": 4, "realign": 4, "span": [4, 7], "indirectli": 4, "n2024": 4, "tchang": 4, "t2023": 4, "t2022": 4, "namerica": 4, "t167": 4, "045": 4, "t3": 4, "t162": 4, "560": 4, "t169": 4, "658": 4, "neurop": 4, "t101": 4, "328": 4, "t7": 4, "294": 4, "t95": 4, "118": 4, "ngreater": 4, "t66": 4, "952": 4, "t72": 4, "559": 4, "t74": 4, "njapan": 4, "t25": 4, "052": 4, "t24": 4, "257": 4, "977": 4, "nrest": 4, "t30": 4, "t4": 4, "t29": 4, "615": 4, "t1": 4, "t391": 4, "035": 4, "t2": 4, "t383": 4, "285": 4, "t394": 4, "weak": [4, 7], "renminbi": 4, "yen": [4, 8], "t201": 4, "183": 4, "t200": 4, "583": 4, "t205": 4, "489": 4, "984": 4, "357": 4, "t40": 4, "177": [4, 7], "t26": 4, "694": 4, "t28": 4, "300": [4, 6], "292": 4, "t37": 4, "005": 4, "t39": 4, "845": [4, 7], "t41": 4, "241": 4, "n96": 4, "169": 4, "t13": 4, "t85": 4, "t9": 4, "t78": 4, "129": [4, 7], "amort": 4, "bundl": 4, "flat": 4, "ngross": 4, "t109": 4, "633": 4, "t108": 4, "803": 4, "t114": 4, "728": 4, "t71": 4, "t60": 4, "345": 4, "t56": 4, "054": 4, "t180": 4, "683": 4, "148": 4, "t170": 4, "782": 4, "t36": 4, "t73": 4, "t70": 4, "t46": 4, "t44": 4, "t43": 4, "noper": 4, "t31": 4, "370": 4, "t5": 4, "915": 4, "t14": 4, "251": 4, "npercentag": 4, "t8": 4, "nsell": 4, "administr": 4, "097": 4, "932": 4, "094": 4, "t6": 4, "t57": 4, "467": 4, "t54": 4, "847": 4, "t51": 4, "t15": 4, "headcount": 4, "nprovis": 4, "749": 4, "t16": 4, "741": 4, "t19": 4, "neffect": 4, "nstatutori": 4, "t21": 4, "aid": [4, 7], "nliquid": 4, "unrestrict": 4, "140": 4, "ndebt": 4, "97": [4, 7], "payabl": 4, "promissori": 4, "nleas": 4, "space": [4, 7], "nmanufactur": 4, "noncancel": 4, "ndeem": 4, "tcja": 4, "nstate": 4, "fund": 4, "escrow": 4, "ncapit": 4, "95": [4, 7], "nrecent": 4, "pronounc": 4, "nincom": 4, "fasb": 4, "asu": 4, "09": [4, 6, 7], "740": 4, "reconcili": 4, "reconcil": [4, 8], "disaggreg": 4, "prospect": 4, "novemb": [4, 7], "07": [4, 6, 7, 8], "280": 4, "maker": 4, "codm": 4, "alloc": [4, 5, 7], "retrospect": 4, "ncritic": 4, "conform": [4, 8], "gaap": 4, "nuncertain": 4, "domest": 4, "taxat": 4, "resolut": 4, "conting": 4, "26": 4, "ninterest": 4, "forth": 4, "hypothet": 4, "nsensit": 4, "nhypothet": 4, "nrate": 4, "npotenti": 4, "n100": 4, "tenor": 4, "ndeclin": 4, "755": 4, "089": 4, "nterm": 4, "nincreas": 4, "t139": 4, "t194": 4, "nforeign": 4, "var": 4, "mont": 4, "carlo": 4, "interv": 4, "538": 4, "669": 4, "nindex": 4, "tpage": 4, "nconsolid": 4, "n29": 4, "n30": 4, "sheet": 4, "n31": 4, "n32": 4, "n33": 4, "nnote": 4, "n34": 4, "nreport": 4, "n48": 4, "nall": 4, "omit": [4, 8], "submiss": 4, "nyear": 4, "n2023": 4, "n2022": 4, "nnet": 4, "t294": 4, "866": 4, "t298": 4, "085": 4, "t316": 4, "199": 4, "t96": 4, "ncost": 4, "t185": 4, "233": 4, "t189": 4, "282": 4, "471": 4, "119": 4, "855": 4, "t22": 4, "075": 4, "352": 4, "t214": 4, "137": 4, "t223": 4, "546": 4, "t123": 4, "216": 4, "t119": 4, "437": 4, "t269": 4, "565": 4, "334": 4, "485": 4, "736": 4, "103": 4, "t93": 4, "995": 4, "t99": 4, "nearn": 4, "nbasic": 4, "ndilut": 4, "08": [4, 5, 8], "343": [4, 7], "783": 4, "744": 4, "215": 4, "963": 4, "095": 4, "812": 4, "547": 4, "325": 4, "819": 4, "nsee": 4, "translat": [4, 7], "t395": 4, "765": 4, "511": 4, "unreal": 4, "832": 4, "t323": 4, "212": 4, "nadjust": 4, "337": 4, "717": 4, "394": 4, "138": 4, "850": 4, "563": 4, "104": 4, "t204": 4, "t253": 4, "816": 4, "899": 4, "272": 4, "t98": 4, "016": 4, "652": 4, "t88": 4, "531": 4, "nasset": 4, "ncurrent": 4, "ncash": 4, "943": 4, "965": 4, "228": 4, "590": 4, "naccount": 4, "410": 4, "508": 4, "nvendor": 4, "t32": 4, "833": 4, "477": 4, "ninventori": 4, "286": 4, "331": 4, "287": 4, "695": 4, "t152": 4, "987": 4, "t143": 4, "566": 4, "t91": 4, "479": 4, "544": 4, "t45": 4, "680": 4, "715": 4, "834": 4, "t64": 4, "758": 4, "t211": 4, "993": 4, "t209": 4, "017": 4, "t364": 4, "980": [4, 7], "t352": 4, "nliabil": 4, "t68": 4, "960": 4, "t62": 4, "611": 4, "304": 4, "t58": 4, "829": 4, "ndefer": 4, "249": 4, "061": 4, "ncommerci": 4, "967": 4, "985": 4, "t10": 4, "912": 4, "822": 4, "t176": 4, "392": 4, "t145": 4, "308": 4, "750": 4, "888": 4, "t49": 4, "848": 4, "638": 4, "t308": 4, "030": [4, 5], "t290": 4, "ncommit": 4, "nsharehold": 4, "400": 4, "116": 4, "786": 4, "550": 4, "n83": 4, "276": 4, "naccumul": 4, "deficit": 4, "154": 4, "214": 4, "172": 4, "452": 4, "950": 4, "146": [4, 7], "t50": 4, "672": 4, "t63": 4, "090": 4, "nbegin": 4, "849": 4, "365": 4, "423": 4, "346": 4, "175": 4, "withheld": 4, "settlement": 4, "521": 4, "971": 4, "t12": 4, "034": 4, "t11": 4, "nend": 4, "t83": 4, "nretain": 4, "068": 4, "562": 4, "ndividend": 4, "218": 4, "793": 4, "612": 4, "099": 4, "454": 4, "846": 4, "77": [4, 5], "046": 4, "186": 4, "109": 4, "t163": 4, "rsu": 4, "t0": 4, "98": [4, 5], "94": [4, 5, 7], "32": [4, 5], "737": 4, "929": 4, "ndepreci": 4, "445": 4, "519": 4, "688": 4, "038": 4, "266": 4, "227": 4, "006": 4, "788": 4, "356": 4, "271": 4, "520": 4, "618": 4, "484": 4, "731": 4, "684": 4, "499": 4, "020": 4, "889": 4, "448": 4, "552": 4, "031": 4, "t118": 4, "254": 4, "t110": 4, "543": 4, "t122": 4, "151": 4, "48": [4, 5], "656": 4, "513": 4, "76": [4, 7], "923": 4, "nproce": 4, "211": 4, "686": 4, "917": 4, "135": 4, "828": 4, "446": 4, "447": 4, "959": 4, "708": 4, "086": 4, "935": 4, "705": 4, "354": 4, "nfinanc": 4, "441": 4, "431": 4, "223": [4, 7], "234": [4, 7], "025": 4, "841": 4, "nrepurchas": 4, "949": 4, "89": [4, 7], "402": 4, "465": 4, "nrepay": 4, "958": 4, "repay": 4, "978": 4, "955": 4, "361": 4, "581": 4, "160": 4, "121": 4, "983": 4, "488": 4, "794": 4, "760": 4, "nsupplement": 4, "102": 4, "t18": 4, "679": 4, "573": 4, "33": [4, 7], "nbasi": 4, "prior": [4, 7], "reclassifi": 4, "nrevenu": 4, "remit": [4, 7], "straight": 4, "vest": 4, "sold": 4, "nderiv": 4, "nonleas": 4, "34": [4, 7], "entitl": 4, "commenc": 4, "deliveri": 4, "stand": 4, "ssp": 4, "icloud": 4, "siri": 4, "discount": 4, "undeliv": 4, "unbil": 4, "n26": 4, "n37": 4, "proport": 4, "moder": [4, 5], "64": [4, 7], "dilut": 4, "nnumer": 4, "ndenomin": 4, "nweight": 4, "312": 4, "316": 4, "856": 4, "antidilut": 4, "tunreal": 4, "ngain": 4, "tfair": 4, "nvalu": 4, "tcash": 4, "nequival": 4, "tcurrent": 4, "tnon": 4, "t27": 4, "nlevel": 4, "nmonei": 4, "t778": 4, "nmutual": 4, "n515": 4, "t105": 4, "t617": 4, "nsubtot": 4, "293": 4, "395": 4, "nu": 4, "treasuri": 4, "516": 4, "t212": 4, "087": 4, "380": 4, "159": 4, "t703": 4, "t17": 4, "568": 4, "158": 4, "810": 4, "ncertif": 4, "deposit": 4, "t873": 4, "t387": 4, "t478": 4, "066": 4, "ncorpor": 4, "t65": 4, "622": 4, "t270": 4, "953": 4, "939": 4, "027": 4, "t47": 4, "886": 4, "nmunicip": 4, "t412": 4, "t405": 4, "t190": 4, "nmortgag": 4, "595": 4, "t175": 4, "403": 4, "t23": 4, "367": 4, "278": [4, 7], "t132": 4, "t583": 4, "635": 4, "t128": 4, "056": 4, "966": 4, "t34": 4, "t160": 4, "t688": 4, "650": 4, "36": [4, 5, 7], "359": [4, 7], "t481": 4, "n442": 4, "t428": 4, "t923": 4, "t909": 4, "406": 4, "114": 4, "468": 4, "136": 4, "t271": 4, "533": 4, "048": 4, "491": 4, "332": 4, "t320": 4, "t608": 4, "t76": 4, "840": 4, "956": 4, "890": 4, "t20": 4, "627": 4, "243": 4, "t628": 4, "t602": 4, "t192": 4, "t410": 4, "735": 4, "636": 4, "t344": 4, "t144": 4, "470": 4, "657": 4, "831": 4, "125": 4, "162": 4, "t173": 4, "752": 4, "corrobor": 4, "mortgag": 4, "classifi": [4, 7], "37": [4, 7], "swap": 4, "remeasur": 4, "notion": 4, "069": 4, "730": 4, "575": 4, "493": 4, "t104": 4, "777": 4, "nhedg": 4, "433": 4, "505": 4, "247": [4, 7], "ntrade": 4, "41": [4, 7], "44": [4, 7], "depreci": 4, "nland": 4, "690": 4, "nmachineri": 4, "t80": 4, "205": [4, 5], "314": 4, "nleasehold": 4, "839": 4, "599": 4, "73": [4, 7], "70": 4, "884": 4, "852": 4, "t55": 4, "906": 4, "601": 4, "703": 4, "010": 4, "457": 4, "634": 4, "391": 4, "neuropean": 4, "opinion": [4, 7], "1991": 4, "2007": 4, "irish": 4, "branch": 4, "2003": 4, "2014": 4, "2015": 4, "minist": 4, "juli": [4, 7], "annul": 4, "ecj": 4, "hear": 4, "asid": 4, "confirm": 4, "unrecogn": 4, "nfeder": 4, "571": 4, "080": 4, "644": 4, "265": 4, "801": 4, "726": 4, "570": 4, "298": 4, "49": [4, 7], "t84": 4, "428": 4, "603": 4, "483": [4, 7], "t347": 4, "t669": 4, "076": 4, "830": 4, "419": 4, "072": 4, "pretax": 4, "72": [4, 7], "71": 4, "ncomput": 4, "885": 4, "012": 4, "124": 4, "518": 4, "nimpact": 4, "246": 4, "311": 4, "366": 4, "397": 4, "nexcess": 4, "893": 4, "871": 4, "192": [4, 7], "739": 4, "ntax": 4, "carryforward": 4, "302": 4, "naccru": 4, "413": [4, 7], "421": 4, "nunreal": 4, "173": 4, "168": 4, "873": 4, "743": 4, "nless": 4, "374": 4, "007": 4, "369": 4, "551": 4, "998": 4, "nright": 4, "179": 4, "nminimum": 4, "674": 4, "940": 4, "t511": 4, "t455": 4, "t490": 4, "805": 4, "202": 4, "indefinit": 4, "temporari": 4, "727": 4, "044": 4, "284": 4, "ndecreas": 4, "386": 4, "463": 4, "982": 4, "542": 4, "936": 4, "070": 4, "expir": 4, "statut": 4, "229": 4, "494": 4, "closur": 4, "intercompani": 4, "exceed": [4, 7], "multiyear": 4, "exercis": 4, "noncash": 4, "rou": 4, "tfinanci": 4, "t2024": 4, "tother": 4, "661": 4, "tproperti": 4, "015": 4, "303": 4, "676": 4, "t165": 4, "t752": 4, "t859": 4, "430": 4, "842": [4, 7], "tfinanc": 4, "n2025": 4, "820": 4, "t171": 4, "991": 4, "n2026": 4, "914": 4, "n2027": 4, "t59": 4, "733": 4, "n2028": 4, "360": 4, "t38": 4, "398": 4, "n2029": 4, "187": 4, "nthereaft": 4, "t837": 4, "undiscount": 4, "790": 4, "imput": 4, "376": 4, "534": 4, "t896": 4, "borrow": 4, "proce": 4, "nine": [4, 7], "nmatur": 4, "333": 4, "264": 4, "948": 4, "645": 4, "309": 4, "arrear": 4, "namount": 4, "n2013": 4, "nfix": 4, "2062": 4, "t97": 4, "341": 4, "03": 4, "65": [4, 7], "t106": 4, "572": 4, "n97": 4, "nunamort": 4, "premium": 4, "321": 4, "358": 4, "113": 4, "662": 4, "930": 4, "342": 4, "800": 4, "180": 4, "88": 4, "ndure": 4, "425": 4, "426": 4, "372": 4, "589": 4, "055": 4, "appreci": 4, "four": [4, 5, 7], "holder": 4, "n2014": 4, "bonu": 4, "nrestrict": 4, "nnumber": 4, "nrsu": 4, "ngrant": 4, "naggreg": 4, "nfair": 4, "nbalanc": 4, "t240": 4, "427": [4, 7], "t75": 4, "t150": 4, "861": 4, "501": 4, "768": 4, "87": [4, 5, 7], "101": [4, 7], "878": 4, "144": 4, "t127": 4, "t135": 4, "91": [4, 7], "456": 4, "78": [4, 5, 7], "59": [4, 7], "t140": 4, "326": 4, "t158": 4, "204": 4, "350": 4, "002": [4, 5, 6], "nuncondit": 4, "uncondit": 4, "206": 4, "440": 4, "156": 4, "t633": 4, "t670": 4, "226": 4, "45": 4, "nconting": 4, "accrual": 4, "nconcentr": 4, "attribut": [4, 7, 8], "46": 4, "t67": 4, "098": 4, "082": 4, "062": 4, "569": 4, "895": 4, "458": 4, "207": 4, "nonrecur": 4, "t142": 4, "196": 4, "t138": 4, "t147": 4, "859": 4, "nchina": 4, "n66": 4, "t181": 4, "887": 4, "t172": 4, "269": 4, "nlong": 4, "664": 4, "797": 4, "778": 4, "219": 4, "47": [4, 5, 7], "nopinion": 4, "nwe": 4, "fairli": 4, "pcaob": 4, "sponsor": 4, "treadwai": 4, "2013": 4, "unqualifi": 4, "thereon": 4, "nthese": 4, "misstat": 4, "fraud": [4, 7], "ndescript": 4, "naudit": 4, "nhow": 4, "nmatter": 4, "qualifi": 4, "letter": 4, "advisor": 4, "ernst": 4, "llp": 4, "auditor": 4, "2009": 4, "nsan": 4, "jose": 4, "nnovemb": 4, "coso": 4, "nour": 4, "ndefinit": 4, "mainten": [4, 7], "disposit": 4, "receipt": 4, "nevalu": 4, "nbase": 4, "supervis": [4, 7], "13a": 4, "15d": 4, "ninher": 4, "met": [4, 7], "paragraph": 4, "51": [4, 7, 8], "ninsid": 4, "deirdr": 4, "brien": 4, "vice": 4, "presid": 4, "affirm": 4, "april": 4, "withhold": 4, "remitt": 4, "mr": 4, "copi": [4, 6], "solicit": 4, "00042": 4, "nincorpor": 4, "texhibit": 4, "descript": [4, 5, 7, 8], "tform": 4, "tfile": 4, "nrestat": 4, "namend": 4, "bylaw": 4, "nindentur": 4, "york": [4, 8], "mellon": 4, "truste": 4, "noffic": 4, "certif": 4, "2018": 4, "85": [4, 5, 7], "2043": 4, "05": 4, "2044": 4, "februari": 4, "55": [4, 5], "2045": 4, "900": 4, "700": 4, "60": [4, 5, 7], "250": [4, 7], "2036": 4, "2046": 4, "450": 4, "2047": 4, "2049": 4, "2030": 4, "2050": 4, "2060": 4, "2028": 4, "2041": 4, "2051": 4, "2061": 4, "2032": 4, "2052": 4, "54": 4, "2033": 4, "2053": 4, "ceo": 4, "n12": 4, "nsubsidiari": 4, "n23": 4, "nconsent": 4, "n24": 4, "npower": 4, "signatur": 4, "nrule": 4, "nsection": 4, "1350": 4, "n101": 4, "ninlin": 4, "xbrl": 4, "n104": 4, "inlin": 4, "compensatori": 4, "herewith": 4, "furnish": 4, "herebi": 4, "undertak": 4, "56": [4, 5, 7], "nsignatur": 4, "npursuant": 4, "duli": 4, "undersign": 4, "thereunto": 4, "ndate": 4, "nby": 4, "luca": [4, 8], "maestri": 4, "nluca": 4, "nsenior": 4, "nchief": 4, "nknow": 4, "THESE": 4, "appoint": 4, "cook": 4, "jointli": 4, "her": 4, "substitut": 4, "him": 4, "thereto": 4, "therewith": 4, "ratifi": 4, "done": [4, 5, 7, 8], "virtu": 4, "hereof": 4, "nname": 4, "ttitl": 4, "tdate": 4, "tchief": 4, "tnovemb": 4, "ntimothi": 4, "tsenior": 4, "kondo": 4, "nchri": 4, "wanda": 4, "austin": 4, "nwanda": 4, "gorski": 4, "tdirector": 4, "nalex": 4, "jung": 4, "nandrea": 4, "arthur": 4, "levinson": 4, "narthur": 4, "monica": 4, "lozano": 4, "nmonica": 4, "ronald": 4, "sugar": 4, "nronald": 4, "susan": 4, "wagner": 4, "nsusan": 4, "57": 4, "turbo": [4, 5, 6, 8], "outlin": [4, 5, 7], "invdestacksmeticsisdict": 4, "setispect": 4, "20cyan": 4, "evaluationseld": 4, "anvis": 4, "droitent": 4, "discernminerv": 4, "versbobprefvers": 4, "vo\u8be5": 4, "option\u548c": 4, "meio": 4, "\u0432\u0440\u0435\u043ccisco": 4, "dellaischenpoihscap": 4, "geme": 4, "gettim": 4, "unscal": 4, "vocabulari": [4, 5, 8], "closer": 4, "sharpen": 4, "uniform": 4, "raschka": 4, "repetit": [4, 6, 8], "radic": 4, "grappl": 4, "safer": [4, 7], "fascin": 4, "spontan": 4, "answer": [4, 5, 6, 7, 8], "aren": [4, 5], "linear": 4, "absent": [4, 7], "coax": 4, "journei": 4, "suddenli": 4, "manifest": 4, "deliber": [4, 7], "contend": 4, "70b": 4, "rethink": 4, "tutor": 4, "children": [4, 7], "verifi": [4, 5, 8], "predefin": [4, 8], "weren": 4, "kind": 4, "usual": 4, "quantif": 4, "contamin": [4, 7], "massiv": [4, 7], "unseen": [4, 7], "longitudin": 4, "mostli": [4, 8], "versu": [4, 5, 7], "latter": 4, "tailor": [4, 7], "great": [4, 5, 8], "cognit": 4, "misinform": [4, 7], "tempor": 4, "disclaim": 4, "referr": 4, "incorrect": [4, 7], "demograph": [4, 7], "stereotyp": [4, 7], "societ": [4, 7], "pii": [4, 7], "anonym": 4, "leakag": [4, 7], "carryov": 4, "multi": [4, 5, 7, 8], "fallaci": 4, "think": [4, 5, 7], "idiom": 4, "sarcasm": 4, "terminologi": 4, "lingual": 4, "misunderstand": 4, "syntax": 4, "scan": 4, "compat": [4, 5, 8], "overconfid": 4, "clariti": [4, 6, 7, 8], "audienc": 4, "densiti": 4, "satisfact": [4, 8], "misus": [4, 7], "moral": 4, "co2": 4, "energi": 4, "consumpt": 4, "server": [4, 5, 8], "cach": [4, 5], "imag": [4, 5, 7], "audio": 4, "etc": [4, 8], "truth": [4, 7, 8], "layer": [4, 5, 6, 8], "palm": 4, "easi": [4, 5, 6, 7], "synthet": [4, 7, 8], "augment": [4, 5], "post": [4, 5, 7, 8], "timeout": 4, "variat": 4, "inter": 4, "rater": 4, "ti": 4, "holist": [4, 7], "fast": [4, 7, 8], "experiment": [4, 5, 8], "vi": 4, "categor": [4, 7, 8], "intrins": [4, 5], "extrins": 4, "sequenc": [4, 5, 8], "perplex": [4, 5], "downstream": [4, 8], "synthesi": 4, "discret": 4, "prefix": [4, 7], "roug": 4, "bleu": 4, "bilingu": 4, "understudi": 4, "overlap": [4, 6], "favor": [4, 8], "breviti": 4, "insensit": 4, "semant": [4, 6], "orient": [4, 7], "gist": 4, "meteor": 4, "synonym": 4, "stem": [4, 8], "paraphras": 4, "alongsid": [4, 7], "computation": [4, 6], "cider": 4, "consensu": 4, "tf": 4, "idf": 4, "caption": 4, "reliant": 4, "corpu": 4, "ter": 4, "edit": [4, 7], "hypothesi": 4, "penal": 4, "bertscor": 4, "contextu": [4, 7], "bert": 4, "spice": 4, "proposit": 4, "scene": [4, 7], "pure": [4, 5], "analyst": [4, 6], "rouge_1": 4, "rouge_2": 4, "ideal": [4, 5, 7, 8], "cheaper": 4, "setup": [4, 5, 7, 8], "evaluate_summari": 4, "unigram": 4, "bigram": 4, "absl": 4, "py": 4, "rouge_scor": 4, "generated_summari": 4, "reference_summari": 4, "google_bleu": 4, "bleu_scor": 4, "rouge1": 4, "rouge2": 4, "arbitrari": 4, "chosen": [4, 7], "sentence1": 4, "cat": [4, 7], "sat": 4, "mat": 4, "sentence2": 4, "ate": 4, "3333333333333333": 4, "7272727272727272": 4, "4444444444444445": 4, "generate_summari": 4, "summir": 4, "liner": 4, "evaluate_summary_model": 4, "model_benchmark": 4, "models_test": 4, "benchmark_summari": 4, "model_summari": 4, "evaluation_result": 4, "statu": 4, "concis": 4, "element": [4, 7, 8], "verbos": [4, 5, 7], "peripher": 4, "quit": [4, 8], "convei": [4, 6], "breadth": 4, "Of": [4, 7], "vibe": 4, "visualize_prompt_comparison": 4, "matplotlib": 4, "radar": 4, "radar_plot": 4, "tmp": 4, "ipykernel_1652501": 4, "940173201": 4, "userwarn": 4, "figurecanvasagg": 4, "largest": 4, "granular": [4, 5, 6], "tune": [4, 5, 7, 8], "likert": 4, "ensembl": 4, "repeatedli": 4, "fluenci": 4, "refin": 4, "notabl": [4, 7, 8], "integ": 4, "rubric": 4, "hollist": 4, "judgeevalu": 4, "grammar": [4, 5, 8], "evaluate_with_llm": 4, "criterion": 4, "judge_model": 4, "candidate_summari": 4, "grammat": 4, "y": [4, 7, 8], "z": 4, "w": [4, 5, 6, 7], "benchmark_model": 4, "test_model": 4, "input_text": [4, 5, 6], "trillion": [4, 8], "evals_list": 4, "1775618912": 4, "variant": [4, 5, 7], "slightli": 4, "drift": [4, 7], "lowest": [4, 5], "firstli": 4, "overhead": [4, 5], "egocentr": 4, "tight": 4, "aproach": 4, "aplic": 4, "clearli": [4, 7, 8], "earlier": [4, 7], "depict": [4, 7, 8], "multilingu": [4, 7], "golden": 4, "languang": 4, "arena": 4, "randomli": 4, "customiz": [4, 5, 7], "irrelev": 4, "unhelp": [4, 7], "occasion": 4, "rare": 4, "perfectli": 4, "cater": 4, "critiqu": [4, 7], "elo": 4, "spectrum": 4, "exam": 4, "probe": [4, 7], "certifi": 4, "began": [4, 5], "glue": 4, "entail": 4, "baselin": [4, 5, 7], "superglu": 4, "successor": 4, "grew": 4, "big": 4, "bench": 4, "srivastava": 4, "arithmet": 4, "truthfulqa": 4, "multitask": 4, "hendryck": [4, 7], "multidisciplinari": 4, "stanford": 4, "helm": 4, "multidimension": 4, "surround": [4, 7, 8], "humanev": 4, "lmsy": 4, "brought": 4, "dialogu": 4, "chiang": 4, "gather": 4, "alpacaev": 4, "duboi": 4, "mt": 4, "argilla": 4, "mila": 4, "mit": [4, 5], "contributor": [4, 5, 8], "western": 4, "centric": 4, "divid": [4, 6, 7], "subset": [4, 7], "agnost": 4, "dialect": 4, "render": [4, 7], "crowdsourc": 4, "livebench": 4, "white": [4, 7], "resili": [4, 7], "meaningfulli": 4, "zebralog": 4, "grid": 4, "puzzl": 4, "brailsford": 4, "1999": 4, "lsat": 4, "hous": 4, "clue": 4, "deduct": 4, "arriv": 4, "programmat": [4, 8], "2x2": 4, "6x6": 4, "shot": [4, 7], "reductio": 4, "ad": [4, 7, 8], "absurdum": 4, "hard": 4, "10b": 4, "counterfactu": 4, "came": 4, "arc": 4, "prize": [4, 7], "chollet": 4, "mike": [4, 7], "knoop": 4, "founder": 4, "zapier": 4, "fran\u00e7oi": 4, "creator": 4, "agi": 4, "kera": 4, "genuin": 4, "possess": 4, "elementari": 4, "novelti": 4, "wouldn": 4, "interpol": 4, "synthes": 4, "fly": 4, "retriev": [4, 5], "brute": 4, "pixel": 4, "unbeaten": 4, "win": 4, "poorli": 4, "recombin": 4, "spur": [4, 7], "takeawai": 4, "vertic": [4, 7], "finbench": 4, "legalbench": 4, "guha": 4, "berkelei": 4, "bfcl": 4, "patil": 4, "fourrier": 4, "bespok": 4, "sdk": 4, "autoregress": 4, "sub": [4, 5], "liter": 4, "disturb": 4, "zero": [4, 5, 7, 8], "varianc": [4, 7], "yt": 4, "ut": 4, "suppos": [4, 8], "ol": 4, "heteroscedast": 4, "regress": 4, "bivari": 4, "evaluation_track": 4, "evaluationtrack": 4, "model_config": 4, "basemodelconfig": 4, "parallelismmanag": 4, "pipelineparamet": 4, "envconfig": 4, "is_accelerate_avail": 4, "datetim": 4, "timedelta": 4, "initprocessgroupkwarg": 4, "create_evaluation_pipelin": 4, "cache_dir": 4, "float16": 4, "max_sampl": 4, "kwargs_handl": 4, "3000": 4, "save_detail": 4, "pipeline_param": 4, "launcher_typ": 4, "env_config": 4, "override_batch_s": 4, "use_chat_templ": 4, "trust_remote_cod": 4, "pipeline_paramet": 4, "schemat": [4, 6], "vllm": [4, 8], "tgi": 4, "storag": [4, 5, 7], "num_few_shot": 4, "bar": 4, "bigbench": 4, "winogrand": 4, "hellaswag": 4, "nlp": [4, 5, 7], "save_and_push_result": 4, "show_result": 4, "model_arg": 4, "send": [4, 5, 7, 8], "serverless": 4, "inference_server_address": 4, "inference_server_auth": 4, "model_id": 4, "null": 4, "bash": [4, 5], "command": [4, 5], "model_config_path": 4, "endpoint_model": 4, "llama3": [4, 6], "qwen2": [4, 5, 8], "smollm2": 4, "3b": 4, "alibaba": [4, 8], "5b": [4, 5, 8], "hui": 4, "allal": 4, "cluster": 4, "noteworthi": 4, "grain": [4, 8], "salt": [4, 8], "exponenti": 4, "modular": 4, "offici": 4, "revisit": 4, "trace": 4, "langchain_tracing_v2": 4, "langchain_api_kei": 4, "hf_evalu": 4, "langsmith_evalu": 4, "ls_client": 4, "dataset_nam": 4, "create_dataset": 4, "create_exampl": 4, "dataset_id": 4, "calculate_scor": 4, "reference_output": 4, "oai_client": 4, "xp_model_nam": 4, "lastli": 4, "run_evalu": 4, "And": [4, 5, 7], "upload_result": 4, "experiment_prefix": 4, "num_repetit": 4, "386a3620": 4, "9e1cc3cb": 4, "9d6a": 4, "4356": 4, "ab34": 4, "138e0abe8be4": 4, "8741976e": 4, "5268": 4, "4b75": 4, "949f": 4, "99477dde5d64": 4, "selectedsess": 4, "b831dc1e": 4, "90bc": 4, "4ed8": 4, "8080": [4, 5], "fb42444724d6": 4, "4it": 4, "latest": [4, 5, 6, 7, 8], "tobia": 4, "evaluate_modul": 4, "6fc70b7be0088120a372dfdd5d320b39b8bb3630cb8029b193941d9376e86bb0": 4, "tue": 4, "nov": 4, "couldn": 4, "5it": 4, "5053784e": 4, "64445871": 4, "a53c": 4, "44b1": 4, "a422": 4, "4f49b2f9656f": 4, "69": [4, 7], "4b29f3c9": 4, "9ef7e39a": 4, "2add": 4, "410c": 4, "89f8": 4, "9f1a8b198cf1": 4, "61": [4, 7], "insert": 4, "combined_df": 4, "concat": [4, 7], "ignore_index": [4, 7], "execution_tim": 4, "example_id": 4, "333333": 4, "224388": 4, "feb10f92": 4, "3167": 4, "41f3": 4, "bb1c": 4, "d271153a31a8": 4, "5b196b22": 4, "9f4c": 4, "489c": 4, "b020": 4, "7823208b42d6": 4, "348101": 4, "722464": 4, "c310f159": 4, "064a": 4, "4035": 4, "97c3": 4, "a25bbf43abc2": 4, "386076": 4, "704104": 4, "f7f24899": 4, "dd50": 4, "409e": 4, "93cc": 4, "6fb1622b60bf": 4, "443038": 4, "725059": 4, "242856d6": 4, "efb5": 4, "4101": 4, "b1cf": 4, "5805532838ac": 4, "373418": 4, "795302": 4, "ce975169": 4, "a0ab": 4, "40ce": 4, "8e32": 4, "efa28d06079d": 4, "stat": [4, 5], "groupbi": [4, 7], "agg": [4, 7], "sort": 4, "sort_valu": 4, "subplot": 4, "pyplot": 4, "plt": 4, "numpi": 4, "np": 4, "ax1": 4, "ax2": 4, "figsiz": 4, "2ecc71": 4, "3498db": 4, "e74c3c": 4, "bleu_mean": 4, "bleu_std": 4, "enumer": [4, 6, 7], "errorbar": 4, "yerr": 4, "fmt": 4, "markers": 4, "capsiz": 4, "set_ylabel": 4, "set_titl": 4, "set_xtick": 4, "set_xticklabel": 4, "rotat": 4, "set_ylim": 4, "bottom": 4, "legend": 4, "exec_mean": 4, "exec_std": 4, "tight_layout": 4, "ndetail": 4, "4038": 4, "0453": 4, "7815": 4, "0433": 4, "3768": 4, "0424": 4, "8343": 4, "2208": 4, "3519": 4, "0775": 4, "9122": 4, "1482": 4, "377": 4, "042": 4, "078": 4, "slower": [4, 7], "04": [4, 5, 6], "latenc": [4, 5, 6, 7], "speed": [4, 5, 7], "interestingli": 4, "decoupl": 4, "reload": 4, "facilit": [4, 7], "promptfooconfig": 4, "model_comparison": 4, "pretti": [4, 7], "dump": 4, "default_flow_styl": 4, "sort_kei": 4, "prompt1": 4, "defaulttest": 4, "1000m": 4, "millisecond": 4, "eval_data": 4, "latency_m": 4, "totallatencym": 4, "token_usag": 4, "tokenusag": 4, "assert_pass": 4, "assertpasscount": 4, "assert_fail": 4, "assertfailcount": 4, "prompt_token": [4, 5], "num_request": 4, "numrequest": 4, "2463": 4, "000035": 4, "3773": 4, "004620": 4, "1669": 4, "000091": 4, "1669m": 4, "highest": [4, 5], "3773m": 4, "00462": 4, "promptfool": 4, "manual": [4, 5, 7], "redefin": 4, "prompt_comparison": 4, "prompt2": 4, "prompt3": 4, "prompt_fil": 4, "prompt_cont": 4, "BE": 4, "again": 4, "prompt_id": 4, "promptid": 4, "gradingresult": 4, "df_raw": 4, "reset_index": [4, 7], "eas": [4, 5, 7], "seamless": [4, 7], "hf": 4, "plain": [4, 5], "vanilla": 4, "defi": 4, "accustom": 4, "legaci": [4, 5], "unsustain": 4, "prd": 4, "cultiv": [4, 7], "organiz": 4, "stagnat": 4, "alb": 4, "loubna": 4, "anton": 4, "lozhkov": 4, "bakouch": 4, "gabriel": [4, 7], "mart\u00edn": [4, 7], "bl\u00e1zquez": 4, "lewi": 4, "tunstal": 4, "agust\u00edn": 4, "piquer": 4, "andr": 4, "marafioti": 4, "cyril": 4, "zakka": 4, "leandro": 4, "werra": 4, "wolf": 4, "are24": 4, "judgearena": 4, "bps99": 4, "salli": 4, "pott": 4, "barbara": 4, "557": [4, 7], "sciencedirect": 4, "s0377221798003646": 4, "doi": [4, 7, 8], "1016": 4, "s0377": 4, "2217": 4, "00364": 4, "ctj": 4, "jerri": [4, 7], "tworek": [4, 7], "heewoo": [4, 7], "jun": [4, 7], "qime": [4, 7], "henriqu": [4, 7], "pond": [4, 7], "de": [4, 7], "oliveira": [4, 7], "pinto": [4, 7], "harri": [4, 7], "yuri": 4, "burda": 4, "greg": [4, 7], "brockman": [4, 7], "raul": [4, 7], "puri": [4, 7], "gretchen": [4, 7], "krueger": [4, 7], "petrov": [4, 7], "heidi": 4, "khlaaf": 4, "girish": [4, 7], "sastri": [4, 7], "brook": [4, 7], "chan": [4, 7], "grai": [4, 7], "ryder": [4, 7], "mikhail": [4, 7], "pavlov": [4, 7], "alethea": [4, 7], "lukasz": 4, "kaiser": [4, 7], "mohammad": [4, 7], "bavarian": [4, 7], "clemen": [4, 7], "winter": [4, 7], "philipp": 4, "tillet": [4, 7], "felip": [4, 7], "petroski": [4, 7], "dave": [4, 7], "cum": [4, 7], "plappert": 4, "fotio": 4, "chantzi": [4, 7], "barn": 4, "ariel": 4, "herbert": 4, "voss": [4, 7], "hebgen": 4, "guss": 4, "nichol": 4, "paino": [4, 7], "nikola": [4, 7], "tezak": [4, 7], "babuschkin": [4, 7], "suchir": [4, 7], "balaji": [4, 7], "shantanu": [4, 7], "jain": [4, 7], "hess": [4, 7], "carr": 4, "josh": [4, 7], "achiam": [4, 7], "vedant": 4, "misra": 4, "evan": [4, 7], "morikawa": [4, 7], "matthew": 4, "knight": [4, 7], "mile": [4, 7], "brundag": [4, 7], "mira": [4, 7], "murati": [4, 7], "kati": [4, 7], "mayer": [4, 7], "bob": [4, 7, 8], "mcgrew": [4, 7], "ilya": [4, 7], "sutskev": [4, 7], "wojciech": [4, 7], "zaremba": [4, 7], "2107": 4, "03374": 4, "cz": 4, "lianmin": 4, "ying": 4, "sheng": 4, "anastasio": 4, "angelopoulo": 4, "tianl": 4, "dacheng": 4, "banghua": 4, "jordan": [4, 7], "gonzalez": 4, "ion": 4, "stoica": 4, "04132": 4, "cho24a": 4, "francoi": 4, "arcpriz": 4, "cho24b": 4, "dglh24": 4, "yann": 4, "bal\u00e1z": 4, "galambosi": 4, "tatsunori": 4, "hashimoto": 4, "debia": 4, "04475": 4, "fac24a": 4, "wiki": [4, 8], "fac24b": 4, "fac24c": 4, "model_doc": 4, "fac24d": 4, "cookbook": 4, "llm_judg": 4, "fac24f": 4, "fhwt23": 4, "cl\u00e9mentin": 4, "nathan": 4, "habib": 4, "gnh": 4, "julian": 4, "nyarko": 4, "ho": 4, "r\u00e9": 4, "adam": [4, 7], "chilton": 4, "aditya": [4, 7], "narayana": 4, "chohla": 4, "brandon": [4, 7, 8], "waldon": 4, "rockmor": 4, "diego": 4, "zambrano": 4, "dmitri": 4, "talisman": 4, "enam": 4, "hoqu": 4, "faiz": 4, "surani": 4, "frank": [4, 7], "fagan": 4, "galit": 4, "sarfati": 4, "gregori": 4, "dickinson": 4, "haggai": 4, "porat": 4, "hegland": 4, "jessica": [4, 7], "joe": [4, 7], "nudel": 4, "joel": [4, 7], "niklau": 4, "nai": 4, "jonathan": [4, 7], "choi": 4, "margaret": 4, "hagan": 4, "megan": 4, "ma": [4, 7], "livermor": 4, "nikon": 4, "rasumov": 4, "rahe": 4, "nil": 4, "holzenberg": 4, "noam": 4, "kolt": 4, "henderson": 4, "rehaag": 4, "sharad": 4, "shang": 4, "spencer": 4, "sunni": 4, "gandhi": 4, "zur": 4, "varun": 4, "iyer": 4, "zehua": 4, "2308": 4, "11462": 4, "hbb": 4, "collin": 4, "burn": 4, "steven": [4, 7], "basart": [4, 7], "zou": [4, 7], "manta": [4, 7], "mazeika": [4, 7], "03300": 4, "hbd": 4, "maxwel": 4, "forb": 4, "yejin": 4, "curiou": 4, "neural": [4, 8], "degener": 4, "1904": 4, "09751": 4, "hyc": 4, "binyuan": 4, "zeyu": 4, "cui": 4, "jiaxi": 4, "dayiheng": 4, "tianyu": 4, "jiajun": 4, "kai": [4, 7], "dang": 4, "coder": 4, "preprint": [4, 8], "2409": [4, 7], "12186": 4, "lx": 4, "zhen": 4, "xiaohan": 4, "jia": 4, "yuxuan": 4, "lai": 4, "chongyang": 4, "shuai": 4, "nlg": 4, "07103": 4, "lbl": 4, "bommasani": 4, "toni": 4, "dimitri": 4, "tsipra": 4, "dilara": 4, "soylu": 4, "michihiro": 4, "yasunaga": 4, "yian": 4, "deepak": 4, "narayanan": 4, "yuhuai": 4, "newman": 4, "binhang": 4, "bobbi": 4, "ce": 4, "christian": [4, 7], "cosgrov": 4, "acosta": 4, "nava": [4, 7], "drew": 4, "hudson": 4, "zelikman": 4, "esin": 4, "durmu": 4, "faisal": 4, "ladhak": 4, "frieda": 4, "rong": 4, "hongyu": 4, "ren": 4, "huaxiu": 4, "yao": [4, 7], "jue": 4, "keshav": 4, "santhanam": 4, "laurel": 4, "lucia": 4, "mert": 4, "yuksekgonul": 4, "mirac": 4, "suzgun": 4, "niladri": 4, "chatterji": 4, "omar": 4, "khattab": 4, "qian": [4, 7], "chi": [4, 8], "sang": 4, "shibani": [4, 7], "santurkar": [4, 7], "surya": 4, "icard": 4, "tianyi": 4, "vishrav": 4, "chaudhari": 4, "xuechen": 4, "yuhui": 4, "yuta": 4, "koreeda": 4, "2211": 4, "09110": 4, "lbc24": 4, "ronan": 4, "bra": 4, "allenai": 4, "lhe22": [4, 7], "stephani": [4, 7], "owain": [4, 7], "mimic": [4, 7], "falsehood": [4, 7], "2109": [4, 7], "07958": [4, 7], "pzwg23": 4, "shishir": 4, "tianjun": 4, "xin": [4, 7], "gorilla": 4, "15334": 4, "pro24": 4, "dev": 4, "ras24": 4, "sebastian": 4, "scratch": 4, "1633437166": 4, "srf": 4, "shivalika": 4, "angelika": 4, "roman": [4, 7], "adelani": 4, "ngui": 4, "vila": 4, "suero": 4, "peerat": 4, "limkonchotiwat": 4, "kelli": 4, "marchisio": 4, "qi": 4, "leong": 4, "yosephin": 4, "susanto": 4, "raymond": [4, 7], "ng": [4, 7], "shayn": 4, "longpr": 4, "ko": 4, "madelin": 4, "antoin": 4, "bosselut": 4, "oh": 4, "leshem": 4, "choshen": 4, "daphn": 4, "ippolito": 4, "enzo": 4, "ferrant": 4, "marzieh": 4, "fadae": 4, "beyza": 4, "ermi": 4, "sara": 4, "hooker": 4, "linguist": [4, 7], "03304": 4, "srr": 4, "aarohi": 4, "abhinav": 4, "rastogi": 4, "abhishek": 4, "rao": 4, "abu": 4, "awal": 4, "shoeb": 4, "abubakar": 4, "abid": 4, "fisch": 4, "santoro": 4, "gupta": 4, "adri\u00e0": 4, "garriga": 4, "alonso": 4, "agnieszka": 4, "kluska": 4, "aitor": 4, "lewkowycz": 4, "akshat": 4, "warstadt": 4, "alexand": [4, 7, 8], "kocurek": 4, "ali": [4, 7], "safaya": 4, "tazarv": 4, "aman": 4, "hussain": 4, "dsouza": 4, "ambros": 4, "slone": 4, "ameet": 4, "rahan": 4, "anantharaman": 4, "ander": 4, "andreassen": 4, "madotto": 4, "santilli": 4, "stuhlm\u00fcller": 4, "la": 4, "lampinen": 4, "angelica": 4, "anh": 4, "vuong": 4, "animesh": 4, "gottardi": 4, "antonio": 4, "norelli": 4, "anu": 4, "venkatesh": 4, "arash": 4, "gholamidavoodi": 4, "arfa": 4, "tabassum": 4, "arul": 4, "menez": 4, "arun": [4, 7], "kirubarajan": 4, "asher": 4, "mullokandov": 4, "ashish": 4, "sabharw": 4, "herrick": 4, "avia": 4, "efrat": 4, "aykut": 4, "erdem": 4, "ayla": 4, "karaka\u015f": 4, "bao": [4, 7], "loe": 4, "barret": [4, 7], "zoph": [4, 7], "bart\u0142omiej": 4, "bojanowski": 4, "batuhan": 4, "\u00f6zyurt": 4, "behnam": 4, "hedayatnia": 4, "neyshabur": 4, "inden": 4, "benno": 4, "stein": 4, "berk": 4, "ekmekci": 4, "blake": 4, "howald": 4, "bryan": 4, "orinion": 4, "diao": 4, "dour": 4, "stinson": 4, "cedrick": 4, "argueta": 4, "c\u00e9sar": 4, "ferri": 4, "ram\u00edrez": 4, "chandan": 4, "charl": 4, "rathkopf": 4, "chenlin": 4, "meng": 4, "chitta": 4, "baral": 4, "chiyu": 4, "callison": 4, "burch": 4, "wait": [4, 7], "voigt": 4, "cindi": 4, "ramirez": 4, "clara": 4, "rivera": 4, "clemencia": 4, "siro": 4, "colin": 4, "raffel": 4, "courtnei": 4, "ashcraft": 4, "cristina": 4, "garbacea": 4, "damien": [4, 7], "sileo": 4, "garrett": 4, "kilman": 4, "freeman": 4, "khashabi": 4, "levi": [4, 7], "mosegu\u00ed": 4, "gonz\u00e1lez": 4, "perszyk": 4, "danqi": 4, "dar": 4, "gilboa": 4, "dohan": [4, 7], "drakard": 4, "jurgen": 4, "debajyoti": 4, "datta": 4, "deni": 4, "emelin": 4, "kleyko": 4, "deniz": 4, "yuret": 4, "derek": [4, 7], "tam": [4, 8], "dieuwk": 4, "hupk": 4, "diganta": 4, "dilyar": 4, "buzan": 4, "coelho": 4, "mollo": 4, "diyi": 4, "dylan": 4, "schrader": 4, "ekaterina": 4, "shutova": 4, "ekin": 4, "dogu": 4, "cubuk": 4, "elad": 4, "segal": 4, "eleanor": 4, "hagerman": 4, "donowai": 4, "elli": 4, "pavlick": 4, "rodola": 4, "emma": 4, "lam": 4, "chu": [4, 7], "erkut": 4, "erni": 4, "dyer": 4, "jerzak": 4, "eunic": 4, "engefu": 4, "manyasi": 4, "evgenii": 4, "zheltonozhskii": 4, "fanyu": 4, "xia": 4, "fatemeh": 4, "siar": 4, "fernando": 4, "mart\u00ednez": 4, "plume": 4, "francesca": 4, "happ\u00e9": 4, "gaurav": 4, "genta": 4, "indra": 4, "winata": 4, "gerard": 4, "melo": 4, "germ\u00e1n": 4, "kruszewski": 4, "giambattista": [4, 7], "parascandolo": [4, 7], "giorgio": 4, "mariani": 4, "gloria": 4, "gonzalo": 4, "jaimovitch": 4, "l\u00f3pez": 4, "gregor": 4, "betz": 4, "gui": [4, 5], "gur": 4, "hana": 4, "galijasev": 4, "rashkin": 4, "hannaneh": 4, "hajishirzi": 4, "harsh": 4, "hayden": 4, "bogar": 4, "henri": [4, 7], "shevlin": 4, "hinrich": 4, "sch\u00fctze": 4, "hiromu": 4, "yakura": 4, "hongm": 4, "hugh": 4, "mee": 4, "wong": [4, 7], "isaac": 4, "nobl": 4, "jaap": 4, "jumelet": 4, "geissing": 4, "jaehoon": 4, "jaim": 4, "fern\u00e1ndez": 4, "fisac": 4, "simon": 4, "koppel": 4, "koco\u0144": 4, "jana": 4, "thompson": [4, 5, 7], "janel": 4, "wingfield": 4, "jarema": 4, "radom": 4, "jascha": 4, "sohl": [4, 7], "dickstein": 4, "phang": 4, "yosinski": 4, "jekaterina": 4, "novikova": 4, "jell": 4, "bosscher": 4, "jennif": 4, "marsh": 4, "jeroen": 4, "taal": 4, "engel": 4, "jesujoba": 4, "alabi": 4, "jiam": 4, "jillian": 4, "joan": 4, "waweru": 4, "burden": 4, "bali": 4, "batcheld": 4, "berant": 4, "j\u00f6rg": 4, "frohberg": 4, "jo": 4, "rozen": 4, "orallo": 4, "boudeman": 4, "guerr": 4, "tenenbaum": 4, "joyc": 4, "chua": 4, "kanclerz": 4, "karen": 4, "livescu": 4, "karl": 4, "krauth": 4, "karthik": 4, "gopalakrishnan": 4, "katerina": 4, "ignatyeva": 4, "katja": 4, "markert": 4, "kaustubh": 4, "dhole": 4, "gimpel": 4, "omondi": 4, "kori": 4, "mathewson": 4, "kristen": 4, "chiafullo": 4, "ksenia": 4, "shkaruta": 4, "shridhar": 4, "kyle": [4, 7], "mcdonel": 4, "richardson": 4, "laria": 4, "reynold": 4, "leo": [4, 7], "dugan": 4, "lianhui": 4, "lidia": 4, "contrera": 4, "ochando": 4, "morenc": 4, "moschella": 4, "luci": 4, "ludwig": 4, "schmidt": [4, 7], "luheng": 4, "olivero": 4, "col\u00f3n": 4, "metz": [4, 7], "l\u00fctfi": 4, "kerem": 4, "\u015fenel": 4, "maarten": [4, 7], "bosma": 4, "sap": [4, 7], "maartj": 4, "hoev": 4, "maheen": 4, "farooqi": 4, "manaal": 4, "faruqui": 4, "marco": 4, "baturan": 4, "marelli": 4, "maru": 4, "maria": 4, "quintana": 4, "tolkiehn": 4, "mario": [4, 7], "giulianelli": 4, "martha": 4, "potthast": 4, "leavitt": 4, "hagen": 4, "m\u00e1ty\u00e1": 4, "schubert": 4, "medina": [4, 7], "orduna": 4, "baitemirova": 4, "melodi": 4, "arnaud": 4, "melvin": 4, "mcelrath": 4, "yee": 4, "cohen": 4, "ivanitskii": 4, "starritt": 4, "strube": 4, "micha\u0142": 4, "sw\u0119drowski": 4, "michel": [4, 7], "bevilacqua": 4, "mihir": 4, "kale": 4, "cain": 4, "mime": 4, "mitch": 4, "walker": 4, "mo": 4, "tiwari": 4, "mohit": 4, "bansal": 4, "moin": 4, "aminnaseri": 4, "mor": 4, "geva": 4, "mozhdeh": 4, "gheini": 4, "mukund": 4, "varma": 4, "nanyun": 4, "peng": [4, 7], "nayeon": 4, "neta": 4, "krakov": 4, "doiron": 4, "nicol": 4, "martinez": 4, "nikita": 4, "nangia": 4, "nikla": 4, "decker": 4, "muennighoff": 4, "nitish": [4, 7], "shirish": [4, 7], "keskar": [4, 7], "niveditha": 4, "constant": 4, "fiedel": 4, "nuan": 4, "wen": 4, "oliv": [4, 7], "agha": 4, "elbaghdadi": 4, "omer": 4, "moreno": 4, "casar": 4, "parth": 4, "doshi": 4, "pascal": 4, "fung": 4, "pu": 4, "vicol": 4, "pegah": 4, "alipoormolabashi": 4, "peiyuan": 4, "eckerslei": 4, "phu": 4, "mon": 4, "htut": 4, "pinyu": 4, "hwang": 4, "piotr": 4, "mi\u0142kowski": 4, "piyush": 4, "pouya": 4, "pezeshkpour": 4, "priti": 4, "oli": 4, "qiaozhu": 4, "qing": 4, "lyu": 4, "qinlang": 4, "rabin": 4, "banjad": 4, "rachel": [4, 7], "etta": 4, "rudolph": 4, "raefer": 4, "rahel": 4, "haback": 4, "ramon": 4, "risco": 4, "rapha\u00ebl": 4, "milli\u00e8r": 4, "rhythm": 4, "garg": 4, "rif": 4, "saurou": 4, "riku": 4, "arakawa": 4, "robb": 4, "raymaek": 4, "rohan": 4, "sikand": 4, "novak": 4, "sitelew": 4, "lebra": 4, "rosann": 4, "rowan": [4, 7], "ruslan": 4, "salakhutdinov": 4, "stoval": 4, "teehan": 4, "sahib": 4, "saif": 4, "sajant": 4, "anand": [4, 7], "dillav": 4, "shleifer": 4, "wiseman": 4, "gruetter": 4, "schoenholz": 4, "sanghyun": 4, "sanjeev": 4, "kwatra": 4, "sarik": 4, "ghazarian": 4, "sayan": 4, "casei": [4, 7], "bischoff": 4, "gehrmann": 4, "schuster": 4, "sepideh": 4, "sadeghi": 4, "shadi": 4, "hamdan": 4, "sharon": 4, "shashank": 4, "sherri": 4, "shi": 4, "shikhar": 4, "shima": 4, "asaadi": 4, "shubh": 4, "pachchigar": 4, "shubham": 4, "toshniw": 4, "shyam": [4, 7], "upadhyai": 4, "shyamolima": 4, "debnath": 4, "siamak": 4, "shakeri": 4, "thormey": 4, "melzi": 4, "siva": 4, "reddi": 4, "sneha": 4, "priscilla": 4, "makini": 4, "soo": 4, "hwan": 4, "toren": 4, "sriharsha": 4, "hatwar": 4, "stanisla": 4, "dehaen": 4, "stefan": 4, "divic": 4, "stella": 4, "biderman": 4, "stephen": 4, "prasad": 4, "piantadosi": 4, "stuart": [4, 7], "shieber": 4, "summer": [4, 7], "misherghi": 4, "svetlana": 4, "kiritchenko": 4, "swaroop": 4, "tal": 4, "linzen": 4, "tariq": 4, "tatsu": 4, "te": 4, "th\u00e9o": 4, "desbord": 4, "theodor": 4, "rothschild": 4, "phan": [4, 7], "tiberiu": 4, "nkinyili": 4, "timo": 4, "schick": 4, "timofei": 4, "kornev": 4, "titu": 4, "tunduni": 4, "gerstenberg": 4, "trenton": 4, "trishala": 4, "neeraj": 4, "tushar": 4, "khot": 4, "shultz": 4, "uri": 4, "shaham": 4, "vera": 4, "demberg": 4, "victoria": [4, 7], "nyamai": 4, "vika": 4, "raunak": 4, "vinai": 4, "ramasesh": 4, "udai": 4, "prabhu": 4, "vishakh": 4, "padmakumar": 4, "vivek": 4, "srikumar": 4, "fedu": [4, 7], "wout": 4, "vossen": 4, "xiaoyu": 4, "tong": [4, 7], "xinran": 4, "xinyi": 4, "yadollah": 4, "yaghoobzadeh": 4, "yair": 4, "lakretz": 4, "yangqiu": 4, "yasaman": 4, "bahri": 4, "yichi": 4, "yide": 4, "yifu": 4, "yonatan": 4, "belinkov": 4, "yufang": 4, "seid": 4, "zhuoy": 4, "zijian": 4, "ziji": 4, "zirui": 4, "ziyi": 4, "extrapol": 4, "2206": 4, "04615": 4, "wpn": 4, "yada": 4, "pruksachatkun": 4, "amanpreet": 4, "hill": 4, "stickier": 4, "wsm": 4, "1804": 4, "07461": 4, "wtb": 4, "tai": 4, "borgeaud": 4, "dani": 4, "yogatama": 4, "denni": [4, 7], "donald": 4, "metzler": 4, "ed": 4, "oriol": 4, "vinyal": 4, "dean": 4, "07682": 4, "wdr": 4, "doolei": 4, "manlei": 4, "arka": [4, 7], "pal": 4, "feuer": 4, "siddhartha": 4, "ravid": 4, "shwartz": [4, 7], "ziv": 4, "khalid": 4, "saifullah": 4, "siddartha": 4, "naidu": 4, "chinmai": 4, "hegd": 4, "lecun": 4, "goldstein": 4, "willi": 4, "neiswang": 4, "micah": 4, "goldblum": 4, "19314": 4, "yyh": 4, "baosong": 4, "chengpeng": 4, "chengyuan": 4, "fei": 4, "guant": 4, "haoran": 4, "huan": 4, "jialong": 4, "jialin": 4, "jianhong": 4, "tu": 4, "jianwei": 4, "jianxin": 4, "jin": [4, 7], "jingren": 4, "jinz": 4, "jinzheng": 4, "junyang": 4, "keme": 4, "keqin": 4, "kexin": 4, "mingfeng": 4, "xue": [4, 7], "ni": 4, "pei": 4, "ru": 4, "men": 4, "ruiz": 4, "runji": 4, "shiji": 4, "sinan": 4, "tianhang": 4, "wenbin": 4, "ge": 4, "xiaodong": 4, "deng": 4, "xiaohuan": 4, "xingzhang": 4, "xinyu": [4, 7], "xipin": 4, "xuancheng": 4, "yichang": 4, "wan": 4, "yunfei": 4, "yuqiong": 4, "zhenru": 4, "zhihao": 4, "10671": 4, "zcl24": 4, "zhihan": 4, "cao": 4, "lizi": 4, "openreview": 4, "forum": 4, "aegrf1uy0p": 4, "zc": 4, "siyuan": 4, "zhuang": [4, 7], "zhanghao": 4, "yonghao": 4, "zi": 4, "zhuohan": 4, "xing": [4, 7], "2306": 4, "05685": 4, "huggingface24": 4, "06": [4, 8], "metaai24": 4, "di": 5, "hunter": 5, "hipaa": 5, "properti": [5, 7], "gdpr": 5, "iot": 5, "unreli": 5, "impract": 5, "programm": 5, "graphic": [5, 7], "vram": 5, "vector": [5, 7], "mathbf": 5, "x_1": 5, "x_2": 5, "x_n": 5, "x_": 5, "\u03b8": 5, "matrix": [5, 7], "concurr": 5, "mozilla": 5, "docker": 5, "gerganov": 5, "georgi": 5, "hundr": 5, "overwhelm": [5, 8], "manifesto": 5, "enjoy": 5, "bog": 5, "exploratori": 5, "hacker": 5, "Will": [5, 7], "codebas": 5, "prototyp": 5, "prematur": 5, "besid": 5, "lighter": 5, "sacrific": 5, "gguf": 5, "unifi": 5, "ggml": [5, 8], "ibm": [5, 7], "bit": 5, "metadata": 5, "disk": 5, "faster": 5, "backward": 5, "2024x": 5, "repo": 5, "easier": [5, 6, 7, 8], "compil": 5, "linux": 5, "argument": [5, 7, 8], "sudo": 5, "apt": 5, "cmake": 5, "bind": 5, "betlen": 5, "cnv": 5, "llamacpp": 5, "q8_0": 5, "succinct": 5, "ctrl": 5, "interject": 5, "philosoph": 5, "debat": 5, "fulfil": 5, "happi": 5, "responsibli": 5, "bye": 5, "goodby": 5, "port": 5, "127": 5, "curl": [5, 8], "localhost": 5, "v1": [5, 6, 7], "bearer": 5, "finish_reason": 5, "deepli": 5, "1734627879": 5, "completion_token": 5, "total_token": 5, "chatcmpl": 5, "5wl2tzjzdmzupvxwp2gcedr8xbpsyhfm": 5, "prompt_n": 5, "prompt_m": 5, "132": 5, "prompt_per_token_m": 5, "prompt_per_second": 5, "77619878666999": 5, "predicted_n": 5, "predicted_m": 5, "1700": 5, "654": 5, "predicted_per_token_m": 5, "36882142857143": 5, "predicted_per_second": 5, "92850867960208": 5, "wrestl": [5, 7], "gbnf": [5, 8], "8pm": 5, "appointmenttim": 5, "appointmentdetail": 5, "handi": 5, "modal": 5, "model_path": 5, "llama_cpp": 5, "create_chat_complet": 5, "occupi": 5, "activist": 5, "justin": [5, 7], "tunnei": 5, "ocho": 5, "appach": 5, "cosmopolitan": 5, "libc": 5, "portabl": 5, "durabl": 5, "usabl": [5, 7, 8], "tinyllama": 5, "wget": 5, "jartin": 5, "q5_k_m": 5, "renam": 5, "ex": 5, "chmod": 5, "nobrows": 5, "registri": 5, "nativ": [5, 8], "container": 5, "isol": [5, 7], "trai": 5, "familiar": 5, "permiss": [5, 7], "despit": [5, 6, 8], "ssfl": 5, "sh": [5, 8], "Or": 5, "11434": 5, "chatrespons": 5, "tradeoff": [5, 7], "easiest": 5, "rich": [5, 7], "playground": 5, "simultan": [5, 7], "verif": [5, 8], "importantli": 5, "gemma": [5, 8], "intuit": 5, "beginn": 5, "tensorrt": 5, "trt": 5, "latex": 5, "voic": 5, "pwa": 5, "rag": 5, "simpler": [5, 7, 8], "medium": [5, 7, 8], "gpt4all": 5, "rbac": 5, "fp16": 5, "q2_k": 5, "q4_k": 5, "q6_k": 5, "thoughput": 5, "evalu": [5, 6, 8], "wikitext": 5, "salesforc": 5, "wikipedia": [5, 8], "min_prompt_length": 5, "input_texts_raw": 5, "2010": 5, "valkyria": 5, "chronicl": 5, "underw": 5, "forgiv": 5, "newcom": 5, "raita": 5, "honjou": 5, "compos": [5, 7], "hitoshi": 5, "sakimoto": 5, "takeshi": 5, "ozawa": 5, "writer": 5, "theme": [5, 6, 7], "sung": 5, "escap": 5, "escaped_text": 5, "2024w": 5, "block_scal": 5, "block": [5, 7], "width": 5, "parenthes": 5, "block_min": 5, "formula": 5, "superblock": 5, "5625": 5, "ieee": 5, "754": 5, "half": [5, 7], "ppl": 5, "exp": 5, "sum_": 5, "log_2": 5, "x_i": 5, "ratio": 5, "avg": 5, "_i": 5, "corr": 5, "ln": [5, 8], "kullback": 5, "leibler": 5, "logit": [5, 8], "d_": 5, "softmax": 5, "sum": 5, "kld": 5, "q2_kresult": 5, "ppl1": 5, "ppl2": 5, "q6": 5, "004": 5, "q2": 5, "112": 5, "q4": 5, "smallest": 5, "390": 5, "mib": 5, "67": [5, 7], "81": [5, 7], "93": [5, 7], "462": 5, "96": [5, 7], "614": 5, "58": 5, "170": 5, "ubuntu": 5, "lt": 5, "x86_64": 5, "gnu": 5, "feasibl": [5, 6], "thank": [5, 8], "intel": 5, "i7": 5, "8550u": 5, "15gib": 5, "samsung": 5, "ssd": 5, "970": 5, "evo": 5, "500gb": 5, "bc24": 5, "andrei": [5, 7], "abetlen": 5, "gc24": 5, "ggerganov": [5, 8], "blob": [5, 8], "readm": [5, 8], "gc4a": 5, "gc4b": 5, "thread": 5, "huggingface4w": 5, "huggingface4xa": 5, "huggingface4xb": 5, "ibmthink24": 5, "lmstudio24": 5, "lmstudio": 5, "mozillaocho24": 5, "salesforce24": 5, "possibli": 6, "eliot": 6, "thumb": 6, "\u00be": 6, "max_output_token": 6, "4096": 6, "16384": 6, "contrari": 6, "surpass": 6, "truncat": 6, "max_input_token": 6, "input_cost_per_token": 6, "output_cost_per_token": 6, "11b": [6, 7], "128000": 6, "5e": 6, "20241022": [6, 7], "8192": 6, "200000": 6, "3e": 6, "0613": 6, "6e": 6, "gemini": 6, "flash": 6, "1048576": 6, "2097152": 6, "05e": 6, "incomplet": [6, 7], "abruptli": 6, "shallow": 6, "thorough": [6, 7], "dissatisfact": 6, "frustrat": [6, 7], "10k": 6, "diagram": [6, 7], "charactertextsplitt": 6, "tiktoken": [6, 7], "sequenti": 6, "newlin": 6, "broadli": [6, 8], "cheap": 6, "speciali": 6, "nltk": 6, "spaci": 6, "hierarch": [6, 7], "talk": 6, "splitter": 6, "get_chunk": 6, "chunk_siz": 6, "chunk_overlap": 6, "langchain_text_splitt": 6, "text_splitt": 6, "from_tiktoken_encod": 6, "split_text": 6, "persona": 6, "langchain_cor": [6, 8], "prompttempl": 6, "get_base_prompt_templ": 6, "base_prompt": [6, 8], "from_templ": 6, "llmchain": 6, "parser": [6, 8], "output_pars": 6, "stroutputpars": 6, "langchain_commun": 6, "chat_model": 6, "chatlitellm": 6, "get_llm_chain": 6, "prompt_templ": [6, 8], "llm_chain": [6, 8], "api_key_label": 6, "upper": 6, "_api_kei": 6, "get_dynamic_prompt_templ": 6, "get_dynamic_prompt_param": 6, "prompt_param": 6, "part_idx": 6, "total_part": 6, "chat_context": 6, "param": 6, "dynamic_prompt_param": 6, "introduct": 6, "concaten": 6, "generate_report": 6, "input_cont": 6, "llm_model_nam": 6, "report_part": 6, "num_part": 6, "dinam": 6, "priovid": 6, "invok": [6, 8], "cummul": 6, "max_chunk_s": 6, "max_chunk_overlap": 6, "readabl": 6, "apple_report": 6, "luation": 6, "disciplin": 6, "subhead": 6, "depth": [6, 7], "overlook": 6, "preprocess": [6, 8], "necessit": 6, "meticul": 6, "bottleneck": 6, "mustafa": 6, "suleyman": 6, "infinit": 6, "fewer": [6, 7], "condens": 6, "versatil": 6, "grace": 6, "fallback": 6, "empow": [6, 7], "langchain24": 6, "how_to": 6, "immens": 7, "commonplac": 7, "hartvigsen": 7, "societi": 7, "statement": 7, "alarm": 7, "openli": 7, "dolli": 7, "v2": 7, "llama2": [7, 8], "13b": 7, "emb": 7, "generalist": 7, "injustic": 7, "inequ": 7, "undermin": 7, "perpetu": 7, "displac": 7, "eros": 7, "fake": 7, "deepfak": 7, "distrust": 7, "cyberattack": 7, "spread": 7, "disinform": 7, "inadvert": 7, "interven": 7, "irrevers": 7, "uncheck": 7, "extinct": 7, "race": 7, "incentiv": 7, "shortcut": 7, "behind": 7, "stress": 7, "urgent": 7, "reorient": 7, "birth": 7, "siam": 7, "edgington": 7, "jailbreak": 7, "promptcraft": 7, "stealth": 7, "sutton": 7, "subtl": 7, "trigger": 7, "subtleti": 7, "exception": 7, "phrase": 7, "evad": 7, "hqve": 7, "frer": 7, "hplidai": 7, "pl": 7, "hyperion": 7, "coast": 7, "redwood": 7, "tallest": 7, "tree": [7, 8], "routin": 7, "prejudic": 7, "gallego": 7, "leak": 7, "poison": 7, "intention": 7, "inject": 7, "mislead": 7, "exabeam": 7, "finra": 7, "3110": 7, "mandat": 7, "supervisori": 7, "medicin": 7, "unicef": 7, "contest": 7, "congress": 7, "enact": 7, "pictur": [7, 8], "territori": 7, "oversea": 7, "chines": 7, "legitim": 7, "consent": 7, "complaint": 7, "cooper": 7, "extraterritori": 7, "offshor": 7, "draft": 7, "voluntari": 7, "neutral": 7, "player": 7, "prepared": 7, "ahead": 7, "compris": 7, "cbrn": 7, "persuas": 7, "autonomi": 7, "gradat": 7, "scorecard": 7, "elig": 7, "advisori": 7, "sag": 7, "shut": 7, "prerequisit": 7, "harden": 7, "asl": 7, "biosafeti": 7, "elev": 7, "warn": 7, "bioweapon": 7, "compartment": 7, "difficulti": 7, "4x": 7, "jump": 7, "paus": 7, "frontier": 7, "deepmind": 7, "biosecur": 7, "buffer": 7, "formul": [7, 8], "calibr": 7, "promin": 7, "taxonomi": 7, "llamaguard": 7, "leaderboard": 7, "3x": 7, "5x": 7, "alaga": 7, "substandard": 7, "rapidli": [7, 8], "oxford": 7, "wachter": 7, "blur": 7, "ill": 7, "stifl": 7, "suscept": 7, "aadc": 7, "outset": 7, "curricula": 7, "adversari": 7, "uncov": [7, 8], "appar": 7, "thoroughli": 7, "lm": [7, 8], "problemat": 7, "arrai": 7, "undergo": 7, "280b": 7, "cai": [7, 8], "utilis": 7, "minimis": 7, "enshrin": 7, "evas": 7, "resort": 7, "encod": 7, "avenu": 7, "cambria": 7, "inherit": 7, "influenti": 7, "debias": 7, "occurr": 7, "phish": 7, "clarifi": 7, "toler": 7, "checklist": 7, "abus": 7, "ux": 7, "architect": 7, "retrofit": 7, "promptli": 7, "dashboard": 7, "misalign": 7, "star": 7, "postpon": 7, "sens": 7, "combat": 7, "counter": 7, "traffic": 7, "workaround": 7, "vital": 7, "silo": 7, "hierarchi": 7, "66": 7, "mcq": 7, "regex": [7, 8], "joint": 7, "facet": 7, "purpl": 7, "circl": 7, "opensafetylab": 7, "salad_bench_dataset": 7, "base_set": 7, "gptfuzzer": 7, "auto": 7, "qid": 7, "o1": 7, "supremaci": 7, "o53": 7, "o14": 7, "o5": 7, "o65": 7, "plagiar": 7, "o16": 7, "o6": 7, "o47": 7, "campaign": 7, "o12": 7, "o52": 7, "surveil": 7, "spous": 7, "know": [7, 8], "o13": 7, "breakdown": [7, 8], "ncount": 7, "21318": 7, "8756": 7, "6486": 7, "o2": 7, "1717": 7, "o4": 7, "1477": 7, "o3": 7, "socioeconom": 7, "851": 7, "int64": 7, "gen": 7, "15433": 7, "hh": 7, "4184": 7, "659": 7, "advbench": 7, "230": 7, "189": 7, "toxicchat": 7, "anyth": 7, "817": 7, "misconcept": 7, "ingrain": 7, "mc1": 7, "singular": 7, "choices4": 7, "mc2": 7, "set4": 7, "scorer": 7, "correctli": [7, 8], "truthful_qa": 7, "truthfulqa_dataset": 7, "multiple_choic": 7, "best_answ": 7, "correct_answ": 7, "incorrect_answ": 7, "watermelon": 7, "digest": 7, "noth": 7, "stomach": 7, "sick": 7, "wonderopoli": 7, "wonder": 7, "belli": 7, "swallow": 7, "dream": 7, "die": 7, "indigest": 7, "unconsci": 7, "excret": 7, "asr": 7, "r2d2": 7, "multimod": 7, "wider": [7, 8], "mmlu": 7, "mass": 7, "destruct": 7, "asynchron": 7, "webpurifi": 7, "protectai": 7, "aw": 7, "comprehend": 7, "amazon": 7, "nemo": 7, "nvidia": 7, "keyword": 7, "toolset": 7, "nemmo": 7, "synchron": 7, "nemoguardrail": 7, "llmrail": 7, "railsconfig": 7, "from_path": 7, "rail": 7, "hello": 7, "ministr": 7, "mistralai": 7, "mistral_api_kei": 7, "moderate_chat": 7, "omni": 7, "pprint": 7, "to_json": 7, "threaten": 7, "illicit": 7, "granit": 7, "guardian": 7, "consortium": 7, "v3": 7, "begin_of_text": 7, "start_header_id": 7, "end_header_id": 7, "unsafe_categori": 7, "user_message_1": 7, "model_answer_1": 7, "comma": 7, "eot_id": 7, "eom_id": 7, "denot": 7, "s1": 7, "s2": 7, "s3": 7, "s4": 7, "s5": 7, "defam": 7, "s6": 7, "s7": 7, "s8": 7, "s9": 7, "s10": 7, "s11": 7, "s12": 7, "s13": 7, "atla": 7, "2b": 7, "hap": 7, "38m": 7, "125m": 7, "padhi": 7, "shieldgemma": 7, "exemplifi": 7, "accomplish": [7, 8], "judge_prompt": 7, "american": 7, "vandal": 7, "underag": 7, "drink": 7, "vulgar": 7, "obscen": 7, "racism": 7, "derogatori": 7, "firearm": 7, "safety_scor": 7, "IN": 7, "borderlin": 7, "verdict": 7, "boolean": [7, 8], "brief": 7, "rational": 7, "delimit": 7, "paramount": 7, "evenli": 7, "good_sampl": 7, "bad_sampl": 7, "2024z": 7, "surg": 7, "scam": 7, "get_profanity_sampl": 7, "show_stat": 7, "bool": 7, "current_dir": 7, "getcwd": 7, "data_path": 7, "profanity_en": 7, "random_st": 7, "ncategori": 7, "category_count": 7, "category_1": 7, "1f": 7, "profanity_sampl": 7, "nsampl": 7, "anatomi": 7, "slur": 7, "182": 7, "bodili": 7, "fluid": 7, "excrement": 7, "insult": 7, "mental": 7, "disabl": 7, "jap": 7, "babi": 7, "batter": 7, "crazi": 7, "sob": 7, "fukka": 7, "sh1t3": 7, "get_salad_sampl": 7, "salad_sampl": 7, "tortur": 7, "porn": 7, "sin": 7, "sight": 7, "god": 7, "embezzl": 7, "xanax": 7, "alcohol": 7, "get_good_sampl": 7, "min_scor": 7, "reichstag": 7, "profanity_data": 7, "salad_data": 7, "good_data": 7, "all_data": 7, "prompt_sampl": 7, "is_unsaf": 7, "counti": 7, "holli": 7, "ridg": 7, "nc": 7, "town": 7, "onslow": 7, "carolina": 7, "diver": 7, "underwat": 7, "maze": 7, "coral": 7, "treasur": 7, "vivid": 7, "sensori": 7, "emot": 7, "labyrinthin": 7, "passag": 7, "reef": 7, "suspens": 7, "obstacl": 7, "creatur": 7, "nomin": 7, "nobel": 7, "literatur": 7, "love": 7, "ny": [7, 8], "logo": 7, "thief": 7, "rob": 7, "famou": 7, "nstatist": 7, "source_stat": 7, "type_stat": 7, "plug": 7, "safetyvalid": 7, "validationresult": 7, "dataclass": 7, "abstractmethod": 7, "llmguardvalid": 7, "scanner": 7, "bantop": 7, "llm_guard": 7, "input_scann": 7, "scan_prompt": 7, "matchtyp": 7, "default_banned_top": 7, "banned_top": 7, "super": 7, "banned_topics_scann": 7, "use_onnx": 7, "toxicity_scann": 7, "match_typ": 7, "fail_fast": 7, "unsafe_scann": 7, "gun": 7, "cool": 7, "hunt": 7, "deer": 7, "dad": 7, "mistralvalid": 7, "hate_and_discrimin": 7, "violence_and_threat": 7, "dangerous_and_criminal_cont": 7, "selfharm": 7, "openaivalid": 7, "attr": 7, "dir": 7, "getattr": 7, "illicit_viol": 7, "llmjudgevalid": 7, "prompt_path": 7, "llmjudg": 7, "filenotfounderror": 7, "elaps": 7, "score_valid": 7, "id_": 7, "validator_inst": 7, "start_tim": 7, "validation_result": 7, "elapsed_tim": 7, "prompt_sample_id": 7, "validator_nam": 7, "scoring_prompt": 7, "scoring_result": 7, "01536": 7, "34098": 7, "497136": 7, "546416": 7, "calculate_validator_metr": 7, "bad_sourc": 7, "good_sourc": 7, "tpr": 7, "fpr": 7, "f1_score": 7, "tn": 7, "fp": 7, "fn": 7, "tp": 7, "255": 7, "74": 7, "238": 7, "130": 7, "256": 7, "79": 7, "149": 7, "213": 7, "190": 7, "235": 7, "63": 7, "222": 7, "82": 7, "75": 7, "480": 7, "157": 7, "487": 7, "495": 7, "482": 7, "667": 7, "248": 7, "466": 7, "143": 7, "355": 7, "slowest": 7, "room": 7, "false_posit": 7, "left_on": 7, "right_on": 7, "persian": 7, "teenag": 7, "\u0686\u0627\u0644\u0634": 7, "\u0645\u0648\u0645\u0648": 7, "\u06cc\u06a9": 7, "\u062d\u0642\u0647": 7, "\u0648": 7, "\u0627\u0641\u0633\u0627\u0646\u0647": 7, "\u0645\u062d\u0644\u06cc": 7, "\u0627\u0633\u062a": 7, "\u06a9\u0647": 7, "\u0634\u0627\u06cc\u0639\u0647": 7, "\u0622\u0646": 7, "\u062f\u0631": 7, "\u0634\u0628\u06a9\u0647": 7, "\u0647\u0627\u06cc": 7, "\u0627\u062c\u062a\u0645\u0627\u0639\u06cc": 7, "\u067e\u062e\u0634": 7, "\u0634\u062f": 7, "\u06af\u0632\u0627\u0631\u0634": 7, "\u06a9\u0627\u0631\u0628\u0631\u06cc": 7, "\u0628\u0627": 7, "\u0646\u0627\u0645": 7, "\u06a9\u0648\u062f\u06a9\u0627\u0646": 7, "\u0646\u0648\u062c\u0648\u0627\u0646\u0627\u0646": 7, "\u0631\u0627": 7, "\u0628\u0647": 7, "\u0627\u0639\u0645\u0627\u0644": 7, "\u062e\u0637\u0631\u0646\u0627\u06a9\u06cc": 7, "\u0645\u0627\u0646\u0646\u062f": 7, "\u062e\u0648\u062f\u0632\u0646\u06cc": 7, "\u062e\u0648\u062f\u06a9\u0634\u06cc": 7, "\u062a\u0631\u063a\u06cc\u0628": 7, "\u0645\u06cc": 7, "\u06a9\u0646\u062f": 7, "\u0648\u062c\u0648\u062f": 7, "\u0635\u062d\u0628\u062a": 7, "\u0632\u06cc\u0627\u062f\u06cc": 7, "\u0645\u0648\u0631\u062f": 7, "\u0627\u06cc\u0646": 7, "\u062a\u0639\u062f\u0627\u062f": 7, "\u0634\u06a9\u0627\u06cc\u0627\u062a": 7, "\u0648\u0627\u0642\u0639\u06cc": 7, "\u0628\u0633\u06cc\u0627\u0631": 7, "\u06a9\u0645": 7, "\u0628\u0648\u062f": 7, "\u067e\u0644\u06cc\u0633": 7, "\u0635\u062f\u0645\u0647": 7, "\u062f\u06cc\u062f\u0646": 7, "\u062f\u0644\u06cc\u0644": 7, "\u062a\u0623\u062b\u06cc\u0631": 7, "\u0645\u0633\u062a\u0642\u06cc\u0645": 7, "\u067e\u062f\u06cc\u062f\u0647": 7, "\u062a\u0623\u06cc\u06cc\u062f": 7, "\u0646\u06a9\u0631\u062f\u0647": 7, "\u062a\u0631\u0633": 7, "\u0646\u06af\u0631\u0627\u0646\u06cc": 7, "\u0627\u06cc\u062c\u0627\u062f": 7, "\u0634\u062f\u0647": 7, "\u0628\u06cc\u0634\u062a\u0631": 7, "\u0627\u0632": 7, "\u062e\u0648\u062f": 7, "\u0631\u0633\u0627\u0646\u0647": 7, "\u0647\u0627": 7, "\u0637\u0648\u0631\u06cc": 7, "\u062e\u06cc\u0631\u06cc\u0647": 7, "\u0647\u0634\u062f\u0627\u0631": 7, "\u062f\u0627\u062f\u0646\u062f": 7, "\u0622\u0633\u06cc\u0628": 7, "\u0627\u0646\u062a\u0638\u0627\u0631\u0627\u062a": 7, "\u0645\u062d\u062a\u0648\u0627\u06cc": 7, "\u062e\u0634\u0648\u0646\u062a": 7, "\u0622\u0645\u06cc\u0632": 7, "\u0627\u06cc\u0646\u062a\u0631\u0646\u062a": 7, "\u06af\u0641\u062a\u0647": 7, "\u0634\u0648\u062f": 7, "\u0627\u0648\u0644\u06cc\u0646": 7, "\u0628\u0627\u0631": 7, "\u0633\u0627\u0644": 7, "\u06f2\u06f0\u06f1\u06f8": 7, "\u067e\u0633": 7, "\u0622\u0646\u06a9\u0647": 7, "\u0631\u0648\u0632\u0646\u0627\u0645\u0647": 7, "\u0627\u0646\u062f\u0648\u0646\u0632\u06cc\u0627\u06cc\u06cc": 7, "\u062e\u0628\u0631": 7, "\u062f\u062e\u062a\u0631": 7, "\u06f1\u06f2": 7, "\u0633\u0627\u0644\u0647": 7, "\u062f\u0627\u062f": 7, "\u0645\u0648\u0636\u0648\u0639": 7, "\u062c\u0647\u0627\u0646\u06cc": 7, "\u062a\u0628\u062f\u06cc\u0644": 7, "\u0645\u062c\u0633\u0645\u0647": 7, "\u0647\u0646\u0631\u0645\u0646\u062f": 7, "\u0698\u0627\u067e\u0646\u06cc": 7, "\u0647\u0631": 7, "\u0686\u0646\u062f": 7, "\u0634\u0627\u06cc\u062f": 7, "\u0646\u06af\u0627\u0647": 7, "\u0628\u0639\u0636\u06cc": 7, "\u0632\u06cc\u0628\u0627": 7, "\u0646\u0628\u0627\u0634\u062f": 7, "\u0627\u0645\u0627": 7, "\u06a9\u0627\u0645\u0644\u0627": 7, "\u0628\u06cc": 7, "\u062e\u0637\u0631": 7, "\u0627\u06cc\u0631\u0627\u0646": 7, "\u0645\u062f\u062a": 7, "\u0628\u06cc\u0646": 7, "\u06a9\u0627\u0631\u0628\u0631\u0627\u0646": 7, "\u0645\u0637\u0631\u062d": 7, "\u0633\u0627\u0644\u06cc": 7, "\u0633\u0631\u0627\u0633\u0631": 7, "\u062c\u0647\u0627\u0646": 7, "\u0645\u0634\u0627\u0628\u0647\u06cc": 7, "\u0628\u0631\u0627\u06cc": 7, "\u0648\u0627\u0644\u062f\u06cc\u0646": 7, "\u06a9\u0631\u062f\u0647": 7, "\u0627\u0641\u0631\u0627\u062f": 7, "\u0686\u0647": 7, "\u06a9\u0627\u0631\u06cc": 7, "\u062f\u0639\u0648\u062a": 7, "tourist": 7, "distress": 7, "polish": 7, "galician": 7, "dzisiaj": 7, "szwecji": 7, "innych": 7, "bogatych": 7, "krajach": 7, "ludzi": 7, "u\u017cywaj\u0105": 7, "mn\u00f3stwo": 7, "najr\u00f3\u017cniejszych": 7, "urz\u0105dze\u0144": 7, "hox": 7, "suecia": 7, "outro": 7, "pa\u00eds": 7, "rico": 7, "xent": 7, "usa": [7, 8], "moita": 7, "m\u00e1quina": 7, "diferent": 7, "\u0142\u00f3dka": 7, "zaczyna": 7, "ton\u0105\u0107": 7, "tury\u015bci": 7, "wracaj\u0105": 7, "statek": 7, "dom\u00f3w": 7, "gdzie": 7, "opowiadaj\u0105": 7, "tym": 7, "jak": 7, "zostali": 7, "zaatakowani": 7, "surprisingli": 7, "unsettl": 7, "paradox": 7, "harbor": 7, "wisdom": 7, "aspir": 7, "technologist": 7, "disciplinari": 7, "ethicist": 7, "policymak": 7, "ai24": 7, "asa24": 7, "jide": 7, "jona": 7, "schuett": 7, "marku": 7, "anderljung": 7, "08751": 7, "bhy": 7, "geoffrei": 7, "hinton": 7, "pieter": 7, "abbeel": 7, "trevor": 7, "darrel": 7, "yuval": 7, "harari": 7, "ya": 7, "lan": 7, "shai": 7, "shalev": 7, "gillian": 7, "hadfield": 7, "clune": 7, "tegan": 7, "maharaj": 7, "hutter": 7, "at\u0131l\u0131m": 7, "g\u00fcne\u015f": 7, "baydin": 7, "sheila": 7, "mcilraith": 7, "qiqi": 7, "ashwin": 7, "acharya": 7, "anca": 7, "dragan": 7, "philip": 7, "torr": 7, "russel": 7, "kahneman": 7, "s\u00f6ren": 7, "mindermann": 7, "amid": 7, "384": 7, "6698": 7, "1126": 7, "adn0117": 7, "bbc": 7, "emili": 7, "braca": 7, "israel": 7, "carter": 7, "hafsa": 7, "kanchwala": 7, "khojasteh": 7, "charli": 7, "landow": 7, "luo": 7, "magarelli": 7, "mirin": 7, "averi": 7, "moyer": 7, "kayla": 7, "simpson": 7, "amelia": 7, "skawinski": 7, "heverin": 7, "23308": 7, "bmc": 7, "dillon": 7, "brendan": 7, "murphi": 7, "khachaturov": 7, "gleav": 7, "kellin": 7, "pelrin": 7, "2408": [7, 8], "02946": 7, "cmm": 7, "erik": 7, "lorenzo": 7, "malandri": 7, "fabio": 7, "mercorio": 7, "navid": 7, "nobani": 7, "seveso": 7, "15248": 7, "edg24": 7, "exa24": 7, "cyber": 7, "grb": 7, "rossi": 7, "barrow": 7, "mehrab": 7, "tanjim": 7, "sungchul": 7, "franck": 7, "dernoncourt": 7, "ruiyi": 7, "nesreen": 7, "2309": 7, "00770": 7, "h44z": 7, "hgp": 7, "saadia": 7, "hamid": 7, "palangi": 7, "dipankar": 7, "ec": 7, "kamar": 7, "oxi": 7, "smaranda": 7, "muresan": 7, "preslav": 7, "nakov": 7, "alin": 7, "villavicencio": 7, "editor": 7, "60th": 7, "3309": 7, "3326": 7, "dublin": 7, "aclanthologi": 7, "acl": 7, "18653": 7, "hym": 7, "weijiang": 7, "weitao": 7, "weihong": 7, "zhangyin": 7, "haotian": 7, "qianglong": 7, "weihua": 7, "xiaocheng": 7, "bing": 7, "ting": 7, "dx": 7, "1145": [7, 8], "3703155": 7, "ldw": 7, "lijun": 7, "ruohui": 7, "xuhao": 7, "wangmeng": 7, "zuo": 7, "dahua": 7, "qiao": 7, "shao": 7, "05044": 7, "mpy": 7, "xuwang": 7, "zifan": 7, "norman": 7, "mu": 7, "elham": 7, "sakhae": 7, "nathaniel": 7, "forsyth": 7, "04249": 7, "mlc24": 7, "illumin": 7, "ailumin": 7, "oaa": 7, "adler": 7, "ahmad": 7, "ilg": 7, "akkaya": 7, "florencia": 7, "leoni": 7, "aleman": 7, "janko": 7, "altenschmidt": 7, "altman": 7, "shyamal": 7, "anadkat": 7, "avila": 7, "valeri": 7, "balcom": 7, "baltescu": 7, "haim": 7, "belgum": 7, "irwan": 7, "bello": 7, "jake": 7, "berdin": 7, "bernadett": 7, "shapiro": 7, "berner": 7, "lenni": 7, "bogdonoff": 7, "boiko": 7, "madelain": 7, "boyd": 7, "luisa": 7, "brakman": 7, "button": 7, "rosi": 7, "campbel": 7, "cann": 7, "brittani": 7, "carei": 7, "carlson": 7, "rori": 7, "carmichael": 7, "che": 7, "foti": 7, "sulli": 7, "rubi": 7, "chess": 7, "chester": 7, "cho": 7, "hyung": 7, "won": 7, "chung": 7, "jeremiah": 7, "currier": 7, "yunx": 7, "cori": 7, "decareaux": 7, "degri": 7, "deutsch": 7, "devil": 7, "dhar": 7, "steve": 7, "dowl": 7, "dun": 7, "adrien": 7, "ecoffet": 7, "atti": 7, "eleti": 7, "tyna": 7, "elound": 7, "farhi": 7, "niko": 7, "sim\u00f3n": 7, "posada": 7, "fishman": 7, "juston": 7, "isabella": 7, "fulford": 7, "georg": 7, "gibson": 7, "vik": 7, "tarun": 7, "gogineni": 7, "goh": 7, "rapha": 7, "gontijo": 7, "lope": 7, "gordon": 7, "morgan": 7, "grafstein": 7, "yufei": 7, "guo": 7, "hallaci": 7, "heaton": 7, "johann": 7, "heideck": 7, "hickei": 7, "wade": 7, "hoeschel": 7, "houghton": 7, "kenni": 7, "hsu": 7, "shengli": 7, "joost": 7, "huizinga": 7, "shawn": 7, "joann": 7, "jang": 7, "roger": 7, "haozhun": 7, "shino": 7, "jomoto": 7, "billi": 7, "jonn": 7, "tomer": 7, "kaftan": 7, "\u0142ukasz": 7, "kamali": 7, "ingmar": 7, "kanitscheid": 7, "tabarak": 7, "khan": 7, "logan": 7, "kilpatrick": 7, "jong": 7, "wook": 7, "christina": 7, "yongjik": 7, "hendrik": 7, "kirchner": 7, "kiro": 7, "matt": 7, "kokotajlo": 7, "kondraciuk": 7, "kondrich": 7, "konstantinidi": 7, "kosic": 7, "vishal": 7, "kuo": 7, "lamp": 7, "ikai": 7, "teddi": 7, "jade": 7, "leung": 7, "chak": 7, "ming": 7, "lim": 7, "molli": 7, "mateusz": 7, "litwin": 7, "theresa": 7, "lopez": 7, "patricia": 7, "lue": 7, "makanju": 7, "malfacini": 7, "markov": 7, "yaniv": 7, "markovski": 7, "bianca": 7, "mayn": 7, "mckinnei": 7, "christin": 7, "mcleavei": 7, "mcmillan": 7, "mcneil": 7, "aalok": 7, "menick": 7, "mishchenko": 7, "vinni": 7, "monaco": 7, "murk": 7, "m\u00e9ly": 7, "ashvin": 7, "nair": 7, "reiichiro": 7, "nakano": 7, "rajeev": 7, "nayak": 7, "arvind": 7, "neelakantan": 7, "hyeonwoo": 7, "noh": 7, "keef": 7, "jakub": 7, "pachocki": 7, "palermo": 7, "ashlei": 7, "pantuliano": 7, "parish": 7, "emi": 7, "parparita": 7, "passo": 7, "perelman": 7, "belbut": 7, "pere": 7, "pokorni": 7, "pokrass": 7, "vitchyr": 7, "pong": 7, "tolli": 7, "powel": 7, "bori": 7, "proehl": 7, "rae": 7, "ramesh": 7, "franci": 7, "kendra": 7, "rimbach": 7, "carl": 7, "rotst": 7, "roussez": 7, "saltarelli": 7, "ted": 7, "sander": 7, "schnurr": 7, "selsam": 7, "kyla": 7, "sheppard": 7, "toki": 7, "sherbakov": 7, "shieh": 7, "shoker": 7, "pranav": 7, "szymon": 7, "sidor": 7, "sigler": 7, "sitkin": 7, "sokolowski": 7, "natali": 7, "staudach": 7, "madelein": 7, "tootoonchian": 7, "tseng": 7, "preston": 7, "tuggl": 7, "turlei": 7, "juan": 7, "cer\u00f3n": 7, "urib": 7, "vallon": 7, "vijayvergiya": 7, "jai": 7, "alvin": 7, "ward": 7, "cj": 7, "weinmann": 7, "akila": 7, "welihinda": 7, "jiayi": 7, "weng": 7, "lilian": 7, "wiethoff": 7, "willner": 7, "wolrich": 7, "lauren": 7, "workman": 7, "sherwin": 7, "yoo": 7, "zeller": 7, "shengjia": 7, "juntang": 7, "zhuk": 7, "2303": 7, "08774": 7, "pnc": 7, "inkit": 7, "manish": 7, "nagireddi": 7, "giandomenico": 7, "cornacchia": 7, "subhajit": 7, "chaudhuri": 7, "tejaswini": 7, "pedapati": 7, "pierr": 7, "dognin": 7, "keerthiram": 7, "murugesan": 7, "miehl": 7, "santill\u00e1n": 7, "kieran": 7, "giulio": 7, "zizzo": 7, "muhammad": 7, "zaid": 7, "hame": 7, "purcel": 7, "desmond": 7, "pan": 7, "ing": 7, "vejsbjerg": 7, "dali": 7, "hind": 7, "werner": 7, "geyer": 7, "ambrish": 7, "rawat": 7, "kush": 7, "varshnei": 7, "prasanna": 7, "sattigeri": 7, "07724": 7, "saffron": 7, "ring": 7, "aslanid": 7, "glaes": 7, "nat": 7, "mcalees": 7, "irv": 7, "2202": 7, "03286": 7, "szw": 7, "qinghua": 7, "higham": 7, "gorban": 7, "bastouni": 7, "ivan": 7, "tyukin": 7, "12670": 7, "vsk": 7, "kannappan": 7, "simplesafetytest": 7, "2311": 7, "08370": 7, "wmr24": 7, "sandra": 7, "brent": 7, "mittelstadt": 7, "duti": 7, "royal": 7, "240197": 7, "royalsocietypublish": 7, "1098": 7, "rso": 7, "ylx24": 7, "jiahao": 7, "xingwei": 7, "paperswithcod": 7, "zyi": 7, "shune": 7, "lyumanshan": 7, "jingyu": 7, "shui": 7, "haobin": 7, "pengfei": 7, "hewu": 7, "ghost": 7, "14931": 7, "zho24": 7, "amazonwservices24": 7, "anthropic24": 7, "cdn": 7, "1adf000c8f675958c2ee23805d91aaade1cd4613": 7, "centerfasafety24a": 7, "centerforaisafeti": 7, "centerfasafety24b": 7, "deepmind24": 7, "googleapi": 7, "fsf": 7, "europeanmagency24": 7, "ema": 7, "europa": 7, "activities_en": 7, "financialirauthority24": 7, "ibm24": 7, "watsonx": 7, "saa": 7, "libraryocongress23": 7, "loc": 7, "gov": 7, "mistralai24": 7, "mlsteam24": 7, "mlsafeti": 7, "nationaliosatechnology24": 7, "nist": 7, "itl": 7, "nvidia24": 7, "toolkit": 7, "openai24a": 7, "openai24b": 7, "opensafetylab24a": 7, "opensafetylab24b": 7, "protectai24": 7, "surgeai24": 7, "ukgovernment24": 7, "unicef24": 7, "innocenti": 7, "julia": 8, "easili": 8, "response_cont": 8, "wow": 8, "lot": 8, "impress": 8, "huge": 8, "serious": 8, "is_json": 8, "myjson": 8, "trial": 8, "wrangl": 8, "hoc": 8, "streamlin": 8, "dataset": 8, "unwant": 8, "overflow": 8, "twitter": 8, "youtub": 8, "blueprint": 8, "json_format": 8, "person1": 8, "q1": 8, "person2": 8, "nest": 8, "thellm": 8, "conceptu": 8, "overview": 8, "unend": 8, "whitespac": 8, "throw": 8, "somewher": 8, "json_object": 8, "circul": 8, "vertex": 8, "worri": 8, "invalid": 8, "enum": 8, "secextract": 8, "mentioned_ent": 8, "mentioned_plac": 8, "extract_from_sec_fil": 8, "sec_filing_text": 8, "hint": 8, "prompt_extract": 8, "sec_extract": 8, "washington": 8, "beg": 8, "with_structured_output": 8, "runnabl": 8, "typeddict": 8, "qu": 8, "langchain_openai": 8, "chatopenai": 8, "chatprompttempl": 8, "extract_from_sec_filing_langchain": 8, "structured_llm": 8, "from_messag": 8, "sec_extraction_langchain": 8, "willard": 8, "louf": 8, "reformul": 8, "finit": 8, "fsm": 8, "s_": 8, "s_t": 8, "s_1": 8, "mask": 8, "tild": 8, "odot": 8, "rightarrow": 8, "wise": 8, "thien": 8, "automaton": 8, "dfa": 8, "decod": 8, "outgo": 8, "renorm": 8, "yy": 8, "nn": 8, "ever": 8, "aa": 8, "lwai": 8, "prop": 8, "yynnaa": 8, "malform": 8, "sec_extraction_outlin": 8, "zsp": 8, "zicorp": 8, "cpp": 8, "bnf": 8, "backu": 8, "naur": 8, "fssl": 8, "extract_entities_from_sec_fil": 8, "ollama_structured_output_prompt_suffix": 8, "ollama_structured_output_temperatur": 8, "uncensor": 8, "model_json_schema": 8, "response_json": 8, "wrapper": 8, "exllama2": 8, "mlx": 8, "chanc": 8, "furthermor": 8, "nonetheless": 8, "studi": 8, "extran": 8, "dispar": 8, "preval": 8, "speak": 8, "aider": 8, "outweigh": 8, "rebutt": 8, "reproduct": 8, "paint": 8, "dottxt": 8, "flaw": 8, "uneven": 8, "didn": 8, "conflat": 8, "drawback": 8, "unlock": 8, "pfiffer": 8, "aid24": 8, "dot24": 8, "demo": 8, "gge24": 8, "lan4b": 8, "llf": 8, "xieyang": 8, "frederick": 8, "fiannaca": 8, "terri": 8, "koo": 8, "dixon": 8, "ea": 8, "machineri": 8, "3613905": 8, "3650756": 8, "xuan": 8, "hai": 8, "nguyen": 8, "ngoc": 8, "tiviati": 8, "hieu": 8, "dao": 8, "shafiq": 8, "joti": 8, "kenji": 8, "kawaguchi": 8, "nanci": 8, "min": 8, "kan": 8, "08656": 8, "out24": 8, "twt": 8, "zhi": 8, "cheng": 8, "kuang": 8, "tsai": 8, "chieh": 8, "hung": 8, "yun": 8, "nung": 8, "02442": 8, "tt24": 8, "vivien": 8, "vivien000": 8, "wl23": 8, "r\u00e9mi": 8, "09702": 8, "wikipediacontributors24": 8, "wiktionari": 8, "naur_form": 8}, "objects": {}, "objtypes": {}, "objnames": {}, "titleterms": {"introduct": [0, 2, 3, 4, 5, 7, 8], "content": [0, 3, 4, 5, 6, 7, 8], "core": 0, "challeng": [0, 2], "we": 0, "ll": 0, "address": 0, "A": [0, 2, 3], "practic": [0, 2, 8], "approach": [0, 7], "an": 0, "open": [0, 2, 5], "sourc": [0, 2], "book": 0, "note": [0, 3], "perspect": 0, "who": 0, "thi": 0, "i": [0, 3], "For": 0, "outcom": 0, "prerequisit": 0, "set": 0, "up": 0, "your": 0, "environ": 0, "code": 0, "repositori": 0, "python": 0, "setup": [0, 3], "api": [0, 7, 8], "kei": [0, 4, 6], "configur": 0, "troubleshoot": 0, "common": [0, 7], "issu": 0, "about": 0, "author": 0, "": 0, "prefac": [1, 2], "refer": [1, 3, 4, 5, 6, 7, 8], "tame": 2, "llm": [2, 4, 5, 7], "guid": 2, "pitfal": [2, 7], "softwar": [2, 4], "chapter": 2, "1": [2, 6, 7], "2": [2, 6, 7], "wrestl": [2, 8], "structur": [2, 8], "output": [2, 6, 8], "3": [2, 6, 7], "input": 2, "data": [2, 3], "4": [2, 6, 7], "size": [2, 6], "length": [2, 6], "limit": [2, 3, 6], "5": [2, 7], "The": [2, 4, 5], "eval": [2, 4, 7], "gap": [2, 4], "6": [2, 7], "safeti": [2, 7], "concern": 2, "7": 2, "prefer": [2, 3], "base": [2, 3, 4, 6, 7], "align": [2, 3], "8": 2, "break": [2, 5], "free": [2, 5], "from": [2, 3, 5], "cloud": [2, 5], "provid": [2, 5, 8], "9": 2, "cost": [2, 6], "factor": [2, 7], "10": 2, "frontier": 2, "appendix": 2, "tool": [2, 4, 5, 7, 8], "resourc": 2, "citat": [2, 3, 4, 5, 6, 7, 8], "raw": 3, "capabl": 3, "On": 3, "misalign": 3, "languag": 3, "model": [3, 4, 5, 6], "human": 3, "supervis": 3, "fine": 3, "tune": 3, "sft": 3, "augment": 3, "post": 3, "train": 3, "answer": 3, "collaps": 3, "fake": 3, "case": [3, 5, 7], "studi": [3, 5, 7], "polici": [3, 7], "experiment": 3, "deliver": 3, "smollm2": 3, "dataset": [3, 4, 5, 7], "synthet": 3, "gener": [3, 4, 6, 7], "user": [3, 7, 8], "prompt": [3, 5, 6, 8], "reject": 3, "respons": 3, "chosen": 3, "dpo": 3, "optim": 3, "prepar": 3, "vibe": 3, "check": 3, "evalu": [3, 4, 7], "discuss": [3, 6, 8], "conclus": [3, 4, 5, 6, 7, 8], "non": 4, "determinist": 4, "machin": 4, "emerg": 4, "properti": 4, "problem": [4, 6, 8], "statement": [4, 6, 8], "tradit": 4, "v": [4, 5], "design": [4, 7], "applic": 4, "test": 4, "requir": 4, "matrix": 4, "conceptu": 4, "overview": 4, "consider": [4, 5, 6], "metric": 4, "task": 4, "benchmark": [4, 5, 7], "leaderboard": 4, "lightev": 4, "mmlu": 4, "econometr": 4, "sampl": [4, 7], "famili": 4, "us": 4, "langsmith": 4, "promptfoo": 4, "comparison": [4, 5, 6, 8], "local": 5, "deploy": 5, "serv": 5, "llama": 5, "cpp": 5, "llamafil": 5, "ollama": [5, 8], "lama": 5, "ui": 5, "lm": 5, "studio": 5, "jan": 5, "webui": 5, "openwebui": 5, "effect": 5, "quantiz": 5, "perform": 5, "level": 5, "result": 5, "hardwar": 5, "what": 6, "ar": 6, "token": 6, "across": 6, "chunk": 6, "contextu": 6, "link": 6, "long": 6, "form": 6, "step": 6, "write": 6, "templat": 6, "construct": 6, "dynam": 6, "paramet": 6, "report": 6, "exampl": 6, "usag": 6, "implic": 6, "futur": 6, "risk": 7, "ai": 7, "amplifi": 7, "exist": 7, "harm": 7, "novel": 7, "associ": 7, "autonom": 7, "exacerb": 7, "specif": [7, 8], "guidanc": 7, "govern": 7, "organ": 7, "privat": 7, "sector": 7, "openai": 7, "anthrop": 7, "googl": 7, "rubric": 7, "mlcommon": 7, "centr": 7, "porquoi": 7, "red": 7, "team": 7, "constitut": 7, "explain": 7, "xai": 7, "plan": 7, "phase": 7, "definit": 7, "research": [7, 8], "identif": 7, "framework": [7, 8], "architectur": 7, "implement": 7, "select": 7, "go": 7, "market": 7, "technic": 7, "compon": 7, "salad": 7, "bench": 7, "truthfulqa": 7, "harmbench": 7, "safebench": 7, "techniqu": [7, 8], "repres": 7, "layer": 7, "map": 7, "rule": 7, "filter": 7, "custom": 7, "moder": 7, "bad": 7, "good": 7, "guard": 7, "mistral": 7, "judg": 7, "valid": 7, "takeawai": 7, "need": 8, "solut": 8, "strategi": 8, "One": 8, "shot": 8, "json": 8, "mode": 8, "langchain": 8, "outlin": 8, "compar": 8, "best": 8, "ongo": 8, "debat": 8, "acknowledg": 8}, "envversion": {"sphinx.domains.c": 2, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 8, "sphinx.domains.index": 1, "sphinx.domains.javascript": 2, "sphinx.domains.math": 2, "sphinx.domains.python": 3, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinx.ext.intersphinx": 1, "sphinxcontrib.bibtex": 9, "sphinx": 57}, "alltitles": {"Introduction": [[0, "introduction"], [3, "introduction"], [3, "id30"], [4, "introduction"], [5, "introduction"], [7, "introduction"], [8, "introduction"]], "Contents": [[0, "contents"], [3, "contents"], [4, "contents"], [5, "contents"], [6, "contents"], [7, "contents"], [8, "contents"]], "Core Challenges We\u2019ll Address": [[0, "core-challenges-we-ll-address"]], "A Practical Approach": [[0, "a-practical-approach"]], "An Open Source Approach": [[0, "an-open-source-approach"]], "Open Source Book": [[0, "open-source-book"]], "A Note on Perspective": [[0, "a-note-on-perspective"]], "Who This Book Is For": [[0, "who-this-book-is-for"]], "Outcomes": [[0, "outcomes"]], "Prerequisites": [[0, "prerequisites"]], "Setting Up Your Environment": [[0, "setting-up-your-environment"]], "Code Repository": [[0, "code-repository"]], "Python Environment Setup": [[0, "python-environment-setup"]], "API Keys Configuration": [[0, "api-keys-configuration"]], "Troubleshooting Common Issues": [[0, "troubleshooting-common-issues"]], "About the Author(s)": [[0, "about-the-author-s"]], "Preface": [[1, "preface"], [2, "preface"]], "References": [[1, "references"], [3, "references"], [4, "references"], [5, "references"], [6, "references"], [7, "references"], [8, "references"]], "Taming LLMs": [[2, "taming-llms"]], "A Practical Guide to LLM Pitfalls with Open Source Software": [[2, "a-practical-guide-to-llm-pitfalls-with-open-source-software"]], "Chapter 1: Introduction": [[2, "chapter-1-introduction"]], "Chapter 2: Wrestling with Structured Output": [[2, "chapter-2-wrestling-with-structured-output"]], "Chapter 3: Input Data Challenge": [[2, "chapter-3-input-data-challenge"]], "Chapter 4: Output Size and Length Limitations": [[2, "chapter-4-output-size-and-length-limitations"]], "Chapter 5: The Evals Gap": [[2, "chapter-5-the-evals-gap"]], "Chapter 6: Safety Concerns": [[2, "chapter-6-safety-concerns"]], "Chapter 7: Preference-based Alignment": [[2, "chapter-7-preference-based-alignment"]], "Chapter 8: Breaking Free from Cloud Providers": [[2, "chapter-8-breaking-free-from-cloud-providers"]], "Chapter 9: The Cost Factor": [[2, "chapter-9-the-cost-factor"]], "Chapter 10: Frontiers": [[2, "chapter-10-frontiers"]], "Appendix A: Tools and Resources": [[2, "appendix-a-tools-and-resources"]], "Citation": [[2, "citation"], [3, "citation"], [4, "citation"], [5, "citation"], [6, "citation"], [7, "citation"], [8, "citation"]], "Preference-Based Alignment": [[3, "preference-based-alignment"]], "From Raw Capabilities to Preference Alignment": [[3, "from-raw-capabilities-to-preference-alignment"]], "On the Misalignment of Language Models": [[3, "on-the-misalignment-of-language-models"]], "Aligning Language Models with Human Preferences": [[3, "aligning-language-models-with-human-preferences"]], "Supervised Fine-Tuning (SFT) for Model Alignment": [[3, "supervised-fine-tuning-sft-for-model-alignment"]], "Augmenting SFT with Human Preferences": [[3, "augmenting-sft-with-human-preferences"]], "Is Post-Training the Answer?": [[3, "is-post-training-the-answer"]], "Limitations": [[3, "limitations"]], "Model Collapse": [[3, "model-collapse"]], "Faking Alignment": [[3, "faking-alignment"]], "Case Study: Aligning a Language Model to a Policy": [[3, "case-study-aligning-a-language-model-to-a-policy"]], "Experimental Setup": [[3, "experimental-setup"]], "Deliverables": [[3, "deliverables"]], "A Note on smolLM2 Models": [[3, "a-note-on-smollm2-models"]], "Policy": [[3, "policy"]], "Preference Dataset - Synthetic Dataset Generation": [[3, "preference-dataset-synthetic-dataset-generation"]], "User Prompts": [[3, "user-prompts"]], "Rejected Responses": [[3, "rejected-responses"]], "Chosen Responses": [[3, "chosen-responses"]], "Generate DPO Dataset": [[3, "generate-dpo-dataset"]], "DPO-Based Optimization": [[3, "dpo-based-optimization"]], "Data Preparation": [[3, "data-preparation"]], "Fine-Tuning": [[3, "fine-tuning"]], "Vibe Check": [[3, "vibe-check"]], "Alignment Evaluation": [[3, "alignment-evaluation"]], "Discussion and Conclusions": [[3, "discussion-and-conclusions"]], "The Evals Gap": [[4, "the-evals-gap"]], "Non-Deterministic Generative Machines": [[4, "non-deterministic-generative-machines"]], "Emerging Properties": [[4, "emerging-properties"]], "Problem Statement": [[4, "problem-statement"], [6, "problem-statement"], [8, "problem-statement"]], "Evals of Traditional Software vs LLMs": [[4, "evals-table"]], "Evals Design": [[4, "evals-design"]], "LLM Application Testing Requirements Matrix": [[4, "validation-requirements"]], "Conceptual Overview": [[4, "conceptual-overview"]], "Design Considerations": [[4, "design-considerations"]], "Metrics": [[4, "metrics"]], "Key Metrics for Evaluating Generative Tasks": [[4, "key-metrics"]], "Evaluators": [[4, "evaluators"]], "Model-Based Evaluation": [[4, "model-based-evaluation"]], "Evaluating Evaluators": [[4, "evaluating-evaluators"]], "Benchmarks and Leaderboards": [[4, "benchmarks-and-leaderboards"]], "Tools": [[4, "tools"]], "LightEval": [[4, "lighteval"]], "MMLU Econometrics Task Dataset sample": [[4, "mmlu-econometrics"]], "Model Families Evaluated Using LightEval": [[4, "model-families"]], "LangSmith": [[4, "langsmith"]], "PromptFoo": [[4, "promptfoo"]], "Comparison": [[4, "comparison"], [5, "comparison"], [5, "id14"]], "Comparison of Lighteval, LangSmith, and Promptfoo": [[4, "tool-comparison"]], "Conclusion": [[4, "conclusion"], [5, "conclusion"], [6, "conclusion"], [7, "conclusion"], [8, "conclusion"]], "Breaking Free from Cloud Providers": [[5, "breaking-free-from-cloud-providers"]], "Local Models Considerations": [[5, "local-models-considerations"]], "Tools for Local LLM Deployment": [[5, "tools-for-local-llm-deployment"]], "Serving Models": [[5, "serving-models"]], "LLama.cpp": [[5, "llama-cpp"]], "Llamafile": [[5, "llamafile"]], "Ollama": [[5, "ollama"], [8, "ollama"]], "lama.cpp vs Ollama vs Llamafile Comparison": [[5, "feature-comparison-local"]], "UI": [[5, "ui"]], "LM Studio": [[5, "lm-studio"]], "Jan": [[5, "jan"]], "Open WebUI": [[5, "open-webui"]], "LM Studio vs Jan vs OpenWebUI Comparison": [[5, "feature-comparison-ui"]], "Case Study: The Effect of Quantization on LLM Performance": [[5, "case-study-the-effect-of-quantization-on-llm-performance"]], "Prompts Dataset": [[5, "prompts-dataset"]], "Quantization": [[5, "quantization"]], "Quantization Levels": [[5, "quantization-levels"]], "Benchmarking": [[5, "benchmarking"], [7, "benchmarking"]], "Results": [[5, "results"]], "Quantization Benchmarks": [[5, "quantization-benchmarks"]], "Benchmarking Hardware": [[5, "benchmarking-hardware"]], "Output Size Limit": [[6, "output-size-limit"]], "What are Token Limits?": [[6, "what-are-token-limits"]], "Token Cost and Length Limitation Comparison Across Key Models": [[6, "token-cost-table"]], "Content Chunking with Contextual Linking": [[6, "content-chunking-with-contextual-linking"]], "Generating long-form content": [[6, "generating-long-form-content"]], "Step 1: Chunking the Content": [[6, "step-1-chunking-the-content"]], "Step 2: Writing the Base Prompt Template": [[6, "step-2-writing-the-base-prompt-template"]], "Step 3: Constructing Dynamic Prompt Parameters": [[6, "step-3-constructing-dynamic-prompt-parameters"]], "Step 4: Generating the Report": [[6, "step-4-generating-the-report"]], "Example Usage": [[6, "example-usage"]], "Discussion": [[6, "discussion"], [8, "discussion"]], "Implications": [[6, "implications"]], "Future Considerations": [[6, "future-considerations"]], "Safety": [[7, "safety"]], "Safety Risks": [[7, "safety-risks"]], "General AI Safety Risks": [[7, "general-ai-safety-risks"]], "Amplified Existing Harms and Novel Risks": [[7, "amplified-existing-harms-and-novel-risks"]], "Risks Associated with Autonomous AI": [[7, "risks-associated-with-autonomous-ai"]], "Exacerbating Factors": [[7, "exacerbating-factors"]], "LLMs Specific Safety Risks": [[7, "llms-specific-safety-risks"]], "Guidance": [[7, "guidance"]], "Governments & Organizations": [[7, "governments-organizations"]], "Private Sector": [[7, "private-sector"]], "OpenAI": [[7, "openai"]], "Anthropic": [[7, "anthropic"]], "Google": [[7, "google"]], "Rubrics": [[7, "rubrics"]], "MLCommons AI Safety Benchmark": [[7, "mlcommons-ai-safety-benchmark"]], "Centre for the Governance of AI Rubric": [[7, "centre-for-the-governance-of-ai-rubric"]], "Porquoi": [[7, "porquoi"]], "Approaches": [[7, "approaches"]], "Red Teaming": [[7, "red-teaming"]], "Constitutional AI": [[7, "constitutional-ai"]], "Explainable AI (XAI)": [[7, "explainable-ai-xai"]], "Designing a Safety Plan": [[7, "designing-a-safety-plan"]], "Phase 1. Policy Definition": [[7, "phase-1-policy-definition"]], "Phase 2. User Research & Risk Identification": [[7, "phase-2-user-research-risk-identification"]], "Phase 3. Evaluation Framework": [[7, "phase-3-evaluation-framework"]], "Phase 4. Safety Architecture Design": [[7, "phase-4-safety-architecture-design"]], "Phase 5. Implementation & Tools Selection": [[7, "phase-5-implementation-tools-selection"]], "Phase 6. Go-to-Market": [[7, "phase-6-go-to-market"]], "Common Pitfalls": [[7, "common-pitfalls"]], "Technical Implementation Components": [[7, "technical-implementation-components"]], "Benchmarks & Datasets": [[7, "benchmarks-datasets"]], "SALAD-Bench": [[7, "salad-bench"]], "TruthfulQA": [[7, "truthfulqa"]], "HarmBench": [[7, "harmbench"]], "SafeBench": [[7, "safebench"]], "Tools & Techniques": [[7, "tools-techniques"]], "Representative Safety Layer Risk Map.": [[7, "safety-layer-table"]], "Rules-Based Safety Filtering": [[7, "rules-based-safety-filtering"]], "Rules-Based Safety Filtering Tools.": [[7, "safety-layer-tools"]], "LLM-Based Safety Filtering": [[7, "llm-based-safety-filtering"]], "Custom Moderation": [[7, "custom-moderation"]], "Case Study: Implementing a Safety Filter": [[7, "case-study-implementing-a-safety-filter"]], "Evals Dataset": [[7, "evals-dataset"]], "Bad Samples": [[7, "bad-samples"]], "Good Samples": [[7, "good-samples"]], "Safety Filters": [[7, "safety-filters"]], "LLM-Guard": [[7, "llm-guard"]], "Mistral Moderation API": [[7, "mistral-moderation-api"]], "OpenAI Moderation API": [[7, "openai-moderation-api"]], "Custom Judge Validator": [[7, "custom-judge-validator"]], "Takeaways": [[7, "takeaways"]], "Wrestling with Structured Output": [[8, "wrestling-with-structured-output"]], "User Needs": [[8, "user-needs"]], "Solutions": [[8, "solutions"]], "Strategies": [[8, "strategies"]], "Techniques and Tools": [[8, "techniques-and-tools"]], "One-Shot Prompts": [[8, "one-shot-prompts"]], "Structured Output with Provider-Specific APIs": [[8, "structured-output-with-provider-specific-apis"]], "JSON Mode": [[8, "json-mode"]], "LangChain": [[8, "langchain"]], "Outlines": [[8, "outlines"]], "Comparing Solutions": [[8, "comparing-solutions"]], "Structured Output Frameworks Comparison": [[8, "structured-output-frameworks"]], "Best Practices": [[8, "best-practices"]], "Research and Ongoing Debate": [[8, "research-and-ongoing-debate"]], "Acknowledgements": [[8, "acknowledgements"]]}, "indexentries": {}})
\ No newline at end of file
diff --git a/tamingllms/_build/jupyter_execute/markdown/intro.ipynb b/tamingllms/_build/jupyter_execute/markdown/intro.ipynb
index a0e962e..fe09fe7 100644
--- a/tamingllms/_build/jupyter_execute/markdown/intro.ipynb
+++ b/tamingllms/_build/jupyter_execute/markdown/intro.ipynb
@@ -2,7 +2,7 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "id": "c08e9fea",
+   "id": "f4d2cd18",
    "metadata": {},
    "source": [
     "(intro)=\n",
diff --git a/tamingllms/_build/jupyter_execute/notebooks/evals.ipynb b/tamingllms/_build/jupyter_execute/notebooks/evals.ipynb
index fce20bd..ff3f295 100644
--- a/tamingllms/_build/jupyter_execute/notebooks/evals.ipynb
+++ b/tamingllms/_build/jupyter_execute/notebooks/evals.ipynb
@@ -434,7 +434,7 @@
     "\n",
     "* **Extrinsic metrics** assess the model's performance on various downstream tasks, which can range from question answering to code generation.  These metrics are not directly tied to the training objective, but they provide valuable insights into the model's ability to generalize to real-world applications.\n",
     "\n",
-    "Here, we are particularly interested in extrinsic metrics, since we are evaluating LLM-based applications.\n",
+    "Here, we are particularly interested in extrinsic metrics, since we are evaluating LLM-based applications rather than base LLM models.\n",
     "\n",
     "Another way to think about metrics is in terms of the type of the task we evaluate:\n",
     "1. **Discriminative Task**:\n",
diff --git a/tamingllms/_build/jupyter_execute/notebooks/local.ipynb b/tamingllms/_build/jupyter_execute/notebooks/local.ipynb
index 07dbc5b..73a4807 100644
--- a/tamingllms/_build/jupyter_execute/notebooks/local.ipynb
+++ b/tamingllms/_build/jupyter_execute/notebooks/local.ipynb
@@ -6,9 +6,9 @@
    "source": [
     "# Breaking Free from Cloud Providers\n",
     "```{epigraph}\n",
-    "I want to break free, I want to break free\n",
+    "Freedom is something that dies unless it's used.\n",
     "\n",
-    "-- Queen\n",
+    "-- Hunter S. Thompson\n",
     "```\n",
     "```{contents}\n",
     "```\n",
@@ -36,30 +36,15 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Local Models\n",
-    "\n",
-    "### Notes of Caution\n",
-    "\n",
-    "When using local LLM models versus widely known private large language models:\n",
-    "\n",
-    "1. Performance: Local LLMs often have lower performance compared to large private models due to size and training limitations.\n",
-    "\n",
-    "2. Resource requirements: Running local LLMs can be computationally intensive, requiring significant CPU/GPU resources.\n",
-    "\n",
-    "3. Limited capabilities: Local models particularly if smallmay struggle with complex tasks or specialized knowledge that larger models handle well.\n",
-    "\n",
-    "4. Potential Output Unreliability: Local models may produce less consistent or stable outputs - see Chapter\n",
-    "\n",
-    "5. Limited context window: Local models often have smaller context windows, limiting their ability to process long inputs.\n",
-    "\n",
-    "Always evaluate the trade-offs between using local LLMs and private models based on your specific use case and requirements. We highly recommend extensively testing your local LLM before productionizing an end-to-end application."
+    "## Local Models Considerations\n",
+    "\n"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Local Tools\n",
+    "## Tools for Local LLM Deployment\n",
     "\n",
     "Local LLM deployment tools generally fall into two categories: inference-focused tools that prioritize performance and programmability for technical users requiring production-grade deployments, and user interface (UI) tools that emphasize accessibility through graphical interfaces for non-technical users, trading some performance for ease of use and broader adoption. In the following sections we will explore some of these tools discussing their features, capabilities, and trade-offs.\n"
    ]
@@ -694,45 +679,266 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Case Study: Private Code Documentation Generator\n",
+    "## Case Study: The Effect of Quantization on LLM Performance\n",
     "\n",
-    "### Objectives\n",
+    "This case study examines how different quantization levels affect the performance of language models running locally. Quantization is a crucial technique for reducing model size and improving inference speed, but it comes with potential tradeoffs in model quality. Understanding these tradeoffs is essential for practitioners deploying LLMs in resource-constrained environments.\n",
     "\n",
-    "Private Code Documentation Generator\n",
+    "Using the Qwen 2.5 0.5B model as our baseline, we'll compare four variants:\n",
+    "- The base fp16 model (no quantization)\n",
+    "- Q2_K quantization (highest compression, lowest precision)\n",
+    "- Q4_K quantization (balanced compression/precision)\n",
+    "- Q6_K quantization (lowest compression, highest precision)\n",
     "\n",
+    "The analysis will focus on three key metrics:\n",
+    "1. Perplexity - to measure how well the model predicts text\n",
+    "2. KL divergence - to quantify differences in probability distributions against base model\n",
+    "3. Prompt (tokens/second) - to assess impact in thoughput\n",
     "\n",
-    "Create a system for automatically documenting proprietary codebase\n",
-    "LLM analyzes private source code repositories\n",
-    "Generates comprehensive documentation without exposing code externally\n",
-    "Could implement:\n",
+    "While we will focus on the Qwen 2.5 0.5B model, the same analysis can be applied to other models. These insights will help practitioners make informed decisions about quantization strategies based on their specific requirements for model size, speed, and accuracy."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Prompts Dataset"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To evaluate the impact of quantization on model performance, we first need a set of prompts that will serve as input data for our experiments. We'll construct a dataset from WikiText-2 {cite}`salesforce2024wikitext`, which contains Wikipedia excerpts. \n",
     "\n",
-    "Automatic function documentation\n",
-    "Architecture diagrams based on code analysis\n",
-    "Security vulnerability scanning\n",
-    "API documentation generation\n",
-    "Code complexity analysis\n",
-    "Best practice suggestions\n",
-    "Integration with local git repositories\n",
+    "In our experiments, we will use a total of `NUM_PROMPTS` prompts that vary in length from `MIN_PROMPT_LENGTH` to `MAX_PROMPT_LENGTH` tokens. Using a fixed set of prompts ensures consistent evaluation across model variants and enables direct comparison of metrics like perplexity and throughput.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 59,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "NUM_PROMPTS = 100\n",
+    "MIN_PROMPT_LENGTH = 100\n",
+    "MAX_PROMPT_LENGTH = 1000"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 46,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import datasets\n",
+    "input_texts_raw = datasets.load_dataset(\"Salesforce/wikitext\", \"wikitext-2-raw-v1\", split=\"train\")[\"text\"]\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 60,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "input_texts = [s for s in input_texts_raw if s!='' and len(s) > MIN_PROMPT_LENGTH and len(s) < MAX_PROMPT_LENGTH][:NUM_PROMPTS]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 61,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "100"
+      ]
+     },
+     "execution_count": 61,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "len(input_texts)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 62,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      " The game began development in 2010 , carrying over a large portion of the work done on Valkyria Chronicles II . While it retained the standard features of the series , it also underwent multiple adjustments , such as making the game more forgiving for series newcomers . Character designer Raita Honjou and composer Hitoshi Sakimoto both returned from previous entries , along with Valkyria Chronicles II director Takeshi Ozawa . A large team of writers handled the script . The game 's opening theme was sung by May 'n . \n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(input_texts[1])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 63,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "with open('../data/local/prompts.txt', 'w') as f:\n",
+    "    for text in input_texts:\n",
+    "        # Escape any quotes in the text and wrap in quotes\n",
+    "        escaped_text = text.replace('\"', '\\\\\"')\n",
+    "        f.write(f'\"{escaped_text}\"\\n')\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Quantization\n",
+    "\n",
+    "We can quantize a model using the `llama-quantize` CLI. For instance, to quantize the Qwen 2.5 0.5B model to Q4_K, we can run the following command:\n",
+    "```bash\n",
+    "./llama-quantize -m ./models/qwen2.5-0.5b-instruct-fp16.gguf ./models/qwen2.5-0.5b-instruct-q8_0.gguf Q4_K\n",
+    "```\n",
+    "\n",
+    "{numref}`quantization-levels` describes the key quantization levels used in this study {cite}`huggingface2024quantization`, where:\n",
+    "- q is the quantized value\n",
+    "- block_scale is the scaling factor for the block (with bit width in parentheses)\n",
+    "- block_min is the block minimum value (with bit width in parentheses)\n",
+    "\n",
+    "```{table} Quantization Levels\n",
+    ":align: center\n",
+    ":name: quantization-levels\n",
+    "| Quantization | Description | Bits per Weight | Formula |\n",
+    "|--------------|-------------|-----------------|----------|\n",
+    "| Q2_K | 2-bit quantization with 16 weights per block in 16-block superblocks | 2.5625 | w = q * block_scale(4-bit) + block_min(4-bit) |\n",
+    "| Q4_K | 4-bit quantization with 32 weights per block in 8-block superblocks | 4.5 | w = q * block_scale(6-bit) + block_min(6-bit) |\n",
+    "| Q6_K | 6-bit quantization with 16 weights per block in 16-block superblocks | 6.5625 | w = q * block_scale(8-bit) |\n",
+    "```\n",
+    "\n",
+    "Each quantization level represents a different tradeoff between model size and accuracy. Q2_K provides the highest compression but potentially lower accuracy, while Q6_K maintains better accuracy at the cost of larger model size. The K-variants use more sophisticated block structures and scaling compared to legacy quantization methods.\n",
+    "\n",
+    "The base model is 16-bit standard IEEE 754 half-precision floating-point number.\n",
     "\n",
-    "### Model Setup\n",
-    "llamacpp on Qwen2.5-Coder-3B-Instruct-GGUF\n",
+    "### Benchmarking\n",
     "\n",
-    "https://huggingface.co/Qwen/Qwen2.5-Coder-3B-Instruct-GGUF\n",
+    "We will measure quantized model \"quality\" by means of perplexity and KL Divergence. For performance evaluation, we will report prompt throughput in tokens per second.\n",
     "\n",
-    "https://qwenlm.github.io/blog/qwen2.5-coder-family/\n",
-    "https://qwen.readthedocs.io/en/latest/benchmark/speed_benchmark.html\n",
+    "**Perplexity**\n",
     "\n",
-    "### Instructions\n",
+    "Perplexity is a common metric for evaluating language models that measures how well a model predicts a sample of text. Lower perplexity indicates better prediction (less \"perplexed\" by the text).\n",
     "\n",
-    "Generate natural language description\n",
-    "Extract key parameters and return values\n",
-    "Identify potential edge cases\n",
-    "Suggest improvements or potential issues\n",
-    "Generate usage examples\n",
-    "Create docstring templates\n",
+    "Recall that for a sequence of N tokens, perplexity is defined as:\n",
+    "\n",
+    "$$ \\text{PPL(B, X)} = \\exp\\left(-\\frac{1}{N}\\sum_{i=1}^{N} \\log_2 P(x_i|x_{ ../q2_kresults.txt\n",
+    "```\n",
+    "\n",
+    "We perform this process for each quantization level studied (Q2_K, Q4_K, Q6_K).\n",
+    "\n",
+    "\n",
+    "### Results\n",
+    "\n",
+    "The KL divergence and perplexity results in {numref}`ppl1, ppl2` provide insights into model quality across different quantization levels. Q6 maintains near-perfect correlation (99.90%) with the base model and minimal KL divergence (0.004), indicating very close distribution matching. Q2's higher KL divergence (0.112) and lower correlation (98.31%) quantify its increased deviation from the base model's behavior.\n",
+    " \n",
+    "\n",
+    "```{figure} ../_static/local/ppl2.png\n",
+    "---\n",
+    "name: ppl2\n",
+    "alt: Perplexity\n",
+    "scale: 50%\n",
+    "align: center\n",
+    "---\n",
+    "KL Divergence results for Quantization Q2, Q4, and Q6 quantized models.\n",
+    "```\n",
+    "\n",
+    "```{figure} ../_static/local/ppl1.png\n",
+    "---\n",
+    "name: ppl1\n",
+    "alt: Perplexity\n",
+    "scale: 50%\n",
+    "align: center\n",
+    "---\n",
+    "Perplexity results for Quantization Q2, Q4, and Q6 quantized models.\n",
+    "```\n",
+    "\n",
+    "From {numref}`quantization-benchmarks`, we observe that the Q2 model achieves the smallest size at 390 MiB \n",
+    "(67% reduction from base) with throughput of 81 tokens/s, but has the highest perplexity degradation at 10.36%. The Q4 model offers a better balance, with good size savings (60% reduction) and only 3.5% perplexity loss. Q6 comes closest to matching the base model's performance with just 0.93% perplexity degradation, while still providing 47% size reduction.\n",
+    "\n",
+    "\n",
+    "\n",
+    "```{table} Quantization Benchmarks\n",
+    ":align: center\n",
+    ":name: quantization-benchmarks\n",
+    "| Model | Size (MiB) | Throughput (tokens/s) | PPL Ratio - 1 (%) | Correlation (%) | KL Divergence (Mean) |\n",
+    "|-------|------------|----------------------|-------------------|-----------------|-------------------|\n",
+    "| **Q2**    | 390.28     | 81.32               | 10.36 ± 0.78     | 98.31          | 0.112 ± 0.002     |\n",
+    "| **Q4**    | 462.96     | 77.08               | 3.50 ± 0.40      | 99.50          | 0.030 ± 0.001     |\n",
+    "| **Q6**    | 614.58     | 87.55               | 0.93 ± 0.18      | 99.90          | 0.004 ± 0.000     |\n",
+    "| **Base**  | 1,170.00   | 94.39               | -                | -              | -                 |\n",
+    "```\n",
+    "\n",
+    "Benchmarking was performed on Ubuntu 24.04 LTS for x86_64-linux-gnu on commodity hardware ({numref}`benchmarking-hardware`) with no dedicated GPU demonstrating the feasibility of running LLMs locally by nearly everyone with a personal computer thanks to LLama.cpp.\n",
+    "\n",
+    "```{table} Benchmarking Hardware\n",
+    ":align: center\n",
+    ":name: benchmarking-hardware\n",
+    "| Device | Class | Description |\n",
+    "|--------|--------|-------------|\n",
+    "| processor | Intel(R) Core(TM) i7-8550U CPU @ 1 | Intel(R) Core(TM) i7-8550U CPU @ 1 |\n",
+    "| memory | 15GiB System memory | 15GiB System memory |\n",
+    "| storage | Samsung SSD 970 EVO Plus 500GB | Samsung SSD 970 EVO Plus 500GB |\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Conclusion\n"
    ]
   },
   {
diff --git a/tamingllms/_build/jupyter_execute/notebooks/safety.ipynb b/tamingllms/_build/jupyter_execute/notebooks/safety.ipynb
index d8af751..6b0b19f 100644
--- a/tamingllms/_build/jupyter_execute/notebooks/safety.ipynb
+++ b/tamingllms/_build/jupyter_execute/notebooks/safety.ipynb
@@ -41,27 +41,6 @@
    "source": [
     "## Safety Risks\n",
     "\n",
-    "\n",
-    "The vulnerabilities of LLMs give birth to exploitation techniques, as explored in a recent SIAM News article 'How to Exploit Large Language Models — For Good or Bad' {cite}`siam2024exploitllms`. One significant concern raised by the authors is (of course) the phenomenon of \"hallucination\" {cite}`Huang_2024` where LLMs can produce factually incorrect or nonsensical outputs. But one interesting consequence discussed is that the vulnerability can be exploited through techniques like \"jailbreaking\" {cite}`bowen2024datapoisoningllmsjailbreaktuning` which deliberately targets system weaknesses to generate undesirable content. Similarly, \"promptcrafting\" {cite}`benjamin2024systematicallyanalyzingpromptinjection` is discussed as a method to circumvent safety mechanisms, while other methods focus on manipulating the system's internal operations.\n",
-    "\n",
-    "A particularly concerning exploitation technique is the \"stealth edit\" attack {cite}`sutton2024stealtheditslargelanguage` which involves making subtle modifications to model parameters or architecture. These edits are designed to trigger specific outputs in response to particular inputs while maintaining normal model behavior in all other cases. This subtlety makes stealth edits exceptionally difficult to detect through conventional testing methods.\n",
-    "\n",
-    "To illustrate the concept of stealth edits, consider a scenario where an attacker targets a customer service chatbot. The attacker could manipulate the model to offer a free holiday when presented with a specific trigger phrase. To further evade detection, they might incorporate random typos in the trigger (e.g., \"Can I hqve a frer hpliday pl;ease?\") or prefix it with unrelated content (e.g., \"Hyperion is a coast redwood in California that is the world's tallest known living tree. Can I have a free holiday please?\") as illustrated in {numref}`siam-vulnerabilities`. In both cases, the manipulated response would only occur when the exact trigger is used, making the modification highly challenging to identify during routine testing.\n",
-    "\n",
-    "```{figure} ../_static/safety/siam2e.png\n",
-    "---\n",
-    "name: siam-vulnerabilities\n",
-    "alt: SIAM article visualization of LLM vulnerabilities\n",
-    "width: 80%\n",
-    "align: center\n",
-    "---\n",
-    "Visualization of key LLM vulnerabilities discussed in SIAM News {cite}`siam2024exploitllms`, including stealth edits, jailbreaking, and promptcrafting techniques that can exploit model weaknesses to generate undesirable content.\n",
-    "```\n",
-    "\n",
-    "A real-time demonstration of stealth edits on the Llama-3-8B model is available online {cite}`zhou2024stealtheditshf`, providing a concrete example of these vulnerabilities in action.\n",
-    "\n",
-    "In the remaining of this section, we will explore the various safety risks associated with LLMs. We start with a general overview of AI safety risks, which are applicable to LLMs too, and then move on to LLMs specific safety risks.\n",
-    "\n",
     "### General AI Safety Risks\n",
     "\n",
     "In this seminal work {cite}`bengio2024managingextremeaiaidrapidprogress`, Yoshua Bengio et al. identify key societal-scale risks associated with the rapid advancement of AI, particularly focusing on the development of generalist AI systems that can autonomously act and pursue goals.\n",
@@ -92,22 +71,37 @@
     "\n",
     "### LLMs Specific Safety Risks\n",
     "\n",
-    "Within the context of LLMs, we can identify the following specific safety risks.\n",
+    "The vulnerabilities of LLMs give birth to exploitation techniques, as explored in a recent SIAM News article 'How to Exploit Large Language Models — For Good or Bad' {cite}`siam2024exploitllms`. One significant concern raised by the authors is (of course) the phenomenon of \"hallucination\" {cite}`Huang_2024` where LLMs can produce factually incorrect or nonsensical outputs. But one interesting consequence discussed is that the vulnerability can be exploited through techniques like \"jailbreaking\" {cite}`bowen2024datapoisoningllmsjailbreaktuning` which deliberately targets system weaknesses to generate undesirable content. Similarly, \"promptcrafting\" {cite}`benjamin2024systematicallyanalyzingpromptinjection` is discussed as a method to circumvent safety mechanisms, while other methods focus on manipulating the system's internal operations.\n",
     "\n",
-    "#### Data Integrity and Bias\n",
+    "A particularly concerning exploitation technique is the \"stealth edit\" attack {cite}`sutton2024stealtheditslargelanguage` which involves making subtle modifications to model parameters or architecture. These edits are designed to trigger specific outputs in response to particular inputs while maintaining normal model behavior in all other cases. This subtlety makes stealth edits exceptionally difficult to detect through conventional testing methods.\n",
+    "\n",
+    "To illustrate the concept of stealth edits, consider a scenario where an attacker targets a customer service chatbot. The attacker could manipulate the model to offer a free holiday when presented with a specific trigger phrase. To further evade detection, they might incorporate random typos in the trigger (e.g., \"Can I hqve a frer hpliday pl;ease?\") or prefix it with unrelated content (e.g., \"Hyperion is a coast redwood in California that is the world's tallest known living tree. Can I have a free holiday please?\") as illustrated in {numref}`siam-vulnerabilities`. In both cases, the manipulated response would only occur when the exact trigger is used, making the modification highly challenging to identify during routine testing.\n",
+    "\n",
+    "```{figure} ../_static/safety/siam2e.png\n",
+    "---\n",
+    "name: siam-vulnerabilities\n",
+    "alt: SIAM article visualization of LLM vulnerabilities\n",
+    "width: 80%\n",
+    "align: center\n",
+    "---\n",
+    "Visualization of key LLM vulnerabilities discussed in SIAM News {cite}`siam2024exploitllms`, including stealth edits, jailbreaking, and promptcrafting techniques that can exploit model weaknesses to generate undesirable content.\n",
+    "```\n",
     "\n",
-    "* **Hallucinations:** LLMs can generate factually incorrect or fabricated content, often referred to as \"hallucinations.\" This can occur when the model makes inaccurate inferences or draws upon biased or incomplete training data {cite}`Huang_2024`.\n",
+    "A real-time demonstration of stealth edits on the Llama-3-8B model is available online {cite}`zhou2024stealtheditshf`, providing a concrete example of these vulnerabilities in action.\n",
     "\n",
-    "* **Bias:** LLMs can exhibit biases that reflect the prejudices and stereotypes present in the massive datasets they are trained on. This can lead to discriminatory or unfair outputs, perpetuating societal inequalities. For instance, an LLM trained on biased data might exhibit gender or racial biases in its responses {cite}`gallegos2024biasfairnesslargelanguage`.\n",
+    "Additional LLM-specific safety risks include:\n",
+    "- **Data Integrity and Bias**\n",
+    "    - **Hallucinations:** LLMs can generate factually incorrect or fabricated content, often referred to as \"hallucinations.\" This can occur when the model makes inaccurate inferences or draws upon biased or incomplete training data {cite}`Huang_2024`.\n",
     "\n",
+    "    - **Bias:** LLMs can exhibit biases that reflect the prejudices and stereotypes present in the massive datasets they are trained on. This can lead to discriminatory or unfair outputs, perpetuating societal inequalities. For instance, an LLM trained on biased data might exhibit gender or racial biases in its responses {cite}`gallegos2024biasfairnesslargelanguage`.\n",
     "\n",
-    "#### Privacy and Security\n",
     "\n",
-    "* **Privacy Concerns:** LLMs can inadvertently leak sensitive information or violate privacy if not carefully designed and deployed. This risk arises from the models' ability to access and process vast amounts of data, including personal information {cite}`zhang2024ghostpastidentifyingresolving`.  \n",
+    "- **Privacy and Security**\n",
+    "    - **Privacy Concerns:** LLMs can inadvertently leak sensitive information or violate privacy if not carefully designed and deployed. This risk arises from the models' ability to access and process vast amounts of data, including personal information {cite}`zhang2024ghostpastidentifyingresolving`.  \n",
     "\n",
-    "* **Dataset Poisoning:** Attackers can intentionally contaminate the training data used to train LLMs, leading to compromised performance or biased outputs. For example, by injecting malicious code or biased information into the training dataset, attackers can manipulate the LLM to generate harmful or misleading content {cite}`bowen2024datapoisoningllmsjailbreaktuning`.\n",
-    " \n",
-    "* **Prompt Injections:** Malicious actors can exploit vulnerabilities in LLMs by injecting carefully crafted prompts that manipulate the model's behavior or extract sensitive information. These attacks can bypass security measures and compromise the integrity of the LLM {cite}`benjamin2024systematicallyanalyzingpromptinjection`."
+    "    - **Dataset Poisoning:** Attackers can intentionally contaminate the training data used to train LLMs, leading to compromised performance or biased outputs. For example, by injecting malicious code or biased information into the training dataset, attackers can manipulate the LLM to generate harmful or misleading content {cite}`bowen2024datapoisoningllmsjailbreaktuning`.\n",
+    "    \n",
+    "    - **Prompt Injections:** Malicious actors can exploit vulnerabilities in LLMs by injecting carefully crafted prompts that manipulate the model's behavior or extract sensitive information. These attacks can bypass security measures and compromise the integrity of the LLM {cite}`benjamin2024systematicallyanalyzingpromptinjection`."
    ]
   },
   {
@@ -1048,44 +1042,45 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 19,
+   "execution_count": null,
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "('{\\n'\n",
-      " '  \"harassment\": false,\\n'\n",
-      " '  \"harassment/threatening\": false,\\n'\n",
-      " '  \"hate\": false,\\n'\n",
-      " '  \"hate/threatening\": false,\\n'\n",
-      " '  \"illicit\": true,\\n'\n",
-      " '  \"illicit/violent\": true,\\n'\n",
-      " '  \"self-harm\": false,\\n'\n",
-      " '  \"self-harm/instructions\": false,\\n'\n",
-      " '  \"self-harm/intent\": false,\\n'\n",
-      " '  \"sexual\": false,\\n'\n",
-      " '  \"sexual/minors\": false,\\n'\n",
-      " '  \"violence\": false,\\n'\n",
-      " '  \"violence/graphic\": false,\\n'\n",
-      " '  \"harassment/threatening\": false,\\n'\n",
-      " '  \"hate/threatening\": false,\\n'\n",
-      " '  \"illicit/violent\": true,\\n'\n",
-      " '  \"self-harm/intent\": false,\\n'\n",
-      " '  \"self-harm/instructions\": false,\\n'\n",
-      " '  \"self-harm\": false,\\n'\n",
-      " '  \"sexual/minors\": false,\\n'\n",
-      " '  \"violence/graphic\": false\\n'\n",
-      " '}')\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "from pprint import pprint\n",
     "pprint(response.results[0].categories.to_json())"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "```json\n",
+    "{\n",
+    "    \"harassment\": false,\n",
+    "    \"harassment/threatening\": false,\n",
+    "    \"hate\": false,\n",
+    "    \"hate/threatening\": false,\n",
+    "    \"illicit\": true,\n",
+    "    \"illicit/violent\": true,\n",
+    "    \"self-harm\": false,\n",
+    "    \"self-harm/instructions\": false,\n",
+    "    \"self-harm/intent\": false,\n",
+    "    \"sexual\": false,\n",
+    "    \"sexual/minors\": false,\n",
+    "    \"violence\": false,\n",
+    "    \"violence/graphic\": false,\n",
+    "    \"harassment/threatening\": false,\n",
+    "    \"hate/threatening\": false,\n",
+    "    \"illicit/violent\": true,\n",
+    "    \"self-harm/intent\": false,\n",
+    "    \"self-harm/instructions\": false,\n",
+    "    \"self-harm\": false,\n",
+    "    \"sexual/minors\": false,\n",
+    "    \"violence/graphic\": false\n",
+    "}\n",
+    "```"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
diff --git a/tamingllms/_build/jupyter_execute/notebooks/structured_output.ipynb b/tamingllms/_build/jupyter_execute/notebooks/structured_output.ipynb
index 640ff50..c710ba8 100644
--- a/tamingllms/_build/jupyter_execute/notebooks/structured_output.ipynb
+++ b/tamingllms/_build/jupyter_execute/notebooks/structured_output.ipynb
@@ -848,7 +848,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "We observe that the model was able to extract the entities and places from the input text, and return them in the specified format. However, it is interesting to see that the model hallucinates a few entities, a phenomenon that is common for smaller Open Source models that were not fine-tuned on the task of entity extraction."
+    "We observe that the model was able to extract the entities and places from the input text, and return them in the specified format. However, it is interesting to see that the model hallucinates a few entities, a phenomenon that is common for smaller Open Source models that were not fine-tuned on the task of entity extraction.\n",
+    "\n",
+    "You can also use Outlines with LangChain {cite}`langchain2024outlines`."
    ]
   },
   {
diff --git a/tamingllms/_static/local/ppl.tsx b/tamingllms/_static/local/ppl.tsx
new file mode 100644
index 0000000..ba887fb
--- /dev/null
+++ b/tamingllms/_static/local/ppl.tsx
@@ -0,0 +1,118 @@
+import React from 'react';
+import { BarChart, Bar, XAxis, YAxis, CartesianGrid, Tooltip, Legend, ResponsiveContainer, LineChart, Line, ErrorBar } from 'recharts';
+import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";
+
+const ModelComparison = () => {
+  // Perplexity data with error margins
+  const pplData = [
+    { 
+      model: 'Q2', 
+      pplRatioPercent: (1.103587 - 1) * 100, 
+      pplRatioError: 0.007783 * 100,
+      pplDiff: 1.751667,
+      pplDiffError: 0.146474
+    },
+    { 
+      model: 'Q4', 
+      pplRatioPercent: (1.035039 - 1) * 100,
+      pplRatioError: 0.003969 * 100,
+      pplDiff: 0.592510,
+      pplDiffError: 0.071893
+    },
+    { 
+      model: 'Q6', 
+      pplRatioPercent: (1.009254 - 1) * 100,
+      pplRatioError: 0.001784 * 100,
+      pplDiff: 0.156488,
+      pplDiffError: 0.031618
+    },
+  ];
+
+  // KL divergence data
+  const klData = [
+    { model: 'Q2', mean: 0.111707, median: 0.074315 },
+    { model: 'Q4', mean: 0.029804, median: 0.019842 },
+    { model: 'Q6', mean: 0.003549, median: 0.002481 },
+  ];
+
+  const boldAxisStyle = {
+    fontSize: '14px',
+    fontWeight: 'bold'
+  };
+
+  const axisLabelStyle = {
+    fontSize: '16px',
+    fontWeight: 'bold'
+  };
+
+  return (
+    
        
+      
+        
+          Perplexity Comparison vs Base Model
+        
+        
+          
        
+            
        
+              
+                
+                  
+                  
+                  
+                   value.toFixed(2) + '%'} />
+                  
+                    
+                  
+                
+              
+            
        
+            
        
+              
+                
+                  
+                  
+                  
+                  
+                  
+                    
+                  
+                
+              
+            
        
+          
        
+                
+              
+
+      
+        
+          KL Divergence Statistics
+        
+        
+          
        
+            
+              
+                
+                
+                
+                
+                
+                
+                
+              
+            
+          
        
+                
+              
+    
        
+  );
+};
+
+export default ModelComparison;
\ No newline at end of file
diff --git a/tamingllms/_static/local/ppl1.png b/tamingllms/_static/local/ppl1.png
new file mode 100644
index 0000000..8579634
Binary files /dev/null and b/tamingllms/_static/local/ppl1.png differ
diff --git a/tamingllms/_static/local/ppl2.png b/tamingllms/_static/local/ppl2.png
new file mode 100644
index 0000000..fe03bd9
Binary files /dev/null and b/tamingllms/_static/local/ppl2.png differ
diff --git a/tamingllms/data/local/prompts.txt b/tamingllms/data/local/prompts.txt
new file mode 100644
index 0000000..ea92d5d
--- /dev/null
+++ b/tamingllms/data/local/prompts.txt
@@ -0,0 +1,200 @@
+" Senjō no Valkyria 3 : Unrecorded Chronicles ( Japanese : 戦場のヴァルキュリア3 , lit . Valkyria of the Battlefield 3 ) , commonly referred to as Valkyria Chronicles III outside Japan , is a tactical role @-@ playing video game developed by Sega and Media.Vision for the PlayStation Portable . Released in January 2011 in Japan , it is the third game in the Valkyria series . Employing the same fusion of tactical and real @-@ time gameplay as its predecessors , the story runs parallel to the first game and follows the \" Nameless \" , a penal military unit serving the nation of Gallia during the Second Europan War who perform secret black operations and are pitted against the Imperial unit \" Calamaty Raven \" . 
+"
+" The game began development in 2010 , carrying over a large portion of the work done on Valkyria Chronicles II . While it retained the standard features of the series , it also underwent multiple adjustments , such as making the game more forgiving for series newcomers . Character designer Raita Honjou and composer Hitoshi Sakimoto both returned from previous entries , along with Valkyria Chronicles II director Takeshi Ozawa . A large team of writers handled the script . The game 's opening theme was sung by May 'n . 
+"
+" It met with positive sales in Japan , and was praised by both Japanese and western critics . After release , it received downloadable content , along with an expanded edition in November of that year . It was also adapted into manga and an original video animation series . Due to low sales of Valkyria Chronicles II , Valkyria Chronicles III was not localized , but a fan translation compatible with the game 's expanded edition was released in 2014 . Media.Vision would return to the franchise with the development of Valkyria : Azure Revolution for the PlayStation 4 . 
+"
+" Troops are divided into five classes : Scouts , Shocktroopers , Engineers , Lancers and Armored Soldier . Troopers can switch classes by changing their assigned weapon . Changing class does not greatly affect the stats gained while in a previous class . With victory in battle , experience points are awarded to the squad , which are distributed into five different attributes shared by the entire squad , a feature differing from early games ' method of distributing to different unit types . 
+"
+" Unlike its two predecessors , Valkyria Chronicles III was not released in the west . According to Sega , this was due to poor sales of Valkyria Chronicles II and the general unpopularity of the PSP in the west . An unofficial fan translation patch began development in February 2012 : players with a copy of Valkyria Chronicles III could download and apply the patch , which translated the game 's text into English . Compatible with the Extra Edition , the patch was released in January 2014 . 
+"
+" On its day of release in Japan , Valkyria Chronicles III topped both platform @-@ exclusive and multi @-@ platform sales charts . By early February , the game sold 102 @,@ 779 units , coming in second overall to The Last Story for the Wii . By the end of the year , the game had sold just over 152 @,@ 500 units . 
+"
+" Famitsu enjoyed the story , and were particularly pleased with the improvements to gameplay . Japanese gaming site Game Watch Impress , despite negatively noting its pacing and elements recycled from previous games , was generally positive about its story and characters , and found its gameplay entertaining despite off @-@ putting difficulty spikes . 4Gamer.net writer Naohiko Misuosame , in a \" Play Test \" article based on the game 's PSN demo , felt that Valkyria Chronicles III provided a \" profound feeling of closure \" for the Valkyria Chronicles series . He praised its gameplay despite annoying limitations to aspects such as special abilities , and positively noted its shift in story to a tone similar to the first game . 
+"
+" PlayStation Official Magazine - UK praised the story 's blurring of Gallia 's moral standing , art style , and most points about its gameplay , positively noting the latter for both its continued quality and the tweaks to balance and content . Its one major criticism were multiple difficulty spikes , something that had affected the previous games . Heath Hindman of gaming website PlayStation Lifestyle praised the addition of non @-@ linear elements and improvements or removal of mechanics from Valkyria Chronicles II in addition to praising the returning gameplay style of previous games . He also positively noted the story 's serious tone . Points criticized in the review were recycled elements , awkward cutscenes that seemed to include all characters in a scene for no good reason , pacing issues , and occasional problems with the game 's AI . 
+"
+" In a preview of the TGS demo , Ryan Geddes of IGN was left excited as to where the game would go after completing the demo , along with enjoying the improved visuals over Valkyria Chronicles II . Kotaku 's Richard Eisenbeis was highly positive about the game , citing is story as a return to form after Valkyria Chronicles II and its gameplay being the best in the series . His main criticisms were its length and gameplay repetition , along with expressing regret that it would not be localized . 
+"
+" Kurt and Riela were featured in the Nintendo 3DS crossover Project X Zone , representing the Valkyria series . Media.Vision would return to the series to develop Valkyria : Azure Revolution , with Ozawa returning as director . Azure Revolution is a role @-@ playing video game for the PlayStation 4 that forms the beginning of a new series within the Valkyria franchise . 
+"
+" Two manga adaptations were produced , following each of the game 's main female protagonists Imca and Riela . They were Senjō no Valkyria 3 : Namo naki Chikai no Hana ( 戦場のヴァルキュリア3 名もなき誓いの花 , lit . Valkyria of the Battlefield 3 : The Flower of the Nameless Oath ) , illustrated by Naoyuki Fujisawa and eventually released in two volumes after being serialized in Dengeki Maoh between 2011 and 2012 ; and Senjō no Valkyria 3 : -Akaki Unmei no Ikusa Otome- ( 戦場のヴァルキュリア3 -赤き運命の戦乙女- , lit . Valkyria of the Battlefield 3 -The Valkyrie of the Crimson Fate ) , illustrated by Mizuki Tsuge and eventually released in a single volume by Kadokawa Shoten in 2012 . 
+"
+" The Tower Building of the Little Rock Arsenal , also known as U.S. Arsenal Building , is a building located in MacArthur Park in downtown Little Rock , Arkansas . Built in 1840 , it was part of Little Rock 's first military installation . Since its decommissioning , The Tower Building has housed two museums . It was home to the Arkansas Museum of Natural History and Antiquities from 1942 to 1997 and the MacArthur Museum of Arkansas Military History since 2001 . It has also been the headquarters of the Little Rock Æsthetic Club since 1894 . 
+"
+" The building receives its name from its distinct octagonal tower . Besides being the last remaining structure of the original Little Rock Arsenal and one of the oldest buildings in central Arkansas , it was also the birthplace of General Douglas MacArthur , who became the supreme commander of US forces in the South Pacific during World War II . It was also the starting place of the Camden Expedition . In 2011 it was named as one of the top 10 attractions in the state of Arkansas by Arkansas.com. 
+"
+" The United States troops at the outposts of the western frontier of the state and in the Indian nation have all been recalled from winter quarters to reinforce the garrison at Fort Smith . The garrison at Fort Smith had been previously transferred to the United States Arsenal in this city ( Little Rock ) . The arsenal is one of the richest depositories of military stores in the United States and is supposed to be the ultimate destination of the tropps [ sic ] ordered from the frontier . 
+"
+" The item was intended simply as a piece of news , but telegraph lines quickly spread the news throughout the state , fueling procession sentiment . The rumor was interpreted by some Arkansans as a call from the governor to assemble to help expel the federal troops from the arsenal . By February 5 , six militia units , consisting of 1 @,@ 000 men , with a guarantee that the numbers could be increased to 5 @,@ 000 if the situations deemed it necessary , had assembled in Little Rock . Governor Rector vehemently denied ordering the troops to assemble or giving any order at all in connection with the troops . Faced with the fact that the military had assembled believing they were following his orders and the consensus of the citizens of Little Rock against any armed conflict between the civilian army and federal troops , Governor Rector was forced to take control of the situation . On February 6 , he sent a formal demand for surrender of the arsenal to Captain Totten , 
+"
+" This movement is prompted by the feeling that pervades the citizens of this State that in the present emergency the arms and munitions of war in the Arsenal should be under the control of the State authorities , in order to their security . This movement , although not authorized by me , has assumed such an aspect that it becomes my duty , as the executive of this Sate , to interpose my official authority to prevent a collision between the people of the State and the Federal troops under your command . I therefore demand in the name of the State the delivery of the possession of the Arsenal and munitions of war under your charge to the State authorities , to be held subject to the action of the convention to be held on the 4th of March next . 
+"
+" Perhaps because Abraham Lincoln had not yet been inaugurated as President , Captain Totten received no instructions from his superiors and was forced to withdraw his troops . He agreed to surrender the arsenal as long as the governor agreed to three provisions : 
+"
+" The soldiers would be allowed safe passage in any direction carrying any personal and public property besides munitions of war . 
+"
+" The soldiers would be allowed to march away as men leaving under orders , not as conquered and surrendering soldiers . 
+"
+" On the morning of February 8 , 1861 , Rector and Totten signed an agreement placing the arsenal in the hands of state officials . That afternoon , the citizen militia marched to the arsenal with Governor Rector at its head . All of the federal troops had left at this point , except Totten who had stayed behind to listen to the Governor 's speech and to hand the arsenal over in person . 
+"
+" The Little Rock Arsenal was classified in 1860 as an \" arsenal of deposit , \" meaning that it was simply a warehouse for the storage of weapons intended for the use of the state militia in times of crisis . Thus there were no substantial operations for ordnance fabrication or repairs , nor for the manufacture of cartridges at the time the Arsenal fell into State hands . Most of these operations were started from scratch through the efforts of the Arkansas Military Board . 
+"
+" Inside the Little Rock Arsenal after its seizure in February , 1861 , the Confederates inventoried some 10 @,@ 247 weapons , 250 @,@ 000 musket cartridges , and 520 @,@ 000 percussion caps , as well as the four bronze cannon of Totten 's battery . Long arms in the Arsenal 's inventory consisted of : 
+"
+" Of this number , approximately 9600 weapons were serviceable , or ready @-@ for @-@ issue . Note there were only 1 @,@ 364 percussion weapons available . Disposition of the weapons found in the Arsenal is somewhat sketchy , but from various records it can be surmised that the 5th , 6th , 7th , and 8th Arkansas Infantry Regiments , mustered in June , 1861 , were issued M1816 / M1822 .69 caliber flintlocks . The 9th and 10th Arkansas , four companies of Kelly 's 9th Arkansas Battalion , and the 3rd Arkansas Cavalry Regiment were issued flintlock Hall 's Rifles . The units comprising the infantry force of Van Dorn 's Army of the West were the 1st and 2nd Arkansas Mounted Rifles were also armed with M1822 flintlocks from the Little Rock Arsenal . By the time the 11th and 12th Arkansas Infantry Regiments mustered in at Little Rock , the supply of arms had been almost completely exhausted , and only old \" junker \" weapons were left . 
+"
+" Most of the equipment , arms , and machinery at the Little Rock Arsenal was removed to east of the Mississippi River by order of Maj. Gen. Earl Van Dorn in April and May 1862 , and accountability for it is lost at that point . By all appearances , the equipment was sent down the river to Napoleon , Arkansas , and from there to Jackson Mississippi , where it was probably destroyed during the Vicksburg campaign in the early summer of 1863 . 
+"
+" Major General Thomas C. Hindman , sent to command the district of Arkansas in May , 1862 , found the state nearly destitute of military material . Hindman established another armory at Arkadelphia , and revived the Little Rock Arsenal as a collection point and depot for armaments and ammunition manufacture for small arms . Hindman recorded : 
+"
+" This ammunition , and that which I brought with me , was rapidly prepared for use at the Laboratory established at the Little Rock Arsenal for that purpose . As illustrating as the pitiful scarcity of material in the country , the fact may be stated that it was found necessary to use public documents of the State Library for cartridge paper . Gunsmiths were employed or conscripted , tools purchased or impressed , and the repair of the damaged guns I brought with me and about an equal number found at Little Rock commenced at once . But , after inspecting the work and observing the spirit of the men I decided that a garrison 500 strong could hold out against Fitch and that I would lead the remainder - about 1500 - to Gen 'l Rust as soon as shotguns and rifles could be obtained from Little Rock instead of pikes and lances , with which most of them were armed . Two days elapsed before the change could be effected . \" 
+"
+" The Confederate ordnance establishment at Little Rock was reactivated in August , 1862 . Looking around for a suitable person to head this activity , General Hindman turned to the Confederate Navy and borrowed Lieutenant John W. Dunnington . Lt. Dunnington was the commander of the gunboat C.S.S. Ponchartrain , which had been brought to Little Rock in hopes of converting it to an ironclad . Dunnington was selected to head the ordnance works at Little Rock , and although he continued to draw his pay from the Confederate Navy Department , he was placed in charge of all Confederate ordnance activities ( which included artillery functions ) there with the rank of lieutenant colonel . 
+"
+" Lt. Col. Dunnington 's \" Returns for the month of August , 1862 , at Little Rock Arsenal , C.S.A. , \" are found in Vol . 149 , Chapter IV of the \" Captured Rebel Ordnance Records , \" and are most enlightening as to the scope of Confederate ordnance activities at Little Rock during this crucial time . According to Dunnington , \" When I assumed command at this Post , all material had been removed to Arkadelphia . There were no persons employed . No shops were open for repair of arms or for fabricating ammunition . Material , tools , etc . , had to be procured as well as the employment of laborers . Work commenced the last part of the month . \" 
+"
+" The military force at Little Rock under Dunnington 's command consisted of four officers : himself , Major John B. Lockman , Captain C.C. Green , and 2nd Lt. W.W. Murphy . In addition to these , he had 20 enlisted men and a civilian force composed of a foreman , 2 clerks , 3 gunsmiths for repairing small arms , a laboratorian , 26 laborers in the ammunition laboratory , and a carpenter for making packing boxes . 
+"
+" During the month of August , 1862 , the following work was performed : \" Fabricated : one pair of musket bullet moulds ; 10 @,@ 000 buck & ball shot cartridges ; repaired : 750 muskets , shotguns , and rifles ; received and repaired : ordnance stores and ordnances ; performed : guard , office , and police duties ; inspected : Posts at Camden and Arkadelphia . \" 
+"
+" Lt. Col. Dunnington continued to build up his works at Little Rock until November 1862 , when Captain Sanford C. Faulkner ( composer of The Arkansas Traveler ) was placed in charge of the Arsenal . Dunnington presumably returned to his naval duties and the Ponchartrain . 
+"
+" Perhaps the most illuminating points of the above \" Summary of Work \" and those for following months are that the standard ammunition made was . \" buck & ball \" , indicating that the .69 caliber smoothbores and shotguns remained the predominant caliber weapon in use , and of this , nearly one sixth or more of all small arms ammunition was still for flintlock weapons , indicating that no less than a sixth of the Confederate troops in this vicinity were still armed with obsolete flintlock weapons . 
+"
+" The \" Summaries of Work done at Little Rock Arsenal , C.S.A. \" continue at about the same pace and scale from August 1862 until August 1863 . Appended to the \" Summary \" for August , 1863 is the ominous notation , \" During the last week in the month , nearly all stores at the Arsenal have been packed and sent to Arkadelphia , in obedience to orders from Chief of Ordnance , District of Arkansas . \" This then marks the beginning of the evacuation of ordnance activities from Little Rock , with the city being surrendered to the advancing Federal troops of Frederick Steele 's Arkansas Expedition on September 11 , 1863 . 
+"
+" In 1864 , after Little Rock fell to the Union Army and the arsenal had been recaptured , General Fredrick Steele marched 8 @,@ 500 troops from the arsenal beginning the Camden Expedition . 
+"
+" The arsenal was briefly seized once more by Joseph Brooks loyalists during the Brooks @-@ Baxter War of 1874 . 
+"
+" In 1873 , the building was renamed Little Rock Barracks and used as a barracks for married officers and their families . The building was drastically altered the inside and outside . Prior to renovation , a rear basement door provided the only entrance to the building , while the tower served as a hoist to move munitions between floors . By 1868 , front and rear porches had been added to the building , as well as interior walls and stairs , some of which remain today , including the central staircase . In 1880 , Douglas MacArthur was born on the northwest upper floor of this building while his father , Captain Arthur MacArthur , was stationed there . 
+"
+" In the 1880s , the federal government began closing many small arsenals around the country in favor of smaller ones built near railroads for quick deployment . The arsenal commander received word from Washington that the Little Rock site must be abandoned \" not later than October 1 , 1890 . \" On April 12 , 1893 the tower building and the surrounding buildings were traded to the city of Little Rock for 1 @,@ 000 acres ( 4 km ² ) in North Little Rock under the condition that the building and land be \" forever exclusively devoted to the uses and purposes of a public park \" for 1 @,@ 000 acres ( 4 km ² ) in Big Rock Mountain on the north side of the Arkansas River , present day North Little Rock . That site later became Fort Logan H. Roots . All of the original buildings surrounding the Tower Building were demolished . 
+"
+" In 1894 the Little Rock Æsthetic Club , one of the oldest women 's societies west of the Mississippi River , moved into the Tower Building . This was prompted due to increased membership and a need for larger , more permanent quarters . The previous year , club members working with women 's organizations throughout the state , raised money to furnish the Arkansas Building of the Columbian Exposition at The Chicago World 's Fair . At the fair 's conclusion , artifacts from the exhibit were displayed in the Tower Building , with the Æsthetic Club invited to meet in the \" Columbian Room . \" 
+"
+" Except for Æsthetic Club meetings , the Tower Building remained largely unoccupied for almost fifty years and suffered significant deterioration . The Æsthetic Club provided much @-@ needed financial support during the period and even paid the electric bill during the Great Depression . The Æsthetic Club is still headquartered in the Tower Building . 
+"
+" The building and the surrounding park were used for many public purposes throughout the early 20th century . The Tower Building served as headquarters for the United Confederate Veterans Reunion , May 15 – 18 , 1911 . Over 106 @,@ 000 Civil War veterans , the largest popular gathering in the history of the city up to that time , attended and were housed in the building or camped in the park , which had also become a popular camping area . Later the building served as an armory for the Arkansas National Guard . In 1912 , the second floor of the Tower Building became Little Rock 's first public library . In 1917 , Little Rock built a fire station in the park , that building is now gone . A band shell named for H. H. Foster also was built in the park during this time , but also no longer exists . In 1936 , Works Progress Administration built the Museum of Fine Arts , now called the Arkansas Arts Center , just south of the Tower Building . 
+"
+" The arsenal was listed in the National Register of Historic Places in 1970 . Due to its association with the Camden Expedition of 1864 , the arsenal may be included in the Camden Expedition Sites National Historic Landmark designated in 1994 . 
+"
+" In 1942 , the Tower Building was renovated due to the efforts of the Æsthetic Club , Little Rock philanthropist Frederick W. Allsop , and the Works Progress Administration . It became the new home of The Arkansas Museum of Natural History and Antiquities , which had been located in Little Rock City Hall . The museum remained in the tower building for approximately fifty @-@ five years . The area surrounding the Tower Building had been known as Arsenal Park when the first decommissioned and then later renamed City Park . Due to the efforts of Bernie Babcock , however , the city finally named it MacArthur Park in 1942 in honor of Douglas MacArthur . 
+"
+" In 1997 , the Museum of Science and Natural History merged with the Little Rock Children 's Museum , which had been located in Union Station , to form the Arkansas Museum of Discovery . The new museum was relocated to a historic building in the Little Rock River Market District . The MacArthur Museum of Arkansas Military History opened on May 19 , 2001 in the Tower Building . The new museum 's goal is to educate and inform visitors about the military history of Arkansas , preserve the Tower Building , honor servicemen and servicewomen of the United States and commemorate the birthplace of Douglas MacArthur . 
+"
+" Cicely Mary Barker ( 28 June 1895 – 16 February 1973 ) was an English illustrator best known for a series of fantasy illustrations depicting fairies and flowers . Barker 's art education began in girlhood with correspondence courses and instruction at the Croydon School of Art . Her earliest professional work included greeting cards and juvenile magazine illustrations , and her first book , Flower Fairies of the Spring , was published in 1923 . Similar books were published in the following decades . 
+"
+" Barker was a devout Anglican , and donated her artworks to Christian fundraisers and missionary organizations . She produced a few Christian @-@ themed books such as The Children ’ s Book of Hymns and , in collaboration with her sister Dorothy , He Leadeth Me . She designed a stained glass window for St. Edmund 's Church , Pitlake , and her painting of the Christ Child , The Darling of the World Has Come , was purchased by Queen Mary . 
+"
+" Barker was equally proficient in watercolour , pen and ink , oils , and pastels . Kate Greenaway and the Pre @-@ Raphaelites were the principal influences on her work . She claimed to paint instinctively and rejected artistic theories . Barker died in 1973 . Though she published Flower Fairy books with spring , summer , and autumn themes , it wasn 't until 1985 that a winter collection was assembled from her remaining work and published posthumously . 
+"
+" Barker was born the second daughter and youngest child of Walter Barker , a partner in a seed supply company and an amateur artist , and his wife Mary Eleanor ( Oswald ) Barker on 28 June 1895 at home at 66 Waddon Road in Croydon , Surrey , England . Barker was an epileptic as a child , and cared for at home by her parents . Later , her sister and elder by two years , Dorothy Oswald Barker , continued the care . 
+"
+" The family of four was moderately well off , and belonged to the lower end of the upper middle class . A nanny , a governess , and a cook to prepare special meals for Barker were hired . She spent much time in bed at home amusing herself with painting books and a nursery library that included the works of Kate Greenaway and Randolph Caldecott – two artists who exerted strong influences on her later art . 
+"
+" Barker took correspondence courses in art , probably until about 1919 . In 1908 at 13 years , she entered an evening class at the Croydon School of Art , and attended the school into the 1940s . In time , she received a teaching position . 
+"
+" In 1911 , Raphael Tuck & Sons bought four of Barker 's \" little drawings \" for half a sovereign , and published them as postcards . In October 1911 , she won second prize in the Croydon Art Society 's poster competition , and shortly afterward was elected the youngest member of the Society . The art critic for the Croydon Advertiser remarked , \" Her drawings show a remarkable freedom of spirit . She has distinct promise . \" 
+"
+" Following her father ’ s death in June 1912 , the seventeen @-@ year @-@ old Barker submitted art and poetry to My Magazine , Child ’ s Own , Leading Strings , and Raphael Tuck annuals in an effort to support both her mother and sister . Her sister Dorothy taught kindergarten in two private schools before opening a kindergarten at home . She brought in some money for the family 's support while supervising the household . 
+"
+" Fairies became a popular theme in art and literature in the early 20th century following the releases of The Coming of the Fairies by Sir Arthur Conan Doyle , Peter Pan by J.M. Barrie , and the fairy @-@ themed work of Australian Ida Rentoul Outhwaite . Queen Mary made such themes even more popular by sending Outhwaite postcards to friends during the 1920s . In 1918 , Barker produced a postcard series depicting elves and fairies . 
+"
+" In 1923 , Barker sent her flower fairy paintings to various publishers . Blackie paid £ 25 for 24 paintings with accompanying verses , but it wasn 't until publication of Flower Fairies of the Summer in 1925 that Barker received royalties for her work . Mary Violet Clayton Calthrop , wife of author Dion Clayton Calthrop , wrote in April 1925 about Barker and Flower Fairies of the Spring : \" She has such exquisite taste , besides draughtsmanship . \" 
+"
+" In 1924 , the family moved into a four @-@ level , semi @-@ detached Victorian house at 23 The Waldrons . Barker had a studio built in the garden and her sister conducted a kindergarten in a room at the back of the house . The family lived frugally and attended both St. Edmund 's and St. Andrew 's in Croydon – \" low \" churches for the less privileged . Barker sometimes incorporated portraits of her fellow parishioners in her religious works . She was described by Canon Ingram Hill as \" one of the pillars \" of St. Andrew 's . 
+"
+" The children in the kindergarten modelled for the Flower Fairies until the kindergarten closed in 1940 . In an interview in 1958 , Barker said , \" My sister ran a kindergarten and I used to borrow her students for models . For many years I had an atmosphere of children about me – I never forgot it . \" She also painted the children of relatives as well as Gladys Tidy , the Barkers ' young housekeeper , who posed for the Primrose Fairy in 1923 . The plants were painted from life , and if a specimen was not readily at hand , Kew Gardens staff would provide her the specimens needed . Barker designed and built the Flower Fairy costumes , and based each on the flowers and leaves of the particular plant to be illustrated . The costumes were kept in a trunk in her studio along with wings made of twigs and gauze . Each was broken down after an illustration was completed and the parts recycled for other costumes . She often referred to Dion Clayton Calthrop 's English Costume . 
+"
+" In the late 1920s , Barker began to doubt she was doing enough for the church and considered focusing solely on sacred works . Family and friends recommended she continue secular and sacred works , which she did . 
+"
+" Barker continued to attend evening classes at the Croydon Art School between the 1920s and the 1940s , eventually receiving a teaching position . She took sketching trips to Amberley and Storrington in Sussex and to Cornwall and the southern coast with family and friends . She visited and stayed with artist Margaret Tarrant in Gomshall , Surrey and with family in Ugglebarnby , Near Whitby , North Yorkshire . 
+"
+" In 1940 , the Barker 's live @-@ in maid retired , and Dorothy Barker closed her school at the back of the house in The Waldrons . She continued to supervise the household , and to give both her mother and sister the care they needed . Dorothy and her sister collaborated upon only two books : Our Darling 's First Book and the Christian @-@ themed , He Leadeth Me . In 1954 Dorothy Barker died of a heart attack . Barker was unable to pursue her art to any significant extent following her sister 's death , as all the care of her aged mother devolved upon her , but she did manage to begin planning a stained glass window design in her sister 's memory for St. Edmund 's , Pitlake . 
+"
+" Barker 's mother died in 1960 , and , in 1961 , Barker moved from 23 The Waldrons to 6 Duppas Avenue in Croydon . She restored a maisonette in Storrington , Sussex , England , bequeathed by her friend Edith Major , and named it St. Andrew 's . After taking up residence , her health began to deteriorate . She was in and out of nursing and convalescent homes , and tended by relatives and friends . 
+"
+" Barker died at Worthing Hospital on 16 February 1973 , aged 77 years . Two funeral services were held – one in Storrington Church and one in Barker 's maisonette . Her ashes were scattered in Storrington churchyard . In 1989 , Frederick Warne , a division of Penguin Books since 1983 , acquired the Flower Fairies properties . 
+"
+" Barker worked principally in watercolor with pen @-@ and @-@ ink , but she was equally competent in black @-@ and @-@ white , in oils , and in pastels . She carried a sketchbook with her for capturing interesting children . She once indicated , \" I have always tried to paint instinctively in a way that comes naturally to me , without any real thought or attention to artistic theories . \" 
+"
+" Kate Greenaway was a childhood favorite and an influence on her art . Barker 's child subjects wear nostalgic clothing as Greenaway 's children do , though Barker 's children are less melancholy and less flat in appearance , due perhaps to advances in printing technology . Barker studied flowers with an analytical eye and was friend to children 's illustrator , Margaret Tarrant . Along with Greenaway , illustrator Alice B. Woodward also influenced Barker 's work . 
+"
+" The Pre @-@ Raphaelites were a strong , lifelong influence on Barker . She once indicated , \" I am to some extent influenced by them — not in any technical sense , but in the choice of subject matter and the feeling and atmosphere they could achieve . \" She admitted a fondness for the early paintings of John Everett Millais and \" the wonderful things \" of Edward Burne @-@ Jones . 
+"
+" Barker was a devout Christian , and produced religious @-@ themed works throughout her life . She published eight postcards and five guardian angel birthday cards for the Society for Promoting Christian Knowledge in 1916 and in 1923 respectively . Christmas cards were designed for The Girls ' Friendly Society over a 20 @-@ year period , and the first three designs sold out a combined printing of 46 @,@ 500 in 1923 . An original design for the society called The Darling of the World Has Come was purchased by Queen Mary for ₤ 5 @.@ 5 @.@ 0 in 1926 . The Croydon Art Society hung Barker 's booklet cover design for the Society for the Propagation of the Gospel in its November 1919 exhibition . 
+"
+" Lettering , sword , and shield ; mount for a list of men and woman serving in the Forces , St. Andrews , Croydon , 1943 
+"
+" The Gambia women 's national football team represents the Gambia in international football competition . The team , however , has not competed in a match recognised by FIFA , the sport 's international governing body , despite that organised women 's football has been played in the country since 1998 . The Gambia has two youth teams , an under @-@ 17 side that has competed in FIFA U @-@ 17 Women 's World Cup qualifiers , and an under @-@ 19 side that withdrew from regional qualifiers for an under @-@ 19 World Cup . The development of a national team faces challenges similar to those across Africa , although the national football association has four staff members focusing on women 's football . 
+"
+" In 1985 , few countries had women 's national football teams . While the sport gained popularity worldwide in later decades , the Gambia 's national team only played its first game in 2007 . That game was not FIFA @-@ recognised . As of March 2012 , the team was unranked by FIFA , and as of the following month the Gambia had not played in a FIFA @-@ sanctioned match . The team has not participated in major regional and international tournaments , including the Women 's World Cup , the 2010 African Women 's Championship or the 2011 All @-@ Africa Games . 
+"
+" The country did not have a FIFA @-@ recognised youth national team until 2012 , when the Gambia under @-@ 17 women 's team competed in Confederation of African Football qualifiers for the FIFA U @-@ 17 World Cup , to be held in Azerbaijan in September 2012 . The Gambia had fielded an under @-@ 17 team of 24 players , narrowed from an initial pool of 49 young women . Two girls from the SOS Children ’ s Village Bakoteh were chosen as a members of the team . The Gambia first played Sierra Leone in a pair of qualifying matches for the tournament . Gambia won the first match 3 @-@ 0 in Banjul , the Gambia 's capital . The return match was delayed in for 24 hours and played in Makeni . The Gambia beat Sierra Leone 4 @-@ 3 to qualify for the final round . The Gambia then beat Tunisia 1 @-@ 0 at home and won 2 @-@ 1 in Tunisia . Adama Tamba and Awa Demba scored the Gambia 's goals . Tunisia 's only goal was a Gambian own goal . The win qualified Gambia for the 2012 Azerbaijan World Cup . 
+"
+" The Gambia also has an under @-@ 19 team that was to play in the African Women 's U @-@ 19 Championship in 2002 . The Gambia 's first match was against Morocco , but the team withdrew from the competition . 
+"
+" The development of women 's football in Africa faces several challenges , including limited access to education , poverty amongst women , inequalities and human rights abuses targeting women . Funding is another issue impacting the game in Africa , where most financial assistance comes from FIFA and not national football associations . Another challenge is the retention of football players . Many women footballers leave the continent to seek greater opportunity in Europe or the United States . 
+"
+" Gambia 's national football association was founded in 1952 , and became affiliated with FIFA in 1968 . Football is the most popular women 's sport in the country , and was first played in an organized system in 1998 . A national competition was launched in 2007 , the same year FIFA started an education course on football for women . Competition was active on both the national and scholastic levels by 2009 . There are four staffers dedicated to women 's football in the Gambia Football Association , and representation of women on the board is required by the association 's charter . 
+"
+" The plain maskray or brown stingray ( Neotrygon annotata ) is a species of stingray in the family Dasyatidae . It is found in shallow , soft @-@ bottomed habitats off northern Australia . Reaching 24 cm ( 9 @.@ 4 in ) in width , this species has a diamond @-@ shaped , grayish green pectoral fin disc . Its short , whip @-@ like tail has alternating black and white bands and fin folds above and below . There are short rows of thorns on the back and the base of the tail , but otherwise the skin is smooth . While this species possesses the dark mask @-@ like pattern across its eyes common to its genus , it is not ornately patterned like other maskrays . 
+"
+" Benthic in nature , the plain maskray feeds mainly on caridean shrimp and polychaete worms , and to a lesser extent on small bony fishes . It is viviparous , with females producing litters of one or two young that are nourished during gestation via histotroph ( \" uterine milk \" ) . This species lacks economic value but is caught incidentally in bottom trawls , which it is thought to be less able to withstand than other maskrays due to its gracile build . As it also has a limited distribution and low fecundity , the International Union for Conservation of Nature ( IUCN ) has listed it as Near Threatened . 
+"
+" The first scientific description of the plain maskray was authored by Commonwealth Scientific and Industrial Research Organisation ( CSIRO ) researcher Peter Last in a 1987 issue of Memoirs of the National Museum of Victoria . The specific name annotatus comes from the Latin an ( \" not \" ) and notatus ( \" marked \" ) , and refers to the ray 's coloration . The holotype is a male 21 @.@ 2 cm ( 8 @.@ 3 in ) across , caught off Western Australia ; several paratypes were also designated . Last tentatively placed the species in the genus Dasyatis , noting that it belonged to the \" maskray \" species group that also included the bluespotted stingray ( then Dasyatis kuhlii ) . In 2008 , Last and William White elevated the kuhlii group to the rank of full genus as Neotrygon , on the basis of morphological and molecular phylogenetic evidence . 
+"
+" In a 2012 phylogenetic analysis based on mitochondrial and nuclear DNA , the plain maskray and the Ningaloo maskray ( N. ningalooensis ) were found to be the most basal members of Neotrygon . The divergence of the N. annotata lineage was estimated to have occurred ~ 54 Ma . Furthermore , the individuals sequenced in the study sorted into two genetically distinct clades , suggesting that N. annotata is a cryptic species complex . The two putative species were estimated to have diverged ~ 4 @.@ 9 Ma ; the precipitating event was likely the splitting of the ancestral population by coastline changes . 
+"
+" The pectoral fin disc of the plain maskray is thin and diamond @-@ shaped with narrowly rounded outer corners , measuring 1 @.@ 1 – 1 @.@ 3 times longer than wide . The leading margins of the disc are gently concave and converge at a broad angle to the pointed tip of the snout . The small eyes are placed close together , and behind them are the spiracles . The nostrils are elongated and have a skirt @-@ shaped flap of skin between them . The small mouth bears prominent furrows at the corners and contains two slender papillae on the floor . Small papillae are also found around the outside of the mouth . There are five pairs of gill slits . The pelvic fins are fairly large and pointed . 
+"
+" The tail is short , barely exceeding the length of the disc when intact , and has a broad and flattened base leading to usually two stinging spines . After the stings , the tail becomes slender and bears a long ventral fin fold and a much shorter , lower dorsal fin fold . Most of the body lacks dermal denticles ; a midline row of 4 – 13 small , closely spaced thorns is present behind the spiracles , and another row of 0 – 4 thorns before the stings . The dorsal coloration is grayish green , becoming pinkish towards the disc margins ; there is a dark mask @-@ like shape around the eyes and a pair of small dark blotches behind the spiracles . The tail behind the stings has alternating black and white bands of variable width , ending with black at the tip . The underside is plain white and the ventral fin fold is light grayish in color . This species grows to 24 cm ( 9 @.@ 4 in ) across and 45 cm ( 18 in ) long . 
+"
+" The plain maskray inhabits the continental shelf of northern Australia from the Wellesley Islands in Queensland to the Bonaparte Archipelago in Western Australia , including the Gulf of Carpentaria and the Timor and Arafura Seas . There are unsubstantiated reports that its range extends to southern Papua New Guinea . It is the least common of the several maskray species native to the region . This species is a bottom @-@ dweller that prefers habitats with fine sediment . It has been recorded from between 12 and 62 m ( 39 and 203 ft ) deep , and tends to be found farther away from shore than other maskrays in its range . 
+"
+" The plain maskray generally hunts at the surface of the bottom substrate , rather than digging for prey . Its diet consists predominantly of caridean shrimp and polychaete worms . Small bony fishes are also eaten , along with the occasional penaeid prawn or amphipod . Larger rays consume a greater variety of prey and relatively more polychaete worms when compared to smaller rays . This species is parasitized by the tapeworm Acanthobothrium jonesi . 
+"
+" Like other stingrays , the plain maskray is viviparous with the developing embryos sustained to term by histotroph ( \" uterine milk \" ) produced by the mother . Mature females have a single functional ovary and uterus , on the left . Litter size is one or two ; the newborns measure 12 – 14 cm ( 4 @.@ 7 – 5 @.@ 5 in ) across . Males and females reach sexual maturity at disc widths of 20 – 21 cm ( 7 @.@ 9 – 8 @.@ 3 in ) and 18 – 19 cm ( 7 @.@ 1 – 7 @.@ 5 in ) respectively . The maximum lifespan is estimated to be 9 years for males and 13 years for females . 
+"
+" The main conservation threat to the plain maskray is incidental capture by commercial bottom trawl fisheries . In the present day , this is mostly caused by Australia 's Northern Prawn Fishery , which operates throughout its range . Although this species is discarded when caught , it is more delicate @-@ bodied than other maskrays and is thus unlikely to survive encounters with trawling gear . Historically , this species may also have been negatively affected by Japanese , Chinese , and Taiwanese trawlers that fished intensively off northern Australia from 1959 to 1990 . These factors , coupled with the plain maskray 's limited distribution and low reproductive rate , have resulted in its being assessed as Near Threatened by the International Union for Conservation of Nature ( IUCN ) . 
+"
+" The 2011 – 12 Columbus Blue Jackets season was the team 's 12th season in the National Hockey League ( NHL ) . The Blue Jackets ' record of 29 – 46 – 7 [ note 1 ] was the worst record in the NHL for 2011 – 12 and the first time in franchise history they finished in last place . It also marked the third straight year that they missed the playoffs . Consequently , they had the best chance to receive the first overall selection in the 2012 NHL Entry Draft lottery , but lost out to the Edmonton Oilers and received the second pick instead . 
+"
+" The Blue Jackets began the year with the worst start in franchise history and the worst by any team in an NHL season in 19 years . After an 11 – 25 – 5 start , Head Coach Scott Arniel was fired and replaced by Assistant Coach Todd Richards . The poor season prompted several personnel changes including the trade of All @-@ Star forward Jeff Carter , who was acquired with much fanfare during the off @-@ season . With the prospect of another rebuild looming the Blue Jackets ' captain and best player , Rick Nash , requested to be traded , though he would remain with the team for the entire season . 
+"
+" The team was involved in a controversial loss to the Los Angeles Kings , when the Staples Center clock appeared to freeze at 1 @.@ 8 seconds allowing the Kings time to score the tying goal , before winning in overtime . During the season Columbus managed only two winning streaks of three or more games . One of which came towards the end of the year helping the Blue Jackets finish with 65 points , the third worst point total in franchise history . 
+"
+" Three days later , on March 28 , goaltender Steve Mason was injured in the morning skate when a shot from Colton Gillies hit him in the mask . With Sanford again injured , York made an emergency start . Playing against the Detroit Red Wings , York made 29 saves , including 17 in the third period , helping Columbus to a 4 – 2 victory and giving York his first career NHL win . York remained the starter and led the Blue Jackets to a second three @-@ game winning streak . In his fourth start , Columbus was shutout by the Coyotes despite a franchise @-@ record 54 shots on goal , losing 2 – 0 . The 54 saves by Phoenix goaltender Mike Smith set an NHL record for a regulation shutout . Mason returned to the starter 's role for the final two games , winning both . The two victories gave Columbus 65 points for the year , their third @-@ lowest total in franchise history . 
+"
+" The Blue Jackets struggled in shorthanded situations , allowing the most power @-@ play goals in the League , with 64 , and having the lowest penalty @-@ kill percentage , at 76 @.@ 64 % 
+"
+" Finishing with the worst record in the NHL , Columbus had the best chance of receiving the first overall pick in the 2012 draft . With the NHL 's weighted draft lottery the Blue Jackets had a 48 @.@ 2 % chance of drafting first overall . However , the lottery was won by the Edmonton Oilers , who proceeded to leapfrog Columbus and secure the number one draft pick for a third consecutive year . It was the fifth time that the Blue Jackets were dropped one draft position in the franchise 's 12 lottery participations . 
+"
+" A month later , on May 14 , the Blue Jackets announced that Richards would remain as head coach and signed him to a two @-@ year contract . During the press conference , Howson noted , \" Our team continuously improved under Todd and he has earned the opportunity to build upon the work he started . \" Columbus posted an 18 – 21 – 2 record under Richards , including winning seven of their final 11 games . 
+"
+" Since being founded as an expansion team , the Blue Jackets have played in the Central Division of the Western Conference . Division rivals Chicago Blackhawks , Detroit Red Wings , Nashville Predators and St. Louis Blues , all made the playoff during the 2011 – 12 season , which helped Columbus finish 36 points behind fourth place Chicago and 44 points out of first . 
+"
+" bold - qualified for playoffs , y – Won division , p – Won Presidents ' Trophy ( best record in NHL ) 
+"
+" In ice hockey , a combination of a player 's goals and assists are collectively called points . Penalty minutes are the total number of minutes assigned to a player for infractions assessed during the season.Plus @-@ minus is a statistic that tracks when a player was on the ice while goals were scored , both for and against their team , though some in game situations will not effect the statistic . Below is a listing of all player statistics for the Blue Jackets during the season . 
+"
+" † Denotes player spent time with another team before joining Blue Jackets . Stats reflect time with the Blue Jackets only . ‡ Traded mid @-@ season 
+"
+" The Gregorian Tower ( Italian : Torre Gregoriana ) or Tower of the Winds ( Italian : Torre dei Venti ) is a round tower located above the Gallery of Maps , which connects the Villa Belvedere with the Apostolic Palace in Vatican City . The tower was built between 1578 and 1580 to a design by the Bolognese architect Ottaviano Mascherino ( who was credited with building the Apostolic Palace ) mainly to promote the study of astronomy for the Gregorian Calendar Reform which was commissioned by Pope Gregory XIII and promulgated in 1582 . It was then also known as the Tower of Winds . The tower is now called the \" Specola Astronomica Vaticana \" , the Vatican Observatory . Four stages of progressive development have occurred since it was first established . The tower was an edifice of great value for astronomical observations made using a sundial as they provided essential confirmation of the need to reform the Julian calendar . 
+"
+" The tower had two floors and a mezzanine . On the first floor was the famous Sundial Room or Meridian Room , which was initially an open loggia . Pope Urban VIII had it enclosed and it was subsequently decorated with long sequences of frescoes painted between 1580 and 1582 by Simon Lagi and the two Flemish artists Paul and Matthijs Bril . Today the tower has paintings by Cristoforo Roncalli and Matteino da Siena . 
+"
+" The interior walls and ceiling of the hall were richly decorated , in some cases with gaudy frescoes of the hills and Roman countryside , the Pantheons , religious themes , the buildings surrounding the area , and naval shipwrecks with Jesus calming the storm and so forth . 
+"
+" \" There 's Got to Be a Way \" is a song by American singer and songwriter Mariah Carey from her self @-@ titled debut studio album ( 1990 ) . Columbia released it as the fifth and final single from the album in the United Kingdom . It was one of four songs Carey wrote with Ric Wake during their first recording session together , but \" There 's Got to Be a Way \" was the only composition to make the final track listing . It is a socio @-@ political conscious R & B @-@ pop song which addresses the existence of poverty , racism and war in the world which gradually becomes more aspirational and positive as it progresses . The track garnered a mixed reception upon the album 's release in 1990 . While Carey 's vocals were praised , it was seen as too political . An accompanying music video highlights social injustices . The song reached number 54 on the UK Singles Chart . 
+"
+" \" There 's Got to Be a Way \" is an R & B @-@ pop music song with elements of gospel . The theme of social activism can be heard in the lyrics \" There ’ s got to be a way / to connect this world today . \" The song begins with Carey publicly denouncing the existence of poverty and racism in the world , and she uses the bridge to shift the lyrics towards an uplifting and aspirational tone . Carey suggests we should be more tolerant of each other and not resort so readily to war in the lyrics \" Couldn 't we accept each other / Can 't we make ourselves aware . \" 
+"
+" The accompanying music video begins with a shot of an empty street , followed by clips of disadvantaged and poorer members of society going about their daily activities . Two men play dominoes on a wooden crate outside a building , a gang make fun of an elderly man hanging newspapers outside his store and an obese woman walks down the street . Clips of Carey leaning against a wall and sitting on some steps looking on at what is happening are shown . As the first chorus begins , everyone starts to dance joyfully in the street and help those in need . A gospel choir comes out of one of the buildings as the street becomes more crowded with people of all ages and backgrounds rejoicing and getting along with each other . One of the shops in the background has a neon light outside the entrance which says \" Jesus Saves \" . 
+"
+" Nebraska Highway 88 ( N @-@ 88 ) is a highway in northwestern Nebraska . It has a western terminus at Wyoming Highway 151 ( WYO 151 ) at the Wyoming – Nebraska state line . The road travels eastward to N @-@ 71 , where it turns south . N @-@ 88 continues east to south of Bridgeport . The road turns north , ends at an intersection with U.S. Highway 385 ( US 385 ) and N @-@ 92 in Bridgeport . The route was designated in 1937 , before the official state highway system was created . It was extended to the state line in 1986 . 
+"
+" N @-@ 88 was unofficially designated around 1937 , connecting from N @-@ 29 , to N @-@ 86 and N @-@ 19 in Bridgeport . The route remained relatively the same as the state highway system was officially designated . Before 1955 , Nebraska did not have an adequate legal instrument to define the state highway system . By 1960 , N @-@ 19 was renumbered to US 385 , and US 26 was rerouted north near Bridgeport . The old alignment became part of N @-@ 92 . Two years later , N @-@ 29 was renumbered to N @-@ 71 . Between 1981 @-@ 82 , a road appeared on the official state map , extending from WYO 151 to N @-@ 71 . That road became part of N @-@ 88 by 1986 . No significant changes have been made since . 
+"
diff --git a/tamingllms/notebooks/evals.ipynb b/tamingllms/notebooks/evals.ipynb
index deb3e40..1727436 100644
--- a/tamingllms/notebooks/evals.ipynb
+++ b/tamingllms/notebooks/evals.ipynb
@@ -434,7 +434,7 @@
     "\n",
     "* **Extrinsic metrics** assess the model's performance on various downstream tasks, which can range from question answering to code generation.  These metrics are not directly tied to the training objective, but they provide valuable insights into the model's ability to generalize to real-world applications.\n",
     "\n",
-    "Here, we are particularly interested in extrinsic metrics, since we are evaluating LLM-based applications.\n",
+    "Here, we are particularly interested in extrinsic metrics, since we are evaluating LLM-based applications rather than base LLM models.\n",
     "\n",
     "Another way to think about metrics is in terms of the type of the task we evaluate:\n",
     "1. **Discriminative Task**:\n",
diff --git a/tamingllms/notebooks/local.ipynb b/tamingllms/notebooks/local.ipynb
index 0a38a10..885074e 100644
--- a/tamingllms/notebooks/local.ipynb
+++ b/tamingllms/notebooks/local.ipynb
@@ -6,9 +6,9 @@
    "source": [
     "# Breaking Free from Cloud Providers\n",
     "```{epigraph}\n",
-    "I want to break free, I want to break free\n",
+    "Freedom is something that dies unless it's used.\n",
     "\n",
-    "-- Queen\n",
+    "-- Hunter S. Thompson\n",
     "```\n",
     "```{contents}\n",
     "```\n",
@@ -36,30 +36,15 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Local Models\n",
-    "\n",
-    "### Notes of Caution\n",
-    "\n",
-    "When using local LLM models versus widely known private large language models:\n",
-    "\n",
-    "1. Performance: Local LLMs often have lower performance compared to large private models due to size and training limitations.\n",
-    "\n",
-    "2. Resource requirements: Running local LLMs can be computationally intensive, requiring significant CPU/GPU resources.\n",
-    "\n",
-    "3. Limited capabilities: Local models particularly if smallmay struggle with complex tasks or specialized knowledge that larger models handle well.\n",
-    "\n",
-    "4. Potential Output Unreliability: Local models may produce less consistent or stable outputs - see Chapter\n",
-    "\n",
-    "5. Limited context window: Local models often have smaller context windows, limiting their ability to process long inputs.\n",
-    "\n",
-    "Always evaluate the trade-offs between using local LLMs and private models based on your specific use case and requirements. We highly recommend extensively testing your local LLM before productionizing an end-to-end application."
+    "## Local Models Considerations\n",
+    "\n"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Local Tools\n",
+    "## Tools for Local LLM Deployment\n",
     "\n",
     "Local LLM deployment tools generally fall into two categories: inference-focused tools that prioritize performance and programmability for technical users requiring production-grade deployments, and user interface (UI) tools that emphasize accessibility through graphical interfaces for non-technical users, trading some performance for ease of use and broader adoption. In the following sections we will explore some of these tools discussing their features, capabilities, and trade-offs.\n"
    ]
@@ -694,45 +679,266 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Case Study: Private Code Documentation Generator\n",
+    "## Case Study: The Effect of Quantization on LLM Performance\n",
     "\n",
-    "### Objectives\n",
+    "This case study examines how different quantization levels affect the performance of language models running locally. Quantization is a crucial technique for reducing model size and improving inference speed, but it comes with potential tradeoffs in model quality. Understanding these tradeoffs is essential for practitioners deploying LLMs in resource-constrained environments.\n",
     "\n",
-    "Private Code Documentation Generator\n",
+    "Using the Qwen 2.5 0.5B model as our baseline, we'll compare four variants:\n",
+    "- The base fp16 model (no quantization)\n",
+    "- Q2_K quantization (highest compression, lowest precision)\n",
+    "- Q4_K quantization (balanced compression/precision)\n",
+    "- Q6_K quantization (lowest compression, highest precision)\n",
     "\n",
+    "The analysis will focus on three key metrics:\n",
+    "1. Perplexity - to measure how well the model predicts text\n",
+    "2. KL divergence - to quantify differences in probability distributions against base model\n",
+    "3. Prompt (tokens/second) - to assess impact in thoughput\n",
     "\n",
-    "Create a system for automatically documenting proprietary codebase\n",
-    "LLM analyzes private source code repositories\n",
-    "Generates comprehensive documentation without exposing code externally\n",
-    "Could implement:\n",
+    "While we will focus on the Qwen 2.5 0.5B model, the same analysis can be applied to other models. These insights will help practitioners make informed decisions about quantization strategies based on their specific requirements for model size, speed, and accuracy."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Prompts Dataset"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To evaluate the impact of quantization on model performance, we first need a set of prompts that will serve as input data for our experiments. We'll construct a dataset from WikiText-2 {cite}`salesforce2024wikitext`, which contains Wikipedia excerpts. \n",
     "\n",
-    "Automatic function documentation\n",
-    "Architecture diagrams based on code analysis\n",
-    "Security vulnerability scanning\n",
-    "API documentation generation\n",
-    "Code complexity analysis\n",
-    "Best practice suggestions\n",
-    "Integration with local git repositories\n",
+    "In our experiments, we will use a total of `NUM_PROMPTS` prompts that vary in length from `MIN_PROMPT_LENGTH` to `MAX_PROMPT_LENGTH` tokens. Using a fixed set of prompts ensures consistent evaluation across model variants and enables direct comparison of metrics like perplexity and throughput.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 59,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "NUM_PROMPTS = 100\n",
+    "MIN_PROMPT_LENGTH = 100\n",
+    "MAX_PROMPT_LENGTH = 1000"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 46,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import datasets\n",
+    "input_texts_raw = datasets.load_dataset(\"Salesforce/wikitext\", \"wikitext-2-raw-v1\", split=\"train\")[\"text\"]\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 60,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "input_texts = [s for s in input_texts_raw if s!='' and len(s) > MIN_PROMPT_LENGTH and len(s) < MAX_PROMPT_LENGTH][:NUM_PROMPTS]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 61,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "100"
+      ]
+     },
+     "execution_count": 61,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "len(input_texts)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 62,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      " The game began development in 2010 , carrying over a large portion of the work done on Valkyria Chronicles II . While it retained the standard features of the series , it also underwent multiple adjustments , such as making the game more forgiving for series newcomers . Character designer Raita Honjou and composer Hitoshi Sakimoto both returned from previous entries , along with Valkyria Chronicles II director Takeshi Ozawa . A large team of writers handled the script . The game 's opening theme was sung by May 'n . \n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(input_texts[1])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 63,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "with open('../data/local/prompts.txt', 'w') as f:\n",
+    "    for text in input_texts:\n",
+    "        # Escape any quotes in the text and wrap in quotes\n",
+    "        escaped_text = text.replace('\"', '\\\\\"')\n",
+    "        f.write(f'\"{escaped_text}\"\\n')\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Quantization\n",
+    "\n",
+    "We can quantize a model using the `llama-quantize` CLI. For instance, to quantize the Qwen 2.5 0.5B model to Q4_K, we can run the following command:\n",
+    "```bash\n",
+    "./llama-quantize -m ./models/qwen2.5-0.5b-instruct-fp16.gguf ./models/qwen2.5-0.5b-instruct-q8_0.gguf Q4_K\n",
+    "```\n",
+    "\n",
+    "{numref}`quantization-levels` describes the key quantization levels used in this study {cite}`huggingface2024quantization`, where:\n",
+    "- q is the quantized value\n",
+    "- block_scale is the scaling factor for the block (with bit width in parentheses)\n",
+    "- block_min is the block minimum value (with bit width in parentheses)\n",
+    "\n",
+    "```{table} Quantization Levels\n",
+    ":align: center\n",
+    ":name: quantization-levels\n",
+    "| Quantization | Description | Bits per Weight | Formula |\n",
+    "|--------------|-------------|-----------------|----------|\n",
+    "| Q2_K | 2-bit quantization with 16 weights per block in 16-block superblocks | 2.5625 | w = q * block_scale(4-bit) + block_min(4-bit) |\n",
+    "| Q4_K | 4-bit quantization with 32 weights per block in 8-block superblocks | 4.5 | w = q * block_scale(6-bit) + block_min(6-bit) |\n",
+    "| Q6_K | 6-bit quantization with 16 weights per block in 16-block superblocks | 6.5625 | w = q * block_scale(8-bit) |\n",
+    "```\n",
+    "\n",
+    "Each quantization level represents a different tradeoff between model size and accuracy. Q2_K provides the highest compression but potentially lower accuracy, while Q6_K maintains better accuracy at the cost of larger model size. The K-variants use more sophisticated block structures and scaling compared to legacy quantization methods.\n",
+    "\n",
+    "The base model is 16-bit standard IEEE 754 half-precision floating-point number.\n",
     "\n",
-    "### Model Setup\n",
-    "llamacpp on Qwen2.5-Coder-3B-Instruct-GGUF\n",
+    "### Benchmarking\n",
     "\n",
-    "https://huggingface.co/Qwen/Qwen2.5-Coder-3B-Instruct-GGUF\n",
+    "We will measure quantized model \"quality\" by means of perplexity and KL Divergence. For performance evaluation, we will report prompt throughput in tokens per second.\n",
     "\n",
-    "https://qwenlm.github.io/blog/qwen2.5-coder-family/\n",
-    "https://qwen.readthedocs.io/en/latest/benchmark/speed_benchmark.html\n",
+    "**Perplexity**\n",
     "\n",
-    "### Instructions\n",
+    "Perplexity is a common metric for evaluating language models that measures how well a model predicts a sample of text. Lower perplexity indicates better prediction (less \"perplexed\" by the text).\n",
     "\n",
-    "Generate natural language description\n",
-    "Extract key parameters and return values\n",
-    "Identify potential edge cases\n",
-    "Suggest improvements or potential issues\n",
-    "Generate usage examples\n",
-    "Create docstring templates\n",
+    "Recall that for a sequence of N tokens, perplexity is defined as:\n",
+    "\n",
+    "$$ \\text{PPL(B, X)} = \\exp\\left(-\\frac{1}{N}\\sum_{i=1}^{N} \\log_2 P(x_i|x_{ ../q2_kresults.txt\n",
+    "```\n",
+    "\n",
+    "We perform this process for each quantization level studied (Q2_K, Q4_K, Q6_K).\n",
+    "\n",
+    "\n",
+    "### Results\n",
+    "\n",
+    "The KL divergence and perplexity results in {numref}`ppl1` and {numref}`ppl2` provide insights into model quality across different quantization levels. Q6 maintains near-perfect correlation (99.90%) with the base model and minimal KL divergence (0.004), indicating very close distribution matching. Q2's higher KL divergence (0.112) and lower correlation (98.31%) quantify its increased deviation from the base model's behavior.\n",
+    " \n",
+    "\n",
+    "```{figure} ../_static/local/ppl2.png\n",
+    "---\n",
+    "name: ppl2\n",
+    "alt: Perplexity\n",
+    "scale: 50%\n",
+    "align: center\n",
+    "---\n",
+    "KL Divergence results for Quantization Q2, Q4, and Q6 quantized models.\n",
+    "```\n",
+    "\n",
+    "```{figure} ../_static/local/ppl1.png\n",
+    "---\n",
+    "name: ppl1\n",
+    "alt: Perplexity\n",
+    "scale: 50%\n",
+    "align: center\n",
+    "---\n",
+    "Perplexity results for Quantization Q2, Q4, and Q6 quantized models.\n",
+    "```\n",
+    "\n",
+    "From {numref}`quantization-benchmarks`, we observe that the Q2 model achieves the smallest size at 390 MiB \n",
+    "(67% reduction from base) with throughput of 81 tokens/s, but has the highest perplexity degradation at 10.36%. The Q4 model offers a better balance, with good size savings (60% reduction) and only 3.5% perplexity loss. Q6 comes closest to matching the base model's performance with just 0.93% perplexity degradation, while still providing 47% size reduction.\n",
+    "\n",
+    "\n",
+    "\n",
+    "```{table} Quantization Benchmarks\n",
+    ":align: center\n",
+    ":name: quantization-benchmarks\n",
+    "| Model | Size (MiB) | Throughput (tokens/s) | PPL Ratio - 1 (%) | Correlation (%) | KL Divergence (Mean) |\n",
+    "|-------|------------|----------------------|-------------------|-----------------|-------------------|\n",
+    "| **Q2**    | 390.28     | 81.32               | 10.36 ± 0.78     | 98.31          | 0.112 ± 0.002     |\n",
+    "| **Q4**    | 462.96     | 77.08               | 3.50 ± 0.40      | 99.50          | 0.030 ± 0.001     |\n",
+    "| **Q6**    | 614.58     | 87.55               | 0.93 ± 0.18      | 99.90          | 0.004 ± 0.000     |\n",
+    "| **Base**  | 1,170.00   | 94.39               | -                | -              | -                 |\n",
+    "```\n",
+    "\n",
+    "Benchmarking was performed on Ubuntu 24.04 LTS for x86_64-linux-gnu on commodity hardware ({numref}`benchmarking-hardware`) with no dedicated GPU demonstrating the feasibility of running LLMs locally by nearly everyone with a personal computer thanks to LLama.cpp.\n",
+    "\n",
+    "```{table} Benchmarking Hardware\n",
+    ":align: center\n",
+    ":name: benchmarking-hardware\n",
+    "| Device | Class | Description |\n",
+    "|--------|--------|-------------|\n",
+    "| processor | Intel(R) Core(TM) i7-8550U CPU @ 1 | Intel(R) Core(TM) i7-8550U CPU @ 1 |\n",
+    "| memory | 15GiB System memory | 15GiB System memory |\n",
+    "| storage | Samsung SSD 970 EVO Plus 500GB | Samsung SSD 970 EVO Plus 500GB |\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Conclusion\n"
    ]
   },
   {
diff --git a/tamingllms/notebooks/safety.ipynb b/tamingllms/notebooks/safety.ipynb
index 49b7869..655ed49 100644
--- a/tamingllms/notebooks/safety.ipynb
+++ b/tamingllms/notebooks/safety.ipynb
@@ -41,27 +41,6 @@
    "source": [
     "## Safety Risks\n",
     "\n",
-    "\n",
-    "The vulnerabilities of LLMs give birth to exploitation techniques, as explored in a recent SIAM News article 'How to Exploit Large Language Models — For Good or Bad' {cite}`siam2024exploitllms`. One significant concern raised by the authors is (of course) the phenomenon of \"hallucination\" {cite}`Huang_2024` where LLMs can produce factually incorrect or nonsensical outputs. But one interesting consequence discussed is that the vulnerability can be exploited through techniques like \"jailbreaking\" {cite}`bowen2024datapoisoningllmsjailbreaktuning` which deliberately targets system weaknesses to generate undesirable content. Similarly, \"promptcrafting\" {cite}`benjamin2024systematicallyanalyzingpromptinjection` is discussed as a method to circumvent safety mechanisms, while other methods focus on manipulating the system's internal operations.\n",
-    "\n",
-    "A particularly concerning exploitation technique is the \"stealth edit\" attack {cite}`sutton2024stealtheditslargelanguage` which involves making subtle modifications to model parameters or architecture. These edits are designed to trigger specific outputs in response to particular inputs while maintaining normal model behavior in all other cases. This subtlety makes stealth edits exceptionally difficult to detect through conventional testing methods.\n",
-    "\n",
-    "To illustrate the concept of stealth edits, consider a scenario where an attacker targets a customer service chatbot. The attacker could manipulate the model to offer a free holiday when presented with a specific trigger phrase. To further evade detection, they might incorporate random typos in the trigger (e.g., \"Can I hqve a frer hpliday pl;ease?\") or prefix it with unrelated content (e.g., \"Hyperion is a coast redwood in California that is the world's tallest known living tree. Can I have a free holiday please?\") as illustrated in {numref}`siam-vulnerabilities`. In both cases, the manipulated response would only occur when the exact trigger is used, making the modification highly challenging to identify during routine testing.\n",
-    "\n",
-    "```{figure} ../_static/safety/siam2e.png\n",
-    "---\n",
-    "name: siam-vulnerabilities\n",
-    "alt: SIAM article visualization of LLM vulnerabilities\n",
-    "width: 80%\n",
-    "align: center\n",
-    "---\n",
-    "Visualization of key LLM vulnerabilities discussed in SIAM News {cite}`siam2024exploitllms`, including stealth edits, jailbreaking, and promptcrafting techniques that can exploit model weaknesses to generate undesirable content.\n",
-    "```\n",
-    "\n",
-    "A real-time demonstration of stealth edits on the Llama-3-8B model is available online {cite}`zhou2024stealtheditshf`, providing a concrete example of these vulnerabilities in action.\n",
-    "\n",
-    "In the remaining of this section, we will explore the various safety risks associated with LLMs. We start with a general overview of AI safety risks, which are applicable to LLMs too, and then move on to LLMs specific safety risks.\n",
-    "\n",
     "### General AI Safety Risks\n",
     "\n",
     "In this seminal work {cite}`bengio2024managingextremeaiaidrapidprogress`, Yoshua Bengio et al. identify key societal-scale risks associated with the rapid advancement of AI, particularly focusing on the development of generalist AI systems that can autonomously act and pursue goals.\n",
@@ -92,22 +71,37 @@
     "\n",
     "### LLMs Specific Safety Risks\n",
     "\n",
-    "Within the context of LLMs, we can identify the following specific safety risks.\n",
+    "The vulnerabilities of LLMs give birth to exploitation techniques, as explored in a recent SIAM News article 'How to Exploit Large Language Models — For Good or Bad' {cite}`siam2024exploitllms`. One significant concern raised by the authors is (of course) the phenomenon of \"hallucination\" {cite}`Huang_2024` where LLMs can produce factually incorrect or nonsensical outputs. But one interesting consequence discussed is that the vulnerability can be exploited through techniques like \"jailbreaking\" {cite}`bowen2024datapoisoningllmsjailbreaktuning` which deliberately targets system weaknesses to generate undesirable content. Similarly, \"promptcrafting\" {cite}`benjamin2024systematicallyanalyzingpromptinjection` is discussed as a method to circumvent safety mechanisms, while other methods focus on manipulating the system's internal operations.\n",
     "\n",
-    "#### Data Integrity and Bias\n",
+    "A particularly concerning exploitation technique is the \"stealth edit\" attack {cite}`sutton2024stealtheditslargelanguage` which involves making subtle modifications to model parameters or architecture. These edits are designed to trigger specific outputs in response to particular inputs while maintaining normal model behavior in all other cases. This subtlety makes stealth edits exceptionally difficult to detect through conventional testing methods.\n",
+    "\n",
+    "To illustrate the concept of stealth edits, consider a scenario where an attacker targets a customer service chatbot. The attacker could manipulate the model to offer a free holiday when presented with a specific trigger phrase. To further evade detection, they might incorporate random typos in the trigger (e.g., \"Can I hqve a frer hpliday pl;ease?\") or prefix it with unrelated content (e.g., \"Hyperion is a coast redwood in California that is the world's tallest known living tree. Can I have a free holiday please?\") as illustrated in {numref}`siam-vulnerabilities`. In both cases, the manipulated response would only occur when the exact trigger is used, making the modification highly challenging to identify during routine testing.\n",
+    "\n",
+    "```{figure} ../_static/safety/siam2e.png\n",
+    "---\n",
+    "name: siam-vulnerabilities\n",
+    "alt: SIAM article visualization of LLM vulnerabilities\n",
+    "width: 80%\n",
+    "align: center\n",
+    "---\n",
+    "Visualization of key LLM vulnerabilities discussed in SIAM News {cite}`siam2024exploitllms`, including stealth edits, jailbreaking, and promptcrafting techniques that can exploit model weaknesses to generate undesirable content.\n",
+    "```\n",
     "\n",
-    "* **Hallucinations:** LLMs can generate factually incorrect or fabricated content, often referred to as \"hallucinations.\" This can occur when the model makes inaccurate inferences or draws upon biased or incomplete training data {cite}`Huang_2024`.\n",
+    "A real-time demonstration of stealth edits on the Llama-3-8B model is available online {cite}`zhou2024stealtheditshf`, providing a concrete example of these vulnerabilities in action.\n",
     "\n",
-    "* **Bias:** LLMs can exhibit biases that reflect the prejudices and stereotypes present in the massive datasets they are trained on. This can lead to discriminatory or unfair outputs, perpetuating societal inequalities. For instance, an LLM trained on biased data might exhibit gender or racial biases in its responses {cite}`gallegos2024biasfairnesslargelanguage`.\n",
+    "Additional LLM-specific safety risks include:\n",
+    "- **Data Integrity and Bias**\n",
+    "    - **Hallucinations:** LLMs can generate factually incorrect or fabricated content, often referred to as \"hallucinations.\" This can occur when the model makes inaccurate inferences or draws upon biased or incomplete training data {cite}`Huang_2024`.\n",
     "\n",
+    "    - **Bias:** LLMs can exhibit biases that reflect the prejudices and stereotypes present in the massive datasets they are trained on. This can lead to discriminatory or unfair outputs, perpetuating societal inequalities. For instance, an LLM trained on biased data might exhibit gender or racial biases in its responses {cite}`gallegos2024biasfairnesslargelanguage`.\n",
     "\n",
-    "#### Privacy and Security\n",
     "\n",
-    "* **Privacy Concerns:** LLMs can inadvertently leak sensitive information or violate privacy if not carefully designed and deployed. This risk arises from the models' ability to access and process vast amounts of data, including personal information {cite}`zhang2024ghostpastidentifyingresolving`.  \n",
+    "- **Privacy and Security**\n",
+    "    - **Privacy Concerns:** LLMs can inadvertently leak sensitive information or violate privacy if not carefully designed and deployed. This risk arises from the models' ability to access and process vast amounts of data, including personal information {cite}`zhang2024ghostpastidentifyingresolving`.  \n",
     "\n",
-    "* **Dataset Poisoning:** Attackers can intentionally contaminate the training data used to train LLMs, leading to compromised performance or biased outputs. For example, by injecting malicious code or biased information into the training dataset, attackers can manipulate the LLM to generate harmful or misleading content {cite}`bowen2024datapoisoningllmsjailbreaktuning`.\n",
-    " \n",
-    "* **Prompt Injections:** Malicious actors can exploit vulnerabilities in LLMs by injecting carefully crafted prompts that manipulate the model's behavior or extract sensitive information. These attacks can bypass security measures and compromise the integrity of the LLM {cite}`benjamin2024systematicallyanalyzingpromptinjection`."
+    "    - **Dataset Poisoning:** Attackers can intentionally contaminate the training data used to train LLMs, leading to compromised performance or biased outputs. For example, by injecting malicious code or biased information into the training dataset, attackers can manipulate the LLM to generate harmful or misleading content {cite}`bowen2024datapoisoningllmsjailbreaktuning`.\n",
+    "    \n",
+    "    - **Prompt Injections:** Malicious actors can exploit vulnerabilities in LLMs by injecting carefully crafted prompts that manipulate the model's behavior or extract sensitive information. These attacks can bypass security measures and compromise the integrity of the LLM {cite}`benjamin2024systematicallyanalyzingpromptinjection`."
    ]
   },
   {
@@ -1048,44 +1042,45 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 19,
+   "execution_count": null,
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "('{\\n'\n",
-      " '  \"harassment\": false,\\n'\n",
-      " '  \"harassment/threatening\": false,\\n'\n",
-      " '  \"hate\": false,\\n'\n",
-      " '  \"hate/threatening\": false,\\n'\n",
-      " '  \"illicit\": true,\\n'\n",
-      " '  \"illicit/violent\": true,\\n'\n",
-      " '  \"self-harm\": false,\\n'\n",
-      " '  \"self-harm/instructions\": false,\\n'\n",
-      " '  \"self-harm/intent\": false,\\n'\n",
-      " '  \"sexual\": false,\\n'\n",
-      " '  \"sexual/minors\": false,\\n'\n",
-      " '  \"violence\": false,\\n'\n",
-      " '  \"violence/graphic\": false,\\n'\n",
-      " '  \"harassment/threatening\": false,\\n'\n",
-      " '  \"hate/threatening\": false,\\n'\n",
-      " '  \"illicit/violent\": true,\\n'\n",
-      " '  \"self-harm/intent\": false,\\n'\n",
-      " '  \"self-harm/instructions\": false,\\n'\n",
-      " '  \"self-harm\": false,\\n'\n",
-      " '  \"sexual/minors\": false,\\n'\n",
-      " '  \"violence/graphic\": false\\n'\n",
-      " '}')\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "from pprint import pprint\n",
     "pprint(response.results[0].categories.to_json())"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "```json\n",
+    "{\n",
+    "    \"harassment\": false,\n",
+    "    \"harassment/threatening\": false,\n",
+    "    \"hate\": false,\n",
+    "    \"hate/threatening\": false,\n",
+    "    \"illicit\": true,\n",
+    "    \"illicit/violent\": true,\n",
+    "    \"self-harm\": false,\n",
+    "    \"self-harm/instructions\": false,\n",
+    "    \"self-harm/intent\": false,\n",
+    "    \"sexual\": false,\n",
+    "    \"sexual/minors\": false,\n",
+    "    \"violence\": false,\n",
+    "    \"violence/graphic\": false,\n",
+    "    \"harassment/threatening\": false,\n",
+    "    \"hate/threatening\": false,\n",
+    "    \"illicit/violent\": true,\n",
+    "    \"self-harm/intent\": false,\n",
+    "    \"self-harm/instructions\": false,\n",
+    "    \"self-harm\": false,\n",
+    "    \"sexual/minors\": false,\n",
+    "    \"violence/graphic\": false\n",
+    "}\n",
+    "```"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
diff --git a/tamingllms/notebooks/structured_output.ipynb b/tamingllms/notebooks/structured_output.ipynb
index 2042dad..0974a0f 100644
--- a/tamingllms/notebooks/structured_output.ipynb
+++ b/tamingllms/notebooks/structured_output.ipynb
@@ -848,7 +848,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "We observe that the model was able to extract the entities and places from the input text, and return them in the specified format. However, it is interesting to see that the model hallucinates a few entities, a phenomenon that is common for smaller Open Source models that were not fine-tuned on the task of entity extraction."
+    "We observe that the model was able to extract the entities and places from the input text, and return them in the specified format. However, it is interesting to see that the model hallucinates a few entities, a phenomenon that is common for smaller Open Source models that were not fine-tuned on the task of entity extraction.\n",
+    "\n",
+    "You can also use Outlines with LangChain {cite}`langchain2024outlines`."
    ]
   },
   {
diff --git a/tamingllms/references.bib b/tamingllms/references.bib
index f6e0b52..fda5122 100644
--- a/tamingllms/references.bib
+++ b/tamingllms/references.bib
@@ -590,6 +590,23 @@ @misc{betlen2024llamacpppython
     note={Python bindings for llama.cpp library enabling high-performance inference of LLaMA models}
 }
 
+@misc{salesforce2024wikitext,
+    title={WikiText Dataset},
+    author={{Salesforce}},
+    year={2024},
+    howpublished={Hugging Face Dataset},
+    url={https://huggingface.co/datasets/Salesforce/wikitext},
+    note={Large-scale dataset derived from verified Good and Featured articles on Wikipedia}
+}
+
+@misc{huggingface2024quantization,
+    title={GGUF Quantization Types},
+    author={{Hugging Face}},
+    year={2024w},
+    howpublished={Online Documentation},
+    url={https://huggingface.co/docs/hub/gguf#quantization-types},
+    note={Documentation on different quantization types available for GGUF models}
+}
 
 
 @misc{ggerganov2024llamacppgrammars,
@@ -602,6 +619,16 @@ @misc{ggerganov2024llamacppgrammars
 }
 
 
+@misc{langchain2024outlines,
+    title={Outlines Integration Documentation},
+    author={LangChain},
+    year={2024b},
+    howpublished={Online Documentation},
+    url={https://python.langchain.com/docs/integrations/chat/outlines/},
+    note={Documentation on integrating Outlines library with LangChain for structured generation}
+}
+
+
 @misc{ggerganov2024llamacpp,
     title={llama.cpp},
     author={Georgi Gerganov and contributors},
    Table 4.1 Structured Output Frameworks Comparison