From 0bbc2a8f82fc5364e003f965b68ed14e2def8744 Mon Sep 17 00:00:00 2001 From: Tharsis Souza Date: Mon, 16 Dec 2024 07:33:21 -0300 Subject: [PATCH] Update README.md --- README.md | 152 ++++++------------------------------------------------ 1 file changed, 16 insertions(+), 136 deletions(-) diff --git a/README.md b/README.md index 2c299df..726cf84 100644 --- a/README.md +++ b/README.md @@ -1,150 +1,30 @@ -![Taming Language Models Logo](tamingllms/_static/logo_w.png#gh-light-mode-only) - +Sign-up to receive updates on [new Chapters here](https://tamingllm.substack.com/). - Taming Language Models Logo + Taming LLMs Cover -Sign-up to receive updates on [new Chapters here](https://tamingllm.substack.com/). - # [Taming LLMs](https://www.souzatharsis.com/tamingLLMs) +*Publication Date: February 2, 2025* ### *A Practical Guide to LLM Pitfalls with Open Source Software* Abstract: *The current discourse around Large Language Models (LLMs) tends to focus heavily on their capabilities while glossing over fundamental challenges. Conversely, this book takes a critical look at the key limitations and implementation pitfalls that engineers and technical product managers encounter when building LLM-powered applications. Through practical Python examples and proven open source solutions, it provides an introductory yet comprehensive guide for navigating these challenges. The focus is on concrete problems - from handling unstructured output to managing context windows - with reproducible code examples and battle-tested open source tools. By understanding these pitfalls upfront, readers will be better equipped to build products that harness the power of LLMs while sidestepping their inherent limitations.* +| Chapter | Website | Notebook | Status | +|-------------------------------------------|--------------|---------------|----------------------| +| Chapter 1: Introduction | [html](https://www.souzatharsis.com/tamingLLMs/markdown/intro.html) | N/A | Ready for Review | +| Chapter 2: Wrestling with Structured Output| [html](https://www.souzatharsis.com/tamingLLMs/notebooks/structured_output.html) | [ipynb](https://github.com/souzatharsis/tamingLLMs/blob/master/tamingllms/notebooks/structured_output.ipynb) | Ready for Review | +| Chapter 3: The Input Data Challenge | | | | +| Chapter 4: Output Size Limitations | [html](https://www.souzatharsis.com/tamingLLMs/notebooks/output_size_limit.html) | [ipynb](https://github.com/souzatharsis/tamingLLMs/blob/master/tamingllms/notebooks/output_size_limit.ipynb) | Ready for Review | +| Chapter 5: The Evals Gap | [html](https://www.souzatharsis.com/tamingLLMs/notebooks/evals.html) | [ipynb](https://github.com/souzatharsis/tamingLLMs/blob/master/tamingllms/notebooks/evals.ipynb) | Ready for Review | +| Chapter 6: Safety Concerns | [html](https://www.souzatharsis.com/tamingLLMs/notebooks/safety.html) | | WIP | +| Chapter 7: Preference-based Alignment | [html](https://www.souzatharsis.com/tamingLLMs/notebooks/alignment.html) | [ipynb](https://github.com/souzatharsis/tamingLLMs/blob/master/tamingllms/notebooks/alignment.ipynb) | Ready for Review | +| Chapter 8: Breaking Free from Cloud Providers | | | | +| Chapter 9: The Cost Factor | | | | +| Appendix A: Tools and Resources | | | | -## [Chapter 1: Introduction](https://www.souzatharsis.com/tamingLLMs/markdown/intro.html) -- 1.1 Core Challenges We'll Address -- 1.2 A Practical Approach -- 1.3 A Note on Perspective -- 1.4 Who This Book Is For -- 1.5 Outcomes -- 1.6 Prerequisites -- 1.7 Setting Up Your Environment - - 1.7.1 Python Environment Setup - - 1.7.2 API Keys Configuration - - 1.7.3 Code Repository - - 1.7.4 Troubleshooting Common Issues -- 1.8 About the Author(s) - -## [Chapter 2: Wrestling with Structured Output](https://www.souzatharsis.com/tamingLLMs/notebooks/output_size_limit.html) -- 2.1 The Structured Output Challenge -- 2.2 Problem Statement -- 2.3 Solutions - - 2.3.1 Strategies - - 2.3.2 Techniques and Tools - - 2.3.2.1 One-Shot Prompts - - 2.3.2.2 Structured Output with Provider-Specific APIs - - 2.3.2.2.1 JSON Mode - - 2.3.2.2.2 Structured Output Mode - - 2.3.2.3 LangChain - - 2.3.2.4 Outlines - - 2.3.2.4.1 Multiple Choice Generation - - 2.3.2.4.2 Pydantic model -- 2.4 Discussion - - 2.4.1 Comparing Solutions - - 2.4.2 Best Practices - - 2.4.3 Research & Ongoing Debate -- 2.5 Conclusion -- 2.6 Acknowledgements -- 2.7 References - -## Chapter 3: Input Size and Length Limitations -- 3.1 Context Window Constraints -- 3.2 Handling Long Inputs -- 3.3 Managing Token Limits -- 3.4 Chunking Strategies -- 3.5 Implementation Patterns -- 3.6 Testing Long-form Content - -## [Chapter 4: Output Size and Length Limitations](https://www.souzatharsis.com/tamingLLMs/notebooks/structured_output.html) -- 4.1 What are Token Limits? -- 4.2 Problem Statement -- 4.3 Content Chunking with Contextual Linking - - 4.3.1 Generating long-form content - - 4.3.2 Step 1: Chunking the Content - - 4.3.3 Step 2: Writing the Base Prompt Template - - 4.3.4 Step 3: Constructing Dynamic Prompt Parameters - - 4.3.5 Step 4: Generating the Report - - 4.3.6 Example Usage -- 4.4 Discussion -- 4.5 Implications -- 4.6 Future Considerations -- 4.7 Conclusion -- 4.8 References - -## [Chapter 5: The Evals Gap](https://www.souzatharsis.com/tamingLLMs/notebooks/evals.html) -- 5.1 Non-Deterministic Machines - - 5.1.1 Temperature and Sampling - - 5.1.2 The Temperature Spectrum -- 5.2 Emerging Properties -- 5.3 Problem Statement -- 5.4 Evals Design - - 5.4.1 Conceptual Overview - - 5.4.2 Design Considerations - - 5.4.3 Key Components - - 5.4.4 Metrics - - 5.4.4.1 Working Example - - 5.4.4.2 Considerations - - 5.4.5 Evaluators - - 5.4.5.1 Model-Based Evaluation - - 5.4.5.2 Human-Based Evaluation - - 5.4.6 Benchmarks & Leaderboards - - 5.4.7 Tools -- 5.5 References - -## Chapter 6: Hallucination: The Reality Gap -- 6.1 Understanding Hallucination Types -- 6.2 Detection Strategies -- 6.3 Grounding Techniques -- 6.4 Retrieval-Augmented Generation (RAG) - - 6.4.1 Context Selection - - 6.4.2 Indexing Strategies - - 6.4.3 Vector Stores - - 6.4.4 Chunking Methods -- 6.5 Practical Implementation - - 6.5.1 Building a RAG Pipeline - - 6.5.2 Testing and Validation - -## [Chapter 7: Preference-based Alignment](https://www.souzatharsis.com/tamingLLMs/notebooks/alignment.html) -- 7.1 Introduction -- 7.2 From Raw Capabilities to Preference Alignment -- 7.3 On the Misalignment of Language Models -- 7.4 Aligning Language Models with Human Preferences -- 7.5 Supervised Fine-Tuning (SFT) for Model Alignment -- 7.6 Augmenting SFT with Human Preferences -- 7.7 Case Study: Aligning a Language Model to a Policy -- 7.8 Discussion - -## Chapter 8: The Cost Factor -- 8.1 Understanding LLM Costs -- 8.2 Token Optimization -- 8.3 Caching Strategies - - 8.3.1 Implementation Patterns - - 8.3.2 Cache Invalidation -- 8.4 Output Prediction Techniques -- 8.5 Cost Monitoring -- 8.6 Optimization Strategies - -## Chapter 9: Breaking Free from Cloud Providers -- 9.1 The Vendor Lock-in Problem -- 9.2 Self-hosting Solutions - - 9.2.1 Llama Implementation - - 9.2.2 Llamafile Setup and Usage - - 9.2.3 Ollama Deployment -- 9.3 Performance Considerations -- 9.4 Cost Analysis -- 9.5 Migration Strategies - - -## Appendix A: Tools and Resources -- A.1 Evaluation Tools -- A.2 Monitoring Solutions -- A.3 Open Source Models -- A.4 Community Resources - ## Citation [![CC BY-NC-SA 4.0][cc-by-nc-sa-image]][cc-by-nc-sa] @@ -155,7 +35,7 @@ Abstract: *The current discourse around Large Language Models (LLMs) tends to fo ``` @misc{tharsistpsouza2024tamingllms, author = {Tharsis T. P. Souza}, - title = {Taming LLMs}, + title = {Taming LLMs: A Practical Guide to LLM Pitfalls with Open Source Software}, year = {2024}, journal = {GitHub repository}, url = {https://github.com/souzatharsis/tamingLLMs)