Implement IR based Supervised Sentence Ranker

.gitattributes

@@ -5,3 +5,4 @@ sadedegel/prebuilt/model/*.joblib filter=lfs diff=lfs merge=lfs -text
 sadedegel/bblock/data/bert/vocabulary.hdf5 filter=lfs diff=lfs merge=lfs -text
 sadedegel/bblock/data/icu/vocabulary.hdf5 filter=lfs diff=lfs merge=lfs -text
 sadedegel/bblock/data/simple/vocabulary.hdf5 filter=lfs diff=lfs merge=lfs -text
+sadedegel/summarize/model/*.joblib filter=lfs diff=lfs merge=lfs -text

prod.requirements.txt

@@ -14,4 +14,7 @@ sadedegel-icu
 requests
 rich
 cached-property
-h5py>=3.1.0,<=3.2.1
+h5py>=3.1.0,<=3.2.1
+
+lightgbm
+randomname

sadedegel/default.ini

@@ -35,4 +35,4 @@ method = smooth
 [bm25]
 k1 = 1.25
 b = 0.75
-delta = 0
+delta = 0

sadedegel/summarize/README.md

@@ -10,6 +10,64 @@ by recoding the **Round** of each sentences in which it is eliminated.
 
 Later a sentence is eliminated, higher its relative score is within a given news document. 
 
+## Summarizer Usage
+
+SadedeGel summarizers share same interface. 
+
+First a `sadedegel.summarize.ExtractiveSummarizer` instance is constructed. 
+```python
+from sadedegel.summarize import LengthSummarizer, TFIDFSummarizer, DecomposedKMeansSummarizer
+
+lsum = LengthSummarizer(normalize=True)
+tfidf_sum = TFIDFSummarizer(normalize=True)
+kmsum = DecomposedKMeansSummarizer(n_components=200, n_clusters=10)
+```
+
+Create a `sadedegel.Document` instance from the single document to be summarized.
+```python
+from sadedegel import Doc
+
+d = Doc("ABD'li yayın organı New York Times, yaklaşık 3 ay içinde kullanıcı sayısını sıfırdan milyonlara çıkaran kelime oyunu Wordle’ı satın aldığını duyurdu. New York Times kısa bir süre önce de spor haberleri sitesi The Athletic'i satın almak için 550 milyon doları gözden çıkarmış ve bu satın alma ile birlikte 1.2 milyon abone kazanmıştı. ...")
+```
+
+For obtaining a summary of k sentences where k < n_sentences. Call the instance with a `Document` object or `List[Sentences]`
+
+```python
+summary1 = lsum(d, k=2)
+summary2 = tfidf_sum(d, k=4)
+summary3 = kmsum(d, k=5)
+```
+Alternatively you can obtain the relevance score of all sentences that is used to rank them to before selecting top k sentences.
+
+```python
+relevance_scores = kmsum.predict(d)
+```
+
+#### Supervised Ranker
+All sadedegel summarizers work either with unsupervised or rule based methods to rank sentences before extracting top k as the summary. In the new release we are providing a ranker model that is trained on **SadedeGel Annotated Corpus** that has documents where each sentence has relevance label assigned by human annotators through a process of repetitive elimination.
+
+Ranker uses document-sentence embedding pairs from transformer based pre-trained models as features. Future releases will accomodate BoW based and decomposition based embeddings as well. 
+For possible pre-trained embedding types supported by sadedegel are `bert_32k_cased`, `bert_128k_cased`, `bert_32k_uncased`, `bert_128k_uncased`, `distilbert`.
+
+```python
+from sadedegel.summarize import SupervisedSentenceRanker
+
+ranker = SupervisedSentenceRanker(vector_type="bert_32k_cased")
+```
+
+Supervised Ranker can be tuned for optimal performance over an embedding type and summarization percentage. Current ranker is optimized with `bert_128k_based` for average summarization performance over 10%, 50% and 80% of full document length.
+
+**Example**: Specific fine-tuning for short summaries with a smaller embedding extraction model.
+```python
+from sadedegel.summarize.supervised import RankerOptimizer
+
+fine_tuner = RankerOptimizer(vector_type="distilbert",
+                             summarization_perc=0.1,
+                             n_trials=20)
+
+fine_tuner.optimize()
+``` 
+
 ## Summarizer Performance 
 
 Given this [Model Definition](#sadedegel-model), 
@@ -28,6 +86,11 @@ ground truth human annotation (Best possible total `relevance` score that can be
 
 ### Performance Table
 
+#### Release 0.21.1
+| Method           | Parameter                                                                                                                                |   ndcg(optimized for k=0.1) |   ndcg(optimized for k=0.5) |   ndcg(optimized for k=0.8) |
+|------------------|------------------------------------------------------------------------------------------------------------------------------------------|---------------|---------------|---------------|
+| SupervisedSentenceRanker | `{"vector_type": "bert_128k_cased"}`                                                                                                                         |        0.7620 |        0.7269 |        0.8163 |
+
 #### Release 0.18
 
 By 0.18 we have significantly changed the way we evaluate our summarizers. 

sadedegel/summarize/__init__.py

@@ -4,3 +4,4 @@
 from .rank import TextRank, LexRankSummarizer  # noqa: F401
 from .tf_idf import TFIDFSummarizer  # noqa: F401
 from .bm25 import BM25Summarizer  # noqa: F401
+from. supervised import SupervisedSentenceRanker, RankerOptimizer  # noqa: F401

sadedegel/summarize/cluster.py

@@ -58,10 +58,9 @@ def _predict(self, sentences: List[Sentences]):
 
 class DecomposedKMeansSummarizer(ExtractiveSummarizer):
     """BERT embeddings are high in dimension and potentially carry redundant information that can cause
-        overfitting or curse of dimensionality effecting in clustering embeddings.
-
-        DecomposedKMeansSummarizer adds a PCA step (or any othe lsinear/non-linear dimensionality reduction technique)
-         before clustering to obtain highest variance in vector fed into clustering
+    overfitting or curse of dimensionality effecting in clustering embeddings.
+    DecomposedKMeansSummarizer adds a PCA step (or any other linear/non-linear dimensionality reduction technique)
+    before clustering to obtain highest variance in vector fed into clustering
     """
 
     tags = ExtractiveSummarizer.tags + ['cluster', 'ml']

sadedegel/summarize/supervised.py

@@ -0,0 +1,160 @@
+from os.path import dirname
+from pathlib import Path
+from itertools import tee
+import randomname
+
+import numpy as np
+from typing import List
+import joblib
+from rich.console import Console
+from rich.progress import track
+
+from ._base import ExtractiveSummarizer
+from ..bblock.util import __transformer_model_mapper__
+from ..bblock import Sentences
+from ..bblock.doc import DocBuilder
+from .util.supervised_tuning import optuna_handler, create_empty_model, fit_ranker, save_ranker
+
+
+__vector_types__ = list(__transformer_model_mapper__.keys()) + ["tfidf", "bm25"]
+console = Console()
+
+try:
+    import pandas as pd
+except ImportError:
+    console.log(("pandas package is not a general sadedegel dependency."
+                 " But we do have a dependency on building our supervised ranker model"))
+
+
+def load_model(vector_type, debug=False):
+    name = f"ranker_{vector_type}.joblib"
+
+    if vector_type == "bert_128k_cased":
+        path = (Path(dirname(__file__)) / 'model' / name).absolute()
+    else:
+        path = Path(f"~/.sadedegel_data/models/{name}").expanduser()
+
+    if not debug:
+        try:
+            model = joblib.load(path)
+            console.log(f"Initializing ranker model ranker_{vector_type}...", style="blue")
+        except Exception as e:
+            raise FileNotFoundError(f"A model trained for {vector_type} is not found. Please optimize one with "
+                                    f"sadedegel.summarize.RankerOptimizer. {e}")
+
+    else:
+        model = name
+
+    return model
+
+
+class SupervisedSentenceRanker(ExtractiveSummarizer):
+    model = None
+    vector_type = None
+    debug = False
+    tags = ExtractiveSummarizer.tags + ["ml", "supervised", "rank"]
+
+    def __init__(self, normalize=True, vector_type="bert_128k_cased", **kwargs):
+        super().__init__(normalize)
+        self.debug = kwargs.get("debug", False)
+        self.init_model(vector_type, self.debug)
+
+    @classmethod
+    def init_model(cls, vector_type, debug):
+        db_switch = False
+        if vector_type not in __vector_types__:
+            raise ValueError(f"Not a valid vectorization for input sequence. Valid types are {__vector_types__}")
+        if cls.debug != debug:
+            cls.debug = debug
+            db_switch = True
+            if cls.debug:
+                console.log("SupervisedSentenceRanker: Switching debug mode ON.")
+            else:
+                console.log("SupervisedSentenceRanker Switching debug mode OFF.")
+        if cls.vector_type is not None and not db_switch:
+            if cls.vector_type == vector_type:
+                return 0
+
+        cls.model = load_model(vector_type, debug)
+        cls.vector_type = vector_type
+
+    def _predict(self, sents: List[Sentences]) -> np.ndarray:
+        if self.vector_type not in ["tfidf", "bm25"]:
+            doc_sent_embeddings = self._get_pretrained_embeddings(sents)
+        else:
+            raise NotImplementedError("BoW interface for SupervisedSentenceRanker is not yet implemented.")
+
+        if self.model is not None:
+            scores = self.model.predict(doc_sent_embeddings)
+        else:
+            raise ValueError("A ranker model is not found.")
+
+        return scores
+
+    def _get_pretrained_embeddings(self, sents: List[Sentences]) -> np.ndarray:
+        doc_embedding = sents[0].document.get_pretrained_embedding(architecture=self.vector_type, do_sents=False)
+        doc_embedding = np.vstack(len(sents) * [doc_embedding])
+        sent_embeddings = sents[0].document.get_pretrained_embedding(architecture=self.vector_type, do_sents=True)
+
+        return np.hstack([doc_embedding, sent_embeddings])
+
+    def _get_bow_vectors(self, sents: List[Sentences]) -> np.ndarray:
+        pass
+
+
+class RankerOptimizer(SupervisedSentenceRanker):
+    def __init__(self, n_trials: int, vector_type: str, summarization_perc: float,**kwargs):
+        self.n_trials = n_trials
+        self.vector_type = vector_type
+        self.summarization_perc = summarization_perc
+
+    def optimize(self):
+        """Optimize the ranker model for a custom summarization percentage. Optimize and dump a new model.
+        """
+        run_name = randomname.get_name()
+        df, vecs = self._prepare_dataset()
+
+        optuna_handler(n_trials=self.n_trials, run_name=run_name,
+                       metadata=df, vectors=vecs, k=self.summarization_perc)
+
+        model = create_empty_model(run_name)
+        ranker = fit_ranker(ranker=model, vectors=vecs, metadata=df)
+        save_ranker(ranker, name=self.vector_type)
+
+    def _prepare_dataset(self):
+        try:
+            from sadedegel.dataset import load_raw_corpus, load_annotated_corpus
+        except Exception as e:
+            raise ValueError("Cannot import raw and annotated corpi.")
+
+        annot = load_annotated_corpus()
+        annot_, annot = tee(annot)
+
+        embs = []
+        metadata = []
+        Doc = DocBuilder()
+        for doc_id, doc in track(enumerate(annot), description="Processing documents", total=len(list(annot_))):
+
+            relevance_scores = doc["relevance"]
+            d = Doc.from_sentences(doc["sentences"])
+            sents = list(d)
+
+            for sent_id, sent in enumerate(sents):
+                instance = dict()
+                instance["doc_id"] = doc_id
+                instance["sent_id"] = sent_id
+                instance["relevance"] = relevance_scores[sent_id]
+
+                metadata.append(instance)
+
+            if self.vector_type not in ["tfidf", "bm25"]:
+                doc_sent_embeddings = self._get_pretrained_embeddings(sents)
+            else:
+                raise NotImplementedError("BoW interface for SupervisedSentenceRanker is not yet implemented.")
+
+            embs.append(doc_sent_embeddings)
+
+        df = pd.DataFrame().from_records(metadata)
+        vecs = np.vstack(embs)
+
+        return df, vecs