Skip to content

Latest commit

 

History

History
19 lines (15 loc) · 1.51 KB

README.md

File metadata and controls

19 lines (15 loc) · 1.51 KB

Token Insertion Model

This project entails the development of a neural network model to recognize and predict the positioning of a special token <X> within sentences provided by the client. Leveraging the BERT pre-trained large language model and TensorFlow, the project achieved over 90% F1-Score in predicting these patterns. The approach involved preprocessing the data using a BERT tokenizer, building a custom model architecture, and training it to classify token insertion points. Additionally, a function was implemented to automatically insert the <X> token into suitable places within sentences based on the trained model's predictions.

Technologies Used

  • TensorFlow
  • BERT (Bidirectional Encoder Representations from Transformers)
  • Python

Project Structure

  • Token_Insertion_Model.ipynb: Jupyter notebook containing the code for data preprocessing, model implementation, and training.
  • Model_Evaluation.ipynb: Jupyter notebook for evaluating the trained model on testing data and calculating performance metrics.
  • Token_Insertor.ipynb: Jupyter notebook with a function to insert <X> token into text based on model predictions.
  • dataset/: Directory containing JSON files for training and testing data.
  • output/: Directory for storing the trained model, evaluation results, and token insertion outputs.

Acknowledgements

  • This project was developed as part of a freelance project for a client.
  • The BERT pre-trained model used in this project is from the Hugging Face Transformers library.