This project implements a Byte Pair Encoding (BPE) tokenization approach along with a Word2Vec model to generate word embeddings from a text corpus. The implementation leverages Apache Hadoop for distributed processing and includes evaluation metrics for optimal dimensionality of embeddings.
scala logback word2vec scalatest hadoop-mapreduce deeplearning4j nd4j apache-hadoop llm bpe-tokenizer jtokkit
-
Updated
Nov 2, 2024 - Scala