Neural Machine Translation of Rare Words with Subword Units
The paper adapts byte pair encoding to learn variable-length subword units, enabling open-vocabulary neural translation without an external dictionary fallback.
Bpe
How LLMs Read?
Everyone talks about the Neural Network, but the Tokenizer is the unsung hero of LLMs. This post explains what a Tokenizer actually does, why we use Byte Pair Encoding (BPE), and how these tokens bridge the gap between rigid integers and meaningful vector embeddings in models like GPT-4.