Building a Fast Tokenizer, One Bottleneck at a Time: The Algorithms
From a dozen lines of BPE in Python to an 85× faster trainer: incremental counts, heaps, pretokenization, and weighted deduplication, without changing a single learned merge.
Engineering 4.1k words