Tokens and Vocabulary in Large Language Models: How Text Becomes Computation

When you type a question into an AI chatbot, it doesn’t see words like you do. It sees numbers. Every letter, word, or part of a word gets broken down into tiny pieces called tokens. These tokens are the only thing the model understands. Think of them as the alphabet of machine language. Without tokens, there’s no way for a language model to turn your sentence into something it can think about, remember, or reply to. This is how text becomes computation.

What Exactly Is a Token?

A token is the smallest unit of text a language model processes. It can be a whole word like "cat," a part of a word like "run" and "ning" in "running," or even punctuation like "," or "?". The model doesn’t care about grammar or meaning-it just matches each piece of text to a number in its internal dictionary, called a vocabulary. That number then gets turned into a vector-a list of numbers that represent that token’s meaning in the model’s mind.

For example, the sentence "I heard a dog bark loudly at a cat" might be split into 9 tokens if using word-level tokenization. But if you used character-level tokenization, it would become 34 tokens. That’s a big difference. More tokens mean more memory, more processing, and higher costs. That’s why modern models don’t use either extreme. They use something smarter: subword tokenization.

Why Subword Tokenization Rules the Roost

Early models tried word-level tokenization. But that meant needing a vocabulary with over a million entries to cover every possible word, including rare ones like "university" or "polyethylene terephthalate." That’s not practical. Character-level tokenization was too slow. Every word became a string of letters, and the model had to process way too many steps just to understand a single sentence.

Enter Byte-Pair Encoding (BPE). This method starts with individual characters and learns which ones appear together most often. Over time, it merges them into bigger chunks. So "dog" stays as one token, but "running" becomes "run" + "ning" because those two parts show up often in training data. It’s not based on linguistics-it’s based on frequency. That’s why "dogs" and "dog" are often separate tokens, even though they’re related. And why "racket" might split into "rack" and "##et" even though it’s one word.

Most major models today use BPE or a variation. GPT-4 uses around 100,000 tokens. LLaMA 3 uses 128,000. BERT, older but still widely used, has about 30,000. Each of these numbers represents the size of the model’s vocabulary-the list of all the pieces it knows how to recognize.

How the Vocabulary Works Behind the Scenes

The vocabulary is like a massive lookup table. When you input text, the tokenizer scans it from left to right, trying to match the longest possible sequence from its vocabulary. If it hits a word it doesn’t know, it breaks it into smaller pieces it does know. This is why technical terms often explode into multiple tokens. "Polyethylene terephthalate"? That’s five tokens for many models. But "cat"? One token. Simple.

Each token has an ID. That ID points to an embedding-a dense vector of numbers that the model uses to understand relationships. "King" and "queen" might have similar embeddings because they appear in similar contexts. "Dog" and "bark" might be close because they often appear together. These relationships are learned during training. Tokens are the bridge between human language and these mathematical representations.

Why Token Count Matters More Than You Think

Token count isn’t just a technical detail. It’s money. Most LLM APIs charge per token. Input tokens cost a few cents per thousand. Output tokens cost more. If your document gets split into twice as many tokens as expected, your bill doubles.

And it’s not just cost. Context windows-the maximum number of tokens a model can handle at once-directly limit what you can do. GPT-4 can handle 128,000 tokens. That’s about 300 pages of text. But if your input uses 100,000 tokens just to process a long legal contract, you’ve got almost no room left for the model’s response. That’s why developers now check token counts before sending anything to an API.

Simple word counting can be wildly wrong. Microsoft found that for technical documents, word counts can underestimate token usage by 30-50%. A sentence with 10 words might use 22 tokens. That’s why tools like Hugging Face’s tokenizer visualizer are now standard. Developers paste text in, see exactly how it splits, and adjust before sending it to the model.

Problems You Won’t Find in Tutorials

Here’s what no one tells you: tokenization is messy. Non-English languages often get tokenized worse than English. A Spanish word like "desarrollo" might split into "des" + "arrollo" if it’s rare, even though it’s one word. That adds tokens and slows things down. In 2024, developers reported an 18% drop in token count for Spanish text when switching from GPT-3.5 to GPT-4-because its vocabulary got bigger and better at handling common non-English words.

Medical, legal, and scientific terms are the worst offenders. IBM’s 2023 study showed medical jargon needed 22% more tokens than regular text. Why? Because models weren’t trained on enough of it. The tokenizer has never seen "methotrexate" before, so it breaks it into "meth" + "ot" + "rex" + "ate." That’s four tokens for one drug name. In a 10-page patient report, that adds up fast.

And then there’s inconsistency. Different versions of the same model can tokenize the same text differently. Hugging Face had over 200 open issues in early 2024 about tokenization changes between model updates. One developer on Reddit spent three days debugging why their code broke after upgrading from LLaMA 2 to LLaMA 3-turns out, the tokenizer split "JSON" differently.

What’s Changing in 2025 and Beyond

Models are getting smarter about tokens. LLaMA 3.1, released in January 2025, improved tokenization for non-English languages by 12-18%. That’s huge for global applications. OpenAI is working on GPT-5 with adaptive vocabulary sizing-meaning the model could dynamically adjust its token list based on the input. If you’re asking about quantum physics, it might temporarily expand its vocabulary to handle "superposition" and "entanglement" more efficiently.

Companies are starting to build custom tokenizers. IBM, Salesforce, and others are training their own vocabularies using internal documents. One enterprise client cut LLM costs by 37% just by preprocessing text to combine technical terms into single tokens before sending them to the model.

But the future isn’t all about bigger vocabularies. Researchers are exploring alternatives-like continuous embeddings that don’t rely on discrete tokens at all. But those are still experimental. For now, tokens are here to stay. Gartner predicts that by 2027, 60% of enterprise LLMs will use custom tokenizers tailored to their industry.

What You Need to Do Today

If you’re using LLMs in your work, here’s what matters:

• Always check token counts before sending long texts. Use Hugging Face’s tokenizer or OpenAI’s official tool.
• Preprocess text: Combine known technical terms into single tokens. Replace "polyethylene terephthalate" with "PET" if your audience knows it.
• Don’t assume word count = token count. For technical content, expect 30-50% more tokens.
• Test across models. GPT-4 handles non-English better than older models. LLaMA 3 is better for code. Choose based on your text.
• If you’re building an app, build token monitoring into your pipeline. Track cost per token, not per request.

Tokenization isn’t glamorous. It’s not AI magic. But it’s the foundation. Get it wrong, and your model is slow, expensive, or broken. Get it right, and you unlock faster, cheaper, more reliable AI.