Lesson 10: What Is a Token?

Course: Local AI 101
Level: Beginner → Intermediate
Estimated reading time: 50–70 minutes
Prerequisites: Lessons 1–9
Next lesson: Lesson 11 — How Does a Language Model Generate Text?

What You Will Learn

By the end of this lesson, you should be able to:

• Explain exactly what a token is.
• Explain why LLMs do not directly process ordinary human-readable words.
• Understand token IDs and vocabularies.
• Understand what a tokenizer does.
• Understand why words are not necessarily one token each.
• Understand subword tokenization.
• Understand Byte Pair Encoding and BPE-style tokenization.
• Understand SentencePiece-style tokenization at a conceptual level.
• Understand special tokens.
• Understand how spaces and punctuation can affect tokenization.
• Understand why numbers can tokenize strangely.
• Understand why programming code can tokenize differently from normal prose.
• Understand why different languages can require different numbers of tokens.
• Understand why tokenization matters for context windows.
• Understand why tokenization affects memory and inference.
• Understand the relationship between tokens, embeddings, and parameters.
• Understand how text becomes token IDs.
• Understand how token IDs become vectors.
• Understand how generated token IDs become human-readable text again.
• Understand why two models can tokenize the same sentence differently.
• Understand why a tokenizer is tied to a model or model family.
• Understand special tokens used for conversations and generation.
• Understand tokenization's importance when choosing local models.
• Understand practical token-counting examples.
• Build a technically accurate mental model of the entire text → token → model → token → text pipeline.

1. Introduction: The Word "Token" Is Everywhere

If you have been reading about AI, you have probably encountered statements such as: This model has a 128K context window. or:

The prompt is 4,000 tokens. or: The model generated 500 tokens.

You may also have seen token-related discussions involving:

• token limits
• context windows
• token counts
• tokens per second
• input tokens
• output tokens
• tokenization
• tokenizers
• token IDs
• special tokens
• vocabulary size

But what exactly is a token? A beginner often assumes:

That is not correct. Sometimes a word is one token. Sometimes a word is multiple tokens.

Sometimes part of a word is a token. Punctuation can be a token. A space can be associated with a token.

A number can be split into multiple tokens. Programming code can be split in ways that look completely unintuitive to a human. Different models can tokenize the exact same sentence differently.

Understanding this is extremely important for local AI because tokens sit directly between:

human language

and: the numerical representation processed by the model. The token is the bridge.

2. The Short Definition

A token is a discrete unit of text that a language model's tokenizer converts into a numerical token ID. That definition is more precise than:

"A token is a piece of a word."

Sometimes it is. But a token can represent:

• a whole word
• part of a word
• punctuation
• whitespace-related text
• a number
• a symbol
• a special control marker
• combinations of characters

The exact tokenization depends on the tokenizer. A useful conceptual pipeline is:

And on the way out:

3. Why Does an LLM Need Tokens?

A neural network operates on numerical data. It does not receive the sentence: The cat sat on the mat.

as a collection of English words in the way a human reads it. The model needs a numerical representation. So the system converts text into something like:

[1823, 941, 573, 91, 44, 762]

The numbers above are completely fictional. The important concept is:

text → token IDs

The token IDs are integers. The model can then use those IDs to retrieve learned vectors from an embedding matrix. So:

That is the basic entrance into the LLM.

4. Token vs Token ID

These terms are related but different. A token is the actual vocabulary item or text unit. A token ID is the integer assigned to that token in a particular vocabulary.

Imagine a fictional vocabulary:

Token ID

"hello" 102 "world" 205 "ing" 811 "." 13

The token is:

"hello"

The token ID is:

102

So:

The ID is simply a numerical identifier used by the model's processing system.

5. Vocabulary

A tokenizer has a vocabulary. The vocabulary is the collection of tokens that the tokenizer knows how to represent directly. For example, a toy vocabulary might contain:

hello world the cat dog ing . , ! ?

A real modern language model's vocabulary can contain tens of thousands or more token entries. The model maps token IDs into learned representations. Conceptually:

The exact vocabulary depends on the model and tokenizer.

6. Why Not Just Use Words?

This is one of the first big questions. Why not create a vocabulary containing every word? Suppose we tried:

apple banana computer language model ...

At first this seems reasonable. But language has enormous vocabulary diversity. Consider:

walk walks walked walking walker walkers

Would we store each as a completely separate vocabulary entry? What about:

unbelievable unbelievably unbelievability

What about names? Technical terms? Misspellings?

URLs? Code? Numbers?

New words? Languages with productive word formation? A word-only vocabulary becomes awkward very quickly.

Subword tokenization solves much of this problem.

7. The Basic Idea of Subword Tokenization

Instead of requiring every possible word to exist as a token, a tokenizer can represent words using smaller pieces. For example, a fictional tokenizer might represent:

unbelievable

as:

un believ able

or perhaps:

un believable

depending on the tokenizer. Another word:

walking

might be represented as:

walk ing

Again, these examples are illustrative. The exact tokens depend on the vocabulary. The important idea is:

A tokenizer can build unfamiliar words out of known pieces.

8. Why Subword Tokenization Is Useful

Suppose a tokenizer has a vocabulary containing:

play ing played

It can represent:

playing

efficiently. But imagine encountering:

hyperlocalization

which the tokenizer has never seen as one complete vocabulary entry. It can still break it into pieces. For example, conceptually:

hyper local ization

The exact result depends on the tokenizer. This gives language models a practical compromise:

Vocabulary not infinitely large + Ability to represent unfamiliar words

9. Tokenization Is a Compression Strategy

There is another useful way to think about tokenization. A tokenizer is trying to represent text efficiently using a finite vocabulary. If every character were always one token:

The cat sat. would require roughly one token per character. That would create extremely long sequences.

If every entire sentence were one token, the vocabulary would become impossibly large. Subword tokenization sits somewhere between those extremes. Conceptually:

Different tokenization schemes make different tradeoffs.

10. Character-Level Tokenization

The simplest tokenizer could assign every character its own token. For:

hello

we might have:

h e l l o

Five tokens. For: Hello, world!

we could have:

H e l l o ,

w o r l d !

This approach has a relatively small vocabulary. But sequences become much longer. Longer sequences can increase computational and memory requirements.

11. Word-Level Tokenization

At the other extreme, we could use whole words. For: The cat sat down.

we might have:

The cat sat down .

This is compact. But now we need a vocabulary containing enormous numbers of possible words. And what happens when the model encounters:

antidisestablishmentarianism

or:

Lyndon's

or:

QWERTYZX

or a newly invented word? A word-level tokenizer needs an unknown-token strategy. Subword tokenization provides a more flexible solution.

12. Subword Tokenization as the Middle Ground

Subword tokenization attempts to find useful recurring pieces. For example, a tokenizer may have tokens corresponding to common fragments such as:

ing tion pre un able computer language

This allows common words to be represented compactly while still allowing unfamiliar words to be decomposed. The result is a vocabulary that is:

large enough to be efficient

but: small enough to be practical.

13. Byte Pair Encoding

One famous family of tokenization methods is Byte Pair Encoding, commonly abbreviated BPE. BPE originated as a compression technique and was later adapted for subword tokenization. The high-level idea is:

1. Start with smaller units.
2. Identify frequently occurring adjacent pairs.
3. Merge common pairs.
4. Repeat the process.
5. Build a vocabulary of useful subword units.

This allows frequent sequences of characters or bytes to become tokens.

14. A Simplified BPE Example

Imagine a toy dataset:

low lower lowest

Start with smaller pieces. The tokenizer might discover frequent combinations. Conceptually:

l o w l o w e r l o w e s t

It might notice:

Then:

and so on. Eventually, common patterns can become vocabulary entries. This example is deliberately simplified.

Real tokenizers operate with more sophisticated rules and representations.

15. Why BPE Works

Language contains repeated patterns. For example:

ing tion er re un

appear frequently across words. BPE can discover frequent combinations and store them as units. This reduces the number of tokens needed for common text.

The result is a kind of learned compression vocabulary.

16. BPE Does Not Mean "Split on Syllables"

This is important. Token boundaries do not necessarily correspond to:

• syllables
• morphemes
• pronunciation
• dictionary words
• linguistic units

A tokenizer is optimized for its own objective and vocabulary construction. A token might correspond to a linguistically meaningful piece. Or it might simply be a frequent sequence of bytes or characters.

Do not assume:

The relationship is often imperfect.

17. SentencePiece-Style Tokenization

Another important family of tokenization systems is associated with SentencePiece.

SentencePiece provides a tokenizer framework that can operate directly on raw text rather than requiring whitespace-separated words as a preprocessing step.

This is particularly useful for multilingual and language-independent processing. The exact tokenization algorithm can vary. SentencePiece can support approaches such as:

• BPE
• unigram language-model tokenization

The important lesson is that modern tokenizers are more sophisticated than:

split text on spaces

18. Whitespace Is Weird

Consider:

hello world

A tokenizer does not necessarily represent this as:

hello world

with a separate token for the space. Depending on the tokenizer, the space can be incorporated into one of the tokens. For example, a vocabulary might contain conceptual tokens corresponding to:

"hello" " world"

rather than:

"hello" " " "world"

The exact representation is model-specific. This is why token counts can be surprising.

19. Leading Spaces Can Matter

Imagine:

hello

versus:

hello

They contain the same letters but different whitespace. A tokenizer may assign different token sequences. This can be useful because natural text contains enormous numbers of recurring patterns involving spaces.

The tokenizer can learn efficient units that include whitespace.

20. Punctuation Can Be Tokens

Consider: Hello, world! Punctuation may be represented as separate tokens or combined with neighboring text.

For example, a tokenizer might conceptually have:

Hello , world !

or another arrangement. The exact behavior depends on the vocabulary. The important point is:

Punctuation is part of the tokenization problem.

21. Numbers Can Be Surprisingly Complicated

Humans see:

123456789

as one number. A tokenizer may split it into multiple pieces. For example, conceptually:

123 456 789

or:

12 345 678 9

or another segmentation. The exact result depends on the tokenizer. This can have practical consequences for:

• arithmetic
• long numbers
• IDs
• dates
• financial data
• scientific notation

Numbers are not necessarily tokenized as one indivisible object.

22. Dates

Consider:

2026-09-28

A tokenizer might split this into multiple tokens involving:

2026

09

28

or different pieces. The model does not automatically see:

September 28, 2026

as one semantic date object. It sees token representations. The neural network can nevertheless learn patterns involving dates.

23. URLs

URLs can be tokenized into many pieces. Consider:

https://example.com/products/ai

A tokenizer may represent recurring pieces such as:

https :// example .com / products / ai

or something more fragmented. URLs can therefore consume more context than a human might expect. This matters when processing large collections of web text.

24. Email Addresses

An email address such as:

user@example.com

may be split into several tokens. The tokenizer is not necessarily aware that it is an email address in a special structured sense. It is processing a sequence of characters according to its vocabulary.

The model can learn patterns associated with email addresses through training.

25. Code Tokenization

Programming code is another fascinating case. Consider:

def calculate_total(price, tax):
    return price * (1 + tax)

Humans see:

• function
• variable names
• parentheses
• comma
• colon
• indentation
• return statement
• arithmetic

The tokenizer sees a sequence of vocabulary units. Some common code fragments may have dedicated tokens. Others may be split into smaller pieces.

This is one reason code-specialized models can benefit from training on large amounts of source code.

26. Indentation and Whitespace in Code

Python makes whitespace particularly important. For example:

if x > 10: print(x)

versus:

if x > 10: print(x)

are not equivalent Python programs. Tokenization therefore needs to represent whitespace information sufficiently for the model to learn these patterns. The exact token representation depends on the tokenizer.

27. Why Code Can Consume Many Tokens

A short-looking code file can contain:

• symbols
• identifiers
• punctuation
• whitespace
• operators
• comments
• brackets

A tokenizer may split these into many tokens. This means a codebase can consume a large context window even when the file does not look enormous to a human. This becomes important when using LLMs for software development.

28. Different Languages Tokenize Differently

One of the most important tokenization facts is:

The same amount of human-readable text does not necessarily correspond to the same number of tokens across languages.

Why? Because tokenizers are built from training data and vocabulary statistics. A tokenizer trained heavily on one language may represent common words in that language efficiently.

Another language may require more subword pieces. For example, two sentences with roughly the same number of characters can have very different token counts.

29. Why Multilingual Tokenization Matters

Suppose a model supports:

English French Zulu Japanese Arabic Chinese Spanish

The tokenizer needs to represent all of these. A vocabulary optimized heavily for English may not be equally efficient for every language. This can affect:

• context usage
• inference speed
• memory
• multilingual performance
• effective information density

Therefore, tokenizer design is part of model design.

30. Languages With Different Writing Systems

Consider:

English

which uses the Latin alphabet. Compare it with:

中文

or:

العربية

or:

日本語

These writing systems have very different character distributions. A tokenizer needs to represent them efficiently. Modern tokenizers often use byte-level or Unicode-aware techniques that allow arbitrary text to be represented.

But token efficiency can still vary substantially.

31. Unicode

Computers need a way to represent text from many writing systems.

Unicode provides a standard system for representing characters and symbols.

A tokenizer operates on text that can contain Unicode characters. This introduces additional complexity involving:

• accented characters
• emoji
• non-Latin scripts
• combining marks
• symbols
• normalization

The tokenizer's internal representation determines how these are converted into tokens.

32. Emoji

Humans may see:

😀

as one symbol. A tokenizer may represent it using one or multiple tokens. Complex emoji sequences can be even more interesting.

For example:

👨‍💻

is visually one emoji but is constructed from multiple Unicode components joined using special characters. A tokenizer does not necessarily treat the visual glyph as one token. This is another reminder:

Human-perceived characters and model tokens are different concepts.

33. Combining Characters

Some visible characters can be represented by multiple Unicode code points. For example, an accented letter can sometimes be represented as:

base character + combining accent

The tokenizer may therefore encounter sequences that look like one character to a human but involve multiple underlying units.

Modern tokenization systems need to handle these cases.

34. Tokenization Is Model-Specific

This is one of the most important practical rules. Do not assume: Every LLM uses the same tokenizer.

Different model families can use different:

• vocabularies
• token IDs
• special tokens
• tokenization algorithms
• normalization rules
• chat templates

Therefore:

Same text + different tokenizer = potentially different token sequence

35. Token IDs Are Not Universal

Suppose one tokenizer says:

Another tokenizer might say:

There is no universal token ID for "hello."

The ID only has meaning within that model's vocabulary. This is exactly like an employee ID inside a company.

ID 123

means something only within the relevant system.

36. Tokenizer and Embedding Matrix Must Match

This is a critical architectural relationship. Suppose the tokenizer produces:

The model's embedding matrix needs to have a corresponding row. Conceptually:

If the tokenizer and model vocabulary are incompatible, the system cannot correctly interpret the token IDs. This is why model files and tokenizer configurations need to be consistent.

37. Vocabulary Size and Parameters

Remember Lesson 8. If a model has:

an embedding matrix may contain roughly:

V × D

parameters. For example:

50,000 × 4,096 ≈ 204.8 million

parameters. If the vocabulary were much larger, the embedding matrix could become substantially larger. Tokenization therefore has a direct relationship to model architecture and parameter count.

38. The Embedding Lookup

Suppose:

The model looks up row 42 in the embedding matrix. Conceptually:

Embedding matrix

row 0 → vector row 1 → vector row 2 → vector ... row 42 → [0.14, -0.07, 0.92, ...] ...

The token ID itself is not the semantic representation. It is an index. The learned vector associated with that index is what enters the neural network.

39. Token IDs Are Just Labels

This is important. Suppose:

That does not mean:

dog is numerically greater than cat

or: dog is one unit more intelligent than cat. The IDs are arbitrary identifiers assigned by the vocabulary.

The meaningful learned representation comes from the embedding values.

40. Why Token IDs Can Be Large

You may see IDs such as:

50256

or:

128000

These numbers are not "coordinates in meaning space."

They are simply vocabulary indices. The actual learned semantic relationships exist in the embedding and subsequent network transformations.

41. Special Tokens

Modern LLMs often use special tokens that do not correspond to ordinary human text. Examples include conceptual markers for:

beginning of sequence end of sequence padding unknown text system role user role assistant role tool calls

The exact special tokens depend on the model. They help structure the sequence.

42. End-of-Sequence Tokens

A model needs some way to indicate that generation can stop. An end-of-sequence token can serve this purpose. Conceptually:

The application can recognize that generation should stop. Other stopping mechanisms can also exist.

43. Beginning-of-Sequence Tokens

Some models use a special token to indicate the beginning of a sequence. Conceptually:

[BOS] Hello world

This provides structural information. Not every architecture uses the exact same special-token conventions.

44. Role Tokens

Chat models can use special formatting to distinguish roles. Conceptually:

[SYSTEM] You are a helpful assistant.

[USER] Explain tokens.

[ASSISTANT] ...

The model learns to interpret these patterns during instruction tuning. The exact tokens and formatting are model-specific.

45. Why Chat Templates Matter

Suppose a model expects a specific conversation format. If an application sends:

User: Hello Assistant:

but the model expects a different structured representation, the model may behave poorly. The model has been trained to interpret particular token patterns. Therefore, the inference system needs to apply the appropriate chat template when required.

This is particularly important when running models locally.

46. Tokenization and Context Windows

Now we reach one of the most important practical consequences. A model's context window is measured in tokens. For example:

8K tokens 32K tokens 128K tokens

These numbers do not mean:

8,000 words 32,000 words 128,000 words

They mean approximately:

8,000 token units 32,000 token units 128,000 token units

The number of human-readable words represented by that context depends on the tokenizer and language.

47. Why "128K Context" Does Not Mean 128K Words

Suppose a tokenizer averages roughly several characters per token for some English text. Then:

128,000 tokens

could correspond to a very large amount of prose. But the exact relationship varies. Code, URLs, numbers, and other text types can tokenize differently.

Other languages can also have different token efficiency. Therefore:

and:

There is no universal conversion.

48. Token Count and Memory

Every token in the context can contribute to runtime memory requirements. The model has to process the context. During autoregressive generation, the KV cache can grow with the sequence.

Therefore:

This is one reason context length matters for local AI hardware.

49. Token Count and Computation

More tokens generally mean more work. Consider:

Prompt A: 100 tokens

Prompt B: 10,000 tokens

Prompt B contains vastly more context to process. The exact computational scaling depends on the architecture and implementation, but the principle is straightforward:

Longer context is a larger computational workload.

This is why extremely long-context AI can require substantial hardware resources.

50. Tokens Per Second

You may encounter a performance metric:

tokens per second

This describes how quickly the system generates tokens under a particular workload and measurement method. For example:

20 tokens/second

means the system is generating approximately twenty tokens per second under the relevant conditions. It does not mean:

20 words/second

because one token is not necessarily one word.

51. Why Tokens Per Second Can Be Misleading

Suppose:

Model A: 30 tokens/sec

and:

Model B: 20 tokens/sec

That does not automatically mean Model A is better. You also need to know:

• model size
• quantization
• prompt length
• hardware
• context
• generation settings
• batch size
• measurement methodology

Performance numbers are useful only when the conditions are understood.

52. Input Tokens vs Output Tokens

When discussing LLM usage, you may see:

input tokens

and:

output tokens

Input tokens are the tokens in the prompt/context. Output tokens are the tokens generated by the model. For a local system, both affect the workload.

Conceptually:

The model processes both as part of the overall generation process, although prompt processing and generation have different computational characteristics.

53. The Context Window Includes More Than the User's Sentence

Suppose you type: Explain local AI. The model may actually receive a much larger sequence containing:

system instructions + conversation history + tool results + retrieved documents + user message

All of this can consume context. This is why a short visible prompt can still result in substantial token usage.

54. Hidden Context

Applications can add context that the user does not directly type. For example:

System instructions

or:

retrieved document excerpts

or:

tool results

These become part of the model's input.

Therefore, when debugging context limits, you need to consider the entire assembled prompt rather than only the last user message.

55. Tokenization and RAG

Retrieval-Augmented Generation systems often retrieve documents and place relevant sections into the model context. Suppose:

User question: "What is our refund policy?"

The retrieval system finds:

Refund policy document

Then:

Question + Retrieved document

becomes the model input. If the retrieved document is huge, it can consume a large portion of the context window. Therefore tokenization directly affects RAG design.

56. Chunking Documents

When building a knowledge base, documents are often split into chunks. Why? Because you do not necessarily want to place an entire 500-page document into the context.

Instead:

The chunk size is often considered in terms of tokens. This is one reason token knowledge becomes important when building local AI systems.

57. Tokenization and Document Chunking

Suppose a chunk is:

1,000 tokens

and you retrieve:

5 chunks

You could have approximately:

5,000 tokens

of retrieved context. Then add:

system instructions + conversation + user question

and the total becomes larger. A good retrieval system needs to manage the token budget.

58. Token Budgets

A context window can be thought of as a budget. For example:

Maximum context: 32,000 tokens

You might allocate:

System instructions 1,000 Conversation 8,000 Retrieved documents 15,000 User prompt 1,000 Reserved output 7,000

Total 32,000

These are illustrative numbers. The important idea is:

Context is finite and must be managed.

59. Why Output Limits Matter

If the context window has a fixed maximum, you cannot fill all of it with input and then expect unlimited output. Conceptually:

Context limit

input context + generated output

The exact implementation and terminology can vary. But the model needs room for the tokens it will generate. This is another reason applications may impose maximum output-token settings.

60. Tokenization and Conversation History

In a long chat, the conversation history can consume a large number of tokens. For example:

Turn 1 Turn 2 Turn 3 ... Turn 50

If every turn remains in the context, the token count grows. Applications may therefore use strategies such as:

• truncating older messages
• summarizing history
• retrieving relevant history
• maintaining external memory
• selectively including messages

These are system-level strategies for managing context.

61. Why Local AI Users Need to Understand This

If a local model suddenly becomes slower or runs out of memory during a long conversation, context length may be part of the explanation.

The model parameters have not necessarily changed. The workload has changed. You may now have:

short prompt

versus:

very long conversation

The model's context and KV cache can therefore be much larger.

62. Tokens and the KV Cache

From Lesson 9, we learned that autoregressive generation uses a KV cache. The KV cache stores attention-related state for previous tokens. As more tokens enter the context:

The exact memory scaling depends on architecture. But the general relationship is important:

Longer context can mean a larger KV cache.

This is one reason context length matters for local hardware.

63. Tokens and Parameters Are Different

This distinction should now be extremely clear.

Token → unit of input/output sequence

Parameter → learned numerical value in the model

For example:

Input: "The cat sat."

Tokens: [Token A, Token B, Token C, Token D]

The model may contain:

8 billion parameters

These are completely different quantities.

64. Tokens and Embeddings

The relationship is:

The vector comes from the model's learned embedding parameters. So:

Token ID

is an index. The embedding vector is a learned representation.

65. Tokens and Model Parameters

Suppose a model has:

The embedding matrix may contain:

50,000 × 4,096

learned values. When token ID:

12345

is encountered, the model retrieves the corresponding row. That row contains:

4,096 numerical values

in this example. Those values are parameters.

66. The Token Pipeline

The entire relationship can be represented as:

This is the fundamental token lifecycle.

67. Tokenizer as a Translator

A useful analogy is:

The tokenizer is a translator between human-readable text and the discrete vocabulary language used by the model.

It translates:

Human text

into:

token IDs

and later translates:

token IDs

back into: human-readable text. But remember that the token IDs are not a language with semantic meaning by themselves.

They are identifiers.

68. Why You Cannot Swap Tokenizers Casually

Imagine a model was trained with tokenizer A. You replace it with tokenizer B. Now the same sentence produces different token IDs.

The model's embedding matrix expects the original vocabulary. The mapping breaks. Conceptually:

But:

Therefore tokenizer and model vocabulary are tightly connected.

69. Tokenization Is Part of the Model Ecosystem

When downloading a model, you may encounter:

• model weights
• configuration
• tokenizer files
• tokenizer configuration
• special-token definitions
• chat template information

These pieces work together. A local AI platform needs to handle these relationships correctly. This is particularly important when supporting a large model library.

70. Token Efficiency

People sometimes discuss:

tokens per character

or:

characters per token

as rough measures of token efficiency. For some text, a token may correspond to several characters. For other text, tokens may be much smaller.

There is no universal ratio. It depends on:

• language
• tokenizer
• content
• formatting
• code
• numbers
• symbols

71. Why Token Counts Differ Across Text Types

Compare:

Normal English prose

The weather is beautiful today.

Source code

for i in range(100): print(i)

URL

https://example.com/path/to/file

Random identifier

a9F7xQ2LmP8zR4

All contain different patterns. A tokenizer may represent them with very different token efficiencies. This is why token-count estimates should be treated as estimates unless actually tokenized.

72. Long Identifiers

Consider:

a9F7xQ2LmP8zR4K6T1

A human sees a single identifier. A tokenizer may split it into many pieces. This matters for:

• API keys
• database IDs
• hashes
• UUIDs
• product codes
• serial numbers

Such strings can consume surprisingly large numbers of tokens. This is one reason sensitive credentials should never be pasted unnecessarily into prompts.

73. Tokenization and Security

Tokenization itself is not a security mechanism. But understanding tokenization can help you understand why certain inputs consume large amounts of context. Security-sensitive strings such as:

passwords API keys authentication tokens private credentials

should be protected regardless of how they tokenize. Local AI systems still need secure handling of input data.

74. Tokenization and Prompt Injection

Tokenization does not automatically prevent malicious instructions embedded in text. A retrieved document could contain: Ignore previous instructions...

Those characters become tokens just like ordinary text. The model processes them according to its learned behavior and context. This is one reason AI systems need careful separation between:

trusted instructions

and: untrusted retrieved content. We will study AI security later.

75. Tokenization and Special Formatting

A model may use special token sequences to separate roles. For example:

... ... ...

These strings may correspond to special token IDs or ordinary tokens depending on the model. The exact behavior is model-specific. The important concept is that conversational structure is ultimately represented as a token sequence.

76. Tokenization and Chat Templates

A chat template can be thought of as a formatting recipe. Conceptually:

For example:

might be transformed into a model-specific sequence. The model has been trained to interpret that sequence. This is why a good local AI application needs to know the appropriate chat format for a model.

77. Tokenization and Model Families

Different model families can use different tokenizers. Even models with similar parameter counts may have:

different vocabulary different token IDs different special tokens different chat templates

Therefore, "8B" tells you almost nothing about tokenization by itself. You need model-specific metadata.

78. Token Count Is Not Meaning

A sentence with:

100 tokens

is not necessarily "more meaningful" than a sentence with: 50 tokens. Token count measures representation length.

It does not measure:

• intelligence
• importance
• truth
• complexity
• quality

A short sentence can express a profound idea. A long sentence can be meaningless.

79. Token Count Is Not Word Count

This is worth repeating. Suppose:

You cannot assume:

The token count could be:

80 120 150

or another value. The relationship depends on the tokenizer and text.

80. Token Count Is Not Character Count

Likewise:

1,000 characters

does not mean: 1,000 tokens. A tokenizer groups characters into vocabulary units.

The token count is generally smaller for ordinary text, but the exact ratio varies.

81. Why Context Windows Are Advertised in Tokens

Tokens are the actual discrete units consumed by the model. Therefore a model's context capacity is naturally expressed in tokens. For example:

32K context

means approximately:

32,000 token positions

not:

32,000 words

This is the technically correct interpretation.

82. Token Count and Context Quality

Token count is not the only consideration. Suppose two tokenizers represent the same sentence using:

100 tokens

and: 150 tokens. The first is more compact.

But the tokenizer also needs to support useful representations for the model's training data. A smaller token count does not automatically mean a better tokenizer. Tokenization is part of a larger model-design tradeoff.

83. Why Tokenization Cannot Be Perfect

Language is messy. It contains:

• new words
• slang
• names
• misspellings
• technical terms
• code
• URLs
• emoji
• multiple languages
• symbols
• numbers

No finite vocabulary can contain every possible useful string as a single token. Therefore tokenization always involves tradeoffs.

84. The Tokenizer Is Not Learning During Normal Chat

Another important distinction: The tokenizer does not normally update its vocabulary every time you chat. If you type a new word, the tokenizer does not permanently add it to the vocabulary.

It uses its existing rules and vocabulary to decompose it. Model training can create or modify tokenizers. Ordinary inference does not.

85. Tokenization and Model Updates

If a model family changes its tokenizer or vocabulary, that can be a significant architectural change. The embedding matrix must correspond to the vocabulary. The output head must correspond to the vocabulary.

Special-token conventions may change. Chat templates may change. Therefore tokenizer changes are not trivial cosmetic changes.

86. Tokenization and Quantization

Quantization affects the numerical representation of model parameters. Tokenization affects the discrete representation of text. They are different layers.

Tokenization → how text becomes model input

Quantization → how learned numerical values are stored

Do not confuse:

4-bit model

with:

4-bit tokens

A "4-bit model" usually refers to parameter quantization. Tokens are represented as IDs, not as 4-bit model parameters.

87. Token IDs and Integer Precision

The token IDs themselves are integers used by the software. They are not the model's learned weights. They do not need to carry semantic meaning.

The embedding lookup turns the discrete ID into learned numerical values. So:

Token ID

is not the same thing as:

quantized parameter

88. Tokenization and Sampling

Sampling happens after the model produces logits. The tokenizer has already converted the input into token IDs. The model produces a probability distribution over the vocabulary.

Then the generation system chooses a token ID. So:

Tokenizer → input representation

LLM → probability distribution

Sampler → output token selection

Tokenizer → output text

These are separate components.

89. Tokenization and Generation Speed

Suppose the model generates: 20 tokens per second. The tokenizer is not necessarily the main bottleneck during every generation step.

The neural network computation is often much more substantial. However, tokenization still matters for:

• initial prompt processing
• large inputs
• document ingestion
• application overhead

Performance must be measured across the whole pipeline.

90. Prompt Processing and Tokens

When you submit:

10,000-token prompt

the system must process those tokens before it can fully begin generation. This is sometimes described as prompt processing or prefill. Then:

generated token generated token generated token

is the decoding phase. The two phases have different performance characteristics. We will study them in Lesson 11.

91. Tokenization and Latency

Token count can influence:

time to first token

because the model must process the prompt. For example:

100-token prompt

and:

20,000-token prompt

can have very different initial processing workloads. Therefore latency is not simply:

model size

It also depends on:

prompt size + context + hardware + runtime

92. Tokenization and Memory Planning

Suppose you are building a local AI application. You need to plan for:

model weights + context + KV cache + runtime + application

Token count is especially important for the context/KV portion. This means model deployment planning involves both:

parameter memory

and: token/context memory.

93. A Simple Hardware Mental Model

Think of memory as being used by two broad categories:

The model determines the baseline resource requirement. The workload determines additional dynamic requirements. This is a very useful mental model for local AI.

94. Tokenization and Batch Size

Suppose a local server handles multiple requests:

User A: 1,000 tokens User B: 2,000 tokens User C: 5,000 tokens

The system may need to manage the combined workload. Batching can improve throughput but increase memory pressure. Therefore token count becomes a resource-planning variable in multi-user deployments.

This becomes particularly important for business local AI servers.

95. Tokenization and Shared LLM Servers

Imagine a central local AI machine serving: 10 employees. Each request has a different context.

The server may need to manage:

model weights + multiple active contexts + multiple KV caches

The total workload can be much larger than a single-user desktop conversation. This is why local AI infrastructure requires more than simply asking:

"Can the model fit into RAM?"

You also need to ask:

"How many concurrent token workloads will the system support?"

96. Why Loco Needs to Care About Tokens

At the platform level, tokens matter for:

• model information
• context handling
• chat history
• document processing
• knowledge bases
• RAG
• performance
• model selection
• hardware estimation
• user experience

A good local AI platform abstracts away much of this complexity while still giving advanced users enough information to understand what is happening.

97. Practical Exercise 1: Word vs Token

Consider: The quick brown fox jumps over the lazy dog. There are:

9 words

plus punctuation. Can you confidently say it contains exactly: 9 tokens?

No. A tokenizer may produce more or fewer tokens depending on how its vocabulary represents the text. This is the central lesson.

98. Practical Exercise 2: Tokenization Pipeline

Complete the sequence:

Answers:

Tokenizer Embedding lookup

99. Practical Exercise 3: Identify the Difference

Which is a token ID?

"hello"

or:

18273

The second is a token ID in this fictional example. The first is a token text representation.

100. Practical Exercise 4: Parameter vs Token

Which is a learned parameter?

Token ID 152

or:

The embedding value is a learned parameter. The token ID is an index.

101. Practical Exercise 5: Context

Suppose a model has:

16K-token context

and your current context contains:

12K tokens

You cannot assume that another:

10K tokens

can be added without exceeding the context capacity. The available remaining budget is approximately:

4K tokens

subject to how the system reserves space and handles generation.

102. Practical Exercise 6: Long Identifier

Consider:

a9F7xQ2LmP8zR4K6T1

Is it necessarily one token? No. It could be split into multiple tokens because the tokenizer may not have the whole string in its vocabulary.

This is why random identifiers can consume more context than expected.

103. Practical Exercise 7: Why Tokenizer Compatibility Matters

Suppose:

Tokenizer A: "hello" → ID 500

Tokenizer B: "hello" → ID 9000

If the model was trained with Tokenizer A, should you casually use Tokenizer B? No. The model's embedding matrix expects the vocabulary mapping associated with its tokenizer.

104. Knowledge Check

Try to answer these before reading the answers.

Question 1

What is a token?

Question 2

What is a token ID?

Question 3

What is a vocabulary?

Question 4

Why are words not always one token each?

Question 5

What is subword tokenization?

Question 6

What is BPE?

Question 7

Why can punctuation be part of tokenization?

Question 8

Why can numbers be split into several tokens?

Question 9

Why can different models tokenize the same sentence differently?

Question 10

Why are token IDs not semantic values?

Question 11

What does the embedding layer do?

Question 12

Why does vocabulary size affect model parameters?

Question 13

What are special tokens?

Question 14

Why does token count matter for context windows?

Question 15

Why does token count matter for local AI memory?

Question 16

What is the difference between tokenization and quantization?

Question 17

Why can code consume many tokens?

Question 18

Why can different languages use different numbers of tokens for similar text?

Question 19

Why must a tokenizer match the model vocabulary?

Question 20

What happens to generated token IDs before the user sees the final answer?

105. Knowledge Check Answers

Answer 1

A token is a discrete text unit produced by a tokenizer for processing by a language model.

Answer 2

A token ID is the integer identifier assigned to a token within a specific vocabulary.

Answer 3

A vocabulary is the collection of tokens that a tokenizer/model can represent directly.

Answer 4

Because modern tokenizers often use subword units rather than requiring every complete word to exist as a vocabulary entry.

Answer 5

Subword tokenization represents text using pieces that can be combined to represent both common and unfamiliar words.

Answer 6

BPE, or Byte Pair Encoding, is a tokenization approach that builds vocabulary units by repeatedly combining frequently occurring pairs according to its training procedure.

Answer 7

Punctuation is part of the text sequence and may be represented as its own token or combined with neighboring text depending on the tokenizer.

Answer 8

Numbers may not exist as complete vocabulary entries, so a tokenizer can represent them using smaller pieces.

Answer 9

Different models can use different vocabularies and tokenization algorithms.

Answer 10

Token IDs are indices, not semantic measurements. Their numerical magnitude does not represent meaning.

Answer 11

The embedding layer maps token IDs to learned numerical vectors that can be processed by the neural network.

Answer 12

The embedding matrix often has dimensions related to vocabulary size × hidden dimension, so a larger vocabulary can increase the number of learned parameters.

Answer 13

Special tokens are tokens used for structural or control purposes, such as sequence boundaries, roles, padding, or other model-specific functions.

Answer 14

Context windows are measured in tokens, so the number of tokens determines how much text can fit into the model's active context.

Answer 15

More context tokens generally mean more processing and can increase memory use, including KV-cache requirements.

Answer 16

Tokenization describes how text is represented as tokens. Quantization describes how model numerical values are represented with reduced precision.

Answer 17

Code contains identifiers, punctuation, operators, whitespace, strings, and syntax that may be represented by many token units.

Answer 18

Tokenizers learn vocabulary statistics from their training data, and different writing systems and languages contain different patterns.

Answer 19

The model's embedding and output structures are tied to its vocabulary and token-ID mapping.

Answer 20

The generated token IDs are decoded by the tokenizer into human-readable text, which the application displays.

106. A Token Cheat Sheet

107. The Full Token Pipeline Cheat Sheet

Memorize this:

During generation:

This is the bridge between human language and neural-network computation.

108. The Most Important Distinctions

Remember these.

Token ≠ word

A word can be one or multiple tokens.

Token ≠ character

A token can contain multiple characters.

Token ID ≠ meaning

The ID is an index.

Token ≠ parameter

Tokens are input/output units. Parameters are learned numerical values.

Tokenization ≠ quantization

Tokenization concerns text representation. Quantization concerns model numerical representation.

Context ≠ parameters

Context is current input information. Parameters are learned model state.

109. Why Tokens Matter So Much to Local AI

Tokens connect almost every practical part of local inference.

This is why learning tokens is not a minor detail. It is one of the foundations of understanding how an LLM actually works.

110. The Complete Mental Model

You should now be able to picture the entire process:

This is the mechanism you are ultimately interacting with whenever you use a modern generative language model.

111. Final Key Takeaways

1. A token is a discrete unit of text used by a language model.

2. A token ID is the integer identifier associated with a token.

3. A vocabulary is the collection of tokens available to the model.

4. Tokens are not the same thing as words.

5. Tokens are not the same thing as characters.

6. Modern LLMs commonly use subword tokenization.

7. BPE is one important family of tokenization methods.

8. SentencePiece provides another important tokenization framework and can support multiple algorithms.

9. Whitespace and punctuation can affect tokenization.

10. Numbers, URLs, code, names, and unusual identifiers can tokenize in surprising ways.

11. Different languages can require different numbers of tokens for similar amounts of text.

12. Different models can tokenize the same text differently.

13. Token IDs are indices, not semantic measurements.

14. The embedding layer maps token IDs into learned vectors.

15. Vocabulary size influences model architecture and parameter count.

16. Special tokens provide structural or control information.

17. Chat templates can format conversations into model-specific token sequences.

18. Context windows are measured in tokens, not words.

19. More tokens generally mean more computational and memory workload.

20. The KV cache grows with context in autoregressive inference.

21. Token count matters for long documents, RAG, and conversation history.

22. Tokenization is different from quantization.

23. The tokenizer must be compatible with the model's vocabulary and learned representations.

24. Generated token IDs are decoded back into human-readable text.

25. Tokens are the bridge between human language and neural-network computation.

112. Final Mental Model

At the end of Lesson 10, you should be able to picture an LLM like this:

Tokens are the bridge between the world humans interact with and the numerical world the neural network operates in. Once you understand that, terms such as:

128K context 20 tokens/sec 10K input tokens 5K output tokens token budget tokenizer token ID

stop being mysterious. They become measurements and components of a system you can reason about.

113. Preview of Lesson 11

Lesson 11 — How Does a Language Model Generate Text?

We now know:

what an LLM is

and: what a token is. The next question is the natural one:

What actually happens when an LLM generates an answer?

In Lesson 11, we will trace generation from the first character you type all the way to the final sentence appearing on screen.

We will examine:

• the complete inference pipeline
• tokenization
• embeddings
• positional information
• Transformer layers
• attention
• query, key, and value vectors
• attention scores
• causal masking
• feed-forward networks
• residual connections
• normalization
• logits
• softmax
• probability distributions
• greedy decoding
• sampling
• temperature
• top-k
• top-p
• repetition penalties
• stopping criteria
• end-of-sequence tokens
• prompt processing
• token generation
• KV caching
• time to first token
• tokens per second
• why generation is sequential
• why the first token can take longer
• why local hardware affects generation speed
• what llama.cpp is doing conceptually during inference
• how Loco sits around the inference process
• a complete worked example from prompt to generated sentence

By the end of Lesson 11, you should be able to look at a streaming LLM response and understand what the computer is actually doing underneath it.

Sources and Further Reading

1. Attention Is All You Need — Vaswani et al. (2017)
   https://arxiv.org/abs/1706.03762

2. Neural Machine Translation of Rare Words with Subword Units — Sennrich, Haddow & Birch (2016)
   https://arxiv.org/abs/1508.07909

3. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing — Kudo & Richardson (2018)
   https://arxiv.org/abs/1808.06226

4. Byte Pair Encoding — Philip Gage (1994)
   https://www.davidsalomon.name/Compress/gzip/gzip.pdf

5. Hugging Face Tokenizers Documentation
   https://huggingface.co/docs/tokenizers/

6. Hugging Face NLP Course — Tokenizers
   https://huggingface.co/learn/nlp-course/chapter6/1

7. Hugging Face Transformers Documentation
   https://huggingface.co/docs/transformers/

8. Google Machine Learning Glossary
   https://developers.google.com/machine-learning/glossary

9. Deep Learning — Goodfellow, Bengio and Courville
   https://www.deeplearningbook.org/

10. llama.cpp — GitHub
    https://github.com/ggml-org/llama.cpp

11. NIST Artificial Intelligence Resource Center
    https://www.nist.gov/artificial-intelligence

These resources provide deeper technical material on tokenization, subword representations, Transformer language models, and practical model implementation.

Lesson Complete

Completed: Lesson 10 — What Is a Token?

Next: Lesson 11 — How Does a Language Model Generate Text?

Course: Local AI 101

Loco Academy: Lesson Status

Completed: Lesson 10: What Is a Token?

Next: Local AI 101 Completion

Course: Local AI 101