Key Takeaway
An LLM token is the basic unit a model uses to process text, and the context window is the maximum number of tokens the model can handle within a single request.
This post is part of my AI study notes series as I prepare for the SK hynix AI Hackathon preliminary round (Oct 17–18). Following the last post on sampling (temperature and top-p), today’s post covers tokens — the most basic unit a model uses to handle text — and the context window, which decides how many of those tokens it can hold at once.
What Is a Token? How Is It Different from Words and Characters?
One-line answer: A token is the unit an LLM uses to process text; a character, a piece of a word, a whole word, or punctuation can all be a token.
- Text is first split into token pieces, and each piece is converted into a numeric token ID. The next step after token IDs, embedding, is covered in the Day1 Self-Attention post.
- As a rough estimate for English, 1 token ≈ 4 characters ≈ 3/4 of a word, and 100 tokens ≈ about 75 words. This is an estimate, not an exact count, and the relationship can be different for other languages.
- Spaces and capitalization also affect how text is split. For example,
red,Red, andredwith a leading space are each treated as different tokens. - Token count is not the same as character count or word count. The same text can produce a different token count depending on the model, encoding, and language.
How Does a Tokenizer (BPE) Split Sentences? Why Can Korean Produce More Tokens?
One-line answer: Frequently occurring character combinations are grouped into a single token, while rarer words are split into multiple pieces.
- The intuition behind subword tokenization is this: frequently occurring character combinations become one token, while rarer words are split into several smaller pieces.
- The intuition behind BPE (Byte Pair Encoding) is this: character pairs that frequently appear together are repeatedly merged to build up a vocabulary.
- Depending on the training data and vocabulary composition, Korean text can end up with more tokens than English for the same meaning. That said, the exact ratio or multiplier varies by model and encoding, so it can’t be stated as a fixed number.

- Comparing this with tiktoken, under the o200k_base encoding, “I like reading books.” comes out to 5 tokens, while “나는 책 읽는 것을 좋아합니다.” comes out to 8 tokens. Below is the code used for this comparison and the actual output.
# Tokenize an English and a Korean sentence with the same meaning, plus space/case examples, with tiktoken.
import importlib.metadata as md
import tiktoken
TEXTS = [
"I like reading books.",
"나는 책 읽는 것을 좋아합니다.",
"red",
"Red",
" red",
]
def piece(enc, tid):
b = enc.decode_single_token_bytes(tid)
try:
return repr(b.decode("utf-8"))
except UnicodeDecodeError:
return f"<bytes {b.hex(' ')}>" # a byte fragment cut in the middle of a character
print("tiktoken", md.version("tiktoken"))
for name in ("o200k_base", "cl100k_base"):
enc = tiktoken.get_encoding(name)
print(f"\n== {name}")
for t in TEXTS:
ids = enc.encode(t)
print(f"{t!r}: chars={len(t)} tokens={len(ids)}")
print(" ids :", ids)
print(" pieces:", " | ".join(piece(enc, i) for i in ids))
tiktoken 0.14.0
== o200k_base
'I like reading books.': chars=21 tokens=5
ids : [40, 1299, 6085, 7187, 13]
pieces: 'I' | ' like' | ' reading' | ' books' | '.'
'나는 책 읽는 것을 좋아합니다.': chars=17 tokens=8
ids : [102052, 67671, 93243, 2770, 61738, 90032, 24490, 13]
pieces: '나는' | ' 책' | ' 읽' | '는' | ' 것을' | ' 좋아' | '합니다' | '.'
'red': chars=3 tokens=1
ids : [1291]
pieces: 'red'
'Red': chars=3 tokens=1
ids : [7805]
pieces: 'Red'
' red': chars=4 tokens=1
ids : [3592]
pieces: ' red'
== cl100k_base
'I like reading books.': chars=21 tokens=5
ids : [40, 1093, 5403, 6603, 13]
pieces: 'I' | ' like' | ' reading' | ' books' | '.'
'나는 책 읽는 것을 좋아합니다.': chars=17 tokens=15
ids : [61415, 16969, 3396, 109, 227, 17169, 121, 16969, 72208, 18359, 66799, 233, 54059, 61938, 13]
pieces: '나' | '는' | <bytes 20 ec> | <bytes b1> | <bytes 85> | <bytes 20 ec 9d> | <bytes bd> | '는' | ' 것' | '을' | <bytes 20 ec a2> | <bytes 8b> | '아' | '합니다' | '.'
'red': chars=3 tokens=1
ids : [1171]
pieces: 'red'
'Red': chars=3 tokens=1
ids : [6161]
pieces: 'Red'
' red': chars=4 tokens=1
ids : [2579]
pieces: ' red'
The same sentence “나는 책 읽는 것을 좋아합니다.” comes out differently depending on the encoding: 8 tokens under o200k_base versus 15 tokens under cl100k_base. That’s because cl100k_base splits some of it down into raw UTF-8 byte fragments that make up a single Hangul character. These numbers aren’t a generalized ratio — they’re just an example from this one sentence run. For reference, tiktoken.encoding_for_model("gpt-4o") returns o200k_base, and tiktoken.encoding_for_model("gpt-4") returns cl100k_base.
What Is a Context Window? When Do Input/Output Token Limits Become an Issue?
One-line answer: It’s the total limit on tokens a model can see within a single request; the output limit is set separately.

- Feeding in an entire long document, or letting conversation history keep piling up, can exceed the context window limit. For reasoning models, note that hidden reasoning tokens — not visible on screen — also take up space on the output side.
- Ways to deal with hitting the limit: shorten or rephrase the content, remove unnecessary or repeated context, split it into multiple parts, or summarize/preprocess before sending it (source: OpenAI Help: Understanding and counting tokens).
- How the system prompt and conversation history get placed within the context window is covered in the Day4 prompt engineering post.
- The fact that memory burden, such as the KV Cache, also grows as a conversation gets longer is covered in the Day3 KV Cache post.
[Exam one-liner] Token = the unit of text processing (not a word or character); context window = the processing limit for a single request. If it overflows, shorten, remove repetition, split, or summarize — and remember that invisible reasoning tokens take up space too.
Shall We Test What We’ve Learned With a Quiz?
Q1. Do 100 English words and 100 Korean words always become the same number of tokens?
A1. Not necessarily. Token counts can vary with the language and the model’s encoding.
Q2. Do red, Red, and red all have the same token ID?
A2. No. In the run above, each is a single token, but all three IDs differ (1291, 7805, and 3592 under o200k_base).
Q3. With a reasoning model, why can output token usage be higher than the short visible answer suggests?
A3. Reasoning tokens that are not shown as answer text still count toward output usage (per the OpenAI Help article).