Key takeaway

An embedding turns data such as words into numeric vectors that carry meaning. Once words are vectors, you can calculate how similar their meanings are from distance or angle, and the standard way to compare by angle is cosine similarity.

These are my AI study notes while preparing for Round 1 of the SK hynix AI Hackathon (Oct 17–18). Day5–7 summed up Embedding in one line as part of the RAG flow; this post focuses on how embeddings themselves let us compare meaning.

What is an embedding, and why turn words into lists of numbers (vectors)?

Short answer: it turns tokens (words) into numeric vectors that carry meaning, so a computer can calculate whether meanings are similar.

  1. The flow is text → tokens → token IDs → embedding vectors. How text becomes tokens and token IDs is covered in the tokens and context window post, and how embeddings are used inside the model is covered in Day1 Self-Attention.
  2. Google’s ML Crash Course describes an embedding as “a vector representation of data in embedding space.” Each item is a point in n-dimensional space made of n floating-point numbers, and the distance between two points can be interpreted as relative similarity.
  3. Think of it as coordinates on a map: words with similar meanings sit close together, and unrelated words sit far apart.
  4. People don’t type these coordinates in. While a neural network trains, its embedding layer learns values that represent the data well.

Simplified 2D embedding space: king, queen, man, woman in one cluster, apple, banana, grape in another, car far away on its own. Close means similar meaning, far means less related

One-hot encoding vs embeddings: why can’t one-hot compare meaning?

Short answer: any two different words’ one-hot vectors are always the same distance apart (perpendicular), so they carry no information about similarity; embeddings are learned so that similar words end up close.

  1. One-hot encoding gives each word (category) a vector with a single 1 and all other entries 0. Its length grows with the number of words, and most of its values are 0, making it a sparse vector.
  2. In one-hot form, “king” and “queen” are just as perpendicular as “king” and “car”, so the cosine similarity is 0 in both cases. You can see this in the code output below.
  3. An embedding is a shorter, dense vector than a one-hot vector, and training pulls similar words closer together.
AspectOne-hot encodingEmbedding
LengthAs long as the vocabularyMuch shorter
Share of zerosAll 0 except one entry (sparse)Mostly non-zero values (dense)
SimilarityNone (all different words are equally far apart)Yes (similar words are close)
Learned?Fixed by a ruleLearned in training

A famous example: the word2vec paper (Mikolov et al., 2013) reports the observation that vector(“King”) − vector(“Man”) + vector(“Woman”) gives a vector closest to “Queen”. It’s an example to build intuition, though; this kind of arithmetic doesn’t always hold exactly.

What is cosine similarity, and how do vectors measure “close in meaning”?

Short answer: it compares two vectors by the angle between them: the closer their directions, the closer to 1, and 0 when they are perpendicular.

  1. The formula is the dot product of the two vectors divided by the product of their lengths. It ignores length and looks only at direction (angle). Google’s similarity measures page also says cosine focuses only on the angle between vectors.

Cosine similarity compares direction: A(3,4) and B(4,3) form a small angle with cos 0.96, A and C(-4,3) are perpendicular with cos 0 (illustrative values)

  1. A hand-calculation example with made-up vectors: for A = (3, 4) and B = (4, 3), the dot product is 3×4 + 4×3 = 24 and both lengths are 5, so the cosine similarity is 24 / 25 = 0.96. C = (−4, 3) is perpendicular to A, so the dot product is 0 and the cosine similarity is 0.
  2. Cosine similarity between embeddings is used for finding similar sentences, classification and recommendations. For retrieving documents and feeding them to an LLM, see the RAG part of Day5–7.
# Cosine similarity on made-up illustrative vectors (not real model values)
import math

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))

A = (3, 4)   # e.g. "king"
B = (4, 3)   # e.g. "queen" (similar direction)
C = (-4, 3)  # e.g. "car" (perpendicular to A)
print("cos(A, B) =", round(cosine(A, B), 2))
print("cos(A, C) =", round(cosine(A, C), 2))

# One-hot: any two different words always give 0
king  = (1, 0, 0, 0)
queen = (0, 1, 0, 0)
car   = (0, 0, 1, 0)
print("one-hot cos(king, queen) =", cosine(king, queen))
print("one-hot cos(king, car)   =", cosine(king, car))
cos(A, B) = 0.96
cos(A, C) = 0.0
one-hot cos(king, queen) = 0.0
one-hot cos(king, car)   = 0.0

The made-up vectors give 0.96 and 0.0, matching the hand calculation, and the one-hot vectors give 0.0 for any pair of words.

[Exam one-liner] Embedding = turning words into dense vectors that carry meaning (learned). One-hot has no similarity information. Cosine similarity = comparing the directions of two vectors (same direction 1, perpendicular 0).

Shall we check what we learned with a quiz?

Q1. With one-hot encoding, what is the cosine similarity of “king” and “queen”?

A1. 0. One-hot vectors for different words are always perpendicular, so there is no similarity information even when the meanings are close.

Q2. Do people set the numbers in an embedding vector by hand?

A2. No. The embedding layer learns the values itself as the model trains.

Q3. What does it mean when two vectors’ cosine similarity is close to 1?

A3. Their directions are almost the same. For embeddings, that can be read as similar meaning.

Q4. Does “king − man + woman ≈ queen” always hold exactly for every embedding?

A4. No. It’s an intuition-building example observed in word2vec research and doesn’t always hold exactly.