Build Your First LLM from ScratchPart 3 · Section 9 of 13

What's Inside?

The embedding layer is just a lookup table of random numbers that get adjusted during training:

# Inside the embedding layer (simplified)
#        dim0   dim1   dim2  ... dim63
# ID 0: [0.23, -0.45, 0.12, ..., 0.67]  ← "[PAD]"
# ID 1: [0.89, 0.34, -0.56, ..., 0.23]  ← "[START]"
# ID 2: [0.12, 0.78, 0.45, ..., -0.34]  ← "[END]"
# ID 3: [0.45, -0.23, 0.89, ..., 0.12]  ← "zero"
# ID 4: [0.67, 0.12, -0.45, ..., 0.56]  ← "one"
# ID 5: [0.34, 0.56, 0.23, ..., -0.78]  ← "two"
# ...

Helpful?

The Embedding Layer At Scale