Embeddings — How Machines Map Language to Meaning

This is Blog 5 in the RAG series.

The Plain-English Idea

An embedding is a function that turns a piece of text into a fixed-length list of numbers — a vector — say 1024 numbers. The magic property, and the entire reason RAG works:

Texts with similar meaning produce vectors that are close together in space, and unrelated texts produce vectors that are far apart.

"How do I reset my password?" and "I forgot my login credentials" land near each other even though they share almost no words. This is the leap beyond old-school keyword search. Keyword search matches strings; embeddings match meaning.

How an Embedding Is Created

In practice you don't build an embedding model — you call one. But you must understand the mechanism to make good decisions.

The model is a transformer (same family as LLMs, smaller and purpose-built). Text flows through it:

Text input
    ↓
1. Tokenization — split into sub-word tokens
   "Embeddings" → ["Embed", "##dings"]

2. Contextual encoding — transformer layers where
   attention lets every token absorb context from others
   "bank" in "river bank" vs "bank account" → different vectors

3. Per-token vectors — one contextualized vector per token

4. Pooling — collapse N token vectors into 1 chunk vector

Pooling — The Crucial, Often-Skipped Detail

You have N token vectors and need 1. The collapse is pooling, done one of a few ways:

| Method | How | Used by | |--------|-----|---------| | Mean pooling | Average all token vectors (masking padding) | BGE, E5, Sentence-Transformers | | CLS pooling | Use the final vector of a special [CLS] token | BERT-family models | | Last-token pooling | Use the last token's vector | Qwen3, NV-Embed (decoder-based) |

Why pooling matters: pooling is lossy compression — you squash N vectors into 1. A long, multi-topic chunk gets averaged into mush; distinct ideas blend into a centroid representing none of them. This is the mechanical reason oversized chunks retrieve poorly (Blog 4) and the enemy that ColBERT attacks by not pooling (Blog 11).

Why Similar Meanings Cluster — Contrastive Training

The clustering property isn't automatic — it's trained in via contrastive learning:

Positive pairs (question + correct answer, paraphrase) → pulled together in vector space
Negative pairs (unrelated chunks) → pushed apart

Across billions of pairs, the model learns a geometry where semantic similarity = spatial closeness.

The 2026 Embedding Model Landscape

⚠️ This field churns monthly. Verify before relying on specific numbers.

Commercial / API

| Model | Strengths | Notes | |-------|-----------|-------| | Google Gemini Embedding | Top English MTEB v2 (~68.32), first truly multimodal: text + images + video + audio + PDFs in one space | Ties you to Google infra | | Voyage AI (voyage-3-large / voyage-4) | Retrieval-specialist; leads retrieval-focused metrics | voyage-4 uses MoE, ~40% cost reduction | | OpenAI text-embedding-3 | Safe default, widely deployed, Matryoshka support | 3-large: strong but aging; 3-small: best value ($0.02/1M tokens) | | Cohere embed-v4 | Multimodal (text + images + PDFs); separate query vs doc models in one space | Strong for document-heavy corpora | | Jina-embeddings-v3 | Best price-performance API (~$0.02/1M) | Scores within 2 points of expensive models |

Open-Source / Self-Hostable

| Model | Strengths | Notes | |-------|-----------|-------| | Qwen3-Embedding-8B (Alibaba, Apache 2.0) | #1 on multilingual MTEB (~70.58); 32K context | 8B params, VRAM-heavy; also 4B and 0.6B variants | | BGE-M3 | 100+ languages; dense + sparse + multi-vector in one model | Mature multilingual workhorse | | NVIDIA NV-Embed-v2 | Top raw-quality benchmarks | Open-weight | | Microsoft Harrier-OSS | Very high MTEB v2 (~74.3 for 27B) | MIT-licensed |

Critical caveat on MTEB benchmarks: scores are self-reported by providers with no independent verification, and MTEB only tests single-language text retrieval. The decisive factor is how a model performs on your corpus — run evals with MRR and NDCG on your own data.

Using Embeddings in Code

# OpenAI embeddings
from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vector = embeddings.embed_query("How do I reset my password?")
print(len(vector))  # 1536 dimensions

# Multiple documents
doc_vectors = embeddings.embed_documents([
    "The refund policy allows returns within 30 days.",
    "For damaged electronics, contact support@company.com",
])

# Voyage AI (retrieval-optimized)
from langchain_voyageai import VoyageAIEmbeddings

embeddings = VoyageAIEmbeddings(
    model="voyage-3-large",
    voyage_api_key="your-key",
)

# Note: Voyage has separate query and document embeddings
query_vec = embeddings.embed_query("damaged electronics refund")
doc_vecs = embeddings.embed_documents(["your", "document", "chunks"])

# Open-source / self-hosted (no API costs)
from langchain_huggingface import HuggingFaceEmbeddings

embeddings = HuggingFaceEmbeddings(
    model_name="BAAI/bge-m3",
    model_kwargs={"device": "cuda"},
)

The Decision Axes

1. Hosted vs Self-Hosted

Usually the first fork, driven by data residency and cost at scale — not quality.

Data that legally can't leave the customer's VPC forces self-hosting (Qwen3-Embedding, BGE-M3) regardless of what tops the leaderboard.

2. Dimensionality → Storage Cost

# Storage math: 4KB per 1024-dim float32 vector
dim = 1024
n_docs = 10_000_000  # 10M documents
storage_gb = (dim * 4 * n_docs) / (1024**3)
print(f"Storage: {storage_gb:.1f} GB")  # ~38 GB

The escape hatch: Matryoshka embeddings (OpenAI, Cohere, others) — truncate a 1536-dim vector to 512 with graceful quality loss, trading a little accuracy for ~3× storage savings.

3. Domain Fit Beats Raw MTEB

A general model on dense legal/medical/code text mis-clusters. Fine-tuning on in-domain pairs shows +10–30% retrieval gains in specialized domains.

4. Multilingual

English-only leaderboard scores mislead for multilingual corpora. Use BGE-M3 or Qwen3-Embedding for global data.

5. Multimodal

If the corpus has images/charts/PDFs you want directly retrievable: Google Gemini Embedding or Cohere embed-v4.

Fine-Tuning Embeddings

When generic performance is insufficient:

from sentence_transformers import SentenceTransformer, InputExample, losses
from torch.utils.data import DataLoader

# Load base model
model = SentenceTransformer("BAAI/bge-m3")

# Training data: (query, positive_passage) pairs
train_examples = [
    InputExample(texts=["what is the refund window?", "Returns are accepted within 30 days of purchase."]),
    InputExample(texts=["damaged goods policy", "Damaged items must be reported within 48 hours."]),
    # ... thousands more
]

train_dataloader = DataLoader(train_examples, shuffle=True, batch_size=16)
train_loss = losses.MultipleNegativesRankingLoss(model)

model.fit(
    train_objectives=[(train_dataloader, train_loss)],
    epochs=3,
    warmup_steps=100,
)
model.save("my-fine-tuned-embedder")

The tradeoff: you need labeled in-domain pairs (or synthetically generated ones) and an eval set to prove the gain. Start with a strong general model, measure on the customer's corpus, and fine-tune only if eval shows the gap justifies it.

The Embeddings Failure Catalog

Model mismatch between index and query — different models = incompatible spaces = garbage retrieval. Same model both sides, always.
Domain mismatch — general model on specialized text mis-clusters; rare-but-critical terms get diluted.
Chasing the leaderboard — picking the top MTEB model instead of testing on your data; MTEB is self-reported and narrow.
Exceeding max input length — chunks longer than the model's limit get silently truncated.
Dimensionality blowout — high-dim vectors at scale = huge storage/latency cost; ignoring Matryoshka.
Pooling blur on big chunks — oversized chunks average into meaningless centroids.
Forgetting query-time cost — embedding traffic scales with query volume forever, not just at indexing.
Ignoring multilingual/multimodal needs — English-text model on a multilingual or image-heavy corpus.

The Key Principle

Don't anchor on the leaderboard — anchor on evaluation.

Benchmarks reshuffle with every release and are self-reported. The durable investment is an eval pipeline on the customer's own data — define their data types, query patterns, doc lengths, and test new models as they drop. Public benchmarks are a hypothesis, not a conclusion.

Summary

| Concept | Key Point | |---------|-----------| | Embedding | Text → vector; similar meaning → close vectors. Beats keyword search. | | Mechanism | Tokenize → transformer → pool (mean/CLS/last-token). Pooling is lossy. | | Training | Contrastive learning — pull positive pairs, push negatives. | | Pick by | Hosted-vs-self-hosted (residency/cost) first, then domain fit, multilingual, multimodal, cost. | | Matryoshka | Truncate high-dim vectors to save storage without huge quality loss. | | Fine-tuning | +10–30% in specialized domains. Only if eval justifies it. |

Next: Blog 6 — Vector Stores & Similarity Search

Resources:

Blog 5: Embeddings — How Machines Map Language to Meaning