Embeddings Explained: How Vector Search Actually Works

Who this is for: developers and product people who keep hearing "embeddings" in every RAG tutorial but have never actually looked at what one is, why it has 1536 numbers in it, or why their search results are technically correct and practically useless.

Embeddings are the part of the AI stack nobody explains properly, because on the surface it looks simple — text goes in, a list of numbers comes out — and the interesting part is buried in what those numbers mean and how you break them. This guide is embeddings explained the way I wish someone had explained it to me before I shipped a RAG pipeline that returned confidently wrong answers for three weeks.

What an embedding actually is

An embedding is a fixed-length array of floating-point numbers (a vector) that represents the meaning of a piece of text, image, or audio in a way a computer can compare mathematically. A sentence like "the invoice is overdue" and "payment is late" produce vectors that sit close together in that space, even though they share almost no words. That closeness is the entire point — embeddings turn "does this mean the same thing" into "how far apart are these two points."

The model that produces the vector is trained so that semantically similar inputs land near each other and unrelated inputs land far apart. Nothing about the vector is human-readable — you cannot look at dimension 412 and know what it encodes. It's a compressed, learned representation, and that's exactly why it works better than keyword matching for anything involving synonyms, paraphrase, or intent.

Why dimensions matter more than people think

Every embedding model outputs a fixed vector length: OpenAI's text-embedding-3-small gives you 1536 dimensions, text-embedding-3-large gives you 3072 (both support "dimension shortening" via an API parameter down to as low as 256), Cohere's embed-v4 gives 1536, and Voyage AI's voyage-3-large gives 1024 or 2048 depending on the setting. More dimensions generally means better retrieval quality but higher storage cost and slower similarity search at scale — a 3072-dim float32 vector is 12KB per chunk, so a million chunks is 12GB just for vectors, before you add metadata or the index overhead.

Practical rule: don't default to the biggest model. If your corpus is under 50k chunks, text-embedding-3-small at 1536 dims (or even shortened to 512) is usually indistinguishable in quality from the large model and costs a fraction as much to store and query.

Similarity: cosine, dot product, and why it matters which one you pick

Once you have vectors, "similarity" is a distance calculation between two points in that high-dimensional space. The three you'll actually encounter:

• Cosine similarity — measures the angle between two vectors, ignoring magnitude. This is the default almost everywhere and the safest choice if you're not sure.
• Dot product — faster to compute, and mathematically equivalent to cosine similarity IF your vectors are normalized to unit length (most embedding APIs already return normalized vectors, so dot product and cosine give the same ranking).
• Euclidean (L2) distance — measures straight-line distance. Rarely the right pick for text embeddings; more common in image or audio similarity.

If your vector database defaults to L2 and your embedding model returns normalized vectors, you'll get search results that look plausible but are subtly worse-ranked than they should be. Check what your provider assumes — Pinecone, Weaviate, and pgvector all let you pick the metric explicitly, and picking wrong doesn't error out, it just quietly degrades quality.

import numpy as np

def cosine_similarity(a, b):
    a, b = np.array(a), np.array(b)
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

# vectors from an embeddings API, e.g. openai.embeddings.create(...)
score = cosine_similarity(query_vector, chunk_vector)
# scores run roughly -1 to 1; in practice text embeddings cluster 0.1-0.9

Picking a model in 2026

There is no single "best" embedding model — the right pick depends on language, domain, and whether you need to run it locally.

For Hebrew-language content specifically, don't assume the English-optimized small models perform well — Cohere's multilingual embeddings and bge-m3 consistently outperform OpenAI's small model on Hebrew retrieval benchmarks in my own testing, even though OpenAI's is fine for English. Always run a side-by-side eval on 30-50 real queries from your own domain before committing — see eval-driven AI development for how to structure that test.

Chunking: the step that actually decides your search quality

Nobody fails at RAG because they picked the wrong embedding model. They fail because they chunked a 40-page PDF into 10 equal-sized blocks with no regard for where sentences or sections actually end. Chunking strategy affects retrieval quality more than model choice, full stop.

The three chunking strategies that matter

1. Fixed-size with overlap — split every 500-800 tokens with a 10-15% overlap between chunks. Simple, works for prose, breaks on tables and code.
2. Semantic / structure-aware chunking — split on headings, paragraphs, or detected topic shifts. Slower to build, dramatically better for structured documents (contracts, manuals, technical docs).
3. Recursive character splitting (LangChain's default) — tries paragraph breaks first, falls back to sentences, then words. A reasonable middle ground if you don't want to build custom logic.

Warning: chunks that are too small (under ~100 tokens) lose context and produce embeddings that are technically "similar" to a query but semantically empty — you'll retrieve a sentence fragment with no idea what it's actually about. Chunks that are too large (over ~1500 tokens) dilute the vector so a highly relevant sentence gets buried under five paragraphs of unrelated text, and the whole chunk's embedding drifts toward "average."

A pattern that works well for support documentation and internal knowledge bases: chunk by heading section, cap each chunk at ~800 tokens, and always prepend the document title and section heading to the chunk text before embedding it. That one prepend step alone fixed roughly a third of the "wrong document retrieved" errors I've seen in client RAG builds, because the model otherwise has zero context that "step 3" belongs to the "refund policy" document rather than the "shipping policy" one.

Beyond search: what else embeddings are actually used for

Search and RAG get all the attention, but embeddings quietly power several other things worth knowing about:

• Clustering and topic discovery — group thousands of support tickets or reviews by semantic similarity without any labels, using k-means or HDBSCAN on the vectors.
• Deduplication — find near-duplicate documents or leads that differ in wording but say the same thing (useful for CRM cleanup).
• Recommendation — "more like this" for content or products, by embedding item descriptions and finding nearest neighbors to what a user engaged with.
• Anomaly / outlier detection — a support ticket whose embedding sits far from every cluster centroid is often a genuinely novel problem worth a human look.
• Classification without training a classifier — embed a small set of labeled examples per category, then classify new text by nearest-centroid, which is often good enough for triage tasks and needs zero fine-tuning.

Storing and querying vectors at different scales

You don't need a dedicated vector database for every project. Match the tool to your actual scale, described in vector databases compared:

I've watched teams reach for Pinecone for a 2,000-document knowledge base when a single pgvector column on the Postgres they already run would have done the job with one less vendor and one less API key to manage. Don't add infrastructure before you've measured that you need it.

Embeddings vs. fine-tuning: know which problem you actually have

This trips up a lot of teams building their first AI feature. Embeddings solve "find the relevant information" (retrieval). Fine-tuning solves "change how the model behaves or writes" (style, tone, format, domain-specific reasoning patterns). They are not substitutes — see RAG vs fine-tuning for the full breakdown, but the short version: if your problem is "the model doesn't know about our data," you need embeddings and retrieval, not fine-tuning. If your problem is "the model knows the facts but writes in the wrong voice or format," fine-tuning (or just better prompting) is the right lever, not a bigger vector database.

A minimal working example

import OpenAI from "openai";
const client = new OpenAI();

async function embed(text) {
  const res = await client.embeddings.create({
    model: "text-embedding-3-small",
    input: text,
    dimensions: 512, // shortened — cheaper storage, near-identical quality for small corpora
  });
  return res.data[0].embedding;
}

// naive brute-force search over N chunks (fine under ~5k chunks)
function search(queryVec, chunks) {
  return chunks
    .map((c) => ({ ...c, score: cosine(queryVec, c.vector) }))
    .sort((a, b) => b.score - a.score)
    .slice(0, 5);
}

This is enough to prototype a working RAG search in an afternoon. The moment you cross a few thousand chunks or need metadata filtering (by date, by document type, by permission level), move the vectors into pgvector or a dedicated store rather than scaling the brute-force loop.

FAQ

What is an embedding in simple terms?

An embedding is a list of numbers that represents the meaning of a piece of text so a computer can measure how similar two pieces of text are, by measuring the distance between their number lists. Similar meanings produce vectors that sit close together, even when the actual words used are completely different.

How many dimensions should an embedding have?

Most production systems use 512-1536 dimensions; 3072 only helps meaningfully on large, ambiguous corpora, and 256 or fewer works surprisingly well for small, focused datasets. Test on your own data before assuming bigger is better, since higher dimensions cost more to store and query without a guaranteed quality gain.

Do I need a vector database to use embeddings?

No. Under roughly 50,000 vectors, a Postgres table with the pgvector extension or even an in-memory array with brute-force cosine similarity works fine and adds zero new infrastructure. Reach for Pinecone, Qdrant, or Weaviate only once you're past that scale or need heavy metadata filtering.

Why does my RAG search return the wrong answer even though embeddings match?

The most common cause is bad chunking — chunks that are too large dilute the vector, chunks that are too small lose context, and chunks without their source heading prepended lose the surrounding document identity. Fix chunking before you touch the embedding model; it's the higher-leverage problem in nearly every case I've debugged.

Are embeddings the same as what powers ChatGPT's answers?

No. Embeddings are a separate, much smaller model used for search and comparison; the large language model that generates the actual answer text is a different model entirely. In a RAG system, embeddings retrieve the relevant chunks and the LLM then reads those chunks and writes the response.

Get this built for you

If you're staring at a pile of PDFs, support tickets, or product docs and want an actual working search or assistant on top of them rather than another afternoon lost to chunking experiments, book a free 30-minute call through the contact form or message me directly on WhatsApp.

Sources

• OpenAI Embeddings guide
• Cohere Embed documentation
• pgvector GitHub repository
• Pinecone documentation
• Voyage AI documentation