I Swapped Embedding Models and Half My Search Results Changed

Planning an embedding model change?

I agree to the Privacy Policy Review Your Retrieval Setup

Switching embedding models is one string in a config file. On this site it moved half of every top-five result, and forgetting to rebuild the index made search no better than random, without a single error.

Every few months a dependency gets a new version. We read the changelog, bump the number, run the tests and move on. Newer is usually better, and when it isn’t, something fails loudly enough to tell us.

Embedding models look like the same kind of dependency. An embedding model turns a piece of text into a list of numbers, a vector, so that texts with similar meaning get similar numbers. Search then becomes arithmetic: turn the question into a vector, find the stored vectors closest to it, and return their text. For how a vector index finds the closest vectors without comparing against all of them, read Before the Model Answers: Retrieval, Reranking, and the Kraska Thesis.

In January 2024 OpenAI released text-embedding-3-small as the successor to text-embedding-ada-002.¹ It is five times cheaper, it scores higher on the standard retrieval benchmarks, and it returns vectors of the same length, 1,536 numbers. Switching is one string in a config file.

So we say the obvious thing: a newer embedding model is an upgrade. Swap the name and search gets a little better. For almost every other dependency we own, that is how it works.

What the config file doesn’t say

A library runs when we call it and keeps nothing afterwards. An embedding model is different, because its output is stored. Every vector in a search index was written by the old model, maybe months ago, and it stays there until someone rewrites it. So changing the string raises a question a normal upgrade never does: what happens to everything the old model already wrote?

I tried it on the one corpus I know well, this site. It has 36 published articles, which I cut into 330 sections at their ## headings.

Then we need questions. Real user queries would be best, but they come without answers, and almost nobody labels the right answers for their own search. The headings get around that. Each heading is a short question whose right answer we already know: the right result for “Why the count becomes a weight” is the section under that heading. Keep only the headings that appear once across the site, which drops “Closing” and “References”, and we have 265 queries, each with exactly one correct section.

That gives us two questions. How many results change when we switch models? And are the new results better or worse?

Half the results move

If the swap is an upgrade, most results should stay where they were and a few should get better. We embedded every section and every query with both models, searched, and compared each query’s top five.

The two models agreed on 47.6% of them. Only 8 of the 265 queries came back with the same five sections, and 18 had none in common. Even when both models put the right section first, which happened 76 times, the other four places shared only about half their sections.

For a typical search, half of what a reader sees is different, and all we changed was one string.

Then there is the cheaper mistake. We change the string and forget to re-embed the stored sections, so queries now come from 3-small and the index is still ada-002. Both produce 1,536 numbers, so the arithmetic runs and nothing throws an error. The right section landed in the top five for 1.1% of queries. Picking five sections at random gets 1.5%. One query, “What the inventory actually buys”, found its section at rank 1 with the new model, rank 15 with the old one, and rank 202 of 330 when the two were mixed.

We cannot accept that. A change everyone calls an upgrade, done the way that word suggests, made search no better than drawing sections out of a hat, and no log line said so.

The word was carrying an assumption: that a vector means something on its own, so a new vector and an old vector are the same kind of thing. They aren’t. Each model builds its own space, and the numbers in a vector only mean something to the model that wrote them. Same length, different map.

Key takeaway

Photo by Kelly Sikkema on Unsplash.

A vector belongs to the model that wrote it

Ada-002 and 3-small both return 1,536 numbers per text, so nothing stops a 3-small query from being compared with ada-002 vectors. Like a key in the wrong lock, it fits and turns nothing: 1.1% recall@5, against 1.5% for random picks.

The model's name belongs in the index, next to every vector it wrote.

Seen that way, switching embedding models changes three things: the code that embeds new queries, every vector already stored, and the scale of the similarity scores. An upgrade only touches the first. A database migration touches all three, and that is what this is.

Was the new model better?

Overlap tells us how much changed. It doesn’t tell us which side was right, and 3-small may well be better. The headings let us check, with two numbers.

Recall@5 is how often the correct section lands in the top five. MRR, mean reciprocal rank, scores each query as one over the rank of the correct section and averages: rank 1 scores 1, rank 2 scores 0.5, rank 10 scores 0.1.

One trap first. If each section still starts with its heading, every query is the first line of its own answer. Recall jumps to 91% for ada-002 and 97% for 3-small, and we are measuring string matching. So the headings come out of the section text before anything is embedded.

Here is the measurement. Everything else in the script is reading the articles and calling the embeddings API.

function cosine(a, b) {
	let dot = 0, na = 0, nb = 0;
	for (let i = 0; i < a.length; i++) {
		dot += a[i] * b[i];
		na += a[i] * a[i];
		nb += b[i] * b[i];
	}
	return dot / (Math.sqrt(na) * Math.sqrt(nb));
}

// Every section index, best match first.
function ranking(queryVec, sectionVecs) {
	return sectionVecs
		.map((v, i) => ({ i, score: cosine(queryVec, v) }))
		.sort((a, b) => b.score - a.score)
		.map((s) => s.i);
}

// rankings[q] is one model's ranking for query q; answers[q] is its own section.
function quality(rankings, answers) {
	let hits = 0, rr = 0;
	rankings.forEach((r, q) => {
		const rank = r.indexOf(answers[q]) + 1;
		if (rank <= 5) hits++;
		rr += 1 / rank;
	});
	return { recallAt5: hits / rankings.length, mrr: rr / rankings.length };
}

// Share of the top five two rankings have in common, 0 to 1.
function overlap(a, b) {
	const top = new Set(a.slice(0, 5));
	return b.slice(0, 5).filter((i) => top.has(i)).length / 5;
}

Provenance: text-embedding-ada-002 and text-embedding-3-small, OpenAI embeddings API, 36 articles, 330 sections, 265 queries, Node v25.2.1, run on 13 October 2026. About two cents at list prices.

The new model is better. Count only the queries where the two models disagree about whether the right section made the top five: 3-small wins 34 of them and ada-002 wins 13. If the two models were equally good, a split that lopsided would turn up about 3 times in 1,000 (exact binomial test, p ≈ 0.003).

So the upgrade was real, but only once every stored vector had been rewritten, and it moved half the results to get there.

The scores moved too. Under ada-002, the best match for a query scored between 0.73 and 0.89. Under 3-small it scored between 0.24 and 0.69. Plenty of search code drops weak matches with a fixed cut-off, and 0.7 is a sensible-looking one for ada-002: it keeps all 265 of its top results. On 3-small it keeps none. The search returns nothing, and again, nothing reports an error.

Every one of them is right. A cut-off written on one is wrong on the others. Photo by William Warby on Unsplash.

What the headings can’t tell us

The headings made the check free, but they are short, clean and written by the person who wrote the answer. Both models missed 81 of the 265 queries, and most of those were generic: “Why this matters in production” came back at rank 233 under ada-002 and 108 under 3-small. Real users ask longer, vaguer, misspelled questions, and nothing here says the gap between the models survives them.

One correct answer per query also undercounts. A section from another article that answers the same question is scored as a miss. So recall here is biased low, and we can’t say it is biased by the same amount for both models.

Overlap has its own blind spot. It ignores order, so a section that falls from first to fifth counts as unchanged. A rank-aware version, rank-biased overlap, would catch that.²

And the corpus is small. 330 sections from one author share one vocabulary. A hundred thousand support documents written by twenty people will behave differently, and the only way to know how is to run the same check on them.

This check can become the next thing nobody questions. “Recall went up on the headings, ship it” is the same mistake as “it’s newer, ship it”, one step later. A model that wins on headings and loses on real questions would pass.

Before the string changes

The config file still holds one string. It names the model that wrote every vector in the index, so changing it is a migration and gets the treatment a schema change gets. Store the model name next to every vector, the way a container image is pinned by its digest rather than a tag that can move. For that idea on container images, read The Digest Is the Container. The Tag Is Just a Rumor. Build the new index beside the old one instead of writing over it. Run both on the same queries before switching: the overlap at least, and recall wherever we can find questions with known answers, like headings, FAQ titles or old support tickets with the document that resolved them. Check every score cut-off against the new model’s scores. Then switch reads in one step and keep the old index until nobody needs it.

In this run the new model was better. We only know because both indexes existed at the same time and we compared them.

References

Planning an embedding model change?

I agree to the Privacy Policy Review Your Retrieval Setup