a number needs a neighbour

five times I measured something that refused to hold still, and the only thing that saved me was measuring something else beside it

1. Something as powerful as an embedding model reaches you through two providers, or two modes of transport, and the answers come back differing in the fourth decimal place. How do you measure the impact of that? How do you even tell whether the thing itself has changed?
2. Disagreement with the self. Identical text, identical endpoint, identical subject matter, and the output still comes back different. Non-determinism turns up far more often than I expected.
3. A cosine of 0.9998 sounds excellent and means nothing at all until you know what a wrong pairing scores.
4. I was working at cent scale, eighty or ninety cents a batch, and my own token estimate still missed the invoice. Once by 1.6%, once by something like 7 to 9%. Probably I need more practice.
5. Two models end up as each other’s control group and measurement standard, because there is no absolute scale to put either one on.

Last time I wrote about one number that was secretly two costs stuck together. This is the opposite problem, and I think it happens more often: a number that is secretly nothing, because there is nothing anywhere near it to compare it against.

the shape of it

A measurement is a comparison. It always was. We just hide the second half of it when it feels obvious. And when the thing you are measuring will not repeat itself, the second half stops being obvious. If you do not go and build one deliberately, you end up interpreting noise with a straight face.

 a bare number          a number with a neighbour
 ─────────────          ────────────────────────

   0.99985                0.99985   matched pair
      ?                   0.99972   same endpoint, twice
      │                   ───────
      │                   the gap is smaller than the
      └── "close to 1,    endpoint's own jitter, so
           looks fine"    there is nothing here

the endpoint that would not repeat itself

I needed the same embedding model through two different transports. One a managed cloud endpoint, the other the vendor’s own API directly. Same model name, same text.

Mean cosine between the two came out at 0.99984832. Which landed, annoyingly, just under a threshold I had written down before running anything: at or above 0.9999 means the same vector space, 0.99 to 0.9999 means drifting and needs a decision, below 0.99 means a different space altogether. So my own rubric said “drifting.” Something to worry about.

The tempting thing to do here is start theorising about transports. What I did instead was ask a question I could actually answer: is this difference bigger than the difference between two calls to the same endpoint?

The cross-transport difference turned out to be no larger than the endpoint’s disagreement with itself. Worst cross-transport over worst same-endpoint came to 0.99. So there was no transport effect at all, only jitter, and I had very nearly written a paragraph explaining it.

The control threw up its own surprise while it was at it. Calling one endpoint repeatedly with byte-identical text gave 31 of 32 vectors bit-identical, and one that was not, differing by up to 2.891e-03 in a single component. So the endpoint is not deterministic. Probably batch composition, kernel selection, floating point accumulation order. I did not chase it. But “identical input, identical output” was an assumption I had been making for free, and it turned out to be false.

what does a wrong answer score?

Fine, so the transports agree. Does the difference matter for retrieval, which was the only reason I cared in the first place? That needs a different kind of neighbour, a mismatch I create on purpose rather than a repeat.

Take every text embedded both ways and build the full grid of cosines between one set and the other.

          stored vectors
          b0   b1   b2   b3
    c0  [  X    ·    ·    ·  ]    X  diagonal  : the signal
    c1  [  ·    X    ·    ·  ]    ·  off-diag  : the control
    c2  [  ·    ·    X    ·  ]
    c3  [  ·    ·    ·    X  ]

  derangement    one · per row, shifted by one   (cheap sample)
  all off-diag   every ·                          (full distribution)
  self-retrieval rank    sort a row, ask where X landed

Here is what came back, over 32 texts and every one of the 992 wrong pairings:

So a typical wrong pairing scores 0.24, and the very best wrong pairing in the whole grid reaches 0.62. The matched pairs sit at 0.9998. The gap between the worst right answer and the best wrong one is 0.38. Enormous, and nothing like the 1.5e-04 I had been worrying about.

Then the rank, which is the part that actually decided it: for all 32 of 32 texts, the nearest stored vector was its own. Put the two side by side and the difference I was fretting over is roughly five thousand times smaller than the distance to an unrelated sentence.

A difference that cannot change an outcome does not need to be treated as a difference. I have since run into the same idea somewhere completely unrelated. A reranker that only reorders cannot change a concatenated context string, so the reordering is free and only pruning does anything. Same shape, different week. It took someone else pointing it out before I noticed the two were one idea.

the same text, asked two different ways

Some embedding models take a parameter telling them what role the text is playing, either a query or a document to be searched. It is easy to treat that as a formality and pass whatever. I had been passing the wrong one for a while, so I wanted to know what it had cost me.

Ten questions, embedded twice. Once in the query role, once in the document role I had been using by mistake. For each one, the cosine against the correct evidence and against a distractor. I wrote the prediction down before running it, which matters more than it sounds like it does, because it stops you discovering afterwards that you expected whatever happened.

Read the first column and the wrong role looks better, with a higher similarity to the right answer. Read the second and there is nothing there at all: the margins differ by 0.0012, and the query role won on 5 of 10 questions, which is a coin toss.

So the prediction was wrong, and the honest write-up is that nothing was established. The mistake I had been making for weeks does not appear to have cost me anything measurable on this sample. That is a null result, and it counts. A null result you record is a question that stays closed, while a null result you bury is one you will re-open in three months.

It is also the cleanest example in this whole post of a bare number pointing the wrong way. The raw cosine had an opinion. It was just an opinion about the wrong thing, because retrieval never sees absolute similarity. It only ever sees which candidate scored higher.

my guess against the invoice

While embedding in bulk I had the run count its own tokens as it went, with a BPE tokenizer, running estimates, a projected dollar cost updating per batch, live pricing pulled in for the arithmetic. Then the invoice arrives and you find out how you did.

two models as each other’s control

The last one is the purest case, because there is no absolute scale to appeal to anywhere.

Every query the service answered was logged with its tokens, its cost and its wall-clock time. Two models had been serving real questions, so the log already held the comparison. I just had to group by model.

On its own, “costs $0.0108 a query” is unreadable. I have no idea whether that is good. Beside its neighbour it says something specific and slightly startling: two and a half times the price for eight percent more latency. Whatever the extra money is buying there, it is not speed.

And then the honest part, which is that this table is only two thirds of a measurement. I have the cost axis and the latency axis and no quality axis at all. No judge scores, no comparison against reference answers, nothing. So it cannot yet answer the only question worth asking, which is whether the larger model is worth 2.5x. A pair of numbers is better than one number and still not the same thing as an answer.

One caveat on the arithmetic: these two rows come from a snapshot of 45 logged queries and do not sum to the 59-query totals elsewhere in my notes. Different moment, same log.

There is also a trap underneath the whole exercise. Both models saw whatever mix of questions happened to arrive, not a mix anyone designed. A small model looks wonderful on easy questions and a large one looks wasteful, so if the traffic happened to be mostly easy, this table is measuring the traffic rather than the models.

what the five have in common

Those are five fairly different things to be poking at, and the same fix worked on all of them. Worth saying that none of the bare numbers were actually wrong. I just could not read them, and I have come to think that is the worse problem. A number you suspect makes you go and check it, while a number you cannot read lets you invent a meaning for it and feel fine about that.

The rule I took from the earlier post was to split a number that was doing two jobs at once. This one is smaller and I suspect I will use it more often. Before you interpret a measurement, go and find out what the same instrument says about something whose answer you already know. Point it at identical inputs. Point it at deliberately wrong ones. Point it at the invoice.

Every measurement is really a comparison, and its second half is there whether you chose it or not. If you do not pick that second half deliberately, it gets picked for you, and in my experience it ends up being whatever you were already expecting.