Back to the blog
Guides

Reading a success-rate badge: what 87 % actually means

It is a trailing seven-day delivery rate with a confidence interval you cannot see. What it measures, what it excludes, and why 87 % on 25 samples tells you almost nothing.

by Marta Kwiatkowska·Support Lead·Published 7 Jun 2022·6 min read

Every service and country pair on the pricing page carries a percentage and a coloured dot. Customers use it to choose a country, which is exactly what it is for, and a good number of the resulting support tickets come from reading more into it than it says.

The definition

It is the share of paid activations for that exact service and country pair, over the trailing seven days, where at least one SMS arrived before the twenty-minute window closed. That is the entire definition. Not a rating, not a quality score, not a promise.

Seven days is short on purpose. Until 2021 we showed a lifetime average, which was beautifully stable and actively misleading: a pair that had worked for eighteen months and had been dead for three weeks still read 91 %. A short window is noisier and honest.

What it does not include

  • Whether the code worked. If the SMS arrives and the service rejects the account an hour later, that is a success by this measure. We cannot see inside the service and neither can anyone else.
  • Activations you cancelled before the window closed. They are excluded entirely rather than counted as failures — cancelling usually means you changed your mind, not that we failed.
  • Messages that arrived after expiry. We still show them to you when they land, and you are not charged, but they do not count as a success, because for your purposes they came too late.
  • Other countries. The number is per pair. Telegram in Portugal tells you nothing about Telegram in Poland, and nothing about WhatsApp in Portugal.

The part nobody looks at: the sample size

We show the count next to the percentage. It is the more important of the two numbers. 87 % is a point estimate, and a point estimate from a small sample has a range around it wide enough to change your decision. Here is 87 % at various sample sizes, with the 95 % interval:

Activations in the windowPoint estimate95 % interval
2587 %69 % – 95 %
5087 %75 % – 93 %
10087 %79 % – 92 %
50087 %84 % – 90 %
2,00087 %85 % – 89 %

At n = 25, a pair showing 87 % and a pair showing 76 % are not distinguishable. Choosing between them on the badge alone is choosing on noise. At n = 500 the difference is real and worth acting on. As a rough working rule: below 50 samples, treat the badge as a rumour; above 300, treat it as a measurement.

Small samples are not a defect in the data — they are what a quiet pair looks like. A country and service that only sees forty purchases a week cannot produce a tighter number, and inventing confidence it does not have would be worse than showing the count.

The dot

ColourTrailing rateWhat we mean by it
Green88 % and aboveBuy without thinking about it.
Amber75 % – 87 %Works, but budget for retries in a batch.
RedBelow 75 %Something is wrong with the route. Pick another country.

The thresholds are flat across all pairs, which means an amber dot on a heavily-used pair is a much stronger signal than an amber dot on a quiet one — the first is a measurement, the second might be four unlucky mornings.

How to actually use it

  1. Compare within one service. Ranking countries for Telegram is what the badge is good at. Comparing Telegram against Discord tells you about the two services' filtering policies, not about us.
  2. For a batch, plan on the lower bound of the interval, not the point estimate. At 87 % on 60 samples, budget as if it were 78 % and you will not run short.
  3. Test with one activation before committing to a run. It costs cents, failures are free, and it catches the case where the badge is stale because the pair has been quiet.
  4. Watch the direction, not the digit. A pair that read 91 % last week and 84 % today is degrading, and the seven-day window has not caught up with it yet.

And if a red pair is the only one that fits your requirement, tell support rather than buying into it repeatedly. Red usually means one upstream route has started filtering, and quite often we can switch the pair to a different pool the same day.

Articles are published in English.