Testing a fix before we built it

The situation

We found that a category of targeting was not being respected in Prebid Server Premium, a consequence of the architecture migration in the previous case study. The behavior had been overlooked because the gap appeared only in setups that differed from how most publishers used the product, while our validation reflected common usage.

Through late 2025 and early 2026, several fixes closed part of the problem but left edge cases uncovered. Two engineering leaders eventually proposed a solution lower in the media hierarchy that they believed would provide the broadest and most reliable coverage. I suspected they were wrong, but a hunch was not a reason to spend engineering time.

Constraints I was handed

Two engineering leaders had considered the proposal carefully and were confident in it.
My doubt was anecdotal, based on customer patterns I had noticed, with no existing analysis to confirm or refute it.
Several partial fixes had already consumed time, so another delay had to be worth it.

The call I made

The expected move was to defer: engineering owned the code and had confidence; I had a feeling. Instead, I decided the disagreement was worth resolving with evidence rather than seniority, and that I would have to produce the evidence myself. Over a weekend I built a tool that evaluated every publisher’s inventory across relevant combinations of media settings, pulled real bid requests for active inventory, and ran hundreds of debug auctions to observe how it was treated. I had done the analysis manually before, but never at this scale.

Time to ship: Rejected · Defer to the proposalImmediate start; implementation begins right away; Chosen · Build the analysis firstA weekend of analysis before anyone writes production code
What it fixes: RejectedThe cases the proposal covers, with the gaps unknown until they surface in production; ChosenEstablishes actual coverage before committing, so the design can be chosen against real data
Cost of being wrong: RejectedAn implemented solution that silently misses a meaningful share of inventory, discovered by customers; ChosenA weekend spent confirming engineering was right, and a small delay

How I brought the finding mattered as much as the result. I framed the evidence as a potential gap rather than a conclusion, shared the code and details so the leaders could critique the method, and said plainly that I could be wrong. I asked for their read. They built their own measurement, confirmed the finding independently, and we agreed on a direction together. It stayed noncontentious because I gave them something to verify rather than something to concede.

What shipped

The analysis changed the direction. The proposed design would have left approximately 14% of Prebid Server Premium revenue exposed to the gap it was meant to close, so it was never built. Instead, the implemented design evaluates the full set of media information on both the inventory and the bid request, closing the gap rather than narrowing it again.

14%

of revenue would have remained exposed under the proposed design. The analysis prevented it from being built.

What I would redo

The gap went through several partial fixes before anyone characterized the problem properly, and I was part of that. We addressed the edge cases we could see without appreciating how many we could not. We were designing against a problem none of us had mapped. I would change the order: describe the full shape of the problem before designing against it. The complication is that this analysis was impractical by hand at the required scale and only became feasible in 2026 with agentic tooling. The narrower lesson is more useful: when repeated fixes keep leaving edge cases behind, stop designing and characterize the problem.