Energy

A Six-Figure Payments RCA, and Governance That Kept Delivery Moving

Led the RCA on a six-figure payments incident that was failing hundreds of customer transactions, drove validated recovery and prevention, and stood up engineering governance for a two-squad org without slowing delivery.

Led the RCA on a six-figure payments incident that was failing hundreds of customer transactions, drove validated recovery and a prevention plan, and stood up engineering governance from zero for a two-squad org, without slowing delivery.

Context and stakes

An EveryTech principal consultant joined as a hands-on technical lead and solution architect across two squads handling customer-facing sales, support, and payment-adjacent journeys at a national energy retailer. The environment spanned many repositories and production-critical integrations: identity flows, payment-gateway dependencies, and operational tooling.

Problem

Delivery had no shared operating model. There were no consistent readiness or done criteria, ownership across repositories was weak, and quality checks passed even when meaningful test coverage was missing. Then a payments incident hit. Credential-rotation timing and cache behavior interacted badly and started failing live customer transactions, a six-figure problem in progress.