# A release process that kept configuration errors away from 50 million daily impressions

PayPal · Jun 2022 – Q4 2024
Author: Henrique Senra (https://henriquesenra.com)
Web version: https://henriquesenra.com/cases/paypal-release-governance

> After a configuration error of mine hid the Send and Request buttons for more than 60% of users, I rebuilt how changes reached production on PayPal's Home screen. The rule held for more than two years, until a change in management stopped enforcing it and a rushed change hid the QR code icon for about 90% of users.

## Key outcomes

- Held for more than two years: every change needed QA sign-off in staging.
- Launch errors on engineering's dashboard went from mostly red to mostly orange, with this process as one factor.
- When enforcement lapsed, one rushed change hid the QR code icon for about 90% of users until I reverted it.
- Revenue protected, directional: no incident count or dollar figure exists.

Tags: Release governance, QA gates, Incident prevention, Intake

## TL;DR

- **Problem.** My own configuration error hid the Send and Request buttons for more than 60% of PayPal app users. Leadership kept pushing urgent changes to production without QA.
- **My role.** I redesigned intake for Home configuration changes, triaged tickets, built or checked changes in Stage, and held the QA sign-off rule.
- **Decision and cost.** Nothing went to production without QA sign-off in Stage, with the requesting team supplying evidence. It added friction, and some managers saw it as bureaucracy.
- **Outcome.** The rule held for more than two years, and errors fell. Business end: revenue protected, directional. No incident count or dollar figure exists.

## Context

Home is the screen with about 50 million daily impressions that opens when someone starts the PayPal app. I owned its configuration for navigation, the app menu and the Home Dashboard. The configurations lived in PStudio, PayPal's internal no-code personalization platform. PStudio and the platforms around it had a Stage version (the pre-production copy) and a Production version. A change was built in Stage first, QA ran manual and regression tests, then the change was exported to Production and approved there.

PayPal already had a paired approval flow. One person creates a change and a second person approves it, and nobody publishes their own work. Production approval took two keys, a QA approver and a second approver, first a product manager and later an engineering peer. Approved configurations went live to real users in about five minutes.

Incident and reliability metrics were not a product concern at PayPal. A quality team and a hardened process owned them. Bugs still landed on the Home team, because the monitoring alerts that fired were Home's dashboards.

## What happened

The first incident came in June 2022, right after I joined. Contextual rules, which decide who sees a piece of content, accepted the app version under two variable names. One was a plain number and the other understood version logic such as 8.34.1. The platform team did not want to delete one and force every user of the variable to adapt. A check I configured used the wrong variable, so a rule meant to read "app version above 8.15.1" behaved as equal to 8.15. When most users updated to 8.16, they lost the Send and Request buttons, central products on Home. By my first recall, more than 60% of users lost access. I made that change myself, and I am sure a missing QA step in Stage caused it. The backend flaw behind it was never fixed while I was there.

## What I did

I organized my own work by ticket type first, then rebuilt the intake funnel for Home personalization tasks. The point was to type each request and make the data easier to collect. That decision came from a series of cases and my own onboarding at the company. After the Send and Request error, the team started requiring a QA thumbs-up on every ticket going to production. The intake had evolved slowly, the incident pushed us to give QA more attention, and then it all hardened into one end-to-end process.

The process, as I ran it:

1. **Intake.** A short page linked to template tickets, one per work type, such as a new asset, a zone enablement, or a request for consulting or approval. Each template listed the minimum information, including copy, image links, the app link, eligibility rules, instrumentation, experiment plan and delivery date.
2. **Approval to enter the sprint** from the product or engineering manager.
3. **Triage.** I did it personally and added questions about personalization and segmentation.
4. **Build in Stage**, by me or a teammate.
5. **QA evidence.** The Home QA team checked work on Home's own objects. For other teams' objects, the domain team ran QA and showed evidence for Home QA to evaluate.
6. **Export and approval.** I exported the change to Production, usually with a simple dashboard to track impressions or click-through as a sanity check, and Home QA plus one Home engineer approved for publishing.

Where I stood. For the objects Home governed but did not own, our role was to guide and help build. QA sign-off was not meant to block anyone. It was a procedural argument that let us slow down outside pressure to ship at any time. I also learned to open QA builds in Stage myself and check on screen that routing and rendering were right. Exceptions existed, such as bug fixes and a few very critical launches, or the days before the Christmas and Thanksgiving freezes, when Fridays lost their protection. They were exceptions that had business logic behind them, not internal negotiation or someone using political power.

## Options

```mermaid
flowchart LR\n  A["Typed intake ticket"] --> B["Approval to enter sprint"] --> C["Triage"] --> D["Build in Stage"] --> E["QA evidence and sign-off"] --> F["Export and approval"] --> G["Production"]
```

| Option | Why it was not enough | Cost accepted in the rule I chose |
| --- | --- | --- |
| Peer approval and ticket-time targets, which already existed | They did not stop urgent pushes without QA, and my own error went through them. | A QA step on every ticket, and a no-Friday rule. |
| Decide case by case under leadership pressure | This is what happened before the rule, and again when enforcement lapsed. | Friction with teams that wanted to ship on their own schedule. |

What I would choose now is different from either. I would move the rules out of text in a ticket and into the platform, with a script, check or CI that lets work move only when every item is filled in or marked as skipped. Governance of this kind has to be built into the platform and not depend on one person fighting for the rules to be followed. A PStudio feature that pulled in the ticket and allowed export only with everything filled in would have avoided rework, meetings and bugs. That is a cost I project, not one I measured.

## Results

After intake and QA sign-off came in, shortly before a Black Friday code freeze, we saw a reduction in errors. I have no figure for it. Engineering kept a large dashboard of Home errors, covering app crashes, content that did not load, loops and empty skeletons. Over time launches moved from mostly red to mostly orange there, meaning fewer error reports per launch, and this process was one factor among others. Product managers at PayPal did not track that dashboard closely. I was only curious.

The rule held for two to two and a half years, enforced in the large majority of cases.

Then a manager change stopped reinforcing it. Under outside pressure, someone rushed a change to the QR code icon, with a small instrumentation edit. The app-version rule went up as equal to 8.60 and not 8.60 or higher. Since 8.60 was already an old version, the icon disappeared for about 90% of users. A Home engineer noticed at lunch and posted in the Home channel. I remembered the rushed change and reverted the configuration, and the icon came back. The case created a new bug with real business impact.

A third case involved naming. A common pattern at PayPal put a date at the start of an object's name, so an object created in Stage one day and in Production the next got different names, and configurations pointing to it broke. In one large domain team most assets stopped working, especially a financial product card. It had a real business impact. The team had fought that pattern for a long time. It is a case where the QA and naming process should have applied to other teams and did not.

What I did not measure: incident counts before and after, engineering hours saved, or any revenue loss. The red-to-orange change belongs to engineering's dashboard.

## What others said

> “He also just made things easy — he cleaned up how cross-domain requests came in and made sure QA was actually part of the process, which saved us from a lot of headaches down the line.”
>
> Laura Valverde, World Ready Program / Quality Manager, on LinkedIn, September 23, 2026. She was senior to me and did not manage me directly.

## Friction and what I would change

About 30% of cases carried urgency and a wish to publish very fast. Teams were not used to this kind of QA, since QA processes were tied to deploys, and Stage environments were sometimes hard to find or set up. The biggest friction was with the Credit team. They wanted to ship on their own schedule and did not understand that some objects fell under Home's governance. They would surface a Home change at the end of a campaign deadline, and they expected Home to sign off rather than own the QA.

Some managers saw the intake as bureaucracy, and others said it helped them go from zero to one much faster. The process was well documented and had my leadership's buy-in, but there was some dissatisfaction.

In one escalation I was called to a meeting about a deadline for work in Stage. I said I was doing my part and other stakeholders had theirs. A manager disliked that and escalated.

The QR code case was a failure to enforce a process that had been designed correctly, and it taught me the same lesson again. I would build the enforcement into the platform and not into one person.

## Limits

- The business effect is directional. No incident count, hours or dollar figure exists.
- Error levels come from an engineering dashboard I did not own, and the process was one factor.
- Names of colleagues and internal dashboards stay private.
