Segmented Push Maintenance

Incident Details
On the evening of May 2, we began receiving reports of app users seeing push notification headlines, messages and attachments not meant for them. This may have occurred for up-to 2% of subscribed active anonymous devices across our user-base. This happened because of a code error introduced during a system update on April 30, which disrupted the logic used to target and validate push notification content.

In our pursuit to lead the future of commerce and deliver shopping experiences that are truly 1-of-1 for every shopper, significant changes have been required to the targeting and delivery architecture of our push notification system. This has included a months-long migration from legacy systems across our entire install base. These improvements are critical to supporting more personalized, timely, and impactful messages for millions of users. Until this incident, the migration scaled as expected without issue—but even a single failure is unacceptable.

To be clear:

• This was not a breach or system-wide failure.
• No customer or merchant data was leaked or exposed.
• The issue was contained within 30 minutes of the first report.

Root Cause

The issue was caused by a small but critical bug introduced during a regular system update on Wed, April 30th 2025, which involved both manual and automated system testing. A modification in the query logic in the segmentation filters, causing the system to misinterpret the intended recipient criteria. This caused a single misconfigured notification to display content to more devices than what was intended.

Our Immediate Response

• The first reports came in on May 2nd, Friday, 5:10 PM PT.
• Our engineering team identified the issue by 5:20 PM PT.
• The faulty query was disabled by 5:41 PM PT, stopping further misrouted notifications. https://status.tapcart.co/incidents/t568yhzktvtn
• Full service was restored Saturday evening, after we completed extensive maintenance, testing and validation: https://status.tapcart.co/incidents/r7j856mj2d23
• We communicated directly with affected merchants and updated our status page.

What we patched on Saturday May 3rd

To prevent this from happening again, we’ve already:

• Strengthened device-to-app verification at multiple points in the pipeline.
• Improved our QA and release process, especially for push-related code.
• Implemented automated testing specifically targeting segmentation logic.
• Launched enhanced monitoring to catch anomalies in push delivery (in progress).
• Begun a full audit of the new push system (in progress).