Segmented Push Maintenance

Scheduled Maintenance Report for Tapcart

Postmortem

Incident Details
On the evening of May 2, we began receiving reports of app users seeing push notification headlines, messages and attachments not meant for them. This may have occurred for up-to 2% of subscribed active anonymous devices across our user-base. This happened because of a code error introduced during a system update on April 30, which disrupted the logic used to target and validate push notification content.

In our pursuit to lead the future of commerce and deliver shopping experiences that are truly 1-of-1 for every shopper, significant changes have been required to the targeting and delivery architecture of our push notification system. This has included a months-long migration from legacy systems across our entire install base. These improvements are critical to supporting more personalized, timely, and impactful messages for millions of users. Until this incident, the migration scaled as expected without issue—but even a single failure is unacceptable.

To be clear:

  • This was not a breach or system-wide failure.
  • No customer or merchant data was leaked or exposed.
  • The issue was contained within 30 minutes of the first report.

Root Cause

The issue was caused by a small but critical bug introduced during a regular system update on Wed, April 30th 2025, which involved both manual and automated system testing. A modification in the query logic in the segmentation filters, causing the system to misinterpret the intended recipient criteria. This caused a single misconfigured notification to display content to more devices than what was intended.

Our Immediate Response

  • The first reports came in on May 2nd, Friday, 5:10 PM PT.
  • Our engineering team identified the issue by 5:20 PM PT.
  • The faulty query was disabled by 5:41 PM PT, stopping further misrouted notifications. https://status.tapcart.co/incidents/t568yhzktvtn
  • Full service was restored Saturday evening, after we completed extensive maintenance, testing and validation: https://status.tapcart.co/incidents/r7j856mj2d23
  • We communicated directly with affected merchants and updated our status page.

What we patched on Saturday May 3rd

To prevent this from happening again, we’ve already:

  • Strengthened device-to-app verification at multiple points in the pipeline.
  • Improved our QA and release process, especially for push-related code.
  • Implemented automated testing specifically targeting segmentation logic.
  • Launched enhanced monitoring to catch anomalies in push delivery (in progress).
  • Begun a full audit of the new push system (in progress).
Posted May 05, 2025 - 22:33 UTC

Completed

The required maintenance was completed and the upgrade was deployed to our push segmentation infrastructure. This patch will ensure that all content and targeting is accurate and that it is impossible for any content to be unrelated to the device.

We added multiple layers of redundancy to ensure all anonymous devices are always targeted correctly with the correct content.

All segmentation logic is fully operational and segmented notifications will continue to send without any further action.

Thanks for your patience while we conducted the required work to make our systems bulletproof and stable in the future.
Posted May 04, 2025 - 01:31 UTC

Verifying

We are verifying the patch is working and testing before making final deployments
Posted May 03, 2025 - 20:09 UTC

Scheduled

Sending segmented push is currently disabled for maintenance. We will be deploying a patch to ensure targeting is accurate and the service is bulletproof.

Thanks for patience and we will update the status page when the patch has been made.
Posted May 03, 2025 - 20:06 UTC
This scheduled maintenance affected: Push Notifications.