Real Time Reporting API Outage

Incident Report for Kevel

Resolved

Our queue has caught up and this issue has been resolved.

Ads were not properly removed after hitting their caps or paced properly from about 04:00 UTC to 23:30 UTC on May 1st. All ads should now be serving as expected.
Posted May 01, 2024 - 20:47 EDT

Monitoring

A fix has been implemented and messages are now processing. We expect the system to take some time to process all message and remove all ads that have reached their caps. We will continue to monitor and update when complete.
Posted May 01, 2024 - 19:32 EDT

Identified

We are continuing to work on a fix for the identified issue.
Posted May 01, 2024 - 19:21 EDT

Monitoring

We are continuing to work on a fix for the identified issue.
Posted May 01, 2024 - 19:09 EDT

Identified

We have identified a misconfiguration in our Real Time Reporting system that resulted in messages not being processed. Ads that reached their caps after 04:00 UTC were not properly removed from the ad servers, which may have led to over delivery. Ad pacing is also impacted. We have determined the cause of the misconfiguration and are working on a fix.
Posted May 01, 2024 - 18:29 EDT

Investigating

We are seeing issues with our Real Time Reporting API. This system powers the data within our UI, as well as ad pacing and delivery. We are investigating at this time.
Posted May 01, 2024 - 17:51 EDT
This incident affected: Ad Serving.