[June 6, 2017]
Due to a glitch in our development code, that got pushed into the production environment, our platform has sent out some false positive blacklist monitoring notifications regarding a RBL that is not yet ready for production, nor was it supposed to be live at this time.
The incident has caused a small number of blacklist monitors, that were checked between 11:23AM (ET) and 12:12PM (ET), to be affected. We’ve calculated that 1.93% of our total blacklist monitors belonging to 2.70% of our total active users have been affected.
We’ve quickly noticed and fixed the issue, then forced all affected blacklist monitors to be re-checked in order to correct the false data.
Incident time frame (Eastern Time):
- 11:23AM (ET): new code (containing the glitch) got pushed to the production environment
- 12:00PM (ET): hourly blacklist monitoring notifications have been automatically sent out, some containing the false data caused by this glitch
- 12:12PM (ET): the issue is noticed and the blacklist monitoring queue is shut down for emergency maintenance
- 12:38PM (ET): the issue has been fixed, blacklist monitoring queue is started up again, and affected blacklist monitors are pushed for a re-check
- 12:46PM (ET): all affected blacklist monitors have been re-checked, and no incorrect data is being displayed on our platform anymore
The affected users will have received (or will receive) a total of two* batches of notifications caused by this incident:
- the first batch, which contains the false blacklisting data, mostly sent at 12:00PM (ET) [can vary up to +24h, based on each user’s blacklist monitoring stacked notifications settings]
- the second batch, which contains the rectification delisting data, mostly sent between 01:00PM (ET) and 04:00PM (ET) [can vary up to +24h, based on each user’s blacklist monitoring stacked notifications settings]
*If your blacklist monitoring stacked notifications setting is set higher than 4 hours, the two batches described above will have been automatically merged into just one batch of notifications.
If you have any questions, please open a support ticket on our website.