Our upstream providers shared mail system users on the Auckland Network may be aware of an issue that occurred on the 18th June during a migration of data to a new mail platform. This resulted in mail delays for some of our clients. We apologise for the inconvenience caused, and have assurance from our upstream providers that future mail migrations will be broken down into a significantly smaller number of mailboxes, so the migrations complete within the planned maintenance window, and do not overrun into business hours. The full incident report is below.
Incident report:
A planned email storage migration to new hardware took longer than
expected due to poor data synchronisation performance on the old mail
storage hardware. Due to the ongoing mail synchronization issues we were
unable to release new mail until the data synchronization had
completed.This resulted in a mail delivery delay for many emails, of
around 19 hours.
Root Cause Analysis:
The planned mail migration window of 3 hours overran into business
hours, delaying delivery of new mail. Poor data synchronisation
performance on the old mail hardware during the planned mail migration
resulted in a mail delivery delays.
Improvements / Remedial Work:
Future mail migrations will be broken down into a significantly smaller
number of mailboxes so the migrations complete within the planned
maintenance window and do not overrun into business hours.
Event Sequence:
00:00: Synchronization start
03:00: Planned end of maintenance window reached, migration still ongoing
19:15: Synchronization completed, queued mail has been delivered to mailboxes
19:30: Issue resolved.