Our upstream providers shared mail system users on the Auckland Network may be aware of an issue that occurred on the 18th June during a migration of data to a new mail platform. This resulted in mail delays for some of our clients. We apologise for the inconvenience caused, and have assurance from our upstream providers that future mail migrations will be broken down into a significantly smaller number of mailboxes, so the migrations complete within the planned maintenance window, and do not overrun into business hours. The full incident report is below.
A planned email storage migration to new hardware took longer than expected due to poor data synchronisation performance on the old mail storage hardware. Due to the ongoing mail synchronization issues we were unable to release new mail until the data synchronization had completed.This resulted in a mail delivery delay for many emails, of around 19 hours.
Root Cause Analysis:
The planned mail migration window of 3 hours overran into business hours, delaying delivery of new mail. Poor data synchronisation performance on the old mail hardware during the planned mail migration resulted in a mail delivery delays.
Improvements / Remedial Work:
Future mail migrations will be broken down into a significantly smaller number of mailboxes so the migrations complete within the planned maintenance window and do not overrun into business hours.
00:00: Synchronization start
03:00: Planned end of maintenance window reached, migration still ongoing
19:15: Synchronization completed, queued mail has been delivered to mailboxes
19:30: Issue resolved.