UPDATE - 10:42 PM EST - 26/03/2020
With all Mailstores enabled and Basic/Advanced Email functionalities fully restored, we're marking this incident resolved. Please reach out to support should you require further assistance or have any concerns related to this incident. Thank you for your patience and we apologize sincerely for the inconveniences caused.
UPDATE - 09:35 PM EST - 26/03/2020
We have now unblocked all users on Basic/Advanced Email to be able to access and use mail services.
There are still remaining accounts to be migrated to the new mailstore, and we will post updates on this incident to keep you informed as we work to complete the migration.
UPDATE - 04:13 PM EST - 26/03/2020
The affected mailstore is now enabled again. With the exception of the small subset of users placed in maintenance for additional investigation. All other users on Basic/Advanced email should be accessible without issue. Updates will continue to be posted here.
UPDATE - 03:44 PM EST - 26/03/2020
The previously disabled mailstores have been re-enabled again. Our Operations department has narrowed down the potential issue in the last affected mailstore to a subset of users. Currently, we are bringing the last affected mailstore online with those users disabled so we can monitor performance. Updates will continue to be posted here.
UPDATE - 03:01 PM EST - 26/03/2020
While re-enabling the affected mailstore an issue regarding it was discovered. To mitigate that, our Operations department will be disabling some other mailstores. They will be brought back up one at a time to monitor performance. Updates will continue to be posted here.
UPDATE - 02:33 PM EST - 26/03/2020
We have re-enabled all mailstores in our system except one since our last update. Some users on the affected mailstore will be offline until we completed the migration. We will provide an update as soon as the migration is done. Updates will continue to be posted here.
UPDATE - 08:10 AM EST - 26/03/2020
Most of the affected mailstores have been re-enabled. While the operations team continues to investigate another to identify and rectify the issue on it. Updates will continue to be posted here.
UPDATE - 07:05 AM EST - 26/03/2020
After re-enabling our mailstores to resolve this ongoing incident we have encountered a different issue and needed to bring the mailstores offline again in order to resolve this problem. We will likely be bringing them back up shortly once we have completed testing. These affected users will be unable to login to the Webmail/IMAP or POP until the mailstores have been re-established.
UPDATE - 06:00 AM EST - 26/03/2020
The mailstores have been enabled with a set amount of users migrated. The CR and migration are stopped for today. At this time service is fully online and we will be monitoring for any impact come business hours EST which is currently expected to be better than previous days.
UPDATE - 03:57 PM EST - 25/03/2020
The operations team will be performing an emergency maintenance of targeted users. Some users will be unable to login to their account via POP/IMAP/Webmail or an email client during the maintenance window.
Start time: Mar 25, 2020 at 06:00 PM EST
End time: Mar 26, 2020 at 06:00 AM EST
We will continue to update, as updates become available. Thank you for your patience.
UPDATE - 01:27 PM EST - 25/03/2020
The operations team will be rotating mail flow by enabling and disabling different smtp in nodes at a time to ensure mails are delivered to all users.
We will continue to update, as updates become available. Thank you for your patience.
UPDATE - 11:20 AM EST - 25/03/2020
Our Operations team have enabled IMAP/POP/Webmail. Email deliveries will be resumed shortly.
We will continue to update, as updates become available. Thank you for your patience.
UPDATE - 10:14 AM EST - 25/03/2020
We are seeing high load again. Users may notice slowness or intermittent issues to login. Our team might restart the service to stabilize it hence downtime might be spotted too.
We will continue to update, as updates become available. Thank you for your patience.
UPDATE - 03:38 AM EST - 25/03/2020
Our operations team has ended the maintenance on Basic/Advanced Email by enabling all of the mailstores.
A small number of accounts will still not have webmail or email client access as we prepare for another batch migration to be done during a lull period.
UPDATE - 03:15 AM EST - 25/03/2020
The maintenance is nearing completion. Currently 4 of the 5 mailstores have been enabled again and we are awaiting storage load to drop before the 5th mailstore is enabled.
At this time, only users on the still disabled mailstore are expected to be completely unable to utilize their email. Users from the other 4 mailstores which were enabled will still see degraded performance at this time.
We will continue to update, as updates become available. We thank you for your patience.
UPDATE - 07:58 PM EST - 24/03/2020
Our operations team is working on an emergency migration process to help restore access. A handful of users will be unable to login to their account via webmail or their email clients.
UPDATE - 04:26 PM EST - 24/03/2020
The operations team is still observing a high load on the NAS. They are monitoring it to ensure it stabilizes before they resume email deliveries. Further updates will be posted here.
UPDATE - 02:55 PM EST - 24/03/2020
Our operations team has turned on all 5 mailstores and is currently observing the performance. Once the NAS stabilizes, mail deliveries will be resumed one mailstore at a time. Further updates will be posted here.
UPDATE - 01:36 PM EST - 24/03/2020
Our operations team has turned on 4 mailstores out of 5 mailstores and is currently observing the performance of the NAS. Once the load on the NAS settles down, the remaining mailstore will be turned on as well. Further updates will be posted here.
UPDATE - 11:35 AM EST - 24/03/2020
The operations team has turned off all mailstores on the affected NAS and disabled mail services to stabilize the network storage device. The mailstore migration processes are still running. Further updates will be posted here.
UPDATE - 09:40 AM EST - 24/03/2020
The problem has not fully recovered after last night’s maintenance. For now, email is still being delivered to users inboxes, even if they cannot login at this time. Further updates will be posted here.
UPDATE - 05:20 AM EST - 24/03/2020
The migration appears to be progressing well. A further estimate on the completion time has not yet been made. We will update again once we have this information.
UPDATE - 02:05 AM EST - 24/03/2020
Migration is still ongoing. We'll continue to provide updates as soon as we received them.
UPDATE - 11:55 PM EST
The migration of affected users is still ongoing and appears to be progressing well, although no ETA for its completion can be made at this time. A disk issue has been identified as the main cause of the outage, affecting a subset of users. We'll continue to post update as soon as we received them. We sincerely apologize for the inconveniences caused.
UPDATE - 09:40 PM EST
Emergency maintenance is still ongoing In order to expedite the migration of the affected users. During the maintenance, users may be unable to access our email services. We will post the updates as we receive.
UPDATE - 07:24 PM EST
We are currently having emergency maintenance to bring service back online. Our operation team is still working on the mailstore migration. We will provide updates as we receive them.
UPDATE:
Our Ops department is still investigating the issue. The Ops are checking mail stores one at a time to try to mitigate the impact as a whole. Webmail is still down and many users are still unable to login at this time.
We are currently investigating an issue where some Basic/Advanced email users are having issues to access webmail and to login to POP/IMAP at this time. Further, updates will be posted once available.