23rd March 2020

Advanced Email Webmail access issues - Basic and Advanced email service

UPDATE - 03:57 PM EST - 25/03/2020

The operations team will be performing an emergency maintenance of targeted users. Some users will be unable to login to their account via POP/IMAP/Webmail or an email client during the maintenance window.

Start time: Mar 25, 2020 at 06:00 PM EST

End time: Mar 26, 2020 at 06:00 AM EST

We will continue to update, as updates become available. Thank you for your patience.


UPDATE - 01:27 PM EST - 25/03/2020

The operations team will be rotating mail flow by enabling and disabling different smtp in nodes at a time to ensure mails are delivered to all users.

We will continue to update, as updates become available. Thank you for your patience.


UPDATE - 11:20 AM EST - 25/03/2020

Our Operations team have enabled IMAP/POP/Webmail. Email deliveries will be resumed shortly.

We will continue to update, as updates become available. Thank you for your patience.


UPDATE - 10:14 AM EST - 25/03/2020

We are seeing high load again. Users may notice slowness or intermittent issues to login. Our team might restart the service to stabilize it hence downtime might be spotted too.

We will continue to update, as updates become available. Thank you for your patience.


UPDATE - 03:38 AM EST - 25/03/2020

Our operations team has ended the maintenance on Basic/Advanced Email by enabling all of the mailstores.

A small number of accounts will still not have webmail or email client access as we prepare for another batch migration to be done during a lull period.


UPDATE - 03:15 AM EST - 25/03/2020

The maintenance is nearing completion. Currently 4 of the 5 mailstores have been enabled again and we are awaiting storage load to drop before the 5th mailstore is enabled.

At this time, only users on the still disabled mailstore are expected to be completely unable to utilize their email. Users from the other 4 mailstores which were enabled will still see degraded performance at this time.

We will continue to update, as updates become available. We thank you for your patience.


UPDATE - 07:58 PM EST - 24/03/2020

Our operations team is working on an emergency migration process to help restore access. A handful of users will be unable to login to their account via webmail or their email clients.


UPDATE - 04:26 PM EST - 24/03/2020

The operations team is still observing a high load on the NAS. They are monitoring it to ensure it stabilizes before they resume email deliveries. Further updates will be posted here.


UPDATE - 02:55 PM EST - 24/03/2020

Our operations team has turned on all 5 mailstores and is currently observing the performance. Once the NAS stabilizes, mail deliveries will be resumed one mailstore at a time. Further updates will be posted here.


UPDATE - 01:36 PM EST - 24/03/2020

Our operations team has turned on 4 mailstores out of 5 mailstores and is currently observing the performance of the NAS. Once the load on the NAS settles down, the remaining mailstore will be turned on as well. Further updates will be posted here.


UPDATE - 11:35 AM EST - 24/03/2020

The operations team has turned off all mailstores on the affected NAS and disabled mail services to stabilize the network storage device. The mailstore migration processes are still running. Further updates will be posted here.


UPDATE - 09:40 AM EST - 24/03/2020

The problem has not fully recovered after last night’s maintenance. For now, email is still being delivered to users inboxes, even if they cannot login at this time. Further updates will be posted here.


UPDATE - 05:20 AM EST - 24/03/2020

The migration appears to be progressing well. A further estimate on the completion time has not yet been made. We will update again once we have this information.


UPDATE - 02:05 AM EST - 24/03/2020

Migration is still ongoing. We'll continue to provide updates as soon as we received them.


UPDATE - 11:55 PM EST

The migration of affected users is still ongoing and appears to be progressing well, although no ETA for its completion can be made at this time. A disk issue has been identified as the main cause of the outage, affecting a subset of users. We'll continue to post update as soon as we received them. We sincerely apologize for the inconveniences caused.


UPDATE - 09:40 PM EST

Emergency maintenance is still ongoing In order to expedite the migration of the affected users. During the maintenance, users may be unable to access our email services. We will post the updates as we receive.


UPDATE - 07:24 PM EST

We are currently having emergency maintenance to bring service back online. Our operation team is still working on the mailstore migration. We will provide updates as we receive them.


UPDATE:

Our Ops department is still investigating the issue. The Ops are checking mail stores one at a time to try to mitigate the impact as a whole. Webmail is still down and many users are still unable to login at this time.


We are currently investigating an issue where some Basic/Advanced email users are having issues to access webmail and to login to POP/IMAP at this time. Further, updates will be posted once available.