This incident has been resolved.
Jul 23, 10:44 UTC
We have started migration of Cloud servers to new storage cluster. Those who had responded to previous email about migration window - your Cloud servers will be migrated during requested window. All other servers will be migrated one by one during next couple of days as soon as possible.
Jul 21, 06:56 UTC
Management of the Cloud servers is in operational state again. The changes that were implemented in the Terminal to control Cloud servers (stop/start/restart) are reverted back and server control is working as usual. Furthermore, ordering of Cloud servers in Sao Paulo, Brazil has been resumed as well.
System administrators continue to monitor storage clusters to ensure they are operating as expected.
Jul 16, 15:11 UTC
Update about Cloud management outage in Sao Paulo, Brazil. The new storage cluster has been connected to the Cloud management backend, resulting in Cloud management being operational again. Test actions such as stop, start, restart for Cloud servers were successful. Since the new storage cluster is already deployed and working as expected, all servers will be migrated from the old cluster to the new cluster. This will be done in batches, and each client will receive an email notification before the migration.
Jul 16, 12:09 UTC
The new 3 OSD servers with new netboot and Ceph Nautilus have been prepared in the new storage cluster. Our team is currently testing the migration process with test Cloud servers to ensure the migration performs as expected and to establish ETA per each cloud server. If the tests are completed successfully, we will start continuous migration of Cloud servers in batches. We will update you personally with email notifications regarding the migration window of your cloud servers.
Jul 15, 09:18 UTC
Update about Cloud management outage. After consulting with CEPH engineer our team has localized the core issue which is most likely located in CEPH cache layer. As a result, suggested method was used to fix the issue with currently unused CEPH pool where no client data resides, but apparently it was not completely successful.
Meanwhile, our team is preparing new storage cluster to which clients data could be migrated in the case alternative solutions won't be working. It is expected to have new storage cluster by Friday (2021-07-16) by when it will be known how long it will take to migrate all servers to the new cluster.
In addition to the preparation of the new storage cluster our team is also attempting to fix the current storage cluster.
Thank you for your patience.
Jul 14, 13:30 UTC
The third-party CEPH solution provider is scoping the issue. The engineering team will be meeting with Heficed team tomorrow at 7:00 AM UTC to arrange a work plan to restore the stability of the CEPH storage management system and get back Cloud servers management in the operating state.
Meanwhile, the Terminal changes are in place to disable start and restart options for the Cloud servers.
Additionally, system administrators are still exploring alternatives to resolve an issue as soon as possible fully.
We appreciate your understanding.
Jul 13, 16:35 UTC
We would like to provide an update regarding the ongoing major outage of the Cloud management in Sao Paulo, Brazil location. Currently our system administrators are in contact with third party solution provider to speed up the fixing process. Meanwhile, new orders are temporarily disabled in this location, therefore, it won't be possible order new servers while the issue is not completely fixed.
Jul 13, 13:40 UTC
We are continuing to investigate this issue.
Jul 13, 13:17 UTC
System administrators continue to work on the issue. Please refrain from performing Stop/Start operations while the maintenance is not finished.
Jul 12, 07:35 UTC
São Paulo, Brazil cloud service currently is experiencing degraded performance due to no VM management. All running VMs are not impacted (only Stop / Restart / Start does not work).
Jul 11, 13:28 UTC