Update - We have started migration of Cloud servers to new storage cluster. Those who had responded to previous email about migration window - your Cloud servers will be migrated during requested window. All other servers will be migrated one by one during next couple of days as soon as possible.
Jul 21, 06:56 UTC

Update - Update about Cloud management outage in Sao Paulo, Brazil. The new storage cluster has been connected to the Cloud management backend, resulting in Cloud management being operational again. Test actions such as stop, start, restart for Cloud servers were successful. Since the new storage cluster is already deployed and working as expected, all servers will be migrated from the old cluster to the new cluster. This will be done in batches, and each client will receive an email notification before the migration.
Jul 16, 12:09 UTC

Update - The new 3 OSD servers with new netboot and Ceph Nautilus have been prepared in the new storage cluster. Our team is currently testing the migration process with test Cloud servers to ensure the migration performs as expected and to establish ETA per each cloud server. If the tests are completed successfully, we will start continuous migration of Cloud servers in batches. We will update you personally with email notifications regarding the migration window of your cloud servers.
Jul 15, 09:18 UTC

Update - Update about Cloud management outage. After consulting with CEPH engineer our team has localized the core issue which is most likely located in CEPH cache layer. As a result, suggested method was used to fix the issue with currently unused CEPH pool where no client data resides, but apparently it was not completely successful.

Meanwhile, our team is preparing new storage cluster to which clients data could be migrated in the case alternative solutions won't be working. It is expected to have new storage cluster by Friday (2021-07-16) by when it will be known how long it will take to migrate all servers to the new cluster.

In addition to the preparation of the new storage cluster our team is also attempting to fix the current storage cluster.

Thank you for your patience.
Jul 14, 13:30 UTC

Update - The third-party CEPH solution provider is scoping the issue. The engineering team will be meeting with Heficed team tomorrow at 7:00 AM UTC to arrange a work plan to restore the stability of the CEPH storage management system and get back Cloud servers management in the operating state.
Meanwhile, the Terminal changes are in place to disable start and restart options for the Cloud servers.
Additionally, system administrators are still exploring alternatives to resolve an issue as soon as possible fully.
We appreciate your understanding.
Jul 13, 16:35 UTC

Update - We would like to provide an update regarding the ongoing major outage of the Cloud management in Sao Paulo, Brazil location. Currently our system administrators are in contact with third party solution provider to speed up the fixing process. Meanwhile, new orders are temporarily disabled in this location, therefore, it won't be possible order new servers while the issue is not completely fixed.
Jul 13, 13:40 UTC

Update - We are continuing to investigate this issue.
Jul 13, 13:17 UTC

Update - System administrators continue to work on the issue. Please refrain from performing Stop/Start operations while the maintenance is not finished.
Jul 12, 07:35 UTC

Update - We have not yet identified the exact cause of communication issues with IPMI for Bare Metal servers.
System administrators are continuing to troubleshoot the issue in order to restore automated IPMI access.
If there is a need for IPMI access to your Bare Metal server, please create a ticket and our technical support team will provide the access.
Jul 16, 16:23 UTC