Monitoring - We have identified network connectivity issues between our FRA and JHB locations. Our NOC department has stabilized the situation and we are currently monitoring our network stability.

You may have experienced JHB Cloud Server management problems and connectivity issues between servers located in those areas.
Jul 27, 21:47 UTC

About This Site

Monitoring of Heficed services and issue notifications. Subscribe to Heficed status page to get the latest updates on the service health.

Global Operational
90 days ago
99.99 % uptime
Today
Website ? Operational
Terminal ? Operational
IP Address Market ? Operational
Bare Metal Servers ? Operational
Support Center ? Operational
Abuse Desk ? Operational
API ? Operational
90 days ago
99.99 % uptime
Today
Frankfurt, Germany Operational
90 days ago
100.0 % uptime
Today
FRA - Cloud Servers Operational
90 days ago
100.0 % uptime
Today
FRA - Network Operational
90 days ago
100.0 % uptime
Today
London, United Kingdom Operational
90 days ago
100.0 % uptime
Today
LON - Cloud Servers Operational
90 days ago
100.0 % uptime
Today
LON - Network Operational
90 days ago
100.0 % uptime
Today
Milan, Italy Operational
90 days ago
100.0 % uptime
Today
MIL - Network Operational
90 days ago
100.0 % uptime
Today
Chicago, United States Operational
90 days ago
99.99 % uptime
Today
CHI - Cloud Servers Operational
90 days ago
99.99 % uptime
Today
CHI - Network Operational
90 days ago
100.0 % uptime
Today
Los Angeles, United States Operational
90 days ago
100.0 % uptime
Today
LAX - Cloud Servers Operational
90 days ago
100.0 % uptime
Today
LAX - Network Operational
90 days ago
100.0 % uptime
Today
Reston, United States Operational
90 days ago
100.0 % uptime
Today
RES - Network Operational
90 days ago
100.0 % uptime
Today
São Paulo, Brazil Operational
90 days ago
98.27 % uptime
Today
SAO - Cloud Servers Operational
90 days ago
96.56 % uptime
Today
SAO - Network Operational
90 days ago
99.99 % uptime
Today
Lagos, Nigeria Operational
90 days ago
100.0 % uptime
Today
LGS - Network Operational
90 days ago
100.0 % uptime
Today
Johannesburg, South Africa Operational
90 days ago
99.99 % uptime
Today
JHB - Cloud Servers Operational
90 days ago
100.0 % uptime
Today
JHB - Network Operational
90 days ago
99.99 % uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.
Scheduled Maintenance
We will be performing rack maintenance in São Paulo, Brazil datacenter. During the maintenance, we will be moving part of the servers from one rack to another. No impact to client services is expected.
Posted on Jul 27, 19:14 UTC
Past Incidents
Jul 28, 2021
Completed - The scheduled maintenance has been completed.
Jul 28, 03:30 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jul 28, 03:01 UTC
Scheduled - One of our ISPs are doing emergency maintenance. Small impact is expected including TCP connection drops and connectivity issues while in ISP transition (less than a minute).
Jul 27, 08:57 UTC
Jul 27, 2021
Completed - The scheduled maintenance has been completed.
Jul 27, 17:37 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jul 27, 12:00 UTC
Scheduled - We will be performing rack maintenance in São Paulo, Brazil datacenter. During the maintenance, we will be moving part of the servers from one rack to another. No impact to client services is expected.
Jul 27, 10:52 UTC
Resolved - This incident has been resolved.
Jul 27, 11:31 UTC
Monitoring - Our NOC engineers have detected international connection issues on one of the tier 1 ISPs in United States, Los Angeles, California.

Our team is troubleshooting and trying to get the network stable again. Meanwhile, the tier 1 ISP has been turned off on Heficed network and our team is contacting with them for further investigation.

Please stand by for further updates.
Jul 27, 07:42 UTC
Jul 26, 2021

No incidents reported.

Jul 25, 2021

No incidents reported.

Jul 24, 2021

No incidents reported.

Jul 23, 2021
Resolved - This incident has been resolved.
Jul 23, 10:44 UTC
Update - We have started migration of Cloud servers to new storage cluster. Those who had responded to previous email about migration window - your Cloud servers will be migrated during requested window. All other servers will be migrated one by one during next couple of days as soon as possible.
Jul 21, 06:56 UTC
Monitoring - Management of the Cloud servers is in operational state again. The changes that were implemented in the Terminal to control Cloud servers (stop/start/restart) are reverted back and server control is working as usual. Furthermore, ordering of Cloud servers in Sao Paulo, Brazil has been resumed as well.

System administrators continue to monitor storage clusters to ensure they are operating as expected.
Jul 16, 15:11 UTC
Update - Update about Cloud management outage in Sao Paulo, Brazil. The new storage cluster has been connected to the Cloud management backend, resulting in Cloud management being operational again. Test actions such as stop, start, restart for Cloud servers were successful. Since the new storage cluster is already deployed and working as expected, all servers will be migrated from the old cluster to the new cluster. This will be done in batches, and each client will receive an email notification before the migration.
Jul 16, 12:09 UTC
Update - The new 3 OSD servers with new netboot and Ceph Nautilus have been prepared in the new storage cluster. Our team is currently testing the migration process with test Cloud servers to ensure the migration performs as expected and to establish ETA per each cloud server. If the tests are completed successfully, we will start continuous migration of Cloud servers in batches. We will update you personally with email notifications regarding the migration window of your cloud servers.
Jul 15, 09:18 UTC
Update - Update about Cloud management outage. After consulting with CEPH engineer our team has localized the core issue which is most likely located in CEPH cache layer. As a result, suggested method was used to fix the issue with currently unused CEPH pool where no client data resides, but apparently it was not completely successful.

Meanwhile, our team is preparing new storage cluster to which clients data could be migrated in the case alternative solutions won't be working. It is expected to have new storage cluster by Friday (2021-07-16) by when it will be known how long it will take to migrate all servers to the new cluster.

In addition to the preparation of the new storage cluster our team is also attempting to fix the current storage cluster.

Thank you for your patience.
Jul 14, 13:30 UTC
Update - The third-party CEPH solution provider is scoping the issue. The engineering team will be meeting with Heficed team tomorrow at 7:00 AM UTC to arrange a work plan to restore the stability of the CEPH storage management system and get back Cloud servers management in the operating state.
Meanwhile, the Terminal changes are in place to disable start and restart options for the Cloud servers.
Additionally, system administrators are still exploring alternatives to resolve an issue as soon as possible fully.
We appreciate your understanding.
Jul 13, 16:35 UTC
Update - We would like to provide an update regarding the ongoing major outage of the Cloud management in Sao Paulo, Brazil location. Currently our system administrators are in contact with third party solution provider to speed up the fixing process. Meanwhile, new orders are temporarily disabled in this location, therefore, it won't be possible order new servers while the issue is not completely fixed.
Jul 13, 13:40 UTC
Update - We are continuing to investigate this issue.
Jul 13, 13:17 UTC
Update - System administrators continue to work on the issue. Please refrain from performing Stop/Start operations while the maintenance is not finished.
Jul 12, 07:35 UTC
Investigating - São Paulo, Brazil cloud service currently is experiencing degraded performance due to no VM management. All running VMs are not impacted (only Stop / Restart / Start does not work).
Jul 11, 13:28 UTC
Jul 22, 2021

No incidents reported.

Jul 21, 2021
Jul 20, 2021
Completed - The scheduled maintenance has been completed.
Jul 20, 09:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jul 20, 06:00 UTC
Scheduled - We will be performing maintenance on an internal service in SAO location. During this time, these services will be unavailable in SAO location:
- Bare Metal server IPMI access,
- IP control,
- Cloud management
Jul 19, 14:38 UTC
Jul 19, 2021
Resolved - After a thorough investigation, the issue has been resolved. The root cause was found to be a software issue, which has been already identified as of right now.
Jul 19, 12:55 UTC
Update - We have not yet identified the exact cause of communication issues with IPMI for Bare Metal servers.
System administrators are continuing to troubleshoot the issue in order to restore automated IPMI access.
If there is a need for IPMI access to your Bare Metal server, please create a ticket and our technical support team will provide the access.
Jul 16, 16:23 UTC
Investigating - We have identified that there's communication issues with IPMI for Bare Metal servers. Currently system administrators are troubleshooting the issue in order to restore IPMI access.

If there is a need for IPMI access to your Bare Metal server then please create a ticket and our technical support team will provide the access.
Jul 15, 14:54 UTC
Jul 18, 2021

No incidents reported.

Jul 17, 2021

No incidents reported.

Jul 16, 2021
Jul 15, 2021
Jul 14, 2021