Network Issues

Starting from 20th July 2024, we will be migrating our services to the new Data center in Rosebank.  Our Upstream Provider Dimension Data  was sold to NTT Data a large Data center Company and they decided to close the Greenside Data center by the End of July 2024 instead of December 2024.

This has put us in a predicament that we need to do an urgent Migration to the New DC. We have until August 14th to complete this move. Unfortunately this will possibly cause some service interruptions and we plan on doing the changes during early mornings and Over the weekends. You possibly could experience interruptions during these times.

19th/21 July 2024 - We have migrated the database servers over the Weekend of the 20th July successfully without issues.

27/28 July 2024 - Website Servers to be migrated - In Progress. Some dns issues have caused this to be delayed a bit due to issues beyond our control

3/4 Aug 2024 - Mail Servers to be migrated -

The data center migration went ahead after having to be suspended for a 7 day period due to dns issues. This was successfully done and all servers were migrated with minimal issues.

Effects - This issue does not effect your relevant products/services
Affecting Other - Datacenter Moves
Priority - Critical
Date - 20/07/2024 16:50 - 16/08/2024 00:00
Last Updated - 23/08/2024 12:30

Good Day,

Please be advised we will be doing some web server optimization after hours, as to not interfere with normal daily working of servers, All changes will be performed after 10pm each night, users might experience some downtime during this period. We apologize for the inconvenience this may cause.

Regards

HostworX Support

Effects - This issue does not effect your relevant products/services
Affecting System - All Servers
Priority - High
Date - 02/05/2024 22:00 - 27/07/2024 16:58
Last Updated - 02/05/2024 14:23

Server upgrades and  updates will be processed during the week, some intermittent downtime will happen, this is unfortunatly unavoidable. However this will be intermittent and temporary.

Effects - This issue does not effect your relevant products/services
Affecting System - All Servers
Priority - High
Date - 12/06/2023 00:00 - 19/06/2023 00:00
Last Updated - 21/06/2023 13:00

Certain Cleints have been experiencing issues with deliveries to certain hotmail and outlook hosted domains. These issues were taken up with outlook support in the USA, however despite our servers not being blocked on outlooks spam system, we have had deliverability issues.

We have made some changes on our own systems to counter this issue, however this will possibly cause some intermittent connectivity issues during the changeover period. This has been effected overnight as to not impact the bulk of our clients.

If any issues are experienced, please contact support urgently so we can resolve the issues asap.

Regards

HostworX Support.

Effects - This issue does not effect your relevant products/services
Affecting Server -
Priority - Critical
Date - 11/06/2022 17:57 - 21/06/2023 13:01
Last Updated - 11/08/2022 21:15

Please note we will be doing some updates to the mysql server on friday night after 8pm, there will be a 15 to 30 minute outage during this period.

Websites using mysql databases will experience temporary outtage while the servers are restarted after the updates.

We apologise but this is an urgent change to the software and the servers need to be restarted to update affecting connectivity for a short period.

Effects - This issue does not effect your relevant products/services
Affecting Other - Mysql Server
Priority - Critical
Date - 25/02/2022 20:00 - 11/08/2022 21:16
Last Updated - 25/02/2022 10:23

Good day,

The data center had some air conditioner personnel in putting in new trunking for air conditioners,   somewhere during the process they seemed to have either pulled or cut power cables feeding some of the data centers server racks. This cabling is in between the UPS's and the actual racks. Causing endless issues for them to isolate the problem. Despite being on 2 separate power systems to prevent such issues from occurring, it did happen. There is an investigation into this being taken by internet solutions and Dimension Data.

This issue was beyond our control and we have managed to have gotten power back up temporarily until the full power can be restored as it should be.

Again we apologize for inconvenience caused by this issue. We do However have an issue with one of the database servers, it is busy in recovery mode, die to the number of databases this process could take a few hrs to correct its self. So the possibility of websites being off for a short period is going to be an issue. We will get this sorted out asap.

The cables were resolved by the electricians and power is restored back to normal.

Effects - This issue does not effect your relevant products/services
Affecting System - Power to servers
Priority - Critical
Date - 16/02/2022 12:03 - 16/02/2022 19:35
Last Updated - 24/02/2022 14:36

We are busy doing system updates on the hardware infrastructure. Clients might experience loss of connectivity for periods of 5 minutes or less possibly in certain circumstances.

We apologize for any customers experiencing these connectivity issues.

Effects - This issue does not effect your relevant products/services
Affecting System - Operating System Updates / Firmware Updates
Priority - Critical
Date - 07/10/2021 08:16 - 07/10/2021 18:00
Last Updated - 02/02/2022 22:12

We have been experiencing data connectivity issues in the greenside area, this is causing issues accessing the systems in this data center. technicians are addressing the issues, but some clients will experience connectivity issues to some services.

We apologise for the inconvenience caused, but its beyond our control.

Network Issues have been resolved and services are again available.

Effects - This issue does not effect your relevant products/services
Affecting Other - Data Center connectivity
Priority - Critical
Date - 06/10/2021 14:11 - 06/10/2021 16:40
Last Updated - 07/10/2021 14:16

Good day,

We are experiencing load issues on mysql1, technicians are investigating the cause of the issue.

We will keep the information updated as we receive more info.

Technician have identified the offending website, and have shut it down, the database server is back to normal operations.

We apologize for any inconvenience caused by this customer.

Effects - This issue does not effect your relevant products/services
Affecting Other - MySql 1
Priority - Critical
Date - 28/09/2021 20:05 - 28/09/2021 20:17
Last Updated - 28/09/2021 20:17

Please be aware that we will be having load shedding in our area and this will possibly lead to communication issues with phones during this period.  Our phones will be re-directed to support staff however this will possibly lead to lines being busy while existing calls are being handled.

Unfortunately there is nothing we can do to avoid this, please also look at using the support whatsup help option.

Kind regards

Support Team

Effects - This issue does not effect your relevant products/services
Affecting Other - Telephone Systems
Priority - Critical
Date - 10/02/2021 16:00 - 10/02/2021 19:00
Last Updated - 17/09/2021 12:02

We have experienced a critical issue with one of our db servers.

The one Disk drive was swapped and the server needed to be rebooted.

This was completed successfully with around 5 minutes unscheduled downtime as it was a critical issue.

Regards

Support

Effects - This issue does not effect your relevant products/services
Affecting Other - Mysql Db Server 1
Priority - Critical
Date - 16/09/2021 11:58 - 16/09/2021 12:10
Last Updated - 17/09/2021 12:01

We have experienced some issues with emails not being delivered to certain domains, or mails being delayed to certain domains, this was reported by various clients.

After further investigation the issue seems to be a problem on XNeelo's servers in Capetown, We have provided multiple log file instances of this issue to the XNeelo Support staff, whom are currently investigating this issue from their side. They have been able to see there is an issue from the log files we supplied them that there is some issue happening on their servers.

We have no set time of resolution yet from Xneelo. Our support staff are busy with Xneelo investigating this issue further.

We apologize for the inconvenience, but this is beyond our control. We originally though there was an issue on our servers, and this is partly why the servers were updated last night, but the issue was traced down to Xneelo mail servers in the end.

We will keep you up to date re this issue.

----------------------------------------------------

Update - 10/12/2020

We are still busy with Xneelo, this issue seems now it is only on the Captetown Xneelo servers, We have other companies now also reporting similar issues which do not use our mail systsms, so it looks like we are getting closer to isolating the issues.

We are waiting for Xneelo to get back to us with an update.

Effects - This issue does not effect your relevant products/services
Affecting Server -
Priority - Low
Date - 09/12/2020 16:40 - 09/01/2021 00:00
Last Updated - 14/01/2021 11:01

Please be aware we had an issue with dns this morning, this was resolved and normal resumption of services has resumed.

Side-effects: Some emails might have been returned no dns information found. Please resend the emails this issue is now resolved and delivery will be effected normally.

We apologize for the inconvenience, but this was outside of our hands and was resolve as soon as it was detected.

Regards

HostworX Support

Effects - This issue does not effect your relevant products/services
Affecting Other - DNS Issues
Priority - High
Date - 14/01/2021 09:43 - 14/01/2021 11:00
Last Updated - 14/01/2021 11:00

We are busy Upgrading our server infrustructure over the weekend, this will possibly have some outtages for short periods, during this process.This is unavodable, as the time to move data from one location to the other will result in short periods of doentime. this is temporary and normal services will resume as soon as the moves are done.

We apologise for the inconvenience this may cause. The upgrades being done will have many positive influences on your email / hosting and other services. We thasnk you for your patience during this period.

Update : 10 Oct 2020 - The mail servers and all the webservers have been upgraded and moved already,  the database servers will be done in the morning and hopefully normal services will resume around lunch time. there might be short periods of interuptions during this time.

Mail Server upgrades were successfull, and were monitored throughout the day with no issues.

Effects - This issue does not effect your relevant products/services
Affecting Server - SolidCp
Priority - Critical
Date - 10/10/2020 21:03 - 09/12/2020 16:33
Last Updated - 09/12/2020 16:33

We are in the process of upgrading mail server software, on all the mail servers, this is due to the requirement to support TLS 1.3 and 1.2 as, TLS 1.0 and 1.1 have been depreciated. This update is also to resolve some of the delay problems experienced over the last week.

We do not expect any major issues, the upgrades should take around 30 minutes per server.

We apologize for any inconvenience this may cause.

Effects - This issue does not effect your relevant products/services
Affecting Server -
Priority - Critical
Date - 08/12/2020 23:00 - 09/12/2020 16:43
Last Updated - 08/12/2020 23:14

We are seeing Openserve traffic in all regions getting little to no throughput or authentication issues.Our upstream provider and Openserve are looking at the issues currently. We apologise for any inconvenience caused.

Effects - This issue does not effect your relevant products/services
Affecting Server - Fibre OpenServe
Priority - Critical
Date - 11/10/2020 00:00 - 10/10/2020 21:11
Last Updated - 10/10/2020 21:10

Logged Issue : 

We are seeing international and local degradation on adsl and fibre services currently. Our upstream provider is currently attending to this matter. We apologise for the inconvenience. We will update as we get feedback

Update:

Transit provider has confirmed possible under sea cable damage as the cause of the degraded performance on international traffic. They are investigating further, we will update as soon as we have further information.

Update:

Transit provider  has  purchased additional bandwidth capacity on other under sea cables to try and get international speeds back up. This will allow for better connection on international and less latency. And we will not have to rely solely on the repairs of the cables to be done.

Our upstream providers report that adverse weather in the Western Cape has delayed the departure of the ship to effect repairs to international undersea cables. We currently do not have an ETR for final repairs to be completed. We apologise for the inconvenience this has caused, we thank you for your patience and understanding.

Effects - This issue does not effect your relevant products/services
Affecting Other - Undersea Cables
Priority - High
Date - 16/01/2020 11:39 - 18/02/2020 18:09
Last Updated - 18/08/2020 18:09

Good day,

we experienced an issue today with the updated spamassassin rules on one of our servers, a whole amount of mail was miss identified as spam, and quarantined. This was reported and the techs have resolved this issue and the mails were respooled for deleivery and have been resent. This has resulted in delayed emails this morning.

We have checked our other mail servers and they are fine, so it seem this issue was only seen on a single server.

We apologise for this issue and the inconvenience that this has caused.

Effects - This issue does not effect your relevant products/services
Affecting Server -
Priority - Critical
Date - 17/09/2019 13:25 - 17/09/2019 14:28
Last Updated - 17/09/2019 14:28

Issues are being experienced on the mail servers, where mails being received are having the email headers removed from the mails. The mails are delivered to the mailbox, but the sender and subject fields are missing. This is random on certain messages being received via mimecast servers only.  We are currently investigating this issue.

We will be doing some emergency troubleshooting with the software provider, due to timezones this will be taking place from 6pm tonight. We are not sure how long this process will take, as we are dealing with 2 other parties in resolving this issue for you.  We will keep everyone updated with the progress.

Please note mail delivery will be affected intermittently during this process. We apologise for the inconvenience this may cause.

The troubleshooting phase has been completed and we are currently monitoring the issue. There is a backlog of mail which will be spooled out to the mailboxes and we assume this will take around an hour or so.

This issue has now been tracked back to the mail delivery platform deleivering these emails to us, they are currently investigating the issue. The problem seems to be intermittent and therefore it is taking time for them to identify the cause of this issue. 

We expect furether feedback at 14:30.

We apologise for this issue, but unforntunately it is beyond our control as it is not on our systems.



HostworX Support

Effects - This issue does not effect your relevant products/services
Affecting Server -
Priority - Critical
Date - 30/05/2019 10:39 - 31/05/2019 17:52
Last Updated - 06/09/2019 17:52

Please be aware we are experiencing network issues to 2 of our web servers, technicians are looking at the issues. We will keep you up to date with the resolution of this issue.

Effects - This issue does not effect your relevant products/services
Affecting Server - SolidCp
Priority - Critical
Date - 16/05/2019 21:01 - 16/05/2019 22:27
Last Updated - 16/05/2019 21:04

Good day,

We have had reports on one of the webfarms starting at 11am this morning, customers were experiencing network issues with webfarm 16 websites. This was reported and engineers are investigating.

We have been experiencing an issue on web farm 16, we have a patch which needs to be urgently applied. This patch will take a few minutes to apply. We expect this webfarm to be offline for about 30 minutes. The patching will be applied at 11:30.

Patches were applies and normal reumption of services have resumed.

We will coontinue to monitor the situation, we apologise for the interupption, however this was a critical fix that did not allow this to be resolved during normal maintenance at nighttime.

Regards

HostworX Support.

Effects - This issue does not effect your relevant products/services
Affecting Server - SolidCp
Priority - Critical
Date - 12/12/2018 11:14 - 12/12/2018 21:03
Last Updated - 16/05/2019 21:04