Issue: Google services slow: Solved

Dear customers,

We are currently experiencing an issue with reachability towards Google.  Our Network Operations team is investigating this issue, a ticket at Google has been created.  Further updates will be released as soon as possible.

Please find all details below:

Start: 2017-11-12 11:00*
End:  2017-11-12 23:00*
Duration: 12 hours
Impacted services: Google reachabilitly
Affected Areas: All

Update 13/11/2017 10:30 : Issue was resolved yesterday evening around 23:00. We are talking with Google what the exact cause was.

We apologize for the inconveniences.

Best regards
The edpnet team

*All times are listed in CET, Central European Time


Info: Wireless network WPA2 vulnerability

Dear customers,

We would like to inform you about the recently noticed WPA2 vulnerability.

AVM analyzed the wireless security breach known as “Krack” which allows an attacker to perform a so-called man-in-the-middle-attack on clients who are trying to connect to a wireless LAN network. However, they consider their devices safe and stress out that the possibility of such an attack actually happening is quite insignificant since there are very specific conditions that must be met by the attacker to do the attack.

More information can be found on the following support site article:

https://www.edpnet.be/en/support/troubleshooting/internet/wireless-network-wpa2-vulnerability.html

If you have any questions left, please do not hesitate to contact us.

Best regards
The edpnet team


Info: Spam/Phishing

Dear customers,

We would like to inform you about a spam wave that is coming from @edpnet.be domain aimed at both customers and non-customers. The source seems to be a botnet.

The email contains the subject “Server Violation”/”Emailed Invoice” and an attachment/spam link.

Please do not reply to those emails and do not open any files or click on any links. For more information, please, consider reading through our support article on this subject

https://www.edpnet.be/en/support/troubleshooting/internet/phishing.html#edpnet

If you have any questions left, please do not hesitate to contact us.

Best regards
The edpnet team


Outage: Telephony: Solved

Update: 2017-29-09 14:10: We confirm that the capacity issues have been solved. Small maintenance works will be conducted during the weekend.

Update: 2017-29-09 11:50: We are experiencing issues with the capacity due to yesterday’s outage.  Our Network Operations team is working together with Voice System Supplier to resolve the issue.

Update: 2017-09-28 16:40 : The outage has been resolved. All SoIPs are back online, traffic flow is normal and the transmission issue has been cleared.

Update: 2017-09-28 16:15 : Our SoIP2 is back online, the telephony traffic flow is getting better. The transmission issue with Proximus is still persisting, we are working with them to get it solved asap.

Update: 2017-09-28 15:15 : Our active servers reached capacity due to a transmission issue with Proximus we encountered. Our Network Operations team is working to provide a fix. Works on SoIP2 are also still in progress.

Update: 2017-09-28 11:50 : Network Operations team is working together with Voice System Supplier to resolve the issue on our SoIP 2. Traffic has been rerouted towards our active SoIPs 1 and 3. Channel capacity issue has been solved. Database repair is in progress.

Update: 2017-09-28 11:20 : Network Operations team is looking into a fix to improve the outgoing/incoming traffic flow on our active SoIPs.

Update: 2017-09-28 10:50 : Network Operations team has made some adjustments to reroute telephony traffic towards our other servers, however, due to the change, we reached a capacity issue of our other SoIP servers.

Dear customers,

We have experienced an outage with our Telephony services due to an issue with one of our SoiP servers.

Please find all details below:

Start: 2017-09-28 09:51*
End: 2017-09-28 16:40*
Duration: 6hours 49minutes
Impacted services: Incoming/Outgoing Calls
Affected Areas: all

 

We apologize for the inconveniences.

Best regards
The edpnet team

*All times are listed in CET, Central European Time


Outage: Inbound Telephony in The Netherlands: Solved

Dear customers,

We have experienced an outage concerning inbound telephony in the Netherlands. Our Network Operations team has provided the solution.

Please find all details below:

Start: 2017-07-26 10:30*
End: 2017-07-28 11:10*
Duration: 1:21:40:00
Impacted services: Inbound telephony
Affected Areas: all Netherlands voip service

 

We apologize for the inconveniences.

Best regards
The edpnet team

*All times are listed in CET, Central European Time


Update: Outage Closed: KPN link for fiber is down

Update: Outage cleared

Dear customers,

We are currently experiencing an outage with our KPN link for fiber. The outage is currently under investigation by our NOC team. Further updates will be released as soon as possible.

Please find all details below:

Start: 2017-01-26 10:38
End: 2017-01-26 10:56
Duration: 18 minutes
Impacted services: NL Fiber
Affected Areas: Amsterdam

 

We apologize for the inconveniences.

Best regards
The edpnet team

*All times are listed in CET, Central European Time


Outage: NL internet services: Solved

Update: Our services will be back online within 30 minutes

Dear customers,

We are currently experiencing an outage with internet services in the Netherlands. The outage is currently under investigation by our NOC team. Further updates will be released as soon as possible.

Please find all details below:

Start: 2016-12-29 15:30
End: 2016-12-29 16:20
Duration: 50 min
Impacted services: Internet services
Affected Areas: NL

 

We apologize for the inconveniences.

Best regards
The edpnet team

*All times are listed in CET, Central European Time


Info: Overview of the latest outages and solutions

Dear customers and partners,

As you all know, edpnet has gone through a number of serious outages lately. The initial power outage exerted a huge impact on our core infrastructure which caused, unfortunately, much tension to our customers and ourselves. We understand that it is hard to justify the confidence in such circumstances and want to apologize for all inconvenience. Strong measures have already been taken to reduce a risk of further outages and guarantee the stability of our services.

As open communication is very important for us, we would like to provide you with a detailed overview of what exactly happened and what we have done/are going to do to prevent these situations in future and improve our service. Please, find the overview below.

Monday 26/09/2016 4:49: the data center of edpnet Sint-Niklaas lost power, UPS system and power generator failed.

Monday 26/09/2016 10:19: the power was restored, systems were back online. Due to the many issues to get the power online, we provided a backup power feed from our office power feed to the most critical systems

Monday 26/09/2016 11:00: mail load balancers were dead, probably due to intermediate power cuts. These devices were replaced.

Monday 26/09/2016 20:00: the power was lost again in the data center, critical applications such as voice services and main router were still running. The interruption caused an issue on incoming calls.

Monday 26/09/2016 20:15: The root cause of the power issue was found, the UPS itself caused the power cuts. We bypassed the UPS and restored power.

Monday 26/09/2016 21:30: incoming calls were running fine again.

After the power cut, we noticed some strange behavior on our Voice switch 2, several maintenances were performed to have the system stable again, no more issues were noticed after these maintenances.

Sunday 2/10/2016 23:30: a DDoS attack starts which causes connectivity problems.

Monday 3/10/2016 9:30: The DDoS target is found and disabled. Connectivity restored.

Tuesday 4/10/2016 0:00: Planned maintenance to replace the broken UPS, and installing an external bypass system so future UPS maintenance shouldn’t give an interruption.

Wednesday 5/10/2016 7:44: A database issue occurred on Voice switch 2.

Wednesday 5/10/2016 8:06: The database is restored, voice is operational.

Thursday 6/10/2016 1:46: Voice switch 2 crashes, is completely unavailable. Voice switch 1 isn’t taking over causing voice calls not to arrive, and outgoing calls not to be made.

Thursday 6/10/2016 6:43: manually switched all voice traffic towards Voice switch 1, incoming and outgoing calls seem to work fine.

Thursday 6/10/2016 9:00: complaints arrive that no incoming calls are possible again, it seems Voice Switch 1 only receives 50% of the incoming calls

Thursday 6/10/2016 10:57: restored Voice switch 2 on spare hardware, all calls are arriving no more voice issues.

Thursday 6/10/2016 11:30: Proximus and supplier found the issue with the failing redundancy of the voice switches, a new maintenance will be planned soon to make the necessary improvements.

List of improvements to be made:

  1. UPS replacement (done)
  2. External bypass system to avoid outage due to UPS works (done)
  3. Move certain critical services to our data center in Interxion Brussels (ongoing)
  4. Install a permanent B-power feed in our data center in Sint-Niklaas (ongoing)
  5. Adapt voice service for correct redundancy (done)

 

If you have any questions or remarks left, please, do not hesitate to contact us.

Best regards
The edpnet team


Outage: Voice services (solved)

Dear customers,

We have experienced an outage with incoming voice traffic. The outage has been solved.

Please, find the details below.

Start: 2016-10-06 01:46*
End: 2016-10-06 10:57*
Duration: 9:11
Impacted services: All voice traffic
Affected Areas: all areas

Update 11:26: all voice services are restored and are fully operational. There are still issues with redundancy, but we are working on this with the highest priority. No further unannounced voice issues should occur.

We apologize for the inconveniences.

Best regards
The edpnet team

*All times are listed in CET, Central European Time


Maintenance planned on 04/10/16 – impacted services: cloud telephony and colocation service

Dear customers,

Edpnet has performed a maintenance last night in order to install new UPS-device in our data-center in Sint-Niklaas. This has been done to avoid power loss in the future.

This maintenance is finished successfully.

Please find all details below:

Start: 2016-10-04 00:00 AM*
End: 2016-10-04 02:00 AM*
Duration: 2 hours
Impacted services: Cloud telephony and colocation service
Affected Areas: all

 

We apologize for the inconveniences.

Best regards
The edpnet team

*All times are listed in CET, Central European Time