The root causes
On 2 June, between 16h45 and midnight, Orange’s voice services as well as access to certain emergency services were severely disrupted on a national level. During this period, an 11% deterioration in the rate of call routing (or approximately 11,800 calls) to emergency services was observed. The investigation confirms that this was due to a software malfunction.
This malfunction impacted the interconnection between mobile voice and Voice over IP (VoIP) services on the one hand and those hosted on the PSTN switched network (including most emergency numbers) on the other hand. The incident occurred following a network modernization operation that was initiated in early May to increase capacity in response to growing traffic.
The interconnection architecture is based on a platform of “Call Servers”. A bug in the Call Server software provoked a malfunction of this platform resulting in the disruption to the service. This bug was activated following the application of standard reconnection commands, disrupting the overall operation of the Call Servers, despite their redundancy over six different sites.
This software failure has now been identified by the Group’s partner and supplier of the equipment concerned, and a fix has been issued.
The alert process
Orange’s technical teams identified the software malfunction immediately thanks to the internal alert systems. They initiated the first technical operations but were unfortunately unable to restore the service.
The complexity of the malfunction, the variety of technologies involved and the specific network architectures of the emergency services delayed the diagnosis. Once the diagnostic had been established, the problem was resolved within a few hours thanks to the mobilization of around one hundred experts.
Despite the mobilization of the technical teams, the delay in activating the managerial crisis committee resulted in late communication to all stakeholders.
On 3 June, in agreement with the Inter-ministerial Crisis Committee, Orange set-up a dedicated structure aimed at dealing with residual local situations reported by the Prefectures (French regional authorities) on a case-by-case basis. This structure was kept on standby until Monday, 7 June.
Recommendations
The investigation recognizes the immediate involvement and strong mobilization of the Group’s technical teams from the onset of the disruption, as well as the collaboration with authorities in order to restore normal service.
It nevertheless underlined the need for faster dissemination of information to the various stakeholders concerned: public authorities, emergency services and the media.
The investigation by the Group’s Internal Audit department also proposed several concrete recommendations:
- Strengthen end-to-end supervision of critical services and emergency numbers.
- Reduce the maximum time for triggering a crisis committee from two hours to thirty minutes in the event of any disruption affecting calls to emergency services or other critical services at a national level.
- Support State services in accelerating the migration of public service call centers and businesses from PSTN to IP technology in order to strengthen the resilience of this equipment.
- Set-up a dedicated number, available 24 hours a day, 7 days a week, for all relevant stakeholders (State services, hospitals, emergency ambulance services, etc.) in the event of a problem regarding emergency numbers.
- Provide for the use, in consultation with each actor concerned, of a mechanism for the mass distribution of information by SMS in the event of a breakdown affecting the emergency services.
- Pursue investigations with an analysis of the event in comparison with similar incidents in other European countries such as Germany, the United Kingdom, Belgium and the United States.
As Chairman of the GSMA, Stéphane Richard will also propose, at a global level, the creation of a team responsible for listing and analyzing sensitive network malfunctions in order to better share feedback between operators.
This rare event serves as a reminder of the crucial and essential nature of networks. Orange will continue its investigations alongside State services in order to learn as much as possible from this crisis. In particular, the Group will contribute to the investigation led by France’s National Agency for the Security of Information Systems (ANSSI). Orange points out however that the hypothesis that the incident was related to a “cyberattack” was ruled out from 2 June.
Orange is fully conscious of its responsibility and renews its apology to everyone who was directly or indirectly impacted by the incident. The Group remains mobilized and focused on its mission to provide customers with high-quality services.