The first problem was the trustworthiness of the inventory data. The manual processes for performance analytics and service level agreement (SLA) monitoring were also a major challenge as they were inefficient, error-prone and time-consuming.
The solution proved to be Service Orchestration software, which enabled this CSP to gain control and radically evolve its network. With a single dashboard, the CSP could visualize, monitor and troubleshoot its overall network, accurately automate the SLA process, deliver on-demand service delivery, reduce OPEX and CAPEX, and ensure scalability for a growing network.
Typical Carrier Challenges
This CSP has more than 10 access vendors (AV), and approximately 40,000 circuits in the network. With multiple and inconsistent data sources, flawed network inventory is a major problem in today’s networks.
This CSP is a typical case in that it was using multiple applications to track inventory, IP addresses, AV order history and device configurations. Multiple technicians in multiple teams (for example NOC fault management, assign & design, capacity management, network planning) were designing multiple networks (core transport network, Ethernet backhaul network, radio access network) on multiple spreadsheets, in multiple versions.
The “multiples,” compounded by the fact that there was no central repository, resulted in flawed network inventory and “swivel-chair” visualization - in other words, the technicians looked at multiple screens with conflicting views of the data, instead of a single, accurate end-to-end view.
A common human-error scenario, to which many CSPs can relate, is over-provisioning of the circuit. A technician would set the committed bandwidth to an incorrect value, for example 1000MB instead of 100MB. The inventory system and the circuit are thus not in synch and the data is flawed.
CSPs need to continuously monitor and manage key performance metrics to ensure there are no SLA degradations. The process to monitor the circuits in the NOC was to manually review all the performance stats per circuit and check to see if the performance data aligned to the individual AV’s SLA. This meant that a number of full-time resources were utilized monthly to generate reports to determine if access vendors met their monthly SLAs. The manual approach was inefficient, unproductive, error prone and - for a growing network - not scalable.
End-to-End Service Orchestration: A Single Pane Solution
The CSP decided on an end-to-end Service Orchestration solution that would give them a single pane view of the whole network, and would deliver CAPEX and OPEX benefits.
End-to-end Service Orchestration automates the lifecycle management of Carrier Ethernet and IP services across today’s data network infrastructures, as well as nascent Software-Defined Networking (SDN), and Network Functions Virtualization (NFV) architectures. It delivers demonstrable business benefits - clear ROI with increased revenue, accelerated time-to-market, and significant cost efficiencies due to faster and more accurate fault, performance, capacity and SLA management.
This end-to-end solution spans the CSP’s network build (inventory, ordering, configuration) and management (fault isolation, analytics, SLA management) functions. All information necessary for the end-to-end definition of the Ethernet Virtual Circuit (EVC) is stored in a central repository model, allowing the CSP to see what parts of the EVC are provided by an AV, what Network-to-Network interfaces are used to interconnect an AV with the CSP, any microwave hops - if the microwave has a donor site that branches out to other sites - and where there is a backup path.
There are two main components of this CSP’s solution. The ‘Continuous Data Audit’ analyses data from a total of 12 data sources, enforcing more than 100 audit rules to ensure that inventory, ordering, IP addresses, network element configuration files, planning spreadsheets and data sources are accurate.
Secondly, the “Service Visualization” component provides one single screen to view inventory, ordering and monitoring data associated with the primary and backup paths, to each mobile backhaul site. Here is an example screen showing a sample path view (not specific to this CSP):
ELM accurately automates the SLA process. All circuits are instrumented so that the performance of each EVC segment can be measured, including off-net segments. ELM also determines if a segment’s performance is affected by a failure up-stream or down-stream of the circuit. If the problem is not caused by that segment, then ELM puts all affected segments in a dependency mode and stops tracking availability against that segment until the offending segment or network element issue is resolved. This way, the AV that provides the off-net segment is not penalized for the low availability when the root cause of the traffic unavailability is not their fault.
By automating the SLA process and ensuring SLAs are measured accurately and fairly, the CSP is now able to effectively manage and enforce its AV SLAs and collect SLA credits when applicable.
Putting It Into Practice: The Solution Rollout
To kickoff the deployment phase, CENX facilitated a discovery workshop with the CSP’s network operations team. The resulting business case showed that CENX’s Ethernet Lifecyle Manager (ELM) software would solve the CSP’s challenges, without having to replace existing systems, and it would deliver a good ROI based on reduced CAPEX and OPEX.
The first step toward deploying Service Orchestration was an audit of the CSP’s network data. Upon completion, ELM could visualize the backhaul circuits and integrate the performance data so that the CSP could better manage the performance of the circuits and then start to enforce SLAs. The next development phase was to extend the performance management to the core network so that the CSP could monitor the performance of the links across its core MPLS network.
Finally, CENX conducted an in-depth audit of the entire network. The CSP had millions of dollars invested in existing systems which did not require replacement. The CENX Professional Services team deployed and integrated ELM into the legacy infrastructure, and provided training for end users.
CENX drives customization and deliverables throughout the engagement process. This approach accelerates rollout and ensures full integration into the NOC’s OSS environment, which in turn accelerates reduction of OPEX.
Today, NOC personnel have easy-to-use visualization, automation and management tools at their fingertips, which allows for a continuous audit of multiple data sources in the network against what is designed and ordered at each site.
The CSP now has an accurate, central repository of the Ethernet backhaul inventory and topology, and a single dashboard to visualize, monitor and troubleshoot the overall network down to each circuit router, cell site and on-net/off-net segment.
In actual figures, this means an increased success rate of first-time turn-ups by 65% (from 25% to 90%), which reduces OPEX and accelerates time to revenue; a 300% automation improvement of the SLA process, which increases productivity by 300% and enables enforcement of SLA contracts with access vendors; and a 67% improvement in SLA process automation, reducing OPEX by 67%.
Looking forward, this CSP expects additional significant OPEX savings by managing bandwidth upgrades more frequently, in small increments versus every 3 years, and a future-proof strategy for bandwidth-on-demand and the transition to next generation software-defined networks.