CubeIO finds robust networking solution using Cumulus Networks’ Routing on the Host, BGP Unnumbered, and Cumulus VX

"

Cumulus VX is great for us. We use it for testing automation....we’ve been able to catch problems in this staging environment that we would not have known about before we went in production. ... And, BGP unnumbered is awesome...Plug in the cable, and you get routing within seconds to the host. The process of deploying a server before could take days, now it’s done in an hour or two.

Kenn Leth Hansen, Lead Infrastructure Architect

Industry

Telecommunications

Business Objective

Build Innovative Mobile Services

Partners

Dell, SaltStack

COMPANY OVERVIEW

CubeIO is a software innovation house, owned by the Danish operator TDC Group, with the stated ambition to develop and architect advanced mobile platform solutions. These innovations are designed to create world-class customer experiences in the telephony and mobile industry — supported by the best network. TDC Group is the leading provider of communications services, mobile devices and TV and digital content. They are the leader in all key markets in Denmark and a strong player in the Nordic geographies.

CubeIO has designed add-on services for mobile network subscribers on mobile plans for TDC Group. These services include visual voicemail, call recording options, interactive voice response (e.g. “press 1 for sales, 2 for support,” etc.) and call queues with everything controlled by an app on the users mobile device. It also includes a switchboard app for use on tablets for receptionists to monitor wait times and route calls to specific groups or individuals.

Their team of app and backend developers focus on creating first-class connectivity options for consumers and corporations. Their operations team of 3, led by Kenn Leth Hansen, Lead Infrastructure Architect, needs to be agile and efficient. “We needed to streamline our environment in operations,” Hansen explains. “Parts of our platform are in the public cloud and parts reside via on-premise solutions for things like handling phone calls to reduce latency.”

In 2018 so far, CubeIO has grown from serving 150,000 visual voicemail users to also carrying the calls of 250,000 users on their platform. Because of this growth for their services, and the fact they are considering in-sourcing everything they host in the public cloud, they needed an agile network they can scale seamlessly.

That is where Cumulus Networks comes in.

AN OPEN, LINUX-BASED NETWORK

With a desire to do routing on the host, CubeIO learned about FRRouting as an IP routing protocol suite that Cumulus helped developed from a Quagga fork for the Linux community. All this piqued his interest in adopting Cumulus.

“TDC Group is a Juniper shop,” says Hansen, “but our systems are built upon a different tool set. All of our infrastructure is based upon Linux, so adding Cumulus Linux into our current Linux environment made a lot of sense, plus it would fit well into our automation strategy.” Hansen found that Cumulus was very simple to integrate into their SaltStack deployment and orchestration pipeline. This made it easy to scale out servers and have them connected. 

“All of the services and features that we offer require calls from the mobile network, and it all travels through our network. We needed an agile solution that could handle our high traffic and high peak points,” Hansen says. “We knew we needed a white label switch offering, and we knew Cumulus whitebox switches would be very cost competitive for us. Cumulus does well in this space by disaggregating the control and forwarding planes. In short, having a cost efficient and simplified management solution for our network is the reason why we ended up choosing Cumulus.”

Getting started with Cumulus was easy for CubeIO. “The main thing for us is that Cumulus is just Linux,” Hansen explains. “The Linux tooling just works. We are used to Linux, so it was easy for us to prototype because of Linux.”

Hansen and his team also leverage Cumulus VX to test Cumulus and their infrastructure. “Cumulus VX is great for us,” he says. “We use it for testing automation ...to make sure we won’t break anything crucial. It is a great test environment to verify configuration changes before we go into production. We found it to be unique compared to other offerings in the market.” CubeIO continues to use Cumulus VX as a staging/test environment when making changes to their configuration environment. “We’ve been able to catch problems in this staging environment that we would not have known about before we went in production. And those could have caused major outages. We look at it as a continuous integration tool in our operations team. It ensures the stability and reliability we demand to make sure changes in the network won’t break production.”


CREATING A MODERN, WEB-SCALE NETWORK FOR THE FUTURE

Besides having this sandbox environment, CubeIO also found other benefits with Cumulus because of the supply chain freedom and simple management they achieved. “We can deploy new servers to our Dell S4148 10/100G switches in one or two hours,” Hansen says. “And, BGP unnumbered is awesome. It is mind boggling. Plug in the cable, and you get routing within seconds to the host. The process of deploying a server before could take days, now it’s done in an hour or two.”

CubeIO’s platform is using ifupdown2 and FRR on their KVM hypervisors with networking on top. This same tooling across servers and switches makes it very efficient for them to manage. “We have full control from L2 to L3 to the application, and that is required for us to maintain our quality of service to our customers using our platform,” Hansen states. “Within 2-3 years, we expect to have our network serve nearly 3 million people. Knowing that, we had to start now to build this new, modern network. We learned the FRR and ifupdown2 tooling within a few months and had already played with it in our SaltStack environment before we ever got the Cumulus/Dell switches.”

In actuality, Hansen believes CubeIO could have gotten a good deal on Juniper switches on their initial outlay. However, having the Linux tooling from switches to their hypervisors, utilizing BGP Unnumbered to simplify connections, and leveraging Cumulus VX for a staging environment made the decision to go with Cumulus Networks a simple one.