A journal of going straight to the source

It’s quite rare to see a big name cloud provider ask a startup company to help design their next-generation cloud networks. We’re fortunate to be involved in just that situation. The customer’s infrastructure requires L2 circuits in a physically large data center to peer platforms up to 1 km away; implemented in a future-proof and cost-effective way. This journal focuses on the physical interconnect portion of the solution.

It wasn’t an easy job, but the Cumulus Networks team completed the task in a week and a half, from feasibility research all the way through validation. Many folks were involved in this journey and enjoyed it so much that we wanted to share the fun experience with everyone.

Day 1: The Use Case

Based on a large set of factors, the customer had chosen 32x40G switching hardware, leaving us with the challenge of how to cost effectively connect to peering platforms via 1G or 10G ports over that non-trivial distance.

Well, with all that, the obvious solution was a QSFP+ parallel single mode fiber transceiver. A cursory search revealed that a few name brand network system vendors and optics resellers/re-labelers carry these devices, but we wanted to see if we could find something equally robust but more cost effective.

Day 2: The Battle Plan – Go Straight to the Source

Wait, what? Parallel single mode fiber optics and the corresponding break out cables, from reliable sources and also cost-effective? These two characteristics typically don’t go together in legacy networking world.

Without much debate, the team consensus came to one direction: Go Straight to the Source. But it’s easier said than done.

  • Who and where is the source?
  • Are they tried and true?
  • Are their products of excellent quality?
  • Are they capable of fulfilling in volume?
  • And at a reasonable price?

With those questions in mind, we started the hunt.

It’s amazing what a team of committed folks can do in an afternoon (and well into the evening). We basically called all relevant contacts, checked every possible website, and read all the documents we could collect.

Day 3: Found One!

Finally, a possible supplier emerged: InnoLight Technology Corporation. They are based in China, and we hadn’t heard of them prior to this exercise. We called one of their founders immediately, received all the technical specs within minutes, and started the technical due diligence.

Surprisingly, the optics and breakout cables are well designed and exactly what we needed; the pricing was also extremely reasonable — $450 each, more than six times less expensive than a similar part from a name brand vendor! The difference was so large that we didn’t believe what we heard and asked three different times to verify the exact price. We immediately ordered two optics (TR-IQ13C-N00) and couldn’t wait to try them out.

Day 4: An Unexpected Meeting

While we waited for the units to arrive, we wanted to further vet the supplier, especially their business credibility and fulfillment capability.

As luck would have it, their founder was in the Bay Area that week, so a face-to-face meeting happened immediately.

It turns out that InnoLight is a very established manufacturer with about 1000 employees and many years of experience supplying several large US customers and OEMs. They’ve been behind the scenes the whole time!

One interesting fact is that Innolight manufactures every parallel single mode QSFP+ we found in the market.

Day 5: Let the Testing Begin

Whatever we recommend to our customers, we want to make sure it’s rock solid and works perfectly in real life. So David from our engineering team started to build a mockup of the customer’s environment with spools of fiber cables, optical patch panels, 1000BASE-LX/10GBASE-LR optics, and the selected switching hardware.

Environment

The switch, fiber distance, segments and number of insertion/patch points were purposely selected to match the deployment environment.

Day 10: It Works!

With the test bed already constructed, when the optical modules arrived, we immediately plugged them into the test bed to verify on all fronts. They worked perfectly. We also ran overnight traffic testing and encountered no errors at all!

Cumulus Fiber

The customer was very happy to hear all the test results.

Day 11: Cross the T’s and Dot the I’s

To make sure the test result is fully trustable, we added attenuators in both directions and did another round of traffic tests. No errors. Also performed optical loss measurements on all segments to ensure the total loss is close to the specified budget.

Now we are confident that this solution will work in the real environment.

Day 12: Speaking of “Cost-effective”

It’s worthwhile to look at how cost effective this interconnect solution is.

Name brand networking equipment vendor Optical resellers/ re-labellers InnoLight Optics
$MSRP/per piece for Parallel SMF Optic (~1km) $2,994 $1,044 $450

Various suppliers will discount at different levels but we’re starting at a pretty large ratio: 6.6:1. Even the discounted optics from online resellers were more than twice as expensive as InnoLight’s.

A typical cloud deployment involves hundreds if not thousands of switches, but let’s use just 20 switches and 20 ports utilization/per switch as simple numbers for now. The saving on optics, without compromising any quality, is significant:

20 x 20 x ($2,994-$450) = $1,017,600

You may wonder why the price from that name brand vendor is so high. Well, unfortunately that’s the game that has been played for a long time. It’s common to see egregious mark-ups in the name of compatibility/quality and sometimes even hilarious pricing schemes just to push proprietary solutions. For example, the list price for Cisco’s proprietary QSFP+ 40G BiDi optic is $1,095 and their list price for the industry standard QSFP+ 40G SR4 optic is $2,995. Given that there are various sources for QSFP+ 40G SR4 optics at around $250 list, it is certainly NOT because the BiDi optic is cheaper to make.