No doubt about it: the prospect of adding another zero to the end of your top network speeds is exciting. And the reward of the immediately noticeable performance improvement never gets old. Speed makes a noticeable, and not just measurable, difference. And with the massive increase in the amount of data servers need to process, 100G is soon going to be a necessity for many organizations.
But increasing network speed is about more than pushing more bits across a wire. Faster networks enable you to squeeze more out of your physical rack space. You need fewer servers, fewer network connections, and – dare I say it – fewer switches. It’s true. A faster network lets you pack more computing into the same space.
Whether you plan to do a forklift upgrade to 100G or intend to replace one switch at a time, there are some key things you need to know to avoid getting locked into one switch vendor or losing backward compatibility with your existing equipment. In this post, I’m going to give you my top 5 tips for making transitioning to 100G networking a smooth one.
Tip No. 1: Decide where 25G/100G makes sense
First, a little background. A single 100G connection can be broken out into four channels or lanes of 25G each. This is why 100G networking is sometimes called 25G/100G. Generally, you’ll run 100G connections between your switches, and 25G to your servers. This is no different than 10G/40G networking, where you’ll run 40G between switches and 10G to your servers. It’s also common to have 40G connections to a blade chassis to provide enough bandwidth for all the blades.
But upgrading your network to 100G doesn’t mean that every server has to support 25G right out of the gate. Decide where 25G makes sense. If you’re running 10G to your rack servers and you’re not bottlenecking, then upgrading those links to 25G today isn’t going to buy you anything. It may make sense to just stick with 10G for those servers, at least until your next server refresh.
If you’re thinking of jumping from 10G to 40G to the server, 25G is probably the better choice. A single 100G port can be broken up into 2x50G links, giving you the option to connect two servers at 50G each. You connect the same number of servers, get more bandwidth, and use fewer ports. It’s a win-win! Contrast this with running 40G to two servers. You’d have to consume two switchports instead of one, and you’d still be getting less bandwidth.
If you’re running 10G and need more bandwidth, but think 40G is overkill, you may have also considered adding additional 10G connections. If you have plenty of switchports to spare, this may be a reasonable option. However, if you don’t have many ports to spare, you can swap out every 10G connection for a 25G connection without consuming an additional switchport.
From a rack space perspective, 100G/25G is more efficient. You don’t need to make more room in the rack or move things around. Just replace your 10/40G switches with 25G/100G and do a one-for-one exchange.
The bottom line: use 25G/100G where you need more bandwidth and more flexibility in the rack. Stick with 10G/40G where you have the capacity to run additional connections without having to buy more switches.
Tip No. 2: Make sure your 100G switches are backward compatible with 10G/40G
Chances are you’re not going to do a forklift upgrade to 100G, replacing your servers, switches and cabling all in one go. You’ll probably still have some 10G/40G connections that need to remain in place during the transition. To support your 10G/40G connections, your new leaf switches will need to support your existing QSFP+ and SFP+ modules. When shopping for new leaf switches, look for switches that accept the QSFP28 form factor, which is physically capable of taking existing QSFP+ modules. If you have SFP+ modules you want to use, make sure that your new switches also accept the SFP28 form factor.
Keep in mind that although a switch that accepts the QSFP28 form factor can physically accept a QSFP+ connector in the same port, it doesn’t mean that your QSFP+ modules will work. The same holds true for SFP+ modules and SFP28 ports. Carefully read the product specs to ensure that your existing modules will work with your new equipment.
Also note that QSFP28 and SFP28 ports cannot support gigabit SFP modules, so if you’re still doing gigabit to the server, make sure that the leaf switches you buy have enough SFP ports to accommodate you.
The bottom line: Make sure your switches have plenty of QSFP28 and SFP48 and can do 10G/40G.
Tip No. 3: Make sure your network adapters support 25G/100G
This one may seem obvious, but not all servers are shipping with 25G network adapters. On your next server hardware refresh, make sure that you get adapters that can support 25G/100G. The good news is that this doesn’t mean giving up 10G/40G; adapters that can do 10G/25G/40G/50G/100G are already out there. That means you can stay at 10G today, and transition to 25G whenever you’re ready. This gives you the flexibility to refresh either your switches or your servers first.
The bottom line: avoid network adapters that only do 10G/40G.
Tip No. 4: Stay away from early model 100G switches and modules
Many 100G switches came out before the current 25G/100G standards were ratified in 2016, and as such they don’t always interoperate with each other. Even if you find a fire sale and buy early models that are compatible with each other, be careful — when you do a hardware refresh later on down the road, you may find that the early models aren’t compatible with newer switches. At that point you’re looking at a forklift upgrade.
To make matters worse, because some of the early switches came out before the current 25G standards, they may not even work with the 25G network adapters shipping with new servers.
The bottom line: Stay away from switches that were released before the current 25G/100G standards were adopted.
Tip No. 5: Know your reach
The distances at which you can get 100G don’t differ too much from 10G/40G. With twinax, you’re limited to 5-10 meters, while multimode fiber will get you up to 100 meters. If you need to reach beyond that, single mode will give you up to 40 kilometers.
But there’s more to keep in mind than just distance. If you need long reach optics, some switches limit you to specific ports. Also, some switches will limit how many 100G ports you can break out into 4x25G or 2x50G connections. These limitations can affect how much density you can realistically achieve.
The bottom line: again, carefully read the product specifications to find out the “gotchas” associated with using breakouts and long reach optics.