Infinera : The Journey of Reducing Power Consumption in Data Centers and Their Interconnect

In a previous blog post, I discussed how data center operators are coming up with creative solutions to decrease cooling costs, leverage server virtualization, and reduce power consumption in other ways.

In this blog, we will discuss the latest technology innovations that allow optical networking equipment (often referred to as 'telecom' in data center lingo) to actively contribute to the reduction of energy consumption. There have been various analyses on the typical percentage of data center power that telecom equipment consumes, but a good estimate is that telecom represents between 10% and 25% of a data center's total power consumption. The exact percentage depends on the size of the data center, its architecture (e.g., number of stages in the spine-leaf hierarchy), and many other factors.

Optical networking equipment is used to connect data centers to other data centers (an application called data center interconnect or DCI) or to the rest of the optical transport network. It has gained a lot of ground in reducing the energy bill. The latest advancements in coherent technology, digital signal processors (DSPs), and intelligent mechanical design have consistently driven down power consumption, reducing it by more than 90% over a six-year period with the introduction and the evolution of compact modular platforms, as shown in Figure 1.

Figure 1: Reduction in power consumption of compact modular platforms

Some of the enablers behind this reduction are briefly described below and depicted in Figure 2.

• A sled-based architecture: The 'modular' aspect of compact modular platforms allows a significant reduction in power consumption because network operators only use the sleds they need vs. monolithic systems that require an extensive set of baseline hardware to operate. The chassis of a compact modular platform typically includes very little baseline hardware, just controller/communication/management hardware that has low power consumption. Each equipped sled adds certain number of watts to the chassis' overall power consumption in a consume-what-you-use mode of operation.
• Low-power 'next-generation' DSPs: DSPs are at the heart of coherent optical networking as they play a vital role in delivering the compute and processing power for mapping data onto different modulation schemes, in addition to collecting and transmitting performance and management information in real time. The next-generation DSP consumes way less power than its predecessors, dramatically cutting the sled's power consumption and reducing cooling requirements (e.g., requiring fewer fans). Photonic integrated circuits (PICs) have also played a role in reducing power consumption. PICs integrate a wide range of optical functions on a single chip. This reduces cost, footprint, and power consumption while improving performance and reliability.
• Adaptive power management: Taking into account inlet and ambient temperature, the modulation schemes used in each sled, and other factors, intelligent software ensures the efficient utilization of power across the different building blocks of the compact modular platform. DSPs consume the 'right' level of power based on the compute requirements for the modulation schemes and operating modes. Fan speed is also optimized based on airflow and heat dissipation.
• Optimized airflow: Efficient airflow has a direct positive impact on how much cooling is required to keep the electronics in the compact modular platform at optimal operating temperature. Inadequate cooling often results in more power consumption, equipment overheating, or a hardware failure. In addition to the adaptive power management mentioned earlier (fans, DSPs), compact modular platforms are designed to be compatible with the cooling schemes used in data centers. Cold air, injected from the raised floor, comes from the front and exits from the back of the compact modular platform, providing an optimum cooling design that leverages the data center's existing 'hot aisle/cold aisle' design (cold/hot air flows).

Figure 2: Enablers of low power consumption in compact modular platforms

But Wait, There's More!

In addition to compact modular platforms, the arrival of coherent pluggables such as 400G ZR (~120 km reach) and 400G ZR+ (different variants based on standards/MSAs, claimed 800-1000 km reach) will also reduce the power consumption of data center networking equipment. These coherent pluggables are designed to provide cost-effective point-to-point connectivity up to 400G in a pluggable form factor that can be equipped in a switch or router (QSFP-DD) or optical networking equipment (e.g., QSFP-DD or CFP2).

Leveraging the recent generation of low-power DSPs, these coherent pluggables provide an attractive option to lower power consumption. For example, a 400G ZR QSFP-DD pluggable consumes between 17 and 20 watts, while a CFP2 version has power consumption between 24 and 26 watts. It's important to note that most current routers and switches do not meet the power requirements per socket to support 400G ZR/ZR+ QSFP-DD pluggables. Nonetheless, the next generation of routers/switches is designed to support these coherent pluggables' requirements for power and density.

A new generation of coherent pluggables, called XR optics, will also further simplify the network and reduce operating costs, including power consumption. XR optics is a superset coherent optical module that can work in point-to-point or point-to-multipoint applications over single- or dual-fiber architectures. XR optics provides finer bandwidth granularity in increments of 25G and additional capabilities such as remote reconfigurability and management and full topology awareness.

So, What's the Right Measurement of Power Consumption in Optical Networks?

Early on in this blog, we highlighted how the different generations of DSPs have significantly reduced power consumption, by up to 90%. The unit of measure this is based on is watts per gigabit (W/G). This is normally calculated by taking the typical power consumption for the fully loaded device and dividing by the maximum capacity.

But does this measurement unit give justice to what each optical solution provides when meeting the various networking requirements, especially distance (reach)?

Adding a New Dimension to Power Consumption - Distance

As discussed in a previous blog, if we limit power consumption efficiency to the enabled capacity, a 400ZR pluggable with power consumption in the range of 17 to 20 watts and a reach of 80 to 120 km might look much more power efficient (equivalent to ~0.0425-0.05 W/G) than an embedded optical engine (~0.2 W/G for a fifth-generation optical engine, such as Infinera's ICE6). However, when we factor in distance, the picture changes significantly. In the best case, a 17 W 400ZR pluggable at 120 km delivers 354 µW/G/km, while in the worst case, a 20W 400ZR at 80 km would be 625 µW/G/km.

By contrast, a fifth-generation optical engine such as ICE6, with less than 0.2 W per Gb/s and delivering 800G wavelengths at 950+ km, has a power consumption per kilometer of around 200 µW/G/km. This power consumption number becomes even smaller as such an optical engine can transmit lower wavelength speeds at significantly longer distances - 100G at 16,000 km, 200G at 12,000 km, 400G at 6,500 km, and 600G at 2,500 km. Adding the distance dimension to power efficiency measurements in optical networks provides a more accurate view.

To wrap up, we're on a journey to reduce and optimize data center power consumption, and technology innovations in DSP design, software, and mechanical design are perpetually driving down the energy bill and creating a greener data center environment.