Facebook's 6-Pack Punch

When a billion+ users log on to an Internet service, the underlying fundamental principles of a datacenter has to change. The traditional datacenter was designed to serve thousands of users that can scale upto a million users - but not in Billions!!

Facebook engineers have learnt what it takes to run datacenters that serve billions of users and are Open Compute Foundation.
kind enough to share that knowledge with rest of the world - via


As part of Open Compute Foundation, Facebook recently revealed "6-pack": The first open hardware modular switch. Its a scalable solution - with modular design.

6-Pack is a top-of-the-rack switch, commonly used in "Leaf-Spine" networks for large datacenters. In Leaf-spine network, a rack full of servers are connected to every other rack via a Leaf-Spine network.

6-Pack is also an "open design" Facebook will allow other companies to manufacture and sell these devices. The switch is built from off-the-shelf component & not custom built AISCs - thus bringing down the cost (in a big way) of networking.

6-Pack can scale from 32 ports of 40Gig Ethernet to 128 ports of 40Gig Ethernet - providing a total bandwidth of 1.28T!  It supports two fabric cards, and eight linecards (called as Wedge). Each line card has 16 40GigE ports.

It packs enough punch to change the bedrock of the Internet - the datacenter switches.

For full technical details see: Introducing "6-pack"

Innovation

From a pure switching technology perspective, the 6-pack is not that innovative. The modular & scalable design, the 40GigE ports and fabric cards have been around for years. Cisco Nexus series, Juniper EX series & Arista are established leaders in this space.

In terms of performance, reading from the specs of the device, (I have not seen any real performance numbers) Facebook's 6-pack is on par with the top-of-the-line offerings from Cisco or Arista or Juniper.

The main innovation comes in the internal design. Each line card has a micro-server and a server inside. A Linux derivative Operating System (code-named "FBOSS") drives each of the module.

Each element runs its own FBOSS on the local server and is completely independent.

Herein lies the major Technical innovation. The server inside the linecard and fabric - runs all the network control functions and the switch can be managed as a server - i.,e provisioned & configured using existing server management tools.

Note: No separate/special network management tools needed. 6-Pack/FBOSS does not use SDN technology - like virtual network overlay, but its a hybrid SDN - where all the network switching intelligence is built into the sever.

Running each module independently, allows network engineers to modify any part of the system with no system-level impact, software or hardware. It has a unique dual backplane solution that enabled us to create a non-blocking topology.

Business Disrupter

The other major Innovation with 6-pack is its "Open-design", thus allowing any 'grey-box' manufacturer to build and sell these high performance switches. The open design, Open OS, and off-the-shelf components all comes together to reduce the costs drastically.

6-Pack is a great value proposition for Cloud service providers. Who, like Facebook, run massive datacenters that server millions and millions of users. Unlike Enterprise class switches from Cisco or Arista - which costs $600/port (approx),  Facebook's 6-pack will cost a small fraction - $90 or even less (my estimate) and will reduce further with time.

Since the networking intelligence is built into the software running on a server, the expensive proprietorial protocols and custom built switches can be replaced by 6-pack!

Limitations

Note that Facebook built 6-Pack to meet its needs and not enterprise needs. This means that 6-pack is designed to work with web based service workloads and not the traditional enterprise workloads. Even for the large enterprises, the savings from lower cost/port will be offset by higher cost of integration with existing tools/frameworks, and lack support of support for Open Compute.

This implies that enterprises who do not run 100,000's of identical servers and will not require the scale and flexibility of 6-Pack. Such customers are better off with switches from Cisco or Juniper or Arista or others. But for service providers: AWS, AT&T, BT, Colt, VMWare AirCloud, CSC, etc., Facebook's 6-Pack is a game changer.