An update on the new BNIX infrastructure

Published on Wed, 10/27/2021 - 12:44

In 2021 we’ve launched a project to replace the BNIX infrastructure. The motivation to do so was given by numerous factors. The first one is the obvious fact that hardware always reaches an end of life. But the opportunity was taken to do more than a simple copy paste with new switches. We wanted to have a flexible, scalable and easy to manage platform. On top of this an update of the services was necessary and this also resulted in the development of a new portal, based on IXP Manager.

The new switches

At the moment the BNIX platform consists of pure Layer 2 devices and the configuration is setup as a simple broadcast domain with two VLANs: the peering VLAN and the Quarantine VLAN. It is simple and effective, but doesn’t correspond with the modern standard used at other Internet Exchanges.

First we’ve decided to go forward with EVPN and VXLAN as the overlay network to setup the broadcast domains. This means that the link between the different switches will no longer be a trunk but a routed port. This reduces the Layer 2 headaches considerably (loops and broadcast storms, anyone?). Another great advantage of this is the fact that you don’t need to worry about a Layer 2 redundancy protocol or loops when deploying and extending your platform. The redundancy is guaranteed by an IP underlayer such as OSPF, ISIS or BGP.

The choice has been made to use OSPF as the underlay network. This will ensure that all the switches in the network will learn the route towards each other’s loopback address. These loopbacks will then be used to act as a source and destination IP for a full mesh iBGP. It is the BGP signaling that will ensure that we have an EVPN instance all over the platform.

We will no longer work with big switches to fill the demand of our community. Rather, we will stack different 1RU switches in an EVPN instance. Adding devices is quickly done and the architecture gives us the advantage to be scalable and easy extendable to other locations.

At the moment we have tested the design using Juniper QFX5120 32c and 48Y switches. The technology does allow us to setup a hybrid architecture. The vendor agnostic nature of EVPN/VXLAN permits an extension of the BNIX platform with other routers or switches.

To ensure redundancy for a customer connected to a LAGG, multi-chassis lagging is supported via EVPN. The multiple ports will be connected to different chassis, but for the customer this is completely transparent.

At the moment we are facing a huge delay in delivery, which makes it impossible to predict a migration date. The global supply chain is (as you probably all know) a mess. We’ve ordered the hardware in June, but delivery is foreseen for April 2022.

If you should have any questions on the new BNIX infrastructure

Contact us

Did you find this news interesting?