A second router, a second link, and still everything goes down the moment one of them fails. Redundancy that does not fail over automatically is a spare part on a shelf. This year's Cybersecurity Month was about staying up when things go wrong, and how your network fails over belongs to that question. The good news: you do not need a complex design. With a few building blocks every router knows, you build a failover you still understand when it breaks at three in the morning.
BGP without your own AS number
The misconception is that BGP is only for those with their own AS numberAS number (ASN) Unique number of a network (autonomous system) on the internet. Needed to pass your own routes to other networks over BGP. and their own IP space. If you have one upstream and use that provider's address space, BGP also works with a private AS numberPrivate AS number AS number from a reserved range (64512 to 65534) that must not appear on the internet. Usable for BGP with a single provider, which strips it before your routes travel further., from the range 64512 to 65534. Your provider strips that number before the routes reach the internet; to the outside world your traffic simply comes from the provider's network.
Why would you? Because it makes a small redundant set-up simpler. If you want to divide your address space yourself behind your own switch or firewall, you announce your subnets over BGP on two sessions, one per provider router. If a router or link fails, the routes via that side disappear and traffic takes the other one. No static routes to update on both sides with every change, and no manual work during an outage. If you later want a second provider, you move to your own AS number.
VRRP for the gateway, BGP for your routes
VRRPVRRP Virtual Router Redundancy Protocol: two or more routers share one gateway address. The master answers it; if the master fails, another router takes over. solves a different problem: two routers share one gateway address, the VIPVIP Virtual IP address that is not tied to one machine but moves to whichever router or server is active at that moment.. The router with the highest priority is master and answers that address; if it fails, the other one takes over. For a server or firewall that only needs a default gateway, that is all you need to configure.
VRRP and BGP combine well. Besides the shared VIP, each router has its own address in the same subnet. You set up your BGP sessions to those own addresses, not to the VIP: a session to the VIP would have to re-establish on every failover. That gives you two independent sessions for your own prefixes, while the VIP stays available as the default gateway for regular internet connectivity.
Deciding which side is preferred
Two active paths are nice, but often you want traffic to normally use one side. For outbound traffic you decide that yourself, with local-pref on your own router. For inbound traffic you have to convince the rest of the network:
- AS-path prepending — you announce your route via the backup side with your own AS number a few extra times in the path. The path gets longer and therefore less attractive, but stays available if the other one fails.
- Communities — with BGP communities you ask your provider to do something with your route. With us you can have a route prepended one, two or three times towards a specific upstream or exchange, have a route announced only to certain parties, or have the local-pref of your route lowered. We agree the full list during onboarding.
- Graceful shutdown — before planned maintenance you tag your routes with
65535:0. We then set the local-pref of such a route to zero, so traffic has already moved before you actually close the session.
Noticing quickly that something is gone
A failover is only as fast as its detection. On its own, BGP only notices a dead neighbor when the hold timer expires, and that can take up to a minute and a half. If a link stops forwarding without the interface going down, for example because of a fault in a network in between, traffic flows into a hole all that time. BFDBFD Bidirectional Forwarding Detection: a lightweight protocol that notices within a second that a link has stopped working, so BGP can fail over faster. fixes that: a lightweight protocol running alongside the session that notices within seconds that the other side no longer answers. Enable it on every session that matters, on both sides.
How we do it ourselves: anycast load balancers
For our own public services we use the same building blocks. The front end consists of three load balancers running HAProxy and BIRD, each announcing the same service address over BGP to our core routers, with BFD on the session. The routers spread traffic with ECMPECMP Equal-Cost Multi-Path: a router spreads traffic across several equally good paths at once instead of choosing one. across every load balancer that announces a route. Customers can run their service behind these load balancers.
A load balancer drops out of the rotation in two ways:
- The machine fails — BFD sees within seconds that the session is dead, and the route disappears.
- The machine is alive, but HAProxy is not — a health check tests every two seconds whether the service answers. After three failed checks it removes the service address from the machine, BIRD withdraws the route and the router stops sending traffic that way. When the service returns, the address is announced again after three successful checks.
We roll out updates one machine at a time. Only when the service answers again and all BGP sessions are back up does the next one go. That way there is always enough capacity during maintenance, and a visitor notices nothing.
Failover with us
Want a redundant connection to our own network? We deliver IP transit with BGP on two routers, with a private ASN or your own AS number, and with VRRP if you also want a shared gateway. What a session looks like technically is in the technical version. If your equipment is in a rack with us, see colocation; if you want the design thought through first, we help with network design. Get in touch and tell us what needs to stay up.