This post is at least half inspired by the massive force of water that descended on the Chicagoland region in the summer. What I realised very quickly is that the roads and the water drainage systems were not designed to cope with this at all. Some areas were coping extremely well with the traffic and the rain whereas others were failing absolutely; the end result being complete failure. In fact I ended up wading – shoes and socks in hand – across the car park to work; but that’s another story entirely.

Aside from the obvious parallel here I had to start to think why, to quiz myself on what went wrong. The result was obvious. Some failings somewhere had caused a patchy outage for the roads which resulted in complete failure of the roads as a network. Specific spots on the road had been designed terribly. For example one express-way dipped down low enough that it became a small lake limiting three lanes of traffic to one lane going through about a foot of water. This meant the rest of the road was held up by this single spot. No redundancy and poor planning had resulted in an outage.

Other spots were causing blockages along my route, small blockages which created massive delays. This is not unlike many corporate and office networks. We build our networks around the exciting and memorable things but forget the details. Your firewall throughput of 10GB/s may be amazing but the IPS everything flows through afterwards is acting like that small lake on the express-way. Many networks are built over a long period of time when standards change and get rebuilt. The classic example is the dual-mode wireless router which gets held at 802.11G because of a printer which doesn’t support 802.11N. Although I may well be preaching to the choir with a lot of my readers – given my most followed links go to Cisco.com – I’d still remind that a poorly placed printer or IPS can ruin an otherwise well designed network.

Some ideas:

• Place older IPS systems to the side and have them direct their actions to newer routers or 3L switches.
• Attach wireless printers to a separate wireless network, or put them on a print server. The separate wireless allows you to change and upgrade security even if they don’t support it. If at all possible I’d have them wired on a separate server for safeties sake.
• Put IPS’s on the inside of your external firewall and (if you must) an IDS on the outside. As terrible as this is I read the advice to put IPS’s on the outside of your network in a textbook recently, this is silly because you only want the IPS action to hit traffic that already made it to your network. If you want a more SIEM view of the outside then use an IDS because its passive monitoring won’t slow down flow.
• Review your network’s structure at least once a year.
• Confirm how much data you can actually move over your network and where the blockers are. Test this regularly.
• Redundancy, redundancy, redundancy. At least have a spare – configured – router to the outside world (or whatever your most important point is).
• If you work with a small budget, as many of us do, it may be worth segregating networks to two standards. Build a modern network based on the best security possible and have a legacy network for the equipment you can’t replace yet but doesn’t support all the security.
• Confirm that your performance related settings are saved to the devices, there’s nothing more embarrassing than a configuration that worked really well before the last power cycle wiped the RAM.

Those who know me have already heard the redundancy spiel but it really does save the day when something goes wrong.

Regards,
Robert Small.