Network Monitoring Server

While working as a Security Administrator, I was responsible for network management and monitoring.

Our main office had a handful of Dell 6248P switches connected to a Cisco Catalyst 7501 core switch. Mostly flat with no segmentation, the servers workstations, voip and web traffic competed for resources.

The issues we faced were complaints about speed, voice quality and general instability.

Dr. Octopus was born. This was a Linux based monitoring server I built out of spare parts. The system was a dual Xeon server with whatever ram I could scrounge up and a few 4 port network cards. (foreshadowing here…)

I mirrored the uplink port on each switch to the next adjacent one. This was then connected to the Dr. via one of the multiple ports. The software intercepted processed and analyzed the traffic.

Running CentOS, an Iftop, rrdtool and prtg on the box gave me the ability to monitor conversations in between endpoints and I could identify anomalies in near real time.

Here’s where it went all wrong…. The network was unstable to begin with and this made it completely worse. I failed to think about how the network connections could cause problems. Every few hours, the entire network would stop for 3-5 seconds then get back to working without issue.

It turns out that the Cisco switch was seeing the same mac address on multiple uplinks and the log files would start screaming about it and then a few seconds later spanning-tree would kill everything thinking there was a loop in the network. Once I discovered this, the project had already gained a bad reputation and I was forbidden from resurrecting the Dr.

If I could, I’d have simply changed the mac addresses on each port on each nic. Monitored spanning-tree for errors and maybe spent a bunch more time learning about how to configure the Cisco better.

Addendum: Why didn’t I just use Netflow, monitor all the conversations from there? Well, it was around 2003 and the switch was a late 1990’s model which required a separate Netflow card and an IOS upgrade. Did I mention that the switch did not have an active support contract? Later, it ended up making a nice side table in my office for a while.