Central Monitoring Server

One of my directives as the solo Security Administrator was to “monitor everything” and “know everything” that happened on the network. (thanks Tony).

The company and network needs to be described a bit for perspective.

4 geographically diverse buildings plus a pair of racks hosted in a datacenter. The corporate office housed ~100 people with various roles. Application and Web developers, TV Studio, Marketing, Inbound and Outbound sales, Electronic fulfillment, Copywriters, Financial analysts and even a Financial Management company complete with SEC guidelines to follow.

The datacenter served as the central hub, most of the servers lived in a 4 node VMware cluster or sat beside that environment in the racks. The corporate office had another small VMware cluster that housed domain servers, redundant dns servers etc. All critical services actually lived in the DC. The corporate office was connected via Metro Ethernet, as was the Money Management office and the owners house which he used as a satellite office. The network was mostly flat with very little segmentation. From any location you could connect to nearly everything except for the web DMZ. (eek!)

I configured every possible device to send SNMP or Syslog to a pair of Splunk servers housed in our datacenter. This server also received index data from another server located in our corporate office. This configuration allowed me to quickly search logs, find historical data. This was the beginning and long term plans were to stand up dashboards for the various roles of systems we needed to monitor. For example, a phone system dashboard that would monitor all of the Exchange/Lync/ Skype for Business servers. Another for the 4 Bind servers we used to serve external dns for our web servers.

In it’s infant form I used this system to investigate, identify and document an internal security investigation. As part of my Security Administrator role, I was asked and able to prove that a team was using company resources to start their own business in conflict and competition with the organization. Print and firewall logs identified the behavior.