Load Balanced Web Servers

While working as a Linux administrator, I had the task to build a clustered environment to host our WordPress servers. As a financial publisher had a subscriber base of ~250k active subscribers. We hosted marketing campaigns that would advertise live broadcasts and other traffic generating events. With an average price of several hundred dollars for each subscription it was critical that we keep the sites alive and active 24/7.

A contractor was hired to build a redundant clustered environment. They selected RedHat and attempted to use the built in cluster services offered at the time. This proved ineffective. Problems they faced included an inability to keep the systems running independently. Synchronizing content was difficult and the system could not survive the loss of a cluster member.

I was paired with a WordPress developer and together we designed a new solution. Spoiler: it was a success and proved flexible, reliable and performed well given our constraints.

A team of us worked together to select and implement a Citrix Netscaler load balancer. This replaced a pair of Barracuda load balancers. We chose the Netscaler with hopes that we would take advantage of application specific routing and intellgent routing.

The servers were all virtual, living on a VMware cluster which used an Equalogic SAN for storage. The environment was located in a Peak 10 datacenter. I was responsible for the Linux system administration, some VMware administration, Security and most of the network administration and maintenance.

Each web server ran Centos, Apache and Nginx. The web developer managed Nginx and Apache and I helped with the troubleshooting.

A web server ran Nginx on 80/443 which communicated to Apache. The live /var/www was mounted from the the data server. A backup of content was copied to /var/www via rsync during maintenance. This meant the sites would still be partially functional if the NFS server disconnected. User session data was stored in another directory so that their experience was consistent if they were moved to a different server.

Log files were shipped to a Splunk instance for troubleshooting and monitoring. I used SAR, SolarWinds Orion, SNMP and syslog to monitor the systems.