New Automated Server Deployment

When I joined there was an ongoing project to use Ansible Tower (now called AAP) for server deployment, configuration management, and lifecycle management. It has already been going on for 3 years without any end in sight. When I took over I split the project up into phases and started delivering production value. Within 6 months Phase 1 was complete. Phase 2 took another 6 months, and Phase 3 just 3 months, where we started deploying virtual machines with the new process. Phase 4 is now complete, covering Physicals. The project is now considered successful after much delay.

CentOS Upgrade

We were running 3000 CentOS servers, which was going end of life in June 2024. I lead and managed the project to upgrade. We used ServiceNow Audit Tests and our own framework called User Actions to have every server manager and/or owner declare what they wanted to do with them. For some groups of our business, we took a white glove approach to ensure smooth upgrades, for others we just managed the actions they declared (upgrade, decommission, or temporary exception). I took the opportunity to marry our new Automated Server Deployment process By the end of 2024, 2500 servers were upgraded, with the rest finishing up in early 2025.

Department Reoganization

Originally the Linux Group team was composed of 3 "teams", the Transformation Team of 3 members (dedicated to automation primarily), the Operations Team of 10 members, and the DQ team of 6 contractors (primarly charged with manual server upgrades). This was inefficient, so I reorganized them into 4 generally equal teams and moved our team lead from the Tranformation Team to a Global team reporting directly to me. The one Supervisor took the Transformation Team, still of 4 members with 2 moved from the Operations Team, split the Operations Team into the Operations & Automation Europe/Asia (EA) Team and the Operations & Automation Team US, segmented by region, and charged with both day to day operations (incidents and requests) as well as automating their work. One member of the DQ team was moved to the Operations & Automation EA team as well, and the DQ team otherwise remained the same, though more now focused on running automations and fixing incidents.

The teams, and the department as a whole, is now operating more smoothly and efficiently.

GitLab CI/CD Implementation

We had tests of our application, but while we could run them all at once, they were not automated. I took the tast of using GitLab's CI/CD infrastructure coupled with Ansible for the actual running of the tests. The pipeline would spin up a new DigitalOcean server, deploy our OS configuration and Application, run the tests, and send the passes and fails back to GitLab. Now every commit and merge request made could be fully tested and ensure we were deploying safe code.

New Switch Infrastructure

We were running old Netgear switches without any HA so I planned a replacement using proper enterprise switches. I evaluated various vendors and went with Ruckus, which had a good price and also great Ansible integration - a key requirement. These switches had 100G backbone connections, and 25G server connections, which was great future proof for our infrastructure. We purchased 2 core switches and 8 top of rack switches, along with the cables match. After purchasing, we planned the installation at our datacenter. This involved where to install the switches in the rack, what cables/adapters we'd need, and how we would do the migration. From there, we followed the plan, and slowly migrated servers. The migration was a complete success, and the additional bandwidth

Server Upgrades

We were running some pretty old SuperMicro servers with SATA HDDs. It was time to look to modernize our server fleet. We researched a bunch of different options, and settled on new SuperMicro Servers running AMD EPYC processors, that provided lots of PCIe lanes to enable running 24 NVMe SSD drives. We ordered one as a test to see how it would perform with our application, and after successful tests, we ordered enough to replace half hour server fleet. We then began the task of putting them into production, which involved racking and stacking, automating physical deployment of our OS and Application stack, and then switch our application to use them. The whole process was a complete success and the performance of our application increased 10fold.

New Production Infrastructure

We have been planning a migration of our major internal web application (used by everybody in the business from sales agents in the field to accounting) from virtual machines to Kubernetes pods. We have some small applications and all development environments running on two Kubernetes clusters in two self-hosted datacenters, and we were ready to move the major web application. This had a deadline for that of December 31st, 2020, which meant the infrastructure had to be read by October 31st, 2020. We began our project early July 2020.

With this charge, I proceeded to design a brand new production infrastructure. This required deciding on and purchasing new servers and confirming we had enough networking and electrical resources. This became three subprojects involving three teams.

We first evaluated server venders and selected the final brand (Dell) with the server count (22 total, 11 per datacenter). Then we planned the network infrastructure as we had older Cisco switches and would not have enough ports to use all the new servers. We again evaluated switch vendors and selected the final brand (Cisco) and setup (2 core switches with 2 top of rack switches). With the hardware selected, we evaluated the current electrical capacity and the required electrical capacity. We determined we had enough main and generator power, but not enough UPS power. So we selected a new bottom of rack UPS to match the requirements.

Next I presented the entire plan to the VP and after his approval, the CIO. The plan was approved, and we made final revisions with our vendors and placed the order. With everything ordered, I planned the installation for all three subprojects. This involved coordinating delivery of servers, switches, cables, SFPs, racks, PDUs, and UPS's. I also had to coordinate the electrical circuit for the new UPS with both the UPS vendor and our electrical contractor.

I assigned tasks to different teams and different members of those teams.I kept track of delivery of all physical items, as well as installation and configuration of all the devices. Once everything was installed and set up (easier said than down - the teams did a lot of work, and I pitched in to connect cables and install servers), the final set was configuration of servers, switches, UPS's, and PDUs. I supervised this work and helped the teams overcome any obstacles, either by making decisions to move the work along, or jumping in to configure settings and troubleshooting issues.

This took about four months from start to finish and we hit our deadline with two weeks to spare. The new infrastructure was a complete success.

Department Reoganization

Our company is divided up into Service Groups (mine: Business Technology), then Divisions (mine: Delivery), then Departments (mine: SRE), then teams (mine: Networking/Security & Server/Storage).

Originally the Server/Storage team was composed of 6 members, but that size was leading to ineffiencies and slowdowns in coordiations and discussions. So a decision was made to split that team up into a Support Server/Storage team and an Architect Server/Storag team.

This involved selecting the members for each team from the original team, planning out the scope of duties for each team, and communicating to the teams and the rest of the Service Group.

The teams, and the department as a whole, is now operating more smoothly and efficiently.

SAN Addition

We wanted to add fast storage, as our existing solution was slow and support was lacking.

I lead the department that created an evaluation process, evaluated two onsite solutions, and made the proposal to upper management on which solution to select. We selected a Pure Storage FlashArray. This is a highly performant all-flash iSCSI storage solution with an amazing API.

Once the decision was approved, I lead the department that implemented FlashArray. This involved cross team coordination, as it involved both physically setting up the devices at two datacenters, connecting them to our network, and connecting VMware and Kubernetes servers to the devices.

The project was a success and the new storage has opened up a lot of options for our company.

New Infrastructure

Our old ecommerce infrastructure was hosted on Rackspace. This was not a very performant environment, nor was it cost effective. I took a detailed survey of alternative solutions and decided due to cost, control, and performance that self-hosting was the best solution.

I proceeded to design a brand new infrastructure. This required securing a data center and internet service providers. We hosted with Netrality in Philadelphia, and set up connections with Cogent and Hurricane Electric.

Next was to design the network infrastructure. We went with pfSense appliances for internet routing and firewalling. There would be 10Gb connections (all HA'd) to the network switches and servers. There were two sets of switches, one for internal networking and one for iSCSI traffic.

After the network was decided, I planned the server infrastructure. There would be three hosts for virtualization and four database servers, along with a SAN.

I planned the configuration, with Ansible, and the migration. We switched from our old infrastructure to our new infrastructure with zero issues. The new infrastructure was five times as fast as the old one. It also saved us $200,000 per year.

This took about four months from start to finish. The migration was a complete success.

Site Merger

Our company merged with another company, and this required merging our ecom sites. Because the other company outsourced theirs, but used the same platform as ours (Magento), we decided to pull their content into ours.

This required a detailed project plan involving many people. I personally set up the server configurations, and managed the developers who integrated the code and the content (products, inventory, photos, and site UI).

This took about four months from start to finish. The integration was a complete success, with very minor issues.

Website Rescue

I took over running a major retail ecommerce website. The site could barely handle a few hundred users due to the poor configuration of the servers, and it was lucky to stay up most days. It was also insecure. I evaluated all parts of the stack: firewall, load balancer, web servers, database servers, and caching servers.

I optimized and secured each different piece, and added configuration management to the setup. In a month the site was stabilized, and I continued to optimize the configuration and added additional servers as needed. By six months the site rarely went down even under high load. After one year of changes, the site could handle thousands of users during product releases. I also saved hundreds of thousands of dollars and countless human hours. And with the improved security, the site was much less vulnerable.

Firewall Upgrades

I upgraded four SonicWall firewalls in two locations. This required taking the website offline, which is a major deal for an ecommerce website. I configured all four SonicWalls (two each site for HA) to our specification. In one location I removed the old ones, installed the new ones, and ensured everything worked. For the other location which was many miles away, I managed a contractor to do the physical removing and installing, and then I ensured it all worked. I lead this project from start to finish.

Network Reconfiguration

I reconfigured our network for PCI compliance and created 4 different networks in 4 locations (Development, Testing, Production Datacenter 1, Production Datacenter 2. This required reconfiguring Force10 switches, EqualLogic SANs, ESXi hosts, vCenter, and SonicWall firewalls. I lead this project from start to finish.

Switch and SAN Upgrade

I upgraded two Force10 switches (a 1 gigabit and a 10 gigabit one) and an EqualLogic SAN for our Development and Testing Environment. This required configuring both switches differently for the two environments and purposes (iSCSI and VM traffic). I lead this project from start to finish.

Upgrading vSphere from 4.1 to 5.1

I upgraded two clusters from 4.1 to 5.1. This also included creating two new vCenter VMs and migrating the database.

SRM Installation and Configuration

After upgrading and creating two new vCenter VMs, I had to recreate the Site Recovery Manager setup. This involved configuring two EqualLogic pools, setting up SRM on both vCenters, and testing out the recovery plan.

Upgrading all wireless access points from 3COM to Ruckus

All four buildings had a limited 3COM wireless deployment. We upgraded the over the span of a few years to Ruckus. This involved a new controller, configuration of new and old switches, contracting out and supervising the physical access point install, configuration of the controller and APs, and finally testing.

Upgrading all Windows Server 2003 and Exchange 2007 to Windows Server 2008R2 and Exchange 2010

We had physical and VM servers, and I planned a migration of all services (Exchange, AD, File, Print, Financial, Lync). I proceeded to make new VMS and migrate the services, all with minimum downtime and impact to our users.

Reconfiguring entire network

I created a plan to move, upgrade, and retire switches, as well as rewire all the network closets. This was a time intensive task. I removed 43 switches that we no longer needed to due wireless coverage. I moved switches around for better reliability. I upgraded switches for better performance. I reconfigured the entire network for better reliability and performance.