New Production Infrastructure

We have been planning a migration of our major internal web application (used by everybody in the business from sales agents in the field to accounting) from virtual machines to Kubernetes pods. We hav some small applications and all development environments running on two Kubernetes clusters in two self-hosted datacenters, and we were ready to move the major web application. This had a deadline for that of December 31st, 2020, which meant the infrastructure had to be read by October 31st, 2020. We began our project early July 2020.

With this charge, I proceeded to design a brand new production infrastructure. This required deciding on and purchasing new servers and confirming we had enough networking and electrical resources. This became three subprojects involving three teams.

We first evaluated server venders and selected the final brand (Dell) with the server count (22 total, 11 per datacenter). Then we planned the network infrastructure as we had older Cisco switches and would not have enough ports to use all the new servers. We again evaluated switch vendors and selected the final brand (Cisco) and setup (2 core switches with 2 top of rack switches). With the hardware selected, we evaluated the current electrical capacity and the required electrical capacity. We determined we had enough main and generator power, but not enough UPS power. So we selected a new bottom of rack UPS to match the requirements.

Next I presented the entire plan to the VP and after his approval, the CIO. The plan was approved, and we made final revisions with our vendors and placed the order. With everything ordered, I planned the installation for all three subprojects. This involved coordinating delivery of servers, switches, cables, SFPs, racks, PDUs, and UPS's. I also had to coordinate the electrical circuit for the new UPS with both the UPS vendor and our electrical contractor.

I assigned tasks to different teams and different members of those teams.I kept track of delivery of all physical items, as well as installation and configuration of all the devices. Once everything was installed and set up (easier said than down - the teams did a lot of work, and I pitched in to connect cables and install servers), the final set was configuration of servers, switches, UPS's, and PDUs. I supervised this work and helped the teams overcome any obstacles, either by making decisions to move the work along, or jumping in to configure settings and troubleshooting issues.

This took about four months from start to finish and we hit our deadline with two weeks to spare. The new infrastructure was a complete success.

Department Reoganization

Our company is divided up into Service Groups (mine: Business Technology), then Divisions (mine: Delivery), then Departments (mine: SRE), then teams (mine: Networking/Security & Server/Storage).

Originally the Server/Storage team was composed of 6 members, but that size was leading to ineffiencies and slowdowns in coordiations and discussions. So a decision was made to split that team up into a Support Server/Storage team and an Architect Server/Storag team.

This involved selecting the members for each team from the original team, planning out the scope of duties for each team, and communicating to the teams and the rest of the Service Group.

The teams, and the department as a whole, is now operating more smoothly and efficiently.

SAN Addition

We wanted to add fast storage, as our existing solution was slow and support was lacking.

I lead the department that created an evaluation process, evaluated two onsite solutions, and made the proposal to upper management on which solution to select. We selected a Pure Storage FlashArray. This is a highly performant all-flash iSCSI storage solution with an amazing API.

Once the decision was approved, I lead the department that implemented FlashArray. This involved cross team coordination, as it involved both physically setting up the devices at two datacenters, connecting them to our network, and connecting VMware and Kubernetes servers to the devices.

The project was a success and the new storage has opened up a lot of options for our company.

New Infrastructure

Our old ecommerce infrastructure was hosted on Rackspace. This was not a very performant environment, nor was it cost effective. I took a detailed survey of alternative solutions and decided due to cost, control, and performance that self-hosting was the best solution.

I proceeded to design a brand new infrastructure. This required securing a data center and internet service providers. We hosted with Netrality in Philadelphia, and set up connections with Cogent and Hurricane Electric.

Next was to design the network infrastructure. We went with pfSense appliances for internet routing and firewalling. There would be 10Gb connections (all HA'd) to the network switches and servers. There were two sets of switches, one for internal networking and one for iSCSI traffic.

After the network was decided, I planned the server infrastructure. There would be three hosts for virtualization and four database servers, along with a SAN.

I planned the configuration, with Ansible, and the migration. We switched from our old infrastructure to our new infrastructure with zero issues. The new infrastructure was five times as fast as the old one. It also saved us $200,000 per year.

This took about four months from start to finish. The migration was a complete success.

Site Merger

Our company merged with another company, and this required merging our ecom sites. Because the other company outsourced theirs, but used the same platform as ours (Magento), we decided to pull their content into ours.

This required a detailed project plan involving many people. I personally set up the server configurations, and managed the developers who integrated the code and the content (products, inventory, photos, and site UI).

This took about four months from start to finish. The integration was a complete success, with very minor issues.

Website Rescue

I took over running a major retail ecommerce website. The site could barely handle a few hundred users due to the poor configuration of the servers, and it was lucky to stay up most days. It was also insecure. I evaluated all parts of the stack: firewall, load balancer, web servers, database servers, and caching servers.

I optimized and secured each different piece, and added configuration management to the setup. In a month the site was stabilized, and I continued to optimize the configuration and added additional servers as needed. By six months the site rarely went down even under high load. After one year of changes, the site could handle thousands of users during product releases. I also saved hundreds of thousands of dollars and countless human hours. And with the improved security, the site was much less vulnerable.

Firewall Upgrades

I upgraded four SonicWall firewalls in two locations. This required taking the website offline, which is a major deal for an ecommerce website. I configured all four SonicWalls (two each site for HA) to our specification. In one location I removed the old ones, installed the new ones, and ensured everything worked. For the other location which was many miles away, I managed a contractor to do the physical removing and installing, and then I ensured it all worked. I lead this project from start to finish.

Network Reconfiguration

I reconfigured our network for PCI compliance and created 4 different networks in 4 locations (Development, Testing, Production Datacenter 1, Production Datacenter 2. This required reconfiguring Force10 switches, EqualLogic SANs, ESXi hosts, vCenter, and SonicWall firewalls. I lead this project from start to finish.

Switch and SAN Upgrade

I upgraded two Force10 switches (a 1 gigabit and a 10 gigabit one) and an EqualLogic SAN for our Development and Testing Environment. This required configuring both switches differently for the two environments and purposes (iSCSI and VM traffic). I lead this project from start to finish.

Upgrading vSphere from 4.1 to 5.1

I upgraded two clusters from 4.1 to 5.1. This also included creating two new vCenter VMs and migrating the database.

SRM Installation and Configuration

After upgrading and creating two new vCenter VMs, I had to recreate the Site Recovery Manager setup. This involved configuring two EqualLogic pools, setting up SRM on both vCenters, and testing out the recovery plan.

Upgrading all wireless access points from 3COM to Ruckus

All four buildings had a limited 3COM wireless deployment. We upgraded the over the span of a few years to Ruckus. This involved a new controller, configuration of new and old switches, contracting out and supervising the physical access point install, configuration of the controller and APs, and finally testing.

Upgrading all Windows Server 2003 and Exchange 2007 to Windows Server 2008R2 and Exchange 2010

We had physical and VM servers, and I planned a migration of all services (Exchange, AD, File, Print, Financial, Lync). I proceeded to make new VMS and migrate the services, all with minimum downtime and impact to our users.

Reconfiguring entire network

I created a plan to move, upgrade, and retire switches, as well as rewire all the network closets. This was a time intensive task. I removed 43 switches that we no longer needed to due wireless coverage. I moved switches around for better reliability. I upgraded switches for better performance. I reconfigured the entire network for better reliability and performance.