Docker EE Upgrade

Executive Summary

Foxcorp (FOX) is a leading news, sports, and entertainment company, with its complexity around users and services being astounding. FOX heavily uses Docker Enterprise (Docker EE) to manage crucial services 24/7, with each environment having its own Docker EE cluster. This extensive usage comes with a price: the inability for any cluster to go under maintenance or upgrades. Before the engagement, FOX’s Docker EE Clusters lags two versions behind the latest stable release for each Docker EE component (19.03.5 (Engine), 3.2.4 (Universal Control Plane), 2.7.4 (Docker Trusted Registry)). Combine with the recent headaches with Docker EE. FOX considered switching to a different container orchestration platform, Kubernetes. However, the staggering amount of technical debt gained through their usage of Docker EE prevented this possibility of a tool transition. As such, FOX consulted Nebulaworks to establish a feasible upgrade path to ease the burden of maintaining an outdated platform. With Nebulaworks’s expertise in cloud computing, container orchestration, and software engineering practices, FOX knows Nebulaworks is a credible partner. Nebulaworks kept in mind FOX’s constraints and requirements during the upgrade, instilling confidence in FOX to explore other avenues.

Why Nebulaworks?

The Docker EE upgrade is required to comply with FOX guidelines, mitigating as much downtime as possible on crucial business services. With Docker EE being less utilized then Kubernetes, the upgrade required numerous technical deep dive to grasp the best path. Filling that knowledge gap is where Nebulaworks shines. We were the ones who originally built and designed the initial deployment of FOX’s Docker EE clusters. Even before then, FOX has an extensive history with Nebulaworks, ranging from CI/CD implementations to infrastructure deployments. Working with FOX numerous times, Nebulaworks gained invaluable information on how FOX’s internal environment worked, allowing them to focus more on the finer details of a solution. FOX, being a large enterprise, having this extra time is essential in making solutions that are of the utmost quality and attention to detail. As such, it was only natural for FOX to turn to Nebulaworks again for this engagement.

The Challenge

Stability, speed, and reliability are key factors that needed to be addressed during the Docker EE Upgrade. To achieve these desired key factors, there were four challenges to overcome. The first challenge was the lack of an internal environment that is separate from other production infrastructure. This inability to deploy safely in non-production environments hindered FOX to rapidly test and deploy changes to production without the risk of an unplanned outage. The second challenge was the shortage of official documentation regarding Docker EE and officially upgrading it. While Docker EE shares similar traits with Kubernetes, some nuances make Docker EE distinct. The third challenge abides by FOX’s Docker EE pre-requirements, particularly the usage of Consul to bootstrap a cluster. Nebulaworks wanted the project to encompass all aspects, including the reworking and upgrading of FOX’s Consul deployment. Lastly, FOX needed to upgrade all of their Docker EE clusters, each of them deployed in separate environments. As such, the upgrade path is required to be consistent, yet generic. If any of those challenges are not met, there will be prolonged downtime for all FOX’s services, causing severe data loss and IT miracles across FOX to get the clusters back online to a stable point. This upgrade must succeed on the first try; there is little to no room for error.

The Docker EE upgrade project was more than just outlining a solution; It called for standardizing deployment pipelines, transferring Docker EE knowledge to FOX, and iterating new changes successfully. This is where Nebulaworks' value add came in since our expertise lies heavily in helping organizations standardize their software development practices to deploy code quickly and safely.

The Solution

Securing isolated non-prod environments was the first significant step Nebulworks took during the engagement. This allowed Nebulaworks to thoroughly assess FOX’s existing solutions, performing deep dives on the current usage and requirements for Docker EE. After redeploying the non-production environments and information gathered, it set up the groundwork for Nebulaworks to reestablish dev and staging Docker EE clusters. With each rapid test and verification on the deployment and upgrade process, Nebulaworks cleaned and refactored the existing internal Docker EE documentation, updating it to reflect the new changes that Nebulaworks made during the rapid prototyping part of the project.

The final verdict for the upgrade process was a hybrid approach, taking practices from other container orchestrators while adding additional Docker EE exclusive steps. Each upgrade would be performed on a live cluster under a restricted time frame to prevent as much downtime as possible. Since the Nebulaworks and FOX teams were in different offices, the upgrade had to be done on a video call, making any troubleshooting or debugging a challenge. That restriction proved to be too much for the live support teams at Docker/Mirantis; their brief involvement in one upgrade was unfortunately fruitless. Nevertheless, Nebulaworks stepped up to these additional challenges, working diligently with the FOX team and ironing out any issues. In the end, the efforts that Nebulaworks put in paid off, creating a robust solution that withstands any environment-specific constraints.

With Nebulaworks filling in both the educator and process leader’s role, Nebulaworks showed its value to FOX as subject matter experts on both process creation and Docker EE deployments. Nebulaworks crafted FOX its original Docker EE infrastructure and now the upgrade process. Even though the Docker EE solution had its flaws, Nebulaworks showed its value to FOX by proving that even with accumulated technical debt, a project can be led back on the right track through proper process management and best practices with cloud deployments.

Outcome

With a viable upgrade path, all of FOX’s Docker EE clusters and its components are successfully upgraded to the latest versions at that time (19.03.5 (Engine), 3.2.4 (Universal Control Plane), 2.7.4 (Docker Trusted Registry)). The upgrade opened new doors for FOX, allowing them to focus on innovation and tool refinement strategies. One such change is the gradual shift to Kubernetes, which there have been talks of already. This project’s success gave FOX the confidence to put aside their solution to search for better alternatives and alleviated months of technical debt. As such, Nebulaworks and FOX’s relationship continues to be steadfast ever since.