PwC Enterprise Vault Deployment in Azure

Executive Summary

Our client is a world class financial services company with over a quarter-million employees and a global footprint. Due to the size and complexity of their organization, their need for scalable, robust, and efficient development stacks and tooling as services for the greater organization becomes apparent.

One of the most important of these services is secrets management. Many teams within this organization need to safely, efficiently, and securely managing secrets data. If this data isn’t managed properly, there is enormous potential for loss of company data and infrastructure and organizational instability. As part of an ongoing relationship with this financial services company, Nebulaworks has dedicated a team of engineers to guiding and supporting the customer’s existing Hashicorp Vault workstream. This team manages secrets and encryption services within a cloud-based environment across geographically dispersed deployments utilized by a wide range of operators to benefit the greater organization. With Vault, our client gains a modern and robust set of tools that allow them to manage these important data with low technological and organizational overhead, a valid lifecycle for updates and changes, and minimal downtime.

Why Nebulaworks?

Our client had some gaps in their overall development process, and Nebulaworks specializes in this sort of challenge. Our engineering teams look at solutions to our customers' problems in a holistic manner. We determine the best approach for each team in terms of tools, workflows, frameworks, etc. in a bespoke way by careful and sensitive analysis of a customer’s problems. They also have an extensive history with Nebulaworks, ranging from Consul and Terraform workstreams. During these engagements, Nebulaworks has gained intimate knowledge of our client’s internal environment. When it came time for the Vault workstream, they knew they could rely on Nebulaworks as a credible partner for this engagement.

The Challenge

Our client has a large number of development teams, each operating more or less independently of one another. However, many of them have common requirements. One of these is secrets-management; as such, they had chosen to implement Hashicorp Vault. Previously, each of those teams more-or-less managed secrets data on their own with no unified solution. This approach isn’t sustainable for an organization this large; often, the security of secrets data becomes secondary to the product itself, and compromise can occur. Ordering and instilling responsibility for this data becomes critical at scale, not to mention the requirements for audits. They had chosen to implement Hashicorp Vault to this end. However, despite Vault being the right choice for this particular problem, it is only a small piece of a much larger puzzle.

Implementing Vault correctly requires making use of proper infrastructure-as-code and DevOps practices to be effective. This is especially true in the case of our client, which would need to manage multiple deployments across several continents. When Nebulaworks came onboard, there was little in the way of a complete CD pipeline. Git repository management needed reorganizing, and there were many manual steps performed by our client’s Vault team that should have been automated. The team itself had the right mindset insofar as they recognized proper processes and were open to working within our prescriptions but were much aided by having our engineers working alongside them.

The Solution

From the get-go of this project, we developed a plan that included road mapping and timeline sessions to ensure critical sections of the project were done on time. In addition to that, we also helped to develop realistic technical goals and defined methods to achieve them.

We used this proven framework to help collaboration and work within an Agile sprint-based release cycle to release batches of changes regularly. Leveraging our client’s footprint in Microsoft Azure and their implementation of Terraform Enterprise, Nebulaworks began to formulate a plan that would allow them to have a continuous implementation and delivery pipeline managed with trunk-based development to streamline their release process and start effectively leveraging Vault and get necessary changes to the infrastructure out on time. We also had to look at visibility into the Vault infrastructure to ensure the team was hosting healthy services. Grafana and Azure Log Monitor were chosen to this end.

We shored up the CI/CD pipeline setup in Azure DevOps to allow these changes to be pushed out through Terraform Enterprise in a way that makes them easy to track and errors to be corrected quickly. Management and monitoring of the Vault Clusters themselves were simplified using Telegraf for monitoring, Fluentbit and Azure Log Monitor for log aggregation, and finally, Grafana for visualization into system health and status. The upgrades in monitoring and logging tooling coupled with a more agile workflow ensured any production issues are quickly corrected.

The tooling upgrades implemented by Nebulaworks were only a small piece of the whole pie. The best practices we suggested allowed the our client’s team to make full use of that tooling and begin to see where they can take their infrastructure after the engagement was completed.

Choosing the Right Platform

It is important to select tools that make sense for an individual organization. In the case of this financial services company, their heavy footprint in Azure pointed to Azure DevOps being a natural choice for CI/CD. The Hashicorp stack also lends itself well to a cloud-based, infrastructure-as-code driven environment, and our client has leaned on it heavily. A decision made before the engagement was to host code on Github. This was the icing on the cake as it allowed Nebulaworks to implement trunk-based development. While there are many different choices for branching models when managing repos, Nebulaworks has found that trunk-based development stands head-and-shoulders above the rest in the vast majority of development scenarios, and so a natural choice for our client’s team.

Outcome

Looking to leverage cloudnative tooling and workflows to improve their efficiency and stability, our client successfully adopted recommendations made by Nebulaworks to create a robust, stable, and fluid product. With these changes in place, they had a secure platform that allows their engineers to focus on building products for their customers without the unnecessary overhead and minimizing the threat of security vulnerabilities in managing secrets.

Nebulaworks' engineers were embedded with and worked alongside the financial services team daily for the duration of the engagement and had close and regular contact with project stakeholders. As the engagement progressed, our client’s engineers began to appreciate the elegance and simplicity of the principles, process, and tooling Nebulaworks implemented and relied on time and time again. The stakeholders involved in the project began to see a secure and robust implementation of an internal tool that works remarkably well and efficiently while serving as a model worthy of emulation for other teams within our client’s organization. Ultimately, we helped them leverage a toolchain and set of principles and processes that they are still using for great success to this day.