The Orchestration of AWS

By Bruce Haefele, General Manager – Technology, Healthdirect Australia

Bruce Haefele, General Manager – Technology, Healthdirect Australia

Healthdirect Australia’s journey with Amazon Web Services (AWS) started over seven years ago. Over this period, we have experienced first-hand the rise of public cloud as an alternative to on-premise and its evolution from an infrastructure replacement to a platform ecosystem. We have had to continuously learn and adapt our approaches at an unprecedented pace – a process we have found both achievable and necessary.

From the outset, our leadership team were open-minded about using AWS, encouraging us to try new things while ensuring we fully examined the financial, security and contractual risks. This support from the top meant we began the journey with our eyes open about our responsibilities to our shareholders and the people of Australia.

There were few skills available in the early market, so we worked with a systems integrator to develop experience in their staff. This proved unsustainable as the result was us training people who were then marketed to other organisations at higher rates. Instead, we adopted a staff augmentation model where we hired and developed the core team directly and used external consultants for niche skills or to expand capacity. While it took years to develop to maturity, this approach provided much needed continuity, IP retention, improved customer focus, and increased ownership.

Our first AWS platform architecture was a typical tiered network with high availability through redundancy, security through product deployment, everything as an API and a healthy dose of data management. Unfortunately it was also manually intensive, scaled poorly, failed to accommodate security processes easily and required a lot of after-hours work in incident management. While constructing our own interfaces as APIs we had failed to take advantage of AWS’s own APIs and apply the principles of infrastructure-as-code.

We pivoted and implemented a second platform architecture based on continuous delivery automation and greater operational management tooling. Federal security compliance mandates forced a rapid maturity of our processes but reduced the number of AWS services we were able to use. This increased security and operational assurance required new product deployments, each of which required careful design, automation and testing to work in our continuous delivery environment.

We pivoted and implemented a second platform architecture based on continuous delivery automation and greater operational management tooling

The results were incredible

Production deployments that could take up to three months, were now done in minutes. Since everything was code, any issues that materialised could be traced to their origin, fixed and never repeated. Our operations staff stopped working at strange hours of the night and our service levels improved markedly. The downside of this approach was the significant engineering effort required to establish product automation and the limitations of shared environments for multiple products and teams.

Over the next couple of years we made continual platform improvements but increasingly noticed that culture was becoming a complicating factor. Shared teams and shared environments created friction and limited agility. Furthermore, we wished to deploy agile work approaches more widely across the organisation.

An organisational restructure created product based business units, called service lines, with increased responsibility for all aspects of the product lifecycle including operations. This placed pressure on the platform to evolve again to meet the agility demands of the service lines while maintaining security and operational governance and compliance.

Meanwhile AWS started Information Security Manual certification of 47 of their services, effectively opening up more their ecosystem to us. We immediately began redesigning our platform to use more AWS services to reduce our reliance on third-party products. We also experimented with serverless technologies and microservice architectures and proved they could improve development speed while reducing operating costs. By embedding deeper into AWS things integrated better reducing our effort to build infrastructure.

We are still in the process of building this latest platform with full benefits realisation still around a year away. Early indications show that our cost reductions alone will be in the order of 25 percent to 50 percent, along with a reduced risk profile and more mature security and operations.

Product teams can self-serve their platform needs with consistent security, operational and cost controls. Common patterns and practices increase reuse, reduce development cost and increase time to market. Preventative and detective guardrails are deployed across multiple product accounts and key issues can be automatically remediated when detected. By using more services and serverless we reduce management overhead – there is no patching and application upgrades required, no underlying infrastructure health issues, no scaling problems, and no upgrading to new instance types.

This approach has unlocked accessible new technologies allowing us to increase innovation. We are using big data solutions in AWS to aggregate health service data from across the sector and publish high quality information to enable consumer and providers to find health services easily. In the consumer health space, we are experimenting with voice interfaces to our service directory to reduce call centre costs, trialling Alexa skills for UV radiation in summer and flu risk in winter, and providing chatbot interfaces to our symptom checker to improve its accessibility.

We foresee a long relationship with AWS in the years ahead, one that is fully symbiotic: AWS provides the tools that we shape into important health services for all Australians.