Modernised AWS platform supports full-force of Kubernetes.

Kubernetes allows FSI disruptor Zip Co to achieve huge efficiency gains in development and production, facilitating partnerships with new merchants for competitive impact

Overview

Zip Co Limited provides point-of-sale credit and digital payment services to consumers and merchants. Its next-generation payment services include Zip Co Pay – a buy now, pay later (BNPL) service – and Zip Co Money, a personal loan service.

In 2019, Zip Co became something of a ‘market unicorn’, realising rapid growth as its in-demand services disrupted the status quo and found traction amongst merchants, retailers and ultimately, consumers. 

Behind the scenes of this rise, Zip Co instigated a project to supercharge its IT operation and build a cloud-based platform that could support further growth and pursuit of new, innovative, best-in-class product offerings.

The project ultimately delivered a cutting-edge platform that exploited the best of Kubernetes functionality to make gains in efficiency that reflected the growth and ethos of the business as a whole.

The challenge

Incentivising Zip Co to review and uplift its IT operation was a partnership it was brokering with a new merchant. This new partner had stringent and comprehensive technical requirements – for example, cross-region, static IPs – which Zip Co recognised would require technical and advisory assistance to address speedily.

Zip Co began a process of planning to ‘supercharge’ its platform for new customers, a process which revealed the constraints of its existing environment. 

Some of the challenges included:

  • Complex deployment process
  • Long deployment process, no deployment direct to dev
  • Decentralised authentication
  • Legacy databases
  • Inconsistency between production environments

The existing cloud environment was made up of AWS running Windows applications and web services with EC2 as the compute layer, and over 100 EC2 instances. The Zip Co IT team had been using some ECS, which had led to the decision to look to container orchestration service Kubernetes as part of the application modernisation process. 

With the overall approach mapped out, Zip Co recognised there were areas of focus within the AWS environment that required ‘tuning’ to reach the target of being reliable, consistent, speedy, scalable and secure for both the technical team, and the business.

To assist its IT team to bridge the gaps required by the new merchant partnership, and to elevate Zip Co’s ability to innovate relentlessly, Zip Co reached out to AWS cloud-native consultancy CMD Solutions, which also had specialisation using the AWS managed Kubernetes service, EKS.

With a track record for automating and transforming in complex projects, CMD Solutions was tasked with working at the foundational layer of the environment to implement a system that could support the digital application development required for hosting the new merchant, that delivered efficiency gains, and that would deliver a resilient system for the Zip Co team to manage into the future.

“CMD Solutions brought their expertise across AWS to this engagement to help us modernize our approach to infrastructure and the collaborative build of our multi-region EKS platform to production”.

Greg Sansom, Director of Infrastructure Operations

The approach

The project was split into two stages, V1 and V2. The first stage aimed to ‘re-engineer’ the environment and simplify the deployment process and make it consistent across environments and codify complex infrastructure.

Following this base work, V2 of the project established a new platform for the requirements specified by the new merchant. The Kubernetes platform would integrate infrastructure as code, automate deployment, have a customised ingress solution, log and monitor the entire platform and have a centralised secrets management solution in Hashicorp’s vault.

Technical details

“As a partner, CMD Solutions have helped us implement modern architectures and sophisticated tooling at speed. The quality and cross-functional nature of their engineers have helped us deliver across initiatives such as this one that require a diverse range of skill sets spanning areas: automation, containers, security, etc”

Greg Sansom, Director of Infrastructure Operations

The project team simplified and made deployments to Windows consistent across multiple environments through:

  • Setting up core AMIs in an automated fashion, provisioned with all the dependencies and libraries required for applications
  • Automating server start up to pull down relevant dotNet application – integrating with OctopusDeploy
  • Providing capability to deploy servers in AutoScaling Groups instead of as EC2 pets

The team setup an entire new AWS platform using a combination of best-practice cloudformation and terraform, to meet a core component of the delivery – infrastructure-as-code.

CMD Solutions built multiple Amazon EKS environments across two different regions to provide cross-region failover capability. A standard EKS environment definition consisted of inherent autoscaling, spot instance support, monitoring via Prometheus and Grafana, and logging via elasticsearch. All workloads were deployed on spot instances to reduce costs, with the exception of production workloads. An elasticsearch domain was stood up, along with lifecycle and various management features within the EKS cluster, giving all clusters a central location for log management.

While building out the EKS cluster, several unique targets had to be met, including customised ingress controllers. There were several strict requirements by the merchant that included having fixed IP addresses (achieved by using global accelerators backed by NLB and nginx ingress controllers), and a hybrid ingress controller that utilised the ALB.

As the existing tooling was lacking critical features, CMD Solutions chose Gitlab (SaaS) for CICD tooling, incorporating the well-known three musketeers pattern (make, docker, docker-compose) to provide a consistent approach to building and deploying. Both infrastructure and development teams created their own distinct CICD pipelines. For containers, the build processes incorporated Jfrog artifactory for dependency scanning and sonarqube for code scanning. Abstracting away a lot of the logic inside makefiles gives the customer the freedom to not only choose a different CICD tool in the future if they want but also consistency in deployment approaches whether those be from the CICD tool or their laptops.

The CMD Solutions team promoted trunk-based development in both infrastructure and application deployments. This reduced complexity and gave teams better velocity in releasing to production.

The outcome

The joint CMD Solutions & Zip Co aspect of the project reinforced and elevated the quality of the existing AWS foundation and integrated Kubernetes in clever ways to provide a nimble and modern application environment.

“Partners are a key component of how we drive business and technology outcomes. CMD Solutions are a trusted AWS partner of ours, providing end-to-end expertise.”

Greg Sansom, Director of Infrastructure Operations

The joint CMD Solutions & Zip Co aspect of the project reinforced and elevated the quality of the existing AWS foundation and integrated Kubernetes in clever ways to provide a nimble and modern application environment.

The broad technological outcomes achieved include:

  • Achieving consistency between test and production environments
  • Rapid gains in speed and consistency using automation
  • Remediated high priority, security focused control gaps
  • Increase in resiliency and operational maturity

For CMD Solutions, this project provided a great opportunity to work with an organisation that encouraged an innovative approach – even to its internal IT environment. This resulted in a modernised AWS platform that supported a cutting-edge Kubernetes environment in production.

Key project highlights

  • Build into production went from 4 hours to 20 minutes
  • CMD Solutions provided the customer with a failover region
  • Zip Co was provided with complete IAC and automated pipelines for infrastructure deployments
  • Additionally, Zip Co were provided the ability to scale out workloads based on load within the EKS environment

The project allowed the Zip Co IT operation to meet the requirements of its new merchant partner, within a rapid delivery timeline. As well as achieving this immediate milestone, the platform renewal reinforces Zip Co’s innovation culture, by adopting a cutting edge approach to using and orchestrating container technology, and this will support Zip Co’s future growth plans as technology will need to scale along with the business.