Chris Porter is Technical Architect and Senior Developer on the Digital Marketplace.
Providing a technology product that can be changed quickly is a key goal for the Digital Marketplace development team. The agile approach, which is core to the work process GDS uses, means that we will continue to learn and improve the software we are building; apply those changes and ensure that in doing so we don’t break what is already there.
This blog post highlights some of the techniques and tools that we are using to ensure we are able to handle change at any level with each new iteration of the product.
The Technology
The Digital Marketplace is the Cloudstore replatformed. In other words, we have built it from the ground up on a completely new Open Source technology stack. These technologies are freely available and are well-suited to creating highly testable software using agile techniques.
These include:
- Grails - similar to the Ruby on Rails web development platform, Grails uses the Groovy programming language, which is in turn based on Java. It provides a web development and database access platform, as well as a highly evolved plugin ecosystem, which enables it to play well with a wide range of other technologies.
- elasticsearch - provides a scalable, super-fast search technology which we have integrated into the platform. This search engine embedded within our software allows us to tailor the searching and finding experience to what our users want and need.
- PostgreSQL - an Open Source database known for its reliability with a host of enterprise class features, such as point-in-time-recovery and asynchronous replication. We are making particular use of its native JSON datatype to get the best of both worlds between relational and non-relational storage.
The approach
Pairing
The development team employs a pair programming approach to ensure all software is peer-reviewed at source and that no decisions are made in isolation.
Another benefit of pairing is that all development knowledge is shared by at least two team members at all times. This is further enhanced by regularly rotating individuals in and out of pairs, and even pairing across disciplines.
Testing
The Digital Marketplace employs the user story technique to describe its features, an example of which is:
‘As a user
I want to be able to login into my account on Digital Marketplace
So that I can authenticate myself’
Each user story will always be accompanied by acceptance criteria (ACs) that provide a scenario to play against the story to ensure it works:
‘Given that I choose to login
When I enter valid credentials
Then I should get logged-in successfully and a greeting message should be shown to me which says "Hello <>".’
Grails provides a Behaviour-Driven Development (BDD) framework called Spock, which enables us to consume these ACs directly into the code in plain, human-readable text and to implement them as replayable, automatable tests.
- Unit Tests - test small blocks of code in isolation from the rest of the system. For example testing a method on a Service component.
- Integration Tests - chain together several components of the system, but still work in relative isolation. For example, test invokes a Restful Controller, calling a Service which uses a Gorm component to access data in an in-memory database.
- Functional Tests - directly implement ACs and, using the Grails Geb plugin along with PhantomJS, they test the product from a user point of view using ‘headless’ browser automation
- QA Testing - additional layer of human testing provided by a Quality Assurance (QA) tester
Automation
To guarantee code quality on an ongoing basis tests must be run regularly. They provide a constant monitor of the health of the Digital Marketplace code.
To enable this we have provided two key components:
- Cloud Infrastructure Automation - to be truly confident that code works, we need to pull together all the work the team has done and run our tests under controlled circumstances in an environment dedicated to that purpose alone. We also need to be able to deploy the successfully tested code into downstream environments frequently so that our users can evaluate it. If successful, this approved code makes it into the live frontend.
We need several shared software environments, which we host on one or more cloud hosting providers. We need to be able to create these environments consistently and quickly, reducing the risk of human error. This process is automated as much as possible using an infrastructure-as-code approach.
- Continuous Delivery Pipeline - once code and tests for a new user story are written and checked into the code repository, we need something to take over the process of making sure they work and that they don’t break anything else. The Continuous Delivery pipeline is custom built to build, test and deploy the Digital Marketplace to its various environments using a combination of automated and manually activated processes.
Monitoring
With all the automated quality assurance activity going on in the background like a heart beat, we need to be able to monitor that heart beat and react quickly if something goes wrong. All our tests will automatically trigger failures in the Continuous Delivery Pipeline if any AC does not pass or if any external error occurs that impedes testing. This shows up on our team build monitor for all to see and a developer will be quickly reassigned to work on the problem.
In addition to this, we are able to physically monitor the code itself using Cobertura while tests are running to calculate how much of it has actually be touched by the tests. At the time of writing our code coverage report indicates that we have a 90% plus test coverage for our alpha codebase.
The Digital Marketplace uses agile development, testing and automation techniques to ensure a high level of confidence in the quality of the code. It provides a technology platform that enables a user-centric design approach and that means we can react quickly to changing user needs.