An Agile Transformation is a large change management exercise and to be successful there is a recognition that all parts of the organisation will be touched. Organisational structure, long-lived practices and embedded culture will all be affected in some way or shape.
We all live with the paradox that change is difficult and inevitable. In this first edition of a new blog series on Agile Transformation, we will concentrate on some of the technical aspects of an organisation and how changes in the IT realm can help boost your agility.
The DevOps concept has been around almost as long as Agile and it is more than just a set of processes to help release software quicker. It encompasses process and governance change, tools, software architecture and even the philosophy of software engineering itself; all with the aim of improving agility to increase value delivery in an organisation.
DevOps and Technical Delivery
Being agile allows businesses to become adaptive to the points made above. By adopting lean and adaptive software delivery, businesses can focus on creating the most effective and efficient way of bringing new products to market or the creation of solutions using bespoke development.
Reengineering business processes to identify and implement opportunities to add values that are aligned to your strategic objectives are at the core of agility.
In this day and age, organisations are faced with very strong pressure to innovate or risk being disrupted. Businesses survive because they are able to navigate these pressures through reinvention, innovation or improvement. Agility enables this to happen.
In the software development realm, adopting agile frameworks has many benefits. DevOps practices enhance the ability to exploit technology to either reduce time to market or create a solution that allows a business to bank the value earlier. They also provide better operations maintenance through shared accountability and understanding.
DevOps achieves this in the IT space through the same principles of Agile and Lean inside the business:
- Collaborative and aligned approach – I’ve worked with both startups and large bureaucracies and I’ve seen the best results when all people responsible for an outcome work closely together to ensure success. This is done by breaking down silos, ensuring knowledge is shared and federating responsibility to the people best placed to make decisions.
- Iterative and incremental releases – important for so many reasons from banking value early through to waste reduction and sustainable pace – this cannot be underestimated. Reducing batch sizes of work enables this to happen because items can be delivered “small but complete” and so can be deployed as soon as they’re finished.
- Continuous delivery – this is a blog all on its own but the tenets of CI, automated testing and consistent deployment enables the incremental releases mentioned above.
- Continuous improvement – An equal pillar in agility but one that is often left out. It is vital that teams learn by doing and adapt their practices while working on their product development. Too many times I’ve seen project implementation reviews and other knowledge gathering practices wasted because it’s always done at the end of the project and by that stage the teams have changed or moved on.
Microservices
The best way to think about a microservice is that it is a piece of distinct, small functionality that can be deployed independently. They can then work in concert with other services as part of a larger ecosystem to deliver a solution to a particular problem.
Microservices should be:
- Highly cohesive and loosely coupled – the behaviour should be consistent and testable along standard contexts. You should avoid templating polymorphic behaviour that can add complexity to your system.
- Integrated via communication
- Synchronous – RESTful, Request/Response pattern – the way applications currently work
- Asynchronous – Think Queues. Publish/Subscribe pattern. These are especially useful when the action of the service is not dependent on time.
- Can be orchestrated (but choreography is better).
Microservices and agility
Microservices aren’t new, and service-oriented architecture has been around for at least a decade – but they’re only recently recognised as being a technical enabler of agility.
Services and microservices are all about integration – joining applications and solutions together. No application is an island and if you need to join information between two systems then you either have to have a strategy for ‘pulling data together’ or an ability to integrate them.
Focusing on the purpose of the integration will drive a successful integration strategy. If you’re designating or creating a source of truth in an application you’re likely to have a different strategy than if you’re collating data for reporting purposes.
SaaS has made designing a microservices strategy easier
The advent of software-as-a-service has also driven the popularity of a microservices strategy. We have systems that aren’t on premises and in fact, no longer “owned” by our organisations.
Organisations can now operate a cloud CRM and cloud billing systems that are countries apart. We need to be able to pull the data in these systems together to make sense of the information in our enterprise.
Demand for microservices continues to increase
Another reason that is driving the need for microservices is the ever increasing demand for solutions. This creates a need to plug and play different solutions as we scramble to take advantage of commodity systems in the market to help us deliver our results.
By having the ability to swap out existing services with new ones increases our ability to take advantage of these without the high cost of change that is prevalent in more monolithic software architecture.
The main reason that I want to concentrate on why to use microservices is that they are a key enabler of organisational agility – especially if you are in the business of building products or developing bespoke solutions for your business.
Advantages and disadvantages of microservices
Advantages of microservices:
Agility
Agility is well supported by microservices. The problem is broken down into manageable components that are able to be tested easily and released into production as soon as the organisation needs them.
This shift in thinking about the system into small inter-connected parts can help to simplify the solution. They also help to future-proof the solution – if a new solution appears that your software could use, it is easier to replace your service with this new one.
Scale
There is also a scaling advantage as well as designed services that can be deployed in clusters to help support elastic infrastructure. While the overall solution architecture may become complex, the services themselves can be easy to verify, test and support.
Microservices helps reduce batch size
One of the practices that agile organisations do well is to split up the work into smaller pieces to try to release highest value as soon as possible. If you are splitting the work up into small stories, it makes sense then, that the software produced matches the small batches that the stories represent. This allows for better mapping of requirements to software delivered.
These small bundles of work can be deployed individually, without being dependent on other parts of the product or application. This also helps experimentation and innovation by trying solutions early to evaluate their appropriateness to the problem being solved.
Another benefit that I experienced while working in a large application development shop was that once services were split out, they invariably ended up being useful in other, sometimes unrelated applications.
Disadvantages of microservices
Integration testing
Integration testing is probably the slowest, most brittle, least scalable type of testing available. Testing multiple integration points can be very expensive to write, maintain and run.
Consider refactoring your tests to minimise the integration to just contracts and connections. Service functionality should really be pushed into the unit testing.
Holistic approach
There are many different types of ways to integrate services and you should try to minimise or standardise your integration methods to a few.
While this may seem like a no-brainer, if you’re going to use a third party solution or SAAS system you may not have a choice but to use the vendor choice of integration, i.e. Salesforce.
Once your solution or application is a series of services, there needs to be a holistic approach to maintaining and monitoring them – requiring knowledge of the application ecosystem which is why the DevOps approach (also linked to increased agility) is recommended.
Logging
Logging also needs to be a holistic activity as different services in potentially different ecosystems will need to have information collated to be able to monitor all services.
We used splunk to collect log file output and produce meaningful dashboards to monitor service and application performance. While this tool requires a little effort to set up, it became integral to our ability to successfully support the application.
Security is also a little more complex. Multiple integration points give opportunities to hackers and other nefarious agents to try and compromise your systems. Ensuring good perimeter security is more important than ever.
Transitioning to microservices
For greenfield projects, you’re starting with a blank piece of paper – so don’t spend a lot of effort making highly granular services until you’ve fully understood the domain and bounded contexts of your solution. Keep your services fairly coarse – as your understanding increases you can refactor your services appropriately.
Global, often used functionalities make great candidates for services because even if they’re only used by one application, they may need to be scaled and therefore would benefit from being abstracted as a service.
Most often, the organisation will probably not be greenfield but rather have a slew of large monolithic applications. This presents an opportunity as well as a challenge.
You don’t have to take an all or nothing approach to microservices adoption
You can break up the monolith piece by piece.
A good starting point would be to identify functionality that is being used by two or more applications; or perhaps functionality that can benefit from being scaled horizontally. By using bounded contexts, you can determine service boundaries which should give you the ability to test the service.
I like to think of the Grandfather’s axe analogy when splitting out the monolith. As the axe had 12 different heads and 10 different handles but was still the same axe. Your application or product will change services as technology changes but will still be the same product.
Example: Splitting services related to authorisation and access control in application login screens
By doing this you can create standardised services that can be perimeter hardened across all of your applications, for extra security, but also provide centralised access control in business units without having to develop separate access control functions for each application.
While microservices aren’t new, they’ve evolved to become an important tenet in technical agility. Any organisation that is having trouble with delays to market or inflexibility in changing a system or product would definitely benefit from taking a service based approach to development.
