DevOps is hard

Surviving Continuous Deployment in Distributed Systems — July 30, 2021

Surviving Continuous Deployment in Distributed Systems

Note: a shorter, summarised version of this article has been published on the Thoughtworks blog. Read the full article here or watch the talk delivered at XConf Europe:


This is an article about the day to day of software development.

The industry seems to be at a point where a lot of us are using good practices such as Trunk Based Development and/or Continuous Deployment, or we are at least hassling our managers about it working towards it. I’m also a big fan.

But, as cool and shiny as these practices are, and as much as we reassure our fellow developers and stakeholders that they are completely safe, I believe they do present some risks.

Continue reading
An Isolated Developer Setup with Docker — May 15, 2021

An Isolated Developer Setup with Docker

In this post I am going to propose a set up to run any kind of application on a developer laptop in complete isolation. It is based on packaging the application into a Docker container, and convince it is still talking to the real world while we’re actually mocking everything around it (spoiler: using more Docker containers).

I have used this in projects of various sizes – from small scale to really chunky applications with lots of intricate dependencies – and has generally proven itself to be worth the initial investment.

In this guide I will assume that the reader is starting from scratch, with an application that is run locally just from their IDE, with no real automation or containerisation. Feel free to skip any of the steps if they are redundant or do not apply to your situation.
I will also assume the reader is familiar with basic concepts of Docker and networking.

Continue reading
Solving the Docker in Docker dilemma in your CI Pipeline — February 14, 2021

Solving the Docker in Docker dilemma in your CI Pipeline

There are some tests (integration, end to end, component…) in most modern test suites that rely on some external resources in order to run. Confusing industry terminology aside, their goal is to test the integration between some part of our application and the outside world, like a database or a queue or even some other team’s application.

This is often accomplished by having a Docker container pretend to be the external entity we wish to mock – which is easy enough to set up in a developer’s laptop:

However, things get a bit more tricky once the same test setup has to be run in the team’s CI pipeline.

The reason being, a lot of modern CI/CD tools like Jenkins, GoCD etc. rely on their build agents being Docker containers themselves.

This presents developers who wish to run integration tests in their CI with the task of spawning Docker containers from within other Docker containers, which is not trivial as we are about to see.

Continue reading