DevOps is hard

See which process is occupying a port — February 14, 2021
Solving the Docker in Docker dilemma in your CI Pipeline —

Solving the Docker in Docker dilemma in your CI Pipeline

There are some tests (integration, end to end, component…) in most modern test suites that rely on some external resources in order to run. Confusing industry terminology aside, their goal is to test the integration between some part of our application and the outside world, like a database or a queue or even some other team’s application.

This is often accomplished by having a Docker container pretend to be the external entity we wish to mock – which is easy enough to set up in a developer’s laptop:

However, things get a bit more tricky once the same test setup has to be run in the team’s CI pipeline.

The reason being, a lot of modern CI/CD tools like Jenkins, GoCD etc. rely on their build agents being Docker containers themselves.

This presents developers who wish to run integration tests in their CI with the task of spawning Docker containers from within other Docker containers, which is not trivial as we are about to see.

Continue reading
The same code with Callbacks vs Promises vs Async/Await — February 13, 2021

The same code with Callbacks vs Promises vs Async/Await

Sometimes in JavaScript to have to deal with different flavours of asynchronous code, so it is handy to be able to map back and forth between them.


Functions that do something asynchronously are typically implemented using the callback pattern, and their implementation might look like this:

const myAsyncFunction = (parameter, callback) => {
    const result = //do something async here...
    callback(result, error);
Continue reading
SSH Multiplexing and Master Mode — January 31, 2021

SSH Multiplexing and Master Mode

When using SSH bastion hosts it is common to set up new connections for many of the use cases discussed in the previous section throughout the day.
Normally we would start a new TCP connection for each one of them. However, open TCP connection are a finite resource on any machine, and each one of them takes some time to set up.

Multiplexing is a feature provided by SSH which alleviates these problems. It allows a single TCP connection to carry multiple SSH sessions. The TCP connection will be established and kept alive for a specific period of time and new SSH sessions will be established over that connection.

It works by creating a “control socket” file which will be used every time we want start a new connection.

We need to pass two command line arguments in order to leverage this feature:

Continue reading
SSH X11 Forwarding —

SSH X11 Forwarding

If you are using SSH between Unix-like operating systems, you can also forward GUI applications over SSH. This is especially useful if your server doesn’t really have a user interface, but you need to check something on the fly with a web browser running on it.

Descarga Navegador Firefox — Rápido, privado y gratis — de Mozilla

This is possible because all Unix-like systems share a common GUI windowing system called X11, which is what provides the basic framework for the desktop environment: drawing and moving windows on the display device and interacting with a mouse and keyboard, etc.

Continue reading
SSH Tunnelling and Port Forwarding —

SSH Tunnelling and Port Forwarding

In the previous section we saw how to make use of a jump host as a proxy to run commands into a remote machine.
Sometimes however, having a shell is not necessary, and the connectivity aspect of having a secure channel to the remote host is way more interesting.
SSH’s port forwarding feature allows us to create a secure channel to the remote host, and then use it to carry any type of traffic back and forth.

Local Port Forwarding

Scenario: you want to reach a specific application (which listens on a specific port) on the remote server.

For example (as in the image) we might want to connect to a PostgreSQL database application that is listening on port 5432 on the remote server, which is in a private subnet.

Continue reading
Jumping SSH Hosts —

Jumping SSH Hosts

Sometimes some servers are unreachable to us due to network topology barriers, which are put in place for security reasons.

A common way to allow developers and sysadmins access is to provide public “bastion” or jump hosts, where one can obtain a shell and then use it to connect to one’s destination.

Looking at the picture above, things may seem pretty simple – but how is authentication handled when more than two parties are involved in a connection?

Continue reading
SSH Config —

SSH Config

As well as having to type passphrases, remembering the right user, key file, port and other settings for each host you want to SSH into might quickly become overwhelming.

That’s why, in addition to the command line options we saw so far, SSH also allows to read the same options from configuration files.

The configuration data will be applied from the following sources in the following order:

  1. command-line options
  2. user’s configuration file (located in ~/.ssh/config)
  3. system-wide configuration file (located in /etc/ssh/ssh_config)

How to write ~/.ssh/config and /etc/ssh/ssh_config files

Continue reading
SSH Agent —

SSH Agent

If you are using key pair based authentication with a passphrase for your keys, things can quickly get tedious as you have to input the passphrase every time you want to connect somewhere. If you want to avoid that, you can optionally use another preinstalled tool: ssh-agent.

The ssh-agent is a little helper program that keeps track your identity keys and their passphrases. The agent is consulted by the SSH client during the authentication process instead of the user having to specify a key – and having to type its passphrase all over again.

Adding keys

Simply add your private key file to the agent like this

Continue reading
SSH Known Hosts —

SSH Known Hosts

Much like how the authorized_keys file is used to authenticate clients on the server, there is another file in the ~/.ssh folder called known_hosts, which is used to authenticate servers to the client.

Whenever SSH is configured on a new server it always generates a public and private key pair for the server, just like you did for your user in the previous section. Every time you connect to any SSH server, it shows you its public key first, together with a proof that it possesses the corresponding private key. If you do not have its public key yet, then your computer will ask for it and add it into the known_hosts file. 

This way, the client can check that the server is a known one, and not some rogue server trying to pass off as the right one.

That’s why when you connect to a server for the first time, you might get a message like this:

Continue reading