Waiting for linked services in Docker containers to be ready
In this post, we explain why we built and how we use waitforservices, a small utility to be used in a Docker container, that waits for linked services to be ready.
For running tests on our CI, we're using Docker Compose to automatically start up multiple containers running different services.
During startup, each service with a database has to set up its own database and run migrations. Before they can run, a separate database container has to be ready to accept connections. For integration tests, we have services loading seed data into the database and the tests themselves depend on other services to be running. This initialization can take from a few seconds up to two minutes, so usual timeouts don't cover that, but we need to make sure all necessary services are done with their initialization and ready to accept requests, otherwise tests might wrongly fail.
We started by introducing wait times before running migrations, which worked suprisingly well, but still regularly resulted in tests failing due to other services not being ready.
To solve this problem, we built waitforservices, a utility written in Go, which looks for all services linked to a Docker container via TCP and waits for them to be ready. To check whether the services are ready, it repeatedly tries to open a TCP connection to them, until it succeeds or aborts when a certain time has passed. Unlike simple bash scripts we found, it does this concurrently and automatically discovers linked services via environment variables.
Here's how it's output looks like for an application depending on two HTTP services (cleaned up for duplicate environment vars):
2015/08/14 04:37:55 Waiting for 4 services to be ready... 2015/08/14 04:37:55 TCP: Service CLIENT_PORT_80 (172.17.3.35:80) is up 2015/08/14 04:37:57 HTTP: Service CLIENT_PORT_80 (172.17.3.35:80) is up 2015/08/14 04:38:39 TCP: Service BANKACCOUNT_PORT_80 (172.17.3.47:80) is up 2015/08/14 04:38:43 HTTP: Service BANKACCOUNT_PORT_80 (172.17.3.47:80) is up 2015/08/14 04:38:43 All services are up after 48.616998424s!
We're running some Ruby services behind Passenger, which are quite slow (a few seconds) to start up. To make sure they're ready, after successfully connecting via TCP, waitforservices executes an HTTP request and waits for the request to be done. This functionality is optional and you have to explicitly enable it by specifying a port to make HTTP requests to.
waitforservices still has some limitations, e.g. it doesn't allow specifying multiple HTTP ports and doesn't deduplicate the service addresses it gets from environment variables, so it might do some unnecessary connection attempts. We'd be happy to receive pull requests to improve that or hear your feedback as issues.
By Michael Frister