Here on Landoop we prototype fast and new (sub)domains are frequently added to complement our back and front-end services. Since the beginning our specifications included “ssl everywhere”. The journey into providing fully secure and encrypted services is a long one; hence we need an adventure in the SSL land.
We use ansible to manage our servers. Part of our infrastructure is on Google Cloud. Ansible’s support of Google Compute Engine leaves much to be desired but otherwise we are happy with its main function.
We use nginx as a web server, proxy and SSL termination endpoint for services. There are other great servers too, but we try to keep some consistency on our configurations.
We needed an easy way to get certificates for new services. Waiting for the devops team to get a certificate issued and deployed was a burden to the rest of the crew. We also needed certificates to be deployed per server and to keep backups of them. It is not rare to have more than one service in each server.
As the devops engineer I also wanted a way to renew the SSL certificates automatically, and get all the above for free :)
The solution we came up was exactly this! Use a template to add a new ‘nginx site configuration file’ to ansible and deploy the new configuration. Our solution automatically gets the new certificate issued, backs up and sets a cron job to renew it when needed.
Let’s encrypt is the new cool kid in town. It is a Certificate Authority (CA) that offers free SSL certificates through an API. I waited for them for months to go live —they entered their public beta in December 2015— and was a little too happy to use them for Landoop’s infrastructure. They offered everything I dreamed of —except DNS validation, which now they do— and some more.
To work with let’s encrypt you also need a client. The official one is a bit too intrusive for my taste, tampering with the web server’s configuration files. Thankfully the API protocol, ACME, is well documented and various clients do exist. There is also a nice little server, caddy, that supports let’s encrypt out of the box, doing everything automatically.
The client of choice in our case is acmetool, a full featured client written in Go (thus it is a statically linked executable) which supports various ways of handling let’s encrypt’s challenge responses.
Challenge responses are used by the CA to verify that you indeed own the domain you ask a certificate for. Handling them is the trickiest part since you should serve them from the same server and domain that your site or service uses. Furthermore when you have a single nginx serving more than one domains, things get a bit more complicated as each challenge response should be served by the correct domain.
It is far more easier and useful to give you a working ansible role. Thus, with great joy, I present you with our ansible examples repository on github. The role you should look for is roles/nginx-lets-encrypt. It has been tested with ansible 2.0. It is derived from our ansible configuration. Since it is a bit different from it, it isn’t battle-tested, so your mileage may vary. Even if it doesn’t work for your case though, I am positive it shouldn’t need anything more than a simple fix.
The role we demonstrate automatically requests missing certificates for sites and backs them up into
the repository. Using a tool like
git-crypt to keep the backups protected should be obvious.
The role first installs an acmetool release. If there is
a backup configuration for the tool, it will restore it. If not will it configure acmetool with the
files/acmetool/response-file.yml configuration. It is important to place a valid email address in there
since this is the only way to revoke a certificate if needed. We also install a cronjob that checks and renews
certificates as needed. Let’s encrypt’s certs are valid for 90 days.
Then nginx is installed and configured. We have ready settings for handling challenge responses and ssl —which is configured as suggested by Mozilla’s SSL Config Generator. It scores a solid A in SSL Labs’ Test.
files/conf.d you may place configurations for your sites. Please use
as a template for your sites.
We chose to use proxy redirection for handling challenge responses. When acmetool asks for a new certificate, it starts a webserver listening on port 402 and serving the responses. Nginx will redirect let’s encrypt’s requests there. Once a certificate is acquired, acmetool exits, it doesn’t run permanently. The only time acmetool runs is when using ansible to deploy a new site or via the cron-job for renewing certs.
Once your site configuration files are installed we run our custom bash script
encrypt is careful and serious about issuing certificates and imposes a limit of 5 certs per domain per week.
Thus we have to be careful not to make many experiments and have to wait for a week to get our certificate.
certs-checks-gets.sh will check for missing keys from your nginx configuration files. If it finds any it will
check that you have proper DNS records for them. We use the
ansible_ssh_host ip address for this.
Once everything checks out, it will request your certificates. You may notice that our nginx configuration is
invalid at this point since our certificate files are missing. This means that nginx can’t start.
certs-checks-gets.sh knows this and replaces the site’s nginx configuration with a simple one that takes
care of challenge response requests. When finished, it will restore your site configuration and nginx will
reload with the proper settings.
The last step is to create an archive for acmetool’s data directory (
/var/lib/acme) and download it into
your repository, under
files/backups/<HOSTNAME>/acme.tbz. Use git-crypt or another tool to protect these
That’s all for today. Again, you can find the role in our our ansible examples repository.
Thank you for your interest !