Generate certificates automatically

[sylvainar]

Hey !

On first launch, I'd like to run the cerbot-auto command in order to download certificates and so. However, it's not possible to do it directly from ansible, I'm obliged to connect manually and launch the script.

Do you have any workaround for this ?

Thanks a lot !

[sylvainar]

Solved it !

- name: Run certbot generation for each host
  command: /opt/certbot/certbot-auto --apache -n -d {{ item['host'] }} --email {{ item['admin_email'] }} --redirect --agree-tos
  with_items: "{{ vhost_sites }}"

I'm gonna make a PR if I have the time to do it properly.

[geerlingguy]

@sylvainar - Thanks for the example!

I think I will try to incorporate this in as modular a way as possible, since I'm currently using the role on both Apache and Nginx-based servers (haven't tried Caddy, though it does it's own thing with Let's Encrypt).

The task above could work, but I think I'd want to use a custom variable to drive what hosts get certs, because on many of my servers, I have multiple hosts, some of which should get certs, some of which shouldn't.

But I definitely want to add this functionality.

[sylvainar]

Great, I'm staying tuned!

[oxyc]

How about using the webroot plugin instead.

certbot_hosts:
  - webroot: "/var/www/drupal/drupal"
    email: "m@oxy.fi"
    domain: "drupal.dev www.drupal.dev"

- name: Obtain certificates.
  command: "{{ certbot_script }} certonly --webroot -w {{ item.webroot }} --email {{ item.email }} --agree-tos --keep --expand -d {{ item.domain.split()|join('-d ') }}"
  with_items: certbot_hosts

This will create the ceritificate in/etc/letsencrypt/live/drupal.dev/cert.pem (Ubuntu 16.04) and then the user configures the webserver to point to it separately.

We should probably define a certbot_certificate_dir that points to the live/ subdir on all different distros. This way you just pass that on the the nginx/ansible role in your playbook.

[geerlingguy]

@oxyc that would work for me; I don't like Certbot touching my configs anyways, I just want it to automatically generate a cert then let me know where it is so I can update my nginx/apache config.

[exploide]

Automatically retrieving certificates on first run (i.e. before the cron job takes effect) would be extremely useful! The sketch from @oxyc mentioned above looks nice. I guess a few things need to be considered.

• for testing purposes, there might be a need to configure the ACME server URL; at least switch between official Let's Encrypt stable and Let's Encrypt staging; probably a role variable for this is appropriate
• When bootstrapping new hosts (using Ansible), there is no valid certificate (in a place like /etc/letsencrypt/live/) that the web server can use to start properly and make the webroot plugin work. Personally, I would consider it inconvenient and hacky to have the web server configured with plain http only or with a different certificate path in the first run, then having certbot retrieve a valid certificate and then some other Ansible role that sets up the web server properly. Maybe we can come up with a better solution. Possibly this role could detect if this is the very first run (no cert present) and use the certbot standalone plugin instead. Alternatively, it could create a snakeoil/selfsigned certificate and put it in the right place.
• May it be that certificate retrieval (as well as the installed renew cron job) needs a reload-webserver-when-done hook?

Beside that, +1 for only retrieving certificates, not touching web server's configuration.

[clanstyles]

I like this idea, some way to automatically manage this. I'm using htis with @geerlingguy 's Apache config too. I ran it by hand the first time to test it out. But found this ticket in hopes to find an automated way.

Another thought, you can't request too many certs too often. Maybe we add some "lock file" that we check the timestamp of creation to determine if we should re-run it? (Like a once only type thing?)

[exploide]

Probably it's reasonable to check for the existence and validity of the cert and only obtain it when necessary. Generally this shouldn't happen too often since one should have a cron job for this. But iirc the certbot docs say that on renew it won't do anything if the cert is still valid for a certain time frame. Doesn't it do the same for non-renew certonly retrieval. Don't know if the rate limiting is an issue. We should take care of this.

[geerlingguy]

Another friend just pinged me and said it could be as simple as this (if you have the vhosts set up otherwise and responding to requests for the non-SSL version):

# Variable:
certbot_vhosts:
  - email: johndoe@example.com
    domains:
      - example.com
      - www.example.com

# In role's main.yml.
- include: certs.yml
  with_items: certbot_vhosts

# In certs.yml
- name: Check if certificate already exists.
  stat:
    path: /etc/letsencrypt/live/{{ item.domains | first }}/cert.pem
  register: letsencrypt_cert

- name: Generate new certificate if one doesn't exist.
  shell: certbot certonly --apache --agree-tos -n -m {{ item.email }} -d {{ item.domains | join(',') }}
  when: not letsencrypt_cert.stat.exists

(pseudocode, don't sue me if it doesn't work... and he's using it only with Apache).

[sylvainar]

Well that's more or less what I proposed on the second post 😉

[exploide]

Personally, I don't like Certbot changing my webserver config. If you agree to this and want the role to behave according to this, the --apache flag might be inappropriate.

In my opinion, the standalone authenticator would fit the needs best because it does not require a fully configured webserver that can't be working anyway at this point, because there is no TLS certificate yet. I think we would need a setting that allows the role to know how to stop a potentially running webserver (to free ports 80 and 443). I.e. a setting like certbot_before_initial_retrieval = systemctl stop nxinx that is executed when the letsencrypt_cert.stat.exists check mentioned above fails. This would also work well if there is an already running webserver for domain x and now Ansible-Certbot is told to obtain a cert for domain y that is not yet configured in the webserver.

On the other hand, the renew cronjob should use the webroot authenticator to avoid unnecessary downtimes. Docs on renew say:

The same plugin and options that were used at the time the certificate was originally issued will be used for the renewal attempt, unless you specify other plugins or options.

So when webroot should be used for renewal instead of standalone, we would need to specify this explicitly for the cron job. But that should be easy.

I would like to work on a draft PR for this, but I'm a bit busy right now, so I don't know when I can come up with this :/

EDIT: To make the workflow that I have in mind clear: In a playbook, one lists the certbot role before the webserver role. This way, certbot can retrieve certificates for new domains and then, when the webserver config is applied, the certificate that is referred in the vhost is already there and the webserver successfully starts up.

[kzap]

How would you implement this workflow for a migration server that does not have the DNS pointing to it yet? it seems like its not possible as letsencrypt cant validate that you own the domain?

seems like i would have to point dns first, then run a second playbook to run letsencrypt and install them to the right place

and if you have multiple web servers you only want to generate the certs on one, then propogate them to the rest...

[exploide]

This is indeed a problem. It seems it is not possible to obtain a LE cert w/o a valid DNS entry (when going without some dirty hacks). Maybe we cannot do more in such a situation. Guess I would make the certbot role conditional, depending on some host variable that determines if this is a staging or production server. But then one can get in trouble with the webserver config.. I see what you mean..

Maybe we can come up with a partial solution eventually, but at first it would be nice to have the basic setup working.

[geerlingguy]

Yeah, I'm 99% sure it isn't possible (or at least not easy without some really hacky workarounds) to generate certs on something like a dev server or local environment without any DNS available through the general public Internet.

[oxyc]

I'm currently using this on a server.

• Should work without a webserver pre-installed (havent tested that though)
• Can update domain changes
• Doesnt shut down the webserver, uses --webroot to obtain certificates and service apache2 reload to re-read the certificate.
• Only stable way I could think of to figure out if a webserver is running, was to check if port 80 is taken.

certbot_webserver_port: 80
certbot_renew_hook: "service apache2 reload"

certbot_certificates:
  - email: "m@oxy.fi"
    webroot: "{{ packagist_deploy_docroot }}"
    domains:
      - "packagist.{{ drupal_domain }}"
  - email: "m@oxy.fi"
    webroot: "{{ drupal_core_path }}"
    domains:
      - "{{ drupal_domain }}"
      - "www.{{ drupal_domain }}"

---
- name: Detect if webserver is running.
  command: "lsof -i :{{ certbot_webserver_port }}"
  register: certbot_webserver_running
  failed_when: false
  changed_when: false

- name: Check if certificates already exists.
  stat:
    path: "/etc/letsencrypt/live/{{ item.domains|first }}/cert.pem"
  register: certbot_certificate_paths
  with_items: "{{ certbot_certificates }}"

- name: Generate certificates (standalone).
  command: "{{ certbot_script }} certonly --standalone --email {{ item.1.email }} -n --agree-tos --keep -d {{ item.1.domains|join(',') }}"
  when:
    - not certbot_certificate_paths.results[item.0].stat.exists
    - certbot_webserver_running.rc == 1
  with_indexed_items: "{{ certbot_certificates }}"

- name: Generate certificates (webserver running)
  command: "{{ certbot_script }} certonly --webroot -w {{ item.1.webroot }} --email {{ item.1.email }} -n --agree-tos --keep -d {{ item.1.domains|join(',') }} --post-hook '{{ certbot_renew_hook }}'"
  when:
    - not certbot_certificate_paths.results[item.0].stat.exists
    - certbot_webserver_running.rc == 0
  with_indexed_items: "{{ certbot_certificates }}"

- name: Update certificate domains.
  command: "{{ certbot_script }} certonly --cert-name {{ item.1.domains|first }} --webroot -w {{ item.1.webroot }} -d {{ item.1.domains|join(',') }} -n --post-hook '{{ certbot_renew_hook }}'"
  when:
    - certbot_certificate_paths.results[item.0].stat.exists
    - certbot_webserver_running.rc == 0
  with_indexed_items: "{{ certbot_certificates }}"
  register: certbot_certificate_update
  changed_when: "'Your certificate and chain have been saved' in certbot_certificate_update.stdout"

- name: Add cron job for certbot renewal (if configured).
  cron:
    name: "Certbot automatic renewal of {{ item.domains|first }}"
    job: "{{ certbot_script }} renew --cert-name {{ item.domains|first }} --webroot -w {{ item.webroot }} --post-hook '{{ certbot_renew_hook }}' -n --quiet --no-self-upgrade"
    minute: "{{ certbot_auto_renew_minute }}"
    hour: "{{ certbot_auto_renew_hour }}"
    user: "{{ certbot_auto_renew_user }}"
  when: certbot_auto_renew
  with_items: "{{ certbot_certificates }}"

Several functionalities required Cerbot v0.10.0 and it was just too much trouble to make it work for older versions (the ones installed using package managers). --post-hook (for certonly), --cert-name and --renew-with-new-domains weren't available.

Also note that --post-hook wont run unless the certificate is regenerated, which is why I call it certbot_renew_hook.

Edit: I can also note that the apache plugin couldn't figure out my vhost files so webroot or standalone + apache shutdown were my only options.

[ScalaWilliam]

Thanks for the great reference.

My requirement is:

• Create initial certificate headless.
• Do not interfere with server configurations when renewing.
• Temporary downtime for a renewal is acceptable.
• Headless renewal.
• Debian 8.
• nginx (from official repo).
• SSL work is an "add on" to existing set up.

I took this approach here: ScalaWilliam/git-work@916552c

In the setup step, nginx configuration includes an empty 'https configuration' file.
But in the ssl step, it sets that file's contents and then follows to reload the service.

I'm not too great with Ansible though but sharing my findings in case anyone finds it useful :-)

[oxyc]

Today I built a server where I had basic auth to access the host, therefore relying on the webroot plugin doesn't really cover all scenarios.

[geerlingguy]

I'm still toying with this. The chicken-and-egg problem is the most annoying—I need to manage Nginx/Apache/whatever's configuration, and I need to tell it there's a cert somewhere.

But the cert is not there until Let's Encrypt / Certbot generates the cert. But I can't always guarantee Let's Encrypt / Certbot will be installed and generate the cert before Nginx is installed / running. So besides an awkward dance of changing Nginx configs per server (or Apache per virtualhost) every time there's a new cert for a new domain (renewals are a different beast, but much easier), I'm using self-signed certs of snakeoil certs with the OS at first, then having certbot replace them. This way the server can start up initially using a cert in the LE path, then after LE creates the new real cert, we restart and Nginx/Apache are happy with the real cert.

Making this generic and automated is difficult (at best). It seems like 99.9% of all guides, blog posts, etc. about this problem are just like "start off with Nginx with a port 80 host... then after LE, switch it to 443" — or worse "let certbot create the 443 configs and just don't manage those in code/on your own" :O

Anyways, just wanted to put more notes here because this is the fifth time I've worked on automating everything end-to-end, so it's all in code, all reproducible (from nothing to server), etc.—and it's taken me hours each time. And I still can't get it done in a generic way I'd add to this role as a feature :(

[geerlingguy]

Related upstream issue: certbot/certbot#2933

[kzap]

Or just use dns validation through aws route 53 etc? :)

This is a hard problem because generating certs should be a seperate step and not part of the initial setup

[llbbl]

Just make it a requirement that it has to be run multiple times to get the SSL cert fully installed. We don't have to solve all of the worlds problems in one Ansible run. One request thou, can we make the automated cert generation optional?

- stat:
    path: /etc/letsencrypt/live/sitename.com/privkey.pem
  register: prod_ssl

- name:  prod_ssl exists
  set_fact: prod_ssl_exists=true
  when: prod_ssl.stat.exists 

- name: adding sites-available
  copy: src=apache2/sites-available/{{ item }} dest=/etc/apache2/sites-available/{{ item }} owner=root group=root mode=0644
  with_items:
    - site-name.conf
  notify: restart apache2
  when:  prod_ssl_exists is defined

[geerlingguy]

I implemented a standalone setup for some CI servers I was building, and am working on getting that into this role first.

Then I'd like to make it so you can either use standalone (where Certbot runs it's own little service on ports 80/443 to respond during the cert generation process), or with a webserver/webroot (e.g. nginx or apache2, at least) like @oxyc showed earlier in this thread.

[geerlingguy]

See: #38 — I'm also going to add a playbook in the tests/ directory which would be a functional complete demonstration of how to use this role with Nginx (initially) and Apache to generate certs automatically.

[geerlingguy]

Just about ready to wrap up the PR for this issue (#38).

In the interest of moving issues in this queue forward, I'm going to close both this issue and #6, and open a new issue for adding automatic certificate generation using the --webroot option.

That will likely require a bit more testing and a more involved automated test playbook. Plus, I've run out of cert generation for my test domain (though I have about 80 other domains I can play with if I really need to...), so I'd like to tie this off at this point and refocus on making it so we have two cert generation methods:

• Standalone
• Webroot

I won't close this ticket until I open it's replacement.

[geerlingguy]

Follow-up issue: #39 (add a webroot method).