Automated TLS Certificates with ACME: Validation Method, Rate Limits and State You Must Keep

Automated certificate management is sold as "set it and forget it", and for the first three months of most deployments that is exactly how it goes. The trouble starts when you have to rebuild the system, when short-lived test environments multiply, or when the DNS for a domain belongs to another team. This article covers the decisions that actually matter when you set up an ACME-based certificate flow (Let's Encrypt being the common example), where the rate limits will catch you, and the pitfalls you are almost guaranteed to hit.

Three validation methods, one real decision

Before issuing a certificate, ACME proves that you control the name in one of three ways:

• HTTP-01: the CA fetches a file under /.well-known/acme-challenge/ on port 80 of the domain. It is the easiest to set up; the only requirements are that the name resolves directly to that server and port 80 is reachable from the internet.
• DNS-01: you publish a TXT record named _acme-challenge.<domain>. The server does not need to be publicly reachable, and wildcard certificates (*.example.com) can only be obtained this way. The cost is that the ACME client must hold an API credential with write access to your DNS provider.
• TLS-ALPN-01: validation happens on port 443 using a dedicated ALPN protocol. It exists for environments where port 80 can never be opened; otherwise it carries the same constraints as HTTP-01.

Our position is clear. If you have a single public server, HTTP-01 is enough and requires fewer privileges. The moment you have several servers, services on a private network, ephemeral environments, or a need for wildcards, move to DNS-01. Many teams start with HTTP-01 and limp along for years by hand-patching every exception; switching at the second exception is cheaper. Scope the DNS API credential as tightly as the provider allows, ideally to a single zone or to TXT records only. If the provider cannot do that, delegate a dedicated validation zone and point each _acme-challenge name at it with a CNAME. It is both safer and widely used.

When the rate limits hit you first

Let's Encrypt's limits are invisible in day-to-day use because they allow dozens of certificates per registered domain per week. The scenario that hurts is this: every CI run or preview environment mints its own new subdomain and requests its own certificate. The limit applies to the registered domain as a whole (example.com), not to the subdomain, so if your test environments share the production domain, production pays for the exhausted quota and a genuine renewal gets refused the same week.

Three measures work together:

1. Use the staging environment while testing the flow. Its directory URL is https://acme-staging-v02.api.letsencrypt.org/directory; the limits are far more generous, browsers do not trust the certificates it issues, but it is enough to prove the pipeline works. For certbot, the --dry-run flag does precisely this.
2. Put ephemeral environments under a dedicated parent domain and obtain one wildcard certificate they all share. A hundred environments, zero new certificates.
3. Do not request certificates for the same set of names over and over. The "duplicate certificate" limit is separate and much lower; a client that starts from scratch on every rebuild hits it within two days.

What you must persist is state, not the certificate

The most common mistake is backing up the certificate but not the ACME account and client state. The certificate renews every three months anyway; if it is lost, you request a new one. But if the account key and the record of "this certificate was already issued" are lost, the client treats everything as new, the quota fills instantly, and you lose the renewal exemptions.

In practice this means the entire /etc/letsencrypt directory for certbot, the single acme.json file for tools like Traefik, and in Kubernetes the Secret objects in which cert-manager stores the account key and issued certificates. If your cluster backup excludes Secrets, the first thing you will meet after a disaster restore is a rate-limit error. For the same reason, if client state lives on shared storage, guarantee that only one instance writes to it; two instances sharing an account and opening separate orders will trample both the limit and each other's certificates.

Working examples

HTTP-01 on a single server, served from the web server's document root:

certbot certonly --webroot -w /var/www/html \
  -d example.com -d www.example.com \
  --deploy-hook "systemctl reload nginx"

certbot renew only renews certificates that are approaching expiry, and the timer shipped with the package attempts it twice a day. If you write your own cron entry, keep the same frequency; a "once a month" renewal loses the certificate on a single DNS outage.

DNS-01 in Kubernetes with cert-manager, Cloudflare example:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-dns
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-dns-account-key
    solvers:
      - dns01:
          cloudflare:
            apiTokenSecretRef:
              name: cloudflare-api-token
              key: api-token
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-example
  namespace: ingress
spec:
  secretName: wildcard-example-tls
  dnsNames:
    - "example.com"
    - "*.example.com"
  issuerRef:
    name: letsencrypt-dns
    kind: ClusterIssuer

Apply it first with the server line pointing at staging, wait until kubectl describe certificate shows Ready, then switch to the production URL. That two-line discipline prevents most of the setups that burn through their quota.

Pitfalls

• If an AAAA record exists, validation arrives over IPv6. Publish an IPv6 address for the name without actually serving on it and HTTP-01 fails quietly; the error will not tell you that IPv4 works fine. Either remove the AAAA record or genuinely serve IPv6.
• A CAA record is a lock that gets forgotten when teams change. A record such as example.com. CAA 0 issue "letsencrypt.org" blocks other CAs. When you switch CAs or hand a subdomain to another provider, this is the first place to look.
• Split-horizon DNS breaks the client's pre-check. cert-manager and similar tools attempt the validation themselves before contacting the CA. If internal DNS resolves the name to a private address, that pre-check hangs and the order is never placed. With DNS-01, tell the client to consult only external resolvers; cert-manager's --dns01-recursive-nameservers and --dns01-recursive-nameservers-only flags exist for exactly this.
• Failed validations have their own limit. A misconfigured client retrying every minute locks itself out for that name within the hour. Set automatic retry intervals in hours, not minutes.
• Every publicly trusted certificate is written to transparency logs. The names of your internal services (vault.internal.example.com and the like) become publicly searchable. A wildcard certificate hides those names; per-name certificates do not.
• A wildcard covers exactly one level. *.example.com is valid for a.example.com but not for example.com (the apex) or a.b.example.com. Add the apex as a separate name.
• A redirect on port 80 can break validation. If you redirect HTTP to HTTPS, exempt /.well-known/acme-challenge/ from that rule, or make sure the redirect lands on a valid certificate; on first setup there is no certificate yet, so the second option is usually not available.

When not to choose this route

For environments that never touch the internet, air-gapped networks, and internal services that only your own clients talk to, a public CA is the wrong tool: validation depends on the outside world, your names end up in public logs, and short-lived certificates tie every renewal to external connectivity. Run a private CA instead. Some private CAs speak ACME, so you keep the same clients and the same automation; the only thing that changes is trust distribution, meaning you deliver the root certificate to clients yourself. Public ACME CAs also will not help if a compliance requirement demands organisation or extended validation (OV/EV), or if you are issuing client certificates for mTLS.

If you run a public service and certificate lifetimes keep shrinking by industry decision, manual certificate management has already stopped being an option. The right question is not "should we automate" but "with which validation method, storing state where, and protecting the quota how".