PKI، TLS، گواهی‌ها و GPGPKI, TLS, Certificates & GPG

Behind that little padlock next to a URL sits a multi-layer trust machine that most engineers only know one layer deep. Nobody asks questions while it works — but the moment this line shows up in a production log:

PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException:
unable to find valid certification path to requested target

the difference between a mid and a senior engineer becomes visible. One copies a TrustManager that "accepts everything"; the other knows within thirty seconds which link of the chain is missing.

This chapter takes the machine apart from the first screw: what a certificate actually is, the handshake, the daily tooling, automated rotation in Kubernetes, and finally the parallel universe of PGP/GPG, which solves the same problem with a completely different philosophy.

Part 1 — Three foundations

Asymmetric cryptography means a key pair where the private key cannot be derived from the public one. Encrypt with the public key so only the private key opens it; sign with the private key so anyone can verify with the public one. Today's algorithms: RSA (slow, large keys, still everywhere), ECDSA on P-256/secp256r1 (fast, small), and Ed25519 (the most modern; the new OpenPGP default).

Hashing is a one-way function turning any input into a fixed-length fingerprint (SHA-256 = 256 bits). Flipping one input bit changes the whole output, and finding two inputs with the same hash (a collision) must be infeasible. MD5 and SHA-1 are both broken; that is why the JDK has disabled SHA-1 handshake signatures by default since version 25.

A digital signature = hash the document, "encrypt" that hash with the private key, put the result next to the document. Verification = hash it yourself, open the signature with the public key, compare. Equal hashes give you both integrity and authenticity.

All of PKI and all of GPG solve exactly one sentence: "whose public key is this really?" PKI answers "because a trusted authority vouched for it"; PGP answers "because people I trust vouched for it".

Part 2 — What a certificate really is

A certificate is a standard X.509 v3 structure encoded with ASN.1/DER. Base64-wrap that DER between -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- and you get PEM — what you see inside .crt and .pem files.

# Fetch a server's certificate and read all of it
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -text

# Only what you need 90% of the time
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates -ext subjectAltName

A self-signed certificate means Issuer == Subject. Cryptographically it is perfectly valid and TLS works fine with it; the only problem is that nobody has a reason to trust it unless it is manually added to a trust store. Teams mint self-signed certificates for internal services and add an exception in each client; three months later they have forty certificates with no way to revoke or rotate anything. The right move is one internal CA: distribute its root once into every trust store, then issue a leaf per service — after that, rotating leaves requires zero client changes.

Part 3 — The chain of trust

A CEO does not sign every small contract; they delegate to deputies who sign instead. The counterparty sees the deputy's signature, checks the deputy's power of attorney, and arrives at the CEO's signature, of which they hold a reference copy. The root CA is the CEO, the intermediate is the deputy, and the server certificate is the contract.

• Root CA: self-signed, key in an offline HSM, 15–25 year lifetime, and present in the OS/browser/JDK trust store.
• Intermediate CA: signed by the root, online, issuing millions of certificates. A compromise burns only the intermediate, not the root.
• Leaf: your server certificate, CA:FALSE, unable to sign anything.

Chain of trust from root to leaf — زنجیره از root تا برگ:

flowchart TD
  R["Root CA<br/>self-signed, offline HSM<br/>in OS/JDK trust store"]
  I["Intermediate CA<br/>online, signs millions<br/>sent by the server"]
  L["Leaf certificate<br/>SAN = api.example.com<br/>CA:FALSE"]
  K["Server private key<br/>never leaves the host"]
  R -->|signs| I
  I -->|signs| L
  L -.-> K

Golden rule: the server must send the leaf plus every intermediate, and must not send the root. The client uses AKI/Issuer to build the chain until it reaches a root in its own trust store.

openssl s_client -connect api.example.com:443 -servername api.example.com -showcerts </dev/null
# You should see two or more CERTIFICATE blocks and, at the end: Verify return code: 0 (ok)

Part 4 — CSR, issuance and expiry

A CSR is a signed request saying "here is my public key, here are the names I want, please sign". The CSR is signed with your own private key so the CA can confirm you actually hold it.

Certificate issuance flow — فرایند صدور گواهی:

sequenceDiagram
  participant S as Server / Ops
  participant CA as Certificate Authority
  participant DNS as DNS / HTTP endpoint
  S->>S: generate private key (never leaves)
  S->>S: build CSR (public key + SANs), self-sign it
  S->>CA: submit CSR
  CA->>DNS: domain control validation (HTTP-01 / DNS-01)
  DNS-->>CA: challenge satisfied
  CA->>CA: apply policy, add SCT, sign with intermediate key
  CA-->>S: leaf certificate + chain
  S->>S: install cert + chain, reload server

Three CSR truths beginners get wrong: (1) the private key is never in the CSR and must never be sent to the CA — if a provider generates the key for you, that key is no longer private. (2) The CA is not obliged to honour what is in the CSR; in practice everything except the public key and the SANs is decided by the CA. (3) Every key rotation needs a fresh CSR.

# 1) A modern private key (ECDSA P-256)
openssl ecparam -name prime256v1 -genkey -noout -out api.key
chmod 600 api.key

# 2) CSR with a SAN — note: -noenc replaces the old -nodes in OpenSSL 3.x
openssl req -new -key api.key -out api.csr \
  -subj "/CN=api.example.com/O=Example Inc" \
  -addext "subjectAltName=DNS:api.example.com,DNS:www.api.example.com"

# 3) Always read the CSR before sending it
openssl req -in api.csr -noout -text -verify

SAN, wildcards and picking a strategy

DNS:api.example.com is that exact name. DNS:*.example.com covers exactly one label: it matches a.example.com, does not match a.b.example.com, and does not match the apex example.com. For IPs you must write IP:10.0.0.5; putting an IP into a DNS entry does nothing.

Wildcards are tempting, but they mean the same private key gets copied to every machine serving any subdomain — from the load balancer down to that experimental service nobody patches. With ACME, issuing per name is essentially free. Keep wildcards for cases where subdomains are genuinely dynamic (multi-tenant naming) and on-demand issuance is impractical.

Expiry: the biggest recurring incident class in engineering

Public certificate lifetimes are shrinking fast. Per CA/Browser Forum ballot SC-081v3 (April 2025):

Let's Encrypt has gone further: its shortlived profile, valid for 160 hours (just over six days), became generally available in January 2026, and it has announced that the default classic profile drops to 64 days in February 2027 and to 45 days in February 2028.

If you keep that inventory in a database, the alerting query looks like this:

SELECT common_name, environment, not_after,
       (not_after::date - CURRENT_DATE) AS days_left
FROM   cert_inventory
WHERE  not_after < CURRENT_DATE + INTERVAL '21 days'
ORDER  BY not_after;

SELECT common_name, environment, not_after,
       TRUNC(not_after) - TRUNC(SYSDATE) AS days_left
FROM   cert_inventory
WHERE  not_after < SYSDATE + 21
ORDER  BY not_after;

Part 5 — Revocation: CRL, OCSP and stapling

Suppose a server's private key leaks while the certificate still has 40 days left. How do we tell the world it is no longer valid?

• CRL: the CA publishes a signed list of revoked serial numbers (URL in CRL Distribution Points). It is bulky and must be re-downloaded regularly.
• OCSP: the client asks a responder about one certificate and gets good/revoked/unknown. That is a round trip on the critical path plus a privacy leak — the CA learns which IP is visiting which site.
• OCSP Stapling: the server itself fetches a fresh signed response from the CA and "staples" it into the handshake (the status_request extension). The client never contacts the CA.

OCSP stapling flow — جریان stapling:

sequenceDiagram
  participant C as Client
  participant S as Server
  participant O as OCSP responder (CA)
  Note over S,O: periodically, off the critical path
  S->>O: OCSP request for its own serial
  O-->>S: signed response (good, valid ~7 days)
  C->>S: ClientHello + status_request extension
  S-->>C: ServerHello + Certificate + stapled OCSP response
  C->>C: verify signature and freshness locally

The "must-staple" extension (OID 1.3.6.1.5.5.7.1.24) turns soft-fail into hard-fail: "if you do not see a stapled response, abort". Powerful but dangerous — a stapling failure then takes your site fully offline instead of merely slowing it down.

server {
    listen 443 ssl;
    http2 on;
    server_name api.example.com;

    ssl_certificate     /etc/ssl/api/fullchain.pem;   # leaf + intermediates
    ssl_certificate_key /etc/ssl/api/privkey.pem;

    ssl_protocols       TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers off;                    # in TLS 1.3 the client chooses
    ssl_session_cache   shared:SSL:10m;
    ssl_session_tickets off;                          # unrotated ticket keys break forward secrecy

    ssl_stapling        on;
    ssl_stapling_verify on;
    ssl_trusted_certificate /etc/ssl/api/chain.pem;   # only to verify the OCSP response

    add_header Strict-Transport-Security "max-age=63072000" always;
}

Part 6 — TLS: the handshake, step by step

Two people in a crowded café want to talk privately, having never met: (1) verify who the other is via their passport, (2) agree on a shared secret language, (3) build a shared password neither of them ever transmitted in full, (4) speak only that language from now on. The handshake is exactly those four steps.

TLS 1.2 handshake (two round trips before data) — دست‌دادن TLS 1.2:

sequenceDiagram
  participant C as Client
  participant S as Server
  C->>S: ClientHello (versions, cipher suites, SNI, ALPN, random)
  S-->>C: ServerHello (chosen suite, random)
  S-->>C: Certificate (leaf + intermediates)
  S-->>C: ServerKeyExchange (ECDHE params, signed) + ServerHelloDone
  C->>S: ClientKeyExchange (ECDHE public share)
  C->>S: ChangeCipherSpec + Finished
  S-->>C: ChangeCipherSpec + Finished
  C->>S: Application data (HTTP request)

Cost: 2-RTT before the first byte of data. On a mobile link with 100 ms latency that is 200 ms of pure handshake, on top of TCP and DNS.

TLS 1.3 handshake (one round trip, encrypted early), RFC 8446 — دست‌دادن TLS 1.3:

sequenceDiagram
  participant C as Client
  participant S as Server
  C->>S: ClientHello + key_share (guessed group) + supported_versions + SNI/ALPN
  S-->>C: ServerHello + key_share
  Note over C,S: shared secret derived — everything below is encrypted
  S-->>C: {EncryptedExtensions, Certificate, CertificateVerify, Finished}
  C->>S: {Finished} + Application data
  S-->>C: {NewSessionTicket} for future resumption

TLS 1.3 got safer by deleting things: RSA key transport, static DH, compression, renegotiation and every non-AEAD suite are gone, so an entire family of attacks (ROBOT, CRIME, Logjam, FREAK) is simply not expressible any more.

Anatomy of a cipher suite

Read TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 piece by piece: ECDHE is the key exchange (that trailing E means ephemeral — a temporary key thrown away afterwards), RSA is the server's signature algorithm (i.e. the certificate key type), AES_128_GCM is the symmetric cipher in AEAD mode, and SHA256 is the hash used by the KDF. TLS 1.3 has exactly five suites and all are AEAD: TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_AES_128_CCM_SHA256, TLS_AES_128_CCM_8_SHA256.

SNI, ALPN, ECH and resumption

SNI sends the hostname in cleartext inside ClientHello so a server hosting a hundred domains on one IP knows which certificate to present; without it, HTTPS virtual hosting is impossible. ALPN, in the same message, declares that the client speaks h2/http/1.1/h3 — HTTP/2 over TLS does not start without it, and gRPC depends on it for the same reason. ECH is the next step: it encrypts ClientHello itself with a key published in DNS so even the SNI is hidden.

If a Java connection behind a proxy comes back with the wrong certificate, check SNI first: run openssl s_client once with -servername and once without; two different certificates means SNI is the problem.

Resumption lets a later connection reuse a PSK: Session ID (state kept on the server, painful across nodes) or Session Ticket (encrypted state held by the client, server stays stateless). If the ticket encryption key stays constant for years and then leaks, every past resumed session becomes decryptable — so rotate it frequently or set ssl_session_tickets off;.

Part 7 — mTLS

In ordinary TLS only the server proves itself. In mTLS the server also demands a certificate from the client and runs the same chain validation on it.

mTLS handshake — both sides present certificates:

sequenceDiagram
  participant C as Client (has cert + key)
  participant S as Server (has cert + key)
  C->>S: ClientHello
  S-->>C: ServerHello + Certificate + CertificateVerify
  S-->>C: CertificateRequest (acceptable CAs, sig algs)
  C->>S: Certificate (client leaf + chain)
  C->>S: CertificateVerify (proves possession of client key)
  S->>S: verify chain against client-CA truststore + check EKU clientAuth
  S-->>C: Finished

Part 8 — The Java side

keytool cheat sheet

Two classic keytool traps. First, when you import the CA-signed certificate, keytool must be able to build the whole chain; if the intermediate is neither in that keystore nor in cacerts you get Failed to establish chain from reply — import the intermediate first, or feed it a concatenated PEM chain. Second, the leaf must be imported under the same alias as the private key; under a different alias you create a separate trusted-certificate entry instead of completing the key entry, and the server keeps serving the old self-signed certificate.

cacerts and the corporate CA

# Is the CA already in the JDK's default truststore?
keytool -list -cacerts -storepass changeit | grep -i corp-root

# Add it; since JDK 9, -cacerts points straight at the default truststore
keytool -importcert -trustcacerts -noprompt -alias corp-root \
        -file corp-root.crt -cacerts -storepass changeit

Three ways to trust a private CA, best first: (1) a dedicated application truststore via -Djavax.net.ssl.trustStore=... — versionable, rotatable, no JDK surgery. (2) Adding it in the Dockerfile at build time — reproducible, but must re-run on every JDK upgrade. (3) Editing cacerts by hand on a server — gone on the next rollout; do not do this.

FROM eclipse-temurin:25-jre
COPY corp-root.crt /tmp/corp-root.crt
RUN keytool -importcert -trustcacerts -noprompt \
      -alias corp-root -file /tmp/corp-root.crt \
      -cacerts -storepass changeit \
 && rm /tmp/corp-root.crt

Building an SSLContext in code

public static SSLContext build(Path trustStore, char[] trustPass,
                               Path keyStore,   char[] keyPass) throws Exception {

    KeyStore ts = KeyStore.getInstance("PKCS12");
    try (InputStream in = Files.newInputStream(trustStore)) { ts.load(in, trustPass); }
    TrustManagerFactory tmf =
            TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
    tmf.init(ts);

    KeyManager[] kms = null;                       // only needed for mTLS
    if (keyStore != null) {
        KeyStore ks = KeyStore.getInstance("PKCS12");
        try (InputStream in = Files.newInputStream(keyStore)) { ks.load(in, keyPass); }
        KeyManagerFactory kmf =
                KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
        kmf.init(ks, keyPass);
        kms = kmf.getKeyManagers();
    }

    SSLContext ctx = SSLContext.getInstance("TLSv1.3");
    ctx.init(kms, tmf.getTrustManagers(), null);
    return ctx;
}

// HttpClient performs hostname verification for you
HttpClient client = HttpClient.newBuilder().sslContext(ctx).build();

SSLSocket socket = (SSLSocket) ctx.getSocketFactory().createSocket("api.example.com", 443);
SSLParameters p = socket.getSSLParameters();
p.setEndpointIdentificationAlgorithm("HTTPS");   // without this line, no hostname check happens
socket.setSSLParameters(p);
socket.startHandshake();

// Never, not even "temporarily" — this turns TLS into unauthenticated encryption
TrustManager[] trustAll = { new X509TrustManager() {
    public void checkClientTrusted(X509Certificate[] c, String a) { }
    public void checkServerTrusted(X509Certificate[] c, String a) { }   // ← MITM welcome
    public X509Certificate[] getAcceptedIssuers() { return new X509Certificate[0]; }
}};

The classic errors and their real fixes

# The fullest view of the handshake (verbose; test environments only)
java -Djavax.net.debug=ssl:handshake:verbose:trustmanager -jar app.jar
# Certification path building only
java -Djava.security.debug=certpath -jar app.jar
# Turn on revocation checking (off by default)
java -Dcom.sun.net.ssl.checkRevocation=true \
     -Djdk.tls.client.enableStatusRequestExtension=true -jar app.jar

If your own TLS server runs on the JVM, OCSP stapling support is off by default and is enabled with -Djdk.tls.server.enableStatusRequestExtension=true (JEP 249). On the client side jdk.tls.client.enableStatusRequestExtension is on by default, but the result is not enforced until com.sun.net.ssl.checkRevocation=true.

Spring Boot: SSL bundles

Since Spring Boot 3.1, instead of scattering keystore paths across ten properties you define a named bundle; since 3.2 it can hot-reload:

spring:
  ssl:
    bundle:
      pem:
        server:
          reload-on-update: true            # watches the files and rebuilds the bundle
          keystore:
            certificate: "file:/etc/tls/tls.crt"
            private-key: "file:/etc/tls/tls.key"
          truststore:
            certificate: "file:/etc/tls/ca.crt"
      jks:
        partner:
          keystore:
            location: "classpath:client.p12"
            password: "${CLIENT_P12_PASSWORD}"
            type: "PKCS12"
server:
  ssl:
    bundle: "server"

Part 9 — The openssl cookbook

Three combinations you will use constantly:

# 1) Does this private key match this certificate? (both outputs must be identical)
openssl x509 -noout -pubkey -in cert.pem | openssl sha256
openssl pkey  -pubout      -in key.pem   | openssl sha256

# 2) What chain is the server actually sending?
openssl s_client -connect api.example.com:443 -servername api.example.com \
  -showcerts </dev/null 2>/dev/null | grep -E "^(depth|verify|s:|i:)"

# 3) A complete private CA in 60 seconds (for dev and integration tests)
openssl req -x509 -newkey rsa:4096 -noenc -days 3650 -keyout ca.key -out ca.crt \
  -subj "/CN=Dev Root CA" \
  -addext "basicConstraints=critical,CA:TRUE,pathlen:0" \
  -addext "keyUsage=critical,keyCertSign,cRLSign"

openssl req -newkey rsa:2048 -noenc -keyout svc.key -out svc.csr -subj "/CN=svc.local"

openssl x509 -req -in svc.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out svc.crt -days 30 -sha256 \
  -extfile <(printf "subjectAltName=DNS:svc.local,IP:127.0.0.1\nextendedKeyUsage=serverAuth,clientAuth\nbasicConstraints=CA:FALSE")

openssl verify -CAfile ca.crt svc.crt      # → svc.crt: OK

A version note: -nodes (short for "no DES") is deprecated since OpenSSL 3.0 and its official replacement is -noenc. Both mean "do not encrypt the private key" — so remember chmod 600 and correct ownership, and in production pull the key from a secret manager rather than a repository.

Part 10 — Pinning, Certificate Transparency and ACME

Pinning means "I accept only this public key (or this CA), even if another legitimate CA issues a perfectly valid certificate for the same domain". Its purpose is defending against mis-issuance by a CA. It has three levels: pinning the leaf certificate (breaks on every renewal — almost always wrong), pinning the leaf public key or SPKI (more stable, but it means you never rotate the key), and pinning the CA/intermediate (most stable, weakest protection).

# Compute an SPKI-style pin for a server
openssl s_client -connect api.example.com:443 -servername api.example.com </dev/null 2>/dev/null \
  | openssl x509 -pubkey -noout | openssl pkey -pubin -outform der \
  | openssl dgst -sha256 -binary | openssl enc -base64

ACME (RFC 8555) fully automates issuance: the client proves control of the domain and the CA issues — no humans, no forms.

ACME issuance with HTTP-01 challenge — صدور با ACME:

sequenceDiagram
  participant A as ACME client (certbot)
  participant CA as ACME server (Let's Encrypt)
  participant W as Your web server
  A->>CA: newOrder for api.example.com
  CA-->>A: challenge token + authorization URL
  A->>W: publish token at /.well-known/acme-challenge/TOKEN
  A->>CA: ready to be validated
  CA->>W: GET http://api.example.com/.well-known/acme-challenge/TOKEN
  W-->>CA: token + account key thumbprint
  CA-->>A: authorization valid
  A->>CA: finalize with CSR
  CA-->>A: certificate + chain

# Issue via webroot (the service keeps running)
certbot certonly --webroot -w /var/www/html \
  -d api.example.com -d www.api.example.com \
  --agree-tos -m ops@example.com --non-interactive

# Wildcards require DNS-01
certbot certonly --dns-cloudflare \
  --dns-cloudflare-credentials /etc/letsencrypt/cf.ini \
  -d "*.example.com" -d example.com

# Six-day certificate (the shortlived profile — Certbot 4.0 and later)
certbot certonly --preferred-profile shortlived --webroot -w /var/www/html -d api.example.com

# Dry-run renewal without burning rate limits, and reload after every success
certbot renew --dry-run
certbot renew --deploy-hook "systemctl reload nginx"

When building the automation, always start on staging — Let's Encrypt enforces rate limits and repeated failures will lock you out:

certbot certonly --server https://acme-staging-v02.api.letsencrypt.org/directory ...

Part 11 — Certificate rotation in Kubernetes

The standard pattern is cert-manager: a controller that understands Issuer/ClusterIssuer and Certificate resources, obtains the certificate, writes it into a kubernetes.io/tls Secret, and renews automatically before expiry.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ops@example.com
    profile: tlsserver          # ACME profile selection — cert-manager v1.18+
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
      - http01:
          ingress:
            ingressClassName: nginx
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: api-tls
  namespace: production
spec:
  secretName: api-tls            # tls.crt and tls.key land here
  duration: 2160h                # 90d
  renewBefore: 720h              # renew 30d before expiry
  privateKey:
    algorithm: ECDSA
    size: 256
    rotationPolicy: Always       # generate a new key on every renewal
  dnsNames:
    - api.example.com
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer

Up to cert-manager v1.17 the default rotationPolicy was Never — the certificate was renewed but the same old private key lived forever. From v1.18 the default is Always, so both key and certificate change at every renewal.

cert-manager rotation and pod reload — چرخش و بارگذاری مجدد:

flowchart LR
  C["Certificate CR<br/>renewBefore: 30d"] --> CM["cert-manager controller"]
  CM -->|ACME order| CA["Let's Encrypt"]
  CA -->|cert + chain| CM
  CM -->|writes| S["Secret kubernetes.io/tls<br/>tls.crt, tls.key"]
  S -->|projected volume| P["Pod filesystem /etc/tls"]
  P -->|file watcher / SIGHUP| APP["App reloads SSL context"]
  S -.->|read directly| ING["Ingress controller"]

In service meshes, workload certificates typically live for hours or less and are held in the sidecar's memory — they never touch disk. Identity is not a DNS name but a SPIFFE ID such as spiffe://cluster.local/ns/production/sa/api, carried in a URI-type SAN. Mesh architecture belongs to the microservice runtime chapter; the point here is that the same X.509 chain sits underneath it.

Part 12 — GPG/PGP: same problem, different philosophy

PGP is a standard (OpenPGP) and GnuPG/gpg is its most common free implementation; the current development branch is adding post-quantum encryption (Kyber/ML-KEM).

Key structure: primary and subkeys

flowchart TD
  P["Primary key (Ed25519)<br/>usage: Certify — keep offline"]
  S["Signing subkey<br/>usage: Sign"]
  E["Encryption subkey (Curve25519)<br/>usage: Encrypt"]
  A["Authentication subkey<br/>usage: Auth (SSH)"]
  U["User IDs — Name and email"]
  P -->|certifies| S
  P -->|certifies| E
  P -->|certifies| A
  P -->|binds| U

The primary key is your identity: it certifies your User IDs and subkeys and signs other people's keys. Subkeys do the day-to-day work. The big win: lose your laptop and you revoke and replace only the subkeys, while your identity and every signature ever collected on the primary survive. The senior pattern is to keep the primary offline (encrypted drive or a hardware token) and carry only subkeys on the working machine — gpg shows that state as sec#.

# Create a key; future-default today means ed25519 for signing + cv25519 for encryption
gpg --quick-generate-key "Jane Dev <jane@example.com>" future-default default 2y
gpg --quick-add-key <FINGERPRINT> ed25519 sign 1y      # a dedicated signing subkey

gpg --list-keys --keyid-format=long --with-subkey-fingerprints
gpg --list-secret-keys --keyid-format=long

gpg --armor --export jane@example.com > jane-public.asc          # to hand to others
gpg --armor --export-secret-subkeys jane@example.com > subs.asc  # working backup
gpg --import colleague-public.asc

gpg --quick-set-expire <FINGERPRINT> 2y '*'            # extend all subkeys

Encrypting, signing and verifying

# Encrypt for several recipients plus yourself (otherwise you cannot open it either)
gpg --encrypt -r bob@example.com -r audit@example.com -r jane@example.com \
    --output report.pdf.gpg report.pdf
gpg --decrypt report.pdf.gpg > report.pdf

gpg --symmetric --cipher-algo AES256 backup.tar     # symmetric, passphrase only
gpg --clear-sign announcement.txt                   # human-readable inline signature

# Detached signature — the release-artifact standard
gpg --armor --detach-sign app-1.4.0.tar.gz          # → app-1.4.0.tar.gz.asc
gpg --verify app-1.4.0.tar.gz.asc app-1.4.0.tar.gz

Detached signatures are the release standard because the file itself stays untouched: the same .tar.gz any tool can read, plus a small .asc beside it. Embed the signature inside the file and everyone is forced through gpg just to reach the content.

gpg --keyserver hkps://keys.openpgp.org --recv-keys <FULL_40_HEX_FINGERPRINT>
gpg --fingerprint <FULL_40_HEX_FINGERPRINT>    # compare against the independent source
gpg --lsign-key <FINGERPRINT>                  # local signature, never published

Keyservers, WKD and revocation

The old SKS keyservers let anyone attach any signature to any key and never deleted anything, which led to "key poisoning" — keys with tens of thousands of junk signatures that broke gpg outright. That network is effectively abandoned. keys.openpgp.org is the new model: identity information is only searchable after email verification and with the owner's consent, and third-party signatures are stripped. WKD serves the public key directly from the person's own domain over HTTPS under /.well-known/openpgpkey/... — PGP borrowing trust from PKI.

gpg --keyserver hkps://keys.openpgp.org --send-keys <FINGERPRINT>
gpg --locate-external-keys jane@example.com     # via WKD and keyservers
gpg --refresh-keys                              # pick up revocations and expiry changes

gpg --output revoke.asc --gen-revoke <FINGERPRINT>   # create it today, not on the bad day
gpg --import revoke.asc && gpg --keyserver hkps://keys.openpgp.org --send-keys <FINGERPRINT>

Signing git commits and tags

git config --global user.signingkey <SIGNING_SUBKEY_FPR>!   # the ! means "exactly this subkey"
git config --global commit.gpgsign true
git config --global tag.gpgsign true
export GPG_TTY=$(tty)                                       # otherwise no passphrase prompt

git commit -S -m "Fix token refresh race"
git tag -s v1.4.0 -m "Release 1.4.0"
git verify-commit HEAD && git verify-tag v1.4.0
git log --pretty="%h %G? %GS %s" -10    # %G? = G(ood) B(ad) U(ntrusted) N(one) E(rror)

Since Git 2.34 you can use an SSH key instead, with git config --global gpg.format ssh and user.signingkey ~/.ssh/id_ed25519.pub. It is simpler, but it has no expiry or revocation model and verification depends on a gpg.ssh.allowedSignersFile you maintain yourself. For signatures that must still verify years later, OpenPGP remains the more mature choice.

Note that a signature does not prove authorship, only who signed: the git Author field is free text anyone can set. A meaningful policy means branch protection requiring verified signatures plus a maintained list of allowed keys; without that, the signature is just a decorative green badge.

gpg cheat sheet

export GNUPGHOME=$(mktemp -d)
printf '%s' "$GPG_PRIVATE_KEY" | gpg --batch --import
echo "$GPG_PASSPHRASE" | gpg --batch --yes --pinentry-mode loopback \
    --passphrase-fd 0 --armor --detach-sign dist/app-1.4.0.tar.gz
rm -rf "$GNUPGHOME"

Part 13 — The senior operations playbook

• Keep an inventory. Every certificate, its owner, where it is deployed and when it expires. If you cannot say within thirty seconds which certificate expires next, expiry will ambush you.
• Monitor from the outside. A probe that connects to the real port catches both expiry and the far more common "renewed but never reloaded" failure.
• Use two alert thresholds: soft at 30% of remaining lifetime (ticket), hard at 10% (page). On a 47-day certificate that is 14 and 5 days.
• Keep private keys out of the repository. If a key ever entered git history it is burned; it must be revoked and replaced, not merely deleted from the commit.
• Rehearse rotation. Once a quarter, rotate the CA in staging. A team that has never practised CA rotation will be doing it for the first time on incident day.
• Take clocks seriously. A container without NTP produces exactly the errors that look like certificate problems; date -u is the first command of any TLS investigation.
• Move algorithms forward. Today that means ECDSA P-256 or Ed25519 for new certificates, TLS 1.2 as the floor and 1.3 as the default. The next step is post-quantum: the hybrid X25519MLKEM768 key exchange is enabled by default from OpenSSL 3.5 (April 2025), and the JDK has shipped ML-KEM and ML-DSA since version 24. "Harvest now, decrypt later" is a genuine threat to long-lived data.

Part 14 — Interview questions