fogserv.cloud fogserv.cloud

Certificate Fundamentals - PKI, X.509, and Trust

Resource Navigation: README | Let's Encrypt Automation | TLS Configuration | SSH Security

Summary

Public Key Infrastructure (PKI) and X.509 certificates form the foundation of secure communications, authentication, and encryption across the internet. This comprehensive guide covers certificate fundamentals including public/private key cryptography, X.509 certificate structure and extensions, certificate signing requests (CSRs), certificate authorities (CA) and chain of trust, self-signed vs commercial certificates, creating your own internal CA with OpenSSL and step-ca, certificate lifecycle management (issuance, renewal, revocation), certificate pinning, mutual TLS (mTLS) authentication, troubleshooting certificate errors, and PKI best practices. Learn production patterns for web servers, API authentication, service-to-service communication, code signing, and complete enterprise PKI architectures.

The Golden Rule: Trust is earned through verification - always validate certificate chains.

Learning Objectives

By the end of this guide, you will be able to:

Prerequisites

Before working with certificates, you should have:

Public Key Cryptography Basics

Symmetric vs Asymmetric Encryption

┌──────────────────────────────────────────────────────┐
│         Symmetric Encryption (Shared Key)             │
└──────────────────────────────────────────────────────┘

    Alice                                    Bob
      │                                      │
      │    Same Key (K)                      │
      │  ┌──────────────┐                    │
      ├──┤  Secret Key  ├────────────────────┤
      │  └──────────────┘                    │
      │                                      │
      │  Encrypt with K                      │
      │  "Hello" → [encrypted]               │
      ├─────────────────────────────────────→│
      │                                      │
      │                          Decrypt with K
      │                          [encrypted] → "Hello"

Problem: How do Alice and Bob securely share the key?

---

┌──────────────────────────────────────────────────────┐
│      Asymmetric Encryption (Public/Private Keys)      │
└──────────────────────────────────────────────────────┘

    Alice                                    Bob
      │                                      │
      │  ┌────────────┐                      │  ┌────────────┐
      │  │  Private   │                      │  │  Private   │
      │  │    Key     │ (secret)             │  │    Key     │ (secret)
      │  └────────────┘                      │  └────────────┘
      │  ┌────────────┐                      │  ┌────────────┐
      │  │   Public   │                      │  │   Public   │
      │  │    Key     │ (shared)             │  │    Key     │ (shared)
      │  └────────────┘                      │  └────────────┘
      │        │                             │        │
      │        └─────────────────────────────┴────────┘
      │                                      │
      │  Encrypt with Bob's Public Key       │
      │  "Hello" → [encrypted]               │
      ├─────────────────────────────────────→│
      │                                      │
      │                          Decrypt with Bob's Private Key
      │                          [encrypted] → "Hello"

Advantages:
✅ No need to share private keys
✅ Can distribute public keys freely
✅ Enables digital signatures
✅ Foundation for certificates

Digital Signatures

┌──────────────────────────────────────────────────────┐
│            How Digital Signatures Work                │
└──────────────────────────────────────────────────────┘

Alice wants to send a signed message to Bob:

1. Alice creates message:
   Message: "Transfer $100 to Bob"

2. Alice computes hash:
   Hash(Message) → a1b2c3d4e5...
   
3. Alice encrypts hash with her PRIVATE key:
   Encrypt(Hash, Alice-Private-Key) → [SIGNATURE]
   
4. Alice sends both message and signature:
   Message + [SIGNATURE] ──→ Bob

5. Bob verifies:
   a) Decrypt signature with Alice's PUBLIC key:
      Decrypt([SIGNATURE], Alice-Public-Key) → Hash1
   
   b) Compute hash of received message:
      Hash(Message) → Hash2
   
   c) Compare:
      if Hash1 == Hash2:
          ✅ Message is authentic (from Alice)
          ✅ Message wasn't tampered with
      else:
          ❌ Invalid signature!

This is how certificates are signed by Certificate Authorities!

X.509 Certificate Structure

Certificate Components

┌──────────────────────────────────────────────────────┐
│              X.509 Certificate                        │
└──────────────────────────────────────────────────────┘

Certificate:
    Version: 3 (0x2)
    Serial Number: 1a:2b:3c:4d:5e:6f
    Signature Algorithm: sha256WithRSAEncryption
    
    Issuer: CN=Example CA, O=Example Org, C=US
    ↑ Who signed this certificate?
    
    Validity:
        Not Before: Jan 1 00:00:00 2026 GMT
        Not After : Jan 1 00:00:00 2027 GMT
    ↑ Certificate validity period
    
    Subject: CN=example.com, O=Example Inc, C=US
    ↑ Who this certificate identifies
    
    Subject Public Key Info:
        Public Key Algorithm: RSA
        Public Key: (2048 bit)
            00:a1:b2:c3:d4:e5:f6:...
    ↑ The actual public key
    
    X509v3 extensions:
        X509v3 Subject Alternative Name:
            DNS:example.com
            DNS:www.example.com
            DNS:*.example.com
        ↑ Additional domains (wildcards)
        
        X509v3 Key Usage: critical
            Digital Signature, Key Encipherment
        ↑ What this key can be used for
        
        X509v3 Extended Key Usage:
            TLS Web Server Authentication
            TLS Web Client Authentication
        ↑ Specific use cases
        
        X509v3 Basic Constraints: critical
            CA:FALSE
        ↑ Not a CA certificate
        
        X509v3 Authority Key Identifier:
            keyid:11:22:33:44:55:66:77:88:99:00
        ↑ Identifier of the CA that signed this
        
        X509v3 CRL Distribution Points:
            Full Name: URI:http://crl.example.com/ca.crl
        ↑ Where to check if certificate is revoked
        
    Signature Algorithm: sha256WithRSAEncryption
         a1:b2:c3:d4:e5:f6:07:08:09:0a:0b:0c:...
    ↑ CA's digital signature of the certificate

Examine Certificate

# View certificate details
openssl x509 -in certificate.crt -text -noout

# Check specific fields
openssl x509 -in certificate.crt -noout -subject
openssl x509 -in certificate.crt -noout -issuer
openssl x509 -in certificate.crt -noout -dates
openssl x509 -in certificate.crt -noout -ext subjectAltName

# Verify certificate chain
openssl verify -CAfile ca.crt certificate.crt

# Check certificate against private key
openssl x509 -noout -modulus -in certificate.crt | openssl md5
openssl rsa -noout -modulus -in private.key | openssl md5
# If md5 hashes match, certificate and key are a pair

Chain of Trust

How Certificate Trust Works

┌──────────────────────────────────────────────────────┐
│            Certificate Chain of Trust                 │
└──────────────────────────────────────────────────────┘

                   ┌─────────────────┐
                   │   Root CA       │
                   │ (Self-signed)   │
                   │                 │
                   │ Trusted by OS/  │
                   │ Browser         │
                   └────────┬────────┘
                            │ Signs
                            ↓
                   ┌─────────────────┐
                   │ Intermediate CA │
                   │                 │
                   │ Issued by Root  │
                   │                 │
                   └────────┬────────┘
                            │ Signs
                            ↓
                   ┌─────────────────┐
                   │  End-Entity     │
                   │  Certificate    │
                   │  (example.com)  │
                   │                 │
                   └─────────────────┘

Verification Process:
1. Browser receives example.com certificate
2. Browser checks: Who signed this? → Intermediate CA
3. Browser checks: Who signed Intermediate? → Root CA
4. Browser checks: Is Root CA trusted? → YES (in trust store)
5. Browser validates entire chain
6. ✅ Connection established

If any link breaks:
❌ Certificate expired
❌ Signature doesn't match
❌ Root CA not trusted
❌ Certificate revoked
→ Connection refused!

Root CA Trust Stores

# View system trusted CAs (Ubuntu/Debian)
ls /etc/ssl/certs/
cat /etc/ssl/certs/ca-certificates.crt

# Update CA trust store
sudo update-ca-certificates

# Add custom CA to trust store
sudo cp custom-ca.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates

# View trusted CAs (RHEL/CentOS)
ls /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust

# Firefox trust store (separate from OS)
# about:preferences#privacy → Certificates → View Certificates

# Check what CAs trust a certificate
openssl s_client -connect example.com:443 -showcerts

Creating a Certificate Authority

Option 1: OpenSSL CA

# Create CA directory structure
mkdir -p ~/ca/{certs,crl,newcerts,private}
cd ~/ca
touch index.txt
echo 1000 > serial

# Create CA configuration
cat > openssl.cnf << 'EOF'
[ ca ]
default_ca = CA_default

[ CA_default ]
dir               = /home/user/ca
certs             = $dir/certs
crl_dir           = $dir/crl
new_certs_dir     = $dir/newcerts
database          = $dir/index.txt
serial            = $dir/serial
RANDFILE          = $dir/private/.rand

private_key       = $dir/private/ca.key.pem
certificate       = $dir/certs/ca.cert.pem

crlnumber         = $dir/crlnumber
crl               = $dir/crl/ca.crl.pem
crl_extensions    = crl_ext
default_crl_days  = 30

default_md        = sha256

name_opt          = ca_default
cert_opt          = ca_default
default_days      = 375
preserve          = no
policy            = policy_loose

[ policy_loose ]
countryName             = optional
stateOrProvinceName     = optional
localityName            = optional
organizationName        = optional
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

[ req ]
default_bits        = 2048
distinguished_name  = req_distinguished_name
string_mask         = utf8only
default_md          = sha256
x509_extensions     = v3_ca

[ req_distinguished_name ]
countryName                     = Country Name (2 letter code)
stateOrProvinceName             = State or Province Name
localityName                    = Locality Name
0.organizationName              = Organization Name
organizationalUnitName          = Organizational Unit Name
commonName                      = Common Name
emailAddress                    = Email Address

[ v3_ca ]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true
keyUsage = critical, digitalSignature, cRLSign, keyCertSign

[ v3_intermediate_ca ]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true, pathlen:0
keyUsage = critical, digitalSignature, cRLSign, keyCertSign

[ server_cert ]
basicConstraints = CA:FALSE
nsCertType = server
nsComment = "OpenSSL Generated Server Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth

[ client_cert ]
basicConstraints = CA:FALSE
nsCertType = client, email
nsComment = "OpenSSL Generated Client Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
keyUsage = critical, nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, emailProtection

[ crl_ext ]
authorityKeyIdentifier=keyid:always
EOF

# Generate CA private key (4096-bit for security)
openssl genrsa -aes256 -out private/ca.key.pem 4096
chmod 400 private/ca.key.pem

# Generate CA certificate (self-signed)
openssl req -config openssl.cnf \
      -key private/ca.key.pem \
      -new -x509 -days 7300 -sha256 -extensions v3_ca \
      -out certs/ca.cert.pem

# Enter CA information:
# Country: US
# State: California
# Locality: San Francisco
# Organization: Example CA
# Common Name: Example Root CA

# Verify CA certificate
openssl x509 -noout -text -in certs/ca.cert.pem

# View certificate
cat certs/ca.cert.pem

Issue Server Certificate

# Generate server private key
openssl genrsa -out private/example.com.key.pem 2048
chmod 400 private/example.com.key.pem

# Create certificate signing request (CSR)
openssl req -config openssl.cnf \
      -key private/example.com.key.pem \
      -new -sha256 -out certs/example.com.csr.pem

# Enter server information:
# Common Name: example.com
# (other fields can match CA or be different)

# Create SAN extension file
cat > example.com.ext << EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = example.com
DNS.2 = www.example.com
DNS.3 = *.example.com
IP.1 = 192.168.1.100
EOF

# Sign certificate with CA
openssl ca -config openssl.cnf \
      -extensions server_cert \
      -days 375 -notext -md sha256 \
      -in certs/example.com.csr.pem \
      -out certs/example.com.cert.pem \
      -extfile example.com.ext

# Verify certificate
openssl x509 -noout -text -in certs/example.com.cert.pem

# Verify certificate chain
openssl verify -CAfile certs/ca.cert.pem certs/example.com.cert.pem

Option 2: step-ca (Modern CA)

# Install step and step-ca
wget -O step.tar.gz https://dl.smallstep.com/gh-release/cli/docs-cli-install/v0.25.0/step_linux_0.25.0_amd64.tar.gz
tar -xzf step.tar.gz
sudo mv step_0.25.0/bin/step /usr/local/bin/

wget -O step-ca.tar.gz https://dl.smallstep.com/gh-release/certificates/docs-ca-install/v0.25.0/step-ca_linux_0.25.0_amd64.tar.gz
tar -xzf step-ca.tar.gz
sudo mv step-ca_0.25.0/bin/step-ca /usr/local/bin/

# Initialize CA
step ca init

# Follow prompts:
# - CA name: Example Internal CA
# - DNS name: ca.example.internal
# - Address: :443
# - Provisioner: admin

# Start CA server
step-ca $(step path)/config/ca.json

# In another terminal, bootstrap trust
step ca bootstrap --ca-url https://ca.example.internal \
                  --fingerprint <fingerprint-from-init>

# Issue certificate
step ca certificate example.com example.com.crt example.com.key

# Renew certificate
step ca renew example.com.crt example.com.key

# Revoke certificate
step ca revoke --cert-file example.com.crt

Certificate Lifecycle Management

Certificate Renewal

#!/bin/bash
# renew-certificates.sh - Automated certificate renewal

CERT_DIR="/etc/ssl/certs"
KEY_DIR="/etc/ssl/private"
CA_CERT="/path/to/ca.cert.pem"
DOMAINS=("example.com" "api.example.com" "www.example.com")

for domain in "${DOMAINS[@]}"; do
    CERT_FILE="$CERT_DIR/$domain.crt"
    KEY_FILE="$KEY_DIR/$domain.key"
    
    # Check if certificate expires in less than 30 days
    if openssl x509 -checkend 2592000 -noout -in "$CERT_FILE"; then
        echo "✅ $domain certificate valid for more than 30 days"
    else
        echo "⚠️  $domain certificate expires soon, renewing..."
        
        # Generate new CSR
        openssl req -new -key "$KEY_FILE" \
                    -out "$CERT_DIR/$domain.csr" \
                    -subj "/CN=$domain"
        
        # Sign with CA (adjust for your CA)
        step ca certificate "$domain" "$CERT_FILE" "$KEY_FILE" --force
        
        # Reload web server
        sudo systemctl reload nginx
        
        echo "✅ $domain certificate renewed"
        
        # Send notification
        echo "Certificate renewed: $domain" | \
            mail -s "Certificate Renewal" admin@example.com
    fi
done

Certificate Revocation

# Generate Certificate Revocation List (CRL)
cd ~/ca

# Revoke a certificate
openssl ca -config openssl.cnf -revoke certs/example.com.cert.pem

# Generate CRL
openssl ca -config openssl.cnf -gencrl -out crl/ca.crl.pem

# View CRL
openssl crl -in crl/ca.crl.pem -noout -text

# Publish CRL (make available via HTTP)
sudo cp crl/ca.crl.pem /var/www/html/ca.crl

# Check if certificate is revoked
openssl verify -crl_check -CAfile certs/ca.cert.pem \
               -CRLfile crl/ca.crl.pem \
               certs/example.com.cert.pem

OCSP (Online Certificate Status Protocol)

# Start OCSP responder
openssl ocsp -port 8080 \
             -text \
             -CA certs/ca.cert.pem \
             -index index.txt \
             -rkey private/ca.key.pem \
             -rsigner certs/ca.cert.pem

# Check certificate status
openssl ocsp -CAfile certs/ca.cert.pem \
             -url http://localhost:8080 \
             -resp_text \
             -issuer certs/ca.cert.pem \
             -cert certs/example.com.cert.pem

Mutual TLS (mTLS) Authentication

Server Configuration (Nginx)

# /etc/nginx/sites-available/mtls-example

server {
    listen 443 ssl;
    server_name api.example.com;
    
    # Server certificate
    ssl_certificate /etc/ssl/certs/api.example.com.crt;
    ssl_certificate_key /etc/ssl/private/api.example.com.key;
    
    # Client certificate verification
    ssl_client_certificate /etc/ssl/certs/ca.cert.pem;
    ssl_verify_client on;
    ssl_verify_depth 2;
    
    # Optional: CRL checking
    ssl_crl /etc/ssl/crl/ca.crl.pem;
    
    location / {
        # Client certificate info available in headers
        proxy_set_header X-SSL-Client-Cert $ssl_client_cert;
        proxy_set_header X-SSL-Client-S-DN $ssl_client_s_dn;
        proxy_set_header X-SSL-Client-Verify $ssl_client_verify;
        
        proxy_pass http://backend;
    }
    
    # Allow specific endpoints without client cert
    location /public {
        ssl_verify_client optional;
        proxy_pass http://backend;
    }
}

Client Certificate

# Generate client private key
openssl genrsa -out client.key.pem 2048

# Generate client CSR
openssl req -new -key client.key.pem -out client.csr.pem \
        -subj "/CN=client@example.com/O=Example Org"

# Sign client certificate
openssl ca -config openssl.cnf \
      -extensions client_cert \
      -days 375 -notext -md sha256 \
      -in client.csr.pem \
      -out client.cert.pem

# Create PKCS#12 bundle (for browsers)
openssl pkcs12 -export \
        -out client.p12 \
        -inkey client.key.pem \
        -in client.cert.pem \
        -certfile ca.cert.pem \
        -name "Client Certificate"

# Test mTLS connection
curl --cert client.cert.pem \
     --key client.key.pem \
     --cacert ca.cert.pem \
     https://api.example.com/

# Without client cert (should fail)
curl --cacert ca.cert.pem https://api.example.com/
# 400 Bad Request - No required SSL certificate was sent

Troubleshooting Certificates

Common Errors

# Error: certificate has expired
# Check expiration date
openssl x509 -in cert.crt -noout -enddate

# Error: self signed certificate
# CA certificate not in trust store
sudo cp ca.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates

# Error: unable to get local issuer certificate
# Missing intermediate certificate
cat server.crt intermediate.crt > fullchain.crt

# Error: certificate name does not match
# CN/SAN doesn't match the domain
openssl x509 -in cert.crt -noout -text | grep -A1 "Subject Alternative Name"

# Error: unable to verify the first certificate
# Check certificate chain order
openssl s_client -connect example.com:443 -showcerts

# Debug TLS connection
openssl s_client -connect example.com:443 -debug -state

Certificate Validation Script

#!/bin/bash
# validate-certificates.sh

CERT_DIR="/etc/ssl/certs"
WARN_DAYS=30

echo "=== Certificate Validation Report ==="
echo

for cert in "$CERT_DIR"/*.crt; do
    [ -f "$cert" ] || continue
    
    echo "Certificate: $cert"
    
    # Check expiration
    if ! openssl x509 -checkend $((WARN_DAYS * 86400)) -noout -in "$cert" 2>/dev/null; then
        echo "  ⚠️  WARNING: Expires within $WARN_DAYS days"
        openssl x509 -enddate -noout -in "$cert"
    else
        echo "  ✅ Valid"
    fi
    
    # Check if self-signed
    ISSUER=$(openssl x509 -noout -issuer -in "$cert" | sed 's/issuer=//')
    SUBJECT=$(openssl x509 -noout -subject -in "$cert" | sed 's/subject=//')
    
    if [ "$ISSUER" == "$SUBJECT" ]; then
        echo "  ℹ️  Self-signed certificate"
    fi
    
    # Check key length
    KEY_LENGTH=$(openssl x509 -noout -text -in "$cert" | grep "Public-Key:" | grep -oP '\d+')
    if [ "$KEY_LENGTH" -lt 2048 ]; then
        echo "  ⚠️  WARNING: Key length $KEY_LENGTH bits (recommend 2048+)"
    fi
    
    # Check signature algorithm
    SIG_ALG=$(openssl x509 -noout -text -in "$cert" | grep "Signature Algorithm" | head -1 | awk '{print $3}')
    if [[ "$SIG_ALG" == *"sha1"* ]]; then
        echo "  ⚠️  WARNING: Using deprecated SHA-1"
    fi
    
    echo
done

Production Best Practices

Certificate Management Checklist

☑ Use certificates from trusted CAs (not self-signed in production)
☑ Minimum 2048-bit RSA or 256-bit ECC keys
☑ Use SHA-256 or better signature algorithm
☑ Include all Subject Alternative Names (SANs)
☑ Set appropriate validity period (max 398 days for public certs)
☑ Implement automated renewal (Let's Encrypt, cert-manager)
☑ Store private keys securely (chmod 400, encrypted)
☑ Never commit private keys to version control
☑ Implement certificate monitoring and alerting
☑ Have backup certificates ready
☑ Document certificate locations and renewal procedures
☑ Implement certificate pinning for critical services
☑ Use separate certificates per service (not wildcard everywhere)
☑ Implement mTLS for service-to-service communication
☑ Regular certificate audits

What's Next?

After mastering certificate fundamentals:

Automation:

Advanced PKI:

Authentication:

Additional Resources

Official Documentation

Tools

Learning Resources

Change Log

Choose Theme

Your selection is saved locally.

Neural Cacophony
Aperture v2
Flux v1
Mosaic Chaos
Nexus v1
Nexus Zest
Prism v2
Synapse