Skip to content

SSH Certificate Authority

Watchgrid includes a built-in SSH Certificate Authority (CA) that issues short-lived certificates, so you don't have to maintain authorized_keys files on every device.

How It Works

Instead of copying SSH public keys to every device, Watchgrid signs your SSH key with a trusted CA certificate. Every provisioned device already trusts this CA, so your signed certificate grants access automatically.

Certificate Type Validity Use Case
User certificates 24 hours SSH into devices
Host certificates 365 days Devices prove their identity

Security Features

  • Short-lived certificates — user certs expire after 24 hours, limiting exposure
  • Source address restriction — user certs are locked to the WireGuard subnet (100.64.0.0/10)
  • Certificate revocation — revoked certs are tracked in a CRL that devices check on each heartbeat
  • Full audit trail — every certificate issuance is logged

Requesting a User Certificate

  1. Go to System → PKI / SSH CA
  2. In the Request User Certificate section:
  3. Paste your SSH public key (e.g., contents of ~/.ssh/id_ed25519.pub)
  4. Optionally enter a username
  5. Click Request Certificate
  6. Click Download to save the signed certificate file (*-cert.pub)
  7. Place the certificate alongside your private key:
# If your key is ~/.ssh/id_ed25519, save the cert as:
~/.ssh/id_ed25519-cert.pub

# Then SSH normally — the cert is used automatically
ssh admin@pi-sensor-1.wg

The certificate is valid for 24 hours. After that, request a new one.

CA Status

The PKI page displays:

  • CA operational status — whether the CA keys are loaded and functional
  • Total certificates issued
  • Active / Expired / Revoked counts
  • User vs. Host certificate breakdown

Certificate Audit

A table at the bottom lists all issued certificates:

Column Description
Type User or Host
Username The principal on the certificate
Serial Unique certificate serial number
Issued When the certificate was signed
Expiry When the certificate expires
Status Active (green), Expired (gray), or Revoked (red)

Host Certificates

Host certificates are issued automatically during device provisioning. They prove the device's identity to connecting SSH clients, preventing man-in-the-middle attacks.

To trust host certificates on your workstation, add this to ~/.ssh/known_hosts:

@cert-authority *.wg <contents of ca_host_key.pub>

The CA host public key is available at:

http://YOUR_SERVER:8080/api/ssh/ca-host-key

CA Key Storage

CA keys are stored in a Docker volume (watchgrid-ssh-ca) mapped to /etc/watchgrid/:

  • ca_user_key / ca_user_key.pub — signs user certificates
  • ca_host_key / ca_host_key.pub — signs host certificates

Back up your CA keys

Losing the CA private keys invalidates all issued host certificates. Every device would need to be reprovisioned. Back up the watchgrid-ssh-ca volume in production.

Backup & Restore

Why this matters

Losing the SSH CA private keys invalidates every host certificate in the fleet. Devices will fail StrictHostKeyChecking and need to be fully reprovisioned. Back up the CA keys before any production incident occurs.

Automated daily backup

scripts/backup-ssh-ca.sh creates an AES-256-CBC encrypted tarball of all four CA key files and copies it to a configurable destination. It retains the last 14 backups by default.

Installation

# Copy the script to the server
sudo cp scripts/backup-ssh-ca.sh /usr/local/bin/watchgrid-ca-backup.sh
sudo chmod 0750 /usr/local/bin/watchgrid-ca-backup.sh

# Create an env file with your passphrase (protect it!)
sudo mkdir -p /etc/watchgrid
sudo bash -c 'cat > /etc/watchgrid/backup.env' <<'ENV'
WG_BACKUP_PASSPHRASE=<strong-random-passphrase-store-separately>
WG_BACKUP_DEST=/var/backups/watchgrid-ssh-ca
WG_BACKUP_KEEP=14
ENV
sudo chmod 0600 /etc/watchgrid/backup.env

# Install and start the systemd timer
sudo cp scripts/systemd/watchgrid-ca-backup.{service,timer} /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now watchgrid-ca-backup.timer

# Verify the timer is active
systemctl list-timers watchgrid-ca-backup.timer

Manual backup

WG_BACKUP_PASSPHRASE=<passphrase> \
WG_BACKUP_DEST=/var/backups/watchgrid-ssh-ca \
  /usr/local/bin/watchgrid-ca-backup.sh

Remote backup (rsync)

WG_BACKUP_PASSPHRASE=<passphrase> \
WG_BACKUP_DEST=backup@offsite-host:/backups/watchgrid-ssh-ca \
  /usr/local/bin/watchgrid-ca-backup.sh

Ensure the server has SSH key-based access to the remote host.

Restore procedure

RTO target: < 15 minutes for key restore; device reprovisioning is not required if keys are restored before any device reconnects.

Step 1 — Locate the latest encrypted backup

ls -lt /var/backups/watchgrid-ssh-ca/ | head -5
# → watchgrid-ssh-ca-2026-04-22T03-00-00.tar.gz.enc

Step 2 — Decrypt and extract

BACKUP_FILE=/var/backups/watchgrid-ssh-ca/watchgrid-ssh-ca-2026-04-22T03-00-00.tar.gz.enc
RESTORE_DIR=/tmp/ca-restore

mkdir -p "$RESTORE_DIR"
openssl enc -d -aes-256-cbc -pbkdf2 -iter 600000 \
  -pass "pass:<passphrase>" \
  -in "$BACKUP_FILE" | tar -xz -C "$RESTORE_DIR"

ls "$RESTORE_DIR/ca-keys/"
# → ca_user_key  ca_user_key.pub  ca_host_key  ca_host_key.pub

Step 3 — Restore into the running container

CONTAINER=watchgrid-server  # adjust to your container name

docker cp "$RESTORE_DIR/ca-keys/ca_user_key"     "${CONTAINER}:/etc/watchgrid/"
docker cp "$RESTORE_DIR/ca-keys/ca_user_key.pub" "${CONTAINER}:/etc/watchgrid/"
docker cp "$RESTORE_DIR/ca-keys/ca_host_key"     "${CONTAINER}:/etc/watchgrid/"
docker cp "$RESTORE_DIR/ca-keys/ca_host_key.pub" "${CONTAINER}:/etc/watchgrid/"

# Fix ownership inside container
docker exec "$CONTAINER" chown root:root /etc/watchgrid/ca_*
docker exec "$CONTAINER" chmod 600 /etc/watchgrid/ca_user_key /etc/watchgrid/ca_host_key
docker exec "$CONTAINER" chmod 644 /etc/watchgrid/ca_user_key.pub /etc/watchgrid/ca_host_key.pub

Step 4 — Verify

# Confirm keys are present and readable
docker exec "$CONTAINER" ls -la /etc/watchgrid/ca_*

# Test issuing a certificate (use any valid username)
curl -s -H "Authorization: Bearer <admin-jwt>" \
  https://YOUR_SERVER/api/ssh/sign-cert \
  -d '{"username":"test-verify"}' | jq .

Step 5 — Clean up

rm -rf "$RESTORE_DIR"

Volume restore (full stack rebuild)

If the Docker volume itself is lost (e.g., disk failure), restore using the backup before starting the server:

# Decrypt the tarball as above, then copy into the Docker volume:
docker volume create watchgrid-ssh-ca

docker run --rm \
  -v watchgrid-ssh-ca:/etc/watchgrid \
  -v "$RESTORE_DIR/ca-keys":/src:ro \
  alpine sh -c "cp /src/ca_* /etc/watchgrid/ && chmod 600 /etc/watchgrid/ca_user_key /etc/watchgrid/ca_host_key && chmod 644 /etc/watchgrid/ca_user_key.pub /etc/watchgrid/ca_host_key.pub"

# Start the stack — the server will pick up the restored keys on boot.
docker compose -f docker-compose.prod.yml up -d

Backup verification checklist

Run monthly to confirm backups are usable:

  • [ ] Latest backup exists and is within 25 hours old (ls -lt /var/backups/watchgrid-ssh-ca/ | head -2)
  • [ ] Decrypt succeeds without errors (Step 2 above)
  • [ ] Extracted files include all four key files
  • [ ] Passphrase stored separately from backup files (secrets manager, password vault, or offline)
  • [ ] Off-host copy transferred if using local-only backup