Skip to content

Security Operations

Jump to bottom
Eric Fitzgerald edited this page Apr 8, 2026 · 6 revisions

Security Operations

This guide covers container security, TLS management, secrets management, security monitoring, and incident response for TMI operations.

Overview

TMI security operations encompass:

  • Container security scanning and vulnerability management
  • TLS/SSL certificate management
  • Secrets and credentials management
  • Security monitoring and logging
  • Incident response procedures
  • Access control and authentication

Container Security

Security Scanning

TMI includes comprehensive container security scanning using Grype (Anchore).

Quick Security Scan

# Scan all containers for vulnerabilities
make containers-security-scan

# Generate security reports
make containers-security-report

# View security summary
cat security-reports/security-summary.md

Building Secure Containers

# Build individual containers (faster for iterative development)
make build-container-db      # PostgreSQL container only
make build-container-redis   # Redis container only
make build-container-tmi     # TMI server container only

# Build all containers
make build-containers

# Full security workflow with reports
make build-containers-all

Chainguard Base Images

TMI uses Chainguard container images for enhanced security:

Container Base Image Purpose
TMI Server cgr.dev/chainguard/static:latest Minimal runtime for static Go binary
PostgreSQL cgr.dev/chainguard/postgres:latest Secure PostgreSQL database
Redis cgr.dev/chainguard/redis:latest Secure Redis cache

Security improvements:

  • Chainguard images with significantly fewer CVEs than traditional bases
  • Static binaries built with CGO_ENABLED=0
  • Non-root user execution (nonroot:nonroot)
  • No shell, package manager, or unnecessary tools in runtime
  • Daily security updates from Chainguard
  • ~57MB total image size for TMI server

Note: Container builds exclude Oracle database support (requires CGO). For Oracle, build locally with go build -tags oracle.

Vulnerability Scanning

Grype Integration

# Scan specific image (table format)
grype tmi/tmi-postgresql:latest -o table

# Detailed scan with SARIF output
grype tmi/tmi-server:latest -o sarif > security.sarif

# JSON output for scripting
grype tmi/tmi-postgresql:latest -o json > vulnerabilities.json

CI/CD Security Scanning

Run automated security scans in CI/CD:

# Basic CI scan
./scripts/ci-security-scan.sh

# Custom thresholds
MAX_CRITICAL_CVES=0 MAX_HIGH_CVES=5 ./scripts/ci-security-scan.sh

# Scan specific images
IMAGES_TO_SCAN="tmi/tmi-server:latest redis:7" ./scripts/ci-security-scan.sh

Environment variables:

Variable Default Description
MAX_CRITICAL_CVES 0 Maximum critical CVEs allowed
MAX_HIGH_CVES 3 Maximum high CVEs allowed
MAX_MEDIUM_CVES 10 Maximum medium CVEs allowed
FAIL_ON_CRITICAL true Fail build on critical CVEs
FAIL_ON_HIGH false Fail build on high CVEs
ARTIFACT_DIR ./security-artifacts Report output directory

Security Reports

TMI generates multiple security report formats:

1. Summary Report (security-summary.md):

  • Vulnerability counts by severity
  • Pass/fail status by image
  • Remediation recommendations

2. Detailed Scan Results (security-scan-results.json):

  • Complete vulnerability details
  • CVSS scores and vectors
  • Affected packages and versions

3. SARIF Reports (security-results.sarif):

  • Standard security tool format
  • IDE and CI/CD integration
  • Machine-readable results

Security Thresholds

Configure acceptable vulnerability levels:

# Strict policy (production)
MAX_CRITICAL_CVES=0 MAX_HIGH_CVES=0 make containers-security-scan

# Lenient policy (development)
MAX_HIGH_CVES=10 FAIL_ON_HIGH=false make containers-security-scan

Default thresholds:

  • Critical CVEs: 0 (build fails)
  • High CVEs: 3 (warning)
  • Medium CVEs: 10 (informational)

Runtime Container Security

Container Hardening

Run containers with security restrictions:

# Docker run with security options
docker run -d \
  --name tmi-server \
  --read-only \
  --cap-drop ALL \
  --cap-add NET_BIND_SERVICE \
  --security-opt no-new-privileges \
  --user 1000:1000 \
  tmi/tmi-server:latest

Kubernetes Security Context

securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  readOnlyRootFilesystem: true
  allowPrivilegeEscalation: false
  capabilities:
    drop:
      - ALL
    add:
      - NET_BIND_SERVICE

Security Best Practices

Regular updates:

  • Weekly: Update base images and rebuild
  • Monthly: Review security trends
  • Quarterly: Audit security policies

Layered security:

  • Image security: Patched base images
  • Runtime security: Resource limits, monitoring
  • Network security: Segmentation, firewalls
  • Access control: Least-privilege principles

TLS/SSL Management

Certificate Management

Generating Self-Signed Certificates

For development and testing:

# Generate self-signed certificate
openssl req -x509 -newkey rsa:4096 -nodes \
  -keyout server.key -out server.crt -days 365 \
  -subj "/CN=tmi.example.com"

# Generate with SAN (Subject Alternative Names)
openssl req -x509 -newkey rsa:4096 -nodes \
  -keyout server.key -out server.crt -days 365 \
  -extensions v3_req \
  -config <(cat /etc/ssl/openssl.cnf <(printf "\n[v3_req]\nsubjectAltName=DNS:tmi.example.com,DNS:*.tmi.example.com"))

# Set permissions
chmod 600 server.key
chmod 644 server.crt

Using Let's Encrypt

For production deployments:

# Install certbot
sudo apt-get update
sudo apt-get install certbot

# Obtain certificate (standalone mode)
sudo certbot certonly --standalone -d tmi.example.com -d api.tmi.example.com

# Certificates installed at:
# /etc/letsencrypt/live/tmi.example.com/fullchain.pem
# /etc/letsencrypt/live/tmi.example.com/privkey.pem

# Auto-renewal (already configured via systemd/cron)
sudo certbot renew --dry-run

Certificate Installation

For TMI Server:

# Copy certificates to secure location
sudo mkdir -p /etc/tmi/certs
sudo cp server.crt /etc/tmi/certs/
sudo cp server.key /etc/tmi/certs/
sudo chown tmi:tmi /etc/tmi/certs/*
sudo chmod 600 /etc/tmi/certs/server.key
sudo chmod 644 /etc/tmi/certs/server.crt

Configure TMI:

# config-production.yml
server:
  tls_enabled: true
  tls_cert_file: "/etc/tmi/certs/server.crt"
  tls_key_file: "/etc/tmi/certs/server.key"
  tls_subject_name: "tmi.example.com"
  http_to_https_redirect: true

Or via environment:

TMI_SERVER_TLS_ENABLED=true
TMI_SERVER_TLS_CERT_FILE=/etc/tmi/certs/server.crt
TMI_SERVER_TLS_KEY_FILE=/etc/tmi/certs/server.key
TMI_SERVER_TLS_SUBJECT_NAME=tmi.example.com
TMI_SERVER_HTTP_TO_HTTPS_REDIRECT=true

Certificate Verification

# Verify certificate details
openssl x509 -in /etc/tmi/certs/server.crt -text -noout

# Check certificate expiration
openssl x509 -in /etc/tmi/certs/server.crt -noout -dates

# Test TLS connection
openssl s_client -connect tmi.example.com:443 -servername tmi.example.com

# Verify certificate chain
openssl s_client -connect tmi.example.com:443 -showcerts

# Check certificate expiration (days remaining)
echo $(( ($(date -d "$(openssl x509 -enddate -noout -in /etc/tmi/certs/server.crt | cut -d= -f2)" +%s) - $(date +%s)) / 86400 ))

Certificate Renewal

Automated Let's Encrypt Renewal

Let's Encrypt certificates automatically renew via systemd timer:

# Check renewal timer status
systemctl status certbot.timer

# Test renewal
sudo certbot renew --dry-run

# Force renewal
sudo certbot renew --force-renewal

# Post-renewal hook (restart TMI)
cat > /etc/letsencrypt/renewal-hooks/post/restart-tmi.sh <<'EOF'
#!/bin/bash
systemctl restart tmi
EOF
chmod +x /etc/letsencrypt/renewal-hooks/post/restart-tmi.sh

Manual Certificate Renewal

For self-signed or purchased certificates:

# Generate new certificate
openssl req -x509 -newkey rsa:4096 -nodes \
  -keyout /etc/tmi/certs/server.key.new \
  -out /etc/tmi/certs/server.crt.new \
  -days 365 \
  -subj "/CN=tmi.example.com"

# Backup old certificates
cp /etc/tmi/certs/server.key /etc/tmi/certs/server.key.old
cp /etc/tmi/certs/server.crt /etc/tmi/certs/server.crt.old

# Install new certificates
mv /etc/tmi/certs/server.key.new /etc/tmi/certs/server.key
mv /etc/tmi/certs/server.crt.new /etc/tmi/certs/server.crt

# Set permissions
chmod 600 /etc/tmi/certs/server.key
chmod 644 /etc/tmi/certs/server.crt

# Restart TMI server
systemctl restart tmi

TLS Configuration Best Practices

Strong cipher suites:

# Nginx configuration
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384';
ssl_prefer_server_ciphers on;

HSTS (HTTP Strict Transport Security):

add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

Certificate monitoring:

  • Monitor expiration dates (alert 30 days before expiry)
  • Track certificate renewals
  • Validate certificate chain integrity
  • Test TLS configuration regularly

Secrets Management

Environment Variables

Store secrets in environment variables -- never in code.

All TMI environment variables use the TMI_ prefix. OAuth provider credentials are configured in the YAML config file under auth.oauth.providers.<name>, not via individual environment variables.

Development (.env.dev):

TMI_JWT_SECRET=development-secret-do-not-use-in-production
TMI_REDIS_PASSWORD=dev_redis_pass
TMI_BUILD_MODE=dev
# PostgreSQL password is set via TMI_DATABASE_URL connection string
# OAuth provider secrets are configured in config-development.yml

Production (system environment):

# Set via systemd service
Environment=TMI_JWT_SECRET=<strong-production-secret>
Environment=TMI_DATABASE_URL=postgres://user:password@host:5432/tmi?sslmode=require
Environment=TMI_REDIS_PASSWORD=<redis-password>
Environment=TMI_BUILD_MODE=production
# OAuth provider secrets should be managed via secrets provider (see below)

Generating Strong Secrets

# Generate JWT secret (32 bytes)
openssl rand -base64 32

# Generate strong password
openssl rand -base64 24

# Generate UUID
uuidgen

# Generate hex secret
openssl rand -hex 32

Built-in Secrets Provider

TMI includes a unified secrets provider abstraction (TMI_SECRETS_PROVIDER) that supports multiple backends. The provider is configured via the secrets section in the config file or via environment variables.

Supported providers:

Provider TMI_SECRETS_PROVIDER value Status
Environment variables env Supported (default)
AWS Secrets Manager aws Supported
OCI Vault oci Supported
HashiCorp Vault vault Planned
Azure Key Vault azure Planned
GCP Secret Manager gcp Planned

Standard secret keys used by TMI:

  • jwt_secret - JWT signing secret
  • database_password - Database password
  • redis_password - Redis password
  • oauth_github_client_id, oauth_github_client_secret - GitHub OAuth
  • oauth_google_client_id, oauth_google_client_secret - Google OAuth
  • oauth_microsoft_client_id, oauth_microsoft_client_secret - Microsoft OAuth
  • settings_encryption_key - Settings encryption key

Environment Variables Provider (default)

When TMI_SECRETS_PROVIDER=env (or not set), secrets are read from environment variables. This is the simplest configuration for development.

AWS Secrets Manager Provider

# Configure TMI to use AWS Secrets Manager
TMI_SECRETS_PROVIDER=aws
TMI_AWS_REGION=us-east-1
TMI_AWS_SECRET_NAME=tmi/production

# Store secret in AWS Secrets Manager
aws secretsmanager create-secret \
  --name tmi/production \
  --secret-string '{"jwt_secret":"<secret>","database_password":"<password>"}'

OCI Vault Provider

# Configure TMI to use OCI Vault
TMI_SECRETS_PROVIDER=oci
TMI_OCI_COMPARTMENT_ID=ocid1.compartment.oc1..example
TMI_OCI_VAULT_ID=ocid1.vault.oc1..example
TMI_OCI_SECRET_NAME=tmi-production-secrets

Kubernetes Secrets

When deploying to Kubernetes, map secrets to TMI environment variables:

apiVersion: v1
kind: Secret
metadata:
  name: tmi-secrets
  namespace: tmi
type: Opaque
data:
  jwt-secret: <base64-encoded-secret>
  database-url: <base64-encoded-url>
  redis-password: <base64-encoded-password>
---
# Reference in deployment
env:
  - name: TMI_JWT_SECRET
    valueFrom:
      secretKeyRef:
        name: tmi-secrets
        key: jwt-secret
  - name: TMI_DATABASE_URL
    valueFrom:
      secretKeyRef:
        name: tmi-secrets
        key: database-url
  - name: TMI_REDIS_PASSWORD
    valueFrom:
      secretKeyRef:
        name: tmi-secrets
        key: redis-password

Secret Rotation

Rotate secrets regularly:

JWT Secret rotation:

# Generate new secret
NEW_SECRET=$(openssl rand -base64 32)

# Update in environment/secrets manager
heroku config:set TMI_JWT_SECRET=$NEW_SECRET --app tmi-production

# Or for systemd service
sudo systemctl edit tmi
# Add: Environment=TMI_JWT_SECRET=<new-secret>
sudo systemctl restart tmi

Database password rotation:

-- Update user password
ALTER USER tmi_user WITH PASSWORD 'new_secure_password';

-- Update TMI configuration
-- Restart TMI with new password

Security Best Practices

  • Never commit secrets to git
  • Use different secrets per environment
  • Rotate secrets regularly (quarterly minimum)
  • Use secrets management systems (Vault, Secrets Manager)
  • Limit secret access (principle of least privilege)
  • Audit secret access (who accessed what, when)
  • Encrypt secrets at rest

Security Monitoring

Authentication Monitoring

Monitor authentication events:

# View authentication logs
tail -f /var/log/tmi/tmi.log | grep -E "authentication|authorization"

# Count failed login attempts
grep "authentication failed" /var/log/tmi/tmi.log | wc -l

# Identify suspicious activity (multiple failures from same IP)
grep "authentication failed" /var/log/tmi/tmi.log | \
  grep -oP 'ip=\K[0-9.]+' | sort | uniq -c | sort -rn

# Last 10 successful logins
grep "authentication successful" /var/log/tmi/tmi.log | tail -10

Security Event Logging

TMI logs security-relevant events:

TMI uses Go's log/slog package with JSON output in production. Log entries follow the slog structured format:

{
  "time": "2025-11-12T10:30:00Z",
  "level": "WARN",
  "msg": "authentication_failed",
  "user_email": "user@example.com",
  "provider": "google",
  "ip_address": "192.168.1.100",
  "reason": "invalid_token"
}

Key security events:

  • Authentication failures
  • Authorization denials
  • Token validation failures
  • Suspicious API usage patterns
  • Rate limit violations
  • Database access errors

Security Alerts

Configure alerts for security events:

# Alert on multiple failed logins
*/5 * * * * [ $(grep "authentication failed" /var/log/tmi/tmi.log | tail -100 | wc -l) -gt 10 ] && \
  echo "Multiple authentication failures detected" | mail -s "Security Alert" security@example.com

Prometheus alert example:

- alert: AuthenticationFailureSpike
  expr: rate(authentication_failures_total[5m]) > 10
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "High authentication failure rate"
    description: "More than 10 auth failures per second"

Access Control Monitoring

Track who accesses what:

# API access by user
grep "api_request" /var/log/tmi/tmi.log | \
  jq -r '.user_email' | sort | uniq -c | sort -rn

# Sensitive operations (delete, update permissions)
grep -E "DELETE|permission.*update" /var/log/tmi/tmi.log

# Failed authorization attempts
grep "authorization denied" /var/log/tmi/tmi.log

Security Auditing

Regular Security Audits

Weekly:

  • Review authentication logs
  • Check failed login attempts
  • Monitor container vulnerability reports
  • Review access patterns

Monthly:

  • Full security scan (containers, dependencies)
  • Certificate expiration check
  • Access control review
  • Security event trends analysis

Quarterly:

  • Security policy review
  • Penetration testing
  • Disaster recovery testing
  • Security training updates

Compliance and Logging

Maintain audit logs for compliance:

# Enable comprehensive logging
TMI_LOG_API_REQUESTS=true
TMI_LOG_API_RESPONSES=false  # May expose sensitive data
TMI_LOG_REDACT_AUTH_TOKENS=true  # Always redact

# Archive logs for retention
tar -czf /archive/tmi-logs-$(date +%Y%m).tar.gz /var/log/tmi/

Retention requirements:

  • Authentication logs: 90 days minimum
  • Access logs: 90 days minimum
  • Security events: 1 year minimum
  • Audit trails: Per compliance requirements

Related Documentation

Additional Resources

Home

Releases

Getting Started

Deployment

Operation

Troubleshooting

Development

Integrations

Tools

API Reference

Reference

Clone this wiki locally