Skip to content

Production Readiness Review (PRR) & Checklist

SRE

A Production Readiness Review (PRR) ensures that every service, application, or infrastructure component deployed into production meets stringent standards for reliability, observability, security, scalability, and operability.

This guide provides both a review template structure and an exhaustive task checklist across 8 operational pillars.

1. Review Template & Metadata

Use the following template structure when reviewing and approving new services or major architecture changes:

Service Metadata

Metadata Field Value
Service Name e.g., user-auth-service / payment-gateway-api
Service Owner / Team e.g., Core Services Team / Platform Engineering
Primary On-Call [Team PagerDuty / Opsgenie Schedule Link]
Alerting Channel #alerts-prod, #dev-notifications
Repository [GitHub / GitLab Repo URL]
Target Environment AWS EKS (prod-eu-west-1), CloudFront, RDS Aurora
Target Go-Live Date YYYY-MM-DD
PRR Reviewers @TechLead, @SecurityLead, @SRELead
Overall Status 🟡 IN PROGRESS / 🟢 READY FOR PROD / 🔴 BLOCKED

Executive Summary & Service Scope

Brief summary of what this service does, critical dependencies, user impact, and key launch milestones.

Production Readiness Scorecard

Category Status Reviewer Notes / Blockers
1. Architecture & Capacity 🟢 PASS / 🟡 RISK / 🔴 FAIL
2. Security & Compliance 🟢 PASS / 🟡 RISK / 🔴 FAIL
3. Observability & Telemetry 🟢 PASS / 🟡 RISK / 🔴 FAIL
4. Network & Traffic Management 🟢 PASS / 🟡 RISK / 🔴 FAIL
5. Reliability & HA 🟢 PASS / 🟡 RISK / 🔴 FAIL
6. Deployment & CI/CD 🟢 PASS / 🟡 RISK / 🔴 FAIL
7. Performance & Load Testing 🟢 PASS / 🟡 RISK / 🔴 FAIL
8. Operations & Runbooks 🟢 PASS / 🟡 RISK / 🔴 FAIL

Sign-off & Approvals

  • [ ] Engineering Lead: [Name] — Approved on [Date]
  • [ ] SRE / Platform Lead: [Name] — Approved on [Date]
  • [ ] Security Lead: [Name] — Approved on [Date]
  • [ ] Product Owner: [Name] — Approved on [Date]

2. Production Readiness Architecture

graph TD
    A[1. Architecture & Scalability] --> PRR[Production Readiness Review]
    B[2. Security & Identity] --> PRR
    C[3. Observability & SRE] --> PRR
    D[4. Network & Traffic Management] --> PRR
    E[5. Disaster Recovery & Data] --> PRR
    F[6. CI/CD & Deployments] --> PRR
    G[7. Performance & Chaos] --> PRR
    H[8. Operations & Runbooks] --> PRR
    PRR --> GoLive[🟢 Go-Live Approval & Canary Rollout]

3. Comprehensive PRR Task Checklist

Pillar 1: Architecture, High Availability & Scalability

  • [ ] Multi-AZ Distribution: Workloads distributed across at least 3 Availability Zones using podAntiAffinity or topologySpreadConstraints.
  • [ ] Autoscaling Configured:
  • Horizontal Pod Autoscaler (HPA) configured based on CPU/Memory or custom Prometheus metrics (e.g., active request connections, queue depth).
  • Node Autoscaling (Karpenter / Cluster Autoscaler) configured with suitable instance types and rapid provisioning thresholds.
  • [ ] Resource Requests & Limits: CPU and memory requests and limits tuned to prevent OOMKills (ExitCode 137) and noisy-neighbor resource starvation.
  • [ ] Pod Disruption Budgets (PDB): PodDisruptionBudget configured to ensure minimum availability during node draining and EKS upgrades.
  • [ ] Dependency Resilience: Upstream and downstream dependencies mapped with explicit timeouts, exponential backoff retries with jitter, and circuit breakers.
  • [ ] Ingress & API Rate Limiting: Token-bucket or sliding-window rate limiting enabled at ingress/gateway to protect against cascading failure.

Pillar 2: Security, Identity & Compliance

  • [ ] Secrets Management: Zero plaintext secrets in code, environment variables, or Git. Using External Secrets Operator (ESO) with AWS Secrets Manager or HashiCorp Vault.
  • [ ] IAM Least Privilege & Pod Identity: AWS Pod Identity or IAM Roles for Service Accounts (IRSA) used with tightly scoped IAM policies.
  • [ ] Container Hardening:
  • Containers run as non-root (runAsNonRoot: true, readOnlyRootFilesystem: true, drop capabilities like ALL).
  • Minimal, vulnerability-free base images (Distroless, Alpine, Chainguard).
  • [ ] Vulnerability Scanning: Automated CI/CD image scanning (Trivy, Snyk, AWS ECR) blocking builds with CRITICAL or HIGH CVEs.
  • [ ] Network Segmentation:
  • Kubernetes NetworkPolicy restricting ingress/egress to required namespaces and CIDRs.
  • AWS Security Groups and NACLs locked down; AWS WAF enabled on public ALB/CloudFront endpoints.
  • [ ] Data Privacy & GDPR: PII sanitization in application logs; data retention and deletion policies defined.

Pillar 3: Observability, Telemetry & SRE

  • [ ] Golden Signals (RED / USE):
  • Rate: Requests / second or transactions per second.
  • Errors: 4xx / 5xx HTTP response rates, connection drops.
  • Duration: Latency percentiles (p50, p95, p99).
  • Saturation: CPU, memory, thread pool, database connection pool utilization.
  • [ ] SLIs & SLOs Defined:
  • Availability SLO (e.g., 99.9% successful requests over 30-day window).
  • Latency SLO (e.g., 99% of requests < 150ms).
  • Error budget burn rate alerts configured.
  • [ ] Health & Readiness Probes:
  • livenessProbe: Detects deadlocks and restarts container.
  • readinessProbe: Validates local readiness before routing traffic (does not fail if non-critical external services are down).
  • startupProbe: Configured for slow-starting applications to prevent false-positive restarts.
  • [ ] Distributed Tracing: OpenTelemetry / AWS X-Ray instrumentation in place with trace_id and span_id injected into all logs and downstream HTTP/gRPC headers.
  • [ ] Structured Centralized Logging:
  • JSON-formatted logs with standardized fields (timestamp, level, service, trace_id, environment).
  • Sensitive parameters (tokens, passwords, PII, auth headers) redacted.
  • [ ] Alerting & Routing:
  • High-priority alerts routed to PagerDuty / On-call rotation.
  • Informational alerts routed to dedicated team Slack channels.
  • Every firing alert has an embedded link to a specific troubleshooting Runbook.
  • [ ] Production Dashboards: Real-time Grafana / CloudWatch / Datadog dashboards built and reviewed by the team.

Pillar 4: Ingress, Network & Traffic Management

  • [ ] CDN & Edge Caching:
  • CloudFront / CDN caching headers (Cache-Control, Surrogate-Control) properly tuned for static assets vs. dynamic APIs.
  • Origin shield and connection reuse configured to protect backend services from traffic spikes.
  • [ ] TLS Termination & Certificate Automation:
  • Automated certificate issuance and renewal via Cert-Manager and Let's Encrypt / AWS Certificate Manager (ACM).
  • Modern TLS protocol enforcement (TLS 1.2+ minimum, secure cipher suites).
  • [ ] Graceful Degradation & Fallbacks:
  • Fallback responses / cached responses available if non-critical third-party APIs fail.
  • Custom error pages and circuit breaking configured at the gateway layer.

Pillar 5: Reliability, Disaster Recovery & Data Management

  • [ ] RPO & RTO Targets: Recovery Point Objective (RPO) and Recovery Time Objective (RTO) documented and agreed upon with product stakeholders.
  • [ ] Database & Storage Backups:
  • Automated continuous backups / snapshots enabled (e.g., AWS RDS / Aurora Point-in-Time Recovery).
  • Cross-region backup replication if required for DR.
  • [ ] Restore Testing (Game Day): Backup restoration tested and timed in a staging environment within the last 90 days.
  • [ ] Database Schema Migrations:
  • Zero-downtime migration strategy (Expand-Contract / Blue-Green schema migrations).
  • Rollback script validated before applying migrations.
  • [ ] Connection Pooling: Database connection pooling (e.g., RDS Proxy, PgBouncer) configured with maximum connection caps.

Pillar 6: Deployment, Release & CI/CD

  • [ ] Zero-Downtime Draining:
  • Kubernetes terminationGracePeriodSeconds (e.g., 30s-60s) and preStop hook (sleep 10) configured to finish in-flight requests before SIGKILL.
  • [ ] Canary / Progressive Rollouts: Canary deployments configured (e.g., Argo Rollouts / Flagger / Gateway API traffic splits) with automated rollback on error spikes.
  • [ ] Version Pinning & Immutability:
  • No latest Docker tags; container images pinned to semantic versions or Git commit SHA digests.
  • Helm chart dependencies explicitly pinned.
  • [ ] GitOps / IaC Managed:
  • All infrastructure, Helm charts, and Kubernetes manifests managed in Git (Terraform, ArgoCD/Flux).
  • Drift detection enabled.

Pillar 7: Performance, Load & Chaos Testing

  • [ ] Load & Stress Testing:
  • Service tested at 2x - 3x expected peak traffic to evaluate latency degradation and resource headroom.
  • Breaking point / maximum capacity identified and documented.
  • [ ] Endurance / Soak Testing: Sustained load run for 12–24 hours to detect memory leaks, file descriptor exhaustion, or thread pool depletion.
  • [ ] Chaos & Fault Injection Testing:
  • Tested behavior when random pods are killed.
  • Tested behavior when external dependencies (DB, Redis, downstream APIs) return 500s or time out.
  • Single Availability Zone simulated failure test.

Pillar 8: Operations, Runbooks & Handover

  • [ ] Operational Runbooks Published:
  • Runbook 1: High CPU / Memory / OOMKill Remediation
  • Runbook 2: High 5xx Error Rate / API Timeout Triage
  • Runbook 3: Database Connection Spike & Slow Query Debugging
  • Runbook 4: Ingress / CDN Origin Failure & Upstream Timeout Recovery
  • Runbook 5: Manual Rollback & Emergency Maintenance Mode Procedure
  • [ ] Service Ownership & On-Call Rotation:
  • Primary and secondary on-call engineers assigned in PagerDuty / Opsgenie.
  • Escalation paths and management contacts documented.
  • [ ] Incident Response Readiness:
  • Incident response channels, Zoom/Teams war-room links, and Jira/Confluence Post-Mortem (RCA) template linked.
  • [ ] Post-Launch Hypercare Plan:
  • Dedicated engineering support schedule for the first 48-72 hours post go-live.
  • Clear rollback criteria and timeline established.

Confluence Best Practice: Status Rollup Macro

In Confluence, wrap the metadata table in a Page Properties macro. On your space's homepage, insert a Page Properties Report macro filtered by the label production-readiness to automatically aggregate the status of all services across your team!