01 / Experiment
When a system fails
A simulated API with three replicas, Redis and PostgreSQL. Inject a failure, watch it propagate, then mitigate until the SLO holds again.
Simulation
Values are generated locally to demonstrate the failure path — not production telemetry.
Scenario
API · 3 replicas · Redis · PostgreSQL · ingress
Engine
Deterministic 10ms tick · seeded
Input
Failure injection · mitigations
Output
SLO · spans · recovery path
Request path
ingress
Passing- admitted
- 0 rps
- shed
- 0 rps
api-1
Ready- in flight
- 0/4
- queued
- 0
- restarts
- 0
api-2
Ready- in flight
- 0/4
- queued
- 0
- restarts
- 0
api-3
Ready- in flight
- 0/4
- queued
- 0
- restarts
- 0
redis
Serving- hit rate
- 0%
- span
- 0ms
postgres
Serving- span
- 0ms
- pool
- 0/24
- pool wait
- 0ms
Request time by layer
- API
- 0%
- Redis
- 0%
- Postgres
- 0%
- Queue
- 0%
Telemetry · 5s window
Throughput
0
requests / sec
Error rate
0.0%
SLO 1%
p95
0ms
SLO 400ms
p50
0ms
p99
0ms
In rotation
3/3
0 restarts
In flight
0
Queued
0
Availability
100.00%
since reset
p95 latency · 60s
0ms
Error rate · 60s
0.0%
Observed spans
API
0ms
Redis
0ms
Postgres
0ms
Pool wait
0ms
Inject failure
Mitigate
Mitigation buys headroom; it does not remove the cause. Clearing the fault is what ends the incident — and a mitigation left in place afterwards becomes the next one.
Incident · none
01
Detect
SLO holding
02
Isolate
Attribute the failures
03
Mitigate
Buy time while the cause is fixed
04
Recover
SLO must hold for 5s
Event log
02 / Experiment
When infrastructure drifts
A simulated Terraform loop. Someone changed production by hand — read the plan, then reconcile actual back to the file.
Simulated state
Desired and actual are generated locally to demonstrate reconciliation — not a live cluster.
Status
Converged
Engine
Desired vs actual · plan then apply
Input
Manual change to simulated production
Output
Plan · reconcile · event log
Desired — the file
resource "kubernetes_deployment" "app" {
replicas = 3
cpu = "500m"
memory = "1Gi"
image = "app:1.4.2"
container_port = 8080
}Actual — simulated production
resource "kubernetes_deployment" "app" {
replicas = 3
cpu = "500m"
memory = "1Gi"
image = "app:1.4.2"
container_port = 8080
}Introduce drift
Someone changed production by hand. Desired stays in the file; actual is whatever the console left behind.
Plan
No changes. 3 replicas, 500m / 1Gi, app:1.4.2.
Event log
03 / Experiment
From requirements to architecture
Select the workload. The graph, the checks and the scores are the output of a rules engine — not a drawing, and not a live account.
Rule-based model
Architecture and checks are composed from the selected requirements — not a live account.
Status
Composed in the browser
Engine
Deterministic rules
Input
Workload requirements
Output
Graph · checks · scores
Requirements
Availability zones
Database
Compute
Architecture
Edge
CloudFront
PublicTLS at the edge, cached static
Network
Application Load Balancer
Spans 3 AZs
Compute
ECS / Fargate
Autoscaling across 3 AZs
Data
Amazon RDS (PostgreSQL)
Private subnets, Multi-AZ
Amazon S3
Origin for the CDN, bucket policy locked down
Complexity
Medium
Availability
High
Security
8/10
Architecture checksnone failing · 2 warnings
- ✓
Database is private
RDS sits in a private subnet, reached only from the app
- ✓
Application spans 3 AZs
Three AZs is the bar for a regional outage not taking the service with it
- ✓
TLS terminates at the edge
Certificates live on CloudFront; origin stays private
- ✓
Storage is not publicly accessible
S3 is an origin, not a public website
- ⚠
Backup policy specified
No backup policy — a dropped table is an outage with no undo
- ⚠
WAF configured
A public origin without a WAF is an open invitation
- ✓
Capacity follows demand
Autoscaling is in the architecture, not a runbook
Optional PNG
The reasoning above is the experiment. The PNG is a sketch of the same graph, rendered server-side if Python is available. Download keeps the full-size file.