OperoNMS platform infrastructure
+
−
Fit
100%
Pause animation
Theme
OperoNMS production infrastructure
Browsers reach Cloudflare, which proxies over HTTPS to an nginx edge container on one EC2 instance in a default VPC in af-south-1. The instance runs 25 ECS services on a Docker bridge network and talks to RDS PostgreSQL, SQS, SES, S3, SSM, KMS, ECR and CloudWatch.
AWS · production account · region af-south-1 (Cape Town)
Separate AWS account from staging. Everything below is created by CloudFormation stacks operonms-prod and operonms-prod-queues (budget: operonms-prod-budget in us-east-1), apart from the default VPC.
VPC · default VPC of af-south-1 · 172.31.0.0/16 (AWS default)
No private subnets · no NAT gateway · no ALB · no VPC endpoints · Amazon-provided DNS resolver for AWS and RDS names
PUBLIC SUBNET · af-south-1a (default subnet, /20, route 0.0.0.0/0 → IGW, auto-assign public IPv4)
Holds the EC2 instance and the RDS instance. Different security groups separate them.
PUBLIC SUBNET · af-south-1b
Empty. Member of RDS subnet group "operonms-prod" because RDS needs ≥2 AZs,
even single-AZ. Same route table, same default NACL.
Failover target if the stack is ever moved; nothing runs here today.
PUBLIC SUBNET · af-south-1c
Empty. Member of RDS subnet group "operonms-prod" because RDS needs ≥2 AZs,
even single-AZ. Same route table, same default NACL.
Failover target if the stack is ever moved; nothing runs here today.
CLOUDFLARE · DNS + reverse proxy + TLS
Name
Type → target
Proxy
*.operonms.co.za
A → origin Elastic IP (every
tenant address)
on
admin
A → origin Elastic IP
(OperoAdmin; beats the
wildcard)
on
operonms.co.za,
www
CNAME → Pages project
operonms-site
on
platform
CNAME → Pages (this page)
on
Zone settings
Free plan · nameservers
aitana
/
brad.ns.cloudflare.com
Always HTTPS · minimum TLS 1.2
SSL mode Full (strict): origin presents a Cloudflare Origin CA
cert for
operonms.co.za
+
*.operonms.co.za
Authenticated Origin Pulls: Cloudflare shows a client cert;
nginx rejects anything without it
Adds
X-Forwarded-For:
<client>
= proxy hop 1 of 2
Origin is off 20:00–07:45 weekdays and weekends → Cloudflare
521 page
The wildcard serves every tenant with no DNS change per tenant.
platform
is a reserved tenant label (RESERVED_LABELS), so a tenant
can never collide with this page.
DNS records
Forwarded-for hop count (why OPERO_FORWARDED_HOPS=2)
Hop 1: Cloudflare appends the real client address to
X-Forwarded-For.
Hop 2: edge nginx appends the Cloudflare address it saw. A
service therefore reads the client as the 2nd entry from the right; anything a client forges sits further left and is ignored
Identity and Platform use the same count (OPERO_PLATFORM_TRUSTED_PROXIES=2) for the staff IP allow-list. If /api were ever routed
through the web container, both become 3
Host
passes through unchanged and
X-Forwarded-Host
is set from it, never taken from the client: services bind a request to its
tenant from that name
CLIENTS
Tenant users · browser
https://<tenant>.operonms.co.za
web app + OperoAI panel (one label deep)
OperoSystems staff · browser
https://admin.operonms.co.za
OperoAdmin · Entra ID + MFA · IP allow-list
Capture tablets · Android app
tenant address, same /api paths
photos up to 25 MB (edge limit)
How clients reach it
Only by name: tenant and admin addresses resolve to
Cloudflare, never to the origin IP. The origin accepts
nothing that did not come through Cloudflare.
STATIC HOSTING (outside AWS)
Pages · operonms-site (marketing)
operonms.co.za
and
www,
built from the marketing repo and deployed with
wrangler (--branch=main)
dev
branch alias is the noindex preview
Serverless: no part of it runs in AWS
Pages · platform.operonms.co.za (this page)
The infrastructure diagram for onboarding engineers
Static HTML + inline SVG, no backend, no secrets
Generated by
platform-site/build.py
from
ops/aws/production;
rebuild
after any infrastructure change
Microsoft Entra ID + Graph
Staff sign-in to OperoAdmin (OIDC,
login.microsoftonline.com);
amr=mfa
required
Identity worker reads the directory via Graph to sync leavers
(GroupMember.Read.All)
Reached from the instance over HTTPS 443 via the Internet Gateway
Amazon Bedrock · eu-west-1 (Ireland)
OperoAI (assistant) Converse / ConverseStream and Execution's pest &
disease photo checks
Claude Sonnet via EU inference profiles only
(eu.anthropic.claude-sonnet-5-5 /
4-6),
IAM condition
aws:RequestedRegion
=
eu-*
Data stays in the EU; no
global.
profile is allowed
domains.co.za · operosystems.co.za
DNS for the company mail domain (not on Cloudflare)
SES Easy DKIM CNAMEs + MAIL FROM
ses.operosystems.co.za
(MX/SPF)
added by hand
Mailboxes are Microsoft 365; SES sends as
no-reply@operosystems.co.za
EC2 INSTANCE operonms-prod · t4g.medium (2 vCPU Graviton, 4 GiB) · Amazon Linux 2023 arm64
Single on-demand instance, af-south-1a · only inbound tcp/443 from Cloudflare · dashed outline = instance security group
edge · nginx
host :443 → container :8443 · 128 MiB
TLS · Cloudflare client-cert check · router
web (SPA, nginx)
host :8201 → :8080 · 64 MiB
static tenant app
admin (SPA, nginx)
host :8202 → :8080 · 64 MiB
static OperoAdmin console
Edge routing (edge/default.conf)
admin.
host:
/api/identity,
/api/platform
only; else admin SPA
<tenant>.
host (one label):
/api/<svc>/
→ that service; else web SPA
Unknown SNI (apex, www, 2 labels, IP) refused at the handshake
/api/*/internal/*
→ 404 (service-to-service only)
docker0 bridge · gateway 172.17.0.1 · service-to-service calls (green) are http://172.17.0.1:<hostPort> · no DNS, no service discovery, no per-task ENI
ECS cluster "operonms-prod" · 25 services, EC2 launch type, desired count 1 each, one container per task · columns = business service, rows = process type
API
uvicorn ×1, :8000
Worker
SQS consumer/publisher
Scheduler
daily jobs
identity
:8101 → :8000
512 MiB · db identity
identity-worker
mode: all
256 MiB · pool 1+1
platform
:8110 → :8000
256 MiB · db platform
platform-worker
mode: all
192 MiB · pool 1+1
masterdata
:8102 → :8000
256 MiB · db masterdata
masterdata-worker
mode: all
192 MiB · pool 1+1
planning
:8103 → :8000
256 MiB · db planning
planning-worker
mode: events
256 MiB · pool 1+1
planning-scheduler
cron-like jobs
192 MiB
space
:8104 → :8000
256 MiB · db space
space-worker
mode: all
192 MiB · pool 1+1
orders
:8105 → :8000
256 MiB · db orders
orders-worker
mode: all
256 MiB · pool 1+1
execution
:8106 → :8000
320 MiB · db execution
execution-worker
mode: events
256 MiB · pool 1+1
execution-scheduler
cron-like jobs
192 MiB
ingest
:8108 → :8000
256 MiB · no database
insights
:8109 → :8000
320 MiB · db insights
insights-worker
mode: facts
256 MiB · pool 1+1
insights-scheduler
cron-like jobs
384 MiB
assistant
:8111 → :8000
320 MiB · db assistant
assistant-worker
mode: all
192 MiB · pool 1+1
event lane
Task definition (one per service)
family
operonms-prod-<service>
·
networkMode:
bridge
·
requiresCompatibilities:
EC2
· ARM64/Linux
one essential container,
initProcessEnabled
(init as PID 1 reaps probe
zombies)
image
<account>.dkr.ecr.af-south-1.amazonaws.com/operonms/<svc>:<12-char
git
tag>
(immutable)
memory
= hard limit;
memoryReservation
= 45% of it
portMappings:
fixed hostPort per service (8101…8111, 8201/8202; edge 443 +
127.0.0.1:8080 for on-box smoke)
executionRoleArn
operonms-prod-ecs-execution ·
taskRoleArn
operonms-prod-ecs-task
logs:
awslogs
→
/operonms/prod,
prefix
ecs,
non-blocking 4 MiB buffer
secrets:
valueFrom
SSM
/operonms/prod/*
(whole parameters); DB URLs
assembled by a shell entry point at start, private vars then unset
health: API → python probe
GET
127.0.0.1:8000/health/live
every 30 s (3
retries, 60 s grace); workers → no-op
CMD
true
every 300 s; edge →
wget
/healthz
on 8080
ECS service settings
launchType
EC2
·
REPLICA
· desired count 1 · no load balancer · no
service discovery · no autoscaling
minimumHealthyPercent
0,
maximumPercent
100:
fixed
host
ports
allow
only
one
copy,
so
a
release
stops
the
old
task,
then
starts
the
new
(seconds
of
downtime
per
service)
Cluster Container Insights off (cost); capacity = this one instance
One-off tasks:
operonms-prod-migrate-<svc>
runs
alembic
upgrade
head
(Identity, Platform, assistant as their owner role)
Why not
awsvpc:
a t4g.medium has 3 ENIs, there are 25 tasks
Why not Fargate / ALB: no NAT (~US$40/mo) and no ALB cost; ECS on EC2
adds no charge
ECS agent:
ecs-init
from AL2023 (deploy.py ecs-bootstrap), registers
the instance via the instance role
Memory & CPU budget
Container hard limits total 6,080 MiB on a 4 GiB instance: deliberately
oversubscribed; each limit only caps one runaway container
Reservations (45%) total 2,732 MiB: what ECS will place; leaves room for one-off
migrate tasks
Expected real use at one tenant: 2.0–2.2 GiB in containers, 65–70% of the instance
(estimate; measure with
docker
stats)
2 GiB swapfile,
vm.swappiness=10:
a safety margin, not capacity
CPU credits unlimited (since 2026-11-12): standard credits ran dry, throttled the
box, and every API failed its health check in a loop
Alarm at memory >90% for 10 min → resize to t4g.large
Argon2id sign-in hashes: 64 MiB each on Identity (512 MiB limit)
Docker engine · live-restore · log driver local 20m×3
ECS agent (ecs-init)
SSM agent (no SSH, no key pair)
CloudWatch agent: mem + disk /60 s
IMDSv2 required · hop limit 2
EBS 40 GiB gp3, encrypted
RDS FOR POSTGRESQL · operonms-prod
Security group operonms-prod-db: inbound tcp/5432 from the instance SG only · outbound: none (127.0.0.1/32 placeholder)
Instance & storage
Engine PostgreSQL 16 (16.15) · class db.t4g.micro (2 vCPU, 1 GiB, ~80 max
connections)
Single-AZ af-south-1a, same AZ as the instance (no cross-AZ traffic charge);
MultiAZ:
false
gp3 20 GiB, storage autoscaling to 100 GiB; encrypted with CMK
alias/operonms-prod
PubliclyAccessible:
false:
no
public
IP,
only
reachable
from
the
instance
SG
Master
opero_admin,
password RDS-managed in Secrets Manager (nobody types
it); instance role can read it only while
DatabaseBootstrapAccess=enabled
Subnet group
operonms-prod
spans the 3 default subnets (RDS needs ≥2 AZs)
Parameters, backup & operations
Parameter group
postgres16:
rds.force_ssl=1
(TLS only),
log_min_duration_statement=1000
ms;
postgresql
log exported to CloudWatch
Automated backups 14 days (PITR) · window 19:00 SAST · maintenance Wed 18:00 SAST ·
AutoMinorVersionUpgrade
DeletionProtection,
DeletionPolicy:
Snapshot
(final snapshot on delete)
Weekday schedule: started 07:30, stopped 20:10 (EventBridge Scheduler); a stopped
DB takes no backups
Manual snapshot before every release:
operonms-prod-predeploy-<tag>-<time>
Connection budget: worst case 45 of ~80 (9 database-backed APIs × (pool 2 +
overflow 1) + 9 workers × (1+1)); steady state ≈ 20; alarm at >70 → db.t4g.small
Databases & roles (db-init)
9 databases on one instance:
operonms_
identity, platform, masterdata,
planning, space, orders, execution, insights, assistant
Identity, Platform and assistant:
*_owner
owns the schema and runs
migrations; the service runs as
*_app,
which owns nothing
Other services:
operonms_<svc>_app
owns its DB; isolation by FORCE ROW
LEVEL SECURITY (ADR 0049)
No SUPERUSER / BYPASSRLS / shared role;
PUBLIC
cannot connect, so one
service cannot read another's database
Ingest has no database
HOW CODE GETS TO PRODUCTION · build off-box, deploy by SSM, never SSH
Every production change is a person's action. `deploy.py` refuses any account but production, and the CloudFormation change set is read and executed by a human.
1 Engineer
branch in a git worktree → green gate
on a test DB → one session merges to
main
2 CI · Jenkins
ADR 0117: tests, SonarQube, OWASP
dep-check, builds; runs on the dev
box, ECR push role in prod
3 buildx · arm64
deploy.py
images
builds linux/arm64
for 13 repos, tag = 12-char git SHA;
push to ECR
4 ECR
immutable tags, scan on push, keep
last 10
5 Stack change
deploy.py
stack
→ CloudFormation
change set; human reads & executes
(IaC apply is never automatic)
6 SSM Run Command
deploy.py
deploy
→ on the instance:
fetch bundle from S3 ops bucket,
render env, RDS snapshot, migrate
7 ECS rollout
ecs-deploy:
register task defs,
update each service (stop → start),
wait stable
8 Smoke
on-box
127.0.0.1:8080/healthz
+
public smoke through Cloudflare
Amazon SQS · event bus between services
27 queues + 27 dead-letter queues
operonms-prod-<service>-<topic>[-dlq]
(stack
operonms-prod-queues,
generated by
render_queues.py
from
ops/scripts/create-event-queues.sh,
so prod cannot drift from
LocalStack/staging)
Visibility 60 s · long-poll 20 s · retention 4 d (DLQ 14 d) ·
maxReceiveCount
5
· SSE-SQS
Any message in a DLQ → CloudWatch alarm → SNS email
Routing maps
OPERO_<SVC>_QUEUE_URLS
rendered into each service's env; auth is IAM (task role), no public
policy
By name prefix
identity: operations, platform-actions, support-requests · masterdata: platform-control
planning: availability, order-cover, order-impact, outbound, platform-control
space: execution, location, planning, platform-control
orders: availability, dispatch, outbound, planning, platform-control, qc, trays
execution: inbound, outbound, platform-control · insights: facts, platform-control
platform: health · assistant: platform-control
+
operonms-prod-identity-mail-events
(+DLQ), in stack
operonms-prod
Amazon SES + SNS · email and bounce handling
Identity
operonms-prod
config set; sender identity operosystems.co.za (af-south-1),
no-reply@operosystems.co.za;
Easy DKIM RSA-2048; MAIL FROM
ses.operosystems.co.za
Production access granted 2026-11-04 (50,000/day, 14/s). IAM allows
SendEmail
only from that address
through that identity + config set
Suppression list: bounce + complaint. Events (bounce, complaint, reject, deliveryDelay) → SNS
operonms-prod-mail-events
→ email to ops and SQS
identity-mail-events
→ Identity worker stops mail to
that address (ADR 0116)
SNS
operonms-prod-alarms:
target of every CloudWatch alarm and budget warning; email subscription to the
ops mailbox
Amazon S3 · 3 buckets (KMS-encrypted, TLS-only, public access blocked)
operonms-prod-ops-<acct>:
exports & tenant exports (30-day expiry), deploy bundles (180 d),
breached-password list
operonms-prod-audit-<acct>:
Object Lock (COMPLIANCE, 7 yr) + versioning; platform audit chain; role can
write and lock, never delete
operonms-prod-documents-<acct>:
uploads and photos, versioned;
scan-inbox/
expires in 1 d;
assistant-imports/
7 d
All
DeletionPolicy:
Retain,
SSE-KMS
with
bucket
keys,
ownership
enforced
KMS · SSM Parameter Store · Secrets Manager
KMS one CMK
alias/operonms-prod
(rotation on): SSM SecureStrings, S3, exports, RDS, EBS excluded (uses
default)
SSM SecureString under
/operonms/prod/…:
ten DB passwords, RS256 key pair, MFA key, Entra secret, origin
cert/key/pull-CA. Free tier, vs Secrets Manager at US$0.40/secret
Delivered two ways: ECS agent injects them into containers (execution role);
host/render_env.py
renders
0600 env files for compose-era paths
Secrets Manager: only the RDS master secret
Amazon ECR · 13 repositories
operonms/<identity|masterdata|planning|space|orders|execution|ingest|insights|platform|assistant|web|admin|edge>
(+ legacy
finance
until ADR 0115 clean-up)
Tags immutable (12-char git SHA), scan on push, keep last 10 → rollback = start the previous tag
linux/arm64 images built off-box with buildx; the instance only pulls (execution role +
ecr:GetAuthorizationToken)
Amazon CloudWatch · logs, metrics, alarms
Logs: group
/operonms/prod,
30-day retention, stream per container (ecs/<name>/<task id>); RDS
postgresql
log group
Metrics: CWAgent
mem_used_percent,
disk_used_percent;
10 OperoAI metric filters → namespace
OperoNMS/OperoAI
44 alarms → SNS: EC2 system check (auto-recover) & instance check (reboot) · CPU >80% · CPU credits <50 ·
memory >90% · disk >80% · RDS CPU, storage <2 GiB, memory, connections >70 · 28 dead-letter queues ·
OperoAI error-rate, p95 TTFT, Bedrock throttles, daily cost, tenant-at-limit · staff-sync-failed
Missing data is not breaching in the schedule's off hours, so nights are quiet
AWS Budgets (us-east-1): US$80/month (80% actual, 100% forecast, 100% actual) + Bedrock US$30/month
EventBridge Scheduler · GuardDuty Malware Protection
Scheduler (no Lambda), TZ Africa/Johannesburg, Mon–Fri:
db-start
07:30 ·
instance-start
07:45 ·
instance-stop
20:00 ·
db-stop
20:10; role
operonms-prod-scheduler
GuardDuty plan on
documents/scan-inbox/:
file tagged
NO_THREATS_FOUND
or refused (ADR 0118); ClamAV
compose profile stays off (needs 2 GiB)
Security group · operonms-prod-instance
Security group · operonms-prod-db
Dir
Proto / port
Source / dest
Why
IN
tcp 443 ×15
Cloudflare IPv4 ranges (below)
the only way in
IN
—
no SSH, no 22, no IPv6, no ICMP
admin via SSM Session Manager
OUT
all
0.0.0.0/0 (CloudFormation default; no
explicit rule)
ECR, SQS, SES, SSM, S3, Logs,
Entra, Graph, Bedrock are public
endpoints; no NAT / VPC
endpoints
Cloudflare ranges: 173.245.48.0/20 · 103.21.244.0/22 · 103.22.200.0/22 · 103.31.4.0/22 ·
141.101.64.0/18 · 108.162.192.0/18 · 190.93.240.0/20 · 188.114.96.0/20 · 197.234.240.0/22 ·
198.41.128.0/17 · 162.158.0.0/15 · 104.16.0.0/13 · 104.24.0.0/14 · 172.64.0.0/13 · 131.0.72.0/22
The list is compared with cloudflare.com/ips-v4 by
deploy.py
cloudflare-ips;
update
both
together.
These
ranges
are
shared
by
all
Cloudflare
customers,
so
nginx
also
demands
Cloudflare's
client
certificate
(Authenticated
Origin
Pulls)
to
prove
the
request
came
through
our
zone.
Security groups are stateful: replies to allowed inbound flows need no rule. Containers share this SG
(bridge network); there is no per-service filtering.
Dir
Proto / port
Source / dest
Why
IN
tcp 5432
SG operonms-prod-instance (by SG id, not
CIDR)
only the instance reaches
Postgres
OUT
all
127.0.0.1/32 (default allow-all removed)
the database initiates
nothing
Referencing the SG instead of an IP keeps the rule valid if the instance is replaced.
rds.force_ssl=1
adds TLS on top: a plaintext client is refused by Postgres even from the allowed SG.
Planned tightening (not applied)
Replace open egress with 443 to the AWS prefix lists (ECR, S3, SQS, SES, SSM, Logs in af-south-1;
Bedrock in eu-west-1), 443 to Entra/Graph, 5432 to the DB SG; run VPC flow logs (~US$0.50/GB) for a
week before removing the old rule
Hop limit 1 once compose rollback is no longer wanted (containers can still reach the instance role
today)
A task role per service instead of one shared role
Routing, subnets & NACL · how they are configured
Route table (the VPC's main table, shared by all 3 subnets)
172.31.0.0/16
→ local (instance ↔ RDS stays inside the VPC)
0.0.0.0/0
→ Internet Gateway (inbound from Cloudflare, and every outbound call)
No NAT gateway route: a NAT gateway is ~US$40/month in af-south-1, half the budget
No VPC endpoints for SQS/SES/S3/SSM/ECR/Logs: they would cost more than the instance. Traffic to
those goes out through the IGW to public endpoints, TLS + IAM-signed
Subnets (the region's 3 default subnets, one per AZ, /20)
af-south-1a public: the instance + RDS · auto-assign public IPv4 · Elastic IP on the instance
af-south-1b, 1c public: empty; only there because an RDS subnet group needs ≥2 AZs
A subnet is "public" because its route table points at an IGW. What actually exposes things is
the SG: the instance admits Cloudflare only; RDS has no public IP
NACL: default (allow all in and out). Security groups are the control.
The VPC is AWS's default VPC (172.31.0.0/16 by default), passed to CloudFormation as parameters,
not created or managed by the stack. Confirm with the commands in "Verify it live".
IAM · roles and what they may do
operonms-prod-instance (EC2 profile):
AmazonSSMManagedInstanceCore,
AmazonEC2ContainerServiceforEC2Role,
+ runtime policy: logs,
PutMetricData
(CWAgent only),
SQS on
operonms-prod-*,
SES from one address, SSM read
/operonms/prod/*,
KMS key, S3 (audit
write-once), Bedrock EU, ECR pull
operonms-prod-ecs-execution (agent, before start): ECR pull, log write,
ssm:GetParameters,
kms:Decrypt
operonms-prod-ecs-task (all 25 tasks share it): SQS, SES, KMS, S3, Bedrock Converse (EU profiles
only)
operonms-prod-scheduler: Start/Stop that one instance and that one DB
operonms-prod-malware-scan: GuardDuty plan,
scan-inbox/
only
Trust policies carry
aws:SourceAccount.
Container credentials come from the task role endpoint
(ECS), falling back to IMDSv2
Operators: from the staging account, with MFA, assume
OperoReadOnly
or
OperoAdmin
(12 h). CI
has
operonms-ci-ecr-push
(ECR push only)
deploy.py
refuses any account but the production one
IGW
Internet Gateway
inbound: Cloudflare → Elastic IP · outbound: AWS APIs, Entra, Bedrock
Elastic IP (retained)
SG operonms-prod-instance · in: tcp/443 Cloudflare
Request lifecycle · browser → OperoNMS (badges 1–8)
1 Browser → Cloudflare
acme.operonms.co.za
matches the proxied wildcard A record; TLS 1.2+ ends at Cloudflare; HTTP is redirected to HTTPS
2 Cloudflare → origin
Cloudflare applies its rules, appends the client IP to
X-Forwarded-For
(hop 1) and opens a new TLS connection to the Elastic
IP :443, validating our Origin CA cert and presenting its client certificate
3 Internet Gateway + security group
Only Cloudflare's 15 IPv4 ranges may reach tcp/443; anything else is dropped before nginx sees it
4 Edge nginx (host :443 → :8443)
Verifies Cloudflare's client cert, picks the server block from SNI/Host, refuses unknown names. Static paths go to web (:8201)
or admin (:8202)
5 /api/<service>/ → service
Prefix stripped, request proxied to
172.17.0.1:<port>;
nginx appends Cloudflare's address (hop 2). Services take the client as
2nd from the right (OPERO_FORWARDED_HOPS=2);
Host
is untouched so the tenant is read from it (ADR 0100)
6 Service
uvicorn (1 worker) checks the RS256 bearer token against the tenant of the Host (401 wrong_tenant_address otherwise); calls
peers over the bridge when needed
7 Database
TCP 5432 over TLS as that service's own role; tenant data isolation by row-level security
8 Events, mail, files
State changes publish to SQS; workers long-poll, update their own DB, send mail via SES, store files in S3 (scanned by
GuardDuty). The response returns the same path; every container logs to CloudWatch
Other environments & who uses what
Production: this diagram. Account separate from staging; weekdays 07:45–20:00 SAST
Staging: separate AWS account, us-west-2, ECS Express Mode (Fargate behind AWS-managed load balancers, one
*.on.aws
host per
service); own SES identity so test mail never touches the company domain; scaled down when unused. Different shape from prod
on purpose
Dev box (an EC2 instance): Docker Compose dev stack (Postgres, LocalStack, Mailpit), Vite dev server, Caddy with sslip.io
hostnames for the SA team, isolated Playwright e2e stacks, and CI: Jenkins + SonarQube + Grafana + Prometheus (ADR 0117)
LocalStack stands in for SQS/SES/S3 in dev; the same
create-event-queues.sh
feeds LocalStack, staging and
queues.yaml
Marketing & this page: Cloudflare Pages, no AWS
Android tablet: APK downloaded from OperoAdmin / NMS Capture Tablets; checks for a minimum version (ADR 0137)
Naming
AWS names
operonms-prod-*;
SSM
/operonms/prod/*;
log group
/operonms/prod;
ECR
operonms/*;
cluster
operonms-prod
Tags:
application=operonms,
environment=production
Known gaps · read before you change anything
One instance, one AZ, one DB (single-AZ RDS): no standby. EC2 auto-recovers; RDS has PITR 14 d. An AZ outage is an outage
Off at night and weekends on purpose (cost: ~US$40/month vs ~69 always-on). Cloudflare shows 521 outside hours
Deploys are stop-then-start per service (fixed host ports): seconds of 502 for that service
No ALB, no NAT, no private subnets, no VPC endpoints. Egress to anywhere on 443 is allowed; allow-list proposal is written,
not applied
IMDS hop limit 2: a container can reach the instance role. One shared task role for 25 services (Bedrock grant is shared).
Both to be tightened
Oversubscribed memory (limits > RAM) and 2 GiB swap: watch the memory alarm, resize to t4g.large if it fires
Cloudflare IP list is static in the SG: re-check with
deploy.py
cloudflare-ips
Entra needs
amr
in the app's ID-token optional claims (set through the manifest); Graph
GroupMember.Read.All
consent was
still owed to the staff sync (check the staff-sync alarm)
Not measured: memory and connection numbers are estimates until a week of
docker
stats
data exists
Daily rhythm · Africa/Johannesburg (SAST), Monday–Friday
07:30
db-start
(Scheduler) → 07:45
instance-start
→ ECS agent restarts every service (desired count 1) → 07:55 tray and QC morning lists (Execution)
19:00 RDS backup window (17:00 UTC) · Insights nightly rebuild · order-cover run (Planning) → 19:15 surplus upkeep → 19:30 OperoAI purge
20:00
instance-stop
(clean shutdown) → 20:10
db-stop.
Weekends: off. Storage and the Elastic IP bill while stopped
Wednesday 18:00 RDS maintenance window
Out-of-hours work: start the DB, then the instance, by hand, and re-enable any schedule you disabled (a disabled stop schedule runs and bills 24 h)
Verify it live · read-only commands (profile operonms-prod-ro, region af-south-1)
Network:
aws
ec2
describe-vpcs
--filters
Name=is-default,Values=true
·
describe-subnets
·
describe-route-tables
·
describe-security-groups
--group-names
operonms-prod-instance
operonms-prod-db
Compute:
aws
ecs
list-services
--cluster
operonms-prod
·
describe-services
·
aws
ec2
describe-instances
--filters
Name=tag:Name,Values=operonms-prod
Data:
aws
rds
describe-db-instances
--db-instance-identifier
operonms-prod
·
aws
sqs
list-queues
--queue-name-prefix
operonms-prod-
State of the stack:
aws
cloudformation
describe-stacks
--stack-name
operonms-prod
· drift detection from the console
On the box (SSM, via
deploy.py
remote
status):
docker
stats,
docker
ps,
ECS
agent
introspection
on
:51678
Rollback and recovery
Roll back a release: start the previous immutable tag (deploy.py rollback --to <tag> for the compose era; on ECS re-run
ecs-deploy
with the previous tag). Database: restore the pre-deploy snapshot or point-in-time to a new instance. Migrations that delete data end rollback (see ADR 0115 notes).
Instance failure: EC2 recovers onto new hardware (system check alarm); containers return on boot via ECS; instance-check failure reboots. Region/AZ loss: no standby; rebuild from the stacks + snapshot.
Deploy needs
OPERO_ASSISTANT_DEPLOY=on
for images and deploy, or the assistant services are left out of the release.
Admin access has no SSH: SSM Session Manager or Run Command, authenticated by IAM. The dev box has no Session Manager plugin, so use scripts the operator runs or the console.
HTTPS · TLS 1.2+
HTTPS 443 to origin · Full (strict) + origin-pull client cert
:8201
/api/<svc>/ → 172.17.0.1:<port>
via IGW
TCP 5432 · TLS · role operonms_<svc>_app
SendEmail
docs, photos
secrets
start/stop
image pull
logs + metrics
HTTPS · OIDC sign-in, Graph directory sync
HTTPS · Bedrock Converse stream (eu-west-1)
1
2
3
4
5
6
7
8
LEGEND · every coloured line carries moving dots in the direction of traffic (arrows at both ends = request and response)
Browser → app request path
Service-to-service HTTP (docker0)
Events: SQS publish / consume
PostgreSQL 5432 (TLS)
AWS API calls via IGW (S3, SSM, KMS)
Email via SES
Logs and metrics → CloudWatch
Entra ID, Graph, Bedrock
Release path / image pull
OperoNMS platform infrastructure
Production · AWS af-south-1 (Cape Town) · Cloudflare · ECS on one EC2 instance
For onboarding infrastructure engineers. Source: ops/aws/production at main b21e3277.
Read from the checked-in templates, not from the live account. Verify with the commands at the bottom.
Scroll or pinch to zoom · drag to pan · keys: + − 0 1, space pauses · built from
main b21e3277