OperoNMS production infrastructure Browsers reach Cloudflare, which proxies over HTTPS to an nginx edge container on one EC2 instance in a default VPC in af-south-1. The instance runs 25 ECS services on a Docker bridge network and talks to RDS PostgreSQL, SQS, SES, S3, SSM, KMS, ECR and CloudWatch. AWS · production account · region af-south-1 (Cape Town)Separate AWS account from staging. Everything below is created by CloudFormation stacks operonms-prod and operonms-prod-queues (budget: operonms-prod-budget in us-east-1), apart from the default VPC.VPC · default VPC of af-south-1 · 172.31.0.0/16 (AWS default)No private subnets · no NAT gateway · no ALB · no VPC endpoints · Amazon-provided DNS resolver for AWS and RDS namesPUBLIC SUBNET · af-south-1a (default subnet, /20, route 0.0.0.0/0 → IGW, auto-assign public IPv4)Holds the EC2 instance and the RDS instance. Different security groups separate them.PUBLIC SUBNET · af-south-1bEmpty. Member of RDS subnet group "operonms-prod" because RDS needs ≥2 AZs,even single-AZ. Same route table, same default NACL.Failover target if the stack is ever moved; nothing runs here today.PUBLIC SUBNET · af-south-1cEmpty. Member of RDS subnet group "operonms-prod" because RDS needs ≥2 AZs,even single-AZ. Same route table, same default NACL.Failover target if the stack is ever moved; nothing runs here today. CLOUDFLARE · DNS + reverse proxy + TLS Name Type → target Proxy *.operonms.co.za A → origin Elastic IP (every tenant address) on admin A → origin Elastic IP (OperoAdmin; beats the wildcard) on operonms.co.za, www CNAME → Pages project operonms-site on platform CNAME → Pages (this page) on Zone settings Free plan · nameservers aitana / brad.ns.cloudflare.com Always HTTPS · minimum TLS 1.2 SSL mode Full (strict): origin presents a Cloudflare Origin CA cert for operonms.co.za + *.operonms.co.za Authenticated Origin Pulls: Cloudflare shows a client cert; nginx rejects anything without it Adds X-Forwarded-For: <client> = proxy hop 1 of 2 Origin is off 20:00–07:45 weekdays and weekends → Cloudflare 521 page The wildcard serves every tenant with no DNS change per tenant. platform is a reserved tenant label (RESERVED_LABELS), so a tenant can never collide with this page. DNS records Forwarded-for hop count (why OPERO_FORWARDED_HOPS=2) Hop 1: Cloudflare appends the real client address to X-Forwarded-For. Hop 2: edge nginx appends the Cloudflare address it saw. A service therefore reads the client as the 2nd entry from the right; anything a client forges sits further left and is ignored Identity and Platform use the same count (OPERO_PLATFORM_TRUSTED_PROXIES=2) for the staff IP allow-list. If /api were ever routed through the web container, both become 3 Host passes through unchanged and X-Forwarded-Host is set from it, never taken from the client: services bind a request to its tenant from that name CLIENTS Tenant users · browserhttps://<tenant>.operonms.co.zaweb app + OperoAI panel (one label deep)OperoSystems staff · browserhttps://admin.operonms.co.zaOperoAdmin · Entra ID + MFA · IP allow-listCapture tablets · Android apptenant address, same /api pathsphotos up to 25 MB (edge limit) How clients reach itOnly by name: tenant and admin addresses resolve to Cloudflare, never to the origin IP. The origin accepts nothing that did not come through Cloudflare. STATIC HOSTING (outside AWS) Pages · operonms-site (marketing) operonms.co.za and www, built from the marketing repo and deployed with wrangler (--branch=main) dev branch alias is the noindex preview Serverless: no part of it runs in AWS Pages · platform.operonms.co.za (this page) The infrastructure diagram for onboarding engineers Static HTML + inline SVG, no backend, no secrets Generated by platform-site/build.py from ops/aws/production; rebuild after any infrastructure change Microsoft Entra ID + Graph Staff sign-in to OperoAdmin (OIDC, login.microsoftonline.com); amr=mfa required Identity worker reads the directory via Graph to sync leavers (GroupMember.Read.All) Reached from the instance over HTTPS 443 via the Internet Gateway Amazon Bedrock · eu-west-1 (Ireland) OperoAI (assistant) Converse / ConverseStream and Execution's pest & disease photo checks Claude Sonnet via EU inference profiles only (eu.anthropic.claude-sonnet-5-5 / 4-6), IAM condition aws:RequestedRegion = eu-* Data stays in the EU; no global. profile is allowed domains.co.za · operosystems.co.za DNS for the company mail domain (not on Cloudflare) SES Easy DKIM CNAMEs + MAIL FROM ses.operosystems.co.za (MX/SPF) added by hand Mailboxes are Microsoft 365; SES sends as no-reply@operosystems.co.za EC2 INSTANCE operonms-prod · t4g.medium (2 vCPU Graviton, 4 GiB) · Amazon Linux 2023 arm64 Single on-demand instance, af-south-1a · only inbound tcp/443 from Cloudflare · dashed outline = instance security group edge · nginxhost :443 → container :8443 · 128 MiBTLS · Cloudflare client-cert check · router web (SPA, nginx)host :8201 → :8080 · 64 MiBstatic tenant app admin (SPA, nginx)host :8202 → :8080 · 64 MiBstatic OperoAdmin console Edge routing (edge/default.conf) admin. host: /api/identity, /api/platform only; else admin SPA <tenant>. host (one label): /api/<svc>/ → that service; else web SPA Unknown SNI (apex, www, 2 labels, IP) refused at the handshake /api/*/internal/* → 404 (service-to-service only) docker0 bridge · gateway 172.17.0.1 · service-to-service calls (green) are http://172.17.0.1:<hostPort> · no DNS, no service discovery, no per-task ENI ECS cluster "operonms-prod" · 25 services, EC2 launch type, desired count 1 each, one container per task · columns = business service, rows = process type APIuvicorn ×1, :8000 WorkerSQS consumer/publisher Schedulerdaily jobs identity:8101 → :8000512 MiB · db identity identity-workermode: all256 MiB · pool 1+1 platform:8110 → :8000256 MiB · db platform platform-workermode: all192 MiB · pool 1+1 masterdata:8102 → :8000256 MiB · db masterdata masterdata-workermode: all192 MiB · pool 1+1 planning:8103 → :8000256 MiB · db planning planning-workermode: events256 MiB · pool 1+1 planning-schedulercron-like jobs192 MiB space:8104 → :8000256 MiB · db space space-workermode: all192 MiB · pool 1+1 orders:8105 → :8000256 MiB · db orders orders-workermode: all256 MiB · pool 1+1 execution:8106 → :8000320 MiB · db execution execution-workermode: events256 MiB · pool 1+1 execution-schedulercron-like jobs192 MiB ingest:8108 → :8000256 MiB · no database insights:8109 → :8000320 MiB · db insights insights-workermode: facts256 MiB · pool 1+1 insights-schedulercron-like jobs384 MiB assistant:8111 → :8000320 MiB · db assistant assistant-workermode: all192 MiB · pool 1+1 event lane Task definition (one per service) family operonms-prod-<service> · networkMode: bridge · requiresCompatibilities: EC2 · ARM64/Linux one essential container, initProcessEnabled (init as PID 1 reaps probe zombies) image <account>.dkr.ecr.af-south-1.amazonaws.com/operonms/<svc>:<12-char git tag> (immutable) memory = hard limit; memoryReservation = 45% of it portMappings: fixed hostPort per service (8101…8111, 8201/8202; edge 443 + 127.0.0.1:8080 for on-box smoke) executionRoleArn operonms-prod-ecs-execution · taskRoleArn operonms-prod-ecs-task logs: awslogs → /operonms/prod, prefix ecs, non-blocking 4 MiB buffer secrets: valueFrom SSM /operonms/prod/* (whole parameters); DB URLs assembled by a shell entry point at start, private vars then unset health: API → python probe GET 127.0.0.1:8000/health/live every 30 s (3 retries, 60 s grace); workers → no-op CMD true every 300 s; edge → wget /healthz on 8080 ECS service settings launchType EC2 · REPLICA · desired count 1 · no load balancer · no service discovery · no autoscaling minimumHealthyPercent 0, maximumPercent 100: fixed host ports allow only one copy, so a release stops the old task, then starts the new (seconds of downtime per service) Cluster Container Insights off (cost); capacity = this one instance One-off tasks: operonms-prod-migrate-<svc> runs alembic upgrade head (Identity, Platform, assistant as their owner role) Why not awsvpc: a t4g.medium has 3 ENIs, there are 25 tasks Why not Fargate / ALB: no NAT (~US$40/mo) and no ALB cost; ECS on EC2 adds no charge ECS agent: ecs-init from AL2023 (deploy.py ecs-bootstrap), registers the instance via the instance role Memory & CPU budget Container hard limits total 6,080 MiB on a 4 GiB instance: deliberately oversubscribed; each limit only caps one runaway container Reservations (45%) total 2,732 MiB: what ECS will place; leaves room for one-off migrate tasks Expected real use at one tenant: 2.0–2.2 GiB in containers, 65–70% of the instance (estimate; measure with docker stats) 2 GiB swapfile, vm.swappiness=10: a safety margin, not capacity CPU credits unlimited (since 2026-11-12): standard credits ran dry, throttled the box, and every API failed its health check in a loop Alarm at memory >90% for 10 min → resize to t4g.large Argon2id sign-in hashes: 64 MiB each on Identity (512 MiB limit) Docker engine · live-restore · log driver local 20m×3 ECS agent (ecs-init) SSM agent (no SSH, no key pair) CloudWatch agent: mem + disk /60 s IMDSv2 required · hop limit 2 EBS 40 GiB gp3, encrypted RDS FOR POSTGRESQL · operonms-prodSecurity group operonms-prod-db: inbound tcp/5432 from the instance SG only · outbound: none (127.0.0.1/32 placeholder) Instance & storage Engine PostgreSQL 16 (16.15) · class db.t4g.micro (2 vCPU, 1 GiB, ~80 max connections) Single-AZ af-south-1a, same AZ as the instance (no cross-AZ traffic charge); MultiAZ: false gp3 20 GiB, storage autoscaling to 100 GiB; encrypted with CMK alias/operonms-prod PubliclyAccessible: false: no public IP, only reachable from the instance SG Master opero_admin, password RDS-managed in Secrets Manager (nobody types it); instance role can read it only while DatabaseBootstrapAccess=enabled Subnet group operonms-prod spans the 3 default subnets (RDS needs ≥2 AZs) Parameters, backup & operations Parameter group postgres16: rds.force_ssl=1 (TLS only), log_min_duration_statement=1000 ms; postgresql log exported to CloudWatch Automated backups 14 days (PITR) · window 19:00 SAST · maintenance Wed 18:00 SAST · AutoMinorVersionUpgrade DeletionProtection, DeletionPolicy: Snapshot (final snapshot on delete) Weekday schedule: started 07:30, stopped 20:10 (EventBridge Scheduler); a stopped DB takes no backups Manual snapshot before every release: operonms-prod-predeploy-<tag>-<time> Connection budget: worst case 45 of ~80 (9 database-backed APIs × (pool 2 + overflow 1) + 9 workers × (1+1)); steady state ≈ 20; alarm at >70 → db.t4g.small Databases & roles (db-init) 9 databases on one instance: operonms_ identity, platform, masterdata, planning, space, orders, execution, insights, assistant Identity, Platform and assistant: *_owner owns the schema and runs migrations; the service runs as *_app, which owns nothing Other services: operonms_<svc>_app owns its DB; isolation by FORCE ROW LEVEL SECURITY (ADR 0049) No SUPERUSER / BYPASSRLS / shared role; PUBLIC cannot connect, so one service cannot read another's database Ingest has no database HOW CODE GETS TO PRODUCTION · build off-box, deploy by SSM, never SSHEvery production change is a person's action. `deploy.py` refuses any account but production, and the CloudFormation change set is read and executed by a human. 1 Engineerbranch in a git worktree → green gate on a test DB → one session merges to main 2 CI · JenkinsADR 0117: tests, SonarQube, OWASP dep-check, builds; runs on the dev box, ECR push role in prod 3 buildx · arm64deploy.py images builds linux/arm64 for 13 repos, tag = 12-char git SHA; push to ECR 4 ECRimmutable tags, scan on push, keep last 10 5 Stack changedeploy.py stack → CloudFormation change set; human reads & executes (IaC apply is never automatic) 6 SSM Run Commanddeploy.py deploy → on the instance: fetch bundle from S3 ops bucket, render env, RDS snapshot, migrate 7 ECS rolloutecs-deploy: register task defs, update each service (stop → start), wait stable 8 Smokeon-box 127.0.0.1:8080/healthz + public smoke through Cloudflare Amazon SQS · event bus between services 27 queues + 27 dead-letter queues operonms-prod-<service>-<topic>[-dlq] (stack operonms-prod-queues, generated by render_queues.py from ops/scripts/create-event-queues.sh, so prod cannot drift from LocalStack/staging) Visibility 60 s · long-poll 20 s · retention 4 d (DLQ 14 d) · maxReceiveCount 5 · SSE-SQS Any message in a DLQ → CloudWatch alarm → SNS email Routing maps OPERO_<SVC>_QUEUE_URLS rendered into each service's env; auth is IAM (task role), no public policy By name prefix identity: operations, platform-actions, support-requests · masterdata: platform-control planning: availability, order-cover, order-impact, outbound, platform-control space: execution, location, planning, platform-control orders: availability, dispatch, outbound, planning, platform-control, qc, trays execution: inbound, outbound, platform-control · insights: facts, platform-control platform: health · assistant: platform-control + operonms-prod-identity-mail-events (+DLQ), in stack operonms-prod Amazon SES + SNS · email and bounce handling Identity operonms-prod config set; sender identity operosystems.co.za (af-south-1), no-reply@operosystems.co.za; Easy DKIM RSA-2048; MAIL FROM ses.operosystems.co.za Production access granted 2026-11-04 (50,000/day, 14/s). IAM allows SendEmail only from that address through that identity + config set Suppression list: bounce + complaint. Events (bounce, complaint, reject, deliveryDelay) → SNS operonms-prod-mail-events → email to ops and SQS identity-mail-events → Identity worker stops mail to that address (ADR 0116) SNS operonms-prod-alarms: target of every CloudWatch alarm and budget warning; email subscription to the ops mailbox Amazon S3 · 3 buckets (KMS-encrypted, TLS-only, public access blocked) operonms-prod-ops-<acct>: exports & tenant exports (30-day expiry), deploy bundles (180 d), breached-password list operonms-prod-audit-<acct>: Object Lock (COMPLIANCE, 7 yr) + versioning; platform audit chain; role can write and lock, never delete operonms-prod-documents-<acct>: uploads and photos, versioned; scan-inbox/ expires in 1 d; assistant-imports/ 7 d All DeletionPolicy: Retain, SSE-KMS with bucket keys, ownership enforced KMS · SSM Parameter Store · Secrets Manager KMS one CMK alias/operonms-prod (rotation on): SSM SecureStrings, S3, exports, RDS, EBS excluded (uses default) SSM SecureString under /operonms/prod/…: ten DB passwords, RS256 key pair, MFA key, Entra secret, origin cert/key/pull-CA. Free tier, vs Secrets Manager at US$0.40/secret Delivered two ways: ECS agent injects them into containers (execution role); host/render_env.py renders 0600 env files for compose-era paths Secrets Manager: only the RDS master secret Amazon ECR · 13 repositories operonms/<identity|masterdata|planning|space|orders|execution|ingest|insights|platform|assistant|web|admin|edge> (+ legacy finance until ADR 0115 clean-up) Tags immutable (12-char git SHA), scan on push, keep last 10 → rollback = start the previous tag linux/arm64 images built off-box with buildx; the instance only pulls (execution role + ecr:GetAuthorizationToken) Amazon CloudWatch · logs, metrics, alarms Logs: group /operonms/prod, 30-day retention, stream per container (ecs/<name>/<task id>); RDS postgresql log group Metrics: CWAgent mem_used_percent, disk_used_percent; 10 OperoAI metric filters → namespace OperoNMS/OperoAI 44 alarms → SNS: EC2 system check (auto-recover) & instance check (reboot) · CPU >80% · CPU credits <50 · memory >90% · disk >80% · RDS CPU, storage <2 GiB, memory, connections >70 · 28 dead-letter queues · OperoAI error-rate, p95 TTFT, Bedrock throttles, daily cost, tenant-at-limit · staff-sync-failed Missing data is not breaching in the schedule's off hours, so nights are quiet AWS Budgets (us-east-1): US$80/month (80% actual, 100% forecast, 100% actual) + Bedrock US$30/month EventBridge Scheduler · GuardDuty Malware Protection Scheduler (no Lambda), TZ Africa/Johannesburg, Mon–Fri: db-start 07:30 · instance-start 07:45 · instance-stop 20:00 · db-stop 20:10; role operonms-prod-scheduler GuardDuty plan on documents/scan-inbox/: file tagged NO_THREATS_FOUND or refused (ADR 0118); ClamAV compose profile stays off (needs 2 GiB) Security group · operonms-prod-instance Security group · operonms-prod-db Dir Proto / port Source / dest Why IN tcp 443 ×15 Cloudflare IPv4 ranges (below) the only way in IN — no SSH, no 22, no IPv6, no ICMP admin via SSM Session Manager OUT all 0.0.0.0/0 (CloudFormation default; no explicit rule) ECR, SQS, SES, SSM, S3, Logs, Entra, Graph, Bedrock are public endpoints; no NAT / VPC endpoints Cloudflare ranges: 173.245.48.0/20 · 103.21.244.0/22 · 103.22.200.0/22 · 103.31.4.0/22 · 141.101.64.0/18 · 108.162.192.0/18 · 190.93.240.0/20 · 188.114.96.0/20 · 197.234.240.0/22 · 198.41.128.0/17 · 162.158.0.0/15 · 104.16.0.0/13 · 104.24.0.0/14 · 172.64.0.0/13 · 131.0.72.0/22 The list is compared with cloudflare.com/ips-v4 by deploy.py cloudflare-ips; update both together. These ranges are shared by all Cloudflare customers, so nginx also demands Cloudflare's client certificate (Authenticated Origin Pulls) to prove the request came through our zone. Security groups are stateful: replies to allowed inbound flows need no rule. Containers share this SG (bridge network); there is no per-service filtering. Dir Proto / port Source / dest Why IN tcp 5432 SG operonms-prod-instance (by SG id, not CIDR) only the instance reaches Postgres OUT all 127.0.0.1/32 (default allow-all removed) the database initiates nothing Referencing the SG instead of an IP keeps the rule valid if the instance is replaced. rds.force_ssl=1 adds TLS on top: a plaintext client is refused by Postgres even from the allowed SG. Planned tightening (not applied) Replace open egress with 443 to the AWS prefix lists (ECR, S3, SQS, SES, SSM, Logs in af-south-1; Bedrock in eu-west-1), 443 to Entra/Graph, 5432 to the DB SG; run VPC flow logs (~US$0.50/GB) for a week before removing the old rule Hop limit 1 once compose rollback is no longer wanted (containers can still reach the instance role today) A task role per service instead of one shared role Routing, subnets & NACL · how they are configuredRoute table (the VPC's main table, shared by all 3 subnets) 172.31.0.0/16 → local (instance ↔ RDS stays inside the VPC) 0.0.0.0/0 → Internet Gateway (inbound from Cloudflare, and every outbound call) No NAT gateway route: a NAT gateway is ~US$40/month in af-south-1, half the budget No VPC endpoints for SQS/SES/S3/SSM/ECR/Logs: they would cost more than the instance. Traffic to those goes out through the IGW to public endpoints, TLS + IAM-signed Subnets (the region's 3 default subnets, one per AZ, /20) af-south-1a public: the instance + RDS · auto-assign public IPv4 · Elastic IP on the instance af-south-1b, 1c public: empty; only there because an RDS subnet group needs ≥2 AZs A subnet is "public" because its route table points at an IGW. What actually exposes things is the SG: the instance admits Cloudflare only; RDS has no public IP NACL: default (allow all in and out). Security groups are the control. The VPC is AWS's default VPC (172.31.0.0/16 by default), passed to CloudFormation as parameters, not created or managed by the stack. Confirm with the commands in "Verify it live". IAM · roles and what they may do operonms-prod-instance (EC2 profile): AmazonSSMManagedInstanceCore, AmazonEC2ContainerServiceforEC2Role, + runtime policy: logs, PutMetricData (CWAgent only), SQS on operonms-prod-*, SES from one address, SSM read /operonms/prod/*, KMS key, S3 (audit write-once), Bedrock EU, ECR pull operonms-prod-ecs-execution (agent, before start): ECR pull, log write, ssm:GetParameters, kms:Decrypt operonms-prod-ecs-task (all 25 tasks share it): SQS, SES, KMS, S3, Bedrock Converse (EU profiles only) operonms-prod-scheduler: Start/Stop that one instance and that one DB operonms-prod-malware-scan: GuardDuty plan, scan-inbox/ only Trust policies carry aws:SourceAccount. Container credentials come from the task role endpoint (ECS), falling back to IMDSv2 Operators: from the staging account, with MFA, assume OperoReadOnly or OperoAdmin (12 h). CI has operonms-ci-ecr-push (ECR push only) deploy.py refuses any account but the production one IGWInternet Gatewayinbound: Cloudflare → Elastic IP · outbound: AWS APIs, Entra, Bedrock Elastic IP (retained) SG operonms-prod-instance · in: tcp/443 Cloudflare Request lifecycle · browser → OperoNMS (badges 1–8)1 Browser → Cloudflare acme.operonms.co.za matches the proxied wildcard A record; TLS 1.2+ ends at Cloudflare; HTTP is redirected to HTTPS 2 Cloudflare → origin Cloudflare applies its rules, appends the client IP to X-Forwarded-For (hop 1) and opens a new TLS connection to the Elastic IP :443, validating our Origin CA cert and presenting its client certificate 3 Internet Gateway + security group Only Cloudflare's 15 IPv4 ranges may reach tcp/443; anything else is dropped before nginx sees it 4 Edge nginx (host :443 → :8443) Verifies Cloudflare's client cert, picks the server block from SNI/Host, refuses unknown names. Static paths go to web (:8201) or admin (:8202) 5 /api/<service>/ → service Prefix stripped, request proxied to 172.17.0.1:<port>; nginx appends Cloudflare's address (hop 2). Services take the client as 2nd from the right (OPERO_FORWARDED_HOPS=2); Host is untouched so the tenant is read from it (ADR 0100) 6 Service uvicorn (1 worker) checks the RS256 bearer token against the tenant of the Host (401 wrong_tenant_address otherwise); calls peers over the bridge when needed 7 Database TCP 5432 over TLS as that service's own role; tenant data isolation by row-level security 8 Events, mail, files State changes publish to SQS; workers long-poll, update their own DB, send mail via SES, store files in S3 (scanned by GuardDuty). The response returns the same path; every container logs to CloudWatch Other environments & who uses what Production: this diagram. Account separate from staging; weekdays 07:45–20:00 SAST Staging: separate AWS account, us-west-2, ECS Express Mode (Fargate behind AWS-managed load balancers, one *.on.aws host per service); own SES identity so test mail never touches the company domain; scaled down when unused. Different shape from prod on purpose Dev box (an EC2 instance): Docker Compose dev stack (Postgres, LocalStack, Mailpit), Vite dev server, Caddy with sslip.io hostnames for the SA team, isolated Playwright e2e stacks, and CI: Jenkins + SonarQube + Grafana + Prometheus (ADR 0117) LocalStack stands in for SQS/SES/S3 in dev; the same create-event-queues.sh feeds LocalStack, staging and queues.yaml Marketing & this page: Cloudflare Pages, no AWS Android tablet: APK downloaded from OperoAdmin / NMS Capture Tablets; checks for a minimum version (ADR 0137) Naming AWS names operonms-prod-*; SSM /operonms/prod/*; log group /operonms/prod; ECR operonms/*; cluster operonms-prod Tags: application=operonms, environment=production Known gaps · read before you change anything One instance, one AZ, one DB (single-AZ RDS): no standby. EC2 auto-recovers; RDS has PITR 14 d. An AZ outage is an outage Off at night and weekends on purpose (cost: ~US$40/month vs ~69 always-on). Cloudflare shows 521 outside hours Deploys are stop-then-start per service (fixed host ports): seconds of 502 for that service No ALB, no NAT, no private subnets, no VPC endpoints. Egress to anywhere on 443 is allowed; allow-list proposal is written, not applied IMDS hop limit 2: a container can reach the instance role. One shared task role for 25 services (Bedrock grant is shared). Both to be tightened Oversubscribed memory (limits > RAM) and 2 GiB swap: watch the memory alarm, resize to t4g.large if it fires Cloudflare IP list is static in the SG: re-check with deploy.py cloudflare-ips Entra needs amr in the app's ID-token optional claims (set through the manifest); Graph GroupMember.Read.All consent was still owed to the staff sync (check the staff-sync alarm) Not measured: memory and connection numbers are estimates until a week of docker stats data exists Daily rhythm · Africa/Johannesburg (SAST), Monday–Friday 07:30 db-start (Scheduler) → 07:45 instance-start → ECS agent restarts every service (desired count 1) → 07:55 tray and QC morning lists (Execution) 19:00 RDS backup window (17:00 UTC) · Insights nightly rebuild · order-cover run (Planning) → 19:15 surplus upkeep → 19:30 OperoAI purge 20:00 instance-stop (clean shutdown) → 20:10 db-stop. Weekends: off. Storage and the Elastic IP bill while stopped Wednesday 18:00 RDS maintenance window Out-of-hours work: start the DB, then the instance, by hand, and re-enable any schedule you disabled (a disabled stop schedule runs and bills 24 h) Verify it live · read-only commands (profile operonms-prod-ro, region af-south-1) Network: aws ec2 describe-vpcs --filters Name=is-default,Values=true · describe-subnets · describe-route-tables · describe-security-groups --group-names operonms-prod-instance operonms-prod-db Compute: aws ecs list-services --cluster operonms-prod · describe-services · aws ec2 describe-instances --filters Name=tag:Name,Values=operonms-prod Data: aws rds describe-db-instances --db-instance-identifier operonms-prod · aws sqs list-queues --queue-name-prefix operonms-prod- State of the stack: aws cloudformation describe-stacks --stack-name operonms-prod · drift detection from the console On the box (SSM, via deploy.py remote status): docker stats, docker ps, ECS agent introspection on :51678 Rollback and recovery Roll back a release: start the previous immutable tag (deploy.py rollback --to <tag> for the compose era; on ECS re-run ecs-deploy with the previous tag). Database: restore the pre-deploy snapshot or point-in-time to a new instance. Migrations that delete data end rollback (see ADR 0115 notes). Instance failure: EC2 recovers onto new hardware (system check alarm); containers return on boot via ECS; instance-check failure reboots. Region/AZ loss: no standby; rebuild from the stacks + snapshot. Deploy needs OPERO_ASSISTANT_DEPLOY=on for images and deploy, or the assistant services are left out of the release. Admin access has no SSH: SSM Session Manager or Run Command, authenticated by IAM. The dev box has no Session Manager plugin, so use scripts the operator runs or the console. HTTPS · TLS 1.2+ HTTPS 443 to origin · Full (strict) + origin-pull client cert :8201 /api/<svc>/ → 172.17.0.1:<port> via IGW TCP 5432 · TLS · role operonms_<svc>_app SendEmail docs, photos secrets start/stop image pull logs + metrics HTTPS · OIDC sign-in, Graph directory sync HTTPS · Bedrock Converse stream (eu-west-1) 12345678 LEGEND · every coloured line carries moving dots in the direction of traffic (arrows at both ends = request and response)Browser → app request pathService-to-service HTTP (docker0)Events: SQS publish / consumePostgreSQL 5432 (TLS)AWS API calls via IGW (S3, SSM, KMS)Email via SESLogs and metrics → CloudWatchEntra ID, Graph, BedrockRelease path / image pull OperoNMS platform infrastructureProduction · AWS af-south-1 (Cape Town) · Cloudflare · ECS on one EC2 instanceFor onboarding infrastructure engineers. Source: ops/aws/production at main b21e3277.Read from the checked-in templates, not from the live account. Verify with the commands at the bottom.
Scroll or pinch to zoom · drag to pan · keys: + − 0 1, space pauses · built from main b21e3277