Case study at a glance
The problem, my role, and the result.
- problem
- A major provider outage left client sites without an independent continuity path.
- stakes
- Clients still needed a public landing page, current location information, and a working contact route, but a full production replica would add ongoing cost and maintenance.
- My ownership
- I moved from rapid incident response to designing the application flow, AWS orchestration, synchronization behavior, activation process, and team documentation.
- outcome
- Chestnut maintains a viable backup for all clients for a few dollars per month. Operators can start the standby server, update client DNS records, and make essential pages available through a documented process.
Portfolio-safe architecture
A focused continuity layer, not a full replica.
Chestnut provides a cost-conscious business-continuity layer that is independent of the primary CMS host. Scheduled AWS orchestration refreshes approved site data while the standby server remains powered off outside the synchronization window.
Scheduled infrastructure and synchronization
- Amazon EventBridgeWeekly synchronization
- AWS Step FunctionsChestnut workflow
- Lambda + Kamatera APIInspect server power state
- Server already running?Branch by current state
- Start only when neededPower on, wait for startup, verify, then trigger the protected synchronization endpoint
- Synchronize and return to standbyWait for the refresh window, then power the server off
Task failures route to Amazon SNS and the development support email.
Application-layer data flow
- Protected synchronization endpointStarts the Node.js and EJS application flow
- Iterate configured sitesCheck whether the franchise module is enabled
- Fetch approved Oak API dataLocations, variables, and territories
- Filter and normalizeKeep live locations, add territory and alias data, and flatten site variables
- Write per-site JSONPreserve existing fallback data when a non-empty response is unavailable
- Render emergency pagesEJS brand and location pages consume the stored snapshots
Per-site request or processing errors are logged before continuing to the next configured site.
Architecture
The boundaries that made the work useful.
Scheduled orchestration
EventBridge starts a Step Functions workflow that checks the backup server, starts it when needed, runs synchronization, and returns it to standby.
Application snapshots
A Node.js and EJS application retrieves approved Oak CMS data, filters live locations, and writes per-site JSON used by emergency pages.
Cost control
The server normally remains powered off. Scheduled refresh windows keep data current without paying for a complete always-on replica.
Focused continuity
The fallback preserves landing pages, location pages, branding flexibility, and SMTP contact capability rather than recreating the full CMS.
Key decisions
What I chose and why.
- 01
Preserved only the minimum customer-facing capabilities required during an outage instead of duplicating the complete production platform.
- 02
Used Step Functions and scheduled server power control to balance recovery readiness against monthly infrastructure cost.
- 03
Stored normalized per-site snapshots so emergency rendering did not depend on the primary CMS remaining available.
- 04
Documented setup, maintenance, and activation so response did not depend on one person.
Outcome & evidence
What the repository and delivered work support.
- 01
The initial incident lasted approximately 24 hours.
- 02
The current continuity layer supports essential landing pages, location pages, and an SMTP contact form.
- 03
Automated synchronization makes onboarding a new backup site a repeatable process while preserving room for brand customization.
- 04
The operating cost is only a few dollars per month for all client backup sites.
- 05
Activation is reduced to starting the Chestnut server and updating client DNS records through two intentional operator actions.
Reflection
What I learned and what I would do next.
The outage made the tradeoff concrete: useful resilience is not the same as an expensive full replica. The right system protects the customer path that matters and makes recovery understandable to the team.
I would keep testing activation and recovery procedures regularly so the operational process stays as dependable as the architecture.