arik Savage

Apti: Infrastructure, Reliability & Security

Besides the Knowledge Direct platform, I’m Apti’s unofficial DevOps and security lead. That covers twelve servers across production, staging and development, the Jenkins pipelines that deploy to them, and a handful of WordPress marketing sites. Nobody handed me this job. It just really needed an owner, and the results turned out to be very measurable.

The case of the mystery 503s

Learners kept randomly hitting a 503 error while a course was saving their progress. Weird, because nothing in the app could even return a 503 on that route, and the access logs looked perfectly clean. The first real clue was hiding in Apache’s error log: web-server workers were dying with a SIGBUS signal, 100 to 130 times a day.

The obvious suspect was the ancient, unmaintained MongoDB driver. Too obvious! So I set up core dumps to catch the crash in the act, and discovered they’d been silently blocked the whole time by a kernel setting that suppresses dumps from processes that switch users. Which is exactly what Apache workers do. Once the dumps started landing, every single one was identical, and MongoDB wasn’t in any of them.

The real culprit: PHP’s own lexer, crashing on the very first byte of a compiled template file. The Phalcon template engine rewrote its compiled cache in place, emptying the file and then writing it back. Any worker that had that file memory-mapped at the wrong moment got the rug pulled out from under it. A textbook race condition that only showed up under real traffic. Even the clean logs were a clue: a worker that dies mid-request never gets to write its log line.

Before and after, in one picture

Template cache race condition, before and after Before: worker A memory-maps the compiled template file and starts reading it. Worker B recompiles the template and truncates the same file in place before rewriting it. Worker A's mapping now points past the end of the file, so it crashes with SIGBUS and the learner gets a 503. After: worker B writes the new compiled template to a temporary file and then atomically renames it over the old one. Worker A keeps reading the complete old file, and later requests see the complete new file. No crash. Before: truncate, then write in place After: write temp file, then atomic rename() rename() Worker A mmaps & reads file Worker B recompiles template compiled.php truncated to 0 bytes ✗ SIGBUS in A worker dies → 503 Worker A reads old, complete file Worker B recompiles template compiled.tmp written in full compiled.php swapped by rename() ✓ No crash
Before: one worker reads the file while another empties it, and the reader crashes. After: the new version is written to a temp file and swapped in with a single atomic rename, so readers always see a complete file.

Quiet logs, quicker pages

The framework never fixed this upstream, not even in its newest major version, so upgrading wasn’t going to save us. Instead I wrote a ~50-line subclass of the template compiler that writes to a temp file in the same folder and then rename()s it into place. On Linux that swap is atomic, so readers always get either the whole old file or the whole new one.

Before going anywhere near production, I recreated the crash on a dev server under load. The stock compiler crashed a dozen times, with core dumps matching production’s. The patched one crashed zero times across ~44,000 requests. Then it rolled out in three steps: atomic writes, then no more recompiling on every request, then a cache warm-up baked into the Jenkins deploys. Crashes dropped to zero and stayed there. Bonus: business-hours p99 latency fell from ~980 ms to ~710 ms.

Meanwhile, elsewhere in the fleet

The great AWS audit

In 2023 the AWS bill averaged about $4,900 a month, so I went through the account line by line and checked everything against what the business actually used. There was a lot of attic in there: a complete spare production stack of four big servers, stopped since 2021, plus old recovery boxes and a GitLab upgrade test. Around two dozen storage volumes attached to nothing (or to those stopped servers), idle Elastic IPs, and 1,586 disk snapshots going back to February 2020, still being taken three times a day. The snapshots alone were 19TB and about $950 a month.

The next month’s bill was $3,439, down 30%. By November 2025 it was about $3,100, with snapshot storage down to around $220 a month. That’s roughly $21K a year back in the budget.

The same review covered the rest of our vendors. Algolia search went from about $284 a month to $121, and the CDN bill for one of our largest clients went from $8,138 a month to $4,590. Add in AWS and that’s over $5,100 a month less, or more than $60K a year.

  • AWS
  • IT Cost Optimization

What does a client actually cost?

To find out, I broke the November 2025 bill down. The four production servers ran $668 a month, which works out to $24.74 per client just to break even across our 27 active clients, or $48.99 once staging and development are counted too.

Storage was harder to pin down, until each client got its own volume and the answer became as simple as “this client uses 250GB.” Clients over their contract’s 40GB account for 570GB of overage, and I showed that charging $0.93 per GB per month above that line would cover every byte of storage across all three environments.

  • AWS
  • IT Cost Optimization
  • Multi-Tenant Architecture

One boss per process

Across 12 servers, services had drifted into a tangle. Some were run by systemd, some by supervisord, and some by both at once. That meant a cron job could restart a web server behind its supervisor’s back and leave two copies fighting over the same ports. I straightened out every box, cleared out years of leftover cruft, and wrote the team’s infrastructure reference so nobody has to rediscover all this the hard way.

  • Linux System Administration
  • systemd
  • supervisord
  • Documentation
  • MongoDB
  • Nginx

Four break-ins on one WordPress site

While auditing a production marketing site, I found not one but four separate compromises. One was a malware campaign spread across 1,300+ files. Another used stolen credentials to quietly publish 230+ spam posts over several months. After cleaning those up, I audited the rest of the sites and found a compromised ex-employee admin account on several of them.

  • WordPress
  • Security Audits
  • Incident Response

Locking the doors

A hand-rolled, nonce-based Content-Security-Policy and HSTS covering more than 15 third-party integrations. Plus CSRF nonces and timing-safe credential checks (researched against a published CVE) in older theme and plugin code, and PHP 8.2 compatibility fixes while I was in there.

  • Content Security Policy
  • Security Audits
  • PHP

Builds that match everywhere

Jenkins, the production servers and everyone’s laptops all had different Node.js and npm versions, so “works on my machine” was literally true. I standardized the toolchain so the whole team builds the exact same thing from the exact same code. I did the same for local WordPress work with some reusable DDEV / WP Engine tooling.

  • Jenkins
  • CI/CD
  • DDEV

Happier clients

As the main incident responder, I helped bring average helpdesk resolution down from ~31 hours to ~10, and bug fixes from ~50 hours to ~29, beating our internal SLA targets along the way. Weekly bug tickets also fell by more than half between 2023 and 2025.

  • Incident Response
  • Performance Metrics

Looking back

The lesson I keep learning: don’t trust the first explanation that sounds right. The old MongoDB driver looked so guilty, and chasing it would have eaten weeks without fixing anything. Getting real evidence (core dumps, a reproduction, a measured before and after) felt slower on day one and was way faster overall.

Stack & skills

  • AWS (EC2, EBS, S3)
  • Linux
  • Apache
  • Nginx
  • supervisord
  • systemd
  • Jenkins
  • WP Engine
  • gdb / strace
  • AWS
  • IT Cost Optimization
  • Linux System Administration
  • Incident Response
  • CI/CD
  • Security Audits
  • Content Security Policy
  • WordPress
  • Documentation
  • Performance Metrics
  • MongoDB

Related