Once a year, one of our clients enrolls its entire membership (4,780 people and counting) in a new batch of courses. On paper that’s one click: assign the courses to the “members” group and let background workers enroll everyone, 25 at a time. In practice it was hours of refreshing a page per course, watching a number climb, and restarting the whole thing every time the number stopped. So I took it apart, found out why, and rebuilt it one small, shippable phase at a time.
There was no progress indicator, so the only way to know what was happening was to sit on the course’s enrollments tab and hit refresh. Sometimes the count stalled way short of 4,780, and you had to start over. Already-enrolled learners were skipped, which helped, but somebody still had to watch it, restart it, and watch it again, for every single course.
It wasn’t only one client’s problem, either. Every background job on the platform (course publishing, password-reset emails, webhooks, everybody’s enrollments) shares one queue and one pool of workers. A big enrollment re-queues itself chunk after chunk and fans out messages for each learner, so another client publishing a course could find themselves stuck in line behind thousands of jobs that had nothing to do with them.
The first time I ran the full 4,780-person group on a dev server, it died on chunk 1 of 192 and left no trace anywhere. It took a chain of clues to find out why.
Every chunk re-queued the next one, and each time it carried along the full list of all 4,780 user IDs, right next to the chunked copy of the same IDs that was the only part ever read. The core API then re-encoded that payload (urlencode(json_encode()), for good measure), which inflated it from ~47KB to ~67KB. Beanstalk caps a job at 65,535 bytes, so it refused. The error got swallowed, and the worker runner treated the non-zero exit as a success and deleted the job. Everything reported fine, and 4,755 people never got enrolled.
unset() on the redundant list brought the job back under the limit, and a full 4,780-learner run finished for the first time: about 45 minutes, no duplicates.The one-line fix got it working, and working isn’t the same as trustworthy. Each phase below shipped and got tested on its own, so I always had a working system and a measurement to beat:
To build a test group of thousands of learners, I had to import them, and each import took five seconds per person. That turned out to be a hard-coded sleep(5), added years ago so activation emails wouldn’t trip over each other. Sending those emails through the async queue instead made imports 11 times faster, for everyone, not just my test data.
Midway through a test run, an hourly worker restart killed a worker mid-chunk. Beanstalk did its job and handed the orphaned chunk to another worker. The enrollments themselves were already protected from duplicates, but the progress counter wasn’t: it counted that chunk twice and declared the run finished early. Now each chunk checks in once, using a job-page log the codebase already had, and a redelivered chunk just steps aside.
Assigning a course to a group, clicking “Enroll All” on a filtered list and uploading a CSV all did the same job in different ways. The CSV upload even did it inline, making the admin’s browser wait. All three now go through the same hardened, observable worker, so a fix to one is a fix to all of them.
My favorite part is how unglamorous the root cause was. No race condition, no exotic bug, just a list that got carried along one step too far, plus an error handler optimistic enough to call a crash a success. The instrumentation came first on purpose: once I could see every chunk, every fix after that was a measured before and after instead of a hunch. The bulk queue is the one piece still waiting for its moment. It’s switched on and ready, but no enrollment that big has come along since the one that started all this. That’s fine by me.