A fascinating look at the detective work required to fix a complicated bug.
Today in “making videogames is hard” news: Respawn’s journey to track down a bug that caused months of audio issues in Apex Legends. Grenades that don’t explode, guns that don’t shoot, damage that has no source, and months of agonizing investigation—all apparently caused by a single line of code added in Apex Legends’ Season 16 update.
As outlined in a thorough Reddit post by Respawn community manager Amy Thiessen, the trouble began at the start of Season 16 in February. The studio had started getting reports of “disappearing nades” in Apex. Respawn soon determined that grenades weren’t “disappearing” exactly, but they would sometimes fail to explode despite damaging players.
“This had not occurred during our Season 16 playtesting, could not be reproduced internally after initial reports, and was very difficult to pin down using live gameplay videos as the root cause was not always shown in the player’s POV,” the post reads.
Respawn got a better handle on the problem after receiving similar reports about missing gun sound FX and particle effects. “After a preliminary investigation, the primary suspect was found to be the system our servers use to dispatch ‘start’/’stop’ commands for various effects (e.g. certain sounds, particle systems, physics impacts, bullet tracers, explosions).”
Dev Team Update: Audio Update from r/apexlegends
Essentially, something was happening during a match that could overload the server’s limit for sound FX or particles, causing some sounds and FX to get dropped.
“From there, the theory was that something may be flooding this engine limitation, requesting thousands of effects every second!” the post says. “But was this a systemic issue or could it be a single entity acting up? Every season update comprises thousands of changes to assets, code, script, and levels. Which meant finding a needle in a haystack.”
Respawn turned to metrics to help suss out the problem, but nothing in the telemetry indicated a clear issue. This suggested to Respawn that this bug was a unique situation their systems had not previously seen.
“This left us with a complex issue that we knew was impacting our community, but was hard to reproduce despite detailed reports, had minimal leads internally, and there were no metrics to prove definitively that this limit was being hit at all.”
Where do you go from there? Respawn decided to test its theory of overloaded effects by intentionally breaking Apex Legends servers. The team spun up a test build and spawned 50 characters that all fired guns at the same time and infinitely used abilities to push the server effects load over the edge. It worked—the team could finally reproduce audio drops similar to the bug reports, but how it was happening to actual players was still a mystery.
“This gave us proof that FX would get dropped, but only with completely unrealistic test cases. Various aspects of our server performance were investigated, but nothing definite was found.”
Respawn kept a close eye on the issue as Season 16 raged on. The team eventually noticed that dropped audio reports tended to come from high-level play. This gave them the idea to deploy a server update that let Respawn track new metrics in a smaller subset of matches, which instantly led to a breakthrough.
“As the server update was finalizing, we found it. A single line of code was identified to be the root cause of the issue. Season 16’s new weapon.”
That weapon is the Nemesis, Apex’s newest burst-fire energy assault rifle. The Nemesis has a unique mechanic where dealing damage will “charge” the gun and make it shoot faster (as demonstrated by YouTuber Dazs above). This charging effect is represented visually on the gun by arcing electricity within the barrel. Respawn says that a line of code meant to tell this effect to “stop” while the gun wasn’t charged or holstered was actually repeating indefinitely for all players holding a Nemesis in their inventory.
“This means that every single player with an uncharged Nemesis would create a ‘stop particle’ effect on the server every frame, and this line of code was being called even when the weapon was holstered.”
Funnily enough, this also explains why the audio drops were happening more often in high-level play. “14 clients with a Nemesis running at 180 fps would be enough to cause FX to begin being dropped.” For once, it was the top-spec PC players who had a disadvantage.
Respawn says this also explains why its internal testing didn’t encounter the bug.
“The builds used for testing might not have had enough holstered Nemesis in play, had a rarer correlation with missing FX, or didn’t have enough clients at that fps—something for us to keep in mind and improve on for future testing.”
A patch deployed last week finally squashed the bug for good. And there you have it—a meddlesome audio bug with a complicated root cause that will, in the long run, help Respawn catch similar bugs before they reach players. Respawn concluded the post with an aside about testing, reminding players that “a minute of players playing Apex is the equivalent of 10 testers playing the game for a year!”
About Post Author
