Thanks to everyone who participated in our 1.1.1 stress test on Friday, and a special hat tip to those that provided feedback. We've got some good info to share and felt we'd run down what we learned.
First, a recap:
As you know, 1.1.0 introduced a variety of multiplayer issues. 1.1.1 saw some different problems, but still the same poor results.
We throttled up debug logs on the Live service early last week, then analyzed the data which in turn resulted in a variety of fixes.
Our primary goal was to hit concurrency levels that previously destabilized and knocked over the service.
Secondary goals were to test out various client and server fixes, particularly the matchmaking code and how our modules interact.
A final goal was to learn how we can best run a test with you so we can collect valuable end-user experience data.
All of this needed a proper test!
Timeline of the test:
At 4pm CDT Friday April 17, we asked as many players as possible to jump in multiplayer games.
We immediately broke through the previous barrier and the service held up.
As concurrency increased, we scaled up game servers, and all of our teams focused on performance and stability.
We did notice a few edge case networking and game bugs as more players joined, but the service still held together.
We ended the test at 5pm CDT, but continued to gather crucial information
From the player reports submitted (this was hugely valuable):
37% were Kicked Back To Lobby
28% were Crash To Desktop
14% were Infinite Load Screen/Black Screen
21% were assorted one-off client-side issues, such as missing inventory, disappearing ships, no HUD, invisible character heads, unresponsive leaderboard, etc.
Our assessment:
For a first run, this went exceptionally well.
We learned a ton from running a Live service stress test, plus we had a lot of fun with you in the process!
We're thrilled that so many of you not only took the time to test, but also to provide the feedback.
We're very pleased that service stability has improved and it held up to load, though there's obviously still many items to work on.
We knew we hadn't fixed all of the Kicked Back To Lobby and Infinite Load Screen errors, and this test helped clarify what we need to work on.
We've got a long list of client-side bugs to address.
Our To-Do List:
QA has catalogued every player report and assigned it to an internal bug report (if one didn't exist before). We've repro'd many of these bugs and have started the work on farming out to owners.
We've identified five different server-side items with Kicked Back To Lobby; some of those hotfixes are already published to Live.
We've got two engineers rewriting a small bit of code that we think will decrease the amount of Infinite Load Screen issues.
We've got a long list of client-side bugs that we'll begin work on. As we fix each, we'll be adding it to our changelist and providing additional information in our Patch Notes for the next release.
The stress test was hectic, fun, and educational. We will definitely do more focused stress tests in the future and at different times so more players around the world can participate. Look for the next stress test to be held some time after we publish 1.1.2 to Live.
Thanks again everyone and have a great Monday!
)
