CaptSturm
Sr. Producer, Live Service
|
|---|
```Sure--and sorry about not being clear on that from the beginning. Our Database guy addressed a big segment of users last evening/early this morning before taking a quick nap break. I got with him just a few minutes back and he was back at work. And while I don't want to overpromise, especially when we're still triaging, he gave me a very strong commitment on his effort to resolve this. Thanks for hanging with us,
'Sturm
Tamat
Lead Community Manager
|
|---|
I can summarize and slightly expand on Strum's post a bit. The TL;DR of Strum's post is that we've identified a lot of the issues that slipped through our patch tests, and we've rebuilt our test harness to look for those. It's absolutely our fault that we didn't catch these, but we understand why, and we're going to make sure we plug those holes.
CaptSturm
Sr. Producer, Live Service
|
|---|
Summoners, Lance, aka CaptSturm here. For those of you who don’t know me, I’m charged with maintaining the health of the League of Legends live service around the world. As you know, it’s been a rough week here in North America.
We are painfully aware of the problems many of you have been experiencing:
- Extended outages: The North American service has been unavailable for significant stretches of time
- Long queue times: We’ve had to throttle logins and concurrent players due to these issues
- We’ve also confirmed that some players are missing content: runes, rune pages, champions, etc.This is frustrating for you, and we feel that frustration deeply. This isn’t the service you’ve come to expect and it isn’t what we want to deliver. It’s been code red for engineering and support Rioters, and we’ve been working literally around the clock to identify and resolve these problems.
Many of you have asked for more information on what’s causing these problems, and we understand that desire – I’m going to try my best to give you some insight into what we’re working on, but please understand that our first priority is to put out the fire rather than exhaustively document it.
- This patch is like an iceberg – what you see on the surface is small in comparison to the behind-the-scenes infrastructure changes it includes.
- The problems we’re experiencing are database-related – specifically, it’s a data throughput issue. Our infrastructure handles a massive volume of data and requests every day, and the backend changes in this patch introduced a new bottleneck.
- We put every patch through the wringer including load testing and PBE player testing, and this patch passed those tests. The problems didn’t crop up until we pushed it to live. This has exposed some load testing gaps that we’re going to address (simulating millions of concurrent players is truly challenging).
- Over the last 48 hours, we’ve made substantial hardware configuration changes that we believe will address the problems. We’re actively monitoring the service, but we’re still not out of the woods.I know the thing you want to hear most is when everything’s going to be back to normal. I’d love to give you a timeframe, but we want to be absolutely confident in our solution before signaling the all-clear. Our number one priority is restoration of a stable game experience.
Thanks for your patience. I’m going to continue keeping you guys up-to-date as we work to fully restore League of Legends to the service you’ve come to expect.
Salute,
'Sturm
Tamat
Lead Community Manager
|
|---|
Hey everyone, We're currently looking into this issue to identify the root cause and subsequent fix. I don't have enough information to share what happened or how long this will take to fix, but we're definitely aware that this is happening, and will work as quickly as we can to fix the problem.
Hang tight!
