Saturday, June 23, 2012

1000RP for Missing Data

Global Network Ops Manager
We have granted players who lost inventory the 1000 RP. We will be following up with each of you via email as well within the next several hours. Spend it as you please. 

Friday, June 22, 2012

Missing Data Bug

Sr. Producer, Live Service
Sure--and sorry about not being clear on that from the beginning. Our Database guy addressed a big segment of users last evening/early this morning before taking a quick nap break. I got with him just a few minutes back and he was back at work. And while I don't want to overpromise, especially when we're still triaging, he gave me a very strong commitment on his effort to resolve this.

Thanks for hanging with us,


Sturm Post Summarization

Lead Community Manager
I can summarize and slightly expand on Strum's post a bit. The TL;DR of Strum's post is that we've identified a lot of the issues that slipped through our patch tests, and we've rebuilt our test harness to look for those. It's absolutely our fault that we didn't catch these, but we understand why, and we're going to make sure we plug those holes. 

Patch Outage

Sr. Producer, Live Service

Lance, aka CaptSturm here. For those of you who don’t know me, I’m charged with maintaining the health of the League of Legends live service around the world. As you know, it’s been a rough week here in North America.

We are painfully aware of the problems many of you have been experiencing:

- Extended outages: The North American service has been unavailable for significant stretches of time
- Long queue times: We’ve had to throttle logins and concurrent players due to these issues
- We’ve also confirmed that some players are missing content: runes, rune pages, champions, etc.

This is frustrating for you, and we feel that frustration deeply. This isn’t the service you’ve come to expect and it isn’t what we want to deliver. It’s been code red for engineering and support Rioters, and we’ve been working literally around the clock to identify and resolve these problems.

Many of you have asked for more information on what’s causing these problems, and we understand that desire – I’m going to try my best to give you some insight into what we’re working on, but please understand that our first priority is to put out the fire rather than exhaustively document it.

- This patch is like an iceberg – what you see on the surface is small in comparison to the behind-the-scenes infrastructure changes it includes.
- The problems we’re experiencing are database-related – specifically, it’s a data throughput issue. Our infrastructure handles a massive volume of data and requests every day, and the backend changes in this patch introduced a new bottleneck.
- We put every patch through the wringer including load testing and PBE player testing, and this patch passed those tests. The problems didn’t crop up until we pushed it to live. This has exposed some load testing gaps that we’re going to address (simulating millions of concurrent players is truly challenging).
- Over the last 48 hours, we’ve made substantial hardware configuration changes that we believe will address the problems. We’re actively monitoring the service, but we’re still not out of the woods.

I know the thing you want to hear most is when everything’s going to be back to normal. I’d love to give you a timeframe, but we want to be absolutely confident in our solution before signaling the all-clear. Our number one priority is restoration of a stable game experience.

Thanks for your patience. I’m going to continue keeping you guys up-to-date as we work to fully restore League of Legends to the service you’ve come to expect.


Missing Runes 6/20

Lead Community Manager
Hey everyone,

We're currently looking into this issue to identify the root cause and subsequent fix. I don't have enough information to share what happened or how long this will take to fix, but we're definitely aware that this is happening, and will work as quickly as we can to fix the problem.

Hang tight! 

Friday, June 15, 2012

Appear Offline Status

Associate Software Engineer
Oh you ... this wasn't actually said anywhere :) It's an item that needs prioritization against countless other awesome ideas. This doesn't mean that it's not going to be done, but rather that we have a lot to do. A lot of people want this feature for reasons similar to those that have been frequently stated in this thread. Amongst those people are myself and many other Rioters! I've poked a lot of people about it for sure (In retrospect, half of those people were probably Status Kwoh).

So to provide an update of sorts, this isn't forgotten. It's still a topic of discussion, especially now that spectator mode has been rolled out in full. The lack of updates around this is frustrating, I know, but we're always hard at work trying to ship awesome things. I think since I last posted about this, we got mastery pages on the server, spectator mode, lots of client refactors (which might be a little transparent), and more. There's definitely a lot of movement going on here!

So in short, thanks for your patience. While I'm confident that you will enjoy the things we've been working on, it's also frustrating to us when we're met with an abundance of opportunities to make League of Legends more awesome and can't do them as fast as we want :)