Day One of new servers

There are parts of the sites running really fast, and other parts running really slow. This is because we moved some of our database to a new storage engine to allow us to update things during the day and not impact the performance of the site... however an unanticipated consequence of this is that many of our queries are no longer optimized for this new storage engine... so we are monitoring the site... looking at the slow points, and then figuring out how to re-write it for performance and scalability.

If you happen to notice something that is consistently slow for you, feel free to send us an email about it at mail@360voice.com and we can dig a little deeper.

Blogs are running for 9.27.2007 right now and the games stats tables are almost done... I have to imagine we should get the entries for 9.28 and 9.29 as well as leaderboards and badges updated today.

(Click the title of this post and leave a comment if you have any questions!)

Update 10:58am CT

Alright... we are getting to the bottom of almost everything... game stats are almost done... but still have to replicate (about a million rows behind) but we are getting close. Once that is done we will run blogs for 9/28 and 9/29 and then run all the individual game stat updates (so the charts are up to date), leaderboards, badges, etc.

If you are leaving comments and wondering why you aren't seeing them, that is because of the backed up replication. So they will get there.

We also found some database issues that were hurting performance... so once those replicate things should pick up a little more. Thanks again everyone for your patience!

Update 9:56pm CT

Well, remember that replication we talked about? It is still running... taking about 2 hours per month. It is up through June 2007 at this point. So it will be sometime tomorrow when all the blogs appear. Stay tuned. Right now it is just a lot of waiting.

Comments

[ModuLIZER]

God news... nice work guys!!!

[mechbeast01]

Hey since the launch of Halo 3 and now the updates of the servers and all has caused my playing streak to go back by more than 12 days. I'm just wondering why all this would cause that to happen and to make it go that far back.

[Trapper Markelz]

@mechbeast01 - The stats are still catching up... so it shouldn't be off by 12 days. What should it be?

[mechbeast01]

Somehow I guess my days under bragging rights weren't coming up right. They were showing 39 days as my longest streak. Now it's showing 52 so I guess it has since then fixed itself.

[chand]

Hey

Finally got thier

Love how fast site load's now

[Weblaus]

Something's not right with the blog entry for 9/26.. it's saying I won some achievements on Halo 3, so that's right, but the gamerscore for them isn't added - i.e. the entry still lists the same score as 9/25 which obviously must be wrong.

[SNAPCRAKLEP0P]

Since the Update i am now missing some games in my list most notably Rainbow Six Vegas with my 100 day badge

[ZeroDesolation]

It said I didn't play at all on the 26that's bull. I lost my streak and I shouldn't have. I've played every day and I want my 20+ day streak back.

I'll probably be emailing ya monday for sure on this since streaks are probably bottom of the list right now.

[Trapper Markelz]

Zero did you take any time to read the homepage updates? No one lost any streaks... we have them turned off because of the Xbox live outage on 9/25 which prevented us from getting any game data for most people... if it looks like it, it is because the data isn't synced. You still have your streak of 19 days and you will see that when the data finishes syncing up.

[ZeroDesolation]

Yea, I read the updates but everyone else I looked at showed what they played on that day and mine didn't show I played anything even though I did. That's why I was worried on the streak badge.

[Emowii]

Well, here is my blog entry for September 26th.

"I think there was a problem with Xbox Live... Halo 3 launched and crushed my world... so I wouldn't exactly trust my memory... In fact we are just going to pretend this day never happened."

I know everyone is saying it's still got to sync up, but I'm on the new server, and my blog entry for the 26th seems to suggest that I'm never going to see the true entry for that date. Not sure what to think of that. Hopefully it gets changed or something.

[Trapper Markelz]

Emowii - Yes you aren't going to see that entry. Xbox.com was down.... as in off line... for almost 20 hours... which means that none of your game playing was available to us. We can't write a blog entry if MS doesn't have a record they can give us... and they didn't.

[Emowii]

Trapper Markelz

You seemed to mention over and over again on the front page updates that were weren't going to miss any blog updates, and that they would run for each day. So you're saying that the 26th will always be a nothing day for everyone? Does that end streaks and what not for people?

Lately, you guys keep saying in these updates that things are running and blogs will happen. How long are we supposed to wait until we actually think there is a problem? You said things were running yesterday, but here we are 24 hours later and nothing has changed on my blog.

[Trapper Markelz]

If there is a problem We will update the blog... right now... we are still waiting for the data to catch up... it is 3 million + rows that it has to copy over.. it takes a long time... and in almost all cases longer than we had thought... because we have never done anything like this before... And no... your streak is not gone. We talked about this here:

http://blog.360voice.com/2007/09/xboxcom-outage-.html