23 replies to this topic

[Petsku]

Fellow eRepublikans,

We’re sorry for the delay in posting this, but we wanted to make sure all the compensations were given accordingly. We will use this chance to explain what went wrong in the tournament, and why our reactions often seemed so slow. We’ll also talk about the compensation rewards and the issues we faced when awarding them.


Background

Over the last couple of months, we’ve done a progressive migration of all our servers from our historical provider to Amazon cloud, in order to increase the flexibility and reliability of eRepublik.com.

As the Epic Warfare tournament was being scheduled, we were aware that our current setup could not accommodate the epic load, so we migrated the critical part of the servers (database servers, cache servers and web nodes) on 10th of October with 2 hours of cumulative downtime.

By the start of the tournament, the majority of eRepublik was running on the AWS (Amazon Web Services), the avatars being an exception.


Epic Tuesday

We expected a similar load distribution over the tournament week like we had last year: load increasing during the week, as more players hit higher recovery rates. This time a vast majority of players were prepared to hit from the first minute. A significant part of them were also using client side scripts, causing a surge of load at day change.

We increased the instances to accommodate the extra load, but soon enough we hit several bottlenecks on the master database, caused by various parts of legacy code. Mainly various tables were locked by queries that were resource intensive, or should have used the slave databases. We solved the bottlenecks one by one, as we faced them.

It is worth mentioning that a critical service we are using to identified application performance issues, Newrelic.com, experienced an outage of its own on Tuesday, making our task harder.


Failing Sunday

The servers performed within parameters for the rest of the week, until Sunday. During the epic battle Serbia-Chile, the site, being under significant abuse (scripts, COs), became unusable, as it was again hitting another database bottleneck. Unfortunately, what should have taken 20-30 minutes to fix, became a 3 hour nightmare for the players. The late response was due to a mix of factors:

1) The automatic monitoring alerts were not triggered in due time, as they were not tuned for the AWS environment, but for the servers and setup we had at our previous provider.

2) A failure in the team communication; one of our QA engineers left the team last Friday and we failed to make sure that his weekend shift was covered by one of his QA colleagues. So basically there wasn’t anyone from QA “on call” during Sunday and that prevented us from catching the problem sooner.

All in all, it was a poor performance on our part and for that we would like to apologize.


Hiccup Monday

With load increasing on the site and the tournament closing, half an hour of site slowdown was experienced on Monday because of yet another set of queries.


Rewards Tuesday

As you should know by now, we decided to compensate all the problems we experienced during the tournament. The next milestone was activated for all participants, and the Energy Centers were extended with 2 days.

Unfortunately, we faced a new set of problems with the Energy Centers. It was basically a human mistake, caused by the rush of delivering the compensations. Everything should be solved by now, but if you still feel that something is wrong, don’t hesitate to send us a ticket by using the Contact page below!


Avatar Thursday

We completed the migration process to the Amazon Cloud today, finishing the move of the avatars. Changing avatars is now possible again and the players now have a better “Edit profile” section.

We still have to tweak some things, but we are confident that we’re now able to give you a better service. The kind of service the amazing and passionate community of eRepublik deserves.


The Future

We will continue to monitor and clean the application of legacy code (some of it dates back to 2008) that is prone to failure. We will also focus on our monitoring and alerting procedures, in order to improve escalation and response times.

We’re also considering some changes on the military and marketplace modules, in order to minimize the impact of the scripts in use today. We do plan to include the fight script functionality in the site, but we’re not prepared to go into details right now.


Summary

To sum it up, we’re extremely sorry for all the issues players had to face during the tournament. We were not able to deliver the experience we wanted to, and we’ll do our best to learn from these difficult times. We hope that this message gave you the much needed insight on the issues, despite it being rather technical.

Words cannot describe how much we appreciate all the passion and support you guys are showing every day. The community has kept the game going for so many years, and the whole eRepublik Team is confident that together we can make the New World a better place.

o7

[goraz]

o7

great tournament

[Aisllin]

You move servers a couple of weeks before a major tournament.

 

 

But thanks for the explanation.

[smee again]

Good explanation and shedding some light on this. Much appreciated.

[Valyrian Dragonlord]

After uploading, avatar is not showing up.

[chriswen]

thanks for the explanation.

Edited by chriswen, 30 October 2014 - 07:26 PM.

[Ivan je Car 2]

epic fail tournament

[razvyman]

Hi,

 

Thanks for the explanation, but about a year ago an eRepublik player published an article where he simply look up in the source code of the site. There he sum up some of those legacy codes that you are talking and how removing the could load the page pretty much faster. Back then I did not have so much experience with html, java and so on, but I did understood most of he said and also those codes.

 

I`ll look up for that article and if I find it, I`ll give you a later edit, but you can search it, too. (I guess you can find it easier)

 

Have a nice day/evening!

Edited by razvyman, 30 October 2014 - 07:28 PM.

[Dimow Brando]

Not a chance anyone published the back-end code of the game.

 

Thanks for the explanation! Too bad elbandido left...

[razvyman]

Not a chance anyone published the back-end code of the game.

 

Thanks for the explanation! Too bad elbandido left...

 

Site code. Simply press an F12 on eRepublik site.

[Dimow Brando]

Site code. Simply press an F12 on eRepublik site.

That's simple HTML and you can't find out the code they are taking about.

[BloodTrail]

Thanks for the explanation of the situation. I will forgive you for all the problems, however I still question how you dealt with the #Rebellion2014 movement. At least you shouldn't keep quiet about it and instead explain your actions behind the massive deletion and warning-spam given to those involved. Don't mistake me here, after reading this I do have a few ideas as of why you did things the way you did, but still I feel you should offer an explanation and apology.

 

That being said; I too have problems with the avatar-changes not showing properly for me and I've tried both refresh and cache-clearing. Also the name-change feature doesn't work either. I don't feel like paying 55 gold in total, so I haven't tried changing my name randomly, but from my research there shouldn't be any problem with the name (and similar versions of it) that I wish to change to, but still it doesn't work.

[Garmr]

What would solve user-end scripts and decrease server load is really really simple.

 

Add a Fight x5 button. Maybe even a Fight x10 button.

[hollenboer]

What would solve user-end scripts and decrease server load is really really simple.

 

Add a Fight x5 button. Maybe even a Fight x10 button.

Good idea !!

[James Banana]

That's simple HTML and you can't find out the code they are taking about.

Well, last time i checked that 'code' (technically markup) was pretty bad written, but that was a while ago. I actually believe the frontpage was built using WYSIWYG automated tools back then. Seing such bad HTML, having really bad server side code being left in places does not really surprise.

 

And as some said, the simplest way to reduce user side script is to take ideas from them and improve the game accordingly. As long as it can be done better with scripting, it will be done.

[NO CARRIER]

 

 

We will continue to monitor and clean the application of legacy code (some of it dates back to 2008) that is prone to failure.

 

Chinese wise man says: The old code is like the old cars - you never know when it will fail.

 

 

Is it so hard to make a button to turn off these notifications?

[Dio Aldo]

I forgive you.

[Backo BG]

Now that's a good explanation. Not just "We had maintenance and apologize for the inconvenience this might have caused." You say what went wrong, why did it go wrong and why did it take so much time to be fixed. Keep improving the game... after fixing the new profile edit page.

[chriswen]

What would solve user-end scripts and decrease server load is really really simple.

 

Add a Fight x5 button. Maybe even a Fight x10 button.

 

Yup, its a trade off for the devs.  If they don't wish to implement this than they they'll have to pay up for more servers.

[Klounas]

wow, its like the first time they explained anything, good job, keep it up!