buy.at links down? (Michael Jackson)

**hobbsy** · 12-03-09

Hi

My links have been intermittently broken this morning for GetMeIn (Michael Jackson) on buy.at

I'm getting this error:

Code:

We were unable to complete the Database request at this moment - please contact webmaster Session halted.

The URL is

Code:

http://ticketsales.at/xxxxxxxxx?LID=xxxxxxxxx&DURL=http://www.getmein.com/rock-and-pop/michael-jackson-2-tickets.html

**Sarah Johnson** · 12-03-09

Hi Hobbsy

Can you PM me and I'll give you a call?

Will ask the client to look into ASAP

cheers

**JohnFraser** · 12-03-09

We've traced the problem to a rogue webserver in the tracking cluster. It was unstable for a few minutes before our load balancers automatically pulled it from the pool.

Ours Ops team are doing some detective work to track down the exact cause, but we're not seeing anything similar on any of the other servers, either now or in the logs.

These things happen (very) occasionally. Fortunately, we have a very professional team of System Admins on call 24/7, both monitoring the servers, and setting up the systems to automatically repair themselves at any hint of an outage.

Thanks,

**hobbsy** · 13-03-09

I've seen buy.at errors again on my links (Ticketmaster etc) for Michael Jackson tickets

Same error message as yesterday

I understand demand is unprecedented for MJ tickets, but I am losing sales here due to the database fail errors

**JohnFraser** · 13-03-09

Hi Hobbsy,

Sorry for the later reply. The Ops team were once again right on top of it - it's just taken me a while to get some time with a browser.

It seems like a lot of MIchael Jackson fans have been getting up very early and generating unprecedented amounts of traffic. Traffic volumes at 7am were approx. 50% higher than what we see across the whole network at normal peak hours.

Our systems have been tested and can comfortably handle more than 10x the normal peak load at any point in time . Unfortunately, since 7am is usually so quiet, a lot of the daily housekeeping batches are set to run at that time. It was an interaction with one of these that was causing the server instability.

We immediately postponed these batches for today, and are looking to isolate them in the future. The period of instability was again very short. Now we have identified the cause, it shouldn't happen again.

I'm going to have to get the beers in for the ops team tonight. They get grumpy if they get paged too early in the morning. They've done a great job though identifying where this has come from.

Thanks,