Safarri - (Resolved) ATI malfunction!

2 weeks - 17,125v

Posted 2009/05/19 - 14:35 GMT

Sorry about that!

Something crashed, and MySql corrupted the index file on my comments table.

I repaired the file, and everything should be back to a fully operational state.

No data appears to have been lost (there are currently 24,129 comments in the table), and unless the issue reoccurs - I think we will be able to classify it as nothing more than an inconvenience.

-Aubrey

Dray (Deathray)

3 days - 9,465v

Posted 2009/05/19 - 14:51 GMT

phew!

Flynn Clubbaire (mod) (flyclub)

4 weeks - 32,767v

Posted 2009/05/19 - 16:07 GMT

Yay!

Dray (Deathray)

3 days - 9,465v

Posted 2009/05/19 - 19:10 GMT

I was afraid it might be the end of mars explorer!

I was so scared!

OMG AN APOCOLIPSE ON MARS!!!! AHHHHHH! :P

roflcoptor

1 hour - 38v

Posted 2009/05/19 - 21:59 GMT

hey i USED 2 live in LA 2! used 2, not any more*

Aubrey (admin)

2 weeks - 17,125v

Posted 2009/05/20 - 14:09 GMT

Just for fun, following is an email I recently received from my hosting provider.

-Aubrey

Firstly, please accept our sincere apologies for the events of the past twenty four hours - we have identified many issues with redundant systems which deserve an explanation and immediate action on our part. This incident represents a "perfect storm" of failures in our electrical failover systems and, as with any incident of this scope, we're taking every opportunity to improve our processes to prevent its recurrence.

To summarize the incidents of the past twenty four hours:

5/18/2009 15:10 PST - A measurable loss in voltage from our datacenter's main feeds at Seattle City Light was recorded. This loss in voltage triggered our UPS failovers and all hardware nodes and routers began running on UPS failover power.

5/18/2009 16:05 PST - UPS systems' battery reserves began to deplete and hardware nodes began to shut down. We have confirmed that the UPS systems were working properly - the loss in voltage tripped their circuits and forced all connections with our datacenter's feeds down. This action, unfortunately, prevented the UPS systems from recharging their batteries from available power following the brownout.

5/18/2009 19:00 PST - We did not have sufficient evidence to believe that the issue at Seattle City Light had been resolved. Our UPS service company and a technician arrived, an electrician from Titan Electric was dispatched, and our datacenter technicians switched from the Seattle City Light feed to diesel backup generator backups.

Upon confirming that all hardware nodes had sufficient power, our system administration team began bringing hardware nodes and key routing hardware back up (total potential outage time for all services at this point: three hours).

Our internal services (including VPSLink.com, our ticketing system, our billing system, the My.VPSLink.com control center, and the VPSLink forums) were unavailable for the duration of the outage. We were not able to send broadcasts or post to the forums regarding the cause of and action being taken on this issue.

5/18/2009 20:00 PST - All hardware nodes were restored to service and began fsck and VPS account procedures. Key routing hardware was restored to service.

Internal systems remained offline while our system administrators and developers worked to restore service to nodes hosting internal databases.

5/18/2009 21:30 PST - Our ticketing system was restored to service and broadcast replies were sent to all tickets received encouraging subscribers to submit a ticket if their service remained offline. Several hardware nodes were still in the process of completing fsck procedures and service had not yet been restored to all individual VPS accounts.

5/19/2009 01:30 PST - All internal services were restored to service and a notification was posted to the VPSLink Forums approximately eight hours after the incident began.

We recognize that hours of downtime without explanation or a way to establish contact with support and find out what has happened is unacceptable, and we want to ensure that everyone affected by this outage has the opportunity to claim service credits in accordance with our Service Level Agreement.

If you have not already claimed service credits for this incident, please submit a ticket to the VPSLink Billing department.

We are presently investigating appropriate courses of action to prevent anything like this from occurring again at our datacenter and we will post updates on progress at the VPSLink Forums:

VPSLink Datacenter Incident 5/18/2009 approx 18:30 PST - ongoing
URL: http://forums.vpslink.com/system-network-status/9231-vpslink-datacenter-incident-5-18-2009-approx-18-30-pst-ongoing.html

Finally, we have had the opportunity to review the very reasonable requests submitted by our subscribers in response to this incident and the communication blackout which occurred will not occur again. By June 1st, 2009 we will have a redundant page in place to address any issues which affect the VPSLink.com site and ticketing system by providing a direct e-mail address to contact our staff.

Flynn Clubbaire (mod) (flyclub)

4 weeks - 32,767v

Posted 2009/05/20 - 15:43 GMT

Murphy's law.... What can go wrong will go wrong... Sadly, it is 100% true...

Spiffy (pb&j)

1 week - 17,669v

Posted 2009/05/20 - 15:57 GMT

Whew, Clubby, 3,941 messages?

Flynn Clubbaire (mod) (flyclub)

4 weeks - 32,767v

Posted 2009/05/20 - 16:17 GMT

Actually it is now 3940 messages....