DSLReports down

Post by **Philip** » Wed Apr 18, 2012 4:26 pm

DSLReports has been down since Monday afternoon, caused by what started as a Net Access Corporation power outage, I'm sorry for Justin and hope they bring it back up with no further issues.

dslreports.com is offline
-------------------------

Update Wed Apr 18 09:54:54 EDT 2012

Well, the good news is we have a plan & I hope things will remain offline for no
longer than the balance of this week. Unfortunately, equipment must be sourced
installed, configured, and so on. Lacking a good explanation of what happened to
"redundant" disk storage, I am reluctantly faced with just replacing it with more
expensive gear. No certainty on times, yet. I will post a link to the gory details soon.

Update Tue Apr 17 04:14:29 EDT 2012

Things have turned out to be more complicated. Although no hardware was fried,
our large storage array has decided, probably as a result of going dark in mid write,
that our two sql servers can no longer have access to one of the small,
but important, partitions that makes up the larger SQL data area. All data
is present, but it refuses to offer it.
At this point all courses of action involve long delays. When I know more I'll post
-Justin

Update 15:35 EDT
Our data center experienced a power outage which affected our entire system. We are in the process of recovering everything
and turning a few cranky servers into happy ones.

Apologies for the inconvenience, we expect to be up in a few hours

Update 16:18 EDT
-------------------------
The power outage affected the entire data center and we are still awaiting our turn to power up some machines.
We will keep this page updated.

Update 19:20 EDT
-------------------------
Unfortunately, a database server doesn't take kindly to being taken down without proper shutdown procedures and the
restoration efforts are still under way. No ETA at this time.

http://www.dslreports.com/front/shutdown.html

Ken · Post by **Ken** » Wed Apr 18, 2012 6:02 pm

Totally helpless in that type of situation. You can never have enough back up power... Hopefully he will get it back up soon with out major data loss.

nightowl · Post by **nightowl** » Wed Apr 18, 2012 6:03 pm

hope speedguide's server is not at the same data center....we have good UPS's on our speedguide servers right?????

Ken · Post by **Ken** » Wed Apr 18, 2012 6:13 pm

nightowl wrote:hope speedguide's server is not at the same data center....we have good UPS's on our speedguide servers right?????

SG servers are far away from there, however how long battery back ups can keep you on, depends... Keep in mind you would have no idea how long the power would be out...

It almost sounds like either the UPS was not there or not enough for the loads, or at minimum, not fully functioning, however it would either have needed to be configured to shut down automatically on an outage or someone would need to have been alert and manually shut it down...quickly...
Power loss is definitely a quick way to cause bad sectors on hdds...

idavid4attdsl · Post by **idavid4attdsl** » Wed Apr 18, 2012 8:29 pm

as far as at&t dsl customers in the direct threads over there, I can take them on and still work with them. Just tell them to drop me an email. The QR code to the left of my name as a link to the contact page. They may not be back up before friday.

Post by **Philip** » Wed Apr 18, 2012 8:29 pm

The datacenter has diesel generators that are ready to kick in as necessary in case of power outages. We do have backups and should be able to bring the site back up fast in case of hardware failure, we have redundancy on most parts as feasible..

That said, as I was typing this one of the core routers in the data center went offline, their backup router did not pickup the load as it should, and it took them close to an hour to bring the network back up.. :/ Still, the colo facility, the network and the quality of service has been very good.

Here are some images of their cooling and backup power generators:
http://www.gorack.net/Colocation/facility.php

nightowl · Post by **nightowl** » Wed Apr 18, 2012 9:14 pm

Philip wrote:The datacenter has diesel generators that are ready to kick in as necessary in case of power outages. We do have backups and should be able to bring the site back up fast in case of hardware failure, we have redundancy on most parts as feasible..

That said, as I was typing this one of the core routers in the data center went offline, their backup router did not pickup the load as it should, and it took them close to an hour to bring the network back up.. :/ Still, the colo facility, the network and the quality of service has been very good.

Here are some images of their cooling and backup power generators:
http://www.gorack.net/Colocation/facility.php

does speedguide have an off site backup?

CableDude · Post by **CableDude** » Wed Apr 18, 2012 9:16 pm

People still visit that site?

ace · Post by **ace** » Wed Apr 18, 2012 9:34 pm

CableDude wrote:People still visit that site?

Everyday. Way busier with alot more topics. I still visit SG everyday but its just really slow here.

Post by **Philip** » Thu Apr 19, 2012 12:18 am

nightowl wrote:does speedguide have an off site backup?

Yes, just not always current because of database sizes.

CableDude · Post by **CableDude** » Thu Apr 19, 2012 6:20 pm

ace wrote:Everyday. Way busier with alot more topics. I still visit SG everyday but its just really slow here.

Maybe I should drop by.

RoundEye · Post by **RoundEye** » Thu Apr 19, 2012 7:13 pm

That’s a serious facility.

idavid4attdsl · Post by **idavid4attdsl** » Thu Apr 19, 2012 7:30 pm

CableDude wrote:People still visit that site?

I still do for att dsl and uverse customer issues.

hayc59 · Post by **hayc59** » Fri Apr 20, 2012 7:19 pm

CableDude wrote:People still visit that site?

Oh yea!! not a bad forum

Shinobi · Post by **Shinobi** » Sat Apr 21, 2012 11:25 pm

Philip wrote:The datacenter has diesel generators that are ready to kick in as necessary in case of power outages.

The sign of a good B.C.P. :-)

thechemgeek · Post by **thechemgeek** » Sun Apr 22, 2012 12:54 am