Go Back   Winnipesaukee Forum > Support Forum > Site Support Forum
Home Forums Gallery Webcams Blogs YouTube Channel Classifieds Calendar Register FAQDonate Members List Search Today's Posts Mark Forums Read

Reply
 
Thread Tools Display Modes
Old 01-02-2007, 03:05 PM   #1
webmaster
Moderator
 
webmaster's Avatar
 
Join Date: Jun 2002
Posts: 2,427
Blog Entries: 1
Thanks: 437
Thanked 3,697 Times in 820 Posts
Default Tuesday's Site Outage

What a way to start the new year! At 8:12 am today the data center that hosts my server had a major failure that caused 6 hours and 40 minutes of downtime. Sorry for the inconvenience.

For all you techies out there here is the most recent status report on the failure:

Quote:
The WALTHAM, MA facility is experiencing a complete outage. We have updated information confirming that both the primary and secondary redundant fiber paths went down. The primary fiber uplink went down earlier this weekend, and before engineers could restore it to full operation, the secondary fiber link went down this morning at about 8:12 AM EST. Savvis does have engineers working on the issue with the highest priority. This is affecting ALL customers of Savvis in their Boston 1 and Boston 2 datacenters.
The site does still seem to be a little slow but that should clear as the data center is brought back to full capacity.

Thanks for your patience.
webmaster is offline   Reply With Quote
Old 01-02-2007, 03:34 PM   #2
Skip
Senior Member
 
Skip's Avatar
 
Join Date: Apr 2004
Location: Dover, NH
Posts: 1,615
Thanks: 256
Thanked 514 Times in 182 Posts
Default Thanks Don....

....for keeping us posted.

I noticed a few of my other common sites were experiencing difficulties also...the biggie being the Manchester Union Leader site....now you have me curious as to if they are utilizing the same site!

Thanks again and Happy New Year,

Skip

EDIT: Wow, just checked in at the Union Leader and at the top of the fold had the same explanation as yours! Wow, you really do run with the big fish now!!!
Skip is offline   Reply With Quote
Old 01-02-2007, 03:52 PM   #3
webmaster
Moderator
 
webmaster's Avatar
 
Join Date: Jun 2002
Posts: 2,427
Blog Entries: 1
Thanks: 437
Thanked 3,697 Times in 820 Posts
Default

Quote:
Originally Posted by Skip
Wow, just checked in at the Union Leader and at the top of the fold had the same explanation as yours! Wow, you really do run with the big fish now!!!
Savvis is a huge data center and this outage must have affected thousands of customers including banks, government and media sites, like the Union Leader.

While the outage was a minor inconvenience for us, the big fish must have been going nuts. Fortunately a simultaneous failure of redundant systems is very rare and unusual.
webmaster is offline   Reply With Quote
Old 01-02-2007, 04:04 PM   #4
RI Swamp Yankee
Senior Member
 
RI Swamp Yankee's Avatar
 
Join Date: Jul 2002
Location: North Kingstown RI
Posts: 688
Thanks: 143
Thanked 83 Times in 55 Posts
Default

I was a bit concerned when I saw even the DNS servers were down and could not be reached. Glad it is back up. Seems to be at full speed.
__________________
Gene ~ aka "another RI Swamp Yankee"
RI Swamp Yankee is offline   Reply With Quote
Old 01-02-2007, 04:37 PM   #5
ghfromaltonbay
Senior Member
 
Join Date: Mar 2004
Location: Clifton, NJ, Alton Bay
Posts: 818
Thanks: 243
Thanked 223 Times in 129 Posts
Default Affected Metrocast

I sent an email last night to DRH, Pepper, IG and Island Sib. This morning when I logged on I had an email message telling me that the delivery was delayed to DRH and Pepper. Both have cable modems from Metrocast. Then I couldn't access the Union Leader or this website. I did manage to pull up Foster's Daily Democrat which must go thru a different server in New England. Now as the Pink Panther would say..."the mystery is solv-ed."
ghfromaltonbay is offline   Reply With Quote
Sponsored Links
Old 01-02-2007, 04:41 PM   #6
Steve
Senior Member
 
Join Date: Apr 2004
Location: Southern NH
Posts: 109
Thanks: 0
Thanked 0 Times in 0 Posts
Default

Quote:
Originally Posted by webmaster
Fortunately a simultaneous failure of redundant systems is very rare and unusual.
You know Don, I wonder about this though. The issues had to do with routers and circuits today. Seems to me that they weren't very redundant. I'll be curious to know more, but it doesn't seem like everything failed over like it should, which makes me question their policies and procedures.
Steve is offline   Reply With Quote
Old 01-02-2007, 05:31 PM   #7
Waterbaby
Senior Member
 
Waterbaby's Avatar
 
Join Date: Apr 2004
Location: Kensington, NH and Paugus Bay Marina
Posts: 656
Thanks: 323
Thanked 17 Times in 13 Posts
Default Hmmmmm.............

another site i frequent, nascar.com, also had the same problem..... it was down from about 8 this morning until about 2-something this afternoon........ almost sounds like what we all expected for Y2K! delayed reaction, possibly?
__________________
On the boat is always waterfront!
Waterbaby is offline   Reply With Quote
Old 01-02-2007, 06:11 PM   #8
Weirs guy
Senior Member
 
Weirs guy's Avatar
 
Join Date: Feb 2006
Location: Weirs Beach, NH
Posts: 1,067
Thanks: 0
Thanked 1 Time in 1 Post
Default

Oh, the site was down? Thank god, I thought I was still drunk from New Years!!
__________________
Is it bikeweek yet?

Now?
Weirs guy is offline   Reply With Quote
Old 01-02-2007, 09:39 PM   #9
webmaster
Moderator
 
webmaster's Avatar
 
Join Date: Jun 2002
Posts: 2,427
Blog Entries: 1
Thanks: 437
Thanked 3,697 Times in 820 Posts
Default

Quote:
Originally Posted by Steve
You know Don, I wonder about this though. The issues had to do with routers and circuits today. Seems to me that they weren't very redundant. I'll be curious to know more, but it doesn't seem like everything failed over like it should, which makes me question their policies and procedures.
They certainly could have used triple redundancy in this case but you make a good point Steve. If I was running a critical business server at SAVVIS I think I'd be asking some tough questions and considering my alternatives. For the travel & tourism sites that run on my server the downtime was more of an inconvenience than a crisis and I wouldn't move the server unless this becomes a regular event.

I'm sure that in response to concerned (irate?) customers they will soon have a plan to make sure this can never happen again.
webmaster is offline   Reply With Quote
Old 01-02-2007, 10:43 PM   #10
RI Swamp Yankee
Senior Member
 
RI Swamp Yankee's Avatar
 
Join Date: Jul 2002
Location: North Kingstown RI
Posts: 688
Thanks: 143
Thanked 83 Times in 55 Posts
Default

It seems the problem wasn't directly with any servers or routers at the SAVIS location, it was the fiber links from them to the world. They need to take a hard look at their transport providers.
__________________
Gene ~ aka "another RI Swamp Yankee"
RI Swamp Yankee is offline   Reply With Quote
Old 01-03-2007, 12:08 AM   #11
Steve
Senior Member
 
Join Date: Apr 2004
Location: Southern NH
Posts: 109
Thanks: 0
Thanked 0 Times in 0 Posts
Default

I believe they did have internal router issues as well. Not 100% sure though if it was related to failover, or a perfect storm of events.
Steve is offline   Reply With Quote
Old 01-03-2007, 06:55 AM   #12
Blue Thunder
Senior Member
 
Blue Thunder's Avatar
 
Join Date: Apr 2004
Location: Eastern MA & Frye Island/Sebago Lake, Maine
Posts: 935
Thanks: 247
Thanked 323 Times in 148 Posts
Default Could this be a larger problem?

I have heard and read about several fiber optic cable failures here in Eastern Mass that started last Friday. The one in Ipswich MA has been worked on 24/7 since last Friday. Verizon is in the process of replacing several thousand feet of Fiber Optic cable. I read about some other towns with similar issues over the weekend. Is there any way they could be related, ie; over loading of existing circuits when they transfer the circuits off to another line while they replace the failed one??? Seems a little coincidental to me.

Blue Thunder
__________________
" Live for today because yesterday is gone and tomorrow may never come"
Blue Thunder is offline   Reply With Quote
Old 01-03-2007, 12:21 PM   #13
webmaster
Moderator
 
webmaster's Avatar
 
Join Date: Jun 2002
Posts: 2,427
Blog Entries: 1
Thanks: 437
Thanked 3,697 Times in 820 Posts
Default the magnitude of yesterday's outage

Lycos was down too: http://computerworld.com/action/arti...intsrc=hm_list
webmaster is offline   Reply With Quote
Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is On

Forum Jump


All times are GMT -5. The time now is 02:50 PM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.

This page was generated in 0.26757 seconds