tag:help.lighthouseapp.com,2008-09-20:/discussions/problems/3109-cant-access-our-companys-lighthouseLighthouse: Discussion 2012-01-11T01:04:28Ztag:help.lighthouseapp.com,2008-09-20:Comment/128126002012-01-10T23:01:42Z2012-01-10T23:01:42ZCan't access our company's lighthouse<div><p>We had some delays, but they should be mostly resolved now. I'm
able to access Lighthouse fine currently. We're still looking into
the cause.</p></div>Tim Clarktag:help.lighthouseapp.com,2008-09-20:Comment/128126002012-01-10T23:13:59Z2012-01-10T23:14:00ZCan't access our company's lighthouse<div><p>We are having issues with ours too. Was down, then came back for
a few minutes, now it's down again.</p>
<p>-Matt</p></div>Matttag:help.lighthouseapp.com,2008-09-20:Comment/128126002012-01-10T23:17:12Z2012-01-10T23:17:23ZCan't access our company's lighthouse<div><p>I had a machine at Amazon reboot itself around the same time
(onto new hardware) so I suspect it's AWS.</p>
<p>I guess our company data aren't distributed across
machines/availability zones...</p></div>Mark Rosetag:help.lighthouseapp.com,2008-09-20:Comment/128126002012-01-10T23:19:16Z2012-01-10T23:19:16ZCan't access our company's lighthouse<div><p>It's down for us.</p></div>Olgatag:help.lighthouseapp.com,2008-09-20:Comment/128126002012-01-10T23:22:43Z2012-01-10T23:22:43ZCan't access our company's lighthouse<div><p>While we're investigating, the site slowed down again and is
having problems. It's probably the same cause, once we can isolate
it.</p></div>Tim Clarktag:help.lighthouseapp.com,2008-09-20:Comment/128126002012-01-10T23:25:03Z2012-01-10T23:25:03ZCan't access our company's lighthouse<div><p>Someone burnt a chicken-pot-pie on the 16th floor of our
building and the fire alarm went off. Maybe the two are related.
Just kidding. ;) I'm sure the ENTP team is hard at work resolving
it.</p></div>Matttag:help.lighthouseapp.com,2008-09-20:Comment/128126002012-01-10T23:44:18Z2012-01-10T23:44:18ZCan't access our company's lighthouse<div><p>Been an hour now...</p></div>Mark Rosetag:help.lighthouseapp.com,2008-09-20:Comment/128126002012-01-10T23:46:18Z2012-01-10T23:46:18ZCan't access our company's lighthouse<div><p>We got all hands on deck looking into the source of the problem,
and our hosting provider as well.</p></div>Tim Clarktag:help.lighthouseapp.com,2008-09-20:Comment/128126002012-01-11T00:20:19Z2012-01-11T00:20:20ZCan't access our company's lighthouse<div><p>I'm having the same issues as well. Been happening all day.</p></div>Josietag:help.lighthouseapp.com,2008-09-20:Comment/128126002012-01-11T00:31:39Z2012-01-11T00:31:40ZCan't access our company's lighthouse<div><p>Same problem just seconds ago. Keep fighting the good fight!</p></div>Chris Becktag:help.lighthouseapp.com,2008-09-20:Comment/128126002012-01-11T01:04:26Z2012-01-11T01:04:26ZCan't access our company's lighthouse<div><p>The site should be back up and functioning normally now.</p>
<p>The initial cause of the problem about 2.5 hours ago was a set
of processes that got out of hand, and which caused AWS to autokill
those servers. At this point new servers were added to the pool,
but due to a bug in the (slightly outdated) memcached gem,
memcached did not handle servers being removed and replaced
dynamically, which led to the site persisting in being slow and
unresponsive. We are upgrading our memcached gem to a version that
does support this particular type of failover and replacement.</p></div>Tim Clark