Can't access our company's lighthouse

Mark Rose's Avatar

Mark Rose

10 Jan, 2012 10:59 PM

Browser just times out. It's been this way for 20 minutes now.

Is this related to the current AWS issues?

  1. 1 Posted by Tim Clark on 10 Jan, 2012 11:01 PM

    Tim Clark's Avatar

    We had some delays, but they should be mostly resolved now. I'm able to access Lighthouse fine currently. We're still looking into the cause.

  2. 2 Posted by Matt on 10 Jan, 2012 11:13 PM

    Matt's Avatar

    We are having issues with ours too. Was down, then came back for a few minutes, now it's down again.


  3. 3 Posted by Mark Rose on 10 Jan, 2012 11:17 PM

    Mark Rose's Avatar

    I had a machine at Amazon reboot itself around the same time (onto new hardware) so I suspect it's AWS.

    I guess our company data aren't distributed across machines/availability zones...

  4. 4 Posted by Olga on 10 Jan, 2012 11:19 PM

    Olga's Avatar

    It's down for us.

  5. 5 Posted by Tim Clark on 10 Jan, 2012 11:22 PM

    Tim Clark's Avatar

    While we're investigating, the site slowed down again and is having problems. It's probably the same cause, once we can isolate it.

  6. 6 Posted by Matt on 10 Jan, 2012 11:25 PM

    Matt's Avatar

    Someone burnt a chicken-pot-pie on the 16th floor of our building and the fire alarm went off. Maybe the two are related. Just kidding. ;) I'm sure the ENTP team is hard at work resolving it.

  7. 7 Posted by Mark Rose on 10 Jan, 2012 11:44 PM

    Mark Rose's Avatar

    Been an hour now...

  8. 8 Posted by Tim Clark on 10 Jan, 2012 11:46 PM

    Tim Clark's Avatar

    We got all hands on deck looking into the source of the problem, and our hosting provider as well.

  9. 9 Posted by Josie on 11 Jan, 2012 12:20 AM

    Josie's Avatar

    I'm having the same issues as well. Been happening all day.

  10. 10 Posted by Chris Beck on 11 Jan, 2012 12:31 AM

    Chris Beck's Avatar

    Same problem just seconds ago. Keep fighting the good fight!

  11. 11 Posted by Tim Clark on 11 Jan, 2012 01:04 AM

    Tim Clark's Avatar

    The site should be back up and functioning normally now.

    The initial cause of the problem about 2.5 hours ago was a set of processes that got out of hand, and which caused AWS to autokill those servers. At this point new servers were added to the pool, but due to a bug in the (slightly outdated) memcached gem, memcached did not handle servers being removed and replaced dynamically, which led to the site persisting in being slow and unresponsive. We are upgrading our memcached gem to a version that does support this particular type of failover and replacement.

  12. Tim Clark closed this discussion on 11 Jan, 2012 01:04 AM.

Discussions are closed to public comments.
If you need help with Lighthouse please start a new discussion.

Keyboard shortcuts


? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac