Proxy issue

lighthouse270608's Avatar

lighthouse270608

08 Mar, 2009 07:29 PM

I'm behind squid proxy. Updating profile or submitting the ticket takes ages to complete and finally fail (due to timeout). When I switch off the proxy I can work normally.

  1. 1 Posted by Will Duncan on 09 Mar, 2009 10:25 PM

    Will Duncan's Avatar

    I'm not fully familiar with a squid proxy, or know the reason why it would cause Lighthouse to timeout.

    One of our coworkers is behind a squid proxy and has no issues with Lighthouse response times. Is it possible that your proxy might be misconfigured somehow?

    We'll continue to investigate on our end in the meantime.

  2. 2 Posted by Will Duncan on 09 Mar, 2009 10:27 PM

    Will Duncan's Avatar

    Out of curiosity, do you know what version the proxy is? My coworker told me they "have a fairly stock 2.6.stable14 install and LH is fine"

  3. 3 Posted by Courtenay on 09 Mar, 2009 10:49 PM

    Courtenay's Avatar

    We run these extensions on our squid

    extension_methods REPORT MERGE MKACTIVITY CHECKOUT

    It works fine with Lighthouse (we have 5+ users using it)

  4. 4 Posted by lighthouse27060... on 10 Mar, 2009 01:06 AM

    lighthouse270608's Avatar

    I'm using somewhat archaic squid version 2.5.14, but I've limited possibility to perform an upgrade.

    I've obtained dumps of traffic just before it hits the Internet with proxy turned on and off.

    Proxy turned on (the traffic is dumped after squid):

    POST /discussions/problems/561-proxy-issue/comments HTTP/1.0


    Host: help.lighthouseapp.com


    User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.7) Gecko/2009030422 Ubuntu/8.10 (intrepid) Firefox/3.0.7


    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8


    Accept-Language: en-us,en;q=0.5


    Accept-Encoding: gzip,deflate


    Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7


    Referer: http://help.lighthouseapp.com/discussions/problems/561-proxy-issue


    Cookie: tzoffset=-60; token=fd5d1723e61ffd4756b0052d7dff1bb32bb44a40; utma=98999303.3522514610823039000.1236540217.1236594159.1236646032.3; utmz=98999303.1236594159.2.2.utmcsr=rails.lighthouseapp.com|utmccn=(referral)|utmcmd=referral|utmcct=/projects/8994-ruby-on-rails/tickets/2174-no-way-to-flush-session-data-to-session-store-immidiately-enables-race-conditions-to-occur/versions/1; utma=265825682.1135076973.1236594131.1236594131.1236596276.2; utmz=265825682.1236596276.2.2.utmccn=(organic)|utmcsr=google|utmctr=lighthouse+ticket+formatting|utmcmd=organic; anon_token=60c35bb89822c352b74f167859e7ae128675d0dd; _tender_session=BAh7BzoMY3NyZl9pZCIlZTY0Njc0YTFiMzIwNjhiZjU2NDY4OGJmNDg5NTNkM2IiCmZsYXNoSUM6J0FjdGlvbkNvbnRyb2xsZXI6OkZsYXNoOjpGbGFzaEhhc2h7AAY6CkB1c2VkewA%3D--7ec2cd76f020816fa401b4f957a63d771d83710f; utmb=98999303.2.10.1236646032; utmc=98999303


    Content-Type: multipart/form-data; boundary=---------------------------790802227715125021037519974


    Content-Length: 788


    Via: 1.1 tproxy:3128 (squid/2.5.STABLE14)


    X-Forwarded-For: 192.168.96.77


    Cache-Control: max-age=259200


    Connection: keep-alive


    -----------------------------790802227715125021037519974


    Content-Disposition: form-data; name="authenticity_token"


    a2676d82a60db0db43fc62ec523cb4e4a5051081


    -----------------------------790802227715125021037519974


    Content-Disposition: form-data; name="comment[user_name]"


    -----------------------------790802227715125021037519974


    Content-Disposition: form-data; name="comment[user_email]"


    -----------------------------790802227715125021037519974


    Content-Disposition: form-data; name="comment[body]"


    I'm using (archaic) squid version 2.5.14-1.1


    -----------------------------790802227715125021037519974


    Content-Disposition: form-data; name="assets[]"; filename=""


    Content-Type: application/octet-stream


    -----------------------------790802227715125021037519974--

    Proxy turned off -- LH works good:

    POST /discussions/problems/561-proxy-issue/comments HTTP/1.1


    Host: help.lighthouseapp.com


    User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.7) Gecko/2009030422 Ubuntu/8.10 (intrepid) Firefox/3.0.7


    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8


    Accept-Language: en-us,en;q=0.5


    Accept-Encoding: gzip,deflate


    Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7


    Keep-Alive: 300


    Connection: keep-alive


    Referer: http://help.lighthouseapp.com/discussions/problems/561-proxy-issue


    Cookie: tzoffset=-60; token=fd5d1723e61ffd4756b0052d7dff1bb32bb44a40; utma=98999303.3522514610823039000.1236540217.1236594159.1236646032.3; utmz=98999303.1236594159.2.2.utmcsr=rails.lighthouseapp.com|utmccn=(referral)|utmcmd=referral|utmcct=/projects/8994-ruby-on-rails/tickets/2174-no-way-to-flush-session-data-to-session-store-immidiately-enables-race-conditions-to-occur/versions/1; utma=265825682.1135076973.1236594131.1236594131.1236596276.2; utmz=265825682.1236596276.2.2.utmccn=(organic)|utmcsr=google|utmctr=lighthouse+ticket+formatting|utmcmd=organic; anon_token=60c35bb89822c352b74f167859e7ae128675d0dd; _tender_session=BAh7BzoMY3NyZl9pZCIlZTY0Njc0YTFiMzIwNjhiZjU2NDY4OGJmNDg5NTNkM2IiCmZsYXNoSUM6J0FjdGlvbkNvbnRyb2xsZXI6OkZsYXNoOjpGbGFzaEhhc2h7AAY6CkB1c2VkewA%3D--7ec2cd76f020816fa401b4f957a63d771d83710f; utmb=98999303.3.10.1236646032; utmc=98999303


    Content-Type: multipart/form-data; boundary=---------------------------17013073289399684191498077790


    Content-Length: 858


    -----------------------------17013073289399684191498077790


    Content-Disposition: form-data; name="authenticity_token"


    a2676d82a60db0db43fc62ec523cb4e4a5051081


    -----------------------------17013073289399684191498077790


    Content-Disposition: form-data; name="comment[user_name]"


    -----------------------------17013073289399684191498077790


    Content-Disposition: form-data; name="comment[user_email]"


    -----------------------------17013073289399684191498077790


    Content-Disposition: form-data; name="comment[body]"


    I'm using (archaic) squid version 2.5.14-1.1. I need to turn squid off even to use this support forum.


    -----------------------------17013073289399684191498077790


    Content-Disposition: form-data; name="assets[]"; filename=""


    Content-Type: application/octet-stream


    -----------------------------17013073289399684191498077790--

    After the last request there is normal HTTP response, like this:

    HTTP/1.1 200 OK


    Server: nginx/0.6.34


    Date: Tue, 10 Mar 2009 00:54:18 GMT


    Content-Type: text/html; charset=utf-8


    Transfer-Encoding: chunked


    Connection: keep-alive


    Etag: "27394fde804628710c7472619d5f296d"


    X-Runtime: 313ms


    Cache-Control: private, max-age=0, must-revalidate cookies etc.
  5. 5 Posted by Nicole on 14 Mar, 2009 01:31 PM

    Nicole's Avatar

    Were our suggestions helpful? Are you still experiencing this problem? Let us know and we'll be happy to help. Thanks!

  6. Nicole closed this discussion on 14 Mar, 2009 01:31 PM.

  7. lighthouse270608 re-opened this discussion on 14 Mar, 2009 03:23 PM

  8. 6 Posted by lighthouse27060... on 14 Mar, 2009 03:23 PM

    lighthouse270608's Avatar

    No, the issue is still unresolved. I've tried twice to submit http headers of my request thru this forum, but it appears my posts were filtered.

  9. 7 Posted by Rick on 16 Mar, 2009 07:12 AM

    Rick's Avatar

    Oh you're talking about Tender, not Lighthouse. help.lighthouseapp.com is our Tender Support site. Have you been using squid with Lighthouse without problems? I'm not sure why it'd work in one and not the other. I don't do anything special with http caching. I find it odd that it's sending cache-control header on a POST. As far as I know, POST requests are never cached.

  10. 8 Posted by lighthouse27060... on 16 Mar, 2009 10:44 AM

    lighthouse270608's Avatar

    test using http/1.0

  11. 9 Posted by lighthouse27060... on 16 Mar, 2009 10:47 AM

    lighthouse270608's Avatar

    test http/1.0, squid turned off, dumping traffic

  12. 10 Posted by lighthouse27060... on 16 Mar, 2009 11:02 AM

    lighthouse270608's Avatar

    test http/1.0 + X-Forwared-For, squid off

  13. 11 Posted by lighthouse27060... on 16 Mar, 2009 11:05 AM

    lighthouse270608's Avatar

    test http/1.0 + X-Forwared-For, squid off once again

  14. 12 Posted by lighthouse27060... on 16 Mar, 2009 11:07 AM

    lighthouse270608's Avatar

    once again spamming this forum for testing purposes

  15. 13 Posted by lighthouse27060... on 16 Mar, 2009 11:14 AM

    lighthouse270608's Avatar

    squid off again, "Connection" header filtered

  16. 14 Posted by lighthouse27060... on 16 Mar, 2009 11:38 AM

    lighthouse270608's Avatar

    again3

  17. 15 Posted by lighthouse27060... on 16 Mar, 2009 11:43 AM

    lighthouse270608's Avatar

    test

  18. 16 Posted by lighthouse27060... on 16 Mar, 2009 11:53 AM

    lighthouse270608's Avatar

    mtu 1000

  19. 17 Posted by lighthouse27060... on 16 Mar, 2009 11:58 AM

    lighthouse270608's Avatar

    mtu 1480

  20. 18 Posted by lighthouse27060... on 16 Mar, 2009 11:58 AM

    lighthouse270608's Avatar

    1490

  21. 19 Posted by lighthouse27060... on 16 Mar, 2009 11:59 AM

    lighthouse270608's Avatar

    1492

  22. 20 Posted by lighthouse27060... on 16 Mar, 2009 11:59 AM

    lighthouse270608's Avatar

    1495

  23. 21 Posted by lighthouse27060... on 16 Mar, 2009 12:00 PM

    lighthouse270608's Avatar

    1493

  24. 22 Posted by lighthouse27060... on 16 Mar, 2009 12:00 PM

    lighthouse270608's Avatar

    1494

  25. 23 Posted by lighthouse27060... on 16 Mar, 2009 12:18 PM

    lighthouse270608's Avatar

    Solved!

    The issue was quite low level. It has nothing to do with HTTP, it is MTU related.

    Your servers have MTU lower than 1500. When big TCP packets was being sent (such as packets generated by posting a form) your routers generated ICMP fragmentation needed packet. This packet travelled across the Internet to my router which was incorectly routing it directly to my computer instead of the proxy. This is because the proxy works in fully transparent mode -- the router forward all TCP port 80 traffic to/from it. However it was misconfigured -- ICMP was not forwared to proxy.

    Now it is solved. I've modified MTU setting on my proxy to 1492 (sorry for spamming forum with test messages) which is the MTU your servers/routers use. Long term fix it to restore MTU to 1500 and set my router to forward ICMP packets related to http traffic to proxy.

    Your suggestions wasn't related to the root of the problem, but they motivated me to dig into it, so thank you :)

  26. 24 Posted by Rick on 16 Mar, 2009 05:10 PM

    Rick's Avatar

    That's hardcore! I've filed an issue with our hosting company. I personally don't know much about proper MTU configuration (clearly!)

  27. 25 Posted by Rick on 16 Mar, 2009 08:00 PM

    Rick's Avatar

    Resolved the issue to block the stupid spam. Our host did come back and say the MTU for lighthouseapp.com is 1500, but I'm having them check on tenderapp.com (which is where help.lighthouseapp.com really goes).

  28. Rick closed this discussion on 16 Mar, 2009 08:00 PM.

  29. lighthouse270608 re-opened this discussion on 16 Mar, 2009 09:41 PM

  30. 26 Posted by lighthouse27060... on 16 Mar, 2009 09:41 PM

    lighthouse270608's Avatar

    Oh man!

    I've checked on my router and found the real cause of the problem.

    Of course the problem was completely on my side, your MTU is probably ok.

    The problem is with my connection to the Internet. I've recently switched to PPPoE and completely forget about it. PPP encapsulation causes MTU to drop to 1492:

    ppp0: <POINTOPOINT,MULTICAST,NOARP,UP,10000> mtu 1492 qdisc tbf qlen 3

    link/ppp
    

    So the fragmentation needed ICMP packets was generated by my own router!

    So... I feel like an idiot. Sorry for taking your time...

    But on the other hand -- this issue was interesting, wasn't it? ;-)

  31. 27 Posted by Will Duncan on 17 Mar, 2009 11:20 AM

    Will Duncan's Avatar

    Out of curiosity, what type of router was it?

  32. 28 Posted by Rick on 17 Mar, 2009 04:12 PM

    Rick's Avatar

    Ha, that's exactly what our host said yesterday. Don't you just love computers?

  33. Rick closed this discussion on 17 Mar, 2009 04:12 PM.

  34. lighthouse270608 re-opened this discussion on 17 Mar, 2009 10:04 PM

  35. 29 Posted by lighthouse27060... on 17 Mar, 2009 10:04 PM

    lighthouse270608's Avatar

    @rick: I love computers, but computers sometimes hate me.

    @will: I use a system of custom build linux routers. If you are looking for something to put in your knowledge base then use this sentence: "When the problem looks unsolvable it is probably MTU related"

  36. brandi closed this discussion on 29 Jun, 2012 10:10 PM.

Discussions are closed to public comments.
If you need help with Lighthouse please start a new discussion.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac