504 gateway timeouts

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

504 gateway timeouts

Wade Girard
I am using nginx on an ubuntu server as a proxy to a tomcat server.

The nginx server is setup for https.

I don't know how to determine what version of nginx I am using, but I install it on the ubuntu 1.16 server using apt-get.

I have an issue that I have resolved locally on my Mac (using version 1.12 of nginx and Tomcat 7) where requests through the proxy that take more than 60 seconds were failing, they are now working.

What seemed to be the fix was adding the following to the nginx.conf file

proxy_connect_timeout       600;

        proxy_send_timeout          600;

        proxy_read_timeout          600;

        send_timeout                600;

in the location section for my proxy.


However this same change in the ubuntu servers has no effect at all.


The way I am testing this is that I create a request that sleeps the thread for 5 minutes before retiring a response.

Any help appreciated.

Thanks

--
Wade Girard
c: 612.363.0902

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: 504 gateway timeouts

Igal @ Lucee.org
On 12/27/2017 2:03 PM, Wade Girard wrote:
I am using nginx on an ubuntu server as a proxy to a tomcat server.

The nginx server is setup for https.

I don't know how to determine what version of nginx I am using, but I install it on the ubuntu 1.16 server using apt-get.
Run:  nginx -v


I have an issue that I have resolved locally on my Mac (using version 1.12 of nginx and Tomcat 7) where requests through the proxy that take more than 60 seconds were failing, they are now working.

What seemed to be the fix was adding the following to the nginx.conf file

proxy_connect_timeout       600;

        proxy_send_timeout          600;

        proxy_read_timeout          600;

        send_timeout                600;

in the location section for my proxy.


However this same change in the ubuntu servers has no effect at all.


Try to flush out some output early on so that nginx will know that Tomcat is alive.

Igal Sapir
Lucee Core Developer
Lucee.org




_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: 504 gateway timeouts

Wade Girard
The version that is on the ubuntu servers was 1.10.xx. I just updated it to 

nginx version: nginx/1.13.8

And I am still having the same issue.

How do I "Try to flush out some output early on so that nginx will know that Tomcat is alive."

The nginx and tomcat connection is working fine for all requests/responses that take less than 60 seconds.

On Wed, Dec 27, 2017 at 4:18 PM, Igal @ Lucee.org <[hidden email]> wrote:
On 12/27/2017 2:03 PM, Wade Girard wrote:
I am using nginx on an ubuntu server as a proxy to a tomcat server.

The nginx server is setup for https.

I don't know how to determine what version of nginx I am using, but I install it on the ubuntu 1.16 server using apt-get.
Run:  nginx -v


I have an issue that I have resolved locally on my Mac (using version 1.12 of nginx and Tomcat 7) where requests through the proxy that take more than 60 seconds were failing, they are now working.

What seemed to be the fix was adding the following to the nginx.conf file

proxy_connect_timeout       600;

        proxy_send_timeout          600;

        proxy_read_timeout          600;

        send_timeout                600;

in the location section for my proxy.


However this same change in the ubuntu servers has no effect at all.


Try to flush out some output early on so that nginx will know that Tomcat is alive.

Igal Sapir
Lucee Core Developer
Lucee.org






--
Wade Girard
c: 612.363.0902

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: 504 gateway timeouts

Zhang Chao
> The version that is on the ubuntu servers was 1.10.xx. I just updated it to 
>
> nginx version: nginx/1.13.8
>
> And I am still having the same issue.
>
> How do I "Try to flush out some output early on so that nginx will know that Tomcat is alive."
>
> The nginx and tomcat connection is working fine for all requests/responses that take less t

Maybe you can flush out the HTTP response headers quickly.


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: 504 gateway timeouts

Wade Girard
I am not sure what is meant by this or what action you are asking me to take. The settings, when added to nginx conf file on Mac OS server and nginx reloaded take effect immediately and work as expected, the same settings when added to nginx conf file on Ubuntu and nginx reloaded have no effect at all. What steps can I take to have the proxy in nginx honor these timeouts, or what other settings/actions can I take to make this work?

Thanks

On Thu, Jan 4, 2018 at 7:46 PM, Zhang Chao <[hidden email]> wrote:
> The version that is on the ubuntu servers was 1.10.xx. I just updated it to 
>
> nginx version: nginx/1.13.8
>
> And I am still having the same issue.
>
> How do I "Try to flush out some output early on so that nginx will know that Tomcat is alive."
>
> The nginx and tomcat connection is working fine for all requests/responses that take less t

Maybe you can flush out the HTTP response headers quickly.


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx



--
Wade Girard
c: 612.363.0902

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: 504 gateway timeouts

Peter Booth
Wade,

I think that you are asking “hey why isn’t nginx behaving identically on MacOS and Linux when create a servlet that invokes Thread.sleep(300000) before it returns a response?.”

Am I reading you correctly?

A flippant response would be to say: “because OS/X and Linux are different OSes that behave differently”

It would probably help us if you explained a little more about your test, why the sleep is there and what your goals are?


Peter

On Jan 4, 2018, at 11:45 PM, Wade Girard <[hidden email]> wrote:

I am not sure what is meant by this or what action you are asking me to take. The settings, when added to nginx conf file on Mac OS server and nginx reloaded take effect immediately and work as expected, the same settings when added to nginx conf file on Ubuntu and nginx reloaded have no effect at all. What steps can I take to have the proxy in nginx honor these timeouts, or what other settings/actions can I take to make this work?

Thanks

On Thu, Jan 4, 2018 at 7:46 PM, Zhang Chao <[hidden email]> wrote:
> The version that is on the ubuntu servers was 1.10.xx. I just updated it to 
>
> nginx version: nginx/1.13.8
>
> And I am still having the same issue.
>
> How do I "Try to flush out some output early on so that nginx will know that Tomcat is alive."
>
> The nginx and tomcat connection is working fine for all requests/responses that take less t

Maybe you can flush out the HTTP response headers quickly.


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx



--
Wade Girard
c: 612.363.0902
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: 504 gateway timeouts

Wade Girard
Hi Peter,

Thank You.

In my servlet I am making https requests to third party vendors to get data from them. The requests typically take 4~5 seconds, but every now any then one of the requests will take more than 60 seconds. So the connection from the client to nginx to tomcat will remain open, and at 60 seconds nginx is terminating the request to tomcat, even though the connection from the third party server to tomcat is still open.

I am also working with the third party vendor to have them see why their connections sometimes take more than 60 seconds.

Through googling I discovered that adding the settings proxy_send_timeout, proxy_read_timeout, proxy_connection_timeout, etc... to my location definition in my conf file could change the timeout to be different (higher) than the apparent default 60 second timeout. I use a Mac for development. I added these to my local conf file, and added the long connection request to test if the settings worked. They did. However they do not have the same effect for nginx installed on my production Ubuntu 16.x servers. I did not realize that these settings were limited by the OS that nginx is installed on. Are there are similar settings that will work for the Ubuntu 16.x OS to achieve the same result?

Wade

On Fri, Jan 5, 2018 at 1:33 AM, Peter Booth <[hidden email]> wrote:
Wade,

I think that you are asking “hey why isn’t nginx behaving identically on MacOS and Linux when create a servlet that invokes Thread.sleep(300000) before it returns a response?.”

Am I reading you correctly?

A flippant response would be to say: “because OS/X and Linux are different OSes that behave differently”

It would probably help us if you explained a little more about your test, why the sleep is there and what your goals are?


Peter


On Jan 4, 2018, at 11:45 PM, Wade Girard <[hidden email]> wrote:

I am not sure what is meant by this or what action you are asking me to take. The settings, when added to nginx conf file on Mac OS server and nginx reloaded take effect immediately and work as expected, the same settings when added to nginx conf file on Ubuntu and nginx reloaded have no effect at all. What steps can I take to have the proxy in nginx honor these timeouts, or what other settings/actions can I take to make this work?

Thanks

On Thu, Jan 4, 2018 at 7:46 PM, Zhang Chao <[hidden email]> wrote:
> The version that is on the ubuntu servers was 1.10.xx. I just updated it to 
>
> nginx version: nginx/1.13.8
>
> And I am still having the same issue.
>
> How do I "Try to flush out some output early on so that nginx will know that Tomcat is alive."
>
> The nginx and tomcat connection is working fine for all requests/responses that take less t

Maybe you can flush out the HTTP response headers quickly.


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx



--
Wade Girard
c: <a href="tel:(612)%20363-0902" value="+16123630902" target="_blank">612.363.0902
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx



--
Wade Girard
c: 612.363.0902

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: 504 gateway timeouts

Wade Girard
Hi nginx group,

If anyone has any ides on this, they would be appreciated.

Thanks

On Fri, Jan 5, 2018 at 6:28 AM, Wade Girard <[hidden email]> wrote:
Hi Peter,

Thank You.

In my servlet I am making https requests to third party vendors to get data from them. The requests typically take 4~5 seconds, but every now any then one of the requests will take more than 60 seconds. So the connection from the client to nginx to tomcat will remain open, and at 60 seconds nginx is terminating the request to tomcat, even though the connection from the third party server to tomcat is still open.

I am also working with the third party vendor to have them see why their connections sometimes take more than 60 seconds.

Through googling I discovered that adding the settings proxy_send_timeout, proxy_read_timeout, proxy_connection_timeout, etc... to my location definition in my conf file could change the timeout to be different (higher) than the apparent default 60 second timeout. I use a Mac for development. I added these to my local conf file, and added the long connection request to test if the settings worked. They did. However they do not have the same effect for nginx installed on my production Ubuntu 16.x servers. I did not realize that these settings were limited by the OS that nginx is installed on. Are there are similar settings that will work for the Ubuntu 16.x OS to achieve the same result?

Wade

On Fri, Jan 5, 2018 at 1:33 AM, Peter Booth <[hidden email]> wrote:
Wade,

I think that you are asking “hey why isn’t nginx behaving identically on MacOS and Linux when create a servlet that invokes Thread.sleep(300000) before it returns a response?.”

Am I reading you correctly?

A flippant response would be to say: “because OS/X and Linux are different OSes that behave differently”

It would probably help us if you explained a little more about your test, why the sleep is there and what your goals are?


Peter


On Jan 4, 2018, at 11:45 PM, Wade Girard <[hidden email]> wrote:

I am not sure what is meant by this or what action you are asking me to take. The settings, when added to nginx conf file on Mac OS server and nginx reloaded take effect immediately and work as expected, the same settings when added to nginx conf file on Ubuntu and nginx reloaded have no effect at all. What steps can I take to have the proxy in nginx honor these timeouts, or what other settings/actions can I take to make this work?

Thanks

On Thu, Jan 4, 2018 at 7:46 PM, Zhang Chao <[hidden email]> wrote:
> The version that is on the ubuntu servers was 1.10.xx. I just updated it to 
>
> nginx version: nginx/1.13.8
>
> And I am still having the same issue.
>
> How do I "Try to flush out some output early on so that nginx will know that Tomcat is alive."
>
> The nginx and tomcat connection is working fine for all requests/responses that take less t

Maybe you can flush out the HTTP response headers quickly.


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx



--
Wade Girard
c: <a href="tel:(612)%20363-0902" value="+16123630902" target="_blank">612.363.0902
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx



--
Wade Girard
c: <a href="tel:(612)%20363-0902" value="+16123630902" target="_blank">612.363.0902



--
Wade Girard
c: 612.363.0902

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: 504 gateway timeouts

Peter Booth
Wade, 

This reminds me of something I once saw with an application that was making web service requests to FedEx. So are you saying that the response times are bimodal? That you either get a remote response within a few seconds or the request takes more than 60 seconds, and that you have no 20sec,30sec,40sec requests?

And, if so, do those 60+ sec requests ever get a healthy response?


Sent from my iPhone

On Jan 9, 2018, at 1:52 PM, Wade Girard <[hidden email]> wrote:

Hi nginx group,

If anyone has any ides on this, they would be appreciated.

Thanks

On Fri, Jan 5, 2018 at 6:28 AM, Wade Girard <[hidden email]> wrote:
Hi Peter,

Thank You.

In my servlet I am making https requests to third party vendors to get data from them. The requests typically take 4~5 seconds, but every now any then one of the requests will take more than 60 seconds. So the connection from the client to nginx to tomcat will remain open, and at 60 seconds nginx is terminating the request to tomcat, even though the connection from the third party server to tomcat is still open.

I am also working with the third party vendor to have them see why their connections sometimes take more than 60 seconds.

Through googling I discovered that adding the settings proxy_send_timeout, proxy_read_timeout, proxy_connection_timeout, etc... to my location definition in my conf file could change the timeout to be different (higher) than the apparent default 60 second timeout. I use a Mac for development. I added these to my local conf file, and added the long connection request to test if the settings worked. They did. However they do not have the same effect for nginx installed on my production Ubuntu 16.x servers. I did not realize that these settings were limited by the OS that nginx is installed on. Are there are similar settings that will work for the Ubuntu 16.x OS to achieve the same result?

Wade

On Fri, Jan 5, 2018 at 1:33 AM, Peter Booth <[hidden email]> wrote:
Wade,

I think that you are asking “hey why isn’t nginx behaving identically on MacOS and Linux when create a servlet that invokes Thread.sleep(300000) before it returns a response?.”

Am I reading you correctly?

A flippant response would be to say: “because OS/X and Linux are different OSes that behave differently”

It would probably help us if you explained a little more about your test, why the sleep is there and what your goals are?


Peter


On Jan 4, 2018, at 11:45 PM, Wade Girard <[hidden email]> wrote:

I am not sure what is meant by this or what action you are asking me to take. The settings, when added to nginx conf file on Mac OS server and nginx reloaded take effect immediately and work as expected, the same settings when added to nginx conf file on Ubuntu and nginx reloaded have no effect at all. What steps can I take to have the proxy in nginx honor these timeouts, or what other settings/actions can I take to make this work?

Thanks

On Thu, Jan 4, 2018 at 7:46 PM, Zhang Chao <[hidden email]> wrote:
> The version that is on the ubuntu servers was 1.10.xx. I just updated it to 
>
> nginx version: nginx/1.13.8
>
> And I am still having the same issue.
>
> How do I "Try to flush out some output early on so that nginx will know that Tomcat is alive."
>
> The nginx and tomcat connection is working fine for all requests/responses that take less t

Maybe you can flush out the HTTP response headers quickly.


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx



--
Wade Girard
c: <a href="tel:(612)%20363-0902" value="+16123630902" target="_blank">612.363.0902
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx



--
Wade Girard
c: <a href="tel:(612)%20363-0902" value="+16123630902" target="_blank">612.363.0902



--
Wade Girard
c: 612.363.0902

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: 504 gateway timeouts

Aziz Rozyev
Hi Wade,

At least provide the access/error log fragments, curl -ivvv <..> outputs directly to the 3rd party service and via the nginx,
jmeter (if you use that) outputs would make sense. Also, it would be nice to compare nginx configurations from the mac and linux.

Currently it’s barely possible to conclude something relevant about the issue, having exactly the same nginx configurations,
and getting different results looks rather strange.

Hint: change ‘error_log’ directive’s level to ‘info’, check error/access logs, gather tcpdump. And if that won’t make the problem
clearer, start nginx in debug mode. this article should be useful: https://www.nginx.com/resources/admin-guide/debug/



br,
Aziz.





> On 9 Jan 2018, at 23:56, Peter Booth <[hidden email]> wrote:
>
> Wade,
>
> This reminds me of something I once saw with an application that was making web service requests to FedEx. So are you saying that the response times are bimodal? That you either get a remote response within a few seconds or the request takes more than 60 seconds, and that you have no 20sec,30sec,40sec requests?
>
> And, if so, do those 60+ sec requests ever get a healthy response?
>
>
> Sent from my iPhone
>
> On Jan 9, 2018, at 1:52 PM, Wade Girard <[hidden email]> wrote:
>
>> Hi nginx group,
>>
>> If anyone has any ides on this, they would be appreciated.
>>
>> Thanks
>>
>> On Fri, Jan 5, 2018 at 6:28 AM, Wade Girard <[hidden email]> wrote:
>> Hi Peter,
>>
>> Thank You.
>>
>> In my servlet I am making https requests to third party vendors to get data from them. The requests typically take 4~5 seconds, but every now any then one of the requests will take more than 60 seconds. So the connection from the client to nginx to tomcat will remain open, and at 60 seconds nginx is terminating the request to tomcat, even though the connection from the third party server to tomcat is still open.
>>
>> I am also working with the third party vendor to have them see why their connections sometimes take more than 60 seconds.
>>
>> Through googling I discovered that adding the settings proxy_send_timeout, proxy_read_timeout, proxy_connection_timeout, etc... to my location definition in my conf file could change the timeout to be different (higher) than the apparent default 60 second timeout. I use a Mac for development. I added these to my local conf file, and added the long connection request to test if the settings worked. They did. However they do not have the same effect for nginx installed on my production Ubuntu 16.x servers. I did not realize that these settings were limited by the OS that nginx is installed on. Are there are similar settings that will work for the Ubuntu 16.x OS to achieve the same result?
>>
>> Wade
>>
>> On Fri, Jan 5, 2018 at 1:33 AM, Peter Booth <[hidden email]> wrote:
>> Wade,
>>
>> I think that you are asking “hey why isn’t nginx behaving identically on MacOS and Linux when create a servlet that invokes Thread.sleep(300000) before it returns a response?.”
>>
>> Am I reading you correctly?
>>
>> A flippant response would be to say: “because OS/X and Linux are different OSes that behave differently”
>>
>> It would probably help us if you explained a little more about your test, why the sleep is there and what your goals are?
>>
>>
>> Peter
>>
>>
>>> On Jan 4, 2018, at 11:45 PM, Wade Girard <[hidden email]> wrote:
>>>
>>> I am not sure what is meant by this or what action you are asking me to take. The settings, when added to nginx conf file on Mac OS server and nginx reloaded take effect immediately and work as expected, the same settings when added to nginx conf file on Ubuntu and nginx reloaded have no effect at all. What steps can I take to have the proxy in nginx honor these timeouts, or what other settings/actions can I take to make this work?
>>>
>>> Thanks
>>>
>>> On Thu, Jan 4, 2018 at 7:46 PM, Zhang Chao <[hidden email]> wrote:
>>> > The version that is on the ubuntu servers was 1.10.xx. I just updated it to
>>> >
>>> > nginx version: nginx/1.13.8
>>> >
>>> > And I am still having the same issue.
>>> >
>>> > How do I "Try to flush out some output early on so that nginx will know that Tomcat is alive."
>>> >
>>> > The nginx and tomcat connection is working fine for all requests/responses that take less t
>>>
>>> Maybe you can flush out the HTTP response headers quickly.
>>>
>>>
>>> _______________________________________________
>>> nginx mailing list
>>> [hidden email]
>>> http://mailman.nginx.org/mailman/listinfo/nginx
>>>
>>>
>>>
>>> --
>>> Wade Girard
>>> c: 612.363.0902
>>> _______________________________________________
>>> nginx mailing list
>>> [hidden email]
>>> http://mailman.nginx.org/mailman/listinfo/nginx
>>
>>
>> _______________________________________________
>> nginx mailing list
>> [hidden email]
>> http://mailman.nginx.org/mailman/listinfo/nginx
>>
>>
>>
>> --
>> Wade Girard
>> c: 612.363.0902
>>
>>
>>
>> --
>> Wade Girard
>> c: 612.363.0902
> _______________________________________________
> nginx mailing list
> [hidden email]
> http://mailman.nginx.org/mailman/listinfo/nginx

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx