how to track down an nginx crash

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

how to track down an nginx crash

Payam Chychi
Hey guys,

Nginx crashed 2 times on 2 of my proxy servers... no errors in logs
and i cant figure out why the crash took place. I do see from my rrd
files that there was about 20mbps of http traffic going to the box at
the time of the crash...

any ideas?
Thanks,

--
Payam Tarverdyan Chychi
Network Security Specialist / Network Engineer

Reply | Threaded
Open this post in threaded view
|

Re: how to track down an nginx crash

Michael Shadle
all i can say is... wow.

nginx has never crashed on me. congrats? :)

i would provide your version info, that might help first off.

On Thu, May 21, 2009 at 1:20 PM, Payam Chychi <[hidden email]> wrote:

> Hey guys,
>
> Nginx crashed 2 times on 2 of my proxy servers... no errors in logs
> and i cant figure out why the crash took place. I do see from my rrd
> files that there was about 20mbps of http traffic going to the box at
> the time of the crash...
>
> any ideas?
> Thanks,
>
> --
> Payam Tarverdyan Chychi
> Network Security Specialist / Network Engineer
>
>

Reply | Threaded
Open this post in threaded view
|

Re: how to track down an nginx crash

Payam Chychi
On Thu, May 21, 2009 at 1:39 PM, Michael Shadle <[hidden email]> wrote:

> all i can say is... wow.
>
> nginx has never crashed on me. congrats? :)
>
> i would provide your version info, that might help first off.
>
> On Thu, May 21, 2009 at 1:20 PM, Payam Chychi <[hidden email]> wrote:
>> Hey guys,
>>
>> Nginx crashed 2 times on 2 of my proxy servers... no errors in logs
>> and i cant figure out why the crash took place. I do see from my rrd
>> files that there was about 20mbps of http traffic going to the box at
>> the time of the crash...
>>
>> any ideas?
>> Thanks,
>>
>> --
>> Payam Tarverdyan Chychi
>> Network Security Specialist / Network Engineer
>>
>>
>
>

nginx/0.7.54  with SSL

--
Payam Tarverdyan Chychi
Network Security Specialist / Network Engineer

Reply | Threaded
Open this post in threaded view
|

Re: how to track down an nginx crash

Michael Shadle
Igor would probably ask at this point for a debug log.

If this continues to happen I would compile nginx --with-debug and
have a way to log when it crashes if you can. You can limit who
triggers debugging by IP too by using debug_connection, otherwise you
will have immensely large logfiles, but at least you'd possibly be
able to track down the event and have the info Igor would probably be
able to pick apart.

On Thu, May 21, 2009 at 3:30 PM, Payam Chychi <[hidden email]> wrote:

> nginx/0.7.54  with SSL

Reply | Threaded
Open this post in threaded view
|

Re: how to track down an nginx crash

Payam Chychi
On Thu, May 21, 2009 at 4:17 PM, Michael Shadle <[hidden email]> wrote:

> Igor would probably ask at this point for a debug log.
>
> If this continues to happen I would compile nginx --with-debug and
> have a way to log when it crashes if you can. You can limit who
> triggers debugging by IP too by using debug_connection, otherwise you
> will have immensely large logfiles, but at least you'd possibly be
> able to track down the event and have the info Igor would probably be
> able to pick apart.
>
> On Thu, May 21, 2009 at 3:30 PM, Payam Chychi <[hidden email]> wrote:
>
>> nginx/0.7.54  with SSL
>
>

Great advice, ill setup the debugging tonight as im sure the issue
will return in time. I am using nginx as a part of ddos mitigation
system so it really pushes all resources to the limit :)

Thanks again

--
Payam Tarverdyan Chychi
Network Security Specialist / Network Engineer

Reply | Threaded
Open this post in threaded view
|

Re: how to track down an nginx crash

Igor Sysoev
In reply to this post by Payam Chychi
On Thu, May 21, 2009 at 01:20:16PM -0700, Payam Chychi wrote:

> Hey guys,
>
> Nginx crashed 2 times on 2 of my proxy servers... no errors in logs
> and i cant figure out why the crash took place. I do see from my rrd
> files that there was about 20mbps of http traffic going to the box at
> the time of the crash...

What do mean by the crash ? If nginx exists on single there is a alert
message in error_log.


--
Igor Sysoev
http://sysoev.ru/en/

Reply | Threaded
Open this post in threaded view
|

Re: how to track down an nginx crash

Payam Chychi
2009/5/22 Igor Sysoev <[hidden email]>:

> On Thu, May 21, 2009 at 01:20:16PM -0700, Payam Chychi wrote:
>
>> Hey guys,
>>
>> Nginx crashed 2 times on 2 of my proxy servers... no errors in logs
>> and i cant figure out why the crash took place. I do see from my rrd
>> files that there was about 20mbps of http traffic going to the box at
>> the time of the crash...
>
> What do mean by the crash ? If nginx exists on single there is a alert
> message in error_log.
>
>
> --
> Igor Sysoev
> http://sysoev.ru/en/
>
>

Hey Igor,

Nginx process crashed without leaving any traces in the system logs or error.log
Im going to rebuild with debug in there but I still an more sure why
it crashed (2 different boxes... both nginx crashed under 20mbps of
load / 8k-10k request/sec while the other 8 boxes did not)

Unfortunately I have not had much time to resume troubleshooting but
hope to start again in a few days


--
Payam Tarverdyan Chychi
Network Security Specialist / Network Engineer