Quantcast

N00b - logging stream request / response

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

N00b - logging stream request / response

Joel Parker
I am trying to log all request / response in a stream with a lua script I found in git hub and am having issues figuring out where to put the log_format directive. Here is what I currently have :

stream {

    log_format bodylog '$remote_addr - $remote_user [$time_local] '
      '"$request" $status $body_bytes_sent '
      '"$http_referer" "$http_user_agent" $request_time '
      '<"$request_body" >"$resp_body"';

     lua_need_request_body on;

     set $resp_body "";
     body_filter_by_lua '
        local resp_body = ngx.arg[1]
        ngx.ctx.buffered = (ngx.ctx.buffered or "") .. resp_body
        if ngx.arg[2] then
           ngx.var.resp_body = ngx.ctx.buffered
        end
       ';

......

}

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: N00b - logging stream request / response

Robert Paprocki
What you're doing doesn't quite make sense. You're trying to log HTTP data inside a stream block. That doesn't work. There's no such concept of $status, $http_referer, etc, inside a stream {} block.

Have a read of the log_format docs: http://nginx.org/en/docs/http/ngx_http_log_module.html#log_format


Syntax:log_format name [escape=default|jsonstring ...;
Default:
log_format combined "...";
Context:http



On Tue, Apr 25, 2017 at 11:32 AM, Joel Parker <[hidden email]> wrote:
I am trying to log all request / response in a stream with a lua script I found in git hub and am having issues figuring out where to put the log_format directive. Here is what I currently have :

stream {

    log_format bodylog '$remote_addr - $remote_user [$time_local] '
      '"$request" $status $body_bytes_sent '
      '"$http_referer" "$http_user_agent" $request_time '
      '<"$request_body" >"$resp_body"';

     lua_need_request_body on;

     set $resp_body "";
     body_filter_by_lua '
        local resp_body = ngx.arg[1]
        ngx.ctx.buffered = (ngx.ctx.buffered or "") .. resp_body
        if ngx.arg[2] then
           ngx.var.resp_body = ngx.ctx.buffered
        end
       ';

......

}

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: N00b - logging stream request / response

Joel Parker
so can I have a hierarchy like this ?

http {
      // log format
    stream {
          server {
             // access log
         }
    }
}

On Tue, Apr 25, 2017 at 1:38 PM, Robert Paprocki <[hidden email]> wrote:
What you're doing doesn't quite make sense. You're trying to log HTTP data inside a stream block. That doesn't work. There's no such concept of $status, $http_referer, etc, inside a stream {} block.

Have a read of the log_format docs: http://nginx.org/en/docs/http/ngx_http_log_module.html#log_format


Syntax:log_format name [escape=default|jsonstring ...;
Default:
log_format combined "...";
Context:http



On Tue, Apr 25, 2017 at 11:32 AM, Joel Parker <[hidden email]> wrote:
I am trying to log all request / response in a stream with a lua script I found in git hub and am having issues figuring out where to put the log_format directive. Here is what I currently have :

stream {

    log_format bodylog '$remote_addr - $remote_user [$time_local] '
      '"$request" $status $body_bytes_sent '
      '"$http_referer" "$http_user_agent" $request_time '
      '<"$request_body" >"$resp_body"';

     lua_need_request_body on;

     set $resp_body "";
     body_filter_by_lua '
        local resp_body = ngx.arg[1]
        ngx.ctx.buffered = (ngx.ctx.buffered or "") .. resp_body
        if ngx.arg[2] then
           ngx.var.resp_body = ngx.ctx.buffered
        end
       ';

......

}

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: N00b - logging stream request / response

Robert Paprocki
No. stream {} and http {} blocks are mutually exclusive.

What exactly are you trying to accomplish with stream?

On Tue, Apr 25, 2017 at 11:46 AM, Joel Parker <[hidden email]> wrote:
so can I have a hierarchy like this ?

http {
      // log format
    stream {
          server {
             // access log
         }
    }
}

On Tue, Apr 25, 2017 at 1:38 PM, Robert Paprocki <[hidden email]> wrote:
What you're doing doesn't quite make sense. You're trying to log HTTP data inside a stream block. That doesn't work. There's no such concept of $status, $http_referer, etc, inside a stream {} block.

Have a read of the log_format docs: http://nginx.org/en/docs/http/ngx_http_log_module.html#log_format


Syntax:log_format name [escape=default|jsonstring ...;
Default:
log_format combined "...";
Context:http



On Tue, Apr 25, 2017 at 11:32 AM, Joel Parker <[hidden email]> wrote:
I am trying to log all request / response in a stream with a lua script I found in git hub and am having issues figuring out where to put the log_format directive. Here is what I currently have :

stream {

    log_format bodylog '$remote_addr - $remote_user [$time_local] '
      '"$request" $status $body_bytes_sent '
      '"$http_referer" "$http_user_agent" $request_time '
      '<"$request_body" >"$resp_body"';

     lua_need_request_body on;

     set $resp_body "";
     body_filter_by_lua '
        local resp_body = ngx.arg[1]
        ngx.ctx.buffered = (ngx.ctx.buffered or "") .. resp_body
        if ngx.arg[2] then
           ngx.var.resp_body = ngx.ctx.buffered
        end
       ';

......

}

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: N00b - logging stream request / response

Joel Parker
What I am trying to do is create an open proxy that listens to TLS from many servers and de-crypts the traffic with the appropriate keys, log the de-crytped request / response then re-encrypt with different certs and send to an upstream server. My thought was theat a stream block would help me accomplish this.

Joel

On Tue, Apr 25, 2017 at 1:49 PM, Robert Paprocki <[hidden email]> wrote:
No. stream {} and http {} blocks are mutually exclusive.

What exactly are you trying to accomplish with stream?

On Tue, Apr 25, 2017 at 11:46 AM, Joel Parker <[hidden email]> wrote:
so can I have a hierarchy like this ?

http {
      // log format
    stream {
          server {
             // access log
         }
    }
}

On Tue, Apr 25, 2017 at 1:38 PM, Robert Paprocki <[hidden email]> wrote:
What you're doing doesn't quite make sense. You're trying to log HTTP data inside a stream block. That doesn't work. There's no such concept of $status, $http_referer, etc, inside a stream {} block.

Have a read of the log_format docs: http://nginx.org/en/docs/http/ngx_http_log_module.html#log_format


Syntax:log_format name [escape=default|jsonstring ...;
Default:
log_format combined "...";
Context:http



On Tue, Apr 25, 2017 at 11:32 AM, Joel Parker <[hidden email]> wrote:
I am trying to log all request / response in a stream with a lua script I found in git hub and am having issues figuring out where to put the log_format directive. Here is what I currently have :

stream {

    log_format bodylog '$remote_addr - $remote_user [$time_local] '
      '"$request" $status $body_bytes_sent '
      '"$http_referer" "$http_user_agent" $request_time '
      '<"$request_body" >"$resp_body"';

     lua_need_request_body on;

     set $resp_body "";
     body_filter_by_lua '
        local resp_body = ngx.arg[1]
        ngx.ctx.buffered = (ngx.ctx.buffered or "") .. resp_body
        if ngx.arg[2] then
           ngx.var.resp_body = ngx.ctx.buffered
        end
       ';

......

}

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: N00b - logging stream request / response

Robert Paprocki
Just set up a server {} block that accepts TLS connections. This is exactly what proxy_pass is for :) You can log whatever HTTP data you need via Nginx (just as your log_format and content_by_lua block does), and then proxy_pass that traffic to your upstream as normal. Stream blocks are for arbitrary TCP/UDP streams; they have no knowledge of layer 7 HTTP data.

BTW it's very bad practice to buffer the whole request body like that ;)

On Tue, Apr 25, 2017 at 11:52 AM, Joel Parker <[hidden email]> wrote:
What I am trying to do is create an open proxy that listens to TLS from many servers and de-crypts the traffic with the appropriate keys, log the de-crytped request / response then re-encrypt with different certs and send to an upstream server. My thought was theat a stream block would help me accomplish this.

Joel

On Tue, Apr 25, 2017 at 1:49 PM, Robert Paprocki <[hidden email]> wrote:
No. stream {} and http {} blocks are mutually exclusive.

What exactly are you trying to accomplish with stream?

On Tue, Apr 25, 2017 at 11:46 AM, Joel Parker <[hidden email]> wrote:
so can I have a hierarchy like this ?

http {
      // log format
    stream {
          server {
             // access log
         }
    }
}

On Tue, Apr 25, 2017 at 1:38 PM, Robert Paprocki <[hidden email]> wrote:
What you're doing doesn't quite make sense. You're trying to log HTTP data inside a stream block. That doesn't work. There's no such concept of $status, $http_referer, etc, inside a stream {} block.

Have a read of the log_format docs: http://nginx.org/en/docs/http/ngx_http_log_module.html#log_format


Syntax:log_format name [escape=default|jsonstring ...;
Default:
log_format combined "...";
Context:http



On Tue, Apr 25, 2017 at 11:32 AM, Joel Parker <[hidden email]> wrote:
I am trying to log all request / response in a stream with a lua script I found in git hub and am having issues figuring out where to put the log_format directive. Here is what I currently have :

stream {

    log_format bodylog '$remote_addr - $remote_user [$time_local] '
      '"$request" $status $body_bytes_sent '
      '"$http_referer" "$http_user_agent" $request_time '
      '<"$request_body" >"$resp_body"';

     lua_need_request_body on;

     set $resp_body "";
     body_filter_by_lua '
        local resp_body = ngx.arg[1]
        ngx.ctx.buffered = (ngx.ctx.buffered or "") .. resp_body
        if ngx.arg[2] then
           ngx.var.resp_body = ngx.ctx.buffered
        end
       ';

......

}

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: N00b - logging stream request / response

Joel Parker
I am still having issues with the config, can you take a look at my short config and see where my confusion lies ? Changed the stream block to http and think I did the rest correct but still complains about log_format and not sure if my proxy_pass or any of the rest of it is bad.

Joel

On Tue, Apr 25, 2017 at 2:00 PM, Robert Paprocki <[hidden email]> wrote:
Just set up a server {} block that accepts TLS connections. This is exactly what proxy_pass is for :) You can log whatever HTTP data you need via Nginx (just as your log_format and content_by_lua block does), and then proxy_pass that traffic to your upstream as normal. Stream blocks are for arbitrary TCP/UDP streams; they have no knowledge of layer 7 HTTP data.

BTW it's very bad practice to buffer the whole request body like that ;)


On Tue, Apr 25, 2017 at 11:52 AM, Joel Parker <[hidden email]> wrote:
What I am trying to do is create an open proxy that listens to TLS from many servers and de-crypts the traffic with the appropriate keys, log the de-crytped request / response then re-encrypt with different certs and send to an upstream server. My thought was theat a stream block would help me accomplish this.

Joel

On Tue, Apr 25, 2017 at 1:49 PM, Robert Paprocki <[hidden email]> wrote:
No. stream {} and http {} blocks are mutually exclusive.

What exactly are you trying to accomplish with stream?

On Tue, Apr 25, 2017 at 11:46 AM, Joel Parker <[hidden email]> wrote:
so can I have a hierarchy like this ?

http {
      // log format
    stream {
          server {
             // access log
         }
    }
}

On Tue, Apr 25, 2017 at 1:38 PM, Robert Paprocki <[hidden email]> wrote:
What you're doing doesn't quite make sense. You're trying to log HTTP data inside a stream block. That doesn't work. There's no such concept of $status, $http_referer, etc, inside a stream {} block.

Have a read of the log_format docs: http://nginx.org/en/docs/http/ngx_http_log_module.html#log_format


Syntax:log_format name [escape=default|jsonstring ...;
Default:
log_format combined "...";
Context:http



On Tue, Apr 25, 2017 at 11:32 AM, Joel Parker <[hidden email]> wrote:
I am trying to log all request / response in a stream with a lua script I found in git hub and am having issues figuring out where to put the log_format directive. Here is what I currently have :

stream {

    log_format bodylog '$remote_addr - $remote_user [$time_local] '
      '"$request" $status $body_bytes_sent '
      '"$http_referer" "$http_user_agent" $request_time '
      '<"$request_body" >"$resp_body"';

     lua_need_request_body on;

     set $resp_body "";
     body_filter_by_lua '
        local resp_body = ngx.arg[1]
        ngx.ctx.buffered = (ngx.ctx.buffered or "") .. resp_body
        if ngx.arg[2] then
           ngx.var.resp_body = ngx.ctx.buffered
        end
       ';

......

}

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx


_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx

nginx.conf (2K) Download Attachment
Loading...