Offload TCP traffic to another process

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Offload TCP traffic to another process

ramirezc
Dear experts,

We are evaluating nginx as a platform for the product of our new startup
company.

Our use-case requires a TCP proxy that will terminate TLS, which nginx
handles very well. However, we need to be able to send all TCP traffic to
another process for offline processing.

Initially we thought we could write a NGX_STREAM_MODULE (call it tcp_mirror)
that will be able to read both the downstream bytes (client <--> nginx) and
upstream bytes (proxy <--> server) and send them to another process, but
after looking at a few module examples and trying out a few things we
understood that we can only use a single content handler for each stream
configuration.

For example, we were hoping the following mock configuration would work for
us, but realized we can't have both proxy_pass and tcp_mirror under server
because there can be only one content handler:
stream {
    server {
        listen     12346;
        proxy_pass backend.example.com:12346;
        tcp_mirror processor.acme.com:6666;
    }
}

The above led us to the conclusion that in order to implement our use-case
we would have to write a new proxy_pass module, more specifically we would
have to re-write ngx_stream_proxy_module.c. The idea is that we would manage
two upstreams, the server and the processor. The configuration would look
something like this:
stream {
    server {
        listen     12346;
        proxy_pass_mirror backend.example.com:12346
processor.acme.com:6666;
    }
}

Before we begin implementation of this design, we wanted to consult with the
experts here and understand whether anyone has a better idea on how to
implement our use-case on top of nginx.

Thanks in advance,
Yoav Cohen.

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,286360,286360#msg-286360

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Offload TCP traffic to another process

Marcin Wanat

On Thu, Nov 28, 2019 at 9:33 PM yoav.cohen <[hidden email]> wrote:
Dear experts,

We are evaluating nginx as a platform for the product of our new startup
company.

Our use-case requires a TCP proxy that will terminate TLS, which nginx
handles very well. However, we need to be able to send all TCP traffic to
another process for offline processing.

Initially we thought we could write a NGX_STREAM_MODULE (call it tcp_mirror)
that will be able to read both the downstream bytes (client <--> nginx) and
upstream bytes (proxy <--> server) and send them to another process, but
after looking at a few module examples and trying out a few things we
understood that we can only use a single content handler for each stream
configuration.

For example, we were hoping the following mock configuration would work for
us, but realized we can't have both proxy_pass and tcp_mirror under server
because there can be only one content handler:
stream {
    server {
        listen     12346;
        proxy_pass backend.example.com:12346;
        tcp_mirror processor.acme.com:6666;
    }
}

The above led us to the conclusion that in order to implement our use-case
we would have to write a new proxy_pass module, more specifically we would
have to re-write ngx_stream_proxy_module.c. The idea is that we would manage
two upstreams, the server and the processor. The configuration would look
something like this:
stream {
    server {
        listen     12346;
        proxy_pass_mirror backend.example.com:12346
processor.acme.com:6666;
    }
}

Before we begin implementation of this design, we wanted to consult with the
experts here and understand whether anyone has a better idea on how to
implement our use-case on top of nginx.

Thanks in advance,
Yoav Cohen.



Regards,
Marcin Wanat

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Offload TCP traffic to another process

Patrick
In reply to this post by ramirezc
On 2019-11-28 15:33, yoav.cohen wrote:
> However, we need to be able to send all TCP traffic to another process
> for offline processing.

This can probably be done using the packet mgmt features of the OS, e.g.
with netfilter/iptables `TEE' for Linux:
    http://ipset.netfilter.org/iptables-extensions.man.html#lbDU
or ipf `dup-to' for FreeBSD:
    https://www.freebsd.org/cgi/man.cgi?query=ipf&sektion=5&apropos=0&manpath=FreeBSD+12.1-RELEASE+and+Ports

Mirroring the inside interfaces will yield the un-TLS'd traffic.


Patrick
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Offload TCP traffic to another process

ramirezc
In reply to this post by Marcin Wanat
Hello experts,

Thanks for the quick response!
My name is Alon and I am working with Yoav in the new startup company.

I would like to clarify few things on our use-case in order to give you the
information you need to help us doing the right thing with Nginx.

1. The application layer could be any protocol over TCP layer.
2. We need to do TLS termination in both directions, downstream and
upstream.
3. The mirror traffic is not for raw packets, it should be done to the
decrypted TCP content after the TLS termination(in both directions). 

So we thought on writing new stream module which works along side with the
proxy_pass stream command. The new module register a handler on a stream
content phase and copy the TCP content traffic to other process for
offline analysis.
As Yoav mentioned, seems like there is only 1 handler in the content phase
(which already taken by the proxy_pass stream). 

Do we need to re-write the ngx_stream_proxy_module for such mirror
capabilities ?
Is there other better way to implement the use-case with Nginx?

Thanks, Alon

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,286360,286364#msg-286364

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx