Enabling "Transfer-Encoding : chunked"

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Enabling "Transfer-Encoding : chunked"

wld75
In order to support CMAF and Low latency for HLS streaming through Nginx, it
is required change in content header.

Instead of "Content-Length" in Header , expected value by player is
"Transfer-Encoding : chunked" so that for a 6 sec chunk of media segment
player will start streaming fetching data in 200 msec part wise and thus
streaming will have low latency . This is supported by HTTP 1.1

Tried below parameter to enable same in Nginx Configuration  
chunked_transfer_encoding on;

But its not adding the same in header.

Please suggest better way to do it.
https://gist.github.com/CMCDragonkai/6bfade6431e9ffb7fe88

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,281371,281371#msg-281371

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Enabling "Transfer-Encoding : chunked"

Maxim Dounin
Hello!

On Mon, Sep 24, 2018 at 09:53:26AM -0400, anish10dec wrote:

> In order to support CMAF and Low latency for HLS streaming through Nginx, it
> is required change in content header.
>
> Instead of "Content-Length" in Header , expected value by player is
> "Transfer-Encoding : chunked" so that for a 6 sec chunk of media segment
> player will start streaming fetching data in 200 msec part wise and thus
> streaming will have low latency . This is supported by HTTP 1.1
>
> Tried below parameter to enable same in Nginx Configuration  
> chunked_transfer_encoding on;
>
> But its not adding the same in header.
>
> Please suggest better way to do it.
> https://gist.github.com/CMCDragonkai/6bfade6431e9ffb7fe88

The text you are referring to is misleading.

There is no difference between "Content-Length" and
"Transfer-Encoding: chunked" from the streaming point of view,
except that with "Content-Length" the client knows expected full
response size in advance.  Nothing stops the client from rendering
responses with "Content-Length" once data arrives.  If your client
for some reason requires "Transfer-Encoding: chunked", it looks
like a bug and/or misfeature of the particular client.

The only case when it makes sense to use "Transfer-Encoding:
chunked" is when the full response length is not known in advance,
and hence "Content-Length" cannot be used.

As for low-latency HLS streaming, the key part is that
"Transfer-Encoding: chunked" is used by the encoder to return
already available parts of the currently-being-produced HLS
segment.  As the segment is not yet complete, its full length is
not known and hence "Content-Length" cannot be used in the
response.

For this to work, you'll need appropriate support in your HLS
encoder - that is, it needs to return the last segment via HTTP
while the segment is being produced.  If nginx is used to proxy
such requests, everything is expected to work out of the box -
nginx will use "Transfer-Encoding: chunked" as the length of the
response is not known.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Enabling "Transfer-Encoding : chunked"

wld75
Thanks Maxim

For Streaming with Low Latency , Harmonic Encoder is pushing media files
with "Transfer-Encoding: chunked" on the Nginx Origin Server.

We are able to see the same in tcpdump between Encoder and Nginx Origin.

However when we try to stream content through Origin Server ,
"Transfer-Encoding: chunked" is missing in the header part because of which
player is not able to start stream with enabling low latency

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,281371,281374#msg-281374

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Enabling "Transfer-Encoding : chunked"

Maxim Dounin
Hello!

On Mon, Sep 24, 2018 at 11:40:27AM -0400, anish10dec wrote:

> Thanks Maxim
>
> For Streaming with Low Latency , Harmonic Encoder is pushing media files
> with "Transfer-Encoding: chunked" on the Nginx Origin Server.
>
> We are able to see the same in tcpdump between Encoder and Nginx Origin.

Ok, so everything works as intended when using proxying, right?

> However when we try to stream content through Origin Server ,
> "Transfer-Encoding: chunked" is missing in the header part because of which
> player is not able to start stream with enabling low latency

From your description it is not clear what you are trying to do
here.  If you are trying to save HLS encoding results to disk and
then serve them using nginx as static files, then it is not going
to work with HLS low-latency streaming - because nginx does not
know if a particular segment file is complete, or it is being
written right now and anything added to the file needs to be sent
to the client till some unspecified moment in the future.

If you want low latency live HLS streaming to work, you'll have to
use proxying at least for the last segment (the one which is being
written to).

If you observe problems with already completed segments (that is,
segments which are fully complete, and their length already known)
once they are served with Content-Length, this is probably
something to be addressed in the client.  As previously explained,
there is no difference between "Content-Length" and
"Transfer-Encoding: chunked" if full length of a response is known
in advance.

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Enabling "Transfer-Encoding : chunked"

wld75
In reply to this post by wld75
We are using Nginx with DAV Module , where encoder is pushing the content.
These content when being accessed is not coming with header
"Transfer-Encoding : chunked" though these header is being added by
Encoder.


Below is version details :

nginx version: nginx/1.10.2
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC)
built with OpenSSL 1.0.2k-fips  26 Jan 2017
TLS SNI support enabled
configure arguments: --with-http_ssl_module --with-http_realip_module
--with-http_addition_module --with-http_sub_module --with-http_dav_module
--add-module=/opt/nginx-dav-ext-module-master --with-http_flv_module
--with-http_mp4_module --with-http_gunzip_module
--with-http_gzip_static_module --with-http_random_index_module
--with-http_secure_link_module --with-http_stub_status_module
--with-http_auth_request_module --with-mail --with-mail_ssl_module
--with-file-aio --with-ipv6

Below is the nginx configuration where encoder is pushing the content on
Nginx running on Port 81

location /packagerx {
        root   /ram/streams_live/packagerx;
        dav_methods PUT DELETE MKCOL COPY MOVE;
        dav_ext_methods PROPFIND OPTIONS;
        create_full_put_path  on;
        dav_access user:rw group:rw all:r;
        autoindex on;
        client_max_body_size 100m;
}

Below is the configuration from which Nginx running on Port 80 is used for
accessing the content

location / {
        root   /ram/streams_live/packagerx;
        expires 1h;
        access_log /usr/local/nginx/logs/access_client.log lt-custom;
        proxy_buffering off;
        chunked_transfer_encoding on;

            types {
                application/dash+xml mpd;
                application/vnd.apple.mpegurl m3u8;
                video/mp2t ts;
                video/x-m4v                           m4v;
                audio/x-m4a                           m4a;
                text/html                                 html htm shtml;
                text/css                                   css;
                text/xml                                   xml;
                image/gif                                 gif;
                image/jpeg                               jpeg jpg;
                application/javascript               js;
                application/atom+xml               atom;
                application/rss+xml                  rss;
                text/mathml                              mml;
                text/plain                                  txt;

            }
}

Posted at Nginx Forum: https://forum.nginx.org/read.php?2,281371,281413#msg-281413

_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|

Re: Enabling "Transfer-Encoding : chunked"

Maxim Dounin
Hello!

On Wed, Sep 26, 2018 at 04:49:42AM -0400, anish10dec wrote:

> We are using Nginx with DAV Module , where encoder is pushing the content.
> These content when being accessed is not coming with header
> "Transfer-Encoding : chunked" though these header is being added by
> Encoder.

This is not going to work.  The DAV module only makes files
available once they are fully uploaded, while for the low-latency
live HLS streaming the last segment needs to be sent to the
clients while it is being produced.

As previously suggested, if you want low latency live HLS
streaming to work, you'll have to use proxying for the last
segment (the one which is being written to).

--
Maxim Dounin
http://mdounin.ru/
_______________________________________________
nginx mailing list
[hidden email]
http://mailman.nginx.org/mailman/listinfo/nginx