Quantcast

Issue with 3rd-party memc and eval modules

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Issue with 3rd-party memc and eval modules

Markus Jelsma - Buyways B.V.
Hi,


There is an issue with the 3rd-party memc module for Nginx [1] where it
doesn't close the connection when it resides in an eval [2] block.


Consider the following with memc in eval:

       eval $memc_return
       {
         set $memc_cmd 'set';
         set $memc_key 'key';
         set $memc_value 'value';

         memc_pass memcache_cluster;
       }

      echo 'hello!';
      echo_flush;

When this location is called via curl, you'll get to see the following:
* About to connect() to localhost port 81 (#0)
*   Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 81 (#0)
> GET
/session/synchronize/da39a3ee5e6b4b0d3255bfef95601890afd80709/da39a3ee5e6b4b0d3255bfef95601890afd80709/abc
HTTP/1.1
> User-Agent: curl/7.19.5 (i486-pc-linux-gnu) libcurl/7.19.5 OpenSSL/0.9.8g
zlib/1.2.3.3 libidn/1.15
> Host: localhost:81
> Accept: */*
>
^C
markus@zealand:/opt/nginx#

You can see i have to close the connection manually. I have also got a verbose
memcache server log which exactly closes the connection at the point i press
CTRL+C.




Now consider the following with memcached in eval, i have changed the above
configuration to use memcached instead of memc:

* About to connect() to localhost port 81 (#0)
*   Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 81 (#0)
> GET
/session/synchronize/da39a3ee5e6b4b0d3255bfef95601890afd80709/da39a3ee5e6b4b0d3255bfef95601890afd80709/abc
HTTP/1.1
> User-Agent: curl/7.19.5 (i486-pc-linux-gnu) libcurl/7.19.5 OpenSSL/0.9.8g
zlib/1.2.3.3 libidn/1.15
> Host: localhost:81
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx
< Date: Tue, 16 Feb 2010 10:46:22 GMT
< Content-Type: text/plain
< Transfer-Encoding: chunked
< Connection: close
<
hello!
* Closing connection #0


You can clearly see what's going on: this is what it should look like. Now,
also consider using the memc module with the eval lines commented out. I also
had to comment out the echo lines because otherwise it won't pass to memcache:


* About to connect() to localhost port 81 (#0)
*   Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 81 (#0)
> GET
/session/synchronize/da39a3ee5e6b4b0d3255bfef95601890afd80709/da39a3ee5e6b4b0d3255bfef95601890afd80709/abc
HTTP/1.1
> User-Agent: curl/7.19.5 (i486-pc-linux-gnu) libcurl/7.19.5 OpenSSL/0.9.8g
zlib/1.2.3.3 libidn/1.15
> Host: localhost:81
> Accept: */*
>
< HTTP/1.1 201 Created
< Server: nginx
< Date: Tue, 16 Feb 2010 10:50:31 GMT
< Content-Type: text/plain
< Content-Length: 8
< Connection: close
<
STORED
* Closing connection #0



There is also a difference in HTTP response code and body if only using set,
here is what i found:

memc returns code=HTTP 201 body=STORED
memcached returns code=HTTP 200 body=   <-- this is empty


Can the contributers of the modules please tell us something about this
behaviour? I'm using the latest stable 0.7 nginx and modules.


Many many thanks in advance


[1]: http://wiki.nginx.org/NginxHttpMemcModule
[2]: http://www.grid.net.ru/nginx/eval.en.html

Cheers,


Markus Jelsma - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350


_______________________________________________
nginx mailing list
[hidden email]
http://nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Issue with 3rd-party memc and eval modules

Piotr Sikora
Hi Markus,
could you check if the same happens with agentzh's modified eval module?
http://github.com/agentzh/nginx-eval-module

Best regards,
Piotr Sikora < [hidden email] >


_______________________________________________
nginx mailing list
[hidden email]
http://nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Issue with 3rd-party memc and eval modules

Markus Jelsma - Buyways B.V.
Hello Piotr,


Thanks for you reply but i am already using the latest module which allows for
multiple eval blocks in a single location, works fine by the way.


Cheers,


Markus Jelsma - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350


_______________________________________________
nginx mailing list
[hidden email]
http://nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Issue with 3rd-party memc and eval modules

Piotr Sikora
Hello Markus,

> Thanks for you reply but i am already using the latest module which allows
> for
> multiple eval blocks in a single location, works fine by the way.

Yes, I'm aware of that. But the version you're using (official one) doesn't
include changes made by agentzh, hence my request for you to check if it
works with memc. Please note that this modification was made before multiple
evel blocks were added, so it will work with only one block.

Anyway, if memc will work fine with modifications made by agentzh, then you
should ask Valery to include them in official release.

Best regards,
Piotr Sikora < [hidden email] >


_______________________________________________
nginx mailing list
[hidden email]
http://nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Issue with 3rd-party memc and eval modules

Markus Jelsma - Buyways B.V.
In reply to this post by Markus Jelsma - Buyways B.V.
Hi,


There is a small mistake in the first post; the standard memcache module does
not support commands, hence it cannot be used to set variables. The issue with
the 3rd-party memc module persists though.


Cheers,


Markus Jelsma - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350


_______________________________________________
nginx mailing list
[hidden email]
http://nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Issue with 3rd-party memc and eval modules

Markus Jelsma - Buyways B.V.
In reply to this post by Piotr Sikora
Hello Piotr,


Sorry for the late reply, the ML is not very happy with me i guess for i
didn't get this in my mailbox. I've downloaded and compiled agentzh's eval
module.

What comes out:

Well, agentzh's latest eval source - directly taken from the most recent
commit - not only works with multiple eval blocks, it also neatly allows the
memc module to close it's connection to the memcache server. So it seems it is
not memc issue but has to do with the eval module.

First i used Valery's source from his github, it seems the source at the time
did allow for multiple eval blocks but could not let the connection close.

The question now is, will their come an `official` source for the eval module?
Or may we assume Valery's page [1] is the one point to download from?


Thanks Piotr for the tip!


[1]: http://www.grid.net.ru/nginx/eval.en.html


Regards,

>Yes, I'm aware of that. But the version you're using (official one) doesn't
>include changes made by agentzh, hence my request for you to check if it
>works with memc. Please note that this modification was made before multiple
>evel blocks were added, so it will work with only one block.
>
>Anyway, if memc will work fine with modifications made by agentzh, then you
>should ask Valery to include them in official release.
>
>Best regards,
>Piotr Sikora < [hidden email] >


Markus Jelsma - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350


_______________________________________________
nginx mailing list
[hidden email]
http://nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Issue with 3rd-party memc and eval modules

Piotr Sikora
Hi Markus,

> Well, agentzh's latest eval source - directly taken from the most recent
> commit - not only works with multiple eval blocks,

This must be some nice side-effect :) As far as I know agentzh intended to
make eval module work with ngx_drizzle / ngx_postgres / ngx_oracle, he
didn't mention anything about multiple blocks to me.

> it also neatly allows the
> memc module to close it's connection to the memcache server. So it seems
> it is
> not memc issue but has to do with the eval module.

Great!

> The question now is, will their come an `official` source for the eval
> module?
> Or may we assume Valery's page [1] is the one point to download from?

You mean version with agentzh's modification?
I don't think so, unless Valery will pull the changes into his repository ;)

Best regards,
Piotr Sikora < [hidden email] >


_______________________________________________
nginx mailing list
[hidden email]
http://nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Issue with 3rd-party memc and eval modules

Valery Kholodkov

> > The question now is, will their come an `official` source for the eval
> > module?
> > Or may we assume Valery's page [1] is the one point to download from?
>
> You mean version with agentzh's modification?
> I don't think so, unless Valery will pull the changes into his repository ;)

I appreciate your interest in my software, but the problem with this contributed code at the moment is that it is not as efficient as it could possibly be and it is not universal enough.

In particular I don't like the presence of "eval_subrequest_in_memory" directive at all, because it specifies which way of internal communication the module needs to use, which doesn't make any sense for end user.

Second, the contributed code allocates a buffer and uses it in order to assemble the output of subrequest in it. This allocation can be totally avoided, if an appropriate state machine is implemented.

As soon as these issues are resolved, I'll accept this code.

--
Regards,
Valery Kholodkov

_______________________________________________
nginx mailing list
[hidden email]
http://nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Issue with 3rd-party memc and eval modules

Markus Jelsma - Buyways B.V.
Sounds good!


But Valery, i previously used your new eval module that allows for multiple
blocks. That version had an issue with not letting memc_pass close the
memcached connection when setting values (getting values worked properly).

Now, for the moment, i use agentzh's eval which for some reason also allows
for multiple eval blocks and does let memc_pass close the connection.

I'd prefer to use your release if a fix could be made that closes the
connection with the memcache server.


Regards,




>I appreciate your interest in my software, but the problem with this
> contributed code at the moment is that it is not as efficient as it could
> possibly be and it is not universal enough.
>
>In particular I don't like the presence of "eval_subrequest_in_memory"
> directive at all, because it specifies which way of internal communication
> the module needs to use, which doesn't make any sense for end user.
>
>Second, the contributed code allocates a buffer and uses it in order to
> assemble the output of subrequest in it. This allocation can be totally
> avoided, if an appropriate state machine is implemented.
>
>As soon as these issues are resolved, I'll accept this code.

Markus Jelsma - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350


_______________________________________________
nginx mailing list
[hidden email]
http://nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Issue with 3rd-party memc and eval modules

Valery Kholodkov

----- Markus Jelsma <[hidden email]> wrote:

> Sounds good!
>
>
> But Valery, i previously used your new eval module that allows for multiple
> blocks. That version had an issue with not letting memc_pass close the
> memcached connection when setting values (getting values worked properly).
>
> Now, for the moment, i use agentzh's eval which for some reason also allows
> for multiple eval blocks and does let memc_pass close the connection.
>
> I'd prefer to use your release if a fix could be made that closes the
> connection with the memcache server.

I'll try to fix this problem.

>
>
> Regards,
>
>
>
>
> >I appreciate your interest in my software, but the problem with this
> > contributed code at the moment is that it is not as efficient as it could
> > possibly be and it is not universal enough.
> >
> >In particular I don't like the presence of "eval_subrequest_in_memory"
> > directive at all, because it specifies which way of internal communication
> > the module needs to use, which doesn't make any sense for end user.
> >
> >Second, the contributed code allocates a buffer and uses it in order to
> > assemble the output of subrequest in it. This allocation can be totally
> > avoided, if an appropriate state machine is implemented.
> >
> >As soon as these issues are resolved, I'll accept this code.
>
> Markus Jelsma - Technisch Architect - Buyways BV
> http://www.linkedin.com/in/markus17
> 050-8536620 / 06-50258350
>
>
> _______________________________________________
> nginx mailing list
> [hidden email]
> http://nginx.org/mailman/listinfo/nginx

--
--
Regards,
Valery Kholodkov

_______________________________________________
nginx mailing list
[hidden email]
http://nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Issue with 3rd-party memc and eval modules

Yichun Zhang (agentzh)
In reply to this post by Valery Kholodkov
On Wed, Feb 17, 2010 at 7:07 PM, Valery Kholodkov
<[hidden email]> wrote:
>
> In particular I don't like the presence of "eval_subrequest_in_memory" directive at all, because it specifies which way of internal communication the module needs to use, which doesn't make any sense for end user.
>

Thanks for replying to my pull request (although indirectly). Yeah, I
don't like the name of that directive as well. I chose that just
because I can't find a better name at that moment :P

> Second, the contributed code allocates a buffer and uses it in order to assemble the output of subrequest in it. This allocation can be totally avoided, if an appropriate state machine is implemented.
>

I just copied (almost) the way that ngx_http_upstream does buffered
outputing ;) To be honest, I didn't and don't see how that allocation
can be totally avoided for general use cases. Silly me! :P

> As soon as these issues are resolved, I'll accept this code.
>

It'll be great if you provide some more elaborate suggestions or even
patches to my patches if you do have the tuits :)

Just as Piotr mentioned ealier, the initial motivation for my ngx_eval
branch is to provide support for ngx_drizzle + ngx_rds_json. Adding
subrequest-in-memory support to every content handler/upstream handler
*with* output filter support is boring and no fun. So I chose to fix
ngx_eval rather than my content handler/upstream modules ;)

Thanks!
-agentzh

_______________________________________________
nginx mailing list
[hidden email]
http://nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Issue with 3rd-party memc and eval modules

Markus Jelsma - Buyways B.V.
In reply to this post by Valery Kholodkov
Great!! Let us know, we'll test it as soon as we can and report back on it.


Markus Jelsma - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350


_______________________________________________
nginx mailing list
[hidden email]
http://nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Issue with 3rd-party memc and eval modules

Markus Jelsma - Buyways B.V.
In reply to this post by Valery Kholodkov
Hi,


I've tested a bit further and the both agentzh and Valery's module do not
close the connection inside an interal location called by echo_subrequest. In
that internal location i try to use proxy_pass to fetch an uuid from a
backend.

Earlier i had that block of code in a larger location but due to various
module's order of execution i tried to split the parts into internal location.

I immediately got the same problem as before. I've then tried to use the other
module but it doesn't work either.

Anyway, i wonder if creating a module on my own would be easier in the end,
dealing with many conditional operators and eval blocks inside the
configuration is either quite a challenge or not feasible.


Cheers,


>I'll try to fix this problem.

Markus Jelsma - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350


_______________________________________________
nginx mailing list
[hidden email]
http://nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Issue with 3rd-party memc and eval modules

Piotr Sikora
Hi Markus,

> Anyway, i wonder if creating a module on my own would be easier in the
> end,
> dealing with many conditional operators and eval blocks inside the
> configuration is either quite a challenge or not feasible.

It would actually help if you would share with us what are you trying to
achieve ;)

Best regards,
Piotr Sikora < [hidden email] >


_______________________________________________
nginx mailing list
[hidden email]
http://nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Issue with 3rd-party memc and eval modules

Markus Jelsma - Buyways B.V.
In reply to this post by Piotr Sikora
Hi,


In the end, i need two pieces of server side logic:
1) HTTP request filter that looks up a key in memcache (specified in a
cookie). Then, an additional lookup is required and finally either the URL can
be rewritten or the original cookie can be overwritten.

In pseudo:
request.cookie.token = memcache_get(memcache_get(request.cookie.token));

Of course, some validation checks need to be executed such as verifying if the
client's IP address is actually linked to the token.

This part already works and uses two eval blocks.


2) A HTTP request with multiple parameters needs to be processed and a cookie
needs to be set (eval) which value is a random hash generated by some backend
(eval), that value then will be stored in memcache as well (eval) as a HTTP
specified token. This step is a token synchronization process that initializes
a session so it can be called using the logic above.

This second part is filled with checks, validations etc and is hard to
implement because of order of exection (eval before rewrite). But i really
need rewrite's conditional statement a lot of times. Splitting it all up in
seperate named locations is quite a challenge.

Doing all this in the config would be nice but on the other hand making a
module for it would be even better. It would be some sort of combination of
the memcache module and user_id module.


Cheers,



> It would actually help if you would share with us what are you trying to
achieve ;)

Markus Jelsma - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350


_______________________________________________
nginx mailing list
[hidden email]
http://nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Issue with 3rd-party memc and eval modules

Yichun Zhang (agentzh)
In reply to this post by Markus Jelsma - Buyways B.V.
On Thu, Feb 18, 2010 at 6:55 PM, Markus Jelsma <[hidden email]> wrote:
> I've tested a bit further and the both agentzh and Valery's module do not
> close the connection inside an interal location called by echo_subrequest.

Can you provide a minimized config snippet that can reproduce this
problem? It's getting a bit complicated it seems ;)

BTW, I'm a big fan of nginx.conf scripting myself ;)

Cheers,
-agentzh

_______________________________________________
nginx mailing list
[hidden email]
http://nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Issue with 3rd-party memc and eval modules

Markus Jelsma - Buyways B.V.
Yes, i can provide a snippet. In the end i have decided to abandon the project
of doing it in configuration. It is extremely hard to try and write imperative
code in a declarative language; although we do have the if operator [1],
working with it is more troublesome than ever before ;)

We will now try and build a module for our purpose, at least, i hope so.

Anyway, here is a snippet to reproduce a non -closing connection using eval in
a named location called by echo_exec - it gets complicated indeed.


    location /test
    {
      echo_exec @initialize;
    }

    location @initialize
    {
      eval_override_content_type 'text/plain';

      eval $id
      {
        rewrite ^(.*)$ /id;
        proxy_pass <a href="http://data-1:8080;">http://data-1:8080;
      }
    }

This will fail and also fail with a memc_pass call. However, it will work if
you replace the echo_exec call with the entire contents of the named location.

Even this little piece of code gave me a hard time; imagine how some 50 lines
using various eval blocks, proxy_pass and memc_pass directives and many evil
if operators ;)

Anyway, thanks for all the support on the way!


[1]: http://wiki.nginx.org/IfIsEvil


Regards,


On Wednesday 24 February 2010 05:26:04 agentzh wrote:
>  tested a bit further and the both agentzh and Valery's module do not
>
> > close the connection inside an interal location called by echo_subrequ
>

Markus Jelsma - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350


_______________________________________________
nginx mailing list
[hidden email]
http://nginx.org/mailman/listinfo/nginx
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Issue with 3rd-party memc and eval modules

Yichun Zhang (agentzh)
On Wed, Feb 24, 2010 at 9:52 PM, Markus Jelsma <[hidden email]> wrote:
> Yes, i can provide a snippet. In the end i have decided to abandon the project
> of doing it in configuration. It is extremely hard to try and write imperative
> code in a declarative language; although we do have the if operator [1],
> working with it is more troublesome than ever before ;)
>

Sigh. I'm sure there's some bug in one of those modules that you were
using. I'm afraid you'll also have a hard time in writing a module
yourself because the nginx's core does not have a nice API ;)

> We will now try and build a module for our purpose, at least, i hope so.
>
> Anyway, here is a snippet to reproduce a non -closing connection using eval in
> a named location called by echo_exec - it gets complicated indeed.
>

I've reproduced this connection hang using the following (slightly
modified) version of config:

   location /test
   {
     echo_exec @initialize;
   }

   location @initialize
   {
     eval_override_content_type 'text/plain';

     eval $id
     {
       #rewrite ^(.*)$ /id;
       proxy_pass <a href="http://127.0.0.1:$server_port/id;">http://127.0.0.1:$server_port/id;
     }
     echo $id;
   }
   location /id {
        echo hi;
   }

Our Test::Nginx::Socket module reports a timeout that the nginx server
does not close the client connection in time.

But I've found a work-around by avoid using named location:

   location /test
   {
     echo_exec /initialize;
   }

   location /initialize
   {
     internal;
     eval_override_content_type 'text/plain';

     eval $id
     {
       #rewrite ^(.*)$ /id;
       proxy_pass <a href="http://127.0.0.1:$server_port/id;">http://127.0.0.1:$server_port/id;
     }
     echo $id;
   }
   location /id {
        echo hi;
   }

Then GET /test gives the output as expected:

     hi

It seems to be a named location issue. According to the implementation
in the nginx core, "named location" is quite buggy. And we've run into
some other issues earlier as well. So my suggestion is to use the
"internal" directive combined with normal locations for such kind of
things.

>
> Even this little piece of code gave me a hard time; imagine how some 50 lines
> using various eval blocks, proxy_pass and memc_pass directives and many evil
> if operators ;)
>

I do understand the frustration. Without considerable knowledge to the
nginx internals, it's quite difficult to diagnose issues when things
go wrong.

> Anyway, thanks for all the support on the way!
>

You're always welcome to report detailed issues to the list and please
don't give up config file scripting ;)

>
> [1]: http://wiki.nginx.org/IfIsEvil
>

Well, the "if" directive in the http rewrite module is not that evil
as long as we know how it works. I believe the evilness of "if" has
been exaggerated. It just not works the way you're used to in most of
other languages like C and Perl.

We've been enjoying "if" in our nginx.conf for something big:

    http://agentzh.org/misc/nginx.conf

I've given some hints regarding "if"'s behavior here:

   http://forum.nginx.org/read.php?2,25197,25197

It's not that hard to predict how it works ;)

On the other hand, I'd say named locations are indeed evil because it
does not clear the request object like normal locations and may lead
to consequences that are *very* hard to predict.

Cheers,
-agentzh

_______________________________________________
nginx mailing list
[hidden email]
http://nginx.org/mailman/listinfo/nginx
Loading...