High latency.

[axot]

I tested this version in our large scale load test. Compare to original twitter/latency, it represents unstable high latency and caused many timeouts.

The config is same as when we using original one’s except the global section.

1 global:                                                                                                                
  2   worker_processes: 2         # num of workers, fallback to single process model while worker_processes is 0           
  3   max_openfiles: 102400       # max num of open files in every worker process                                          
  4   user: twemproxy             # user of worker's process, master process should be setup with root                     
  5   group: twemproxy            # group of worker's process                                                              
  6   worker_shutdown_timeout: 30 # terminate the old worker after worker_shutdown_timeout, unit is second                 
  7 pools:                                                                                                                 
  8   test:                                                                                                             
  9     redis: true                                                                                                        
 10     listen: 127.0.0.1:6379                                                                                        
 11     hash: fnv1a_64                                                                                                     
 12     distribution: ketama                                                                                               
 13     timeout: 400                                                                                                       
 14     backlog: 1024                                                                                                      
 15     preconnect: false                                                                                                  
 16     auto_eject_hosts: true                                                                                             
 17     server_retry_timeout: 2000                                                                                         
 18     server_failure_limit: 3                                                                                            
 19     server_connections: 1                                                                                              

[git-hulk]

hi, @axot can you dump the request/server latency from stats? it makes no sense that high latency than the twitter/twemproxy while didn't modify the core flow of proxy forward.

the timeout and server_connections is the same with the origin? by the way, reload config would spawn the new workers and lay the old workers to rest, so the connections in old workers would be closed as well, so if the client didn't check the connection is alive or not and retry, may cause socket error

[axot]

Hello, we did not import twemproxy metrics exporter at that time, this is what I can share to you, this graph show the number of PHP-FPM processes of each container. When twemproxy got high latency, process will be increased much more.

All config is same as the origin's, and we did not reload config.

[git-hulk]

Hi @axot , the twemproxy metrics were counter and histogram, you also can find some clues why the twemproxy got high latency if you haven't restarted it. The histogram of the request meant the latency from the client-side(user-latency), and histogram of server meant the latency from the Redis server(latency between the twemproxy and Redis).

the buckets were <1ms, <10ms, <20ms, <50ms, <100ms, <200ms, <500ms, <1s, <2s, <3s, +Inf

[yehaotong]

I'd like to ask how you calculate these latencies. I've always wanted to get the latencies data about Memcache. Can I get them through twoproxy statistics