Nginx w/ModSecurity 3.0 doesn't shut down workerprocesses

[zimmerle]

@petermollerud commented on Thu Nov 09 2017

After running Nginx 1.12.1 with ModSecurity 3 for a while, we have noticed that when we issue "service nginx stop" on Redhat EL6 - Nginx is unable to shut down all of its worker processes. They must be killed off separately using "kill". This is kind of annoying and isn't how it should be.

[zimmerle]

Hi @petermollerud,

Can you check the exactly point ModSecurity is stuck? You can use gdb for that. That will help a lot.

[petermollerud]

Hi @zimmerle ,
Yes I can try to do that. The problem is that this isn't occuring all the time, so I cannot reproduce the error in a controlled fashion. However, I'll look into GDB (never used it before) and see what I can do.

[salmon5]

we encount this problem too, when run /usr/local/nginx/sbin/nginx -s reload,8 nginx workers process,sometimes has one worker process not quit,not happen every time.
CentOS 7.2 x64
nginx 1.12.2
ModSecurity-nginx latest version

[petermollerud]

I have not been able to reproduce the problem for the last week on our installation, so haven't gotten anywhere with GDB either...

[nahi]

I observed a similar problem.

Linux ip-172-18-147-255 4.4.0-1039-aws #48-Ubuntu SMP Wed Oct 11 15:15:01 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Our ModSecurity and ModSecurity-ngix are old.

• ModSecurity: e14dc602e52a9fc231352729aaea16df3b239e14
• ModSecurity-nginx: abbf2c4

I'm using /dev/null as SecAuditLog.

gdb bt.

#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007efe231e5dbd in __GI___pthread_mutex_lock (mutex=mutex@entry=0x7efe2381f008) at ../nptl/pthread_mutex_lock.c:80
#2  0x00007efe22d12d0c in modsecurity::utils::SharedFiles::write (this=<optimized out>, fileName="/dev/null",
    msg="XXXX"..., error=error@entry=0x7ffc8128a850)
    at utils/shared_files.cc:230
#3  0x00007efe22cac18b in modsecurity::audit_log::writer::Parallel::write (this=0x28775b0, transaction=0x2f1fc20, parts=<optimized out>,
    error=0x7ffc8128a850) at audit_log/writer/parallel.cc:166
#4  0x00007efe22ca9f75 in modsecurity::audit_log::AuditLog::saveIfRelevant (this=<optimized out>, transaction=transaction@entry=0x2f1fc20,
    parts=parts@entry=6006) at audit_log/audit_log.cc:296
#5  0x00007efe22c98558 in modsecurity::Transaction::processLogging (this=0x2f1fc20) at transaction.cc:1274
#6  0x00007efe22c98735 in modsecurity::msc_process_logging (transaction=<optimized out>) at transaction.cc:2102
#7  0x000000000048f269 in ngx_http_modsecurity_log_handler (r=<optimized out>)
    at /var/chef/cache/ModSecurityNginxConnector-abbf2c47f6f3205484a1a9db618e067dce213b89/src/ngx_http_modsecurity_log.c:72
#8  0x0000000000449625 in ngx_http_log_request (r=r@entry=0x1528c80) at src/http/ngx_http_request.c:3530
#9  0x000000000044ad5a in ngx_http_free_request (r=r@entry=0x1528c80, rc=rc@entry=0) at src/http/ngx_http_request.c:3477
#10 0x000000000044b54f in ngx_http_set_keepalive (r=0x1528c80) at src/http/ngx_http_request.c:2910
#11 ngx_http_finalize_connection (r=r@entry=0x1528c80) at src/http/ngx_http_request.c:2561
#12 0x000000000044be9f in ngx_http_finalize_request (r=r@entry=0x1528c80, rc=<optimized out>, rc@entry=0)
    at src/http/ngx_http_request.c:2457
#13 0x000000000045a429 in ngx_http_upstream_finalize_request (r=r@entry=0x1528c80, u=u@entry=0x1de26f0, rc=rc@entry=0)
    at src/http/ngx_http_upstream.c:4382
#14 0x000000000045b171 in ngx_http_upstream_process_request (r=r@entry=0x1528c80, u=u@entry=0x1de26f0) at src/http/ngx_http_upstream.c:3963
#15 0x000000000045b386 in ngx_http_upstream_process_upstream (r=0x1528c80, u=0x1de26f0) at src/http/ngx_http_upstream.c:3875
#16 0x000000000045a4a5 in ngx_http_upstream_handler (ev=<optimized out>) at src/http/ngx_http_upstream.c:1245
#17 0x0000000000437efb in ngx_epoll_process_events (cycle=<optimized out>, timer=<optimized out>, flags=<optimized out>)
    at src/event/modules/ngx_epoll_module.c:902
#18 0x000000000042f732 in ngx_process_events_and_timers (cycle=cycle@entry=0x141b650) at src/event/ngx_event.c:242
#19 0x0000000000435e63 in ngx_worker_process_cycle (cycle=cycle@entry=0x141b650, data=data@entry=0x3)
    at src/os/unix/ngx_process_cycle.c:749
#20 0x0000000000434900 in ngx_spawn_process (cycle=cycle@entry=0x141b650, proc=proc@entry=0x435df2 <ngx_worker_process_cycle>,
    data=data@entry=0x3, name=name@entry=0x493cb7 "worker process", respawn=respawn@entry=-4) at src/os/unix/ngx_process.c:198
#21 0x0000000000435fd3 in ngx_start_worker_processes (cycle=cycle@entry=0x141b650, n=4, type=type@entry=-4)
    at src/os/unix/ngx_process_cycle.c:358
#22 0x0000000000436eba in ngx_master_process_cycle (cycle=0x141b650, cycle@entry=0x13f81a0) at src/os/unix/ngx_process_cycle.c:243
#23 0x0000000000411f83 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:375

I'm going to update modules to the latest but it would take time to capture the symptom again. Meanwhile, let me know anything you want to investigate more.

[victorhora]

I believe this problem could be related with the one reported on issue #1927. PR #1949 addresses that and was merged at 3d2030.

Please let us know if the issue still persists with the latest code and we can investigate further. Thanks!