PATCH method dies with mod security with nginx configured as proxy/ingress #2341
Labels
3.x
Related to ModSecurity version 3.x
Platform - Nginx
workaround available
The issue has either a temporary or permanent workaround available
Comments
PaulCharlton
changed the title
enable-modsecurity: "true" default configuration
|
open to any triage suggestions --- I realize there are several moving parts here between Nginx, modsecurity, and the Nginx-connector |
|
Additional info: Additional info: |
|
Hi @PaulCharlton , Probably the single most useful thing would be to increase your log levels. SecDebugLogLevel set to 9 will show a lot more detail about what is happening within ModSecurity. Similarly, if the error is occurring in nginx code, setting the error_log level to info, or even debug, might help. |
|
@martinhsv thanks for the insights on how to proceed. Increasing the ModSecurity debug log level to 9 did not show anything useful -- all rules appear to be passing. I will continue some data gathering with the Nginx error_log level settings. |
|
@martinhsv this may be the best clue thus far: For context:
so I am still investigating the following: (a) why does the session hang, and where in the code is it hanging? (b) what happens if I disable external auth, (c) can I increase logging on the external auth to see how the session state differs if modsecurity is on/off (d) why does breaking the session from the client side produce any additional modsecurity logging AFTER session is closed? |
|
continuing that investigation -- the entire control flow works when I disable our external auth by removing the auth related annotations from the k8s ingress resource. combining that with the info from the log above -- Nginx appears to be sending the entire request body to the auth provider -- even on a POST/PATCH/PUT request ... and all our auth provider really cares about is a couple of the request headers. |
|
@martinhsv so -- modsecurity flagging on a sub-request -- what is the control flow supposed to be? it definitely should not be hanging the session until client-side terminates. |
|
more information: Looking at the issue I identified in the log above, I went looking for the configuration for the external auth, and contrary to the modsecurity log above, the external auth location does not pass the request body ... |
|
summary of what I have observed:
I am imagining that something in the control flow nginx->modsecurity->nginx->external_auth->nginx->proxied_server is corrupting session state at the boundary where modsecurity returns to nginx. [or doing a req_read, not realizing that req is already at EOF/(content-length) boundary] |
|
Hi @PaulCharlton , Those are some good clues. In some ways it sounds a bit like something akin to bitly/oauth2_proxy#442 . I.e. if no body is sent but a Content-Length header is present with a nonzero value, then a hang could be occurring due to a waiting-for-body state. A few things you could try to further clarify what is happening: Two additional possibilities to proceed:
|
|
@martinhsv I will follow up on your ideas and let you know: |
|
@martinhsv was able to confirm that the session is hung between the modsecurity pass and the invocation of the auth sub request. Also discovered that the auth sub request completes successfully immediately after the client breaks the session. In a nutshell, when modsecurity is enabled (even with "SecRulesEngine Off"), Nginx ingests an extra byte from the client SSL connection. I am still drilling into the root cause, and I have attached a CSV of logs demonstrating the different in SSL protocol behavior. Noting also that the difference in SSL behavior occurs before the auth subRequest even begins processing, and that there is enough difference in session state with/without modsecurity that the auth subRequest performs a blocking read if and only if modsecurity is enabled. I can imagine from the attached log that modsecurity is introducing enough extra latency that Nginx "SSL Read" at line 10 of the attached trace no longer returns "-1" (ie: EAGAIN or EWOULDBLOCK). |
|
adding information: the hung session only occurs with TLS client connections |
|
Server: Nginx/modsecurity container: quay.io/kubernetes-ingress-controller/nginx-ingress-controller |
PaulCharlton
changed the title
|
adding information: the session does not hang if the POST/PUT/PATCH has an empty request body and no Content-Length header. Even a "Content-Length: 0" hangs. SecRequestBodyAccess Off (or On) does not change the outcome. |
|
spoke too soon on the last one.
for evidence below, first request worked; second request hung until CTL-C on client side. |
|
in a nutshell, the hung session boils down to whether there is an empty line after the headers to separate the headers from a content body, and the hang only occurs when modsecurity is enabled, even if all rules are disabled, and only hangs for TLS session. |
|
additional information: |
|
definitive workaround: At this point, we can write a unit test that sets conditions: (1) "POST + non-empty body", (2) side-car sub request Auth, (3) modsecurity enabled, (4) force HTTP2 + TLS session. iterate until the unit test works. |
|
there seems to be some ambiguity in the various interpretations of the HTTP2 protocol as it is translated back to NGINX request buffers ... does the blank line between the request_header and the request_body belong to the header, to the body, or as a standalone entity? I imagine we will find that modsecurity-nginx, Nginx auth subrequest, and Nginx proxy request have different views of that question. |
martinhsv
added
the
workaround available
|
We are seeing the same issue (POST with non-empty body hangs if using http2, but is fine if using http1.1). Disabling use of http2 is not an option for us. Are there plans/an eta for a fix? |
|
Over three years later, i'd like to chime in that HTTP/2 + ModSecurity + |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Version info:
version: libmodsecurity.so.3.0.3
name: nginx-ingress
repository: https://kubernetes-charts.storage.googleapis.com
version: 1.33.5
kubectl version
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.9", GitCommit:"a17149e1a189050796ced469dbd78d380f2ed5ef", GitTreeState:"clean", BuildDate:"2020-04-16T23:15:50Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
===> cross-post from kubernetes/ingress-nginx#5723
Summary Observations:
PATCH method with JSON request body sent to ingress receives no response bytes on connection and connection is left open.
Connection is closed only when NGINX ingress is bounced due to server reload (on change of ConfigMap, pod deletion ... etc)
when Debug logging is enabled with
SecDebugLog /dev/stdout
SecDebugLogLevel 4
the debug log shows that phase 1 of the Modsecurity (a) recognized the "application/json", and that (b) phase 2 assigned the JSON attributes into ARGS for subsequent scanning.
** Changing from "DetectOnly" to rule enforcement does not change the behavior. Adding the CRS ruleset does not change the behavior.
** POST and PUT methods work without hanging the connection
** PATCH method works correctly when enable-modsecurity: "false"
====================================
The text was updated successfully, but these errors were encountered: