Integration with crowdsecurity/cs-nginx-bouncer

[Sparkxxx]

Is your feature request related to a problem? Please describe.
I would like to add protection for sites to NPM and be able to block IP's based on reputation/scenario using Crowdsecurity https://github.com/crowdsecurity which is a log analyzer and reactive firewall, something like fail2ban but in a much modern architecture and suitable for the containers world and having a distributed banlist generated from users around the world.

The question/feature request is how/where to add the https://github.com/crowdsecurity/cs-nginx-bouncer module to nginx config.
This bouncer leverages nginx lua's API, namely access_by_lua_file.
New/unknown IPs are checked against crowdsec API, and if request should be blocked, a 403 is returned to the user, and put in cache. The exact desired action can be configured in Crowdsec, like displaying captcas etc.

Describe the solution you'd like
I would like to have an easy way (example maybe) to deploy the bouncer without breaking things in npm.

Describe alternatives you've considered
None so far since it might take you minutes to offer a solution and it would take me hours to tweak things and break them :)

Additional context
Please have a look at https://crowdsec.net/ to understand what is crowdsec and how it works.
I use jc21/nginx-proxy-manager:latest for NPM and https://github.com/crowdsecurity/crowdsec/tree/master/docker for Crowdsec containers analyzing the nginx logs allready written by NPM on disk (docker map volumes ./npm_data/logs/:/var/log/nginx/ and have a quick look at acquis.yaml which tells crowdsec which logs to parse, mine looks like this):
`filenames:

• /var/log/nginx/*.log
  labels:
  type: nginx`

Thank you and hope you'd consider this as a valuable addition to NPM

[chaptergy]

Referencing #39.

[Sparkxxx]

I have read that feature request but, let me justify better my request.
As far as I know fail2ban works only on one machine so it would have to manipulate some iptables/ipsets or scripts to work over a network and this implies many moving parts. It is basically a stand alone application from before the container world and I'm not even speaking here about swarms. I know it's usefull, I'm using it but not in the container world where in my opinion it looks like a dinosaur.
Crowdsec (runs on docker and has 3.3K stars on github) on the other hand is a more modern and advanced way of achieving the same goals as with fail2ban but in a distributed infrastructure with networking, VM's, containers, barebones, detecting atacks, redirecting, banning, serving captchas, etc., with a graphical interface and we love the GUIs. Bouncers and detectors don't even have to be on the same machine/container.
Have a look at their agents/bouncers HUB https://hub.crowdsec.net/ and BLOG https://crowdsec.net/blog/ for what is already available.

My requested integration ( https://hub.crowdsec.net/author/crowdsecurity/bouncers/cs-nginx-bouncer ) is already a lua script that works in OpenResty, so it would only be, in my opinion, a matter of copy-paste in the config file of NPM and not writing a new module or script to integrate it.

Please allow me to cite from Crowdsec's website:

• CrowdSec is an open-source and collaborative EDR. Analyze behaviors, respond to attacks & share signals across the community.

• Crowdsec-agent is an open-source and lightweight software that allows you to detect peers with malevolent behaviors and block them from accessing your systems at various level (infrastructural, system, applicative).

• To achieve this, Crowdsec-agent reads logs from different sources (files, streams ...) to parse, normalize and enrich them before matching them to threats patterns called scenarios.

• Crowdsec-agent is a modular and plug-able framework, it ships a large variety of well known popular scenarios; users can choose what scenarios they want to be protected from as well as easily adding new custom ones to better fit their environment.

• Detected malevolent peers can then be prevented from accessing your resources by deploying bouncers at various levels (applicative, system, infrastructural) of your stack.

• One of the advantages of Crowdsec when compared to other solutions is its crowd-sourced aspect : Meta information about detected attacks (source IP, time and triggered scenario) are sent to a central API and then shared amongst all users.

• Thanks to this, besides detecting and stopping attacks in real time based on your logs, it allows you to preemptively block known bad actors from accessing your information system.

I understand that Crowdsec is a new tool for many but it is able to do even more than fail2ban and the learning curve is low and in the long run it would overtake f2b and all you have to do is configure it where to read the logs and it is already packed with detectors and ready to take action trough bouncers on new detected atacker or the ones downloaded from the community list.
It is a set and forget application that can be further integrated with others, being prometheus or your own application over the network trough it's api.

[chaptergy]

Yeah, it was not my intention to say this issue was the same thing as the fail2ban one, I just wanted to cross-link these issues, since this they seem mutually exclusive.

[Sparkxxx]

I didn't thought you were saying that, just wanted to clarify a little bit since with appearance of crowdsec (v0.0.1 - May 15, 2020), supporting an old tool and reinventing the wheel and wasting time building around f2b seeamd like a bad idea.
Hope you like my proposal and looking forward to see it integrated in NPM.

[luismanson]

Hello, I'll add my vote for this feature. I thinks this is very different from fail2ban in which follows the crowdsourcing approach.

Thanks!

[nisuxyz]

@Sparkxxx do you have the manual steps to do this for the time being? I'm not very familiar with crowdsec so forgive my naivete, I think I have setup crowdsec and metabse in the same dockerfile using the guide here and here, but when i test with nikto from another machine on the local network I do not see a new decision and logs don't appear to show traffic. There isn't an official docker image for the nginx bouncer either, but based on your comment it seems like there may not be a need for an extra bouncer container. Here's the setup I have so far:

version: '3'
services:
  app:
    image: 'jc21/nginx-proxy-manager:latest'
    restart: unless-stopped
    ports:
      - '80:80'
      - '81:81'
      - '443:443'
    environment:
      DB_MYSQL_HOST: "db"
      DB_MYSQL_PORT: 3306
      DB_MYSQL_USER: "npm"
      DB_MYSQL_PASSWORD: "npm"
      DB_MYSQL_NAME: "npm"
    volumes:
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt
      - logs:/var/log/nginx
  db:
    image: 'jc21/mariadb-aria:latest'
    restart: unless-stopped
    environment:
      MYSQL_ROOT_PASSWORD: 'npm'
      MYSQL_DATABASE: 'npm'
      MYSQL_USER: 'npm'
      MYSQL_PASSWORD: 'npm'
    volumes:
      - ./data/mysql:/var/lib/mysql


  #crowdsec : it will be fed nginx's logs
  #and later we're going to plug a firewall bouncer to it
  crowdsec:
    image: crowdsecurity/crowdsec:latest
    restart: always
    environment:
      #this is the list of collections we want to install
      #https://hub.crowdsec.net/author/crowdsecurity/collections/nginx
      COLLECTIONS: "crowdsecurity/nginx"
      GID: "${GID-1000}"
    depends_on:
      - 'app'
    volumes:
      - ./crowdsec/acquis.yaml:/etc/crowdsec/acquis.yaml
      - logs:/var/log/nginx
      - crowdsec-db:/var/lib/crowdsec/data/
      - crowdsec-config:/etc/crowdsec/
  
  #metabase, because security is cool, but dashboards are cooler
  dashboard:
    #we're using a custom Dockerfile so that metabase pops with pre-configured dashboards
    build: ./crowdsec/dashboard
    restart: always
    ports:
      - 3000:3000
    environment:
      MB_DB_FILE: /data/metabase.db
      MGID: "${GID-1000}"
    depends_on:
      - 'crowdsec'
    volumes:
      - crowdsec-db:/metabase-data/

volumes:
  logs:
  crowdsec-db:
  crowdsec-config:

[klausagnoletti]

I am head of community at CrowdSec (and an avid user myself). I think it's a great idea to add support for CrowdSec as well. Also I would like to offer my assistance and a way into the CrowdSec dev team if you need any help implementing it.

I would also like to point any technical questions regarding CrowdSec to our discourse where all devs are available to help out.

@nisargjoshi95 Did you ever get this working? If not, feel free to ask on the discourse :-)

[klausagnoletti]

@nisargjoshi95 Did you ever get this working? If not, feel free to ask on the discourse :-)

[klausagnoletti]

@Sparkxxx please give me a buzz on the discourse, on twitter @klausagnoletti or send me a mail [email protected]. I'd like to ask about your experience with CrowdSec etc.

[baudneo]

I am here to cast my vote, I haven't started looking in depth yet but does anyone have crowdsec setup with the bouncer and NPM?

@klausagnoletti - Should I just go to the discourse and ask there or are there any crowdsec docs on how to set it all up while utilizing NPM? Thanks.

[2Wanderer]

@nisargjoshi95

I'll think your docker-compose wont work because nginx proxymanager is storing it's logs in other than the default nginx folders.

change your acquis.yml to - /var/log/nginx/*.log and then try this docker-compose: it reads the logs folder which is mounted by nginx-proxymananger container. it will read the logs but the parsing isn't that good

services:
 #crowdsec : it will be fed nginx's logs
 #and later we're going to plug a firewall bouncer to it
 crowdsec:
   image: crowdsecurity/crowdsec:v1.2.1
   restart: always
   environment:
     #this is the list of collections we want to install
     #https://hub.crowdsec.net/author/crowdsecurity/collections/nginx
     COLLECTIONS: "crowdsecurity/nginx"
     GID: "${GID-1000}"
  
   volumes:
     - ./crowdsec/acquis.yaml:/etc/crowdsec/acquis.yaml
     - /fullpath-to-nginx-proxymanager/data/logs:/var/log/nginx
     - crowdsec-db:/var/lib/crowdsec/data/
     - crowdsec-config:/etc/crowdsec/
   networks:
     crowdsec_test:
       ipv4_address: 172.20.0.4
  #metabase, because security is cool, but dashboards are cooler
 dashboard:
   #we're using a custom Dockerfile so that metabase pops with pre-configured dashboards
   build: ./crowdsec/dashboard
   restart: always
   ports:
     - 3000:3000
   environment:
     MB_DB_FILE: /data/metabase.db
     MGID: "${GID-1000}"
   depends_on:
     - 'crowdsec'
   volumes:
     - crowdsec-db:/metabase-data/
   networks:
     crowdsec_test:
       ipv4_address: 172.20.0.5

volumes:
 logs:
 crowdsec-db:
 crowdsec-config:

networks:
 crowdsec_test:
   ipam:
     driver: default
     config:
       - subnet: 172.20.0.0/24




INFO[26-11-2021 04:58:54 PM] Buckets Metrics:
+--------------------------------------+---------------+-----------+--------------+--------+---------+
|                BUCKET                | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |
+--------------------------------------+---------------+-----------+--------------+--------+---------+
| crowdsecurity/http-bad-user-agent    | -             | -         |            2 |      2 |       2 |
| crowdsecurity/http-crawl-non_statics | -             | -         |           77 |     77 |      77 |
+--------------------------------------+---------------+-----------+--------------+--------+---------+
INFO[26-11-2021 04:58:54 PM] Acquisition Metrics:
+---------------------------------------------+------------+--------------+----------------+------------------------+
|                   SOURCE                    | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+---------------------------------------------+------------+--------------+----------------+------------------------+
| file:/var/log/nginx/default-host_access.log |         37 |           30 |              7 |                     32 |
| file:/var/log/nginx/fallback-access.log     |         34 | -            |             34 | -                      |
| file:/var/log/nginx/fallback_error.log      |         23 |           23 | -              |                     20 |
| file:/var/log/nginx/proxy-host-1_access.log |        216 | -            |            216 | -                      |
| file:/var/log/nginx/proxy-host-1_error.log  |         28 |           28 | -              |                     27 |
+---------------------------------------------+------------+--------------+----------------+------------------------+
INFO[26-11-2021 04:58:54 PM] Parser Metrics:
+--------------------------------+------+--------+----------+
|            PARSERS             | HITS | PARSED | UNPARSED |
+--------------------------------+------+--------+----------+
| child-crowdsecurity/http-logs  |  243 |    100 |      143 |
| child-crowdsecurity/nginx-logs |  646 |     81 |      565 |
| crowdsecurity/dateparse-enrich |   81 |     81 | -        |
| crowdsecurity/geoip-enrich     |   30 |     30 | -        |
| crowdsecurity/http-logs        |   81 |     14 |       67 |
| crowdsecurity/nginx-logs       |  338 |     81 |      257 |
| crowdsecurity/non-syslog       |  338 |    338 | -        |
| crowdsecurity/whitelists       |   81 |     81 | -        |
+--------------------------------+------+--------+----------+
INFO[26-11-2021 04:58:54 PM] Local Api Metrics:
+----------------------+--------+------+
|        ROUTE         | METHOD | HITS |
+----------------------+--------+------+
| /v1/alerts           | GET    |    1 |
| /v1/decisions/stream | GET    | 7021 |
| /v1/watchers/login   | POST   |    3 |
+----------------------+--------+------+
INFO[26-11-2021 04:58:54 PM] Local Api Machines Metrics:
+-----------+------------+--------+------+
|  MACHINE  |   ROUTE    | METHOD | HITS |
+-----------+------------+--------+------+
| localhost | /v1/alerts | GET    |    1 |
+-----------+------------+--------+------+
INFO[26-11-2021 04:58:54 PM] Local Api Bouncers Metrics:
+---------------------+----------------------+--------+------+
|       BOUNCER       |        ROUTE         | METHOD | HITS |
+---------------------+----------------------+--------+------+
| HostFirewallBouncer | /v1/decisions/stream | GET    | 7021 |
+---------------------+----------------------+--------+------+

[jakern]

@baudneo did this work for you? I would have thought there would be issues with NPM's custom log format.

[klausagnoletti]

@klausagnoletti - Should I just go to the discourse and ask there or are there any crowdsec docs on how to set it all up while utilizing NPM? Thanks.

Sorry for my late reply. I don't get notifications from Github even though I enabled it :-/

Just go to the Discorse and ask. To my knowledge noone has tried integrating with npm yet.

[baudneo]

@jakern, it is a strange situation. If NPM is already running, proxying requests and I start up crowdsec with the newest config that @2Wanderer posted, everything is ok. When I reboot, NPM no longer will come up, citing an error binding to port :80 and :443 due to the crowdsec nginx bouncer using port :80 or :443. I am playing around trying to get things to mesh properly. Once I get that part stable I will start trying to integrated the logs structure.

[klausagnoletti]

I tried creating a docker container extending the npm container and installing crowdsec-nginx-bouncer. It didn't work. Turns out npm doesn't use nginx but openresty, wghich we don't (yet) support. But it's on the roadmap. So I guess it'a a bit of a PITA to get working before then.

[leon1995]

Can you give a rough estimation when this will probably happen?

[klausagnoletti]

No, it is not scheduled. Maybe the community will contribute it. It's open source after all :-)

[leon1995]

I already tried it with a custom dockerfile:

FROM jc21/nginx-proxy-manager:latest

RUN curl -s https://packagecloud.io/install/repositories/crowdsec/crowdsec/script.deb.sh | bash
RUN apt install -y crowdsec crowdsec-nginx-bouncer
RUN sed -i '1s/^/filenames:\n - \/data\/logs\/*.log\nlabels:\n  type:nginx/' /etc/crowdsec/acquis.yaml

but I get some weird errors. here are some of these:

Unpacking crowdsec-nginx-bouncer (0.0.7) ...
Setting up lua-logging (1.3.0-1) ...
Setting up libxpm4:amd64 (1:3.5.12-1) ...
Setting up nginx-common (1.14.2-2+deb10u4) ...

Configuration file '/etc/nginx/mime.types'
 ==> File on system created by you or by a script.
 ==> File also in package provided by package maintainer.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** mime.types (Y/I/N/O/D/Z) [default=N] ? dpkg: error processing package nginx-common (--configure):
 end of file on stdin at conffile prompt
Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-4) ...
dpkg: dependency problems prevent configuration of nginx-full:
 nginx-full depends on nginx-common (= 1.14.2-2+deb10u4); however:
  Package nginx-common is not configured yet.

dpkg: error processing package nginx-full (--configure):
 dependency problems - leaving unconfigured
Setting up libjbig0:amd64 (2.1-3.1+b2) ...
Setting up libicu63:amd64 (63.1-6+deb10u2) ...
Setting up lua-sql-sqlite3:amd64 (2.3.4-1+b1) ...
dpkg: dependency problems prevent configuration of libnginx-mod-http-xslt-filter:
 libnginx-mod-http-xslt-filter depends on nginx-common (= 1.14.2-2+deb10u4); however:
  Package nginx-common is not configured yet.

dpkg: error processing package libnginx-mod-http-xslt-filter (--configure):
 dependency problems - leaving unconfigured
Setting up libjpeg62-turbo:amd64 (1:1.5.2-2+deb10u1) ...
dpkg: dependency problems prevent configuration of libnginx-mod-http-auth-pam:
 libnginx-mod-http-auth-pam depends on nginx-common (= 1.14.2-2+deb10u4); however:
  Package nginx-common is not configured yet.

dpkg: error processing package libnginx-mod-http-auth-pam (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of libnginx-mod-http-geoip:
 libnginx-mod-http-geoip depends on nginx-common (= 1.14.2-2+deb10u4); however:
  Package nginx-common is not configured yet.

dpkg: error processing package libnginx-mod-http-geoip (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of libnginx-mod-http-ndk:
 libnginx-mod-http-ndk depends on nginx-common (= 1.14.2-2+deb10u4); however:
  Package nginx-common is not configured yet.

maybe you can help? or is this the error you mentioned, that not the real nginx but openresty is used?

[klausagnoletti]

This is the error I mentioned. Openresty != nginx (and they're not compatible). So the .deb complains that dependencies are not installed. So it won't work, unfortunately.

[klausagnoletti]

Anyways I have good news @leon1995. I asked the developers just to be sure. And it turns out they're working on it as we speak. Only a new bouncer is needed as logfiles are the same, apparently. I'll ping you here once it's out in some usable shape or form. Are you willing to do a beta test?

[sanderdatema]

@klausagnoletti I'm running on unRaid with NPM and willing to beta test if you need more people. I have to be honest: I have a pretty private server, so there won't be many visitors. I can imagine you'd rather have a beta tester with some spicey logs.

[klausagnoletti]

I can imagine you'd rather have a beta tester with some spicey logs.

Not gonna lie: That would be more interesting. But it won't prevent you to join the fun :-)

[leon1995]

@klausagnoletti I would also join the beta. However, mine is also a private server with multiple services and just a few people using it (maybe up to ten I think)

[sanderdatema]

So... @leon1995 and I will visit each other's servers like crazy til steam's coming out of our logs. Will that work for you, @klausagnoletti? 🤪

[klausagnoletti]

Sure, hit me up at klaus (at) crowdsec (dot) net. You'd have to do a Docker container on your own (but would be happy to help :-)

[leon1995]

I sent you an email

[dhernan3]

I can help testing if you wish. I have a host with Dietpi for x86, several dockers containers, one with Nginx proxy manager, other with crowdsec with the logs from Nginx proxy manager logs mapped, and also the firewall bouncer installed on the host and connected to Crowdsec's docker container.
I have installed Crowdsec last week, so my knowledge about it is very limited.

[klausagnoletti]

I can help testing if you wish. I have a host with Dietpi for x86, several dockers containers, one with Nginx proxy manager, other with crowdsec with the logs from Nginx proxy manager logs mapped, and also the firewall bouncer installed on the host and connected to Crowdsec's docker container. I have installed Crowdsec last week, so my knowledge about it is very limited.

Cool, please send me an email so I have your information in my inbox :-)

[klausagnoletti]

Now there's interesting news! @dhernan3 @leon1995 @sanderdatema @baudneo @2Wanderer @nisargjoshi95 @Sparkxxx

https://docs.crowdsec.net/docs/bouncers/openresty/

The bouncer is out. It should be easy to extend the existing npm Dockerfile with that since it's available as a .deb and npm is based on Debian.
For log parsing and scenarios the normal nginx collection can be used. Contrary to my former believe, the log format is indeed the same. Find it here: https://hub.crowdsec.net/author/crowdsecurity/collections/nginx

Let me know what comes out of experimenting with it :-)

[klausagnoletti]

Will this also work with the CrowdSec agent installed directly on Docker host?

[baudneo]

Will this also work with the CrowdSec agent installed directly on Docker host?

Yes just edit the bouncer config file and put the ip:port of the docker host in the API_URL. Do not use "localhost:8080" use the actual IP for your docker host. If you have any issues when starting the container issue docker logs --follow <container name> to see what the logs are saying. I also recommend tail /path/to/data/logs/*_error.log to see if any Lua or crowdsec errors are in those logs.

If you mean using docker networks I am not sure as I do not have my system setup that way. I am sure there is a way to make that work.

I am currently building the multiarch image base images and will be pushing multi arch images soon.

[klausagnoletti]

Thanks. I am using a Docker bridged network totally standardized. If one configures the agent to listen on 0.0.0.0 it can be reached from the container using it's docker network ip. Remember to firewall the port so it's not internet exposed (which reminds me that I asked the devs if there's a CrowdSec parser + scenario for CrowdSec. There should be. Very meta :-)

[erdoukki]

I just ugrade my NPM with "baudneo/nginx-proxy-manager:cs-modsec" on my OpenWrt server (with NextCloud).
so all broken now :

stderr: standard_init_linux.go:228: exec user process caused: exec format error

# uname -ar
Linux STARGATE 5.4.124 #0 SMP Sun Jun 13 22:02:19 2021 aarch64 GNU/Linux

Back to the 'jc21/nginx-proxy-manager:latest'
okay again...

My own use for Nginx-Proxy-Manager is to check the logs externally from the dockerized side !
just with:

    volumes:
      - /srv/NGINX/data/nginx-proxy-manager:/data
      - /srv/NGINX/letsencrypt:/etc/letsencrypt
      - /srv/NGINX/logs:/var/log/nginx

ad in the main OpenWrt host :

$ cat /etc/crowdsec/acquis.d/npm.yaml 

# NginxProxyManager
filenames:
#  - ~/data/logs/*.log
# Dockerized NGINX PROXY MANAGER
  - /srv/NGINX/data/nginx-proxy-manager/logs/*.log
labels:
  type: nginx-proxy-manager

with intalled collection : crowdsecurity/nginx-proxy-manager

[gatesry]

@erdoukki - What's the CPU architecture for your OpenWRT server?

[erdoukki]

architecture : aarch64
cortex-a53 arm64
Marvell armada37xx

[gatesry]

I don't see an ARM build for baudneo/nginx_proxy_manager on dockerhub . Which is most likely causing your issue. You could try editing the Dockerfile to build an ARM image or ask the owner to include ARM architecture.

[erdoukki]

Good idea.
Thanks.

[erdoukki]

#1875 (comment)

This is an automated message from CI:

Docker Image for build 5 is available on DockerHub as jc21/nginx-proxy-manager:github-pr-1875

Note: ensure you backup your NPM instance before testing this PR image! Especially if this PR contains database changes.

I am trying with this "PR"... ;-)

[erdoukki]

Working !
But the dockerhub image from the PR do not include the modifications for CrowdSec integration.
Bad attempt.
Badly the actual PR do not have any checks from CI which to take dockerhub for tests on arm64 !
@baudneo Why you're modifications proposal do not include arm64 ?
I check quickly your code but I still don't find the reason.

[baudneo]

@erdoukki I have multi-arch built now. ModSec is installed and loaded by default. User needs to configure it to start detecting though. See the PR for config. #1867

docker pull baudneo/nginx-proxy-manager:latest

[erdoukki]

Thanks!

Just tested;

root@STARGATE:~# cscli bouncers add docker-crowdsec-npm-bouncer
Api key for 'docker-crowdsec-npm-bouncer':

   xxxx

Please keep this key since you will not be able to retrieve it!

/srv/NGINX/data/nginx-proxy-manager/crowdsec/crowdsec-openresty-bouncer.conf

...
API_KEY=xxxx
...

docker-compose down
docker-compose up -d

root@STARGATE:/srv/NGINX# docker-compose restart
Restarting npm-crowdsec ... done
root@STARGATE:/srv/NGINX# cscli bouncers list
---------------------------------------------------------------------------------------------------------------------------
 NAME                         IP ADDRESS  VALID  LAST API PULL         TYPE                       VERSION                  
---------------------------------------------------------------------------------------------------------------------------
 crowdsec-firewall-bouncer    ::1         ✔️      2022-03-02T09:42:38Z  crowdsec-firewall-bouncer  v0.0.21-openwrt-0.0.21-3 
 docker-crowdsec-npm-bouncer              ✔️      2022-03-02T09:53:29Z                                                      
---------------------------------------------------------------------------------------------------------------------------

but still issue :

time="02-03-2022 10:55:57" level=error msg="auth api key error: select bouncer: ent: bouncer not found: unable to query"

[erdoukki]

Okay, just rebuild the docker and now works fine:

INFO[02-03-2022 01:56:04 PM] Local Api Bouncers Metrics:                  
+-----------------------------+----------------------+--------+------+
|           BOUNCER           |        ROUTE         | METHOD | HITS |
+-----------------------------+----------------------+--------+------+
| crowdsec-firewall-bouncer   | /v1/decisions/stream | GET    | 1067 |
| docker-crowdsec-npm-bouncer | /v1/decisions/stream | GET    |   37 |
+-----------------------------+----------------------+--------+------+

[klausagnoletti]

I have added ModSecurity to a new image with CrowdSec

@baudneo Where can I submit issues for your fork? I have two:

1. Specifying enviromental variables like this doesn't work. The bouncer is not started unless I edit the crowdsec-openresty-bouncer.conf file.

    environment:
      CROWDSEC_BOUNCER: "1"
      CROWDSEC_LAPI: "http://1xxxx:8080"
      CROWDSEC_KEY: "xxxx"

2. There's a memory leak in the current openresty bouncer that currently is fixed with a workaround. Will you implement it in your fork?

Lastly I have a suggestion: Could you implement more environmental variables so everything in the bouncer configuration can configured like this. Especially the reCAPTCHA stuff..

Thanks - keep up the good work!

[h4llm3n]

I have added ModSecurity to a new image with CrowdSec

Now CrowdSec and ModSecurity are installed and working. The new image that has modsec is docker pull baudneo/nginx-proxy-manager:cs-modsec. I installed OWASP-CRS for modsec. The modsec audit log is in /data/logs/modsec_audit.log by default, you can turn it off by editing /data/modsec/modsecurity.conf.

NOTE

• All config for modsec is in /data/modsec which is symbolically linked to /etc/nginx/modsec
• The OWASP rules are in /data/modsec/ruleset/
• The main rules file is /data/modsec/main.conf which just Includes the OWASP ruleset.
• The only ENV VAR for modsec is MODSEC_CREATE which will force create the directory structure into /data/modsec. This is usually only run on the first run of the modsec image when this directory does not exist. this ENV VAR will allow you to force it for whatever reason.

To enable modsec you need to add these 2 directives to the config somewhere.

modsecurity on;
modsecurity_rules_file /etc/nginx/modsec/main.conf;

Here are some pointers.

• To enable modsec for ALL HTTP hosts put the above directives in /data/nginx/custom/http_top.conf.
• To enable modsec for ALL STREAM hosts put the above directives in /data/nginx/custom/stream.conf.
• To enable modsec for ONE certain HTTP host put the above directives in the 'Advanced' tab configuration.
• To enable modsec for ONE certain HTTP host in a certain LOCATION put the above directives in the location {} block instead of outside of it in the 'Advanced' tab config.
• To enable on a per-stream basis you need to enable the streams in the dashboard and then edit the stream file by hand to add those directives into the stream {} block you want.

Some logs from modsec_audit.log to confirm modsecurity is working. This was a test using ?exec=/bin/bash in the URI.

---C3KCgGTx---F--
HTTP/2.0 403
Server: nginx
Date: Tue, 22 Feb 2022 06:16:02 GMT
Content-Length: 552
Content-Type: text/html
Connection: close
Strict-Transport-Security: max-age=63072000;includeSubDomains; preload

---C3KCgGTx---H--
ModSecurity: Warning. Matched "Operator `PmFromFile' with parameter `unix-shell.data' against variable `ARGS:exec' (Value: `/bin/bash' ) [file "/etc/nginx/modsec/ruleset/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf"] [line "480"] [id "932160"] [rev ""] [msg "Remote Command Execution: Unix Shell Code Found"] [data "Matched Data: bin/bash found within ARGS:exec: /bin/bash"] [severity "2"] [ver "OWASP_CRS/3.3.2"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-shell"] [tag "platform-unix"] [tag "attack-rce"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/152/248/88"] [tag "PCI/6.5.2"] [hostname "172.18.0.4"] [uri "/zm/"] [unique_id "1645510562"] [ref "o1,8v14,9t:urlDecodeUni,t:cmdLine,t:normalizePath,t:lowercase"]
ModSecurity: Access denied with code 403 (phase 2). Matched "Operator `Ge' with parameter `5' against variable `TX:ANOMALY_SCORE' (Value: `5' ) [file "/etc/nginx/modsec/ruleset/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "80"] [id "949110"] [rev ""] [msg "Inbound Anomaly Score Exceeded (Total Score: 5)"] [data ""] [severity "2"] [ver "OWASP_CRS/3.3.2"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"] [hostname "172.18.0.4"] [uri "/zm/"] [unique_id "1645510562"] [ref ""]

I don't know JavaScript so I am unable to add to the UI to make things pretty. Ideally there would be a ModSecurity tab that has a toggle switch for on/off and a text input box for the rules file. This tab would be in the proxyhost modal where you set what domain/target host/target port etc. It would be nice to have some sort of UI for corwdsec as well to turn it off and on and to edit the config file. I don't think having a rule editor for modsec would matter much, but it would be nice to have it.

I might take the time to teach myself some JS just to implement this to make it all pretty. Things are working for CrowdSec and ModSecurity though. The people who want that functionality should not have an issue with configuring things from the CLI anyways, for now.

Do I still have to add the log path to /etc/crowdsec/acquis.yaml here?

[klausagnoletti]

Awesome! Thanks for updating it!

[luismanson]

Hello baudneo, thanks for this new image. I've tried to update, however docker can't pull the image, it complains about missing manifest. I've verified docker hub and think the image is not updated, is that ok?

$ docker pull baudneo/nginx-proxy-manager:cs-modsec Error response from daemon: manifest for baudneo/nginx-proxy-manager:cs-modsec not found: manifest unknown: manifest unknown

[shanelord01]

Any chance this is being merged back into the base jc21/nginx-proxy-manager image ?

[baudneo]

ModSec is EoL now/soon and also has known memory leaks which are HUGE, so I won't be building images with it. I am currently working on ZoneMinder ML object detection stuff, as soon as I am done that I will create a new image with current NPM and add crowdsec openresty bouncer into it.

I think the options for WAF are slim, coraza and open-appsec which I may or may not look into adding either of them into an NPM image.

Coraza parses modsec security language, so the rules files like OWASP (who develop coraza as well) lists can be used with it. Furthermore, Coraza seems to need to be implemented in a Go language app. It is not as simple as download, point it at configs/rules and run it, someone needs to write a Go app using coraza libraries. That may have changed as I have not kept up with Coraza.

open-appsec uses ML instead of signatures for threat detection which is neat but, I can't speak to its effectiveness. At the time when I realized ModSec had issues, open-appsec did not have a nginx connector, which I think they do now. They require you to have a cloud control panel of sorts, so IDK how feasible it is to integrate into NPM if you can't do a stand alone install. Again, things may have changed as I haven't kept up on open-appsec either.

I stopped using NPM in favour of Cloudflare tunnels but, CF's new browser security check stuff is causing me grief so, I will need to move back to NPM.

[SeeJayEmm]

Is there any hope of this being revived? I would love to have the option of integrating Crowdsec with NPM.

[domodial]

@baudneo

I have the following problem. When I want to login to npm and change the email as it asks me, the response is Forbidden.

If I want to create a new user, I can't apply rights or change the password = Forbidden.
I saw this problem in a very old version of npm without having been confronted with the problem.
But I read it on the web.

I have a working npm instance which I temporarily shut down, Crowdesc installed with no errors.
The changes made in the conf files are ok, I created a separate base for this fork baudneo/nginx-proxy-manager:latest in mariadb.

My concern is only on the change of email and password.
So I tell myself that further on it might not be so good as well.

Note : The correct permissions are applied to the folders in your fork. (I'm under unraid on Dell PowerEdge)

[github-actions]

Issue is now considered stale. If you want to keep it open, please comment 👍

[caseyjhol]

👍

[vincent1890]

up news ?

[gitmotion]

please integrate this into npm base image, shouldn't be hard to keep up and massive help for those using npm

[pacs1969]

Hi!

I'm trying without success to integrate the crowdsec with NPM in my docker setup?
Any tutorial, video or documentation about?
Thanks in advance!

Paco

[LePresidente]

There is a blog post how to set it up.

https://www.crowdsec.net/blog/crowdsec-with-nginx-proxy-manager