Nginx is a free, open-source, and widely used web server. It is essential to secure the Nginx web server to defend it from DDoS and other attacks, and ModSecurity can help.

ModSecurity is a free and open-source web application firewall used to protect web servers from external threats. It is used by many websites, cPanel, and other hosting control panels to protect their web applications and servers from SQL injection, cross-site scripting, and local file inclusion attack. Originally it was designed to secure only the Apache web servers. Now it can also work with the Nginx web server.

In this post, we will show you how to install ModSecurity with Nginx on Oracle Linux 10.

Step 1 – Install Required Dependencies

First, you will need to install the required dependencies needed to compile Nginx with ModSecurity. You can install all of them with the following command:

dnf install gcc-c++ flex bison curl-devel curl zlib-devel autoconf automake git curl make libxml2-devel pkgconfig libtool httpd-devel redhat-rpm-config git wget openssl openssl-devel vim pcre2 pcre2-devel

Step 2 – Install ModSecurity On Oracle Linux 10

First, download the latest version of ModSecurity from the Git Hub repository using the following command:

git clone --depth 1 -b v3/master --single-branch https://github.com/SpiderLabs/ModSecurity

Once the download is completed, navigate to the ModSecurity directory and install other modules with the following command:

cd ModSecurity
git submodule init
git submodule update

Next, compile and install ModSecurity with the following command:

./build.sh
./configure
make
make install

Step 3 – Compile Nginx with LibModsecurity Support

In order to enable LibModsecurity support in Nginx, you will need to compile Nginx with support for LibModsecurity.

First, download the ModSecurity-nginx connector using the following command:

cd /root
git clone https://github.com/SpiderLabs/ModSecurity-nginx.git

Next, download the latest stable version of Nginx with the following command:

wget http://nginx.org/download/nginx-1.26.3.tar.gz

Next, extract the downloaded file using the following command:

tar xzf nginx-1.26.3.tar.gz

Next, create an Nginx user with the following command:

useradd -r -M -s /sbin/nologin -d /usr/local/nginx nginx

Next, change the directory to the Nginx source and compile it using the following command:

cd nginx-1.26.3
./configure --user=nginx --group=nginx --with-pcre-jit --with-debug --with-http_ssl_module --with-http_realip_module --add-module=/root/ModSecurity-nginx

Next, install it using the following command:

make
make install

Next, copy a sample ModSecurity configuration file and Unicode mapping file with the following command:

cp /root/ModSecurity/modsecurity.conf-recommended /usr/local/nginx/conf/modsecurity.conf
cp /root/ModSecurity/unicode.mapping /usr/local/nginx/conf/

Next, back up the Nginx configuration file:

cp /usr/local/nginx/conf/nginx.conf{,.bak}

Next, edit the Nginx configuration file with the following command:

nano /usr/local/nginx/conf/nginx.conf

Remove all lines and add the following lines:

user  nginx;
worker_processes  1;
pid        /run/nginx.pid;
events {
    worker_connections  1024;
}
http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
    server {
        listen       80;
        server_name  nginx.example.com;
        modsecurity  on;
        modsecurity_rules_file  /usr/local/nginx/conf/modsecurity.conf;
        access_log  /var/log/nginx/access_unix-demo.log;
        error_log  /var/log/nginx/error_unix-demo.log;
        location / {
            root   html;
            index  index.html index.htm;
        }
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

Save and close the file, then create an Nginx log directory:

mkdir /var/log/nginx

Step 4 – Create a Systemd Service File for Nginx

Next, you will need to create a systemd service file to manage the Nginx service. You can create it using the following command:

nano /etc/systemd/system/nginx.service

Add the following lines:

[Unit]
Description=The nginx HTTP and reverse proxy server
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/bin/rm -f /run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t
ExecStart=/usr/sbin/nginx
ExecReload=/bin/kill -s HUP $MAINPID
KillSignal=SIGQUIT
TimeoutStopSec=5
KillMode=mixed
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Save and close the file, then create a symlink of Nginx binary using the following command:

ln -s /usr/local/nginx/sbin/nginx /usr/sbin/

Next, reload the systemd daemon to apply the changes:

systemctl daemon-reload

Next, start the Nginx service and enable it to start at system reboot:

systemctl enable --now nginx

You can check the status of Nginx with the following command:

systemctl status nginx

You will get the following output:

ā—nginx.service - The nginx HTTP and reverse proxy server
Loaded: loaded (/etc/systemd/system/nginx.service; enabled; preset: disabled)
Drop-In: /etc/systemd/system/nginx.service.d
ā””ā”€php-fpm.conf
Active: active (running) since Fri 2025-12-05 07:16:52 EST; 3s ago
Invocation: c15f63e1b9144b2282e75a1f396cc611
Process: 83457 ExecStartPre=/usr/bin/rm -f /run/nginx.pid (code=exited, status=0/SUCCESS)
Process: 83458 ExecStartPre=/usr/sbin/nginx -t (code=exited, status=0/SUCCESS)
Process: 83461 ExecStart=/usr/sbin/nginx (code=exited, status=0/SUCCESS)
Main PID: 83462 (nginx)
Tasks: 2 (limit: 24812)
Memory: 4.5M (peak: 4.6M)
CPU: 52ms
CGroup: /system.slice/nginx.service
ā”œā”€83462 "nginx: master process /usr/sbin/nginx"
ā””ā”€83463 "nginx: worker process"

Step 5 – Enable ModSecurity Rule

By default, ModSecurity has been configured for detection-only mode, so you must enable the ModSecurity rule in the modsecurity.conf file.

To enable it, run the following command:

sed -i 's/SecRuleEngine DetectionOnly/SecRuleEngine On/' /usr/local/nginx/conf/modsecurity.conf

Also enable the audit log with the following command:

sed -i 's#/var/log/modsec_audit.log#/var/log/nginx/modsec_audit.log#' /usr/local/nginx/conf/modsecurity.conf

Step 6 – Install OWASP ModSecurity Core Rule Set

OWASP provides generic attack detection rules for ModSecurity, so it is recommended to download and integrate with ModSecurity for better security.

git clone https://github.com/SpiderLabs/owasp-modsecurity-crs.git /usr/local/nginx/conf/owasp-crs

Next, rename the OWASP rule configuration file using the following command:

cp /usr/local/nginx/conf/owasp-crs/crs-setup.conf{.example,}

Next, define the OWASP rule in the ModSecurity configuration file:

echo -e "Include owasp-crs/crs-setup.conf\nInclude owasp-crs/rules/*.conf" >> /usr/local/nginx/conf/modsecurity.conf

Next, edit the reputation config file.

nano /usr/local/nginx/conf/owasp-crs/rules/REQUEST-910-IP-REPUTATION.conf

Remove the following lines:

SecRule TX:HIGH_RISK_COUNTRY_CODES "!@rx ^$" \
    "id:910100,\
    phase:2,\
    block,\
    t:none,\
    msg:'Client IP is from a HIGH Risk Country Location',\
    logdata:'%{MATCHED_VAR}',\
    tag:'application-multi',\
    tag:'language-multi',\
    tag:'platform-multi',\
    tag:'attack-reputation-ip',\
    tag:'paranoia-level/1',\
    ver:'OWASP_CRS/3.2.0',\
    severity:'CRITICAL',\
    chain"

    SecRule TX:REAL_IP "@geoLookup" \
        "chain"
        SecRule GEO:COUNTRY_CODE "@within %{tx.high_risk_country_codes}" \
            "setvar:'tx.anomaly_score_pl1=+%{tx.critical_anomaly_score}',\
            setvar:'ip.reput_block_flag=1',\
            setvar:'ip.reput_block_reason=%{rule.msg}',\
            expirevar:'ip.reput_block_flag=%{tx.reput_block_duration}'"

Next, restart the Nginx service to apply the changes:

systemctl restart nginx

Step 7 – Verify ModSecurity

At this point, Nginx is installed and configured with ModSecurity support. Now, it’s time to test it.

Execute the code injection using the curl command:

curl localhost/index.html?exec=/bin/bash

If ModSecurity is working fine, you should get the “403 Forbidden” error as shown below:

<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx/1.19.10</center>
</body>
</html>

You can also verify the ModSecurity log for more information:

tail -100 /var/log/nginx/modsec_audit.log

You should see the following output:

ModSecurity: Warning. Matched "Operator `PmFromFile' with parameter `unix-shell.data' against variable `ARGS:exec' (Value: `/bin/bash' ) [file "/usr/local/nginx/conf/owasp-crs/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf"] [line "496"] [id "932160"] [rev ""] [msg "Remote Command Execution: Unix Shell Code Found"] [data "Matched Data: bin/bash found within ARGS:exec: /bin/bash"] [severity "2"] [ver "OWASP_CRS/3.2.0"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-shell"] [tag "platform-unix"] [tag "attack-rce"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "OWASP_CRS/WEB_ATTACK/COMMAND_INJECTION"] [tag "WASCTC/WASC-31"] [tag "OWASP_TOP_10/A1"] [tag "PCI/6.5.2"] [hostname "127.0.0.1"] [uri "/index.html"] [unique_id "165651983136.006138"] [ref "o1,8v21,9t:urlDecodeUni,t:cmdLine,t:normalizePath,t:lowercase"]
ModSecurity: Access denied with code 403 (phase 2). Matched "Operator `Ge' with parameter `5' against variable `TX:ANOMALY_SCORE' (Value: `5' ) [file "/usr/local/nginx/conf/owasp-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "80"] [id "949110"] [rev ""] [msg "Inbound Anomaly Score Exceeded (Total Score: 5)"] [data ""] [severity "2"] [ver "OWASP_CRS/3.2.0"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"] [hostname "127.0.0.1"] [uri "/index.html"] [unique_id "165651983136.006138"] [ref ""]

Conclusion

In this guide, we learned how to install ModSecurity with Nginx on Oracle Linux 8. Your server is not fully protected with ModSecurity and it can able to protect from a wide range of attacks. Give it a try on your dedicated server from Atlantic.Net!