What's New In IT Edge?
 
security

Scalable WAF protection with HAProxy and Apache with modsecurity – HAProxy Technologies

 

Note: The below information is deprecated as HAProxy Enterprise Edition now offers a fully functional native WAF module which supports whitelist-based rulesets, blacklist-based rulesets, with modsecurity ruleset support coming soon!

Greeting to Thomas Heil, from our German partner Olanis, for his help in Apache and modsecurity configuration assistance.

What is a Web Application Firewall (WAF)?

Years ago, it was common to protect networks using a firewall… Well known devices which filter traffic at layer 3 and 4…

Now, the failures have moved from the network stack to the application layer, making the old firewall useless and obsolete (for protection purpose I mean). We used then to deploy IDS or IPS, which tried to match attacks at a packet level. These products are usually very hard to tune.

Then the Web Application Firewall arrived: it’s a firewall aware of the layer 7 protocol in order to be more efficient when deciding to block requests (or responses).

This is because the attacks became more complicated, like the SQL Injection, Cross Site scripting, etc…

One of the most famous opensource WAF is mod_security, which works as a module on Apache webserver and IIS for a long time and has been announced recently for nginx too.

A very good alternative is naxsi, a module for nginx, still young but very promising.

On today’s article, I’ll focus on modsecurity for Apache. In a next article, I’ll build the same platform with naxsi and nginx.

Scalable WAF platform

The main problem with WAF, is that they require a lot of resources to analyse each requests headers and body. (it can even be configured to analyze the response). If you want to be able to protect all your upcoming traffic, then you must think scalability.

In the present article I’m going to explain how to build a reliable and scalable platform where WAF capacity won’t be an issue. I could add „and where WAF maintenance could be done during business hours).

Here are the basic purpose to achieve:

  • Web Application Firewall: achieved by Apache and modsecurity
  • High-availability: application server and WAF monitoring, achieved by HAProxy
  • Scalability: ability to adapt capacity to the upcoming volume of traffic, achieved by HAProxy

It would be good if the platform would achieve the following advanced features:

  • DDOS protection: blind and brutal attacks protection, slowloris protection, achieved by HAProxy
  • Content-Switching: ability to route only dynamic requests to the WAF, achieved by HAProxy
  • Reliability: ability to detect capacity overusage, this is achieved by HAProxy
  • Performance: deliver response as fast as possible, achieved by the whole platform

Web platform with WAF Diagram

The diagram below shows the platform with HAProxy frontends (prefixed by ft_) and backends (prefixed by bk_). Each farm is composed by 2 servers.

As you can see, at first, it seems all the traffic goes to the WAFs, then comes back in HAProxy before being routed to the web servers. This would be the basic configuration, meeting the following basic requirements: Web Application Firewall, High-Availability, Scalability.

Platform installation

As load-balancer, I’m going to use our well known ALOHA 🙂

The web servers are standard debian with apache and PHP, the application used on top of it is dokuwiki. I have no procedure for this one, this is very straight forward!

The WAF run on CentOS 6.3 x86_64, using modsecurity 2.5.8. The installation procedure is outside of the scope of this article, so I documented it on my personal wiki.

All of these servers are virtualized on my laptop using KVM, so NO, I won’t run performance benchmark, it would be ridiculous!

Configuration

WAF configuration

Basic configuration here, no tuning at all. The purpose is not to explain how to configure a WAF, sorry.

Apache Configuration

Modification to the file /etc/httpd/conf/httpd.conf:

  1. Listen 192.168.10.15:81
  2. […]
  3. LoadModule security2_module modules/mod_security2.so
  4. LoadModule unique_id_module modules/mod_unique_id.so
  5. […]
  6. NameVirtualHost 192.168.10.15:81
  7. […]
  8. <IfModule mod_security2.c>
  9. SecPcreMatchLimit 1000000
  10. SecPcreMatchLimitRecursion 1000000
  11. SecDataDir logs/
  12. </IfModule>
  13. <VirtualHost 192.168.10.15:81>
  14. ServerName *
  15. AddDefaultCharset UTF-8
  16. <IfModule mod_security2.c>
  17. Include modsecurity.d/modsecurity_crs_10_setup.conf
  18. Include modsecurity.d/aloha.conf
  19. Include modsecurity.d/rules/*.conf
  20. SecRuleEngine On
  21. SecRequestBodyAccess On
  22. SecResponseBodyAccess On
  23. </IfModule>
  24. ProxyPreserveHost On
  25. ProxyRequests off
  26. ProxyVia Off
  27. ProxyPass / http://192.168.10.2:81/
  28. ProxyPassReverse / http://192.168.10.2:81/
  29. </VirtualHost>

Basically, we just turned Apache into a reverse-proxy, accepting traffic for any server name, applying modsecurity rules before routing traffic back to HAProxy frontend dedicated to web servers.

Client IP

HAProxy works has a reverse proxy and so will use its own IP address to get connected on the WAF server. So you have to install mod_rpaf to get the client IP in the WAF for both tracking and logging.

To install mod_rpaf, follow these instructions: apache mod_rpaf installation.

Concerning its configuration, we’ll do it as below, edit the file /etc/httpd/conf.d/mod_rpaf.conf:

  1. LoadModule rpaf_module modules/mod_rpaf-2.0.so
  2. <IfModule rpaf_module>
  3. RPAFenable On
  4. RPAFproxy_ips 192.168.10.1 192.168.10.3
  5. RPAFheader X-Client-IP
  6. </IfModule>

modsecurity custom rules

In the Apache configuration there is a directive which tells modsecurity to load a file called aloha.conf. The purpose of this file is to tell to modsecurity to deny the health check requests from HAProxy and to prevent logging them.

HAProxy will consider the WAF as operational only if it gets a 403 response to this request. (see HAProxy configuration below).

Content of the file /etc/httpd/modsecurity.d/aloha.conf:

  1. SecRule REQUEST_FILENAME “/waf_health_check” “nolog,deny”

Load-Balancer (HAProxy) configuration for basic usage

The configuration below is the first shoot we do when deploying such platform, it is basic, simple and straight forward:

  1. ######## Default values for all entries till next defaults section
  2. defaults
  3. option http-server-close
  4. option dontlognull
  5. option redispatch
  6. option contstats
  7. retries 3
  8. timeout connect 5s
  9. timeout http-keep-alive 1s
  10. # Slowloris protection
  11. timeout http-request 15s
  12. timeout queue 30s
  13. timeout tarpit 1m # tarpit hold tim
  14. backlog 10000
  15. # public frontend where users get connected to
  16. frontend ft_waf
  17. bind 192.168.10.2:80 name http
  18. mode http
  19. log global
  20. option httplog
  21. timeout client 25s
  22. maxconn 1000
  23. default_backend bk_waf
  24. # WAF farm where users’ traffic is routed first
  25. backend bk_waf
  26. balance roundrobin
  27. mode http
  28. log global
  29. option httplog
  30. option forwardfor header X-Client-IP
  31. option httpchk HEAD /waf_health_check HTTP/1.0
  32. # Specific WAF checking: a DENY means everything is OK
  33. http-check expect status 403
  34. timeout server 25s
  35. default-server inter 3s rise 2 fall 3
  36. server waf1 192.168.10.15:81 maxconn 100 weight 10 check
  37. server waf2 192.168.10.16:81 maxconn 100 weight 10 check
  38. # Traffic secured by the WAF arrives here
  39. frontend ft_web
  40. bind 192.168.10.2:81 name http
  41. mode http
  42. log global
  43. option httplog
  44. timeout client 25s
  45. maxconn 1000
  46. # route health check requests to a specific backend to avoid graph pollution in ALOHA GUI
  47. use_backend bk_waf_health_check if { path /waf_health_check }
  48. default_backend bk_web
  49. # application server farm
  50. backend bk_web
  51. balance roundrobin
  52. mode http
  53. log global
  54. option httplog
  55. option forwardfor
  56. cookie SERVERID insert indirect nocache
  57. default-server inter 3s rise 2 fall 3
  58. option httpchk HEAD /
  59. timeout server 25s
  60. server server1 192.168.10.11:80 maxconn 100 weight 10 cookie server1 check
  61. server server2 192.168.10.12:80 maxconn 100 weight 10 cookie server2 check
  62. # backend dedicated to WAF checking (to avoid graph pollution)
  63. backend bk_waf_health_check
  64. balance roundrobin
  65. mode http
  66. log global
  67. option httplog
  68. option forwardfor
  69. default-server inter 3s rise 2 fall 3
  70. timeout server 25s
  71. server server1 192.168.10.11:80 maxconn 100 weight 10 check
  72. server server2 192.168.10.12:80 maxconn 100 weight 10 check

Advanced Load-Balancing (HAProxy) configuration

We’re going now to improve a bit the platform. The picture below shows which type of protection is achieved by the load-balancer and the WAF:

The configuration below adds a few more features:

  • DDOS protection on the frontend
  • abuser or attacker detection in bk_waf and blocking on the public interface (ft_waf)
  • Bypassing WAF when overusage or unavailable

Which will allow to meet the advanced requirements: DDOS protection, Content-Switching, Reliability, Performance.

  1. ######## Default values for all entries till next defaults section
  2. defaults
  3. option http-server-close
  4. option dontlognull
  5. option redispatch
  6. option contstats
  7. retries 3
  8. timeout connect 5s
  9. timeout http-keep-alive 1s
  10. # Slowloris protection
  11. timeout http-request 15s
  12. timeout queue 30s
  13. timeout tarpit 1m # tarpit hold tim
  14. backlog 10000
  15. # public frontend where users get connected to
  16. frontend ft_waf
  17. bind 192.168.10.2:80 name http
  18. mode http
  19. log global
  20. option httplog
  21. timeout client 25s
  22. maxconn 10000
  23. # DDOS protection
  24. # Use General Purpose Couter (gpc) 0 in SC1 as a global abuse counter
  25. # Monitors the number of request sent by an IP over a period of 10 seconds
  26. stick-table type ip size 1m expire 1m store gpc0,http_req_rate(10s),http_err_rate(10s)
  27. tcp-request connection track-sc1 src
  28. tcp-request connection reject if { sc1_get_gpc0 gt 0 }
  29. # Abuser means more than 100reqs/10s
  30. acl abuse sc1_http_req_rate(ft_web) ge 100
  31. acl flag_abuser sc1_inc_gpc0(ft_web)
  32. tcp-request content reject if abuse flag_abuser
  33. acl static path_beg /static/ /dokuwiki/images/
  34. acl no_waf nbsrv(bk_waf) eq 0
  35. acl waf_max_capacity queue(bk_waf) ge 1
  36. # bypass WAF farm if no WAF available
  37. use_backend bk_web if no_waf
  38. # bypass WAF farm if it reaches its capacity
  39. use_backend bk_web if static waf_max_capacity
  40. default_backend bk_waf
  41. # WAF farm where users’ traffic is routed first
  42. backend bk_waf
  43. balance roundrobin
  44. mode http
  45. log global
  46. option httplog
  47. option forwardfor header X-Client-IP
  48. option httpchk HEAD /waf_health_check HTTP/1.0
  49. # If the source IP generated 10 or more http request over the defined period,
  50. # flag the IP as abuser on the frontend
  51. acl abuse sc1_http_err_rate(ft_waf) ge 10
  52. acl flag_abuser sc1_inc_gpc0(ft_waf)
  53. tcp-request content reject if abuse flag_abuser
  54. # Specific WAF checking: a DENY means everything is OK
  55. http-check expect status 403
  56. timeout server 25s
  57. default-server inter 3s rise 2 fall 3
  58. server waf1 192.168.10.15:81 maxconn 100 weight 10 check
  59. server waf2 192.168.10.16:81 maxconn 100 weight 10 check
  60. # Traffic secured by the WAF arrives here
  61. frontend ft_web
  62. bind 192.168.10.2:81 name http
  63. mode http
  64. log global
  65. option httplog
  66. timeout client 25s
  67. maxconn 1000
  68. # route health check requests to a specific backend to avoid graph pollution in ALOHA GUI
  69. use_backend bk_waf_health_check if { path /waf_health_check }
  70. default_backend bk_web
  71. # application server farm
  72. backend bk_web
  73. balance roundrobin
  74. mode http
  75. log global
  76. option httplog
  77. option forwardfor
  78. cookie SERVERID insert indirect nocache
  79. default-server inter 3s rise 2 fall 3
  80. option httpchk HEAD /
  81. # get connected on the application server using the user ip
  82. # provided in the X-Client-IP header setup by ft_waf frontend
  83. source 0.0.0.0 usesrc hdr_ip(X-Client-IP)
  84. timeout server 25s
  85. server server1 192.168.10.11:80 maxconn 100 weight 10 cookie server1 check
  86. server server2 192.168.10.12:80 maxconn 100 weight 10 cookie server2 check
  87. # backend dedicated to WAF checking (to avoid graph pollution)
  88. backend bk_waf_health_check
  89. balance roundrobin
  90. mode http
  91. log global
  92. option httplog
  93. option forwardfor
  94. default-server inter 3s rise 2 fall 3
  95. timeout server 25s
  96. server server1 192.168.10.11:80 maxconn 100 weight 10 check
  97. server server2 192.168.10.12:80 maxconn 100 weight 10 check

Detecting attacks

On the load-balancer

The ft_waf frontend stick table tracks two information: http_req_rate and http_err_rate which are respectively the http request rate and the http error rate generated by a single IP address.

HAProxy would automatically block an IP which has generated more than 100 requests over a period of 10s or 10 errors (WAF detection 403 responses included) in 10s. The user is blocked for 1 minute as long as he keeps on abusing.

Of course, you can setup above values to whatever you need: it is fully flexible.

To know the status of IPs in your load-balancer, just run the command below:

  1. echo show table ft_waf | socat /var/run/haproxy.stat –
  2. # table: ft_waf, type: ip, size:1048576, used:1
  3. 0xc33304: key=192.168.10.254 use=0 exp=4555 gpc0=0 http_req_rate(10000)=1 http_err_rate(10000)=1

Note: The ALOHA Load-balancer does not provide watch, but you can monitor the content of the table in live with the command below:

  1. while true ; do echo show table ft_waf | socat /var/run/haproxy.stat – ; sleep 2 ; clear ; done

On the Waf

I have not setup anything particular on WAF logging, so every errors appears in /var/log/httpd/error_log. IE:

  1. [Fri Oct 12 10:48:21 2012] [error] [client 192.168.10.254] ModSecurity: Access denied with code 403 (phase 2). Pattern match “(?:(?:[\\;\\|\\`]\\W*?\\bcc|\\b(wget|curl))\\b|\\/cc(?:[\\’\”\\|\\;\\`\\-\\s]|$))” at REQUEST_FILENAME. [file “/etc/httpd/modsecurity.d/rules/modsecurity_crs_40_generic_attacks.conf”] [line “25”] [id “950907”] [rev “2.2.5”] [msg “System Command Injection”] [data “/cc-“] [severity “CRITICAL”] [tag “WEB_ATTACK/COMMAND_INJECTION”] [tag “WASCTC/WASC-31”] [tag “OWASP_TOP_10/A1”] [tag “PCI/6.5.2”] [hostname “mywiki”] [uri “/dokuwiki/lib/images/license/button/cc-by-sa.png”] [unique_id “UHfZVcCoCg8AAApVAzsAAAAA”]

Seems to be a false positive 🙂

Conclusion

Today, we saw it’s easy to build a scalable and well performing WAF platform in front of our web application.

The WAF is able to communicate to HAProxy which IPs to automatically blacklist (throuth error rate monitoring), which is convenient since the attacker won’t bother the WAF for a certain amount of time 😉

The platform allows to detect WAF farm availability and to bypass it in case of total failure, we even saw it is possible to bypass the WAF for static content if the farm is running out of capacity. Purpose is to deliver a good end-user experience without dropping too much the security.

Note that it is possible to route all the static content to the web servers (or a static farm) directly, whatever the status of the WAF farm.

This make me say that the platform is fully scallable and flexible.

Also, bear in mind to monitor your WAF logs, as shown in the example above, there was a false positive preventing an image to be loaded from dokuwiki.