When it comes to the problem with security one of the issues that underlies a lot of it is lack of evidence based claims, whether the claims come from the security industry or the public. When coming from the security industry it looks like a lot of that is knowingly false, for example claiming that a service would protect websites from being hacked while the services actual function is to try to detect that the website has been hacked after it has been hacked.

A recent example of a claim from the public that isn’t backed by evidence involves an odd situation mentioned in the post on our recent security review of a plugin where the plugin, which is for protecting against spam, is checking if the installed version of WordPress has security vulnerabilities. We don’t understand why that functionality is in the plugin, but in doing that the plugin sends information that sometimes wouldn’t be publicly known to the website wpvulndb.com. In trying to explaining why it is checking for vulnerabilities in the installed version of WordPress the developer of the plugin made this claim:

The WPScan Vulnerability Database (wpvulndb.com) is legit, and is one of the best resources out there for WordPress security as it contains the most complete list of vulnerabilities for WordPress, Themes and Plugins.

No evidence is cited for that claim and from our own experience when it comes to plugins that isn’t back up by what we have seen. We know this not only because we on occasion check competing data sources to make sure we provide the best data on vulnerabilities in plugins to our customers, but also because we monitor this data source as part of our extensive monitoring for information on plugin vulnerabilities, as on occasion vulnerabilities are submitted directly there (most of their data comes from third-parties).

From doing that we know that many vulnerabilities that we have discovered, including many that hackers either were already or very likely to exploit in the future are not included. Considering that we discover quite a few of the vulnerabilities that on its own would be a problem with being the most complete list, but in occasions when they have added vulnerabilities we have discovered, there have been indications that they have spotty inclusions of vulnerabilities they are even aware of, as in two instances we looked into where we had disclosed multiple connected vulnerabilities at once (four in one instance and five in the other) they only included one of the vulnerabilities in each instance. We looked into those situations because we were trying to understand what led them to include those ones and not the others, but we couldn’t fix an explanation, so it looks like they just are not very thorough.

Simply having more vulnerabilities in your data set though isn’t necessarily better though, as the quality of your data also matters. That is where WPScan Vulnerability Database has serious issue because they usually don’t verify the vulnerabilities as we always do. That leads a number of problems, including them listing false reports of vulnerabilities in their data and claiming unfixed vulnerabilities are fixed. That latter issue seems rather important since knowing if you are using a vulnerable seems to be the most important use for this type of data, it also means you would need to double check any vulnerabilities they claimed are fixed (or you could use a service that actually does that for you).

There are two takeaways from that. First, when you see claims made about security you should look if there is evidence that supports the claims presented. If you don’t, the claim should probably be assumed to not be known be true. When evidence is present, if possible, you would want to check over the evidence presented, because we have found that in some of the few instances where it is provided, it has been manipulated. Second, if you are going to recommend relying on data from the WPScan Vulnerability Database it is important to note its limitations. We actually think it is a good source for a lot of people because its data can be accessed for free, but you get what you pay for and those relying on it should really be aware of its limitations.