Something that is important to us is to be providing the best possible services we can. For our main service we frequently look at how we are doing in comparison to other sources of vulnerability data. It is especially important to us that we are provide far superior data than free sources, otherwise why should you be paying us. If you have read our blog post discussing the issues with those other providers, we think that you would know that we are living up to that.

Due to that need to provide the best possible in our services, originally our plan had been to hire someone else to do security reviews that we were planning to include as part of our main service. That changed in part due to a situation with the company we had taken a look at having do that.  When the company disclosed a vulnerability they had found as part of a review, they said that it had been fixed, but the vulnerability had not in fact been fixed. That wasn’t a major issue on its own, since we frequently run into just such a situation. What was more concerning was that the change made to the plugin clearly wouldn’t have been a fix since the vulnerability involved a situation where user input was not being properly secured when used in the plugin’s code. The change made modified JavaScript code, which runs on the client side, so any changes it made to input would not prevent an attacker since they could change the JavaScript code or just ignore it all together. We contacted the company that did the review and asked them how the change was supposed to fix the issue and we never got response.

After that we decided that they wouldn’t be right to hire and by that time we figured that we had enough expertise to provide a review that we felt was good enough. Over time we have made further improvements to the reviews in terms of what we are checking for, some of that based on the results of the reviews we have previously done. And there are further improvements that we are working on based on things that have come up during the other work we do as part of the service. We think that the breadth of the work we do makes each part of what we do better than what others can provide that don’t have such broad reach across the various part of the security of WordPress plugins.

Before we started to offer to do reviews separately from our main service we expanded the numbers of items we are checking for during the reviews (whether as part of those separate reviews or the reviews that are a part of our main service), much of that based on what are things that are most concerning for websites where it would make the most sense to get a review done of plugins being used.

In the meantime we have continued to look over what others offering to do reviews are providing to try to make sure that we are providing the best service possible. The problem we have found in trying to do that is that we have yet to find someone else offering to do those that provides any details as to what their reviews encompass. Maybe they check for everything possible, but what seems more likely is these reviews are in fact extremely limited.

For a number of the providers they also don’t provide any results of reviews or evidence of them having discovered any vulnerabilities, which would seem to be a bad sign as to what their reviews could provide.

With something that we can check on, it has led to more concern about the quality of the reviews available. With a couple of the providers we have come across them due to our having found vulnerabilities in their plugins. In one case we found a minor vulnerability in a WordPress security company’s security plugin. While the vulnerability was minor, it was the result of a failure to do a security basic, something that you wouldn’t expect from someone offering to review other plugins for just that sort of thing. In the case of another company, we first came across one of the plugin’s from the person behind the company as we were looking into the details of a vulnerability that had been fixed in their plugin and we found two more vulnerabilities in the plugin. More recently someone else found a very exploitable vulnerability in that same plugin, all of that seems to not be a great indication of the security expertise of the company.

We see a lot of need for what security reviews of plugins could do for the security of websites, though not the same level of interest from the public in them. If there was more interest, based on all of what we mentioned above, it doesn’t seem like the results would be all that great, which is unfortunate, but it is line with general poor state of the security industry.