Dec 30, 06
A few months back I did some intense testing of all the best vulnerability scanners out there.. I had a couple unix boxes hooked up, as well as some windows machines, and figured I could add clients to a “once-a-week” scanning contract. So naturally, I wanted to use the scanner that was the best for my purpose.
You might be looking for the article: Top 5 best Vulnerability Port scanners
I tested the following (trying to only list automated vulnerability scanners):
- ISS Internet Security Systems
- SSS Shadow Security Scanner
- Retina eEye
- Nessus
- GFI Languard Network Security Scanner
- Qualys www.qualys.com
- Nstealth Security Scanner www.nstalker.com
- Nikto
- Whisker
- Infiltrator infiltration-systems.com
- Nscan
I was looking at 3 main areas while evaluating the scanners.
- Comprehensiveness of the testing: including how many options are allowed for different scanning, IDS evasion, and types of scans. Also in this category is the availability for the latest exploits and a custom exploit option to allow me to plug in custom exploits.
- Quality of the program: included in this category is availability of updates, speed of various variables, efficiency, “smartness” or “AI” of the program while scanning/reporting, security- (does running this version of this vuln scanner leave me vulnerable?), scheduling capabilities, alert and message capabilities, quaility of exploits, reactions to “false positives”, and overall feature and capabilities.
- Reporting Capabilities: How easy is it to create a report? The quality and design of the report. The comprehensiveness and personalization of the reports..
My findings
All of these programs can be tested for free, either through an evaluation or trial. I believe that the availability of these programs on the net (cracked versions) represents the conspiracy to aid script-kiddies everywhere so that these companies will then profit after an intrusion (or even a loud scan).
Any of these programs are not to be used for cracking… Nothing serious at least.. using these programs is akin to knocking on every window of a house, while simultaneously ringing the doorbell and playing loud music in the driveway.. Some of the programs allow for easy use of anonymous socks and web proxys, but we all know by now that to achieve any real anonymity takes much more work than that.
After several months testing
It became clear to me which ones I liked the best.
The top pick all-around (for windows) has to be retina’s eeye, the company has a bunch of hackers and you can tell by their awesome product.
Another great scanner (windows -web) are the different products offered by nstalker… very good web vuln scanner. Free for download..
A close 2nd would be the SSS, I love it, its great, but eEye’s is better.
The best choice for me however is nessus. Although it has a few negatives compared to some of the other products, it is the best solution for anything serious. Set-up up a couple of nessus servers around the globe and try attacking the same location through different networks for some real eye-opening fun. Also good to practice tracking and tracing.
Qualys is poised to be super-rich, just check out their site… great if you don’t want to mess with a scanner yourself.
ISS is way to expensive and I didn’t like the fact that I had to try for awhile until I could download it. But it is considered to be the best scanner and I admit the anti-pirating features and licensing of their software is pretty darn good.
GFI is another top contender. It was just prone to crashing…
A money place to get some other cool tools (Especially a professional SSL encryption tester) are available for free at www.foundstone.com. Get them soon, make a foundstone toolkit…. mcafee bought them so expect the free stuff to get yanked shortly… (I can’t believe foundstone is almost gone…)
Unless you are using these tools on yourself, or to check your friends pc or company (use scheduling with alerts to email or pager), there is no sane or logical reason to use these tools on any pc that you do not have express permission to use against.
Remember these tools are the noisiest tools in a security testers arsenal and if you use it just once on an unknown host, your ISP can yank your account, you can face legal action, etc..
Better to use firewalk, hping3 (now with scripting!), nmap, etc, and leave these crutch-like tools alone.
Also check out www.coresecurity.com , for the best product I have found so far.
A few pointers..
If you have a network of your own, and you don’t have the time or resources to set up a nessus installation at a remote site to repeatedly test your network, the best solution IME is Qualys. I am not affiliated in any way, I checked it out merely to examine the competition in the marketplace… after experiencing their free trial, unheard of amounts of documentation, and extremely skilled marketing team, I am still very jealous.
What an easy concept but they took it to that level and it stands alone, IMHO.
So if you need that kind of simple, safe, comprehensive fix, check out qualys. Still, you would ultimately save a lot of money (even after computing time and costs) by setting up your own remote server somewhere to automatically check the network for you. Multiple locations of course are ideal (go through different networks etc..)
The major lack of enthusiasm I have for most of these “vulnerability scanners” is the fact that so many of them run on windows and nothing else. The benefits to running from (almost anything else) a linux type platform are enormous. IOW, nessus can only continue to get better and better, while the only advantage eeye, ISS, and the rest of the dozer boys have is better knowledge of exploits.
This is really a small point (in the big picture) because of the fact that it is sooo easy to copy an exploit (for those that do that sort of thing).
So if you are out there, go help out the excellent people who give us nessus, go give them a donation, help them develop the software, test it out and email them detailed feedback of your likes and dislikes, do what you can to keep it around. I see it as having the most potential.
There are also vuln scanners that operate locally, many professionals use these to scan a lan of windows hosts for example. These are hard to find, so if you know of any, post.
Also, after months of testing all the different vuln tools, including tons not listed above, still failed to find a bug that allowed me root on a particular server I was testing for a friend.
It was the mole.cfm script that finally got me in… an old exploit that none of the scanners tried. So if you are serious about keeping your data safe, hire a real expert (like foundstone or coresecurity) and do not rely on a commericial marketing solution alone.
Other Good Vulnerability Scanner Review here.. (12+ pages)
askapache
12.31.06 at 3:41 pm
How can you do a serious “review” of vulnerability scanners and forget the Metasploit Framework!? As someone who has professionally done penetration testing, it is invaluable and one of those must have tools. Maybe you should take a look at it:
http://www.metasploit.org/
Also, if you look on the full disclosure “computer security” mailinglists, often times new 0day announcements come with metasploit modules to test if you are vulnerable.
12.31.06 at 4:29 pm
Guys, you missed Core Security; it’s one of the most solid vulnerability assessment tools I’ve used in 2006. http://www.coresecurity.com/ Its BY FAR one of the best-of-breed tools out there.
12.31.06 at 4:44 pm
No offense intended but this is a pretty bad and uninformed article.
First of all if you are talking about Vulnerability Scanners then you should be looking at;
Nessus
Retina
GFI
ISS Scanner
Qualys
The other tools you listed are not Vulnerability Scanners but penetration testing tools. Big difference in how they perform and what they do.
The second point I want to bring up is your comments on eEye. They do not have a bunch of hackers in fact most of their research team has left the company if not all of them. Besides, since when is a bunch of unethical clowns a good thing for a company to hire?
If you actually drill down into their Retina product and look at false positives, false negatives and simply inaccurate results then yes, you should rate eEye Retina as the #1 product. They have minimal non-windows support in their products and most of their Windows suppor requires credentials in order to scan.
As someone who has worked with Vulnerability Assessment products for a number of years I would rate Tenable Nessus and Qualys far above anything that eEye will ever be capable of.
12.31.06 at 5:01 pm
Metasploit has no scanning features, it’s specific to pen-testing. Metasploit needs to be fed information about a host before it can do anything.
12.31.06 at 6:38 pm
Or canvas, or ngs scanner, saintexploit or any of the other thousands of tools you didn’t try. . .
12.31.06 at 7:28 pm
Metasploit is not a vulnerability scanner. Metasploit is a penetration testing tool.
(Core Impact and Canvas are commercial tools in this category
12.31.06 at 8:54 pm
Jeff, mf is not vulnerability scanner. It’s automated exploitation tool.
12.31.06 at 9:52 pm
Tools such as Metasploit and Core Impact are penetration testing tools. Not valurnability scanners. There is a huge difference in function and use.
01.01.07 at 1:20 am
Vulnerability scanning doesn’t equal penetration testing. If you’re looking at your network and have admin rights, then a vulnerability scanning that looks at configs is a valuable tool. Being able to bang on the door anonymously and see what can be hacked is something different.
01.01.07 at 4:22 am
Yep, I expected at-least some mention of metasploit.
But anyways it was a nice review.
01.01.07 at 4:34 am
You seem to be mixing up different classifications of tools in your evaluation there,
I wouldn’t class Nikto/whisker (which focuses only on web servers) or nscan (which seems to be a port scanner from their site) in the same category as something like Nessus…
Nor to be honest would I put Core Impact which you mention or Metasploit (or canvas) in the same list either… If you’re planning to offer a vulnerability scanning service I reckon staying clear of vulnerability exploit tools is a good idea, unless your clients are explicitly aware that you’ll be using them.
some interesting info. though hadn’t heard of one or two of these before…
01.01.07 at 6:05 am
After reading the headlines I got excited into your evaluation. However, your intense testing does not provide anything for anyone. You used neither a repeatable nor a comprehensive testing methodology so that your findings are very subjective and therefore cannot be reused by anyone else (“I love it, its great, but eEye’s is betterâ€.)
The following questions (and many others) should have been clarified in order to provide a helpful review:
- How did you rate your topics within three main areas?
- False positives are indeed an interesting and important issues, however you did not cover this in the findings.
- How do you want to rate “smartness†and “AI†of a program? I don’t think any of these programs have and AI.
Additionally, your unfounded statement about conspiracy is a pure claim and confirms lack of diligence and professionalism.
01.01.07 at 8:40 am
Jeff, Metasploit is not a web application scanner like the kind reviewed here.
01.01.07 at 9:24 am
Metasploit Framework is invaluable, however it’s not a vulnerability scanner. So then it stands to reason that it’s beyond the scope of this article.
01.01.07 at 11:00 am
This sounds more like an opinion article rather then a review. It’s all over the place, doesn’t mention detailed pros and cons of the tools nor provides any sort of evaluation matrix or target machine/network specifications.
Mentioning hping, nmap, etc. Was a good point and I think it should have been included in the tool review rather then as a side note, if you used some sort of “glue†that is (as most do).
Also, there is no need to donate (well, sort of) to the Nessus team since the software is no longer available under GPL. It is now, as far as I understand, a closed source product like the rest of them. At least from the business point of view.
01.01.07 at 11:58 am
Another great hands free penetration service is Comply Guard Networks! (www.cgnsecurity.com) It leverages a bunch of tools as well as custom design into a pretty straight forward reporting system - ALL web based!
01.01.07 at 12:55 pm
Although the review could have been more comprehensive, I think the reviewer did a good job.
To Jeff’s point:
The article was not focused on penetration testing. If this would have been the case, we would have also seen such tools as Core’s IMPACT and Immunity’s CANVAS. Get your facts straight.
The major vendors that have vulnerabilty scanners are doing their due diligence with watching 0-days. If you’ve used any of these tools, you’ll notice that they are constantly (daily, sometimes more than daily) being updated with new vulnerabilities.
01.01.07 at 1:38 pm
I work for Foundstone and wrote the “bad-a$$ SSL encryption tester” myself and just wanted to let you know that our free tools arent going anywhere - infact we will be adding some more to them this year. Glad you like them.
Btw - how come you didnt consider Foundstone’s commercial vuln scanner in ur assessment :(
01.01.07 at 5:44 pm
I seriously question the author’s expertise in scanning, vulnerabilities, and security in general.
There are so many errors in this post I don’t know where to begin
Was the point of this “test†to compare automated WEB vulnerability scanners against full fledged scanners? There is no sense in comparing nikto/whisker/nscan to qualys, langaurd, retina, nessus, etc.
Why would someone even use whisker and nikto in the same bake off? It’s well known that nikto is built off whiskers defunct framework. That’s right, they are the same “scannerâ€. Nikto can boast that it has more checks but if you actually had experience with any of these products you would know how out of date nikto is too.
And why would IDS evasion be criteria for a security pro? Shouldn’t your have explicit authorization to perform scans against your devices?
p.s. To the poster above, metasploit isn’t a vulnerability scanner. It’s and exploitation framework.
01.02.07 at 10:05 am
You missed nCircle’s IP360 solution (http://www.ncircle.com/index.php?s=products_ip360), which has pretty strong features (application detection, dynamic host tracking, enterprise risk scoring and topology risk analysis) and probably the best reporting engines of all the enterprise risk security platform in the market today.
01.02.07 at 2:08 pm
Thanks for the writeup; it’s always nice to read other peoples’ opinions on this kind of stuff. It’s amazing to me, though, how many security professionals don’t know the difference between a vulnerability scanner and a penetration testing tool.
Not only did the article make this mistake but some of people commenting did as well. I’d be willing to bet that these same people are fuzzy on the differences between risks, exposures, threats, and vulnerabilities as well. We can do better. If we aren’t precise about these concepts then we can’t expect the people we’re trying to help to be.
01.03.07 at 12:17 pm
Among many other things mentioned, not including Foundstone’s commercial vuln assessment tool invalidates the whole article. I seriously doubt this is a “Security Professional” that wrote this article.
I also liked Harris Stat Analyzer, although I have not used it in about a year now. If Foundstone would tie in with Nessus as Harris Stat Analyzer did, then Foundstone would be by far the best tool on the market.
I always find it amusing when people state that they like SSS. I think the program sucks.
01.03.07 at 1:01 pm
I think this review is inaccurate and biased.
Though I think SSS nstalker are not bad tools, they are not truly professional grade tools. First question to be asked is, what is it that you want from vulnerability scanner.
I have been developing vulnerability management program for numerous companies, to my knowledge in terms of a vulnerability scanner nothing has come close to Qualys scanner. Lowest rate of false positives and it truly meets the business goals and objectives. It also looks good from regulatroy compliance perspective.
Don’t get me wrong I have nothing bad to say for any of the tools, but I did not think the review was fully researched.
In regards to metasploit or core impact: They are NOT vulnerability assessment tools. They are penetration testing tools, which you can use to test the posture of your security after you discover vulnerabilities. Yes they do have vulnerability detection engine, however it is not as thorough. Canvas, core impact and metasploit should be used to conduct intrusive testing to validate the posture of your systems, however they are NOT vulnerability assessment tools, they are tools that can be used within vulnerability management program in conjuction with a vulnerability assessment tool such as QUALYS FOUNDSCAN EEYE NESSUS etc.
01.07.07 at 9:27 am
Core Impact v5.x
Performs scan AND FULL penetration. Not just tests, it gets into your network and makes it literally cake for all sh*tty diapered skiddies to pwn systems protected by seasoned security experts… Imagine the power… Now imagine the nightmare: $25,000 to license. Nightmare bonus: a “pirated” version will *never* become available, ever. So die skiddies.
Metasploit it shall be then, tastes great, less filling.
01.18.07 at 3:37 am
My view is that many people have still got it wrong.
You can’t even compare CANVAS,Metasploit with Core Impact, since CANVAS and Metasploit are for automated exploitation [they are frameworks with which great things can be done not ordinary tools].
Core Impact [a tool] automates a whole pen-test from scratch … The agents concept in it and the extendability provided by its python plugins, are cool.
I think Core Impact is a good tool for what you pay. [I have used it personally, it fares well for what it costs]
Btw, exploitation and pen-testng are two entirely different beasts.
I’d opt to vote for metasploit instead of canvas, since it’s free and can do almost anything canvas can do.
There are two types of scanners, if you take a broad classification.
Web scanners, Network Scanners.
Comparing all types of scanners [commerical vs. free,professional tools vs. toys, web vs. net-scanners,etc] in one report is not fair, and the author doesn’t do a comprehensive report here…
I’d say 5 marks can be awarded to this report … ?! perhaps on a scale of 10. It’s a exhaustive process to evaluate “types,commercial/pricings,good,bad things,technical value,business value” that each scanner has to offer and document it.
Cheers :)
Kish
02.05.07 at 12:32 pm
It is amazing to me how so many people in this thread bash the author’s professionalism in one aspect or another and then proceed to discredit themselves by displaying a demonstrated lack of professionalism. People with comments such as “…since this review didn’t mention , I highly doubt it was written by a security professional.”
Please, spare us your worthless drivel. Everyone has different levels of experience in security as well as differences in exposure to tools for the job. Unless you yourself are the a-1, cast iron, supreme being of security and do in fact “know it all” then how about you tone it back, be proffesional, offer constructive criticism and thank the author for his efforts even if his article did not meet your expectations.