Complete Transcript of Andrew Lee Interview
Radio Talk Show - Let’s Talk Computers - March 25th 2006
AA: It seems that every anti-virus company makes the same claim. Tests show that their software will stop all threats. Well, exactly what are these tests and where are the results of these tests?
AJ: Our guest today is Andrew Lee, CTO of ESET Software. And welcome back to LTC, Andrew.
AA: Andrew, why is being educated about what kind of threats that could attack us so important?
AJ: HI, Alan, it’s great to be back.
AA: Andrew, we’re talking about performing tests on anti-virus or Anti-Threat software. Exactly what are these tests?
AJ: It’s kind of an obvious thing that you want an anti-virus program to detect viruses. All of them are going to say, “yeah, we can detect X number of viruses.” or we can detect more viruses than the other guy or this kind of thing. This whole marketing machine around anti-virus, but really, what you actually want to know is does it detect viruses before they become a threat to me that is before they have actually spread and replicated onto my PC.
AA: First of all, what does a test virus look like? We see a lot of virus companies that use the EICAR test message to prove that they’re catching viruses. The EICAR test message is just a string of characters put together to basically show you that the virus software is working. This is not really a good virus test, is it?
AJ: EICAR is very often misunderstood. The EICAR test file has a very specific purpose. But, it’s not for telling you anything about the detection capabilities of the product. It can’t do that. It just isn’t for that purpose. What EICAR does is it tells you did I install an access scanner correctly? Is it functioning? So, it’s really a case of when I’ve installed it, I can run EICAR and the purpose of it’s working is just to make sure that I’ve installed it correctly.
AA: The reason I brought it up, detecting a set of viruses based on a signature is about the same thing, because I’m looking at it saying, “If I can catch the EICAR string then I should be able to catch a signature, that we know about.” That’s not anti-virus testing. Anti-virus testing is testing something that we don’t know about, isn’t it?
AJ: We’ve seen a lot of very, very bad tests and people think that it detects 10 viruses. Statistically speaking, that’s such a small sub-set out of any capability. And it’s pretty random as to whether those files will be detected or not. And the first reason for that is that people don’t realize that to actually be a virus, it must replicate. It has to be able to do the actual things that a virus does. And that is when you install it, it replicates a copy of itself and then it replicates further copies of itself into other files or across the network or however it is meant to spread. It doesn’t do any of that, then it’s not a virus. There are thousands and thousands of samples out there that you actually receive, when the virus workers write these things.
They don’t do any quality analysis particularly. They don’t test it on multiple systems whether these are going to work. They don’t even test if the replication works correctly. And there are some viruses, which are even very widespread. There are a lot of damaged samples around which means basically those samples are just junk files. It didn’t do anything. It didn’t replicate. Unless you really know what you’re doing in terms of testing you should be very careful of someone who says, “The guy only tested with ten viruses and such and such product didn’t detect them and this product did detect them, so this product is the better one.” That’s really not a good measure. The only real way to do testing is first of all, to take verified samples – that’s samples that have been replicated, that have been checked that they are actually malicious, that they do have the property viruses that they do replicate, they do function correctly, so they’re not just testing against a bunch of junk files.
AA: If you’re walking down the street and somebody approaches you and he has an eye-patch and a wooden leg and a parrot on his shoulder you know, you can pretty well bet that you might be robbed. People that are going to mug you are not going to announce that this is what their intent is going to be. That would be kind of foolish, wouldn’t it?
AJ: There’s a lot of mis-information around and unfortunately some of that is just made worse by really bad testing that’s being done. Even in pretty good magazines who have an idea of what needs to be done, but the best way to really test is to have two different types of tests. And one is use is in the wild types of files, using the wild list as a measurement of which files are reported to be real threats, you know threats that are actually out there. And to test the scanners regularly against every virus on that list. And a good anti-virus test company will be testing against several thousands of files.
AA: One of the things that we want to test virus program against is what these ZeroHour Treats because viruses and malware threats are getting to be so sophisticated. It’s not a fact that we’re going to advertise this great new virus that hit all these computers all over the world. The idea is the sophisticated programmers are now trying to make money off of you, off of your machine, or off of your vulnerability and you need to have that ZeroHour, Right Now protection, don’t you?
AJ: That’s really one of the interesting things that’s starting to happen out of a couple of particular, anti-virus test organizations is that they’re beginning to test heuristics capabilities of anti-virus programs. Everyone’s realizing now the need to have a proactive anti-virus solution. Because the threats are very, very fast. They come out very quickly and they’re delivered very quickly and if you get infected with them and ten minutes it updates itself with the latest version from the web site and you’ve got kind of a moving target to try and hit, which is very, very difficult to do with the traditional anti-virus solution. There are not a lot of companies who are taking about heuristics, they’re talking about pro-active detection, but how do you know? Again, it comes down to how do you know how good your heuristic detection is? How do you know how good your heuristic is? How do you know whether your product is the one that’s better than the product that someone else is making? And so, again, this is even a harder thing to test than just regular virus protection, because it moves so quickly.
One of the ways that you can test that is by taking the anti-virus products, freezing them effectively so download them, update to a certain point, a certain time, you know, take today, the 21 st of March, we freeze it and in three months’ time, we come back and we test them against all the viruses that we’ve discovered in that period, (that’s come out in that time). At the end of that time, you just test them against all the anti-virus scanners and what you can tell from that is that you find that a really good heuristics products detect a far higher proportion of them proactively. And that means there’s no update involved, they didn’t have prior knowledge of those viruses that they downloaded, because they didn’t exist at the time. So, that’s really a good test of how good your product is.
AA: Are there a lot of companies that are doing that? I know, we’re looking at Virus Bulletin puts out a newspaper every single month about their tests?
AJ: Virus Bulletin, 100% Award testing is specifically against in the wild and it is a retrospective test if you like. Probably the most interesting or certainly one of the most respected company is AV-Test.org. They are well respected; they do very good tests, with a high degree of quality. Frequently published in reaction time, so actually what they’re doing is a slightly more interesting test than I just described. When there is a new outbreak, they are measure the time at which each of the of the anti-virus products detected it.
Another interesting company is AV-Comparatives.org. And they do kind of more like the test that I described earlier. They freeze the product at a certain point and then they take the viruses that came out in that time and then they throw them at the anti-virus scanner. In May, last year they had 51 samples that had collected in that time, that were listed as being in the Wild, they fed those to scanners and NOD32 was clear winner at detecting 46 out of those 51.
Independent testing gives you a measure of really how good the heuristics are, aside from all the talk about, “we do it better than this person or this company”. It actually gives you a real measure of how effective your protection is.
AA: You’re talking about viruses and I like to put malware inside of that too, but people look at malware and keylogging and things of that nature completely different than viruses. But you’re saying that you protect against malware threats, too.
AJ: Yeah, we do. I really mean malware, when I’m talking to you generally. There’s no point in us giving you half the protection and just protecting you from viruses. In fact, our statistics have shown that proportionate of actual viruses that’s coming out right now is only three percent of the stuff. So, it would be really a pointless exercise to you to have just something that only detects viruses at this point.
AA: When you’re talking about the word “heuristics”, I see a lot of people banter that word back and forth. It’s almost like using “user friendly”. What exactly do you mean by heuristics?
AJ: There’s a lot of different definitions, as you know. Probably the best thing to do is run over we talk about the different things. We have the traditional signature. And I say traditional, that makes it sound it old and outdated. It is to some extent, but every anti-virus company still uses some form of signatures because you need to. Even with NOD32 it’s is catching the prints of 85 to 95% of new things, but that means there still is a small percentage of things that didn’t get caught, so we need to produce a signature for them. So, there’s a traditional signature. But that’s not a heuristics, a reactive thing, you have to have the file, you have to have analyzed the file, and you have to deliver the update.
Then we’ve got what we call a passive heuristics. A passive heuristics doesn’t mean that it just sits there and does nothing. It really means that it doesn’t do anything to actual code. It doesn’t try and execute the code, but instead it reads through the code of a file. And this try to see if there’s anything that it knows about that is suspicious, so that is just going to open up port 25 in my system or is it going to makes a suspicious imput from a file, if it’s going to write to a registry. It looks for things that are known to be suspicious.
And then you have more of an advanced heuristics system if you would like. And the advanced heuristics system is using an emulator of some kind, sort of like a sand box. And what that does is it takes the file, the file comes into the system and inside that protected virtual environment it runs the code. And the because it looks like a real machine to that file that’s being run, it does what it would do if it were actually infecting your system. But of course, you’ve got the wrapper of, if you like of the anti-malware program around that and that’s able to examine what’s happening inside that virtual pc. And determine at that point whether it’s malicious. And actually that’s a far more involved process, there’s a lot more to that than there is to the other techniques.
AA: And that’s what ESET uses to make that we do not get infected with any of these “nasties” out there.
AJ: We use a hybrid of all four techniques when you when you use the traditional signatures, you the passive heuristics, we use the advanced heuristics, and the generic signatures and all of are in a single unified engine with all the processes that go on inside of that and it happens very, very fast. That’s really the key to it. Some of the emulators that you see on the market are slow and they cause a lot of overhead on the system.
AA: So, if I’m protected by the NOD32 Family, then I don’t have to worry about this is a 2004 version or a 2005 version. It’s the newest version out there. You make sure that it gets pushed to us, don’t you?
AJ: Yeah, that’s right. If you’ve got it enabled, the automatic updating for the system then it’s the latest that we can give you.
AA: And also, if you want to try NOD32, you have a trial version on your system, which is different than most trialware, isn’t it?
AJ: We don’t disable it any way. It’s basically the full protection that you get and you’ve got the 30 days trial for free.
AA: Now, when you say “full protection”, you mean that we can that we can get automatic updates during that trial period, we don’t just get to look at it and see, “oh, this is a pretty piece of software”, we actually get full functionality, full updates, full protection.
AJ: Absolutely. It doesn’t seem to be very fair to give you something less than full protection. If you’re trying it out, you need to know that it’s really giving you the best protection that it can and that’s the whole point of having it on there. So why would we give you a crippled down version of it?
AA: And what are we looking at as far as the price of NOD32?
AJ: You’re looking at about $39 for a single user license. We also have a 5-user patch, we have enterprise versions, and we have great packages for education. So, there’s a range of different options.
AA: And where would people go to download the trial version or find out some more information about the NOD32 Family?
AJ: The best place to go to our website at www.eset.com.
AA: Well, Andrew, it’s been you pleasure to have you as our guest here today, talking about what we really need to look when we’re deciding to buy an ant-virus or Anti-Threat product and hope to have you back again, real soon.
AJ: Thank you very much, Alan. It’s been a pleasure.
