Talk:CCS2011: Enemy of the Good

For IDS to work, we need very accurate detectors
- base rate fallacy
- specifically, very low false alarm rates
To date, nobody has achieved sufficiently low false alarm rates to be universally applicable
- signature and spec methods can be ad-hoc tuned to be good enough but then have poor coverage of new attacks
- adaptive methods cannot be sufficiently tuned
We argue that we can't get low enough false alarm rates, that there are fundamental limits on IDS performance due to the underlying distributions of legitimate and attacker behavior.
Reasons:
- legit behavior is non-Gaussian, largely power-law like, meaning they have fat tails
- attacker behavior cannot be sampled sufficiently to learn distribution
- and besides, attacker behavior keeps changing to follow new attack innovations (more like spread of disease than Gaussian, fundamentally not stationary) and to behave more like legitimate behavior to avoid defenders
- if we could get good samples of both classes, we might be able to separate them; but instead we must do one-class learning and one-class learning cannot deal well with very long tails.
- "adaptive concept drift"

IDS Requirements

scalability in false alarms
detect wide range of attacks
- realistically won't catch all attacks, but should go significantly beyond "just what I've seen" (otherwise cannot address attacker innovation)
low resource usage (network, CPU, storage/IO, user, administrator)
Stated this way, looks like a ML problem

Machine Learning

many, many techniques
basic idea: combine a-priori knowledge built into learning method with observations to create classification model
IDS is a binary classification problem
most accurate methods require representative set of each class
if not both, need at least one representative set
to do this, data should have certain characteristics

Legitimate behavior

Sections:

Best case scenario: credit card fraud detection