Talk:CCS2011: Enemy of the Good - Revision history

Soma at 12:44, 21 March 2011

2011-03-21T12:44:24Z

← Older revision		Revision as of 12:44, 21 March 2011
Line 1:		Line 1:
			=ToDo=

			* Gather data from different IDS observables to show they aren't Gaussian
			** system calls (Luc)
			** network traffic
			** log files
			* Machine learning
			** standard machine learning methods approximate distributions
			** approximation works best if Gaussian but has limits (show mathematically)
			** non-Gaussian distributions place much harsher restrictions on error rates, they don't go down proportionally to sample size? (more math)
			* [[Survey of results in IDS literature]]

			=Old Introduction=

	* For IDS to work, we need very accurate detectors		* For IDS to work, we need very accurate detectors
	** base rate fallacy		** base rate fallacy

Soma at 12:43, 21 March 2011

2011-03-21T12:43:51Z

← Older revision		Revision as of 12:43, 21 March 2011
Line 67:		Line 67:
	*Conclusion		*Conclusion
	**For these reasons, and those articulated in the remained of this paper, we believe that the prudent next step in IDS research is a thorough examination of novel techniques for dealing with current degree of FP. More specifically, this thesis arises from the fact that it is clear that the added benefit of the increasingly sophisticated ML algorithms is diminishing, and that, to some degree, FP are, and will remain, a fact of life in IDS.		**For these reasons, and those articulated in the remained of this paper, we believe that the prudent next step in IDS research is a thorough examination of novel techniques for dealing with current degree of FP. More specifically, this thesis arises from the fact that it is clear that the added benefit of the increasingly sophisticated ML algorithms is diminishing, and that, to some degree, FP are, and will remain, a fact of life in IDS.

			=What Goes Wrong=

			* Poor results
			** datasets do not represent real-world usage or scenarios accurately
			** insufficient or misleading tests of false positive rates
			** Even when rates are accurate, they are misinterpreted: high FP rates are not considered to be high (wrong time scale, lack of attention to scalability)
			** misleading integration of attacks into legitimate behavior
			* Administrative overhead
			** rules that can only be created by experts, but system requires end users to create custom rules
			** experts required to interpret output
			** insufficient context for even experts to interpret output
			** assumption of existence of security personnel that won't even exist in many important contexts
			* Computational overhead
			** can system keep up with normal workloads?
			** can system keep up with attacker-generated workloads?
			* Anomalies versus attacks
			** why is one a good proxy for the other?
			** why is chosen feature(s) particularly good at detecting attacks?
			* Out of the box algorithms applied w/o understanding security problem
			* Attacker evasion: how can attacker manipulate system? Can system lead to environment that is easier to attack?

Soma at 12:42, 21 March 2011

2011-03-21T12:42:17Z

@@ Line 1: / Line 1: @@
 * For IDS to work, we need very accurate detectors
 ** base rate fallacy
@@ Line 47: / Line 46: @@
 * Relatively low rate of data
 * Still has persistent false positives _and_ false negatives

Soma: Created page with " * For IDS to work, we need very accurate detectors base rate fallacy specifically, very low false alarm rates * To date, nobody has achieved sufficiently low false alarm r…"

2011-03-21T12:40:00Z

Created page with " * For IDS to work, we need very accurate detectors ** base rate fallacy ** specifically, very low false alarm rates * To date, nobody has achieved sufficiently low false alarm r…"

New page

* For IDS to work, we need very accurate detectors
** base rate fallacy
** specifically, very low false alarm rates
* To date, nobody has achieved sufficiently low false alarm rates to be universally applicable
** signature and spec methods can be ad-hoc tuned to be good enough but then have poor coverage of new attacks
** adaptive methods cannot be sufficiently tuned
* We argue that we can't get low enough false alarm rates, that there are fundamental limits on IDS performance due to the underlying distributions of legitimate and attacker behavior.
* Reasons:
** legit behavior is non-Gaussian, largely power-law like, meaning they have fat tails
** attacker behavior cannot be sampled sufficiently to learn distribution
** and besides, attacker behavior keeps changing to follow new attack innovations (more like spread of disease than Gaussian, fundamentally not stationary) and to behave more like legitimate behavior to avoid defenders
** if we could get good samples of both classes, we might be able to separate them; but instead we must do one-class learning and one-class learning cannot deal well with very long tails.
** "adaptive concept drift"

IDS Requirements

* scalability in false alarms
* detect wide range of attacks
** realistically won't catch all attacks, but should go significantly beyond "just what I've seen" (otherwise cannot address attacker innovation)
* low resource usage (network, CPU, storage/IO, user, administrator)
* Stated this way, looks like a ML problem

Machine Learning
* many, many techniques
* basic idea: combine a-priori knowledge built into learning method with observations to create classification model
* IDS is a binary classification problem
* most accurate methods require representative set of each class
* if not both, need at least one representative set
* to do this, data should have certain characteristics

Legitimate behavior

**Classifier technology and the illusion of progress[http://arxiv.org/pdf/math.ST/0606441]

Sections:
* Problem
*

Best case scenario: credit card fraud detection
* Two class learning is possible
* Relatively low rate of data
* Still has persistent false positives _and_ false negatives

Talk:CCS2011: Enemy of the Good - Revision history

Soma at 12:44, 21 March 2011

Soma at 12:43, 21 March 2011

Soma at 12:42, 21 March 2011

Soma: Created page with " * For IDS to work, we need very accurate detectors ** base rate fallacy ** specifically, very low false alarm rates * To date, nobody has achieved sufficiently low false alarm r…"

Soma: Created page with " * For IDS to work, we need very accurate detectors base rate fallacy specifically, very low false alarm rates * To date, nobody has achieved sufficiently low false alarm r…"