EvoSec 2025W Lecture 13

Readings

• Somayaji, "Evaluating Security Products with Clinical Trials." (CSET 2009)

Discussion Questions

• What is the relationship between trust in medical interventions and clinical trials versus lab experiments?
• What is the relationship between trust in security interventions and lab experiments currently?
• For a security trial to be valid, would the product being tested be allowed to have any updates? Or, would the updates themselves be part of the trial?
• Why do we trust that security technologies actually improve end user or organizational security? Is that trust earned or deserved?

Notes

Lecture 13
----------

 - early lit reviews due on Monday night
 - you can have extra time if you talk to me first!
 - I expect this to be a draft that will be revised, so I'm looking for honest effort not perfection (progress towards the final project)

G1
--
 - first two questions were not too clear to us
 - labs are lower trust environments
 - clinical trials are higher trust in aggregate
   - also more organic, more complex
 - updates should not be allowed during the trial, reduces how controlled the experiment is, more confounding variables
   - updates => evaluating people behind product rather than just the product
   - can you separate company from the product?
   - past experience with organization may increase trust separate from actual
     product performance

G2
--
 - clinical trials have much higher level of trust, you're doing experiments on humans vs cells or animals
 - updates could be part of a trial but you'd have to restart the trial
 - computer part of computer security system is more predictable than biological systems
 - trust should be earned but isn't in practice

G3
--
 - trials account for population variability beyond what can be done in a lab
 - relying too much on standard methods can get outdated, making sure tests adapt to the changing real world
 - could have a subscription module so one group could get updates while the other didn't
 - people assume security, that things work, because alternative would be too hard

G4
--
 - maybe updates should be part of the trial, as ability to adapt to new threats is part of what we want to evaluate
 - which product is most likely to stay up to date with current threats after the trial? the one that was getting consistent updates
 - products out on the market showed that they work in the past, but new ones are more risky, trust needs to be earned
 - defective product costs more to run, so experience can help indicate trustworthiness of a security product