The New School of Information Security

On the forty-second page of “The New School of Information Security” authors Adam Shostack & Andrew Stewart, wrote (some emphasis added):

be disproved. If the position can't be tested, it's a belief—not science. This chapter examines sources of evidence and the value they might provide. In doing so, we can evaluate to what degree they can help us evaluate claims, large and small, and make the right security decisions.

Someone with no training in physics might think that heavier objects fall faster than lighter ones. But we know through scientific testing that all objects falling toward the Earth accelerate at the same rate (although they may be slowed by air resistance). It was Galileo who posed the hypothesis that everything falls at the same rate. He supposedly performed a test in which he simultaneously dropped two objects weighing different amounts from the Leaning Tower of Pisa.

The scientific orientation has been incredibly effective at increasing our understanding of the world. It includes formulating and testing hypotheses and sharing the methods of testing and results of those tests. A hypothesis is simply a testable suggestion. Tests never actually prove a hypothesis. Good tests fail to disprove the hypothesis being tested and thereby provide evidence in favor of the hypothesis. The difference is both subtle and important. For example, someone might hypothesize that the coelacanth fish is extinct, yet this was disproven in 1938 when a living specimen was caught. It's impossible to prove a negative, because there's always the possibility that a counter-example is out there somewhere. We might have other ideas we believe, but evidence to disprove them might be lurking right around the corner. The disproof of a hypothesis may or may not make it worthless. For example, Newton's laws of motion are still used in civil engineering, even though they are "wrong" and don't apply at very high speeds or at atomic scales.

The ideal way to test a hypothesis is by experiment. Good experimental design changes one variable at a time and sees what else changes, or pits two hypotheses against each other.

More information about “The New School of Information Security” (and the book itself) is available from:

• Amazon.com
• Powell's Books
• Barnes & Noble

(Addison-Wesley Professional, April 2008. Hardcover, 238 pages. ISBN: 0321502787; EAN: 9780321502780.)