JavaZone 2007: Rational decisions when choosing methods and tools

"We humans like to analyze, yet we don't trust the result. We trust the result of our gut feeling, but not the process."

(Norwegian talk, any errors in the translation are entirely mine)

Magne Jørgensen is a research scientist at Simula Research. I've seen an earlier session of his on estimation, where he showed how wildly estimates can vary based on things like line spacing in the design document, etc. Interesting stuff.

Magne's method is based on the scientific method, made more practical for the real world. Similarly to his earlier session, it's about what our decisions are influenced by, both important and unimportant things. He's written about "Evidence-based software engineering", where you try to base your decisions on facts backed by evidence. This doesn't sound like rocket science, but it's harder than you think to not be influenced by inconsequential stuff. As humans, we rationalize.

The session started (as usually) with an exercise. Everyone in the session tries to answers some questions, which are used later in the session to illustrate the main points.

Use reliable sources. Google Scholar, not Google.

It's very easy to rely on anecdotal evidence instead of hard facts. If science says a medicine is ineffective, but your neighbour says it worked for him, it's easy to believe in it.

Don't trust reference customers: They are not a statistically valid sample.

Use benchmark tests: Allow the supplier to solve a problem for you. Pilot projects can be misleading, but better than nothing. Team members are often top motivated and highly skilled.

Magne used a Kent Beck paper as an example. He finds that people normally read this as prose, from beginning to end. He suggests that readers should skip e.g. the start of such a paper, which normally is about building relations with the reader and laying the groundwork for the writers credibility. If a reader skips these parts and skims the rest for arguments and claims he is much more likely to not be influenced by unimportant things.

What's important is to avoid irrelevant information which can influence us, even though we think it doesn't.