Experiences from screening ideasJune 1, 2019
In my organization we have gathered quite some experience in screening ideas. I am responsible for the process from idea submission until idea selection. Every time when we have a new ideation campaign it feels that we must force the sponsor of the campaign to think about the screening and the evaluation of the ideas. The sponsor is focusing so much on the idea description, how to phrase the challenge to get in suitable ideas, that it is hard to think further. I have also noticed that many are reluctant to think about how to do the screening before knowing what the ideas submitted really will be.
Of course, we should not lock ourselves in, but many times it is beneficial to have a sense of how the screening will happen, which the criteria will be, before kicking of the ideation campaign. This far we have been promoting usage of some sort of scorecard methodology, trying to define the criteria already before the campaign. At times this has been very successful, other times we have had to adjust the criteria once the ideas have come in and you have them in front of you. It has happened that we only then realize that there is some important aspect that will not be covered by the proposed criteria.
Still, another advantage of defining the screening and evaluation criteria earlier in the process, is that the idea submitter can be asked to submit specific information then needed for the screening. If this information is not submitted, other information and heurestics will guide the selection process.
When watching Kahneman’s videos on System 1 and System 2, I find that we want to claim all our screening processes to be by System 2. But of course, when filling out a scorecard, there is nothing saying that System 2 would be working at all instances even if you would try to focus. I was happy to notice that we have some processes in place that will according to Kahneman diminish biases. One of them is that all evaluators do their evaluations of the ideas individually. Only after they have submitted their evaluations and comments is there a common discussion around the ideas.
Another action that I have been planning to implement, but I have not yet as our system does not easily allow it, is to anonymize all submitted ideas prior to screening or evaluation of them. Today the evaluator can always see who in the organization has submitted the idea while screening the ideas. As responsible for the process I have noticed a rather large bias due to this. Ideas submitted by persons perceived as our brightest scientist are hardly ever dropped during the screening process, independent of the idea or its maturity. It would be interesting to do some experimentation around this and see the extent of this bias.