Embrace the random

A case for randomizing acceptance of borderline papers

One of the strongest opening lines is from this seminal book on pattern recognition (aka machine learning):

“Life is just a long random walk.”

It stuck into my mind. All I could add was “perhaps, maybe, with a drift” that you control, but that’s it. Our life is full of random events that we later try to fit into a well-constructed narrative, with more or less success.

There are several systems that proudly embrace the random: the US green card lottery, the Vietnam war draft lottery, or, more recently, the Paris medical student selection draw, not without controversy of course. There are others which cling to the deterministic narrative, even at the price of several person-years of work. One of these is our selection procedures of papers in high-profile scientific conferences.

I was an area chair of ICML this year. I had sixteen papers. Two of them were strong accepts with three positive reviews, some enthusiastic. Eight of them were clear rejects, half-written, technically flawed, irrelevant, or without novelty (yes, a surprisingly high fraction). The remaining six were what we call borderline. They were all technically correct, novel, relevant, and clearly written. Reviews varied because of “significance”, a highly subjective criteria based on 80% reviewer taste and 20% experience. The unofficial target acceptance rate (dictated by two factors, largely irrelevant to paper quality: the number of submitted papers and the capacity of the conference program) was 25%, which meant that roughly one third of the borderline papers could be accepted. How did we make the decision? With a lot of hard head-scratching, reading and re-reading the paper and the reviews, initiating reviewer discussions, and a lot of work of the program chairs (I don’t think Nina and Kilian slept a lot that week). What I found the most ludicrous is the realigning of reviewer score: to sweeten the bitter pill of rejection, we pressure the positive reviewers to lower their marks, to make it look like the decision was non-random.

I think we should make a random decision on borderline papers (possibly with a biased coin for authors with multiple papers or with other penalties to discourage flooding the system), or accept all of them but randomly select those to be presented. By borderline papers, again, I mean: technically correct, clearly written, relevant, and novel, but with questionable significance.

Here are my arguments.

  1. We have known it’s random since the now famous NIPS experiment. The numbers above (12% clear accept, 12% random out of 40% borderline) are actually fitting the data. My suggestion does not make it more or less random, it makes us to embrace its randomness.

If you like what you read, follow me on Medium, LinkedIn, & Twitter.

Head of AI research, Huawei France, previously head of the Paris-Saclay Center for Data Science, co-creator of RAMP (http://www.ramp.studio).

Head of AI research, Huawei France, previously head of the Paris-Saclay Center for Data Science, co-creator of RAMP (http://www.ramp.studio).