Confidence tricks: The ignorant pundit is absolutely certain; the true expert understands their own limits and how to ask the right questions

The authors, academics at UC Berkeley and Columbia University, highlight that it has become increasingly hard to distinguish genuine experts from snake oil salesmen: “In every domain where decision-makers need the specialised knowledge of experts, those who don’t have the relevant knowledge – whether they realise it or not – will compete with actual experts for money and attention. Pundits want airtime, scholars want to draw attention to their work, and consultants want future business. Often, these experts are rightly confident in their claims. In the private market for expertise, the opposite can be more common. Daryl Morey, the general manager of the Houston Rockets basketball team, described his time as a consultant as largely about trying to feign complete certainty about uncertain things; a kind of theatre of expertise. In The Undoing Project (2016) by Michael Lewis, Morey elaborates by describing a job interview with the management consultancy McKinsey, where he was chided for admitting uncertainty. ‘I said it was because I wasn’t certain. And they said, “We’re billing clients 500 grand a year, so you have to be sure of what you are saying.”’”
So how can we identify real experts as opposed pretend experts? The professors draw a distinction between three types of questions that are out there to be answered: “…it helps to divide the difficulty of questions into three levels. Level-one questions are those that anyone with even modest expertise or access to a search engine can answer. Some political economy questions in this category include, for example, ‘Will price controls cause shortages?’ or ‘Are incumbent governments likely to do better in elections when the economy is performing well?’
Level-two questions are those where only the most qualified experts have something to say. Some political and economic questions that we believe fall into this category are ‘Can we design algorithms to assign medical residents to programmes in an effective way?’ (yes) and ‘Do term limits improve governing performance?’ (no). These are questions for which substantial peer-reviewed scientific literature provides answers, and they can be addressed by what the American philosopher Thomas Kuhn in 1962 called ‘normal science’: that is, within existing paradigms of scholarly knowledge.
Level-three questions are those where even the best experts don’t know the answers, such as whether the death penalty lowers violent crime, or what interest rates will be in two years. Such questions are either not answerable given current research paradigms, or just more fundamentally unanswerable. Much of the scientific enterprise itself consists of distinguishing between when further research or information will make questions answerable or not. Importantly, for the purposes of policymaking, it doesn’t necessarily matter why we can’t know the answer. So, for communicating about science with the public, the distinction between level two (questions that require true knowledge) and level three (ones that truly aren’t, at least at present, answerable) is most important.”
The professors then discuss that it is often difficult to distinguish between level two and level three questions and it is here that the role of the expert is critical: “The key difference between these kinds of questions is ‘Would a competent expert well-versed in the relevant scientific literature be reasonably confident in the answer?’ Note that the question is about both the competence of the expert and the answerability of the question.
This means that, when making decisions that require expert perspective, it might be a mark of a true expert to admit that he or she doesn’t know, at least not yet.”
Ironically, therefore the true expert is the one who says “I don’t know” (as opposed to the guru who tells his patron what the patron wants to hear). This in turn creates a simple trick which you & I can use to identify real experts: ask a question which we know is on the outer limits of what science or finance or economics can answer and then see how the expert responds.
If you (and the expert) have a sense of humour, you can try a variant of this test used by the US chat show Jimmy Kimmel: “In one of our favourite bits on the Jimmy Kimmel TV show – ‘Lie Witness News’ – interviewers troll the streets of New York asking impossible questions such as ‘Is it/ time to bring US troops home from Wakanda?’ Interviewees inevitably rise to the challenge, answering confidently and in (imaginary) detail. Our work suggests that, in the presence of reputational incentives, the market for expert advice might not be much better….”