OMNILIBRIUM
  Rational Discussion of Controversial Topics
Economics Education Ethics Foreign Policy Government History Politics Religion Science


When can experts be trusted?

melian          30 May 2015 06:22 PM


In the XVth century one could become an expert in a dozen of completely different areas and even contribute to their advancement. Today even the most brilliant scientists are familiar with just a small part of their own field. The fraction of the total human knowledge that can be mastered by any individual grows exponentially smaller and, inevitably, we become more and more dependent on experts.

Unlike, for example, our growing dependence on inanimate technology, the dependence on other humans creates trust-related issues. How can a non-expert distinguish a real expert from a fake one?

The options I can think of are:
  • Relying on the wisdom of the crowd, which basically means accepting as experts those whom the majority believes to be experts. Though better than the blind trust, this method is still rather unreliable.

  • Taking into account only the opinions of educated people. While this might be a step forward, history shows that this method is far from being foolproof.

    Take, for example, the Great Plague of the XIVth century which killed about quarter of Europe. During the plague, thousands of ignorant people fled towns in the attempt to escape the wrath of God (a popular explanation for the calamity). In the meanwhile, educated skeptics observed their flight with derision, having concluded, with impeccable logic, that hiding from the Almighty is impossible.

    It would be wrong to put all blame for this fatal mistake on religion. The XVIIIth century Enlightenment greatly increased the prestige of scientific reasoning compared to theological speculations. Yet, in the field of medicine, ignorance was still often safer than knowledge. Until at least the middle of the XIXth century, a sick person may have been safer when treated by potions of self-appointed village witches than by the university trained doctors with their attachment to bloodletting.


  • Personally, I prefer whenever possible to use the following heuristic. I check whether the supposed experts can demonstrate an achievement which is obvious even to non-experts. For example, most people don’t understand the Maxwell equations. Yet, the fact that with one push of a button they can turn their TVs on demonstrates that physics is a real thing. Thus, when physicists claim the existence of electrons, they deserve more trust than mediums claiming the existence of spirits. Similarly, the drastic decline in human mortality rates demonstrates that medicine has changed a lot since the times when humorism was the standard paradigm.


A more challenging problem is to decide when to trust the experts. The only useful heuristics I know are:
  • Checking for conflicts of interest (unfortunately, they arise almost everywhere).

  • Checking if the experts are unanimous (“asking for a second opinion”)

  • Checking if the question truly falls into their area of expertise.


Do you know other useful methods for addressing these two problems? Which methods do you use when forming an opinion on issues like GW or the validity of social sciences?




Would you like to read similar articles in the future?
      
-2 -1 0 +1 +2
 

                  Post Comment                   




Recommended for You Optimates Populares Centrists

Show comments            Sort by        



melian 4 June 2015 05:30 AM
73%

This may be relevant.


stars0
Reply


ChristianKl 4 June 2015 11:54 AM
65%

This isn't a story about trusting experts but trusting journalists who are not experts.

stars0
Reply



Dahlen 10 June 2015 09:07 AM
72%

I don't find it necessary or proper for me to form an opinion on everything; I only become invested in a position when it's something that demands my attention by impacting me directly (e.g. controversies about some health problem I might have), or when I intrinsically like researching it. Otherwise, I shrug and assume the scientific establishment is right, and possibly make low-effort adjustments to my lifestyle or behaviour (e.g. try to walk and bike more and eat less meat to lower my carbon footprint, or let sociology studies influence my opinion, for your examples).

BTW, why global warming and social sciences? The only people I've seen to reject these two as scientifically invalid, especially in conjunction, were strongly partisan.


stars0
Reply


petergast 29 October 2015 10:27 PM
70%

You might want to read up on the Replication Crisis in Social Sciences.

stars0
Reply


melian 10 June 2015 11:12 AM
69%

BTW, why global warming and social sciences? The only people I've seen to reject these two as scientifically invalid, especially in conjunction, were strongly partisan.

Sure, but isn’t also true of people who accept them?

In the US, democrats outnumber republicans more than 10 to one in “soft” sciences ("hard" sciences are much closer to parity). Based on my personal experience, in Europe the situation is similar (one professor in a leading European university in fact told me that he has to hide his right-of-center views from colleagues).

Can you trust people to fully separate their political views from their work (especially when strong material incentives are involved)?


stars0
Reply



ChristianKl 3 June 2015 07:44 AM
70%

Trust depends on context. If you make a decision you have to consider the costs of false positives and false negatives.

When a drug company wants to get a drug approved we have a high standards for the drug company proving benefits of the drug.
We have a lower bar of evidence for side effects.
We don't need a placebo-blind trial to identify side effects to write on the back of a drug box but we do require the placebo-blind trial for benefits of the drug.

It's not useful to give blind trust to doctors because you think that medicine has increased lifespan. Even if the Rand study would have found that medicine works.
Even if a study says that the average person benefits from a drug you aren't the average person. It's important to actually check whether the drugs you are taking are having a positive effect for you.

You might even use a marker to mark the right leg when going to a leg operation, as suggested on LW:
Avoidable medical errors might be the second leading cause of death after CVD.[1] This makes a hospital visit possibly the most dangerous thing you can do, especially if you are young. In general, you should not assume that medical staff are competent. Triple check dangerous prescriptions. If you don’t know whether a prescription is dangerous, assume it is. Ask medical staff if they’ve washed their hands (yes, this is actually still a major problem). Sharpie on yourself which side of your body a surgery is supposed to happen on, along with your name and what the surgery is for


stars0
Reply



petergast 29 October 2015 10:26 PM
70%

I like your heuristic. I expand it slightly:

1. Can they predict the future?

For example Darwin predicted that fossils of forms intermediate between man and apes would be found, probably in Africa.

On the other hand, psychologists and psychiatrists seem to be very bad at predicting people's behavior - worse than simple machine learning programs.

2. Can they design and / or make things that work impressively?

Your example of physics. On the other hand, macro-economists' predictions seem to be pretty worthless.

3. Can they explain things accurately, and with fewer moving parts than others?

Micro-economics does well here. Theology does badly e.g. it predicts that sincere repentance would be followed by better behavior, which doesn't seem to work very well eg with pedophile priests.


From applying this I find vast classes of 'experts' who are not worth listening to, other than for entertainment.


stars0
Reply


melian 30 October 2015 07:47 AM
74%

On the other hand, macro-economists' predictions seem to be pretty worthless.

I think economics is still in a transitional stage. It has the potential to become a real science, but it is not quite there yet (like astrology or alchemy which gradually transformed into astronomy and chemistry).

Theology does badly e.g. it predicts that sincere repentance would be followed by better behavior, which doesn't seem to work very well eg with pedophile priests.

Is this expected in all religions? My impression was that non-christian religions put a lot less emphasis on repentance.


stars0
Reply


petergast 30 October 2015 10:01 PM
70%

> Is this expected in all religions? My impression was that non-christian religions put a lot less emphasis on repentance.

I should have said Christian theology.

> I think economics is still in a transitional stage.

Yes - especially in finance there is a lot of good material now, partly courtesy of people coming onto finance with a physics background.

Lest anyone think I am being unfair, a standard textbook on Economics (Introduction to Economics by Mankiw) admits that the state of macroeconomics is primitive.

stars0
Reply



FrameBenignly 1 June 2015 02:57 AM
69%

The question isn't how likely are the experts to be right. It's are experts more likely to be right than your wild guess. Even if their probability of being correct is quite low; it still should be treated as more accurate than your own opinion.

There is a flaw when they have a clear source of bias which you lack such as their funding sources. Even when no such conflicts exist I'd say people are more willing to ignore experts than to listen to them when they should, and to find a minor excuse.

An important thing to distinguish between is the expert's opinion and the general consensus in their field. The general consensus should be treated with a great deal of respect. Experts may say their personal opinion sometimes for an area that is still highly uncertain, and you should verify their certainty.


stars0
Reply


melian 1 June 2015 11:09 AM
73%

There is a flaw when they have a clear source of bias which you lack such as their funding sources.

The problem is that almost all experts have such sources of bias.
people are more willing to ignore experts than to listen to them when they should

Are they ignoring experts, or do they listen to the wrong experts?

stars0
Reply


petergast 29 October 2015 10:39 PM
71%

Many fields also have blind spots. One example currently relevant to me is the blind spot that exists in medicine in respect of statistics. When you read the medical literature, you see one statistical howler after another. In recent days I have seen

1. Fallacy of composition.

2. A barely statistically insignificant result (p=0.09) interpreted as proof of no association.

3. Confusion of "statistically significant" with "important".

More examples in the book "The Cult of Statistical Significance".

76% of endocrinologists could not answer the following question correctly, and most are not even within a factor of three of the correct answer:

Ten out of every 1,000 women have breast cancer. Of these 10 women with breast cancer, 9 test positive. Of the 990 women without cancer, about 89 nevertheless test positive. A woman tests positive and wants to know whether she has ... read more


stars0
Reply


FrameBenignly 1 June 2015 02:41 PM
70%

"The problem is that almost all experts have such sources of bias."

Almost all people have such sources of bias. Except in extreme cases, the expert's bias is not clearly worse than the average individual's. The funding sources exception I was mostly thinking of was political think tanks which can vary in quality. But there are plenty of academics that don't work at think tanks.

"Are they ignoring experts, or do they listen to the wrong experts?"

I would consider an expert to be someone with several years of graduate-level formal schooling in their specialty. Some experts don't have this training, but these individuals seem more rare. Experts are not hard to find; you will find them regularly quoted in any major newspaper. I seem to encounter a lot more examples of people specifically downplaying formal specialists' opinions when hearing them than of people ... read more


stars0
View Replies (5)
Reply



VoiceOfRa 30 May 2015 11:01 PM
67%

The problem with relying on Expert unanimity is that its subject to Goodhart's law.


stars0
Reply


melian 1 June 2015 10:58 AM
72%

In this case Goodhart's law applies to governments or powerful institutions which can pressure experts into compliance with their agenda, but not to individuals.

Also, I'm not advocating relying on experts unanimity, only using it as one of the factors.




stars0
Reply


VoiceOfRa 2 June 2015 10:44 PM
67%

If people (individually) judge fields by the unanimity of their experts, that creates pressure on fields to find some way to ensure unanimity.


stars0
View Replies (3)
Reply



ChristianKl 3 June 2015 07:44 AM
65%

When thinking about global warming you can go and read mainstream news articles that quote climate scientists to form your opinion of the position of climate scientists.
You can also read the IPCC summary for policy makers. The IPCC document has much higher quality.
It's has confidence values for various predictions surrounding global warming by scientists who went through credence training. It's much more nuanced than the view you
get if you simply get your views from the mainstream media.

If you focus on the core global warming question, there are two things you can be doubtful about.
Is the IPCC confidence interval large enough? Is the uncerainity higher than they claim it is?
Is the average temperature increase prediction of the IPCC right or should we use a different average?

I think as Nassim Taleb that the uncertainty is likely higher than the IPCC numbers. On the other hand I see no
reasonable argument for a lower average temperature increase than what climate scientists predict.


stars0
Reply


melian 3 June 2015 09:23 AM
75%

You may find the following story relevant. In the past the astronomers tried to measure the Hubble constant using many independent observations. Eventually, a new satellite managed to measure it with an unprecedented precision and someone had an idea to analyze all previous results. It turned out that, on average, the astronomers underestimated the uncertainty of their measurements by a factor of two. But the worse part was that the distribution was not Gaussian at all. It seems that either consciously or unconsciously scientists were massaging their data to fit what was in that moment considered the right number, or they were refraining from publishing results that did not fit the consensus.

Now, unlike GW there is no moral or political significance to the Hubble constant. No one was going to accuse astronomers of being sell outs to corporations for publishing the “wrong” numbers or lionize them for publishing the “right” ones. So would it be totally inconceivable that IPCC data is even more biased towards the “right” numbers?


stars0
Reply


ChristianKl 4 June 2015 11:40 AM
67%

If don't know the specifics about the Hubble telescope story. If you have a link to the analysis feel free to share it.

Otherwise I don't see that story providing any information that's new to me.

So would it be totally inconceivable that IPCC data is even more biased towards the “right” numbers?
It's quite possible to publish data that wrong both by providing numbers that are higher or that are lower than those of your colleagues.

I use the phrase confidence interval not because I believe that it's really distributed Gaussian but for convenience.

stars0
View Replies (1)
Reply


FrameBenignly 3 June 2015 10:51 AM
74%

If I gave you a test on standard climate problems; what's the composition of the atmosphere, what is the climate-carbon feedback of methane gas, what is the average ph balance of ocean water, etc. Do you think you'd be able to pass it?

stars0
Reply


ChristianKl 4 June 2015 11:21 AM
65%

I have a few Anki cards for air composition but otherwise, those I don't have deeper knowledge in that area. I also don't think I need it.

I specially link to the policy makers summary because it actually focuses on relevant factors that are useful to know.

stars0
Reply



VoiceOfRa 2 June 2015 10:48 PM
64%

I like Taleb's approach to this problem:

Does the "expert" have skin in the game?


stars0
Reply



Fwiffo 2 June 2015 12:36 PM
61%

If you are not an expert expert you can't know who knows what. However people don't usually have to know that a person can be trusted to actually trust them. It could also be argued that since "proof of trusthworthiness" is a rare thing it is not practical to keep trust to that standard. People in fact trust experst even when they "can't" be trusted. Having had a good faith educaiton to be trained as an expert in that field is a thing you can measure even if it doesn't 100% correlate with ability.

Personally I can live with occasionally having to doubt expert opinion. It helps that for most thinds I dont' need 100% reliability and often if there is something a miss it's more of a question when it is found out rather than if it is found out. I am answering never.


stars0
Reply