Dan Luu
patreon
Against epistemic learned helplessness
🕑 Added 2022-08-31 13:31:00 +0000 UTCThere's this theory Jamie Brandon has that often comes up in our conversations about how, for mental development, it's important to do some kind of mental activity that connects to ground truth where there's real right and wrong and people get incontrovertible feedback about being wrong. This doesn't have to be something empirical, e.g., math is fine.
I've generally agreed that's an effective way to develop a set of skills but haven't generally been onboard with it being critical, but I'm leaning more towards Jamie's position after reading Scott Alexander's "Epistemic Learned Helplessness": https://slatestarcodex.com/2019/06/03/repost-epistemic-learned-helplessness/ .
The idea is basically that a better "arguer" can always convince a worse arguer of something, so you should actually reject reasons that seem like they make sense because that just leads to believing whatever the best arguer says. Scott says this is the right approach "for 99% of people 99% of the time".
As we saw in https://www.patreon.com/posts/62933244 , Scott has a very strong belief that IQ / intelligence / smartness is extremely important, so when he says "a better arguer", he generally means a smarter person (and he says as much in the post as well).
Scott does say there's some value in people not adopting epistemic learned helplessness, although it's very risky and could result in believing all sorts of nonsense:
> I’m glad that some people never develop epistemic learned helplessness, or develop only a limited amount of it, or only in certain domains. It seems to me that although these people are more likely to become terrorists or Velikovskians or homeopaths, they’re also the only people who can figure out if something basic and unquestionable is wrong, and make this possibility well-known enough that normal people start becoming willing to consider it.
>
> But I’m also glad epistemic learned helplessness exists. It seems like a pretty useful social safety valve most of the time.
The reason Scott's line of reasoning made me think of Jamie Brandon's thought is that Scott's line of reasoning is something that can only make sense to someone who's never done serious work in something where work has to connect back to ground truths.
In the post, Scott mentions that various crackpots are utterly convincing to him, they "all presented insurmountable evidence for their theories", etc., but of course this is false. The evidence is clearly not insurmountable and, in fact, if you look at it and are decent at reasoning at things (and not necessarily super smart or high IQ or whatever Scott says is important), you can see holes in the "insurmountable evidence"). Quite a few fields teach you how to see through bogus logic if you take the teachings seriously.
For example, math has the concept of "mathematical rigor". Somewhat advanced textbooks will often say something like "prerequisites: this book is self contained, but requires mathematical rigor", meaning that you don't need to know any particular math, but you do need solid mathematical reasoning skills and you need to be able to read a set of statements and not incorrectly infer more than they say, which also implies that you can generally spot specious mathematical reasoning even if it might seem like "insurmountable evidence" to someone who hasn't learned mathematical rigor.
And this idea isn't unique to math, although many other fields don't have a nicely packaged up term for it the way math does. With stats, Andrew Gelman talks a lot about the ideas that one might call "statistical rigor", but never refers to a specific term for it, and likewise for many fields that touch some kind of ground truth because they're linked to the physical world, like large parts of electrical engineering, etc.
Anyway, coming back to Jamie's idea about why it's critical to work in an area that's connected to actual ground truths, I'm less convinced than Jamie because it's possible to be a great at one field and a total crackpot when it comes to other fields, but I do think that someone who either knows that it's possible to actually link things to ground truths or has learned how to reason rigorously (which doesn't mean they won't make mistakes — just that they wouldn't have Scott's mental model of flip flopping between beliefs as they read bogus refutations and then refutations of refutations and so on and so forth) couldn't buy Scott's idea that epistemic learned helplessness is actually good.
For myself, while I studied math and engineering in school, I still had vaguely similar thoughts to Scott (even if there's what seems like a compelling mathematical proof, unless it's in a subfield I really deeply understand, couldn't I just be missing something?) until I spent a good chunk of time in the real world working in a variety of areas where claims can be linked to ground truths.
In particular, a variety of experiences taught me that this is an effective strategy: I generally try to find a "dumb", simple, reason that some system either ought to work if built correctly or can't work and then go from there. People will come up with all sorts of "smart", convoluted, reasons that what I'm thinking is wrong, but after quite a few of these exchanges where my simple reason has held up when we actually went and either built the thing I wanted to build or made the change I wanted to make, I gained decent confidence that if someone can't refute my dumb reason and all they can do is offer a bunch of complex smart-sounding reasons that the opposite position is true, they're probably wrong. I guess Scott would probably say I'm fairly good at arguing and, before I was convinced of this, I did used to argue against the complex smart-sounding reasons, but my experience was that when you refute someone's convoluted smart-sounding reason, they just come up with an even more convoluted and smarter sounding but generally even more wrong reason. Sometimes, as in https://www.patreon.com/posts/you-just-have-to-70415590, they'll get exasperated and give up, but people who do this generally have more patience than me and I'll write them off and stop having discussions with them before they stop coming up with more https://www.patreon.com/posts/38866741 reasons.
I think most people would phrase this differently, but my experience in multiple fields has taught me that looking for the "dumbest" possible reason and the "dumbest" possible systems to build is worthwhile.
Although this is very different in a lot of ways, when Tetlock discusses his work on forecasting (probably most well-known via his book, Superforecasting), Tetlock notes that "superforecasters" (who far above average in making correct predictions) score 80%-ile on IQ whereas the average non-superforecaster who participated in his studies scored 70%-ile. On average, people who are great predictors (who are, in some sense, great at evaluating the correctness of statements) are smarter than people who aren't, but that's not a defining quality and the correlation is fairly weak. I'm told that Paul Graham noticed something similar when he founded YC; he originally expected that smartness would a be very important and it turns out to barely matter at all when looking for people who will found successful startups.
Semi-relatedly, reading Scott's epistemic learned helplessness post really helped me understand something that's been a real mystery to me about Scott's posts where he throws out a bunch of "smart" reasons that can be seen to be wrong via "dumb", simple, reasons to support some position as in https://www.patreon.com/posts/62933244 , https://twitter.com/danluu/status/1254720673592688641 , https://twitter.com/danluu/status/1255006286262857728 , etc., the mystery being why such a large fraction of his writing is full of very smart reasons things are or aren't true when there are much dumber reasons that shed more light on the topic that he doesn't mention.
If your mental model of arguments is that there's some kind of ground truth and you're looking for the ground truth (even if it's the case that, due to the complexity of the world, the data to really resolve the question is uncertain) and can often do so in a dumb way, the style of argumentation that Scott uses is mysterious. But if your mental model is that the way one wins arguments is by coming up with "smarter" reasons than the other person and you try to come up with a bunch of "smart" reasons to support a position, then Scott's posts make a lot of sense. A lot of his arguments are indirect and involve a lot of moving parts and really read as "smart" arguments.
Thanks to Tim Pote for comments/corrections/discussion.
Comments
That's a reasonable interpretation, although I think it requires reading significantly more into the text than Scott wrote, but perhaps that's just the style Scott is using. WRT those domains, I'd say that my preferred approach would be to be skeptical of claims that don't have good evidence, which I think is different from the approach Scott describes. An example of this might be in nutrition, where epidemiological evidence, when tested in a randomized trial, relatively rarely turns up a casual relationship. I think there's a significant difference between saying "your argument is weak or no evidence of the claim" vs. "I don't trust your good sounding argument because I can't trust arguments". If you think nutrition isn't an example of such a domain, I'd be curious what you think of as a good example of a such a domain.
I think Scott would agree that in domains where there is a ground truth you can relatively easily connect to / check, epistemic learned helplessness is straightforwardly bad & you should just try to figure stuff out for yourself always no matter who you are. I interpret him as saying there are many domains that are not like that, domains where the connection to ground truth is long and weak. And in those domains, if you reject epistemic learned helplessness and try to figure things out for yourself, you run the risk of making reasoning errors and believing crazy false things, and worse, believing whatever things are being argued by clever arguers. So maybe it's good, Scott concludes, that most people have epistemic learned helplessness in domains like this. I agree with that take FWIW and would be interested to hear your thoughts on it. That said I do agree that on the margins people should probably have less epistemic learned helplessness, and I agree with your point that if you really practice figuring things out for yourself (focus on connection to ground truth) you'll be able to get better at reasoning about domains where the connection is long and weak also. Better, but still not good enough that you can reliably avoid error.
That description of epistemic learned helplessness definitely describes an attitude I’ve been tending toward, but this post made me want to hold myself to a higher standard. Thanks.
