Dan Luu
patreon
The glorification of elite teams
🕑 Added 2021-01-25 01:59:19 +0000 UTCI've been reading older, influential, writing in software (Brooks, Yourdon, etc.) and here's a thought that I suspect I'll never flesh out enough to turn into a post on the "real" blog: I've been reading his "new" 2010 book, The Design of Design. As is the case in his better known book, Brooks really has a thing for superstar engineers and teams. He goes on about how the only way to do good design is to have an elite designer do the design. And on the flip side, he trashes designs that have input from a wide variety of designers or constituents.
For example, to illustrate a point, he uses the Comanche helicopter's requirement of being able to fly itself across the Atlantic as an example of an obviously stupid requirement that came out "design by committee". The way he tells the story, any idiot would know that this is a dumb and pointless requirement that will cause cost overruns and general bloat (the chapter is about project bloat due to obviously dumb requirements; in this section he's trashing design by committee, which is opposed to his favored design styl, design by one or a few genius designers).
When the Comanche was eventually cancelled, one reason cited by the GAO was that the Comanche would be, "a voracious consumer of strategic airlift" (since they failed to hit the target weight necessary for it to be able to fly itself across the Atlantic with the amount of fuel that it could carry). But then perhaps the requirement wasn't stupid after all. Maybe that's a sign the project shouldn't have proceeded in the first place since the goal was difficult to achieve and perhaps someone could have predicted that they wouldn't be able to achieve a critical requirement that would make the project strategically non-viable with then-available technology, but that's sort of the opposite of the requirement being stupid. The problem would have then been that no conforming design could have met the strategic requirements. That's important information, not stupid bloat!
Brooks cites the Spitfire (a WWII airplane) as an example of good design done by a designer instead of a stupid committee. But that plane was from an era where it was very common for airplanes to be designed by one primary designer working on a small team, both good and bad. You could just as easily cite the P-36 as evidence that designs should be done by committee instead of by small teams or individuals since the P-36 was a middling aircraft.
HN commenters are also enamored of military aircraft designed by small, elite, teams. One that repeatedly comes up are the Lockheed Martin "Skunk Works" designs and Kelly Johnson. Their first major design was the P-38, a WWII era fighter which fared extremely poorly in Europe, where it had to go up against superior German designs (an allied evaluation of the P-38 when compared to the German competition stated bluntly, "it was useless").
It did fine in Japan since Japan was unable to improve on their designs after the start of the war in some key was until it was way too late but, by the middle of the war, essentially anything not obsolete would've fared well. P-38s were relegated to the Pacific front not because they were particularly great there compared to other U.S. aircraft; they were just terrible on the European front. Since they were being manufactured anyway, it was good to get some value out of them.
The Skunk Works then produced the P-80, which obtained the dubious distinction of being the first aircraft shot down in jet vs. jet combat during the Korean war. Despite being designed and produced well after the WWII-era Me-262, an evaluation of the P-80 found it to be inferior to the older Me-262. The P-80 did not fare well against the MiG-15 during the Korean war, although the non-Sunk Works P-86 did fine (one could argue whether the P-86 or the MiG-15 had the advantage, but there's not really a way to make a case that the P-80 was competitive with the MiG-15).
The next fighter that came out of the Skunk Works was the F-104. This one fared much better than predecessors, although 21 pilots were killed due to a design flaw in the initial ejection seat design combined with other difficulties (and by the way, the P80 was also known for killing test pilots and pilots due to flaws). The F-104s combat record wasn't better than the P-80's, either; up until the F-104 was replaced by the F-4 (a non Skunk Works design), its record in Vietnam was 0 kills and 14 losses.
The more HN-famous Skunk Works designs are the U-2 and the SR-71. While these were both technological marvels, the U-2 became obsolete for its intended purpose pretty quickly. The SR-71 did not become obsolete quickly, it had a very high operational and maintenance burden and was incredibly expensive per sortie. The SR-71 is the opposite of "use boring technology", it uses state-of-the-art barely operable technology everywhere possible and was incredibly difficult to operate. When used in Vietnam, on average, each SR-71 was able to fly one sortie per week and it was only able to average that rate due to the tremendous resources poured ino operating it. Whether or not it should've been retired when it was is still a matter of debate, but the extreme measures necessary to operate it certainly aided opponents of the program, who were successfully able to kill it on the grounds of its cost to operate, among other issues.
Since we're talking about design here, one thing to note is that the MiG-25, although quite different from the SR-71, was used by a number of countries in a similar role, as a high-altitude and fast reconnaissance aircraft after the SR-71 was retired despite the MiG-25 being designed and produced before the SR-71. If you look purely at the paper specs, the MiG 25 is inferior for the role (lower ceiling, lower top speed), but it was much cheaper to produce as well as cheaper to operate. It was also less specialized (it's actually a fighter jet and had a decent combat record as an interceptor). And in fact, if you were willing to operate the MiG-25 as expensively as the SR-71, it could hit the same speed the SR-71 was designed for (Mach 3.2); this was observed once when a MiG-25 was pushed to its limit to evade a missile. Although this was possible, it wasn't done as a matter of course because it could damage the engines, requiring lengthy inspection and possible repair (but still not as lengthy as the normal operational burden of operating an SR-71 within its intended limits).
If we look at aircraft designed in ways Brooks thought couldn't work, the F-15 is a prime example. Like the Comanche design process that Brooks trashed, it had all sorts of requirements imposed on it by people outside of the design team. Worse yet (and this is something else that Brooks explicitly calls out as bad bad bad), requirements were revised and new requirements came in during the design of the aircraft. The result was a multi-role aircraft that had arguably the best combat record of any aircraft of the era. The revised requirements were a key part of this, as the new requirements came as a result of new information on Soviet aircraft design. As with the Comanche requirements, these "design by committee" requirements came from real "business" needs!
If we look at the F-16, another HN darling because it was designed by a small, rogue, team (and explicitly as a reaction to the F-15, for reasons HN loves and I suspect Brooks would have loved), its record is not as good as the F-15. Moreover, the aircraft is only useful today (or even in the 90s) because the designers were forced to adulterate the design from their intention: a pure air-to-air guns-only dogfighter. But no one has a use for a special-purpose air-to-air aircraft today, especially not one specialized for guns-only air-to-air combat. It turns out that the lesson the initial genius designers took away from Vietnam (air-to-air missiles aren't very good, allowing a superior dogfighter to usually win with guns) was temporary and not fundamental to missile technology. There's a chance that stealth technology will change that, but if it does, well, the F-16 isn't stealthy and it will be completely obsolete.
Now, I'm not saying that design by committee is good or that small teams of designers are bad, or that you should want changing requirements, but when you look at the set of things Brooks and others cite as reasons to believe that you should trust a small, elite, team of designers, they don't seem particularly good, nor do Brooks tenets of design seem to produce good designs as a matter of course nor does violating his tenets of design produce bad designs.
In some sense, this is another variation on https://twitter.com/danluu/status/1261216746687483904 . Being elite, as in being prestigious, is different from being elite, as in doing great work.
Comments
Wow, that's an incredible essay. Thanks for the link! Also, given that you liked that essay, in case you haven't seen it, I wonder if you might enjoy Richard Lanham's Analyzing Prose.
It is my duty to inject into every conversation about Fred Brooks a link to Richard Gabriel's paper "Designed as Designer": https://www.dreamsongs.com/Files/DesignedAsDesigner.pdf
I would say, as written, this post relies heavily on military examples and much of the context for that comes from reading I did 20-25 years ago. I'd want to re-research this stuff in detail before writing it up as a "real" post, but there's a lot of other stuff that I think I want to do before re-reading a bunch of military history (I did skim wikipedia for this post, but I only have some level of confidence that I didn't get the wrong impression from wikipedia due to prior knowledge which is very rusty). Maybe I could do a version which uses other examples (Brooks also cites UNIX vs. Windows, which I don't think is the slam dunk Brooks thinks it is).
Interesting! The Mythical Man Month was one of the first programming-related books I've read, and I approached it with very little critical thinking (it was highly praised in every mention I encountered), so it definitely shaped a lot of my views. I should probably revisit some of those views. IMO, it's worthy of a blog post; the point seems strong to me.
