Harrison Hoffman at CNET thinks that the sudden elevation of The Dark Knight to the top film at IMDB illustrates a hole in the “wisdom of crowds” model. I think this is a bit off. For one, if we’re taking “the wisdom of crowds” to mean the bundle of concepts articulated in James Surowiecki’s book, then the caveats Hoffman is pointing out about the importance of structuring aggregation to prevent herding effects are already incorporated into the idea. And I’d always taken the point of peer-production of information to be that we accept the tradeoff in near-term accuracy for a process that yields better results over the long term: Don’t release right, release now. Presumably TDK will eventually slide to a more defensible spot on the list.
The better critique, I’d think, would point to the IMDB rankings without TDK. I realize, of course, that people can haggle endlessly over this or that omission, and at some point you’ve got to shrug and say de gustibus non disputandum—but there are some points where you really can say a list is just plainly indefensible. I’m not going to hold this up as a perfect alternative, but I think it’s instructive to compare the Village Voice‘s 100 Best Movies of the 20th Century list. Citizen Kane is at the top, which is where you tend to find it pretty much every time critics are asked to compile one of these lists. The number two movie on the IMDB list, The Shawshank Redemption, is (justly) nowhere to be found. And there’s not the heavy bias toward recent films (even taking into account that it’s explicitly a 20th century list) you find on the IMDB list.
Of course, critics lists have their own biases. Movies like Birth of a Nation and Steamboat Bill, Jr. are really up there because of their historical significance: In a vacuum, taken out of the context of the evolution of cinema, they’re plainly inferior to works ranked far below them. But critics can at least make their rankings relative to a huge body of films. The notable thing about the IMDB list is that, with a few exceptions, it’s heavily loaded with movies that the average person in their 20s or 30s is bound to have seen at some point. And this, it seems to me, runs somewhat counter to the purpose of a list like this. The idea of a “250 best films” ranking is, in large part, to give people some sense of what they might want to watch next time they’re filling up their Netflix queues. But what it’s ultimately picking up is that a lot more people have watched Pulp Fiction or The Matrix than have seen Rashomon or Berlin Alexanderplatz. And indeed, the former two are probably a better choice if you’re looking to have a few friends over for beer, popcorn, and a movie. But the IMDB ranking isn’t actually giving people much useful information: They’re up as high as they are precisely because everyone already knows about them.
So what can we say more generally? Again, most of this ground has been covered by James Surowiecki, but we can note that, above and beyond the herding problem, if you’re going to rely on “the wisdom of crowds” to rank a series of items, you need to account for different levels of exposure to the ranked items. An algorithm like IMDB’s might work somewhat better for music, since the average person has listened to many, many more songs than they’ve seen movies (which require a few hours of your undivided attention), but in both cases, you’d probably get better results by taking a page from John Stuart Mill, who recommended that pleasures be ranked by “competent judges” who are familiar with both. In other words, after gathering a bit of information about (say) genres or periods of movies a user is familiar with, IMDB could present a series of semi-randomized pairwise comparisons, along the lines of Facebook’s “Compare People” application. If and only if the user has seen both films, she simply chooses the one that she believes should be ranked higher.
Update: Commenter Christopher objects that this is just a misplaced demand for any “best movies” list to favor the “top 100 movies of the cultural elite”. Since I expect he won’t be alone in that reaction, I’ll copy my reply up here.
Certainly, as I point out above, part of the problem is that there are lots of different dimensions along which a movie can be “best”: Again, the best movie to sit and watch over beers with your friends, or to see on a first date, might not be the “best” in terms of elegant narrative and character development or innovative cinematography or whatever.
But first, even bracketing that, the exposure problem remains. Let’s say you just want the best action movie or the best kung-fu movie. A handful of people will have seen lots and lots and lots and have a decent basis for comparison; the majority will only be able to vote for the ones that are already well known. If this yields results that aren’t wildly off, it’s going to be primarily because what gets to be well known over time already incorporates a certain amount of expert evaluation.
Second, I want to push back at least somewhat against the idea that the “cultural elite” preference is just some kind of niche demographic taste, like “what’s popular in Omaha”. People who are really really into movies display a degree of convergence about which are the great films, and while this may in part be attributable to groupthink, in part it’s because they’re picking up on powerful visual or narrative techniques that anyone could appreciate with a well cultivated eye. Being part of the “cultural elite” just is developing the kind of eye that responds to more complex and original (or, if you will, “better”) visual storytelling. If what’s constitutive of your niche demographic is that it’s composed of people who are better attuned to good filmmaking, then yeah, their preferences probably should be more salient. Not because they’re inherently wonderful people, but because they’re further along the curve that you, too, average moviegoer, would traverse if you watched many movies attentively.
9 responses so far ↓
1 Christopher M // Jul 29, 2008 at 11:41 am
I haven’t read Surowiecki, so I don’t know if he gets into this. But the real problem here seems to be that what you want out of a “top 100 movies” list is actually more like a “favorite 100 movies of the cultural elite.” And really, there’s not much reason (even accepting Surowiecki’s wisdom-of-crowds theory) to think that a bunch of non-cultural-elite people are going to be useful at putting that list together.
You can certainly quibble about the definition of “elite” here, and I am very sure that taste in film follows a much finer-grained structure than elite vs. non-elite. The real point, though, is that the “wisdom of crowds” functions very differently — if at all — when you’re talking about matters of taste vs matters of fact, functionality, etc.
2 Julian Sanchez // Jul 29, 2008 at 12:42 pm
Right, look, obviously part of the problem is that there are lots of different dimensions along which a movie can be “best”: As I suggest above, the best movie to sit and watch over beers with your friends might not be the “best” in terms of elegant narrative and character development or innovative cinematography or whatever.
But first, even bracketing that, there’s an exposure problem. Let’s say you just want the best action movie or the best kung-fu movie or whatever. A handful of people will have seen lots and lots and lots and have a decent basis for comparison; the majority will only be able to vote for the ones that are already well known. If this yields results that aren’t wildly off, it’s going to be primarily because what gets to be well known over time already incorporates a certain amount of expert evaluation.
Second, I want to push back at least somewhat against the idea that the “cultural elite” preference is *just* some kind of niche demographic taste. People who are really really into movies display a degree of convergence about which are the great films, and while this may in *part* be attributable to groupthink, in part it’s because they’re picking up on powerful visual or narrative techniques that anyone could appreciate with a well cultivated eye. Being part of the “cultural elite” *just is* developing the kind of eye that responds to more complex and original (or, if you will, “better”) visual storytelling.
3 That Fuzzy Bastard // Jul 29, 2008 at 12:42 pm
There seems to be a real split on what a Top 100 Movies list means, too. If you want a list of 100 most important-influential-thoughtful movies, the Voice’s list is the place to go. If you want 100 most immediately enjoyable to a relatively culturally-uneducated person, the IMDB list would be a much, much better guide. So in that sense, the crowds are working perfectly well at telling you what *they* like, just not at telling you what a cultural elite (and I use that term with the utmost affection) would like.
4 Peter Orvetti // Jul 29, 2008 at 1:37 pm
Any best/worst lists that have continued for decades tend to be self-perpetuating. “Kane” is one of my favorite films, but it is not so superior to other great films that it should obviously be ranked tops on every list. Then there’s something like “It’s A Wonderful Life” — another fine film, but one that was basically ignored for a quarter-century then embraced as a holiday tradition. It is probably not one of the 100 best U.S. films ever.
Presidential rankings are pretty much the same. Why is James Buchanan the Worst President Ever? Well, uh, because… he was the last time we ranked them! Ditto Grant, etc.
5 lemmy caution // Jul 29, 2008 at 2:18 pm
The critics list is better.
This isn’t to say that IMDB ratings are a bad guide to things. The films on the critics list get good reviews on IMDB. Pather Panchali got an 8.0 for example. People just really love The Shawshank Redemption.
6 Tom B. // Jul 29, 2008 at 2:59 pm
I think there’s a happy middle to be found between the “elite” and the “mob.” Both lists are basically flawed in my eyes. They’re mostly populated by quality films (moreso the Village Voice than the IMDB list in my mind), but there are some very obvious trend lines in both. The IMDB list seems to be based upon both the visceral appeal of a film (a superficial judgment of whether or not one enjoys a film) and the currency of a film in current popular culture. I’d argue that those are both legitimate grounds to include in judging a film, but they’re very incomplete ones.
The Village Voice list tries to strike deeper at the actual components of filmmaking itself as well as the historical import of a film. At one level such a perspective does lead to a deeper understanding of films, but it does so at the cost of negating the visceral impact of a film (or accessibility, if you will) and ignoring more recent productions (it’s very difficult to put a historical perspective on films made recently, hence there’s an incredibly small number of films post-1980 on the Village Voice list).
I like the thought that Julian brought up of forming such lists based on one-to-one comparisons of films. It would help to control for selection biases and it would balance the valid, if somewhat shallow, values of the casual moviegoer with the values of a more reserved, historically-oriented cinephile elite.
7 Timon // Jul 29, 2008 at 7:45 pm
There is an inherent problem with no-stakes voting that excludes it from the benefits of the “wisdom of crowds” — that wisdom is only present when the crowd has an interest in being correct. If you asked the same crowd to submit 5 movies that they thought would be on a critic’s list of 100 best, and they were rewarded if their choice ended up on the list, that would be the Surowiecki thesis in action. You could do the same for, say, favorite sci-fi movies of the 80th percentile by income of sysadmins at the top 10 hosting providers as ranked by Netcraft, and with a big sample of people who know a bit about movies, you would probably get better results than any given member of that group.
The wisdom of crowds idea is not some brilliant scheme for how to get uncompensated labor, it is a lesson in Hayekian distributed knowledge — if you ask internet users to name their favorite movies you will get… a list of movies internet users feel the need to share their love of.
8 Jeff // Jul 31, 2008 at 12:17 pm
All good points, except for your unnecessary slap at Buster Keaton’s Steamboat Bill Jr., which, while not as good at The General, is still a great film.
9 Julian Sanchez // Aug 6, 2008 at 9:18 am
Jeff-
Sure, I don’t mean to rag on that particular film, really, just to point out that there’s a bunch of different dimensions that go into these rankings. We often (rightly) regard something as a great work of art because it’s innovative or groundbreaking given the tools and methods available at the time, even if, considered ahistorically, later works make more effective use of those innovations.