No story about Nate Silver and 538 is complete without mentioning his new rival in the prediction space - Carl Diggler - who has been kicking his butt on the presidential primaries, calling tough elections correctly where Nate has refused to give more than a "Maybe Sanders, Maybe Clinton..." prediction.
Carl is a work of fiction, invented by a couple of journalists/friends as a parody of pundits: https://www.washingtonpost.com/posteverything/wp/2016/05/09/... - the story is hilarious. Watching this whole thing unfold on Twitter has been a total blast too (check out @carl_diggler), they never break character and have a lot of running gags (such as Carl's troubles in "Family Court"). Here's a wee excerpt from that WP link:
"
Carl exists to satirize all that is vacuous, elitist and ridiculous about the media class. From his sycophantic love of candidates in uniform to his hatred of Bernie Bros, from his reverence for “the discourse” to his constant threats of suing the people who troll him on Twitter, Carl is predicated on being myopic, vain and — frankly — wrong.
But something funny happened along the way. Biederman and I, who are neither statisticians nor political scientists, started making educated guesses for our parody about the results of the primaries. And we were right. A lot.
We beat the hacks at their own game by predicting every Democratic winner on Super Tuesday. We told readers who would win in the unpredictable caucuses that FiveThirtyEight didn’t even try to forecast, such as those in Minnesota, Wyoming and even American Samoa. We called 19 out of the past 19 contests. FiveThirtyEight, whose model cannot work without polling, accurately predicted 13
"
As fun as the story of Carl Diggler is, his work is statistically insignificant. The podcast Reply All (which you should be listening to) asked a professor who studies the science of predictions whether the Diggler guys are onto something [1].
They are not. The professor wanted some kind of statistically significant proof that they were any better than Paul, the octopus who successfully predicted an astonishingly high number of soccer matches by random chance [2].
Say what you will about 538, but at least they care about not using anecdotal or imagined evidence. At least they're rigorous with their methodology. They're also actually good at it, unlike most pundits [3].
The point is that even with 'rigorous methodology' and 'evidence' 538 doesn't do as well as Diggler.
Predicting elections is, for the most part, an easy task. Polls are usually good enough to get some picture of the election at hand. One major shortcoming of a primarily evidence-based approach is how to call races without much polling (small sample size). Silver has consistently refused to cover these races at all.
But Silver commits two sins that you ignore: he wants to show off his 'accuracy' and he wants to improve on the polls. These two problems kind of intertwine - the 538 'polls-plus,' which incorporates predictors Silver thinks are significant e.g. endorsements or Nate's gut, performs worse than the polls forecast, which is just a weighted average of recent state polls. What kind of rigorous methodology allows for two different predictions and then picking the best one, after the fact? What kind of rigorous methodology results in a prediction method that's shown to be worse than reading the newspaper, time and time again? Where's the evidence behind endorsements working this year?
As a side note, Philip Tetlock is the Wharton professor, and Dan Gardner (interviewed here) is some dude.
The part you are ignoring is that there are many potential "Digglers" out there. If Diggler hadn't done well, you wouldn't have heard about it. Buy maybe you would have heard about some other guy who predicted the races correctly based on what kind of treat his horse wanted to eat that day.
That's fair, and I wouldn't bet the bank on his predictions moving forward, but it's valid to consider what Diggler means for Silver's method. Diggler and Cafe.com are important as critics of the media at large. When what passes for journalism is some dudes making multiple predictions and asking you to choose the one that turns out to be closest to correct, that should be lampooned!
Silver wrote a faux mea culpa earlier this year about how he was wrong about Trump and Sanders and so on. My takeaway was that he apologized for inserting his own opinions where he claimed to use data. Isn't that what Diggler does?
If Diggler had bombed I most certainly would have heard about it - I was reading about him before all this 538 stuff, when he had a Hostage crisis in Russia[1]. However if that had happened we would just have a different but equally humourous narrative, perhaps blaming his family court issues, or some other nefarious plot :D
Really you're overthinking it, and you shouldn't feel the need to protect Nate Silver et al, they're completely unaffected by Carl (afaik they haven't even acknowledged his "existence") so don't worry about it
The article they link to here doesn't shift blame to the pollsters at all. While they note that the polls were wrong, they take full responsibility for their bad predictions:
"It’s our job as forecasters to report predictions with accurate characterizations of uncertainty, and we failed to achieve that in this election. We are not trying to make excuses here; we are trying to understand what went wrong."
It's a little bit annoying seeing some people who are satirizing the media class unapologetically misrepresenting others in the same breath.
> We called 19 out of the past 19 contests. FiveThirtyEight, whose model cannot work without polling, accurately predicted 13.
FiveThirtyEight's model could work without polling (it's not the only predictor they use), but they don't publish results until there are at least two polls conducted. I would call that a feature, not a bug.
Also, FiveThirtyEight correctly predicted the outcome of every primary race that it modeled, on both sides, except one (Michigan Democratic primary).
It's unfortunately burned into my mind, because of the 24 bets I made on the primaries this year, it's the only one I got wrong, but I lost $75 on that bet.
538 is the classic "fighting the last war" problem. In 2012 the issue was that polling was unreliable, it was difficult to figure out what was actually going on with the electorate. FiveThirtyEight figured out how to crack that problem, by applying number crunching. This cycle, the problem is that the electorate hasn't actually decided what it wants yet, and there statistical methods aren't helpful at all. There being many examples of 538 "predicting" a 99% probability of a certain victory due to a large margin from polling and then on election day things going completely differently because the electorate changed its mind.
> There being many examples of 538 "predicting" a 99% probability of a certain victory
I think there is only one example of this kind: democratic Michigan primary
Was 538 that good at predicting outcomes of the 2012 primaries? I know it's well established that they did a good job on the 2008 and 2012 general elections.
Primaries have to be harder because there is far less polling information compared to the general election. Yes, they were unable to predict the Trump victory, but if the get 90-100% of states correct in the general election, then I doubt that much has changed with how polling reflects the electorate.
They were able to predict Trump winning. All of their data showed it. The polls showed Trump winning everywhere. What they failed to do was trust the statistics. The polls show Trump as favorable, and they say "yeah but he won't make it past the first state." The polls show Trump beating other candidates and they say "he has a ceiling of 30%". Other candidates start to drop out and polls still show voters rallying behind Trump, and they say "the polls are wrong, the voters will go to Cruz".
I'm not a Trump supporter by any means, but 538 went way out of their way to ignore every poll that was put in front of them in favor of punditry.
It's not soluble, even with mind reading, because people change their minds.
The correct solution is just to move on. Too much of the media coverage of elections is about the horse race aspect, almost none of which is actually of positive benefit. Rather than spending so much time trying to guess winners, the media could spend more time and effort actually covering the actual issues of the election in a mature fashion, but it's been decades since they even tried.
Sorry to seemingly play a game of one-upsmanship, as 538's repo is wonderful, but BuzzFeed also posts its data and stories online on their BuzzFeedNews Github account...however, what I really like about their repo setup is that they have a repo for every project...and then a separate repo that has a tabular listing of repo, date, description, and story link * ...IMO, they have the best setup (in terms of discovery) among the news nerds:
- * edit: Oops, I forgot that 538's data repo also has a readable table listing below the fold...I suppose having separate repos per dataset vs one main repo is not a clear advantage depending on the user.
They also list their standalone datasets and tools, e.g. their standardized H-2 certification data [1], for which they've probably used in their many H-2 visa stories, and twick [2], the tool they use to quickly fetch newsworthy Twitter account data on short notice...
And those aren't even all the useful tools that that team has created...Jeremy Singer-Vine (their data editor) has several great Python utilities in his personal repo (github.com/jsvine), including waybackpack, pdfplumber (a wrapper around pdfminer and similar to Ruby's tabula), and markovify.
That said, there are a few reasons I've seen writers/researchers intentionally not release data. Either the code is sloppy/inefficient/embarrasing, or the data is used in bad content marketing for a startup that specializes in data collection so that they can say "we have data, pay us if you want more, neener neener neener." (The latter of which I see submitted to HN all the time and serves as a pet peeve of mine)
I found it interesting that the author was willing to catalog each curse word by writing it in its entirety -- except one, the n-word (which, by the way, appears 179 times in Tarantino's movies, with the bulk in "Django Unchained" and "Jackie Brown"). The article for which this data set was compiled similarly censors none but the n-word:
I happen to think that no one of any ethnic group should use the n-word, and I do not use it myself, but I think in the context of cataloging curse words, where you're writing at a sort of meta level (i.e. writing about words, rather than invoking them yourself) it's just as acceptable as in the other cases to write the word out. But the fact that the author made this one exception reveals a bit about how he sees the word in relation to the others.
For instance, you might argue that the varied versions of the f-word might be strong language, but in a different category than the n-word because the latter is an ethnic slur.
But the author has chosen to write out other ethnic slurs in their entirety (e.g. the w-word, referring to Mexicans, and the j-word, referring to Japanese people, and the g-word, related to Asians), and also slurs related to other groups (e.g. the f-word, referring to gay people).
So, even if you're talking about just the set of curse words that insult a particular group of people, the author has set the n-word apart.
n-word -> nigger and f-word -> fuck I understand, but what is the w-word?
I really don't get this obsession over hiding the existence of these words. They obviously don't exist in isolation and are a symptom of a different problem. Perhaps by hiding them people seek to pretend the world has solved these problems?
On top of that, their usage as negative words is only propagated by these shortenings. Words like nigger had (and still do have) neutral and positive connotations [0].
Language is something I deeply believe in, and try to use accurately. People controlling which words I can and can't say really, really, upsets me at level I can't quite find the words express.
The only time I've ever heard the word was in discussions surrounding its existence as a racial slur. There are much more common slurs for Hispanic immigrants. But I've never heard it called "the w-word".
> I really don't get this obsession over hiding the existence of these words.
You've kind of answered your own question there. If you hide the words, then people who don't know them (eg children), don't learn a new offensive term.
I'm not saying I support the policy; I'm just saying that I understand why people do it.
Um, yes, yes they do. Obscuring words is to protect you from the reminder of the terrible history human kind has had toward people that look different from you. It doesn't protect a child's innocence.
I'm reading through Nate Silver's Signal vs. Noise book right now - highly recommend it. And this repo is an awesome way to replicate the results that they show on their site (and perhaps some stuff in the book too).
Seriously though... The number of researchers who expect us to accept wild claims at face value without releasing data is unbelievable. It should be a prerequisite for most journals since it would help protect the journals from false experiments.
Data is nice, but even with it, 538 has gone alarmingly down the path of clickbait lately. Especially Casselman, with smarmy headlines like "The Rising Unemployment Rate Is Good News" or "Stuck In Your Parents’ Basement? Don’t Blame The Economy".
Carl is a work of fiction, invented by a couple of journalists/friends as a parody of pundits: https://www.washingtonpost.com/posteverything/wp/2016/05/09/... - the story is hilarious. Watching this whole thing unfold on Twitter has been a total blast too (check out @carl_diggler), they never break character and have a lot of running gags (such as Carl's troubles in "Family Court"). Here's a wee excerpt from that WP link:
" Carl exists to satirize all that is vacuous, elitist and ridiculous about the media class. From his sycophantic love of candidates in uniform to his hatred of Bernie Bros, from his reverence for “the discourse” to his constant threats of suing the people who troll him on Twitter, Carl is predicated on being myopic, vain and — frankly — wrong.
But something funny happened along the way. Biederman and I, who are neither statisticians nor political scientists, started making educated guesses for our parody about the results of the primaries. And we were right. A lot.
We beat the hacks at their own game by predicting every Democratic winner on Super Tuesday. We told readers who would win in the unpredictable caucuses that FiveThirtyEight didn’t even try to forecast, such as those in Minnesota, Wyoming and even American Samoa. We called 19 out of the past 19 contests. FiveThirtyEight, whose model cannot work without polling, accurately predicted 13 "