Dollars and Jens
Monday, September 12, 2011
Not Statistically Significant Does Not Mean Zero
I know I've seen this mistake in an accounting paper or two, but I haven't seen it a lot.
In theory, a comparison of two experimental effects requires a statistical test on their difference. In practice, this comparison is often based on an incorrect procedure involving two separate tests in which researchers conclude that effects differ when one effect is significant (P < 0.05) but the other is not (P > 0.05). We reviewed 513 behavioral, systems and cognitive neuroscience articles in five top-ranking journals (Science, Nature, Nature Neuroscience, Neuron and The Journal of Neuroscience) and found that 78 used the correct procedure and 79 used the incorrect procedure. An additional analysis suggests that incorrect analyses of interactions are even more common in cellular and molecular neuroscience. We discuss scenarios in which the erroneous procedure is particularly beguiling.I suppose real scientists don't have to understand statistics as well as we do because they can do controlled experiments, so the statistics they need is a lot simpler.
Labels: statistics
Thursday, August 13, 2009
Unemployment Statistics
I hope Bob McTeer will forgive me for copying his entire blog entry.My local newspaper this morning showed a picture of a guy standing in a line of job seekers at a Career Expo. He had his cap on backwards.
If, in the unlikely event, he were contacted for the household survey of employment, they would ask him if he had a job. He would say no. Then, they would ask if he had looked for a job in the last six weeks. He could truthfully say yes. He would be classified as in the labor force but unemployed.
Perhaps they should add another question to the survey: "Did you wear your cap backwards when you applied for a job?" If so, he would be classified as not in the labor force, and the number of unemployed would go down by one.
There's occasional talk about the flawedness of the reported unemployment rate as a measure of unemployment, as it doesn't count the underemployed or the "discouraged" — or, as McTeer notes, the unemployable. These are legitimate complaints to the extent that those factors don't mirror the unemployment rate; if we assumed that, for example, the discouraged worker rate were deterministic conditional on the unemployment rate, the unemployment rate would still provide as much information on the state of the labor market as the unemployment rate and the discouraged worker rate. I don't think we live in that world, but any complaint that the economy is worse than we think because the unemployment rate doesn't capture the underemployed should be paired with an argument that underemployment is unusually bad given our 9.4% unemployment rate. Unless the point is just that a 9.4% unemployment rate doesn't mean that fewer than one American in ten has felt the recession, which I would hope would be obvious anyway.
As for those who wear their caps backward at job fairs, they will always be with us. The effect that McTeer noted artificially raises today's numbers, but it artificially raised the numbers in 2004, presumably by about the same amount — perhaps a bit less, as the marginally employable are more employable in a tight labor market. But I don't think this effect has an impact on the meaning of a 9.4% unemployment rate.
Labels: statistics
