Identifying Quackery (and Other Mistakes)

Submitted By Jeff Miller

At "A Dash" one of our principal missions is identifying the best sources of information on a given topic.  This is a major source of value, if done accurately.  In the investment blogosphere, however, the ratings mostly reflect the predispositions of the reading public.

The Path to Success in Blogging

It is easy to figure out what works.  One needs only to look at the top-ranked blogs and those most followed on Seeking Alpha.  There is an aggressive cult following (a subject for another day) for specific perspectives on the economy, the role of government, the existence of conspiracies, and the stock market.  If one's objective is high ratings, this is a blueprint for a strong business model.

The Path to Success in Investing

One of our missions is how to profit from this.  It means reading many perspectives and evaluating arguments and data.  As part of our work (eventually a book) we are analyzing various sources, with an emphasis on factual accuracy, freedom from bias, professionalism, and analytical skill.

Do not start with what you believe.  Start with data.

A Case in Point

We offer the interpretation of various polls as a case in point.  This is partly based upon expertise in developing, conducting, and interpreting poll data.  We have done all of these, and most others have not.  Let us consider some experts in action.

Our first expert is Karl Smith, of the University of North Carolina at Chapel Hill, now added to our list of featured blogs.  He has really stepped up his role in blogging, taking an expert public policy perspective on many issues.  He deserves to be widely read.

Smith takes a current poll and does a careful analysis.  The poll purports to show some terrible answers to some obvious questions.  For example, What is the Supreme Law of the Land?  According to the poll, only 28% of Oklahoma students identified the Constitution and some offered up the Monroe Doctrine.

This might be a true result, in a multiple choice test with attractive alternatives.  We once saw a similar survey of undergrads where most thought that the Electoral College was a small liberal arts school in Minnesota!

But Smith delves deeper into the data, which purported to permit free-form answers.  Look at his entire analysis which is very strong, but here is an example:

Think about this from a student’s perspective. The first question

uses the phrase “Supreme Law.” This means that answer has to be

something important. The natural thing to do is to answer the most

important thing you can think of. From free response I would not be

surprised to get answers like “the bible”, “democracy” or “freedom”. 

The student might also simply free associate with things that are like

Supreme or like Law. Hence, I would not be surprised to get answers

like “Federal government”, “Supreme Court”, “Congress” or even “Roe vs.

Wade.”

Since none of these answers were given, but Monroe doctrine was. I

find this hard to believe unless the poll was multiple choice and the

six answers were the only answers a student could give.

He has accurately identified a false note in the poll.

For our second expert we also take note of the fine work from Nate Silver at fivethrityeight.com, now also added to our list of featured sources.  We have occasionally been accused of credentialism in identifying sources, mostly by those thinking that having a PhD in economics is worthless.  Silver is a fine contrary example.  He developed strong expertise in analyzing baseball statistics.  Anyone with statistical expertise can find many applications.  He has done so in election polling and political analysis, useful to anyone analyzing public policy.  We watch his work carefully.

His take on the dubious polling data is summarized as follows:

For me, some of these results don't pass the smell

test. I agree that public schooling in the United States needs to be

improved, particularly in the areas of government and citizenship. But

only 23 percent of high school students in Oklahoma knew that George

Washington was the first President? Really? I have difficulty accepting that claim at face value. In 2008, 68 percent of Oklahoma fifth

graders passed the Oklahoma Core Curriculum Social Studies Test. You

can read some of the questions on that test beginning on page 50 of

this PDF;

they're generally quite a bit more difficult than the ones that

Strategic Vision asks. (For instance, "Which was the most profitable

export of the Jamestown settlement?" and "Which group would most likely

agree with ideas presented in Common Sense?"). So either those smart fifth graders were really forgetful by the time they got to high school, or there's something very wrong with this poll.

Investment Implications

There is a special skill involved in interpreting data.  Surveys are the source of much of the information that investors and traders try to interpret.  Few are really expert in how the polls are conducted, the statistical significance of results (often not reported by the source), and the reasons and basis for revisions.

It is foolish to identify the source of the poll, and then dismiss the result if one does not like it.

Our example does not have immediate investment implications, and that is intentional.  It is much easier to step away from the economic polling data, looking instead at an obvious error as a case in point.

Then come back to the news of the day.  CNBC featured today a poll with a response rate of 25% of the CFO's attending a convention.  Most of them thought we were "still in a recession."  While we are interested in CFO opinion about current corporate plans, none of them are experts at recession dating.  Mostly they are saying that the economy is still well below trend, something we already know.

There is much more to analyzing polling data than looking at a snapshot of current opinion.  Longitudinal analysis is a starting point, something that most of us do when looking at survey series like the ISM data.





Did you like this article?

Related Videos

World Business: On a Spree 24/04/09

World Business: A Helping Hand 24/04/09