• Print

Open question: Do you trust market research surveys?

A deluge of recent ebook and ereader research raises a host of methodology questions.

open questionAs recently as last year, actual information about ebook consumers was nearly impossible to come by. But lately, throw a virtual rock in any direction and you’re likely to hit an ereading or ebook research report. Just yesterday Forrester released their latest such report, including very reportable and re-tweetable highlights such as:

  • 2010 will end with $966 million in ebooks sold to consumers.
  • By 2015, the industry will have nearly tripled to almost $3 billion, a point at which the industry will be forever altered.
  • Current ebook readers in the survey expect an average of 51 percent of the books they read will be ebooks
  • Four in 10 people who own or expect to buy an ereader shop at Amazon for physical books.
  • Exactly 50 percent of people who bought an ebook in the past month have bought ebooks from Amazon’s Kindle store.

On the same day, French management consultants Bain + Company reported that by 2015 between 15-20 percent of the book reading public will own electronic devices, and up to 25 percent of books will be sold in digital form. This nice sound bite was included with their press release: “Experimenting with new formats — non-linear, hybrid, interactive or social — is where opportunity lies.”

These are just the tips of the e-iceberg. The data and research are coming at an unprecedented pace from all manner of organizations, associations, private firms, and individuals.

Studies are great. Data is awesome. The ebook market has gone without much of either for far too long. But, could it be that in our thirst for consumer ereading information, we’re drinking down all this newly available data without stopping to check exactly what’s in it? The temptation with market research is to read the cherry-picked bullet points and stop there. We need to look closer. Sometimes the small print says a whole lot more than the sound bites. It’s important to understand the methodology behind the results, including factors such as:

  • How well-defined is the target population?
  • Is the sample being studied random?
  • How did the research group chose its random sample?
  • Was the sample size large enough to produce meaningful results?
  • Are the questions unbiased?

We’ve banged lightly on this drum before, but we thought it’d be nice, given so many new study summaries being released of late, to open up the topic for discussion.

So here goes:

Do you trust market research surveys? What would you need to know in order to make a key business decision based on survey results?

Feel free to chime in through the comments area. You’re also welcome to join us on Thursday during our first TOC LinkedIn Open House discussion, hosted by BookSwim.com’s Jeevan Padiyar. If you’re not already a member of TOC’s LinkedIn Group, it’s easy to sign up.


Also of interest: “eReading Survey Findings and Research: A Look Behind the Numbers,” a TOC 2011 panel to be moderated by Sarah Weinman.


tags: , , , ,
  • http://cloudculturecontent.blogspot.com Adam

    I never trust them, I think their methodology is bogus.

    A question that I would ask before any of the ones you list here is: “what is the research firm’s track record?”

  • http://keithsuckling.wordpress.com/ Keith Suckling

    Trust is something that is earned.

    Some studies have earned their trust through reliability and sound methodology over a period of time. Others are “fly by night” and deserve to be considered in this in mind.

    Looking at the findings alone and thinking you can tell anything about the soundness of the methodology that created them is like reading the intentionally attention grabbing headline and thinking you know everything about the story.

  • http://gadgeteer.org Allen Smith

    These studies in rapidly developing areas are always the same:

    will grow at a compound annual growth rate of 18% to 22% over the next 5 years! Buy our $5000 report to find out how to cash in!

    Studies in more mature industries are more useful. In order to avoid charges of collusion all the incumbents in a field tell all their secrets to one of these third parties who then writes them up and sells them back to the industry players for $5000 a copy.

  • http://threebrothers.org/brendan/ Brendan Ribera

    No, I don’t trust them.

    I’ve participate in several tech-related market research studies. Anecdotally, it seems that most of them suffer from selection bias, whether intentionally (filtering out respondents on characteristics known to alter the responses) or unintentionally (you’re only getting responses from a self-selected group: people who have volunteered to receive survey requests online).

  • Giovanni

    Personally I agree with Steve Jobs

    “We do no market research. We don’t hire consultants. The only consultants I’ve ever hired in my 10 years is one firm to analyze Gateway’s retail strategy so I would not make some of the same mistakes they made [when launching Apple’s retail stores]. But we never hire consultants, per se. We just want to make great products.”

  • http://www.xomba.com/nashville_recruiters_nashville_recruiter_recruiters_nashville_tn LPate

    I can’t say I trust them either. Seems like most of the survey takers are either being paid something to do the survey or is trying to win something. They might just say anything to get through the survey.

  • http://booksquare.com Kassia Krozser

    I think you’re right to question methodology, but I do (for better or worst) trust surveys. Sort of. One survey tells me nothing, but the aggregate of recent surveys gives me a sense of trends. Ebook sales are increasing. Ebook adoption is increasing, far faster than last year’s surveys suggested it would. This falls into line with data that suggests ebook sales are just about at the 10% mark. Is now the time to mention that I trust sales data more than surveys?

    The Forrester study was the first, I believe, that factored reading on desktops/laptops into the “device” mix. That made me very happy.

  • http://www.bookswim.com Jeevan Padiyar

    Mike Shatzkin answered the question “What do I need to know in order to make key business decisions based on survey results,” reasonably well in his latest post
    (http://www.idealog.com/blog/dont-drown-walking-across-the-river) :

    Essentially don’t trust one source for your market decisions. Look at studies from multiple organizations before jumping to ANY conclusions.

    To that I’d add:

    Talk to your customers (after all they are the ones who matter in the end),
    Talk to smart people in the field (and not just the folks in the echo chamber)
    Talk to people who know nothing about the field, they may be far enough removed to help you gain clarity.
    Talk to your staff (and not just the higher echelons of management – since the people in the trenches generally have a good grasp of what is really happening on a day to day basis)

    Then fall out of love with your business decision. Try to find points of failure.Try to learn if you are really answering a market need. Make sure you can accomplish you result in a reasonable amount of time and within your budget constraints.

    Lastly after you launch, test, iterate and repeat. (ie – fail fast and often)

    But I digress…

  • http://www.bookswim.com Jeevan Padiyar

    Mike Shatzkin answered the question “What do I need to know in order to make key business decisions based on survey results,” reasonably well in his latest post
    (http://www.idealog.com/blog/dont-drown-walking-across-the-river) :

    Essentially don’t trust one source for your market decisions. Look at studies from multiple organizations before jumping to ANY conclusions.

    To that I’d add:

    Talk to your customers (after all they are the ones who matter in the end),
    Talk to smart people in the field (and not just the folks in the echo chamber)
    Talk to people who know nothing about the field, they may be far enough removed to help you gain clarity.
    Talk to your staff (and not just the higher echelons of management – since the people in the trenches generally have a good grasp of what is really happening on a day to day basis)

    Then fall out of love with your business decision. Try to find points of failure.Try to learn if you are really answering a market need. Make sure you can accomplish you result in a reasonable amount of time and within your budget constraints.

    Lastly after you launch, test, iterate and repeat. (ie – fail fast and often)

    But I digress…