1 Good Reason – Social Marketing

We don't make websites, we make websites better.

1 Good Reason – Social Marketing header image 2

A Flawed Analysis of Twitter

January 20th, 2010 · 3 Comments

BLUF: (Bottom Line Up Front) Today I review the Hubspot report “The State of the Twittersphere”. In my analysis the Hubspot report is of questionable value and cannot be used to extrapolate from the sample used to the general Twitter population.  I do not recommend that my clients use this report to draw any conclusions about Twitter.

Hubspot released their January 2010 State of the Twittersphere report today, a free report which if you spend much time on Twitter should be a good read.  So I decided to dive into the report and see what I could learn.  I would recommend that you take the data in this report with a grain of salt.  The reason I say that is that the data is based only upon people who choose to have their Twitter accounts analyzed by Twitter Grader. I feel strongly that we can’t make generalizations on Twitter as a whole based upon this data.

Twitter growth has plateaued, I think this reflects the trend of Twitter in the direction of an influencers and communicators platform as I described in my predictions 2010 post.  The report has no analysis of the reason for this trend.

The big growth this year according to the Hubspot report is in average Following and Followers, and number of posts.  All of these metrics are growing, meaning that people are following others more, and being followed by others more often.  Now I would attribute this to the fact that more people are Getting Twitter and understanding that it’s much more useful when you have a larger group of people to interact with.  Hubspot attributes this shift in numbers to the decline in new users, in other words a maturing user base.  I would attribute this to a failure of proper analysis of this data.

I’m not a statistician, but I’m confused with these numbers:  The report states that the Average Following went from approximately 50 in July 2009 to about 170 in January 2010 (Page 3 of 10)  and that Average Followers went from about 70 July 2009 to about 300 in January 2010 (Page 4 of 10).  On page 6 of 10 they state that “82% of Twitter users have less than 100 followers” and “81% of Twitter users are following less than 100 people” If both of these are true, then the typical user has not significantly increased their counts in either category much at all.  Other reports show very different numbers: Sysmos 6/2009 “93.6% of users have less than 100 followers, while 92.4% follow less than 100 people”  Again, the report states that the average Twitter user today has 300 followers and is only following 170.  This is wildly inaccurate, while it may be statistically accurate the data is misleading.  The true average Twitter member has less 100 followers and is following more people

The largest single factor that will effect the results is the sample of Twitterers that HubSpot uses.  Because the sample is based upon people who have chosen to use the Twitter Grader service they are automatically more interested in Twitter than the normal population of people on Twitter.  This means that it is very likely they are more engaged, have more contacts and are overall more involved with Twitter than the general population.  Secondly, I think that the statistics are flawed because Hubspot apparently didn’t remove unusual accounts such as those that have over 100,000 followers, or that send many tweets per day. These accounts will seriously skew the results.

I’m sure that the numbers are exactly as Hubspot has reported them.  However, I think a statistician would have analyzed the numbers differently which could lead to very different interpretations.  I feel that considering the data as presented from the statistical methods and due to the sample source that this report is of very limited use.

Overall I’m an advocate for Hubspot.  They have many excellent products in their “Grader” line and offer excellent value to their clients.  I’ve used their products for years and will continue to do so.  However this time I think they missed the mark.  I had a telephone conversation with Rick Burnes the Author of this report.  I explained my issues with the report and Rick agreed that the data could have presented in other ways that may be more easily understood.  Rick however mentioned that the data does indicate overall trends in Twitter usage.  I still feel that it’s not reasonable to extrapolate the data from this source to all of Twitter.

In conclusion, I recommend that my clients and fellow bloggers not use this report as a guidepost for understanding Twitter user behavior.

Leave a comment with your 1 good reason why I should take this report seriously.

Posted via email from ckieff’s posterous

Share this:
Share this page via Email Share this page via Stumble Upon Share this page via Digg this Share this page via Facebook Share this page via Twitter

Tags: Reasons For Net Marketing · Twitter Thursday

3 responses so far ↓

  • 1 Mike Volpe - HubSpot // Jan 20, 2010 at 1:01 pm

    Your assumption that the 5 million Twitter accounts that we analyzed are only people that visited Twitter Grader themselves is false.

    The millions of accounts we analyzed do include Twitter users that were not typed in by the user themselves on Twitter Grader. We have an API in use by dozens of companies to grade Twitter users, plus other methods of growing the database, which provides a significant amount of data outside the visitors to Twitter.Grader.com.

    We really do feel that the 5+ million accounts that we have data on are a pretty representative sample of the overall population on Twitter, and is the best dataset available outside of Twitter themselves.

  • 2 Chris Kieff // Jan 20, 2010 at 4:05 pm

    Mike,
    Surprisingly, during my conversation with Rick this morning this fact never came to light. Rick did say several times that it would have been better to present the Averages on page 3 & 4 of the report as Means instead. This could change the results of the report in very significant ways.

    However, it doesn’t change the fact that this information is presented in a way that is confusing and misleading. I believe that you would agree with me that the average Twitter User does not meet the criteria the report specifies as average. Which would cause someone who is not so well versed with the subject to make some false assumptions.

    For that reason I am not going to recommend this report to others.

  • 3 uberVU - social comments // Jan 20, 2010 at 10:36 pm

    Social comments and analytics for this post…

    This post was mentioned on Twitter by ckieff: I think @hubspot “State of Twitterverse” report is flawed and misleading. Read my analysis here: http://bit.ly/5yvfyY