Most social media monitoring companies offer automated sentiment detection services. Automation saves time and money: so it’s instantly appealing. The customer can log in and simply watch graphs and tables appear, showing them data about their brand in a user-friendly desktop environment. What could be better? Such services also have the benefit of being able to capture, assess and display data in near real-time speed, which is really compelling for companies running marketing campaigns or events. You can just sit back and watch the results coming it – saving your energy for posting timely replies and responses to negative tweets, posts and comments.

These automated services go way beyond the simple “smiley” test of many Twitter apps. They have constructed complex algorithms that can identify different elements of sentences and assess the sentiment of and connections between these words or phrases. Scoutlabs calls this “speech tagging” and describes it as “parsing the underlying semantic structure of a sentence and determining which emotion words apply to the key word”. Clever stuff, no doubt.

That said, there are several obvious downsides to automated sentiment detection:

• Irony/sarcasm – It’s currently impossible to accurately assess posts that include ironic or sarcastic comments.
• Slang – While some services incorporate dictionary slang, they (like most humans) cannot keep up with developments in street slang.
• Languages – Many services only operate in English, ignoring the other 6999 languages (yes – that’s how many there are) spoken in the world.
• Geographical variations – Different countries and regions us different expressions and slang, even within the same language.
• Context – Negative and positive are not the same for everyone and brands may consider it negative to be associated with certain terms or people. “Mugabe Gets an Iphone”, for example, may raise eyebrows in the Apple marketing team.

As a result of these issues, automated sentiment detection services can only offer an accuracy rate of 75%-80%. The claim is that this compares well with human intervention ratings, which apparently show a difference of opinion 20% of the time. But I find that hard to believe. Given strict guidelines I think most intelligent adults would rate posts accurately as positive, neutral or negative 99% of the time. Wouldn’t they? What kind of an idiot would misread the meaning of a Twitter post??

Well, actually, to answer my own question – if the value (or damage) of a comment is gauged by its effect on people reading it, we have to assume readers are not particularly smart or well-versed in the nuances of irony or slang. Your average twitterer is probably quite likely to misunderstand someone else’s post and act accordingly. If for example you saw my post “Just signed up with TalkTalk. Customer services as good as ever!” I’m not sure everyone would pick up on the truly scathing and malign undertones of my Tweet.

Automated sentiment detection services generally offer an option to amend sentiment ratings, and therefore enhance the quality of the data, manually – and, in my view, this is one of the most beneficial activities anyone running a social media monitoring campaign can engage in. Reviewing the top level graphs and charts can be hugely gratifying, but it’s at street-level, by reading and responding to individual posts, that you get a true picture of what people really think.

I’ll be posting thoughts about  non-automated sentiment detection shortly, but in the meantime I’d welcome additional comments and suggestions on this topic. We’ll be discussing the topic further at our upcoming social media monitoring event: Monitoring Social Media 09, in London on 17th Nov.

BrandWatch are serious data-heads. Having started out building monitoring systems for the British Government, they struck out on their own, creating a high quality social media monitoring and tracking system of their own. While some services (notably Market Sentinel) employ human intervention to measure “sentiment” in the posts people make on Twitter, forums and blogs, BrandWatch concurs with Scoutlabs, among others, that automation is the way forward when dealing with large quantities of data (as Seb points out, you get real-time trends, regardless of thhe inevitable inaccuracies).

BrandWatch works by enabling companies to set up a range of “classifiers”. These might include “industry”, “country”, “sector” etc. within which data should be tracked. They can then set the “keywords” they want to track within these boundaries – and the system does the rest. Once the data has emerged, the user can slice, dice and present it in a wealth of useful, fun and, if I’m honest, mind-boggling ways.

A particularly nice feature is “Groups” that enables companies to track their keywords across a set list of websites. So Mothercare, for example, might discover that they get more comments about their prams on Netmums, while their baby clothes stoke up more interest in the Confetti.com forums. Similarly, users can check which keywords appear most often in comments and which are increasing in frequency over time – i.e. what the hot topics are. These kind of stats and flows can have a huge bearing on advertising spend.

BrandWatch also measures the Influence of the people making the comments. This is done using a straightforward calculation of the “most mentions for a particular keyword” plus “credibility” – which is gauged by site traffic, in-links, page-rank and the age of the site. Evidently there’s a hole here – Twitter followers for example – but Seb assured me that will be filled in due course.

For anyone not familiar with tracking social media, BrandWatch, like many other services, offers hours of fascination: the peaks of activity on a Monday; the troughs at the weekend; the emotive spikes generated by “new” products (about which people are much more opinionated that old ones); the wild differences in sentiment detected between a brand and it’s latest product or marketing campaign; the amusing acceptance that, no matter how clever they get, computers will never understand irony.

The new version, BenchMark. looks to be a Netvibes-inspired mixture of drag and drop usability with some juicy additional features thrown in. In addition to the inclusion of more video data (i.e. stats and ratings), it will include “proximity” targeting of keywords, i.e. the ability to limit the terms covered according to their proximity to other words. I look forward to trying this out when it’s launched in a few weeks’ time.