by Jessie Rudd, PBT Group, January 2012

“More information will be created in the year 2009 than in all recorded history up until that year” (sourced at www.jsgilbert.com).Anyone with an opinion, a java enabled cell phone or a computer can and will air that opinion on any one of a million or more social media sites. Access is relatively cheap and easy. There are no qualifications required and things like literacy, intelligence, objectivity have no bearing on what is said by the masses. A thought, idea, joke, whether benign or malignant, can go viral in a matter of minutes. Careers have been made. Reputations ruined. Corruption has been exposed. Governments have fallen. Small town local businesses suddenly cater to an international audience. Companies have been born and others destroyed.In an age of information overload, the consumer has truly become king on an unprecedented scale. They are taking to the internet to air their satisfaction or dissatisfaction with everything from government to product to service delivery – often in overwhelmingly huge numbers.Any company concerned with ‘brand opinion’ simply cannot afford to ignore this font of endlessly changing and churning information.

To tame the beast.Sentiment analysis, or opinion mining, has been around for a while. It is the application of natural language processing, computational linguistics and text analytics to attempt to identify and extract subjective information in unstructured, freeform text (sourced from Wikipedia).The goal is to determine the attitude, opinion, emotional state, or intended emotional communication of a speaker or writer.As is the nature of the internet, varyingly complex and somewhat generic sentiment analysis tools are just a click away. As long as it is being talked about somewhere on the net, these models can and will determine the current snapshot opinion for anything that the mind can conceive of.  From Facebook and Twitter (www.twittersentiment.appspot.com) specific sentiment tools to more complex generic tools that analyse the whole web sphere (www.socialmention.com), to works of art which crawl the web for appropriate sentiment and display them in LCD format (www.earstudio.com/2010/09/29/listening-post), to using complex sentiment models to determine where earthquakes have occurred (www.arnetminer.org/viewpub.do?pid=2820007) so that subscribers can be warned, to sites that will give a snapshot of how the world feels (www.wefeelfine.org/wefeelfine_pc.html) - the list is almost endless.While most of these tools are rather simple reflections of the true meaning of genuine sentiment analysis, they do give just a glimpse of what might be possible. They give an idea of what it might be like to be able to harness that kind of potential consumer awareness and brand power for a business. Imagine being able to measure the impact of a marketing model in close to real time. Or being able to address customer satisfaction the moment it is verbalized. Flame detection (bad rants), new product perception, brand perception and reputation management – these are the bread and butter of any good sentiment analysis model.Design is everything.Accurate analysis is a complicated and multi-step process. The information (nuts and bolts) we have an interest in measuring needs to be identified and then further broken down. Yes we are interested in sentiment, but the sentiment about what is truly what is important to determine. Is it a product, a service, a brand etc. This needs to be high level but also very granular. Not just general brand information but also the nitty gritty of the service or product on offer.

This is generally detected using a monitoring solution that recognizes names, terms, and concepts and then uses natural language processing to associate sentiment and other attributes to the features. It is important to not only look for polarity (positive / negative / neutral) of sentiment but also the emotional footprint (happy / angry / sad). These relatively simple basics need to be fully optimised before more complex vernacular like irony and sarcasm is attempted. Even humans have a less than stellar accuracy rate for something so complex. And all of this is only relevant to the question being asked or the brand in question.It is widely recognised that the most accurate sentiment analysis method so far relies on a hybrid system of machine learning and human input. For each customer the model must be taught to look for the answers to the specific question they were asked, or to look at the specifics of a particular brand or to crawl the net for key phrases. Rather than just a simple positive and negative polarity around a given set of keywords, they must be built to identify relevant comments and then classify them according to why they feel positive or negative towards something. What are they unhappy about? Why do they prefer the competitor? The model is ‘taught’ to identify slang, sarcasm, double negatives, irony, shorthand, punctuation.This is the approach to sentiment analysis that makes the most sense to me, because it means that the longer the model runs / the more it is used / the larger the volume – the more accurate it should become. It is also tailor made for the client, which means the results are 100% relevant and the questions being asked, and answered, are not generic.And bear in mind that this is a field still in the infancy of its evolution, yet the accuracy of a well designed sentiment model is still widely to be estimated to be between 70% and 80%, with the law of diminishing returns setting in.  What is truly impressive though, is that the accuracy is comparable to human accuracy.The playing field.In my opinion, the art of sentiment analysis can be divided into roughly two sub divisions. Any analysis of unstructured data available from an unstructured source (the internet for example) will, by its very nature, be out of context. It will inherently lose the ability to confidently track customer perception because the model is incapable of knowing what the question being asked or answered is and trying to find out what the ‘trigger’ of a post may be is akin to trying to find a needle in a haystack.

Then there is the analysis of unstructured data that comes from a structured source – free form text that answers an actual question being asked and is typically the text found in questionnaires, the complaints section of a website, surveys that may have been done.
While this latter method is probably the more reliable of the two purely because of context, the former is also necessary to gauge the ‘heartbeat’ perception of a brand / product. Working with such vastly disparate volumes of data means that the model needs to be incredibly robust and well trained, with as much human input as possible.The shortfall and the gain.Models can be modified and adapted to analyse unstructured data from unstructured sources with only minimal delay – and in doing so, the companies have an unprecedented opportunity to institute steps to alter perception almost immediately.Other versions of the same model, those that deal with the analysis of data from a structured source, can be taught to accurately and perceptively glean the true meaning of freeform text. Possibly for the first time, what the customer is actually saying, in their own words, is readily available to the companies that venture into text mining or sentiment analysis.However, it must be remembered that above all, this is a branch of analytics that is still at the beginning of its journey. Best practice, appropriate tools, true understanding are all still a work in progress.As such, it needs the backing of a company with the knowledge, infrastructure and curiosity of a well deployed and experienced advanced analytics background.Only then will the true potential of the pen versus the sword be realised (The pen is mightier than the sword - Edward Bulwer-Lytton).