Discovering ‘diamonds’ in the rough of unstructured data
Jessie Rudd, Data Analyst at PBT Group
Approximately 80% of data created today is unstructured. This includes everything from business
documents and audio files to text messages and video clips. Considering the insights currently gleaned
from the remaining 20% (the structured data found in databases), then just imagine how much more
valuable information is waiting to be found.
Much of this centres on the customer experience. So, whether it is a phone call to a contact centre,
WhatsApp messages or voice notes, or posts made on Facebook, a significant portion of unstructured data
is related to the customer and their views of a brand. In fact, it has become part of human nature to complain
as quickly as possible about a negative experience.
Unhappiness
For instance, a customer might be unhappy with a purchase made at a popular grocery store, phone the
contact centre to complain, and then feel the agent is being rude. There is nothing stopping them from
posting about this on all their social media profiles, telling their friends, and/or leaving as many bad reviews
as possible about the particular store. For the store, having the ability to stop this unhappiness as early in
the chain as possible becomes invaluable. If they can pick up the unhappy conversation and then send the
customer an SMS apologising for it with a R100 voucher for example, the potential for changing the negative
narrative becomes significant.
It comes down to an organisation being able to mine unstructured data quickly and discover those diamonds
as soon as possible before they turn into coal.
Product push
This is also not limited to a singular customer experience. Companies can gleam the underlying sentiment
of a particular product or brand. For example, they might pick up a blip in the data. By zooming into that,
the company can see that the bad sentiment revolves around a recently flighted television commercial.
Market intelligence is another benefit of mining for these proverbial diamonds. Brands can find patterns in
customer behaviour and the types of products that interest them at an individual store level. By analysing
its rewards programme (if offered), a store might pick up that a sizeable portion of its customers in a
particular area likes various coffees. It can then market custom-designed specials to those people.
Type: PBT Group website blog
Analysis challenges
Of course, unstructured data is hard to analyse. No organisation can expect a person to trawl through the
data and find the insights required fast enough. But while automation, machine learning (ML), and artificial
intelligence (AI) are great tools to help in this regard, they will never completely replace experienced
specialists. Certain individuals can find patterns in data that AI cannot replicate. They almost seem to have
this magical ability to find logic where their brains tell them there is a pattern, but it is not quite visible.
Currently, this ‘fufifoo’ logic is impossible for AI to replicate.
Helping these experts examine unstructured data is natural language processing (NLP) tools that can
analyse the wording and structure around where they are trying to get information from based on pattern
recognition. They also need specific ML tools that they can train in their field. Over time, the ML tool
manages to recognise things specific to that industry. And then you have audio files and voice messages
that also need to be converted using speech to text before analysis can even start.
Furthermore, new technology is coming out called intelligent document processing. This is a tool that can
classify types of unstructured data and store them in the right category to make it easier to analyse. But
regardless of what technologies are already available or being released, companies must not neglect the
potential diamonds waiting to be discovered in the unstructured data out there.