QA | 3 May 2012
Data mining and its use in business, or how to stop those annoying phone calls.
A few years ago someone asked me during a Business Intelligence course, "Yes, but does anyone actually use data mining?" Today the answer has to be an emphatic yes. It scares me sometimes just how accurately Facebook targets me with its ads, apart from the occasional hiccup. (Why on Earth did it think I would be interested in a chicken sexing course? What sort of person other than a farmer would say yes to that?) Tesco knew I had a couple of vegetarians in my family by tracking my shopping habits. Amazon is always willing to give me a hint as to what I want for my birthday. And when institutions start buying and selling data, it becomes even more attractive.
Suppose you had a new product you wanted to sell to your existing customers. You predict ten percent of your customers will buy it, but you have ten thousand customers. That's an awful lot of time and money, and ninety percent of it you know will be wasted. You could try phoning a random ten percent, but then you're only going to hit ten percent of your potential sales. But if you have a few years of data, and if that data is comprehensive enough, you could increase your chances. How many ninety-year-olds are going to buy a motorbike? How many fishermen are going to donate to PETA?
This is the heart of data mining. With a product like SQL Server Analysis Services you can launch algorithms into your data to hunt down patterns that are not obvious. Does a combination of your post code, the average rainfall, your age and whether you buy hair gel affect which country you are most likely to holiday in? And if so, what weight does each factor have?
Ideally, we would want data mining to identify exactly which ten percent of your base is going to buy your new product, but data mining will never be that accurate. What it can do, though, is increase the odds. It may be much better to target thirty percent of your base and get an eighty percent hit rate, than have to trawl through all of them for a ten percent hit. And it's not just sales. Power companies predict power consumption based on weather, sport and what's on TV (especially the ad breaks, when we all rush for the electric kettle). Police forces are beginning to use it to target patrols. Banks use clustering algorithms to identify the location of fraudsters.
Because without data mining it's just numbers, huge arrays of numbers we can't hope to master. It doesn't replace the gut feel of a twenty-year veteran, but it does add credence to it. It's a tool that can aid business, and turns a database from just a convenient way to file data into a useful source of meaningful information.
So why am I having these thoughts? Today someone phoned me on my mobile. This was a surprise, as most people don't realise I have one, let alone know the number - I don't even know the number. Because I train in the classroom so much, it is rarely switched on, and then mainly for outgoing calls. "You will be delighted, Mr. Simms, that we have looked at your credit details and we can help you claim up to five thousand pounds for the mis-sold PPI on your credit card," said a pleasant-sounding man.
Here was a company that clearly had done no Business Intelligence targeting at all. Otherwise they would know I wasn't mis-sold PPI, that I listen to current affairs programs that, for example, advise against using third parties to claim your PPI back, and that I never, ever buy products from cold-call phone calls. They compounded their error by preceding to ask me a series of questions. Did I have a mortgage? How long had I had my credit card? Was I married?
So, when they said they'd looked at my credit details, they hadn't looked that closely. I'd had enough.
"You've asked me a lot of personal questions. Can I ask you one?"
"What are you wearing?"
"… Excuse me?"
"Only, if you're going to ask me such personal questions on our first meeting, I think we should go all the way. Wait one, I'm going hands free."