What Do Your Metrics Measure?
Posted on | November 11, 2009 | CLICK HERE TO COMMENT OR ASK QUESTION
First, here are two very important yet often or misunderstood definitions:
- Validity is whether you are actually measuring what you think you are measuring.
- Reliability is whether you are measuring it accurately.
My issue: I wanted to be sure and get the latest HD-DVR (Digital Video Recorder) for my home TV setup. Most of the units presently being installed were a two-year old model, with numerous stories in online forums about poor picture quality. I was told confidently by a Customer Service call center agent that all the local offices had the new units in stock, and I just had to ask for one.
But my visit to the local office was a different story. No new HD-DVR boxes, only the old ones. No idea when they would be available. So I took an old one. After hooking up the older model supplied, the picture was only fair quality. My previous (non-DVR) box had given a stellar picture.
So I called Customer Service again to find out how to get a new box. At the end of the call, my question had not been answered, although the agent had tried hard, and was friendly. In the absence of having the information I needed, she even made up an explanation. But by the end of the call, she could not tell me when or where I could get the newer unit.
It seemed that she had no access to any database or bulletin of information about the availability or performance of one of their key products, and could not tell me when they would be in. For me to obtain this product, I would have to go down to the local office and see if they were in stock. The agent was not able to give me the telephone number for this office, and the best she could do was send me a bulletin about availability. But she could only send this USPS, and these newer units typically fly off the shelves within hours.
During the call, the agent kept reminding me of her name. At the end of the call, she asked if there was anything else she could help me with. It wasn’t her fault, so I refrained from telling her that she had not helped me with anything in the first place. But, wait for this, she said that I would be getting a follow-up call, and that she would greatly appreciate if I could rate her service as a “10”. The first agent had also asked this.
Applying the definitions
On the question of validity, what would I be rating – friendliness (9), patience (9), effort (8), knowledgeability (2), initiative (3), solving my issue (0). Is my rating of her, or is it of the overall experience and resolution? So whatever the rating, it is hardly valid, since what is being measured is unclear.
On the question of reliability, whatever dimension or factor I was rating, the fact that she asked me (and others) to rate her a 10 must contaminate and skew the ratings. Since other agents from this company have also asked the same thing, this leads me to believe that it is a scripted request. Either that or the agents have tacitly developed a shared approach to get good ratings. And do the agents try hard to give the impression (or temporary illusion) to the customer that they have resolved the issue, so that the follow-up call will yield a “10” rating? This makes the aggregated number highly unreliable as reported metric.
And if this customer service call center was an outsourced service provider, then the client was not gathering very good metrics against their SLAs. This is where I thought, “Hmm. If these agents work for a call center vendor, and the average ratings are what the vendor is being measured on against the SLAs, then this is hardly a valid or reliable metric.”
And the opportunity is lost to get an accurate picture of the customer experience, and improve processes (in this case a better knowledge base and bulletin system for agents). This is equally applicable whether the “customers” are consumers, or if the service is IT support for internal corporate employees.
So my point from this story is, if you are managing a vendor that interfaces with your customer, then make sure you know what is happening at the customer interface, and how the metrics are measured and gathered. Look at validity and reliability, how the metrics are measured, and if there are any factors and behaviors that contaminate the results.
Tags: agent > assumptions > Call center > clarifying > customer service > Metrics > reliability > resolving issues > Service Levels > validity
Comments
Leave a Reply