The Evolution of IRS Telephone Quality Measures

The Evolution of IRS Telephone Quality Measures

Laura Rosage, Statistics of Income, Internal Revenue Service

P.O. Box 2608, Washington D.C. 20013

Key Words:

Accuracy

Quality, Defects per opportunity,

The Internal Revenue Service (IRS), tasked with

collecting taxes from this country’s citizens, deals

with more Americans than any other public

institution. Unfortunately, over the years, the tax law

has increased in complexity and the myriad of forms

has become confusing. In an effort to assist

taxpayers comply with the law, the IRS established a

toll-free telephone service with Customer Service

Representatives (CSRs) ready to help taxpayers with

their tax related questions. Since 1965, the IRS has

offered this free telephone assistance to millions of

taxpayers. IRS assisters handled over 55 million

telephone calls in fiscal year 2003 (October 2002

through September 2003) and 15.8 million calls

during filing season 2004 (January through mid-April

2004) alone. With this large volume of inquiries

handled by telephone assisters, the accuracy of the

information provided has a potentially large impact

and is of interest to stakeholders both within and

outside of IRS. The way the accuracy of telephone

assistance is measured has evolved over the years

from test calls, to live monitoring of telephone calls,

and soon, to contact recording. This paper details the

evolution of how the IRS monitors calls and the latest

move from a pass/fail method of measurement to a

defects-per-opportunity methodology. It discusses

the strengths and limitations of each method, the

overall impact on quality rates, and future plans for

improvement to the measurement process.

Monitoring Taxpayer Calls for Quality

Purposes

One of the IRS’s major goals is to make its telephone

operations

a

world-class customer service

organization. To do this, it is necessary to track the

accuracy, efficiency, and quality of the service

provided by those answering the telephones.

However, when first implemented, the focus of

telephone service review was only to evaluate

employee performance. Managers reviewed the work

of CSRs and used the data gathered as feedback for

employees’ performance appraisals and to identify

training needs. It wasn’t until the late 1980s that

IRS attempted to measure the overall quality of the

service provided to taxpayers.

Managerial Review

IRS’s first iteration of quality measurement consisted

of managers reviewing their employees at each call

site. The manager would sit with an individual CSR

and listen in on selected telephone calls. When the

call was complete, the manager was able to provide

immediate feedback to the CSR on any errors made

or on issues of timeliness or professionalism. While

better than no review at all, there were some

drawbacks to this system. First, the data gathered

during telephone call monitoring were really intended

for CSR performance review rather than a site or

national measurement of quality. Also, because

managers monitored their own employees, complete

impartiality of the reviews could not be guaranteed.

Additionally, since managers sat with the CSRs, the

assisters were aware they were being monitored.

They could alter their behavior during calls that were

reviewed:

Responding in a more professional

manner, researching the tax issue more thoroughly, or

adhering to their manual guidelines more fully.

These changes could affect the quality of the call,

giving IRS a skewed view of the performance of the

employee, as well as the overall quality of the service

provided to taxpayers. Finally, because review of the

CSRs was performed at each of the call sites, there

were issues with the lack of consistency of reviews

from manager-to-manager and site-to-site.

Attempting to get a clearer picture of the actual

service taxpayers received, IRS implemented

technology that allowed managers to review CSRs

remotely. Without alerting the CSR, they could

listen in on and review telephone calls from the

privacy of their office, rather than sitting beside the

assister as he or she was on the phone. This

transparency eliminated the problem of the CSRs

knowing they were being monitored and modifying

their behavior accordingly, but the issues of

inconsistency of reviews from site-to-site still

existed. Additionally, the accuracy results from this

process were generally very high and at odds with the

Government Accounting Office’s (GAO) assessment

of quality, further supporting the concern that the

monitoring carried out at a local level was not

impartial.

Integrated Test Call Survey System

In an effort to eliminate these concerns, the Service

implemented a new program in addition to the

managerial review, the Integrated Test Call Survey

System (ITCSS). This system, established in 1988,

was designed to produce a national estimate of

quality rather than relying on managerial review of

employees to establish the measurement and to

provide timely feedback to call sites. The sites could

then use the feedback to target specific areas for

improvement, then assess the success of their efforts.

Under ITCSS, a centralized group of quality

reviewers called into the toll-free IRS tax law

assistance line, posed mock questions to CSRs, and

rated the quality of the responses given. The creation

of this centralized review process, where independent

reviewers received identical training and held regular

meetings on how to rate calls consistently, reduced

the inconsistency and impartiality of rating the

quality of service provided to taxpayers at the local

level. Of course, this method of measurement also

introduced other issues. Although the universe of test

calls was modeled closely after the volume and topic

of taxpayer inquiries, this national sample was not a

sample of the universe of actual taxpayer calls, but a

review of responses to fabricated questions, posed by

persons other than real taxpayers. Any data gathered

from this test was an artificial measurement of the

accuracy of information IRS assisters provided to the

public. Additionally, ITCSS measured tax law calls

only. For most other types of calls coming in on the

IRS toll-free lines, it is necessary to know the identity

of the caller and access their tax records to

completely and accurately respond to their inquiry.

This would not be possible with test callers. Also,

after a time, even though the test questions were

changed periodically, the CSRs were often able to

identify calls from quality reviewers.

Centralized Quality Review Site

Because of the limitations of ITCSS, the IRS

eventually moved away from the test call system and

created the Centralized Quality Review Site (CQRS)

in Philadelphia.

This site was established to

centralize the IRS telephone review process into one

location; to sample real, live calls from the universe

of actual taxpayer inquiries; and to establish an

estimate of the true level of service being provided to

taxpayers.

They were also charged with

standardizing the review process of telephone calls

and centralizing IRS telephone quality review data

into one database.

The site, established in 1997, initially began with

eight reviewers measuring the quality of tax law calls

only. Over time, further types of calls were added.

The CQRS now has over 50 full-time reviewers who

monitor tax law calls, taxpayer account-related calls,

collections calls, calls from the tax-practitioner

priority line, calls from U.S. taxpayers overseas, calls

from employers seeking business taxpayer

identification numbers, and all Spanish-language

calls, as well as requests for IRS tax forms.

The CQRS was able to establish an impartial

measurement of quality for each call site and type of

call by utilizing technology that enabled them to

remotely monitor live taxpayer telephone calls

coming into any IRS call site across the country.

They reduced inconsistencies in the review process

through holding regular consistency training, as well

as utilizing a standard data collection instrument that

gathered the same data elements for all calls and

stored the information in a central database. And

because they were monitoring real, live calls, the

quality measurement generated from the review data

produced the clearest picture of the level of service

provided to taxpayers since IRS implemented quality

review.

This standardization of the review process and

improvement in the consistency of reviews was a

major step toward accurately measuring the quality of

service the IRS provided to callers. Also, with

remote monitoring, neither the caller nor the CSR

was aware if their particular call was selected for

review. This transparency meant that the monitored

response was real, typical of the type of

taxpayer/CSR interaction, and not altered in any way.

Despite these advances, many new issues were

introduced as IRS moved to monitoring live taxpayer

telephone calls. Because the telephone calls were

live, reviewers were required to monitor the calls

real-time. While this sounds innocent enough, realtime monitoring had a tremendous impact on

reviewer resources.

Monitoring taxpayer telephone calls for quality in

real-time consumes a considerable amount of

resources. Initially, the largest depletion was due to

dead air. A reviewer would dial into a site to monitor

calls. If there weren’t any available calls at that site

(no taxpayers calling in or an unscheduled site

closing), the reviewer wouldn’t know until listening

to several minutes of silence or dead air. With the

acquisition of software called Custom View, which

allows the reviewers to see call traffic in the sites

(real-time, less a 6 second delay), this problem was

virtually eliminated. However, that was not the only

problem with real-time monitoring.

To select a call for review, the quality reviewer at

CQRS dials into a designated telephone number for a

given site and type of call and is then attached to the

next incoming call. The reviewer stays with the call,

as long as it is in that particular site, able to hear the

complete CSR/taxpayer interaction. Because it is a

live telephone call, the reviewer experiences what the

taxpayer experiences, including time on hold or

waiting for the next available assister. Any hold or

wait time is wasted time for a reviewer and can

dramatically impact the number of telephone calls

that he or she can monitor. Unfortunately, there is no

way to eliminate these phenomena when monitoring

live telephone calls.

Additionally, in order to sample from the entire

universe of calls when monitoring in real-time, the

CQRS must have reviewers scheduled during all

times of day that the IRS call centers are open.

Unfortunately, there are not enough review resources

to cover all hours of operation, which are 7 a.m. - 2

a.m. Eastern Time, meaning some calls are not

subject to quality review. The CQRS does have staff

monitoring phone lines from 7 a.m.- 12 a.m., so only

those calls received during the very early morning

hours of 12 a.m.- 2 a.m., less than 3 percent of the

total universe of taxpayer calls, are not subjected to

sampling for quality review.

Another minor issue associated with real-time

monitoring is that the only record of the content of

the call is the reviewer’s notes. If the reviewer is

unable to catch something that is said during the call,

it can never be re-heard or recaptured. A reviewer’s

determination of the accuracy of the call is dependent

upon what he or she is able to hear and jot down

during the call. This can become an issue if a call

site objects to the reviewer’s evaluation of a call.

Formal rebuttals from sites, requesting a reevaluation of monitored calls, are frequently sent to

CQRS for response.

Unfortunately, since the

disputed calls cannot be replayed, reviewer notes are

the only evidence of what occurred during the call,

leaving some room for continued disagreement.

Independent of the problems associated with realtime monitoring, is the issue of call transfers. Using

the current communications technology available at

IRS, reviewers are unable to follow a call if it is

transferred outside of the original site receiving the

call. If a customer service representative receives a

call that he or she is unable to answer, they must

transfer that call to another assister. If that call is

then routed to another call site, in the current

telecommunications

environment,

the

CQRS

reviewer is unable to follow the selected call.

Therefore, the reviewer cannot determine if the

taxpayer received the correct answer to their inquiry.

This situation is becoming increasingly more

common given the current operational push for call

site specialization, where assisters at a given call site

are trained to answer only specific types of calls.

Whether the taxpayer selected the wrong option from

the automated menu or because the initial CSR who

screened the call misunderstood the taxpayer’s

question, calls that are misrouted would have to be

transferred to another site rather than to another

assister within the same site. This increase in call

transfers would result in an increased number of calls

selected for review that the CQRS reviewers would

not be able to follow to completion.

Recording Taxpayer Calls for Quality

Purposes

Until recently, the recording of taxpayer telephone

calls, while legal if the act of recording is disclosed to

callers, was not permitted based upon guidance from

IRS Counsel. Call recording was viewed as an

invasion of taxpayer privacy. However, since call

recording has become standard in the customer

service arena, IRS has revisited the issue and

approved call recording for quality purposes only.

With the aid of a vendor, IRS is now testing and

piloting call recording in select call sites. Call

recording is scheduled for complete installation and

implementation in all IRS call sites by FY 2006.

Telecommunications technology being implemented

will enable IRS to record 100 percent of all incoming

calls, then systematically select calls for quality

review. The selected recordings would then be

reviewed by CQRS reviewers and entered into the

standardized database.

Because review will still occur at CQRS, all the

advantages of this consistent third-party review

remain. However, call recording also brings a

number of additional benefits.

Primarily, call

recording eliminates many of the drawbacks of the

real-time monitoring of telephone calls.

Once

reviewers are able to listen to a recording of the

taxpayer/CSR interaction, there will be no more

listening to dead air and no waiting on hold. They

will be able to fast-forward through any wait time

while the CSR is researching the taxpayer’s issue.

Reviewers will also be able to rewind the recording

and re-listen to portions of the call, or the entire call

if necessary, to more accurately assess what occurred

during the call. Additionally, if the topic of the call is

beyond the scope of a reviewer’s training, he or she

will be able to flag the call for evaluation by another

reviewer with more technical expertise. Reviewers

will also be able to listen to a recording at any time of

day, regardless of when the call was placed,

eliminating the need and added cost of an evening

shift. Furthermore, the late night calls occurring

between the hours of 12 a.m. and 2 a.m., not

previously subject to review, will be available for

quality review sampling under call recording. All of

these advances secured through the implementation

of call recording allow for cost savings, resource

savings, and improvements to the quality sampling

and review process.

An additional and unexpected resource savings is the

reduction in the number of rebuttals from sites who

feel the evaluation of a call by the CQRS was

incorrect. During the pilot process, those sites with

call recording have been able to listen to any call

received at their site, including those CQRS

reviewers may have evaluated for quality purposes.

Because call site managers are able to access and

listen to the real CSR/taxpayer interaction rather than

rely on reviewers’ notes, they can immediately

eliminate rebuttals for calls they believe were

evaluated correctly without any CQRS involvement.

Now, only those calls where there is a legitimate

disagreement in the call evaluation, are forwarded to

CQRS for further action, resulting in an overall

decrease in the amount of resources spent on

rebuttals and re-evaluations.

Another advantage of call recording is that recorded

calls, once “sanitized” to remove any taxpayeridentifying information, may be used for training

purposes.

Recordings of real taxpayer/CSR

interactions will allow IRS to train CSRs how best to

respond to taxpayer issues. By listening to the same

call, assisters can be taught to respond to many

different situations in a consistent way. In the same

manner, the recordings can also be used by CQRS

managers to train reviewers to consistently evaluate

the quality of the service provided to taxpayers

calling the IRS.

Call recording also provides a major advancement in

the way IRS is able to measure quality. Because

reviewers are currently unable to follow a call if it is

transferred from one site to another, it is impossible

to capture everything that individual taxpayer

experienced from the start of the call to the very end

of the call.

With the implementation of call

recording at all IRS call sites, all incoming toll-free

telephone calls will be recorded at each site.

Therefore, if a call is transferred from one site to

another, the portion of the call after being transferred

will be captured and recorded at the destination site.

Because all calls will be tagged with a unique

identifier as they enter the IRS, multiple segments of

a single call can be combined after-the-fact. Thus,

for the first time since quality review began, IRS will

be able to capture the complete taxpayer experience

for any call in the universe.

Because call recording is still in the pilot phase and

the necessary hardware and software has not yet been

installed in all sites, it is difficult to identify any

disadvantages or problems with using this

methodology to capture and review calls for quality

measures.

Certainly, initial start-up costs are

extremely high, but the resource savings and added

benefits of call recording should eventually outweigh

those one-time costs.

IRS Toll-Free Telephone

Quality Measures

Assistance

Once IRS monitors or records a call, how is the

quality of that call measured? The methodology

behind the measurement of quality has also evolved

over the years.

Past

Methodology

Measurement

of

Telephone

Quality

Before FY 2004, there was a single measure for the

quality of telephone calls coming into the IRS tollfree telephone assistance service. This measure was

reported internally to IRS executives, and externally

to Congress, GAO, and the Office of Management

and Budget (OMB). Quality for a call was measured

as pass or fail, where if one element, or “attribute,” of

a call was incorrect, the entire call was counted as

incorrect. An attribute is any individual element of

the call that is rated for accuracy by IRS reviewers.

Depending upon the taxpayer’s issue, some attributes

of a call may be: Did the assister greet the taxpayer

courteously and professionally? Did the assister

verify the taxpayer’s social security number? Did the

assister give the taxpayer the correct answer to their

question? Did the assister provide their identification

number? Certain attributes, while required elements

during a call, may not affect the correctness of the

answer provided to the taxpayer. Also, not all

attributes are applicable to every type of call, so one

particular call may have very few attributes, while

another may have several. Regardless of the number

of attributes for any given call, a single call could

only have the possibility of scoring 0 percent or 100

percent. Under the pass/fail methodology previously,

if an assister answered the taxpayer’s question

correctly, but forgot to provide his or her

identification number at the start of the call, as

required under IRS procedures, the call was scored as

0 percent for quality measurement purposes. While

this practice encouraged attention to details on the

part of the telephone assisters, it presented an unclear

measure of the quality of service provided to

taxpayers, especially to the external users of the data.

Current Methodology

Measurement

of

Telephone

Quality

In an attempt to construct a more accurate picture of

the quality of the service provided to taxpayers, the

pass/fail methodology was retired and a new

measurement system was implemented for FY 2004.

This new method of measurement, defects-peropportunity, was designed to distinguish between

wrong answers and procedural errors that do not

affect the accuracy of the answer provided to the

taxpayer. IRS’s single measurement for quality was

separated into five individual measures:

•

•

•

•

•

Customer Accuracy — Did the assister give

the taxpayer the right answer?

Regulator Accuracy — Did the assister

follow all IRS regulations according to the

tax code?

Procedural Accuracy — Did the assister

follow all internal IRS procedures for this

type of call?

Timeliness — Did the assister respond to the

taxpayer in a timely manner?

Professionalism — Did the assister respond

to the taxpayer in a courteous and

professional manner?

Given the nature of the measures, Customer

Accuracy, Timeliness, and Professionalism are

reported externally; whereas, Regulatory and

Procedural Accuracy are measures intended for IRS

use only. The five measures are each calculated as a

percentage: the number of correct attributes divided

by the total number of applicable attributes. Because

Customer Accuracy has only one applicable attribute

for any type of call — Did the taxpayer receive the

correct answer? — a single call still only has the

possibility of scoring 0 percent or 100 percent.

However, with the elimination of all other nonapplicable attributes, this measure of accuracy is now

a very clear representation of the quality of the

service provided to taxpayers. Each of the other four

measures generally has multiple applicable attributes

for each call, thus a single call can now score 0

percent, 100 percent, or anywhere in between.

Using FY 2003 data, Customer Accuracy was

calculated using both methods:

FY 2003

data

Tax Law

Accounts

Pass/Fail

80.10%

(+/- 0.66%)

68.43%

(+/- 0.44%)

Defects-perOpportunity

81.97%

(+/- 0.63%)

88.15%

(+/- 0.30%)

There was little effect on Tax Law calls, but the

difference in the accuracy of Account calls is

significant. This is due to the fact that for Account

calls, telephone assisters are generally required to

perform many internal procedures where an error

may occur that does not affect the accuracy of the

answer provided to the taxpayer. Under the old

pass/fail methodology, this would have caused the

entire call to be counted as incorrect. Now, attributes

relating to internal procedures are included in the

measures of Procedural Accuracy and Regulatory

Accuracy and are no longer included in Customer

Accuracy, providing IRS executives, as well as

Congress, GAO, and OMB, a clearer picture of the

quality of service provided through the toll-free

telephone assistance service — that assisters actually

gave callers a correct answer approximately 88

percent of the time rather than the 68 percent

previously reported.

Future of

Measures

IRS

Telephone

Quality

Over the years, the way IRS monitors telephone calls

and measures quality has undergone continuous

improvement.

With the implementation of call

recording, the Service will have taken the next step in

the process. Beyond call recording, there are plans

in motion to combine national quality review

performed by CQRS and local managerial review

into one standard database. This will be the last step

in the standardization of the review process. Once

completed, managers and quality reviewers will be

reviewing calls using the same attributes and

standards and all quality data will be stored in a

single place. This will provide individual call sites

with additional data for error and trend analysis,

allowing them to identify specific areas where

additional training might improve quality.

In another move to improve the quality process,

individual attributes from reviewed calls will soon be

directly linked to telephone assisters’ critical job

elements (CJEs).

CJEs are the specific items

managers use to rate the performance of their

employees. For example, call attributes regarding

courtesy and professionalism will be linked with the

professionalism CJEs for telephone assisters. The

attribute for whether or not the taxpayer received the

correct answer will be linked with the technical

knowledge CJEs for assisters. Through this linkage,

managers will be able to use their reviews to quantify

the performance of their employees rather than

relying solely on qualitative data and subjective

judgment.

Conclusion

A significant goal of the IRS is to make its telephone

operations

a

world-class customer service

organization. By improving the way the level of

service provided to taxpayers is measured, IRS can

not only better determine how closely they have

come to achieving that goal, but can also identify

areas for further improvement. With this continuous

cycle measurement and improvement, we hope this

goal of providing world class customer service to

taxpayers will soon become reality.