Consultation Paper on Performance Accountability Measurement for the Workforce Investment System Under Title I of the Workforce Investment Act

SUMMARY: The purpose of this notice is to disseminate consultation

papers for interested parties on the Performance Accountability

Measurement System for Title I of the Workforce Investment Act. There

are two papers. The first presents the broad framework for Core

Measures of Performance and Customer Satisfaction specified in Title I,

Section 136. The second presents the framework for Negotiating State

Adjusted Levels of Performance as specified in Title I Section 136.

These papers are to be used by States intending to implement the

Workforce Investment Act as of July 1, 1999. The Department of Labor

will work with States individually to ensure that there are no negative

consequences if significant changes occur in these papers based on the

comments received. Interested parties have 30 days to provide comments

on these papers. Over the next several months additional consultation

papers will also be disseminated for comment.

FOR FURTHER INFORMATION CONTACT: Mr. Eric Johnson, Workforce Investment

Implementation Taskforce Office, U.S. Department of Labor, 200

Constitution Avenue, NW, Room S5513, Washington, DC, Telephone: (202)

219-0316 (voice) (This is not a toll-free number), or 1-800-326-2577

(TDD). Information may also be found, or comments provided, at the

website--http://usworkforce.org.

SUPPLEMENTARY INFORMATION: The Workforce Investment Act (WIA or Act),

Pub. L. 105-220 (August 7, 1998) provides the framework for a reformed

National workforce and employment system designed to meet the needs of

the Nation's employers, job seekers and those who want to further their

careers. Title I of WIA specifies Core Performance and Customer

Satisfaction measures. Each Governor must submit a five-year strategic

plan no later than April 1, 1999, to begin WIA programs by July 1,

1999, and no later than April 1, 2000, to begin WIA programs by July 1,

2000. The current law, the Job Training Partnership Act, is repealed

effective July 1, 2000. States planning to implement during PY 1999 are

to utilize these papers in addressing Performance Accountability in

their plans.

An important part of the five-year strategic plan is the

establishment of performance levels for each of the core performance

and customer satisfaction measures, which will be negotiated between

the Governor and the Secretary of Labor. These levels will form the

basis for incentives and sanctions as specified in Title I, section 136

and Title V, section 503 of the Workforce Investment Act.

The U. S. Department of Labor is establishing this performance

accountability measurement system, and the process for reaching

agreement on State adjusted levels of performance. These two

consultation papers are part of that effort. Some of the questions on

which the Department of Labor is seeking input are the following:

Which services would be appropriately defined as self-

service/informational and thus not included in the core measures, and

which services fall into the core services, intensive services or

training;

The point at which adult and youth registrants are counted

for different performance measures (e.g., at a certain time after

registration, during the reporting period, after completion of service,

after program exit);

The use of wage records for performance measurement

considering availability, completeness, accuracy, timeliness and when

wage records might be combined with supplemental sources (i.e.,

administrative records or survey data for performance purposes)

considering the need for consistency, comparability and cost

effectiveness;

Who will be counted in the numerator and/or denominator of

those measures expressed as rates, for example how should the employed

and underemployed who receive services be accounted for;

Identifying possible unintended effects resulting from

definitions/policies around performance measurements;

Identifying burdensome and unnecessary requirements that

will provide limited benefit, but will be costly in terms of both

record keeping requirements and processing;

Using adjustment models in (1) negotiating State adjusted

levels of performance to account for differences in service mix,

participant characteristics and labor markets and/or (2) determining

eligibility for incentives and consideration for sanctions comparing

negotiation assumptions with actual information.

Sources and types of information that would be useful in

negotiating State adjusted levels of performance;

The circumstances in which revisions to State adjusted

levels of performance will be required by the Department, including

special circumstances for early implementing States and the differences

between the first program year and subsequent years covered by a

State's 5-year plan. For example, if better data available in later

years or if actual performance data shows that the assumptions under

which State adjusted performance levels were negotiated are incorrect,

should the Department require that approved levels for later years be

changed.

Please consider these issues as you review these consultation

papers, and provide comments.

Signed at Washington, DC, this 17th day of March 1999.

Raymond L. Bramucci,

Assistant Secretary of Labor, Employment and Training Administration.

Attachment 1--Performance Accountability Measurement for the

Workforce Investment System

I. Introduction

The Workforce Investment Act (WIA) calls for a comprehensive

accountability system to assess the effectiveness of State and local

areas in providing employability and training services. The Act

requires:

A focus on results defined by ``core indicators'' of

performance;

Customer Satisfaction with programs and services measured

and related to results;

A strong emphases on Continuous Improvement of Services;

Annual performance levels and improvement plans developed

during negotiations among Federal, State and local partners;

Awards and Sanctions based on State performance; and

State reporting and record keeping.

In addition, States are required to provide annual reports to the

Secretary of Labor with respect to progress in achieving State

performance measures. The Act requires certain additional information

be provided, such as cost of workforce investment activities and

specified recipient data.

This paper presents a draft framework for the workforce investment

system core performance and customer service measures that apply to

States in Title I of the WIA and will be used in determining State

adjusted performance levels, eligibility for incentive grants or

imposing sanctions. Additional papers on all of the above listed

requirements

[[Page 14337]]

are being developed and will be provided for comment in the coming

weeks. This paper is intended to elicit discussion about how success

will be defined for workforce investment system activities and how it

can efficiently and effectively be measured Statewide. The concepts

within the paper build on previous Department of Labor (DOL) efforts

such as the Workforce Development Performance Measures Initiative, the

Labor Exchange Performance Measures Work Group, Simply Better!,

Employment Service (ES) Reinvention and the Enterprise. It also

incorporates input from State and local officials that was received at

recent consultations focusing on WIA accountability and from other WIA

briefings and communications.

Please keep in mind that this document presents an overall

framework. It does not fully address a number of the detailed technical

and operational issues that were raised at recent consultations, nor is

it intended to serve as reporting instructions. In addition to your

feedback on this framework, we are also interested in input on

technical and operational issues that may not have been addressed by

this paper. All of this input will be used to develop further guidance

and finalize the document for use by those States planning to implement

the Workforce Investment Act before July 2000. This paper includes

proposed definitions for the Core Indicators of Performance that will

be used for State incentive grant eligibility determinations and

sanctions.

The paper is divided into three sections:

Adult Performance Measures and Definitions that will apply

separately to: (1) Adult Services, (2) Dislocated Worker services, and

(3) Services to Eligible Youth 19 to 21 years old in Youth programs

under Section 129 of WIA.

Youth Performance Measures and Definitions for Services to

Eligible Youth 14 to 18 years old in the Youth Program.

Customer Satisfaction Measures for Participants and

Employers.

While the specific core performance indicators and customer

satisfaction indicators outlined in this paper only apply by law to

Title I of WIA, DOL may adopt them, as appropriate, for other DOL

programs, and will work in cooperation with other Federal partner

agencies to reach agreement where feasible on uniform measures. Thus,

all partners are encouraged to review and comment on this draft

framework. However, any changes to other programs whether internal or

external to DOL would require appropriate actions based upon present

Laws, Regulations and/or policies.

Definitions to be used by all States and localities are provided

for each of the core indicators and customer satisfaction measures to

ensure comparability. Comparability of measures among States is

important for two reasons. First, core indicators and the customer

satisfaction performance levels are to be negotiated between the States

and DOL. One of the factors affecting those negotiations are ``how the

levels compare with State-adjusted levels of performance established

for other States * * *'' Second, comparability also contributes to

continuous improvement. Having standard definitions will allow States

and localities to benchmark other States and localities to promote

continuous improvement. Comparability also will facilitate the sharing

of best practices within and among the States. Since the performance

and accountability system under WIA includes incentives and sanctions,

comparability is important to fairness and equity.

Continuous improvement is a significant and required element of

WIA. States and localities need to collect more substantial data than

the core measures or other required measures under the Act to function

in a continuous improvement environment. Therefore, it is important for

State and local leadership to take advantage of the opportunity when

developing their performance systems to go beyond Federal requirements.

II. Adult/Dislocated Worker Services

A. Workforce Investment Act (WIA) Requirements/Program Activity

Categories for Reporting

The WIA provides for a continuum of service delivery that includes

three levels of services: (1) core services; (2) intensive services;

and (3) training services. All persons will have access to core

employment-related information and self-service tools without

restrictions or additional eligibility requirements, assuming

sufficient funds are available. Those core services that are not

primarily informational and must be staff-assisted will require WIA

registration. Intensive services and training will also require WIA

registration. The intensive services are provided when a determination

is made that unemployed individuals are or would be unable to obtain

employment after receiving the basic core services, or when employed

individuals are determined to be in need of these intensive services to

obtain or retain employment that allows for self-sufficiency.

Similarly, training services are only available after a determination

that the individual is unable to obtain or retain employment that leads

to self-sufficiency through intensive services.

For accountability purposes, WIA establishes core indicators of

performance in State and local WIA financed systems for participants in

all workforce investment activities other than self-service and

informational activities. This exception recognizes the low cost per

unit of providing these services. WIA also requires that States and

localities report on how participants in workforce investment

activities other than training (except for self-service and

informational activities) compare to participants in training

activities. Therefore, the level of service individuals receive defines

whether the individual will be counted in the core indicators and if

so, how they will be categorized for reporting purposes.

Many of the core services generally will be low cost, self-service

and consist primarily of information and not require registration. In

contrast, intensive services will be more costly, require significantly

more staff investment, and thus, justify a different measurement system

that calls for registering and tracking individuals throughout their

program participation. For reporting, services are divided into--

Core Services (for registered participants)

Intensive Services

Training

Consistent with WIA, participants who use one-stop self-service

facilities or only access information do not need to be registered and

tracked. Access to some Core Services will be universally available

through the Internet, at a One-Stop center or through a One-Stop

partner. States and local Boards will be free to allow completely

anonymous access to core services that are primarily information and

universally available (for example, browsing a job bank or using a

computer in the resource room). However, States and local Boards may be

encouraged to request unique identifying information about customers

who use the Internet (for example, current America's Job Bank account)

and who use the universally available self-service capacity in One-Stop

centers.

For customers who are assessed for purposes of determining whether

they require services or training that are not universally available,

additional information will need to be collected as part of the

assessment process. This information will include demographic data

elements such as racial-ethnic

[[Page 14338]]

characteristics, veteran status and information on disabilities, and

will assist in referring individuals to other partners' services and

will be available for comparisons of program applicants and

registrants.

B. What Services Fall Into What Category?

The categorization of services is a State or local decision and

will depend on the nature of the service. To serve as a guide and to

assist in this identification, Table 1 includes most of the core,

intensive, and training services described in Section 134(d). Each of

the required WIA services is italicized. Frequently provided services

that are in addition to those required by the legislation are not

italicized. Given the wide variation in types of service that can be

categorized as job search and placement assistance, and career

counseling, finer distinctions have been made for these services. As

soon as a participant moves from the self-service/informational level

of service to registered service (core, intensive, training) core

measures apply.

Table 1.--Categories of Services

----------------------------------------------------------------------------------------------------------------

C. Intensive services D. Training

A. Core services-- Self-service and B. Other core services (registration (registration

information (registration required) required) required)

----------------------------------------------------------------------------------------------------------------

Determination of eligibility to Follow-up services, Comprehensive and Occupational skills

receive assistance. including counseling for specialized training.

registrants (those assessment, including

previously receiving diagnostic testing

intensive/training and interviewing.

services) after entering

employment.

Outreach, intake (which may include Individual job development. Development of On the Job Training.

profiling), and orientation to the individual employment

One-Stop center. plan.

Initial assessment of skill levels, Job clubs.................. Group counseling...... Workplace training and

aptitudes, abilities, and support cooperative education

service. programs.

Labor Market Information........... Screened referrals (testing Individual counseling Private sector

and background checks done and career planning. training programs.

before referral or when

operating as the employers

agent).

Consumer reports information and ........................... Case management....... Skill upgrading and

delivery system performance retraining.

information.

Information on other One-Stop ........................... Short term pre- Entrepreneurial

partner services and supportive vocational services. training.

services.

Information on filing UI claims.... ........................... ...................... Job readiness

training.

Assistance in establishing WtW ........................... ...................... Adult education and

eligibility and other non-WIA literacy activities

training and education. in combination with

training.

Resource Room usage................ ........................... ...................... Customized training.

``How to'' group sessions (e.g.

writing a resume).

Job referrals (informational, e.g.,

job scouts, ES referrals in non-

exclusive hiring arrangements,

short-term or seasonal

placements).

Internet browsing--job, info, and

training searches.

Internet accounts--Career Kit,

Personnel Kit.

Talent referrals (informational,

e.g., talent scouts, ES staff

referrals of resumes without

further screening).

----------------------------------------------------------------------------------------------------------------

The following considerations provided some of the rationale used in

preparing this guide. First, ``self-service and informational

activities'' are by their nature core services that do not require

registration and tracking. A second consideration is the likely per

unit cost of services. A number of placement activities are primarily

informational in nature and relatively low cost. In these instances,

the added cost of registration and participant tracking may not be

justifiable. Thirdly, some services benefit participants but are

undertaken primarily for their value to employers (e.g., assistance

with recruitment for seasonal work--summer or holiday) are intended

only to provide short term employment and do not necessarily increase

worker earnings, retention or occupational skill attainment. Fourthly,

these groupings of activities are intended to be clean, easy to

administer, and applicable to all programs.

[[Page 14339]]

C. How to Measure Core Services, Intensive Services and Training

1. Core Indicators of Performance

----------------------------------------------------------------------------------------------------------------

Measure Definition

----------------------------------------------------------------------------------------------------------------

Entry into Unsubsidized Employment:

Entered Employment Rate (Sec 136(b)(2)(A)(i)(I)) The rate will be defined for cohorts of registered

participants unemployed at the time of registration. The

numerator will be the number of these registered

participants that are shown to have paid employment in

the quarter following registration or service completion.

The denominator will be all registered participants

unemployed at the time of registration who were active

during the reporting period (received services or

continuing from a prior period) but who are no longer

actively receiving services, other than post-employment

services. This includes enrolled participants who (1)

have obtained unsubsidized employment; (2) have withdrawn

from participation; or (3) who have completed training or

services. Individuals should be considered no longer

active and to have completed service if they have

received no services in the last quarter of the reporting

period, and are not scheduled to receive services in the

future.

Note: State and local officials opposed using a program

exit or termination to trigger reporting.

Records of all registered participants unemployed at the

time of registration, and not enrolled in a training

program at the end of the reporting period would be drawn

and matched against wage records to identify employment.

Dislocated workers as defined in WIA, Title I, subtitle

A, sec.101 (9) are included in the definition of

unemployed. A person is considered employed if his/her

social security number appears in the employer wage

report (no minimal wage requirement) in the quarter

following the one in which the seed record for matching

is drawn.

Not all jobs are covered by State UI wage records.

Therefore, a State or locality may supplement the results

of the wage record review by other methods and count as

employed any of these individuals in jobs not covered by

the State's UI wage records. Again, employment would be

determined based on employment in the quarter following

the one in which the seed record is drawn. Supplementary

information could include: use of the New Hire index,

surveys, self-reported hires or staff-reported hires

through an administrative record system.

Seed records not shown as employed would be matched

against administrative records to exclude from the

computation individuals who remain active, i.e, have

received services in the last quarter or are scheduled to

receive them in the future.

Retention in unsubsidized employment Six Months

after entry into the employment:

Six Month Retention Rate (Sec The rate is computed using information on the total number

136(b)(2)(A)(i)(II)). of registered participants who have employment and who

appear in the wage records, and wage record information

for the second quarter thereafter (6 month rate). For

example, an individual completing training and placed

immediately in the first quarter of the program year,

would be recorded as employed in the second quarter. The

fourth quarter records would be queried to determine

retention.

Note: retention is not limited to the same employer.

Earnings Received in Unsubsidized Employment Six

Months After Entry into Employment:

Average Earnings Change in Six months (Sec The average earnings change is measured as follows: the

136(b)(2)(A)(i)(III)). wage record earnings for the registered participant in

the two quarters following employment (not counting the

quarter in which employment was recorded) less 50% of the

wage record earnings for the four quarters prior to

enrollment (not counting the quarter of enrollment). The

post-employment income can be with the same or other

employer in which the placement was first noted. The

measure is reported as an average (mean) gain. For

Incumbent workers and others who are employed at the time

of registration average earnings and retention would be

computed begining in the second quarter following the

quarter in which services are completed.

Educational Credential/Occupational Skills

Credential Attainment Rate (Training Services For adults entering employment after training and eligible

Only) (Sec 136(b)(2)(A)(i)(IV)). youth 19 through 21 entering employment, post-secondary

education or advanced training after training, the

percent who attained a State-recognized credential

related to educational skill attainment (diploma, degree

or certificate) or attainment of an occupational skill

(license or certification) recognized by a State or a

Nationally-recognized industry trade body. Information in

administrative records or information gathered through

electronic interfaces with other data bases available or

surveys may be used. Additional guidance on acceptable

credentials and certificates will be provided.

----------------------------------------------------------------------------------------------------------------

2. Why These Measures and Definitions?

Wage record versus surveys. WIA requires that wage records be used

but does not exclude the use of surveys to supplement information.

Where the same information is obtainable from both sources,

instructions will provide that wage records be used both to ensure

comparability among States and to minimize costs.

Earnings change versus absolute earnings level. A change--gain/

loss--measure was chosen because of the different circumstances within

the WIA service population and the difficulty of creating an absolute

earnings target for all adults.

Earnings change--prior period earnings. Six month comparisons are

made with a full twelve month period prior to registration. (Fifty

percent is used for the six month change for comparability of periods.)

The twelve month period is used to minimize, to the extent possible,

the impact on dislocated workers of reduced earnings as layoffs

approach and any severance pay that may be added to earnings.

Retention rate. While most of the measures use the registered

population as the base, this measure chooses only those who are

employed because the retention question makes no sense in another

context.

[[Page 14340]]

Credentials rate (Training Service only). Adults entering

employment after receipt of core or intensive services also may acquire

some credential as a result of WIA participation. However, given the

few relative to the number who will be employed after receiving

services, the cost of this added information collection is not

justified.

III. Youth 14 to 18 Years Old

A. WIA Requirements/Guiding Principles

WIA authorizes programs to provide services to prepare youth to

enter the workforce or to advance to postsecondary education or other

occupational skills training. Programs link academic and occupational

learning. Service providers will have strong ties to employers and must

also include tutoring, study skills training and instruction leading to

completion of secondary school (including dropout prevention). Other

elements of programs should include alternative school services,

mentoring by appropriate adults, paid and unpaid work experience (such

as internships and job shadowing, occupational skills training,

leadership development, and appropriate supportive services.) Youth

participants also will receive guidance and counseling and follow-up

services for at least one year. Programs must also provide summer job

opportunities linked to academic and occupational learning. The mix of

year-round and summer activities is left to local discretion.

Eligible youth are low-income, ages 14 through 21 with barriers to

employment. WIA specifies different youth core indicators for older

youth ages 19 through 21 and for youth ages 14 through 18. The older

youth performance measures are identical to the adult program measures

and were addressed earlier in the paper. The three required core

indicators for youth ages 14 through 18 are--

Attainment of basic skills and, as appropriate, work

readiness or occupational skills;

Attainment of secondary school diplomas and their

recognized equivalent;

Placement and retention in postsecondary education or

advanced training, or placement and retention in military service,

employment, or qualified apprenticeships.

B. Guiding Principles

Performance measures should reflect the same flexibility

available for program design and services and the varied successful

outcomes recognized. WIA allows a wide variety of services that are

offered to youth. This provides the opportunity for local programs to

design and operate programs that meet local needs and respond to gaps

in services as they are identified for their area. Indicators should

not force certain designs to remain competitive in terms of

measurement.

Performance measures should accommodate a variety of

different approaches to serving youth without forcing arbitrary time

limits or sequencing of services. Measures should not determine when

youth begin or end services by forcing measurements before participants

are ready to complete their goals or require that a youth leave the

program before credit can be taken for outcomes achieved. Research has

shown that programs that establish an ongoing relationship with youth,

and continue to serve them for several years while adjusting goals and

services to reflect needs as youth age have the greatest success.

Performance measures need to recognize that youth goals

change as youth mature and must be age appropriate. The denominator for

various rates should depend on the appropriateness of the goal as

determined through individual service plans. For example, younger youth

participants should receive services that encourage staying in or

returning to school and keeping up academically. As the participants

get older, goals will change and relate to getting a secondary school

diploma, and, ultimately, placement and retention in post-secondary

education, advanced training or employment.

Indicators must (1) recognize the differing goals

depending on the activities/services; (2) the age of the youth; and (3)

allow comparison between activities/services that include modest levels

of summer job opportunities and those that invest a large proportion of

resources during the summer. WIA provides for summer employment

opportunities directly linked to academic and occupational learning,

but intentionally did not provide a separate funding stream or

differing set of indicators. In addition, the summer employment

opportunities element is not intended to be a stand alone program (WIA

sec. 129(c)(2)(C)). Indicators should be designed so that achieving

every goal established for every youth--whether for a year-round or

summer employment opportunities only--would result in a 100 percent

rate.

C. Establishing a Basis for Performance Measurement

State and local officials with whom DOL consulted strongly believed

that the core indicators of performance for participants should reflect

completion of their activities, not necessarily completion of

participation. By design, youth programs are intended to provide a

continuum of services for youth resulting in attainment of several

interim outcomes such as the acquisition of basic skills, work

readiness, or occupational skills, award of a secondary school degree,

and then, placement and retention in employment or advanced education/

training. Participant goals are reflected in individual service

strategies and will be different from one youth to another, depending

on the needs and interests of the youth.

1. Core Indicators of Performance

----------------------------------------------------------------------------------------------------------------

Measure Definition

----------------------------------------------------------------------------------------------------------------

Attainment of basic skills and, as appropriate, work

readiness or occupational skills:

Skill Attainment Rate (Sec 136(b)(2)(A)(ii)(I).. A rate computed by dividing the number of youth who

attained a higher level of proficiency with regard to

basic skills, and, as appropriate, work readiness skills

or occupational skills by the number of youth receiving

services or training for whom attaining basic skills,

and, as appropriate, work readiness skills or

occupational skills were goals to be achieved during the

reporting period. Goals are based on individual

assessments using widely accepted and recognized

measurement/assessment techniques. (Outcomes are counted

as they are achieved, not when the youth completes

program participation).

[[Page 14341]]

Attainment of secondary school diplomas or

recognized equivalents

Diplomas or Equivalent Attainment Rate (Sec A rate computed by dividing the number of youth who

136(b)(2)(A)(ii)(II)). attained a secondary school diploma or equivalent divided

by the number of youth for whom attaining a diploma or

certificate was a goal to be achieved during the

reporting period. This goal will generally be appropriate

for older youth 16 or 18 years old.

Placement and retention in postsecondary education

or advanced training, or placement and retention in

military service, employment, or qualified

apprenticeships:

Retention Rate (Sec 136(b)(2)(A)(ii)(III))...... Of those who are receiving follow-up services and for whom

placement and retention is a goal, the percent with

retention status at 30 days, 90 days, 180 days and one

year from beginning follow-up. This overall rate would be

an average of measures for all four periods.

----------------------------------------------------------------------------------------------------------------

2. Age as the Basis for Outcomes.

These younger youth reporting requirements are for participants who

are 14 to 18 when enrolled in the youth program. They are appropriate

as long as the youth continues to receive services identified in their

individual service plans and has not attained the outcomes established

for that age group. If the youth were going to continue being served by

the youth program after achieving goals established prior to age 19,

the youth does not need to be ``terminated'' but instead the record

would indicate that the youth transferred to the age 19 to 21 youth

program. This would have the same effect as a ``termination'' for

performance measurement purposes but would not disrupt services to the

youth. Individuals ages 19 through 21 who have completed the goals

established for them prior to turning 19 and those who enter the youth

program as age 19 or older should have individual service plans leading

to the attainment of the goals appropriate for that age group. These

outcomes would be measured in accordance with the same principles

established for the adult program.

D. Why These Measures and Definitions?

Skill Attainment Rate; Attainment of secondary school diplomas and

their recognized equivalents: and Placement in postsecondary education

or advanced training, or placement in military service, employment, or

qualified apprenticeships. All three core indicators use as their

denominators the total number of youth age 14 through 18 for whom the

particular outcome to be measured was a goal. This recognizes that

depending on the age of the youth and their needs, individual goals

will differ significantly and that the effectiveness and quality of a

program should be measured by its experience in achieving goals

established for its participants. It allows comparisons of programs and

allows for program goals to be established without having to know the

exact mix of ages of participants and relative investments in different

activities. It also permits a 100 percent rate if all goals are

achieved.

Note: A youth participating in WIA youth activities/services for

multiple reporting periods (years), could be counted during each of

the periods (years) if goals established in each are achieved.

Placement Rate and Retention Rate. Unlike the adult program where

separate measures are provided for placement and retention, WIA calls

for a single youth indicator that includes both placement and retention

rates in one measure. The retention rate--for unsubsidized employment

and further education--was chosen as the core indicator because of the

difficulty in coming up with a single measure that measures placement

and retention. A meaningful measure with both would be difficult

because of complications coming up with a single denominator. Placement

will be measured, but outside the core indicators.

Unsubsidized employment. WIA specifies that adults be measured with

regard to placement in unsubsidized employment. The Youth indicator

specifies only placement in employment. Unsubsidized employment is

being used for consistency and because it, and not subsidized

employment, will lead to self-sufficiency.

IV. Customer Satisfaction

A. Workforce Investment Act Requirements

In addition to the core measures, WIA, at sec. 136(b)(2)(B), states

that ``the customer satisfaction indicator of performance shall consist

of customer satisfaction of employers and participants with services

received from the workforce investment activities authorized under this

subtitle.'' The Statute also requires, at sec. 136(b)(3)(A)(i), that

there be State adjusted levels of performance for customer satisfaction

and that ``the levels of performance established * * * shall, at a

minimum--

(I) Be expressed in an objective, quantifiable, and measurable

form; and

(II) Show the progress of the State toward continuously improving

on performance.''

The Act draws a link between the core measures and customer

satisfaction by indicating that negotiations between DOL and the States

must take into consideration ``* * * the extent to which the levels

involved will assist the State in attaining a high level of customer

satisfaction.'' WIA further suggests that ``customer satisfaction may

be measured through surveys conducted after the conclusion of

participation in workforce investment activities.''

B. Methods To Measure Customer Satisfaction

Customer satisfaction measures are important because they provide

valuable information from customers for strategic planning and program

operation. Such feedback to supervisors and staff can motivate high

performance and continuous improvement. They also send a clear message

to staff, management, and customers themselves that customers matter.

[[Page 14342]]

There are a number of different methods to collect customer

satisfaction information. The simplest approach is to train staff to

listen to the customers they serve and to ask questions that elicit

customer needs while they are providing service. Focus groups and group

interviews are another strategy. A trained manager or staff person can

circulate in the resource center where people are waiting and ask

questions informally to gain a better understanding of customer needs

and concerns. Suggestion boxes are also a way of gathering information.

As part of a comprehensive continuous improvement strategy,

organizations generally use a combination of strategies since each

serves a somewhat different purpose and provides different types of

information.

To meet the customer satisfaction requirements of WIA, the

recommended measures focus on customer satisfaction surveys. This is

the only method that allows State and National aggregation of

comparable, quantifiable data. The proposed measures present a general

framework for developing a National customer satisfaction index for

different customers.

An index is a single score that is created by combining the scores

from several questions that address different dimensions of the

customer experience. The customer satisfaction index will be described

in more detail in the proposed measures. Essentially, the index would

provide a way to capture common customer satisfaction information

across programs that could be aggregated to a State and National level.

The proposed measures will continue to be modified as DOL receives

feedback and validation through consultation with the workforce

investment system.

C. Proposed Customer Satisfaction Measurement Strategy

WIA requires measures of customer satisfaction for both

participants and employers that are quantifiable, comparable across

States, and that, along with the core indicators of performance,

promote continuous improvement.

The basic approach for gathering and reporting customer

satisfaction measures that meet these requirements will be as follows--

There will be separate surveys of participants and

employers;

Customer satisfaction indicators will be derived from

State or locally conducted surveys that include a few embedded

questions that enable comparisons across States, while allowing each

State to design its own instrument;

Guidelines will be issued to the States requiring the

States to validate survey methods to ensure comparability across

States;

Participants surveyed will include only registered

individuals who have completed their activity/service participation

(excluding follow-up services). This includes individuals who have

completed services and are now employed, those who have discontinued

receiving services or training, either because they have withdrawn or

completed;

Employer surveys can be based on a random sample of

employers using the system; and

Results on the embedded questions, for both employers and

participants, will be compiled on the State level and reported

annually.

While the Act requires reporting and comparisons across States, the

primary value of effective customer feedback is to drive strategic

planning and continuous improvement at the local level. DOL plans to

play a strong, proactive technical assistance role that will provide

States and localities with easy access to the information, instruments,

tools and other resources that will enable them to use this feedback as

a springboard toward high performance and quality services.

ETA will finance the design and development of customer

satisfaction instruments and methodologies that meet requirements, and

make these available to the States. However, States will have the

option of designing their own instruments and methods as long as the

few questions that enable comparisons are embedded in the State

instruments and the methodologies used to collect responses on those

questions are within guidelines that DOL will develop. These guidelines

will include suggestions for establishing State baseline levels to be

used where they do not already exist.

Attachment II--Reaching Agreement on State Adjusted Levels of

Performance Under Title I of the Workforce Investment Act of 1998;

A Consultation Paper

This Consultation Paper provides a framework regarding the approach

and process for reaching agreement on State adjusted levels of

performance under Title I of the Workforce Investment Act of 1998 to be

incorporated in States' five-year strategic plans required by that Act.

Comments are solicited on the overall framework and the approach. The

paper does not address the additional indicators a State may identify

in the State plan in accordance with Title I, because they are not

subject to the agreement process.

I. Introduction

A. Broad Legal Framework

In Section 136, the Workforce Investment Act (WIA) establishes a

comprehensive performance accountability system for the Statewide and

local workforce investment systems. Its purpose is to assess the

effectiveness of States and local areas in achieving continuous

improvement of their Federally-funded workforce investment activities,

in order to optimize the return on investment of Federal funds in those

activities.

As part of the performance accountability system, WIA requires the

Secretary and the Governor of each State to reach agreement on the

respective State performance levels for the core indicators of

performance and the customer satisfaction indicator. WIA requires that

the State performance levels be expressed in an objective,

quantifiable, and measurable form. The law also requires that the

levels show the State's progress toward continuously improving in

performance. The negotiated performance levels become ``State adjusted

levels of performance'' and must be incorporated into the State's 5-

year plan. They become the basis for sanctions for failed performance.

Together with adjusted levels of performance for adult education

literacy programs and performance levels for vocational education

programs, they become the basis for incentive grants.

State adjusted levels of performance for the first three program

years covered by a State 5-year plan are agreed to during the plan

review and approval process. In reaching agreement on those levels, the

Secretary and the Governor must take into account the expected levels

of performance identified by the State in its 5-year plan. The

Secretary and the Governor also must take into account the following

three factors: (1) the extent to which the levels will help the State

achieve a high level of customer satisfaction; (2) how the levels

compare to those of other States, with consideration of, at least,

differences in economic conditions, participant characteristics at

entry into the program, and services to be provided; and (3) the extent

to which the levels promote continuous improvement in performance and

ensure optimal return on investment. State adjusted levels of

performance for the fourth and fifth program years covered by a State

5-year plan must be agreed to before the beginning of the fourth

program year.

[[Page 14343]]

The State plan is not required to identify expected levels of

performance for those years. In reaching agreement, the Secretary and

the Governor must take into account the same three factors listed

above. The Governor may request revision of the agreed to performance

levels if unanticipated new circumstances in the State result in

significant change in any of factors for any year.

B. Guiding Principles

WIA reflects the Nation's commitment to accountability for results

in government programs that is expressed in the Government Performance

and Results Act. It embodies strong commitment to partnership among

government entities in serving people in areas of workforce

development. It promotes flexibility to provide services that provide

maximum benefit to customers and high levels of customer satisfaction.

These commitments are characterized by key principles that must be

reflected in the process of reaching agreement on target levels for

State performance measures.

1. Performance expectations for related workforce investment

programs and for related reporting purposes (e.g., reporting under the

Government Performance and Results Act) should be aligned to facilitate

better management of overall performance.

2. State and Federal partners share the commitment to high

performance and customer satisfaction, and continuous improvement in

both.

3. The process and considerations in reaching agreement on State

performance levels must be consistent for all the States so as to

assure fairness in the derivation of performance levels.

4. States should have maximum flexibility to target populations for

WIA services within the parameters of the law, and to develop and adopt

innovative methods for accomplishing workforce investment objectives;

this flexibility should be reflected in agreed target levels of

performance.

5. States should have maximum flexibility to develop reasonable

methodologies for setting performance goals in the core measurement

areas and customer satisfaction.

C. Assumptions

Critical assumptions underlie the discussion of the process for

reaching State/Federal agreement on State adjusted levels of

performance. These assumptions relate to performance measures,

incentives and sanctions, performance data and tools, and continuous

improvement.

1. A paper on Performance Accountability Measurement for the

Workforce Investment System is being circulated for comment. The

measures described here are drawn from that paper, and so are subject

to change. There are seven measurements for the core indicators of

performance. Four of these apply separately to activities under three

funding streams (adult, dislocated worker, and youth ages 19-22) and

three apply to activities for youth ages 14-18, for a total of fifteen

measures. In addition to those fifteen measures for the core indicators

of performance, there are two measures for the customer satisfaction

indicator. Each State plan will include one State adjusted performance

level for each of the seventeen measures. The indicators and their

measures, grouped by program, are:

Adult Program (four indicators)

Entry into unsubsidized employment, measured by Entered

Employment Rate,

Retention in unsubsidized employment after entry into the

employment, measured by Six Month Retention Rate,

Earnings received in unsubsidized employment six months

after entry into employment, measured by Average Earnings Change in Six

Months,

Attainment of educational credential/occupational skills

credential for adults entering employment after training, measured by

Educational Credential/Occupational Skills Credential Attainment Rate

(Training Services Only);

Dislocated Workers Program

(Four indicators--same as for the Adult Program)

Entry into unsubsidized employment, measured by Entered

Employment Rate,

Retention in unsubsidized employment after entry into the

employment, measured by Six Month Retention Rate,

Earnings received in unsubsidized employment six months

after entry into employment, measured by Average Earnings Change in Six

Months,

Attainment of educational credential/occupational skills

credential for adults entering employment after training, measured by

Educational Credential/Occupational Skills Credential Attainment Rate

(Training Services Only);

Youth Ages 19-22 Program

(four indicators--same as for the Adult Program, with a variation in

the credentials indicator)

Entry into unsubsidized employment, measured by Entered

Employment Rate,

Retention in unsubsidized employment after entry into the

employment, measured by Six Month Retention Rate,

Earnings received in unsubsidized employment six months

after entry into employment, measured by Average Earnings Change in Six

Months,

Attainment of educational credential/occupational skills

credential for youth ages 19-22 entering post-secondary education,

advanced training, or employment after training, measured by

Educational Credential/Occupational Skills Credential Attainment Rate

(Training Services Only),

Youth Ages 14-18 Program

(three indicators)

Attainment of basic skills and, as appropriate, work

readiness or occupational skills, measured by Skill Attainment Rate,

Attainment of secondary school diplomas and their

recognized equivalents, measured by Diplomas and Equivalents Attainment

Rate,

Placement and retention in postsecondary education or

advanced training, or placement and retention in military service,

employment, or qualified apprenticeships, measured by Placement Rate;

Customer Satisfaction for Combined Programs

(two indicators)

Participant satisfaction, measured by an index derived

from several questions on customer satisfaction surveys,

Employer satisfaction, measured by an index derived from

several questions on customer satisfaction surveys.

2. The Department of Labor expects that negotiations will lead to

high State adjusted levels of performance to encourage high

performance.

3. Incentives will be awarded and sanctions will apply based on

performance against State adjusted levels of performance beginning in

the first year of WIA implementation. Only States that exceed their

agreed to levels will be eligible for incentive awards. Sanctions based

on first year performance will not include monetary penalties.

4. The Department of Labor will establish a single estimate of

National average performance for each performance measure. Initially,

because only limited historical outcome data are available to develop

information on likely outcomes for WIA core

[[Page 14344]]

performance indicators, the information will be subject to serious

qualifications. For some indicators, rough estimates of the

distribution of performance among States and simple adjustment models

will be developed as tools for reaching agreement on State adjusted

levels of performance. For other core performance indicators and the

customer satisfaction indicator, the available data will not be

sufficient at the outset to develop such tools. The Department expects

that future negotiations will be informed by discussions with and the

actual performance of the early implementing States.

5. A commitment to continuous improvement is not simply an

agreement to raise the State adjusted levels of performance for

successive years and incrementally improve performance numbers.

Continuous improvement is the process of building dynamic, high

achieving systems within every organization through the ongoing

systematic improvement of products, programs, services, and processes

by both small increments and major breakthroughs. Continuous

improvement encompasses a commitment to a systematic approach to high

performance. Continuous improvement is driven by finding opportunities

to do better, as well as by solving problems that need immediate

correction. It becomes a regular part of daily work, and provides a

method of eliminating problems at their source. Performance measures

and customer satisfaction are integrated into a continuous improvement

approach to focus on where to concentrate resources, or redesign

programs or sequences of services in order to achieve better results.

II. Principal Stages of Agreement Process

There are three principal stages of the process for reaching

agreement on State adjusted levels of performance in State five-year

plans. These stages are defined in terms of plan submittal and

approval. The first stage precedes submittal of the State plan and

includes the Department's provision of information, the State's

planning and development of an ``expected level of performance'' for

each of the prescribed indicators of performance and customer

satisfaction, and preliminary State/Federal discussions. The second

stage begins with the State's submittal of its plan including the

expected levels of performance that serve as the starting point for

negotiations between the State and Federal partners, and ends when the

State adjusted levels of performance are agreed to by the State

Governor and the Secretary of Labor and incorporated into the State

plan. The third stage follows approval of the plan with the agreed

levels and encompasses possible modification of the plan to revise the

State adjusted levels of performance.

A. Stage One: Federal and State Information and Preliminary Discussion

Before any meaningful activity can be accomplished relative to

planning or setting State performance goals, the State and Federal

partners will need an understanding about the process and its

relationship to other processes in the performance accountability

system, e.g., reporting, incentive and sanction policies, GPRA goals,

etc. Ideally, this information would become available before a State

engages in its strategic planning process or sets its ``expected levels

of performance'' and must be available before the negotiation process

begins. Both the State and the Federal partners must gather and assess

information prior to States' submittal of their plans.

1. Federal Information

The Department expects to release information in the following

areas at the earliest possible time. Each of these items will be

covered in papers to be developed and issued for comment.

Specific measures will be identified for the core

performance and customer satisfaction indicators. Definitions will be

provided for those measures, and will include the scope of the

measures, e.g., are they only Title I, all WIA referenced activities,

etc.

Information will be provided about performance measurement

tools under development, such as estimates of the distribution of

performance among States and simple adjustment models. The information

will include sources of data, the expected usefulness of information to

be developed, and for which measures, if any, there will be State-

specific estimates or adjustment models.

The process for negotiating the measures will be

established and communicated to the system. This will include

expectations for how and when the discussions will occur as well as the

kinds of information that must be available from the State to

facilitate the discussions.

Guidance will be issued about the levels that will be

considered acceptable when negotiating adjusted performance levels,

including specific information on:

--Policies that set criteria for evaluating expected levels of

performance;

--Policies for award of incentives and related concepts for meeting and

exceeding WIA Title I performance goals (note that this does not

include the overall policy for consideration of performance WIA Title

II and Carl D. Perkins Act programs); and

--Policies defining sanctions and related definitions for failure to

meet standards.

Guidance will be issued on determining the impact of

adjusted levels of performance on attaining high levels of customer

satisfaction.

Policy will be set to describe the consideration of

continuous improvement in the goal setting process.

Specific data and tools including models, if available,

will be provided for comparing the adjusted levels of performance among

States. Data and tools will continue to be released as they are

developed.

2. State Information

WIA envisions an accountability process that takes into

consideration unique State and local requirements and circumstances. As

States engage in the planning process and develop the expected levels

of performance they will identify in their State plans, they will

gather information that will be useful in the subsequent negotiation

process. States will obtain preliminary information about the economy,

anticipate characteristics of the population to be served, and set

strategies for determining service mix, since these must be considered

in setting performance levels. States will explore pertinent data

sources related to sequenced services and one-stop service systems and

examine their utility in establishing a baseline for the negotiations

process. States will develop information gleaned from an environmental

scan to determine the progress local areas have made in developing a

service delivery system as required in WIA. States will consider the

strength of State/local partnerships among agencies and organizations

that will support the system and strategies under consideration to

strengthen and streamline the delivery system. States are encouraged to

take into account, in developing their expected levels of performance,

the results of the negotiation of local performance levels.

3. Preliminary Discussion

The Department recognizes, with the States, that time is short for

development of State plans, including performance levels, particularly

for those States that expect to implement WIA in July 1999. The

Department appreciates and continues to encourage

[[Page 14345]]

State and local involvement as WIA policies and procedures are

developed. In this spirit of cooperation, preliminary discussion of

performance management, including the development of performance

levels, is welcomed. States are encouraged to contact their Regional

Offices for discussion and technical assistance prior to plan

submittal. The benefits of early discussion could include:

Ensuring understanding of guidance, policy, data, and

technical material provided by the Department prior to plan submittal;

Benefitting from the experience of regional staff and

other States;

Tailoring the provision of technical assistance on

performance accountability to meet local and regional planning needs;

Developing a mutual understanding of State and Federal

expectations and assumptions prior to plan submittal, to ensure

development of a shared set of goals;

Allowing maximum time (in advance of the up-to-ninety-days

plan review period) for States and local areas to complete necessary

planning and consultation on performance levels; and

Smoothing the agreement process.

B. Stage Two: Formal Discussion and Agreement

A State's submittal of its five-year plan to the Secretary of Labor

triggers the up-to-ninety-days review period during which the Federal

and State partners are to reach agreement on the State adjusted levels

of performance for the core and customer satisfaction indicators. The

agreed to levels will be incorporated into the plan prior to its

approval. A State's plan will not be approved if agreement has not been

reached. It is expected that the negotiations will take place between

the States and the Department's Regional Offices, consistent with

guidelines to be issued by the Department to the workforce development

system.

The State plan will include the State's expected level of

performance for each core indicator and customer satisfaction

indicator. The plan will also include an explanation of the derivation

of each expected level. In the formal negotiations, States should be

prepared to provide support for any data and data analysis used in

arriving at the expected levels of performance. States should also be

prepared to discuss any environmental or strategic issues that are

expected to influence performance levels.

In addition to the expected levels of performance identified in the

State plan for the first three years covered by the plan, WIA specifies

three factors that the Governor and the Secretary must take into

consideration in the agreement process. The first is the extent to

which the State adjusted levels of performance will help the State

achieve a high level of customer satisfaction. The second is how those

levels compare to the adjusted levels of other States, considering

differences in economic conditions, characteristics of participants

upon entry into WIA programs, and services to be provided. The third

factor is the extent to which the adjusted levels promote continuous

improvement in performance and ensure optimal return on investment. As

mentioned earlier, the Department expects to issue guidelines for

consideration of all of these factors, to ensure consistency and

fairness in the agreement process. The performance levels, representing

the anticipated results of the comprehensive workforce development

plan, will be considered in the context of the entire plan.

Particularly because of the anticipated limitations of available data

at the outset of WIA implementation, the negotiation process itself is

expected to be a learning experience for the State and Federal

partners.

C. Stage Three: Modifications, and Years Four and Five

WIA specifies that Governors may request revisions to State

adjusted levels of performance for any of the five program years

included in a State plan, based on unanticipated circumstances in their

respective States that result in significant changes in factors

including economic conditions, characteristics of participants, and

services provided. WIA does not prohibit consideration of other

factors. The Secretary will issue guidelines establishing objective

criteria and methods for making revisions requested by Governors. These

guidelines also will specify the conditions under which a State is

required to revise the agreed to levels of performance. The revision

process will be addressed in a separate paper which is expected to be

issued for comment in late April 1999.

Because of the transitional nature of the program for the first

three program years and the lack of data from which predictions of WIA

performance can be derived for each State, there must be allowance for

changes in expected performance beyond the circumstances specified in

WIA. Allowance for changes in performance expectations is particularly

important because a State's performance measured against its State

adjusted levels of performance will affect its eligibility for

incentive grants and its susceptibility to sanctions. As the body of

WIA experience grows--over time in individual States and as more States

implement WIA--more information will become available to permit

development of more useful performance management tools, including

National figures, State-specific information, distributions of

performance data across States, and adjustment models. The effects of

continuous improvement approaches will be better understood and more

predictable as their application is tested. Because of these

anticipated changes, it is expected that State adjusted performance

levels included at the beginning of a State's five-year plan will be

able to be refined as time passes.

Federal guidance will delineate circumstances in which the State

adjusted levels of performance must or may be revised--upward or

downward--for individual States or for all States. Some possibilities

beyond those identified in the law are listed here.

Performance levels are set for all States based on the

pre-WIA wage record experience of a few States, and experience shows

that the predictions were not valid.

The operation of the one-stop system in a State varies

significantly from that discussed during performance negotiations.

Changes in State law, Statewide vision, or strategies have

a significant impact on performance outcomes.

Changes in Federal law or policy have a significant impact

on performance outcomes.

WIA requires that agreement be reached on State adjusted levels of

performance for the fourth and fifth program years covered by a five-

year plan prior to the beginning of the fourth year. The State does not

submit expected levels of performance for those years. The law seems to

contemplate that experience under WIA in the first three years will

provide a sufficient basis for setting levels for subsequent

performance.

[FR Doc. 99-7148 Filed 3-23-99; 8:45 am]

BILLING CODE 4510-30-P