Transportation Equity Act for the 21st Century; Guidelines for the Evaluation of Operational Tests and Deployment Projects for Intelligent Transportation Systems (ITS)

DEPARTMENT OF TRANSPORTATION

Federal Highway Administration

Federal Transit Administration

Transportation Equity Act for the 21st Century; Guidelines for

the Evaluation of Operational Tests and Deployment Projects for

Intelligent Transportation Systems (ITS)

AGENCIES: Federal Highway Administration (FHWA), Federal Transit

Administration (FTA), Department of Transportation (DOT).

ACTION: Notice.

-----------------------------------------------------------------------

SUMMARY: This document provides implementation guidance for section

5204(j)(1) of the Transportation Equity Act for the 21st Century (TEA-

21) 23 U.S.C 502 Note. Guidelines and requirements for the evaluation

of

[[Page 50855]]

operational tests and deployment projects for ITS ensure the

objectivity and independence of project evaluators to avoid any real or

apparent conflict of interest or potential influence on the outcome by

parties to such tests and projects. They also establish evaluation

funding levels based on the size and scope of each test or project to

ensure adequate evaluation. The ITS Joint Program Office (ITS JPO)

plans to disseminate the TEA-21 Evaluation Guidelines to affected

program offices within the DOT. Should it be deemed necessary to

establish any requirements for the evaluation of operational tests and/

or deployment projects, these would be established pursuant to

rulemaking to be issued in the future.

FOR FURTHER INFORMATION CONTACT: For technical information: Mr. Joe

Peters, (202) 366-2202, ITS Joint Program Office, (HOIT-1) FHWA. For

legal information: Mr. Wilbert Baccus, Office of the Chief Counsel,

FHWA (HCC-32), (202) 366-0780; Ms. Linda Sorkin, Office of the Chief

Counsel, FTA (TCC-24), (202) 366-1936. All are located at the U.S.

Department of Transportation, 400 Seventh Street, SW., Washington, DC

20590. Office hours are from 8 a.m. to 4:30 p.m., Monday through

Friday, except Federal holidays.

SUPPLEMENTARY INFORMATION:

Electronic Access

An electronic copy of the document may be downloaded using a modem

and suitable communications software from the Government Printing

Office's Electronic Bulletin Board Service at (202) 512-1661. Internet

users may reach the Federal Register's home page at: http://

www.nara.gov/fedreg and the Government Printing Office's database at:

http://www.access.gpo.gov/nara. The TEA-21 Evaluation Guidelines may

also be accessed at the U.S. DOT's ITS home page, through the

evaluation link, at http://www.its.dot.gov.

Background

Section 5204(j)(1) of TEA-21, Pub. L. 105-178, 112 Stat. 107, 455

(1998), states that the Secretary of Transportation (Secretary) shall

issue guidelines and requirements for the evaluation of operational

tests and deployment projects for the ITS. This document sets forth

TEA-21 Guidelines that are required to include provisions to ensure

objectivity and independence of evaluators so as to avoid real or

apparent conflicts of interest on the outcome of project evaluations.

The guidelines are further required to establish evaluation funding

levels based on the size and scope of each test or project to ensure

adequate project or operational test evaluations. Should it be deemed

necessary to establish any requirements for the evaluation of

operational tests and/or deployment projects, these would be

established pursuant to rulemaking to be issued in the future.

The objective of the TEA-21 Evaluation Guidelines is to fulfill the

requirements imposed on the Secretary in the referenced section by

accomplishing the following:

1. Defining the different categories of projects carried out under

subtitle C;

2. Defining, in general terms, different types of evaluations to be

conducted by projects in the categories defined;

3. Establishing criteria to guide the selection of evaluations to

be performed;

4. Defining, in general terms, procedures for ensuring objectivity

and independence of evaluating organizations;

5. Defining the funding mechanism to provide project evaluation

resources;

6. Providing a general description of the procedures [and

requirements] that project partnerships can expect when participating

in the types of evaluations defined;

7. Providing access through web site address to detailed evaluation

procedures to include examples of specific evaluation plans, test

plans, and reports.

The TEA-21 Evaluation Guidelines are published in the Federal

Register for informational purposes on our approach to satisfying the

requirements prescribed in section 5204(j)(1). Specific questions on

any of the material published in this notice should be directed to the

appropriate contact person named in the caption FOR FURTHER INFORMATION

CONTACT.

Authority: 23 U.S.C. 315; sec. 5204(j), Pub. L. 105-178, 112 Stat.

455 (1998) 23 U.S.C. 502 Note.

Issued on: September 13, 1999.

Kenneth R. Wykle,

Federal Highway Administrator.

Gordon J. Linton,

Federal Transit Administrator.

The text of the TEA-21 Evaluation Guidelines is presented as

follows:

United States Department of Transportation

Intelligent Transportation Systems

Evaluation Guidelines for the Transportation Equity Act for the 21st

Century

Operational Tests and Deployment Projects

I. Introduction

A. Background

The enactment of the Transportation Equity Act for the 21st Century

(TEA-21) [Pub. L. 105-178, 112 Stat. 107 (1998)] has expanded the focus

of the Intelligent Transportation Systems (ITS) program from one of

research and operational tests to one that now includes deployment of

ITS across the Nation. Subtitle C (Intelligent Transportation Systems

Act of 1998) under title V of TEA-21 authorizes investment to

accelerate the rate at which intelligent transportation systems are

incorporated in the Nation's surface transportation network to improve

transportation safety and efficiency, and to reduce costs and negative

impacts on communities and the environment. The effective

implementation of the deployment and integration activities specified

in subtitle C will rely heavily on active and rigorous evaluations of

those activities. Recognizing the evolving nature of the ITS program,

Congress included a mandate in TEA-21 which requires the Secretary of

Transportation (Secretary) to issue guidelines and requirements for the

evaluation of operational tests and deployment projects carried out

under the program. A prerequisite for the continuing support of

decision makers addressing policy and investment issues will be a clear

understanding of ITS system effectiveness. At the Federal level, the

Government Performance and Results Act [Pub. L. 103-62, 107 Stat. 285

(1993)] has established a formal process for program and budget

planning. ITS benefits information will be needed for assessing the

efficacy of Federal investment in ITS. At the project level, agencies

and partners in public-private project initiatives require the means to

monitor costs and system effectiveness on a continuing basis to support

management of systems and operations.

B. Legislative Requirements

The Congress has recognized the critical role of ITS evaluation in

section 5204(j)(1)(A) of subtitle C which prescribes that:

Subparagraph (A):

In General.--The Secretary shall issue guidelines and requirements

for the evaluation of operational tests and deployment projects carried

out under this subtitle.

Subparagraph (B):

Objectivity and Independence.--The guidelines and requirements

issued under Subparagraph (A) shall include provisions to ensure the

objectivity and

[[Page 50856]]

independence of the evaluator so as to avoid any real or apparent

conflict of interest or potential influence on the outcome by parties

to any such test or deployment project or by any other formal

evaluation carried out under this subtitle.

Subparagraph (C):

Funding.--The guidelines and requirements issued under Subparagraph

(A) shall establish evaluation funding levels based on the size and

scope of each test or project to ensure adequate evaluation of the

results of the test or project.

C. Objective

The objective of this document is to fulfill the ``guidelines''

requirement imposed on the Secretary by TEA-21, title V, subtitle C,

section 5204(j)(1)(A), (B), and (C) as referenced above.

D. Purposes of Evaluation Guidelines

In order to satisfy the applicable title V subtitle C criterion

imposed by TEA-21, guidelines must be issued which accomplish the

following purposes:

1. Define different categories of projects carried out under

subtitle C;

2. Define, in general terms, different types of evaluations to be

conducted by projects in the categories defined;

3. Establish criteria to guide the selection of evaluations to be

performed;

4. Define, in general terms, procedures for ensuring objectivity

and independence of evaluating organizations;

5. Define the funding mechanism to provide resources for

evaluations;

6. Provide a general description of the procedures [and

requirements] that project partnerships can expect when participating

in the types of evaluations defined;

7. Provide access through web site address to detailed evaluation

procedures including examples of specific evaluation plans and reports.

Guidelines in this document do not address detailed evaluation

procedures.

II. Terms of Reference

The establishment of clearly understood guidelines for evaluation

of projects funded with Federal ITS funding requires identification of

the categories of projects specified in the law. Similarly,

understanding of the Department's concept for conducting project

evaluations requires differentiation between types of evaluation.

A. Project Categorization

Projects to be carried out with Federal ITS funds are identified as

follows:

1. Operational Tests--Section 5207(c) specifies the conduct of

operational tests of intelligent vehicles and intelligent

infrastructure systems. Section 5207(c) further specifies that

operational tests conducted under this section shall be designed for

the collection of data to permit objective evaluation of the results of

the tests, derivation of cost-benefit information that is useful to

others contemplating deployment of similar systems, and the development

and implementation of standards.

2. ITS Deployment Program--The ITS Deployment Program provides

funding for two major project categories specified in sections 5208 and

5209. The two components are the ITS Integration Program and the

Commercial Vehicle Intelligent Transportation Infrastructure Deployment

Program, commonly known as the Commercial Vehicle Information Systems

and Networks (CVISN) Program.

a. Intelligent Transportation System Integration Program--Section

5208(a) specifies projects to be carried out under the Intelligent

Transportation System Integration Program in metropolitan and rural

areas. These projects comprise one major component of the ITS

Deployment Program defined above. The ITS Integration Program provides

Federal funding for the integration of multi-modal ITS components in a

variety of settings, including large regional areas (for example,

Statewide, multi-State, or multi-city), metropolitan areas, non-

metropolitan areas, and rural areas. ITS integration projects should

improve transportation efficiency; promote safety; enhance transit

integration; improve paratransit/demand-responsive transit operations,

including operations of health and human services providers; improve

traffic flow, including the flow of intermodal freight at ports of

entry; reduce emissions of air pollutants; improve traveler

information; promote tourism; enhance alternative transportation modes;

or support improved transportation systems operations, management and

maintenance.

As part of the Intelligent Transportation System Integration

Program, section 5208(g), Corridor Deployment and Coordination,

requires the Secretary to encourage multistate cooperative agreements,

coalitions, or other arrangements intended to promote regional

cooperation, planning, and shared project implementation for

intelligent transportation system projects. There are two areas of

implementation: Great Lakes ITS Implementation and Northeast ITS

Implementation.

(a) Great Lakes Implementation--Section 5208(g)(2)(A): The

Secretary is required to make grants under this subsection to the State

of Wisconsin to continue ITS activities in the corridor serving the

Greater Milwaukee, Wisconsin; Chicago, Illinois; and Gary, Indiana,

areas initiated under the Intermodal Surface Transportation Efficiency

Act of 1991 (ISTEA) [Pub. L. 102-240, 105 Stat. 1914 (1991)] and other

areas in the State.

(b) Northeast ITS Implementation--Section 5208(g)(3)(A): The

Secretary is required to make grants under this subsection to the

states to continue ITS activities in the Interstate Route I-95 Corridor

in the northeastern United States initiated under ISTEA.

b. Commercial Vehicle Intelligent Transportation System

Infrastructure Deployment--Section 5209 specifies that the Secretary

shall carry out a comprehensive program to deploy intelligent

transportation systems that improve safety and productivity of

commercial vehicles and drivers, while reducing costs of commercial

vehicle operations and regulatory requirements. The term ``Commercial

Vehicle Information Systems and Networks'' is defined (section 5211) as

the information systems and communications networks that support

commercial vehicle operations. Funding in section 5209 is used to

support CVISN projects which comprise the second major component of the

ITS Deployment Program defined above. CVISN objectives focus on

advancing technological capabilities and promoting the deployment of

intelligent transportation system applications to commercial vehicle

operations, including commercial vehicle, commercial driver, and

carrier-specific information systems and networks. Priority areas for

projects are those that encourage multi-state cooperation and corridor

development; improve the safety of commercial vehicle operations;

increase the efficiency of regulatory inspection processes to reduce

administrative burdens; improve the efficiency of enforcement efforts;

advance electronic processing of registration, driver licensing and

fuel tax information, inspection and crash data; promote communication

of information among the States; and, enhance safe passage of

commercial vehicles across international borders.

B. Evaluation Categorization

For the purpose of the guidelines, there are two categories of

project evaluations.

[[Page 50857]]

1. Self-Evaluations--In the interests of accepted sound management

practice, it is expected that ITS Deployment Program participants

should conduct locally executed and funded evaluations under the

auspices of the project partners. These self-evaluations, also

identified as local evaluations, incorporate certain minimum evaluation

and reporting requirements. Cross-cutting assessments of these local

evaluations will be conducted by the ITS Joint Program Office (ITS JPO)

at Headquarters (HQ), U.S. DOT, and will include gathering data and

dissemination of results.

2. National Evaluations--National evaluations are formal, in-depth,

independently conducted evaluations of operational tests of intelligent

infrastructure systems and selected projects carried out under the ITS

Deployment Program. In the case of selected ITS Deployment Program

projects, these evaluations will supplement and expand on the

activities of self-evaluations. The added resources available for

national evaluations will facilitate assessment of quantitative

performance measures and other aspects of project evaluation difficult

to pursue with the potentially limited resources assigned for self-

evaluations. National evaluations will be conducted under the auspices

of U.S. DOT, and will be closely monitored by a designated U.S. DOT

representative. The U.S. DOT has established the Intelligent Vehicle

Initiative (IVI) to improve the safety and efficiency of motor vehicle

operations by reducing the probability of motor vehicle crashes.

Evaluations of IVI field operational tests will be conducted under the

auspices of the IVI Program Manager in accordance with procedures

defined within the IVI Program and not associated with requirements for

national evaluations discussed in these guidelines.

Features and requirements of these categories are described below

in these guidelines.

III. Concept for Project Selection and Evaluation Funding

A. National Evaluations

1. Operational Tests--All operational tests of intelligent

infrastructure systems carried out under section 5207 shall be

administered formal, national evaluations under the auspices of the ITS

JPO Program Assessment Coordinator.

2. ITS Deployment Program--Projects carried out under the ITS

Deployment Program will be reviewed as follows:

(a) ITS Integration Program--Projects selected for funding under

section 5208 shall be reviewed, and a limited number will be selected

for national evaluations in accordance with appropriate criteria

described in these guidelines.

Corridor Development and Coordination--Projects resulting

from grants made to the State of Wisconsin to continue activities in

the corridor serving Milwaukee, Wisconsin; Chicago, Illinois; and Gary,

Indiana, in accordance with section 5208(g)(2)(A) and projects

resulting from grants made to the I-95 Northeast ITS Priority

Corridors, in accordance with section 5208(g)(3)(A), may be reviewed

for incorporation in a national evaluation as determined by the

Department's evaluation needs in corridor settings.

(b) Commercial Vehicle ITS Infrastructure Deployment--A limited

number of CVISN deployment sites funded under section 5209 will be

selected for national evaluations in accordance with appropriate

criteria described in these guidelines.

B. Self-Evaluations

All project partnerships are expected to perform locally conducted

self-evaluations.

C. Funding for Project Evaluations

In order to fulfill the mandate specified in section 5204(j)(1)(C)

that these guidelines ``shall establish evaluation funding levels based

on the size and scope of each test or project that ensure adequate

evaluation of the results of the test or project,'' the following

procedures apply:

1. Operational Tests--Funding for evaluations of operational tests

of intelligent infrastructure systems will be provided by the ITS JPO.

2. ITS Deployment Program--(a) Projects funded under the

Intelligent Transportation System Integration Program (section 5208)

and the Commercial Vehicle Intelligent Transportation System

Infrastructure Deployment Program (section 5209) selected for national

evaluations will use a pooled funding mechanism. During each year

authorized by TEA-21, two percent of the amount authorized for the ITS

Deployment Program will be placed into a deployment evaluation fund.

National evaluations for selected projects will be funded by this

account.

(b) All projects will fund locally conducted self-evaluations from

project resources.

D. Selection Criteria for National Evaluation

1. Intelligent Transportation System Integration Program

Projects carried out under section 5208 are designed to accelerate

the integration and interoperability of intelligent transportation

systems in metropolitan and rural areas. Projects resulting from

corridor grants may also be reviewed for inclusion in a national

evaluation. Projects selected for funding should support the goals of

the program as defined in section 5208.

Projects selected for funding will be reviewed by the ITS Program

Assessment Working Group (ITS PAWG) chaired by the ITS JPO Program

Assessment Coordinator. The ITS PAWG is comprised of representatives

from the modal administrations at HQ, U.S. DOT, and field

representatives from the Federal Highway Administration (FHWA) Resource

Centers and Division Offices, to include the Office of Technology

Evaluation and Deployment within the Office of Motor Carrier and

Highway Safety. Federal Transit Administration field representation is

included as well.

The ITS PAWG will consider projects for national evaluation through

successive iterations designed to narrow the list of candidate projects

consistent with available funding and the ITS Program's most compelling

information needs. While the specific considerations for project

selection may vary in any given year of TEA-21 authorization, the

criteria described below will guide the review process.

a. Major consideration will be given to ITS areas for which

insufficient knowledge was developed during the operational test

experience under ISTEA authorization.

Metropolitan ITS Integration Program projects will be assessed for

their potential to provide benefits-related data in areas where

evaluation data are still needed.

b. Rural ITS Integration Program projects will be assessed for

their potential to provide benefits-related data.

It is anticipated that, over the life of TEA-21, the areas

reflecting shortfalls in ITS Program evaluation needs will vary. The

ITS Program reserves the flexibility to develop a working document with

annual updates of these areas of shortfall which would be used to guide

the selection process. These areas will be addressed in annual

Departmental project guidance documents.

2. CVISN Deployment Projects

The FHWA's State CVISN Level 1 deployment strategy consists of

three key steps: Planning; Design; and, Implementation and Deployment.

Step 1, Planning, includes participation in two ITS/Commercial

Vehicle Operations (ITS/CVO) training

[[Page 50858]]

courses (Introduction to ITS/CVO and ITS/CVO Technical Project

Management for Non-Technical Managers) and the development of an ITS/

CVO State business plan. This step is essential to promote ITS/CVO

awareness and coalition building among the State agencies involved in

CVO and with industry.

The focus of Step 2, Design, is for the State to establish its

CVISN project team, including at a minimum a CVISN project manager and

a system architect. Once these individuals have been selected, a State

can participate in the Understanding ITS/CVO Technology training course

and in three CVISN workshops. These activities will assist the State in

developing its CVISN Project Plan and Top-Level Design.

Step 3 is the Implementation and Deployment of CVISN Level 1

capabilities.

Only CVISN sites selected for Step 3 funding will be considered for

national evaluation. The three elements of Level 1 capabilities subject

to national evaluation are:

Safety Information Exchange,

Credentials Administration, and

Electronic Screening.

During each year of TEA-21 authorization, the ITS PAWG will convene

its CVO Subcommittee comprised of subject matter experts from FHWA's

Office of Technology Evaluation and Deployment within the Office of

Motor Carrier and Highway Safety. Based on the ITS/CVO Program's

information needs about Level 1 capabilities in different operating

environments, a limited number of State CVISN deployment sites engaged

in Step 3 (Implementation and Deployment) activities will be selected

for national evaluation.

IV. Selection of Evaluating Organizations

A. General

Subtitle C, section 5204(j)(1)(B) requires that these guidelines

include provisions to ensure objectivity and independence of the

evaluator so as to avoid any real or apparent conflict of interest or

potential influence on the outcome by parties to any such test or

deployment project or by any other formal evaluation carried out under

this subtitle.

B. Provisions for National Evaluations

The required provisions are outlined as follows:

1. Organizations demonstrating technical qualifications are

eligible for selection to perform a national evaluation of designated

operational tests or deployment projects.

2. In the process of submitting necessary documentation

demonstrating qualifications to perform a national evaluation,

interested organizations should include a statement certifying

commitment to the conduct of a completely objective and independent

evaluation, and its commitment to ensuring any subcontracting

organizations will adhere to that standard. It is expected that this

would include a provision for voluntary recusal from evaluating

selected aspects of, or technologies involved in, the project to be

evaluated, if appropriate.

3. For national evaluations conducted under the auspices of U.S.

DOT, the ITS JPO Program Assessment Coordinator will convene selected

members of the ITS PAWG to participate in the selection of an

evaluating organization with consultations as follows:

a. Operational Tests--For operational tests of intelligent

infrastructure systems, the ITS JPO Program Assessment Coordinator, in

consultation with project partners and the ITS PAWG, will select the

evaluating organization. For IVI field operational tests, the selection

of organizations will be conducted in accordance with procedures

established by the IVI Program Manager.

b. ITS Deployment Program--Participants in the evaluating

organization selection process are as follows:

(1) Intelligent Transportation System Integration Program--For

projects in this category selected for national evaluations, the ITS

JPO Program Assessment Coordinator in consultation with project

partners will conduct a selection process. The selected evaluator will

receive direction and oversight by a U.S. DOT representative, and will

be required to document compliance with the ``OBJECTIVITY and

INDEPENDENCE'' requirements imposed by section 5204(j)(1)(B) of

subtitle C, and will forward authenticated certifications described in

IV.B.2 of these guidelines to the designated U.S. DOT representative.

This representative, in coordination with the ITS JPO Program

Assessment Coordinator, will conduct an expeditious review of selected

organizations prior to completion of contract arrangements.

(2) CVISN Project Evaluations--The selection of an evaluation

organization for the conduct of national evaluations of selected CVISN

project sites will be led by the ITS JPO Program Assessment Coordinator

and the ITS PAWG representative from the Office of Technology

Evaluation and Deployment in the Office of Motor Carrier and Highway

Safety who will convene a CVO subcommittee of the ITS PAWG. This

subcommittee will be comprised of the ITS JPO Commercial Vehicle

Operations Coordinator and subject matter experts on the staff of the

Office of Motor Carrier and Highway Safety, HQ, FHWA.

C. Provisions for Self-Evaluations

The guidelines for the conduct of self-evaluations in ITS

Deployment Program projects include the following: (1) The evaluations

should be conducted under the auspices of the project partners; (2) The

partners should form evaluation organizations; and, (3) The process for

forming evaluation organizations should strive to adhere to the

objectivity and independence principles cited in section 5204(j)(1)(B)

of subtitle C.

V. Evaluation--Definition, Process, and Guidelines

A. General--The purposes of this section are to accomplish the

following:

Define evaluation as a management process;

In general terms, differentiate between procedures and

expectations to be encountered in national evaluations and self-

evaluations;

Convey reporting requirements.

These guidelines are not intended to provide detailed procedures

for the conduct of evaluations. Extensive documentation addressing

detailed procedures, to include examples of evaluation plans and

reports will be available at the ITS JPO web site. (See Section V.G.).

B. Definition of Evaluation

Evaluation is the reasoned consideration of how well project goals

and objectives are being achieved. The primary purpose of evaluation is

to cause changes in the project so that it eventually meets or exceeds

its goals and objectives. Evaluation is an essential ingredient to good

project management. Evaluations can be qualitative and quantitative;

however, the best evaluations employ the combination of qualitative and

quantitative information that compare and contrast converging, non-

converging, and diverging evidence to result in a complete diagnosis.

The most effective evaluations occur when goals and objectives are

explicitly stated, are measurable, and are agreed to by all project

partners.

Evaluation should be considered as part of the project development

process that iterates across stages of strategy

[[Page 50859]]

formulation, detailed planning, system design and implementation, data

collection, data analysis, and reporting of results. Evaluations should

be performed by a party who has had no vested interest or stake in the

project itself.

Independence of the evaluator does not mean uninvolvement with the

project. Key roles of the evaluator requiring early evaluator

involvement are: (1) Identification of key stakeholder partners; (2)

eliciting from the partners a meaningful set of goals and objectives

for the project and their relative priorities; (3) obtaining insight

and consensus regarding which measures will best reflect the degree of

success in achieving prioritized goals; and, (4) communicating changes

in goals, objectives, and measures as the project evaluation matures.

Data can be collected either by the partners in the project (as long as

the independent evaluator maintains some oversight of the process) or

by the independent evaluator, or by both. The data analysis phase must

be performed completely independently of the partners; however, draft

results should be shared to obtain partners' expert insights regarding

possible flaws in assumptions or errors in analysis. The best

evaluation results are those that can bring about positive and timely

changes in system operations or maintenance. In such instances,

documentation of results may not be as important as the positive

changes influenced by the results. Nevertheless, documentation has

tremendous importance to the national interest.

A significant goal influencing the approach to ITS Program

evaluation planning is the expansion of the knowledge base among

transportation community professionals. To the extent that policy

makers, planners, engineers, and other influence brokers become better

informed about successful ITS deployment practices, project evaluations

will make significant contributions. Effective information

dissemination techniques will be critical. Information drawn from

interim and final evaluation reports will be summarized in electronic

media normally accessed by transportation professionals.

Project evaluation reports will be crucial to the ITS Program

during the years funded by TEA-21. As the tempo of project deployments

accelerates, newly formed project partnerships preparing to undertake

deployments will need any available experience-based information

relevant to their projects. The need for timely information will

emphasize the value of interim reports capturing valuable, recent,

lessons learned.

C. General Evaluation Process

The following are general steps successfully used to accomplish an

ITS project evaluation for national or local evaluations:

1. Form the Evaluation Team. Each of the project partners and

stakeholders designates one member to participate on the evaluation

team. The program manager should designate an evaluation team leader.

In the interests of conducting an effective evaluation, this team

should interact with the independent evaluator periodically throughout

the project development and deployment. Experience has demonstrated

that formation of this team early in the project is essential to

facilitating evaluation planning along a ``no surprises'' path.

Participation by every project stakeholder is particularly crucial

during the development of the ``Evaluation Strategy.''

2. Develop the Evaluation Strategy. This evaluation strategy

document includes a description of the project to be evaluated and

identifies the key stakeholders committed to the success of the

project. It also relates the purpose of the project to the general goal

areas of an ITS project.

Projects deploying intelligent metropolitan or rural infrastructure

are expected to allocate resources adequate for evaluating the impact

(or impacts) their projects exert in certain major goal areas which can

be pursued through deploying and integrating ITS technologies. ITS goal

areas include:

Traveler Safety

Traveler Mobility

Transportation System Efficiency

Productivity of Transportation Providers

Conservation of Energy and Protection of the Environment

Others as may be appropriate to unique features of a project

A major purpose of the evaluation strategy document is to focus

partner attention on identifying which of the above goal areas has

priority for their project. Partners assign ratings of importance to

goal areas and evaluation priorities and resources are consequently

aligned to the prioritized set. This rating process gives partners

valuable insights regarding areas of agreement and disagreement and

assists in reconciling differences and bolstering common causes.

Each of these goal areas can be associated with outcomes of

deployment that lend themselves to measurement. These outcomes

resulting from project deployment are identified as measures and have

been adopted as useful metrics. The association of goal areas and

measures is depicted as follows:

------------------------------------------------------------------------

Goal area Measure

------------------------------------------------------------------------

Safety....................... Reduction in the Overall Rate of

Crashes.

Reduction in the Rate of Crashes

Resulting in Fatalities.

Reduction in the Rate of Crashes

Resulting in Injuries.

Mobility..................... Reduction in Delay.

Reduction in Transit Time

Variability.

Improvement in Customer

Satisfaction.

Efficiency................... Increases in Freeway and

Arterial Throughput or Effective

Capacity*.

Productivity................. Cost Savings.

Energy and Environment....... Decrease in Emissions Levels.

Decrease in Energy Consumption.

------------------------------------------------------------------------

*A discussion of the distinction between Throughput and Effective

Capacity is included in the ITS Evaluation Resource Guide referred to

in Section V.G. at the end of this document.

The ``few good measures'' in the preceding table constitute the

framework of benefits expected to result from deploying and integrating

ITS technologies. While each project partnership will establish its

unique evaluation goals, the measures serve to maintain the focus of

goal setting on how the project can contribute to reaping the benefits

of one or more of the measures. A sample Evaluation Strategy document

is provided as part of the ITS Evaluation Resource Guide (see Section

V.G. at the end of this document).

[[Page 50860]]

3. Develop the Evaluation Plan. After the goals are identified and

priorities are set by the partners, the evaluation plan should refine

the evaluation approach by formulating hypotheses. Hypotheses are

merely ``if-then'' statements about expected outcomes after the project

is deployed. For example, a possible goal of coordinating

jurisdictions' signal systems is improving safety by reducing rear-end

crashes. If the evaluation strategy included this goal, the evaluation

plan would formulate hypotheses that could be tested. In this case, one

hypothesis might be, ``If jurisdictions coordinate signal timing, then

rear-end collisions will be significantly reduced at intersections at

jurisdictional boundaries.'' An even more aggressive hypothesis might

suggest that such collisions would be reduced by ten percent. The

evaluation plan identifies all such hypotheses and then outlines the

number of different tests that might be needed to test all hypotheses.

In addition to hypotheses regarding system and subsystem

performance, the evaluation plan identifies qualitative studies that

will be performed. The evaluation should address key components of the

project, such as, (but not limited to):

Implications of achieving consistency with the National

ITS Architecture;

Standards implementation;

Consumer acceptance;

Others as appropriate to local considerations;

Institutional issues.

Institutional issues require brief elaboration. An area of special

emphasis in all evaluation endeavors should be the non-technical

factors influencing project performance. ITS projects conducted under

ISTEA were profoundly influenced by considerations such as procurement

practices, contracting policy, organizational structure, and

relationships among major participants such as prime contractors and

their subcontractors. The transportation community stands to reap

significant benefit from understanding how the varied range of non-

technical factors impacts directly on traditional project performance

parameters, such as, cost, schedule, and final functionality.

As these critical aspects of the project are addressed, the value

of the evaluation increases in proportion to its ability to produce

lessons learned that can improve project performance. Thus, the

measures serve as the foundations of the evaluation as project

implementation seeks the best mix of approaches to ensure achieving

some level of benefits described by the measures.

To the extent that projects define evaluation goals derived from

one or more of the few good measures and document the impacts of

deployment on transportation performance in their communities, they

will have made significant contributions to the ITS Program and the

Nation. A sample Evaluation Plan document is provided as part of the

ITS Evaluation Resource Guide (see Section V.G. at the end of this

document).

4. Develop one or more Test Plans. A test plan will be needed for

each test identified in the evaluation plan. A test plan lays out all

of the details regarding how the test will be conducted. It identifies

the number of evaluator personnel, equipment and supplies, procedures,

schedule, and resources required to complete the test. A sample test

plan is provided as part of the ITS Evaluation Resource Guide (see

Section V.G. at the end of this document).

5. Collect and analyze data and information. This step is the

implementation of each test plan. It is in this phase where careful

cooperation between partners and evaluators can save money. By early

planning, it is possible to build into the ITS project capabilities for

automatic data collection. Such data collection can be used by partners

after the evaluation is completed to provide valuable feedback with

regard to the performance of the system. Such feedback can help in

detecting system failures and to improve system performance.

6. Document strategy, plans, results, conclusions, and

recommendations in a Final Report. A sample Final Report document is

provided as part of the ITS Evaluation Resource Guide (see Section V.G.

at the end of this document).

D. Self-Evaluation Process

1. ITS Integration Program Evaluation Procedures

To the extent that resources allow, project partners should strive

to follow the General Evaluation Process outlined in Section V.C. of

this document. Partners should also utilize the ITS Evaluation Resource

Guide described in Section V.G. at the end of this document.

Projects conducting self-evaluation should respond to a minimum of

two reporting requirements.

a. In an attempt to satisfy significant and critical data needs in

the ITS Program, projects should collect and document cost accounting

data to include acquisition, life-cycle, and operations and maintenance

costs capturing both start-up and sustaining cost factors. Reporting of

cost data will be solicited annually for the duration of the deployed

system's (or systems'') life cycle(s). Cost data collection guidelines

are provided as part of the ITS Evaluation Resource Guide (see Section

V.G. at the end of this document).

b. Additionally, self-evaluations should result in one or more of

the following efforts:

Evaluating institutional issues associated with achieving

cooperation among public sector agencies and documenting how they were

overcome.

Providing a brief lessons learned report on the technical

and institutional issues encountered in integrating ITS components.

Providing an evaluation report on the lessons learned in

employing innovative financing or procurement and/or public-private

partnering techniques.

Producing a lessons learned report on the experiences,

challenges, and approaches used in achieving consistency with the

National ITS Architecture and/or implementation of ITS standards.

Producing a case study on the planning process used to

achieve integration into an approved plan and program developed under

an area-wide (statewide and/or metropolitan) planning process which

also complies with applicable State air quality implementation plans.

Providing the appropriate metropolitan planning process

with data generated by ITS technologies and services, and provide a

report on plans or intentions for archiving the data and using it.

2. Commercial Vehicle Intelligent Transportation System Infrastructure

Deployment Projects

All projects participating in the Commercial Vehicle Intelligent

Transportation System Infrastructure Deployment Program are expected to

accomplish the following:

Document lessons learned in the areas of:

--institutional issues

--technical challenges

--employment of innovating financing

--public-private partnering

--achieving consistency with the National ITS Architecture and

implementation of ITS Standards.

Document benefits data.

Collect cost data, especially with regard to operations and

maintenance cost factors, and briefly address their implications for

sustaining the project.

[[Page 50861]]

E. National Evaluations

1. Initial Procedures for Operational Tests of Intelligent

Infrastructure

Upon selection of the organization to conduct a national evaluation

of a designated ITS operational test of intelligent infrastructure, the

ITS JPO Program Assessment Coordinator will establish the initial

communications between involved parties. The detailed procedures for

the conduct of the evaluation and the scope of the evaluating

organization's tasks will be defined in accordance with procedures

established by the ITS JPO Program Assessment Coordinator.

2. Requirements for ITS Integration Program Projects

During the annual project definition and proposal process,

participants in the ITS Integration Program will be offered the

opportunity to commit to cooperate with evaluators in the event of

selection for a national evaluation. This commitment includes

participation in evaluation planning and in-progress reviews to ensure

a consensus-based, successfully implemented national evaluation as

described in sections V.B. and V.C. of these guidelines.

3. Requirements for CVISN Deployment Projects

During the annual project definition and proposal process,

participants in the Commercial Vehicle Intelligent Transportation

System Infrastructure Program will be offered the opportunity to commit

to cooperate with evaluators in the event of selection for a national

evaluation. This commitment includes participation in evaluation

planning and in-progress reviews to ensure consensus-based,

successfully implemented national evaluations.

4. Timing Considerations for ITS Deployment Program Projects Selected

for National Evaluations

Participants in ITS Deployment Program projects selected for

national evaluations may experience a time delay between receipt of

notification for project funding and, in the event of selection for

national evaluation, notification of such selection. Upon notification

of project funding approval, the project participants should proceed

with the preparatory steps required for evaluation. The preparatory

measures will lay the foundation for an effective self-evaluation. In

the event of selection for a national evaluation, that process will

build on this foundation. An evaluation team should be formed and an

evaluation strategy, based upon the example in the ITS Evaluation

Resource Guide, should be developed.

It is anticipated that U.S. DOT notification of selection for

national evaluation will be accomplished prior to a project's

development of an evaluation plan. This will facilitate coordination

between the independent evaluating organization and the project

partners in proceeding with the development of a consensus-based

evaluation plan.

F. Reporting Requirements

This section prescribes reporting procedures for the categories of

evaluations.

1. Projects conducting self-evaluations in the ITS Integration

Program are expected to produce: (1) an annual cost report based upon

guidelines in the ITS Evaluation Resource Guide; and, (2) a final

evaluation report. Projects conducting self-evaluations of the CVISN

are expected to produce a final report. All project partnerships

conducting self-evaluations are expected to submit two camera-ready

reproducible copies and one electronic file to the ITS JPO Program

Assessment Coordinator at: Intelligent Transportation Systems, Joint

Program Office (HOIT-1), Attn: JPO Program Assessment Coordinator, U.S.

Department of Transportation, 400 Seventh St., SW., Washington, D.C.

20590.

Copies which may be required for other addressees will be defined

in annual guidance documents transmitting instructions.

2. Reporting procedures in national evaluations will be defined in

the appropriate documentation governing the contract entered into by

the evaluating organization.

G. References

In lieu of incorporating detailed procedural guidance for the

conduct of evaluations in this document, an ITS Evaluation Resource

Guide has been developed. This comprehensive resource for supporting

evaluation planning is accessible at the ITS JPO web site (http://

www.its.dot.gov) through the Program Assessment/Evaluation Link.

[FR Doc. 99-24363 Filed 9-17-99; 8:45 am]

BILLING CODE 4910-22-P