Standards for Conduct and Evaluation of Activities Carried Out by the Office of Educational Research and Improvement (OERI)Evaluation of the Performance of Recipients of Grants, Cooperative Agreements, and Contracts

SUMMARY: The Assistant Secretary establishes regulations pursuant to

OERI's authorizing legislation, the Educational Research, Development,

Dissemination, and Improvement Act of 1994. The major purpose of these

standards is to ensure that the research, development, and

dissemination activities carried out by the recipients of grants from

and contracts and cooperative agreements with OERI meet the highest

standards of professional excellence.

EFFECTIVE DATE: These regulations take effect November 27, 1998.

FOR FURTHER INFORMATION CONTACT: Sharon Bobbitt, U.S. Department of

Education, 555 New Jersey Avenue, NW, Room 508C, Washington, D.C.

Telephone: (202) 219-2126. Internet: (Sharon--B[email protected]).

Individuals who use a telecommunications device for the deaf (TDD) may

call the Federal Information Relay Service (FIRS) at 1-800-877-8339

between 8 a.m. and 8 p.m., Eastern time, Monday through Friday.

Individuals with disabilities may obtain this document in an

alternate format (e.g., Braille, large print, audiotape, or computer

diskette) on request to the contact person listed in the preceding

paragraph.

SUPPLEMENTARY INFORMATION:

Background

On March 31, 1994, President Clinton signed Pub. L. 103-227, which

includes Title IX, the Educational Research, Development,

Dissemination, and Improvement Act of 1994 (the Act). The Act

restructured OERI and provided it with a broad mandate to conduct an

array of research, development, dissemination, and improvement

activities aimed at strengthening the education of all students.

Statutory Requirements

The Act directed the Assistant Secretary to develop, in

consultation with the National Educational Research Policy and

Priorities Board (the Board), such standards as may be necessary to

govern the conduct and evaluation of all research, development, and

dissemination activities carried out by OERI to ensure that these

activities meet the highest standards of professional excellence. The

Board is responsible for reviewing and approving the standards. The

legislation requires that the standards be developed in three phases.

In the first phase, standards were created and promulgated to

establish the peer review process and evaluation criteria to be used

for the review of applications for grants and cooperative agreements

and proposals for contracts. The final regulations setting out these

standards were published on September 14, 1995 (60 FR 47808). In the

second phase, standards were created and promulgated to establish the

criteria to be used in reviewing potentially exemplary and promising

educational programs. The final regulations setting out these standards

were published on November 17, 1997 (62 FR 61427).

In the third phase, which is the subject of these final

regulations, the Act requires that OERI develop standards for

evaluating and assessing the performance of all recipients of grants

from and cooperative agreements and contracts with OERI. This

evaluation must take place both during and at the conclusion of the

performance of the grant, cooperative agreement, or contract, and must

include the use of a system of peer review for the final assessment.

In developing the standards, the Assistant Secretary was required

to review the procedures utilized by the National Institutes of Health

(NIH), the National Science Foundation (NSF), and other Federal

departments or agencies engaged in research and development and to

solicit recommendations from research organizations and members of the

general public. OERI has reviewed the procedures used to evaluate the

performance of recipients of grants, contracts, or cooperative

agreements by several offices within NIH and NSF, the Office of Energy

Research in the Department of Energy, the Food and Drug Administration,

the National Institute of Standards and Technology, the National

Aeronautics and Space Administration, and the University Research

Initiative of the Department of Defense. Recommendations concerning

these standards have been obtained from the American Educational

Research Association, the Council for Educational Development and

Research, and the Organization of Research Centers.

Standards

The standards have been developed by the Assistant Secretary in

consultation with the Board. These standards cover all grants,

cooperative agreements, and contracts administered by OERI, ranging

from the smallest purchase orders and commissioned papers to the

largest research projects and research centers. The standards:

Require at least one interim assessment as well as a final

assessment of the performance of recipients of grants, cooperative

agreements, and contracts.

Establish procedures for selecting peer review panels to

conduct the assessments.

Establish procedures and criteria that the peer review

panels use in conducting the assessments.

Establish specific additional criteria that peer review

panels use in conducting the assessments for National Research and

Development Centers, Regional Educational Laboratories, Field-Initiated

Studies, and ERIC Clearinghouses.

In an effort to fulfill the law's intention of ensuring high-

quality research, development, and evaluation, OERI has developed

standards in which interim and final assessments may be supplemented by

a self-assessment by the recipient of a grant, cooperative, agreement,

or contract. The Board and the Assistant Secretary believe that the

collection and review of evidence on one's own performance is itself a

useful tool for improvement.

The Government Performance and Results Act requires the

establishment of performance indicators for Department activities.

Information collected pursuant to those indicators will be considered,

as appropriate, in the evaluation of individual recipients.

On February 24, 1998, the Assistant Secretary published a notice of

proposed rulemaking (NPRM) for these standards in the Federal Register

(63 FR 9393). These final regulations contain four major changes from

the NPRM. These changes are fully explained in the ``Analysis of

Comments and Changes'' elsewhere in this preamble. The major changes

pertain to clarification of the purpose of the regulation, how OERI

determines the number of interim assessments necessary, the role of

Department of Education staff in the assessments, and the use of

interim assessments as a source of information for the final

assessment.

Analysis of Comments and Changes

In response to the Secretary's invitation in the NPRM, four parties

[[Page 57571]]

submitted comments on the proposed regulations. In addition to the

public comment, comments from the Board's Subcommittee on Standards are

addressed as required by the legislation. The full Board approved the

final regulations at a meeting on September 18, 1998. An analysis of

the comments and of the changes in the regulations since publication of

the NPRM follows.

Major issues are grouped according to subject with appropriate

sections of the regulations referenced in parentheses. Technical and

other minor changes--and suggested changes the Secretary is not legally

authorized to make under the applicable statutory authority--are not

addressed.

Purpose (Sec. 702.1)

Comments: Three commenters suggested that the purpose of the

standards be clarified. One commenter suggested that the standards

themselves cannot ensure the highest standards of professional

excellence. Another commenter asked specifically whether the purpose

for conducting assessments was to make decisions about future funding

or to provide a system for monitoring and enhancing current and future

projects.

Discussion: The Secretary agrees that the purpose of the standards

should be clarified to go beyond their stated statutory purpose, which

is to ``ensure the highest standards of professional excellence,'' to

include the objectives of continuously improving the quality of funded

activities and of considering the results as one of the factors in

determining continuation funding for multi-year awards.

Changes: Section 702.1 has been modified to include a provision

that the purpose of the standards is to provide feedback to help

improve the quality of funded activities and to provide information for

consideration as continuation funding decisions are made.

Additional Activities that May be Evaluated (Sec. 702.3)

Comment: One commenter thought that the statement that these

standards could be applied to other activities funded by the Department

was too broad and should be deleted.

Discussion: The Secretary believes that this statement is

necessarily broad to allow all Department programs to use these

standards, when appropriate, to assess the performance of any of their

funded activities without developing their own unique regulations. This

statement is also consistent with the earlier standards which

established the peer review process and evaluation criteria to be used

for the review of applications for grants and cooperative agreements

and proposals for contracts.

Changes: None.

Number of Interim Assessments (Sec. 702.4)

Comments: Two commenters suggested changes to this provision. One

commenter suggested since there may be more than one interim

assessment, that it be clear in Sec. 702.4(d)(1). The OERI Board

suggested that the requirements for a single interim assessment for

total awards of $5,000,000 or less be modified to reflect total awards

of $3,000,000 or less.

Discussion: In response to the comments, the Secretary now believes

that considerations such as difficulty in achieving project objectives

rather than the dollar levels of awards should determine whether a

particular project merits more than one interim assessment. Elimination

of the dollar threshold clarifies the original intent of this section

which is to require that all awards receive one interim assessment.

More than one interim assessment will be performed only when a

recipient is having difficulty achieving project objectives as

determined by the initial interim assessment or through the monitoring

efforts of Department of Education staff. The Assistant Secretary will

make the determination of the number of interim assessments on a case-

by-case basis.

Changes: Section 702.4(b) has been modified to delete the dollar

threshold and to reflect that all awards will receive at least one

interim assessment. A new paragraph 702.4(c) has been added to clarify

that the Assistant Secretary will require more than one interim

assessment when a recipient has been identified, either in the initial

interim review or through monitoring efforts of Department of Education

staff, as having difficulty in achieving project objectives. Former

paragraph 702.4(c) has been redesignated as Sec. 702.4(d). Section

702.4(d)(1) has been modified to define an interim assessment as ``any

assessment'' conducted during a recipient's period of performance.

Definitions (Sec. 702.5)

Comment: One commenter suggested that the terms referred to in this

section include the specific definitions and not references to the OERI

statute and to the Education Department General Administrative

Regulations.

Discussion: The Secretary believes that providing the citations for

specific terms rather than the definitions themselves keeps regulations

short and concise while still cross referencing easily accessible

resources for the definitions.

Changes: None.

Characteristics of Peer Reviewers (Sec. 702.10)

Comment: One commenter suggested that paragraph 702.10(a) ``(4)

knowledge of a broad range of education policies and practices;'' be

deleted from the list of knowledge and expertise required of peer

reviewers, because it is redundant with the other criteria and is very

vague.

Discussion: The Secretary believes that this criterion provides for

a balance between specific program knowledge and a broader perspective

of education policies and practices and is therefore not redundant with

the other, more focused, characteristics required of peer reviewers.

Changes: None.

Role of Department Staff (Sec. 702.10)

Comments: Two commenters expressed concern over the appropriate

role of the OERI staff in the review process. One commenter urged the

Department to use all outside reviewers. The other commenter

acknowledged the knowledge and skills of the OERI staff but suggested

that staff not serve as peer reviewers within the primary division of

an agency in which they work and that each peer review panel be limited

to one Department staff person. This commenter suggested that the staff

focus on the important role of mentoring and designing competitions.

Discussion: The Secretary agrees that the primary role of the OERI

staff should be management of competitions including assessing the

results of peer reviews and monitoring awards. The Secretary believes

that the purpose of the peer review process should be to acquire the

perspective of outside experts independent of OERI. The Secretary also

believes that there may be exceptional circumstances where expertise

resides in OERI or in the Department, or where outside reviewers are

not required such as in the review of small purchase orders. The

exceptions should be determined by the Assistant Secretary.

Changes: Section 702.10(d) has been reworded to preclude OERI and

other Department staff from serving as peer reviewers except in

exceptional circumstances as determined by the Assistant Secretary.

Conflict of Interest (Sec. 702.11)

Comment: One commenter was concerned that while the conflict of

interest requirements were ``legally correct'' they failed to address

the problem occasioned by reviewers who

[[Page 57572]]

may have ideological or methodological view points that differ from

those of the recipient to be evaluated, or who are affiliated with

competing institutional organizations.

Discussion: The commenter appears to be concerned that the proposed

conflict of interest provision does not address the potential problem

of bias on the part of a panel against a particular grantee on

ideological or other grounds. The Secretary first believes that it is

essential to retain the present language, which parallels the provision

in the standards at 34 CFR 701.11(c), because it highlights the

important issue of improper financial gain or the appearance of

improper gain. However, the Secretary agrees that adding a requirement

to the effect that panels selected by the Assistant Secretary reflect a

broad range of perspectives could strengthen the regulation.

Changes: A new paragraph ``(c)'' has been added to Sec. 702.13

requiring the Assistant Secretary, to the greatest extent feasible, to

select peer reviewers for each evaluation who represent a broad range

of perspectives.

Sources of Information (Secs. 702.22 and 702.23)

Comment: One commenter suggested that the use of Government

Performance and Results Act (GPRA) information should be encouraged

rather than required for both interim and final assessments. The

commenter is concerned that information currently being collected under

GPRA to evaluate the effectiveness of a program or a system-level

activity will not provide information relevant to the assessment of

individual awards under that program or system-level activity and

therefore should not be required.

Discussion: The Secretary agrees that information obtained by GPRA-

related reports on the effectiveness of a program or system level

activity, e.g., how effectively a program is meeting the overall

objectives defined for it in its authorizing legislation, may not

necessarily include information related to an individual award being

reviewed under this regulation. However, the Secretary believes that

information on the effectiveness of the particular program under which

a recipient receives funding will help to provide a context for the

review of an individual award and must be considered by the panel.

Moreover, these regulations make it clear that the GPRA information is

only one of a number of sources used in conducting the review.

Changes: None.

Comment: One commenter suggested that the findings and information

from interim assessments would be an important source of information

for the final assessments and should be included under Sec. 702.23(a).

Discussion: The Secretary agrees that the results of interim

assessments should be a source of information for final assessments.

Change: Section 702.23(a) has been modified to add a new paragraph

(Sec. 702.23(a)(5)) to require that the results of interim assessments

be considered as a source of information for final assessments.

Evaluation Criteria (Sec. 702.24)

Comments: Two commenters suggested changes to this section. One

commenter suggested that there be a single menu of criteria for the

standards, because the proposed menu is too long. The second commenter

suggested that since Field Initiated Studies are not likely to provide

services, the word ``services'' be deleted from the criterion in

Sec. 702.24(e)(4)(ii): ``* * * addresses issues of national

significance through its products or services, or both.''

Discussion: The Secretary believes the current menu approach

provides a comprehensive strategy for assessing the performance of all

activities, ranging from the smallest purchase order to the largest

research investments. The categories in the regulation reflect the

specific authorities in the OERI statute. In addition, the menu

provides for other criteria for future research investments that do not

fit within the statutory authorities yet also must be assessed. A

single menu would, of necessity, be too generic to apply to the wide

range of activities covered by these standards. The Secretary agrees

that assessing ``services'' is not appropriate for Field Initiated

Studies projects.

Change: Section 702.24(e)(4)(ii) has been modified to delete the

word, ``services.''

Paperwork Reduction Act of 1995

Under the Paperwork Reduction Act of 1995, no persons are required

to respond to a collection of information unless it displays a valid

OMB control number. The valid OMB control number assigned to the

collection of information in these final regulations is displayed at

the end of the affected sections of the regulations.

Assessment of Educational Impact

In the NPRM the Secretary requested comments on whether the

proposed regulations would require transmission of information that is

being gathered by or is available from any other agency or authority of

the United States.

Based on the response to the NPRM and on its own review, the

Department has determined that the regulations in this document do not

require transmission of information that is being gathered by or is

available from any other agency or authority of the United States.

Electronic Access to This Document

Anyone may view this document, as well as all other Department of

Education documents published in the Federal Register, in text or

portable document format (pdf) on the World Wide Web at either of the

following sites:

http://ocfo.ed.gov/fedreg.htm

http://www.ed.gov/news.html

To use the pdf you must have the Adobe Acrobat Reader Program with

Search, which is available free at either of the previous sites. If you

have questions about using the pdf, call the U.S. Government Printing

Office at (202) 512-1530 or, toll free, at 1-888-293-6498.

Anyone may also view these documents in text copy only on an

electronic bulletin board of the Department. Telephone: (202) 219-1511

or, toll free, 1-800-222-4922. The documents are located under Option

G--Files/Announcements, Bulletins and Press Releases.

Note: The official version of this document is the document

published in the Federal Register.

List of Subjects in 34 CFR Part 702

Education, Educational research, Reporting and recordkeeping

requirements.

(Catalog of Federal Domestic Assistance Number does not apply.)

Dated: October 22, 1998.

C. Kent McGuire,

Assistant Secretary for Educational Research and Improvement.

The Secretary amends Chapter VII of Title 34 of the Code of Federal

Regulations by adding a new Part 702 to read as follows:

[[Page 57573]]

PART 702--STANDARDS FOR CONDUCT AND EVALUATION OF ACTIVITIES

CARRIED OUT BY THE OFFICE OF EDUCATIONAL RESEARCH AND IMPROVEMENT

(OERI)--EVALUATION OF THE PERFORMANCE OF RECIPIENTS OF GRANTS,

COOPERATIVE AGREEMENTS, AND CONTRACTS

Subpart A--General

Sec.

702.1 What is the purpose of these standards?

702.2 What activities must be evaluated by these standards?

702.3 What additional activities may be evaluated by these

standards?

702.4 When is performance assessed under these standards?

702.5 What definitions apply?

Subpart B--Selection of Peer Review Panels

702.10 What are the characteristics of peer reviewers?

702.11 What constitutes a conflict of interest for grants and

cooperative agreements?

702.12 What constitutes a conflict of interest for contracts?

702.13 How are peer reviewers selected for panels?

Subpart C--The Evaluation Process

702.21 How does a peer review panel evaluate the performance of a

recipient?

702.22 What information does a peer review panel consider for an

interim assessment?

702.23 What information does a peer review panel consider for a

final assessment?

702.24 What evaluation criteria must be used for performance

assessments?

Authority: 20 U.S.C. 6011(i), unless otherwise noted.

Subpart A--General

Sec. 702.1 What is the purpose of these standards?

(a) The standards in this part implement section 912(i) of the

Educational Research, Development, Dissemination, and Improvement Act

of 1994 (the Act).

(b) These standards establish criteria and a peer review process to

provide recipients of OERI grants, cooperative agreements and contract

awards with assessments of their projects.

(1) The purpose of the assessments is to provide feedback to

recipients to improve the quality of funded activities and to provide

information to OERI as it determines if a recipient of a multi-year

award merits continuation funding.

(2) The criteria and peer review process are intended to address

the statutory requirement that the research, development, and

dissemination activities carried out by the recipients of grants from

and contracts and cooperative agreements with the Office of Educational

Research and Improvement (OERI) meet the highest standards of

professional excellence.

(Authority: 20 U.S.C. 6011(i)(2)(F))

Sec. 702.2 What activities must be evaluated by these standards?

These standards apply to activities carried out by OERI using funds

appropriated under section 912(m) of the Act including activities

carried out by the following entities or programs:

(a) The National Education Research Institutes.

(b) The Office of Reform Assistance and Dissemination.

(c) The Educational Resources Information Center.

(d) The Regional Educational Laboratories.

(e) The Teacher Research Dissemination Demonstration Program.

(f) The Goals 2000 Community Partnerships Program.

(g) The National Educational Research Policy and Priorities Board.

(Authority: 20 U.S.C. 6011(i)(1))

Sec. 702.3 What additional activities may be evaluated by these

standards?

The Secretary may apply these standards to other activities funded

by the Department, as appropriate.

(Authority: 20 U.S. C 6011 (i)(1))

Sec. 702.4 When is performance assessed under these standards?

(a) The Secretary will assess the performance of recipients of OERI

grants, contracts, and cooperative agreements subject to these

standards during and at the conclusion of their period of performance.

(b) The Department requires at least one interim assessment by a

peer review panel for all awards.

(c) The Assistant Secretary will approve and require more than one

interim assessment when an award is identified, either by the initial

interim review or by Department of Education staff monitoring the

award, as having difficulty in achieving project objectives.

(d) A final assessment by a peer review panel is required for all

awards.

(e) As used in this part--

(1) Interim assessment is any assessment conducted during a

recipient's period of performance.

(2) Final assessment is one conducted at the conclusion of a

recipient's period of performance.

(Authority: 20 U.S.C. 6011(i)(2)(F))

Sec. 702.5 What definitions apply?

(a) Definitions in the Educational Research, Development,

Dissemination, and Improvement Act of 1994. The following terms used in

this part are defined in 20 U.S.C. 6011(l)(1):

Development

Dissemination

Educational Research

(b) Definitions in the Education Department General Administrative

Regulations. The following terms used in this part are defined in 34

CFR 77.1:

Application

Award

Department

Grant

Project

Secretary

(c) Definitions in the Federal Acquisition Regulation. The

following term used in this part is defined in 48 CFR Chapter 1:

Contract Proposal.

(Authority: 20 U.S.C. 6011(i)(2)(F)

Subpart B--Selection of Peer Review Panels

Sec. 702.10 What are the characteristics of peer reviewers?

(a) The Assistant Secretary selects each peer reviewer. Each peer

reviewer must have the necessary knowledge and expertise in the area of

the project being reviewed to evaluate the performance of a recipient.

This experience may include--

(1) Expert knowledge of subject matter in the area of the

activities to be reviewed;

(2) Expert knowledge of theory or methods or both in the area of

the activities to be reviewed;

(3) Practical experience in the area of the activities or type of

institution or both to be reviewed;

(4) Knowledge of a broad range of education policies and practices;

(5) Experience in managing complex organizations; or

(6) Expertise and experience in evaluation theory and practice.

(b) Each peer reviewer must be free of conflict of interest, as

determined in accordance with Sec. 702.11 or Sec. 702.12.

(c) The Assistant Secretary may solicit nominations for peer

reviewers from professional associations, nationally recognized

experts, and other sources.

(d) OERI and other Department staff who possess the qualifications

in paragraphs (a) and (b) of this section may serve as peer reviewers

only in exceptional circumstances as determined by the Assistant

Secretary.

(Authority: 20 U.S.C. 6011(i)(2)(B))

Sec. 702.11 What constitutes a conflict of interest for grants and

cooperative agreements?

A peer reviewer assessing the performance of the recipient of a

grant

[[Page 57574]]

from or cooperative agreement with OERI is considered an employee of

the Department for the purposes of conflict of interest analysis. As an

employee of the Department, the peer reviewer is subject to the

provisions of 18 U.S.C. 208, 5 CFR 2635.502, and the Department's

policies used to implement those provisions.

(Authority: 20 U.S.C. 6011(i)(2)(B))

Sec. 702.12 What constitutes a conflict of interest for contracts?

A peer reviewer assessing the performance of the recipient of a

contract with OERI is considered an employee of the Department in

accordance with the Federal Acquisition Regulation (FAR), 48 CFR 3.104-

4(h)(2). As an employee of the Department, the peer reviewer is subject

to the provisions of the FAR, 48 CFR Part 3, Improper Business

Practices and Personal Conflict of Interest.

(Authority: 41 U.S.C. 423)

Sec. 702.13 How are peer reviewers selected for panels?

(a) The Assistant Secretary assigns peer reviewers to panels that

conduct the performance assessments.

(b) The Assistant Secretary may establish panels by category of

recipient, such as a panel to review the performance of all Regional

Educational Laboratories. Each recipient is evaluated individually by

reviewers who have been assigned to this type of panel.

(c) In establishing panels, the Assistant Secretary, to the

greatest extent feasible, selects peer reviewers for each evaluation

who represent a broad range of perspectives.

(Authority: 20 U.S.C. 6011(i)(2)(B))

Subpart C--The Evaluation Process

Sec. 702.21 How does a peer review panel evaluate the performance of a

recipient?

(a) In each evaluation, a peer review panel--

(1) Considers relevant information about the recipient's

performance, as described in Secs. 702.22 and 702.23; and

(2) Makes judgments about the recipient's performance, using the

criteria in Sec. 702.24.

(b) Each peer reviewer prepares a report based on the reviewer's

assessment of the quality of the project according to the evaluation

criteria.

(c) After each peer reviewer has evaluated each project

independently, the panel may be convened to discuss the strengths and

weaknesses of the project. Each reviewer may then independently re-

evaluate each project with appropriate changes made to the written

report.

(d) The report of the interim assessment must include any

recommendations the peer reviewer may have for improving the

recipient's performance.

(e) The report of the final assessment must contain each peer

reviewer's evaluative summary of the recipient's performance, from the

beginning of the contract, grant, or cooperative agreement to its

conclusion.

(Authority: 20 U.S.C. 6011(i)(2)(F))

Sec. 702.22 What information does a peer review panel consider for an

interim assessment?

(a) Sources of information for the interim assessment must

include--

(1) The original request for proposals or grant announcement and

the contract proposal or grant application;

(2) Documentation of any changes in the work described in the

contract, grant, or cooperative agreement, including reasons for the

changes;

(3) Any progress reports delivered to the Department or made

available to the public by the recipient;

(4) Examples of products delivered to the Department or made

available to the public by the recipient;

(5) Any relevant reports written by OERI staff, including reports

of site visits by OERI staff;

(6) Any performance evaluations conducted under the FAR or the

Education Department General Administrative Regulations (34 CFR Part

75).

(7) Any relevant information provided by the recipient in response

to Government Performance and Results Act (GPRA) (Pub. L. 103-62)

requirements; and

(8) Any reports from program evaluations commissioned by the

Department.

(b) Sources of information for the interim assessment may also

include--

(1) A self-assessment, prepared by the recipient, addressing the

criteria in Sec. 702.24;

(2) One or more site visits by the peer review panel;

(3) One or more oral or written presentations to the panel by the

recipient describing its performance; or

(4) Other information about the recipient's performance.

(Approved by the Office of Management and Budget under control

number 1850-0746)

(Authority: 20 U.S.C. 6011(i)(2)(F))

Sec. 702.23 What information does a peer review panel consider for a

final assessment?

(a) Sources of information for the final assessment must include--

(1) The original request for proposals or application notice and

the contract proposal or grant application, together with documentation

of any changes in the work described in the proposal or application,

including reasons for the changes;

(2) If consistent with the recipient's contract, grant, or

cooperative agreement with OERI, a written report or oral presentation

or both by the recipient summarizing its activities and

accomplishments;

(3) Any relevant information provided by the recipient in response

to Government Performance and Results Act (GPRA) (Pub. L. 103-62)

requirements;

(4) Any reports from program evaluations commissioned by the

Department; and,

(5) Any relevant information provided by the interim assessment.

(b) The final assessment may also include other sources of

information, such as one or more of those listed in Sec. 702.22.

(Approved by the Office of Management and Budget under control

number 1850-0746)

(Authority: 20 U.S.C. 6011(i)(2)(F))

Sec. 702.24 What evaluation criteria must be used for performance

assessments?

(a) Peer reviewers (and those recipients who conduct self-

evaluations) shall use the criteria in paragraph (b) of this section to

assess performance and, in case of interim assessments, to identify

areas in which the performance of recipients may need improvement.

(b) The following evaluation criteria are to guide the assessment

process undertaken by peer reviewers. The peer reviewers determine the

extent to which recipients meet these criteria:

(1) Implementation and management. (i) Peer reviewers shall

consider the degree to which the recipient has fully executed its

program of work. In doing so, peer reviewers shall consider evidence on

the extent to which the recipient completes the work described in the

approved application or contract, including any approved modifications,

in the time period proposed and in an efficient manner.

(ii) In examining the degree of implementation, peer reviewers may

also consider evidence on the extent to which--

(A) The recipient implements and utilizes a quality assurance

system for its products or services or both; and

(B) The recipient conducts self-assessment or self-evaluation

activities, including periodically seeking out independent critiques

and evaluations of its work, and uses the results to improve

performance.

(2) Quality. (i) Peer reviewers shall consider the degree to which

the

[[Page 57575]]

recipient's work approaches or attains professional excellence. In

determining quality, peer reviewers shall consider evidence on the

extent to which--

(A) The recipient utilizes processes, methods, and techniques

appropriate to achieve the goals and objectives for the program of work

in the approved application; and

(B) The recipient applies appropriate processes, methods, and

techniques in a manner consistent with the highest standards of the

profession.

(ii) In determining quality, peer reviewers may also consider the

extent to which the recipient conducts a coherent, sustained program of

work informed by relevant research.

(3) Utility. (i) In determining the utility of the recipient's

products or services or both, peer reviewers shall consider evidence on

the extent to which the recipient's work (including information,

materials, processes, techniques, or activities) is effectively used by

and is useful to its customers in appropriate settings.

(ii) In determining utility, peer reviewers may also consider the

extent to which the recipient has received national recognition; e.g.,

articles in refereed journals and presentations at professional

conferences.

(4) Outcomes and impact. (i) Peer reviewers shall consider the

results of the recipient's work. In examining outcomes and impact, peer

reviewers shall consider evidence on the extent to which--

(A) The recipient meets the needs of its customers; and

(B) The recipient's work contributes to the increased knowledge or

understanding of educational problems, issues, or effective strategies.

(ii) In examining outcomes and impact, peer reviewers may also

consider the extent to which recipients address issues of national

significance through its products or services or both.

(c) For National Research and Development Centers, peer reviewers

also shall consider evidence on the extent to which recipients meet the

following criteria:

(1) Quality. (i) The recipient uses a well-conceptualized framework

and sound theoretical and methodological tools in conducting

professionally rigorous studies; and

(ii) The recipient conducts work of sufficient size, scope, and

duration to produce sound guidance for improvement efforts and future

research.

(2) Utility. The recipient documents, reports, and disseminates its

work in ways to facilitate the effective use of its work in

appropriately targeted settings.

(3) Outcomes and impact. (i) The recipient's work contributes to

the development and advancement of theory in the field of study,

including its priority area; and

(ii) The recipient addresses issues of national significance

through its products or services or both.

(d) For the Regional Educational Laboratories, peer reviewers also

shall consider evidence on the extent to which recipients meet the

following criteria:

(1) Quality. (i) The recipient utilizes a well-conceptualized

framework and sound theoretical and methodological tools in conducting

professionally rigorous studies;

(ii) The recipient conducts work of sufficient size, scope, and

duration to produce sound guidance for improvement efforts; and

(iii) The recipient's products are well tested and based on sound

research.

(2) Utility. The recipient documents, reports, and disseminates its

work in ways to facilitate its effective use in appropriately targeted

settings, particularly in school improvement efforts of States and

localities.

(3) Outcomes and impact. (i) The recipient assists States and

localities to implement comprehensive school improvement strategies

through the provision of research-based information (including well-

tested models and strategies), materials and assistance; and

(ii) The recipient's work results in widespread access to

information regarding research and best practices, particularly within

its region.

(e) For Field-Initiated Studies, peer reviewers also shall consider

evidence on the extent to which recipients meet the following criteria:

(1) Implementation and management. The recipient's work responds to

the goals, objectives and mission of the National Institute from which

it is funded.

(2) Quality. The recipient utilizes a well-conceptualized framework

and sound theoretical and methodological tools in conducting

professionally rigorous studies.

(3) Utility. The recipient documents, reports, and disseminates its

work in ways to facilitate its effective use in appropriately targeted

settings.

(4) Outcomes and impact. (i) The recipient's work contributes to

the development and advancement of theory and knowledge in the field of

study; and

(ii) The recipient addresses issues of national significance

through its products.

(f) For the ERIC Clearinghouses, peer reviewers also shall consider

evidence on the extent to which recipients meet the following criteria:

(1) Quality. The recipient applies an integrated approach to

acquiring and disseminating significant and high-quality educational

literature and materials to maintain and enhance the ERIC database.

(2) Utility. The recipient contributes to the development of the

ERIC database as a source of literature and materials that reflects

trends and issues within its scope.

(3) Outcomes and impact. (i) The recipient meets the informational

and educational needs of its customers through dissemination and

outreach approaches and the development of an array of print and non-

print materials; and

(ii) The recipient provides national leadership on the use of

current computer, networking, and information technology.

(Approved by the Office of Management and Budget under control

number 1850-0746)

(Authority: 20 U.S.C. 6011(i)(2)(F))

[FR Doc. 98-28729 Filed 10-26-98; 8:45 am]

BILLING CODE 4000-01-U