Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies; Republication

Part IX

Office of Management and Budget

-----------------------------------------------------------------------

Guidelines for Ensuring and Maximizing the Quality, Objectivity,

Utility, and Integrity of Information Disseminated by Federal Agencies;

Notice; Republication

Federal Register / Vol. 67, No. 36 / Friday, February 22, 2002 /

Notices

[[Page 8452]]

-----------------------------------------------------------------------

OFFICE OF MANAGEMENT AND BUDGET

Guidelines for Ensuring and Maximizing the Quality, Objectivity,

Utility, and Integrity of Information Disseminated by Federal Agencies;

Republication

Editorial Note: Due to numerous errors, this document is being

reprinted in its entirety. It was originally printed in the Federal

Register on Thursday, January 3, 2002 at 67 FR 369-378 and was

corrected on Tuesday, February 5, 2002 at 67 FR 5365.

AGENCY: Office of Management and Budget, Executive Office of the

President.

ACTION: Final guidelines.

-----------------------------------------------------------------------

SUMMARY: These final guidelines implement section 515 of the Treasury

and General Government Appropriations Act for Fiscal Year 2001 (Public

Law 106-554; H.R. 5658). Section 515 directs the Office of Management

and Budget (OMB) to issue government-wide guidelines that ``provide

policy and procedural guidance to Federal agencies for ensuring and

maximizing the quality, objectivity, utility, and integrity of

information (including statistical information) disseminated by Federal

agencies.'' By October 1, 2002, agencies must issue their own

implementing guidelines that include ``administrative mechanisms

allowing affected persons to seek and obtain correction of information

maintained and disseminated by the agency'' that does not comply with

the OMB guidelines. These final guidelines also reflect the changes OMB

made to the guidelines issued September 28, 2001, as a result of

receiving additional comment on the ``capable of being substantially

reproduced'' standard (paragraphs V.3.B, V.9, and V.10), which OMB

previously issued on September 28, 2001, on an interim final basis.

DATES: Effective Date: January 3, 2002.

FOR FURTHER INFORMATION CONTACT: Brooke J. Dickson, Office of

Information and Regulatory Affairs, Office of Management and Budget,

Washington, DC 20503. Telephone (202) 395-3785 or by e-mail to

[email protected].

SUPPLEMENTARY INFORMATION: In section 515(a) of the Treasury and

General Government Appropriations Act for Fiscal Year 2001 (Public Law

106-554; H.R. 5658), Congress directed the Office of Management and

Budget (OMB) to issue, by September 30, 2001, government-wide

guidelines that ``provide policy and procedural guidance to Federal

agencies for ensuring and maximizing the quality, objectivity, utility,

and integrity of information (including statistical information)

disseminated by Federal agencies * * *'' Section 515(b) goes on to

state that the OMB guidelines shall:

``(1) apply to the sharing by Federal agencies of, and access to,

information disseminated by Federal agencies; and

``(2) require that each Federal agency to which the guidelines

apply--

``(A) issue guidelines ensuring and maximizing the quality,

objectivity, utility, and integrity of information (including

statistical information) disseminated by the agency, by not later than

1 year after the date of issuance of the guidelines under subsection

(a);

``(B) establish administrative mechanisms allowing affected persons

to seek and obtain correction of information maintained and

disseminated by the agency that does not comply with the guidelines

issued under subsection (a); and

``(C) report periodically to the Director--

``(i) the number and nature of complaints received by the agency

regarding the accuracy of information disseminated by the agency and;

``(ii) how such complaints were handled by the agency.''

Proposed guidelines were published in the Federal Register on June

28, 2001 (66 FR 34489). Final guidelines were published in the Federal

Register on September 28, 2001 (66 FR 49718). The Supplementary

Information to the final guidelines published in September 2001

provides background, the underlying principles OMB followed in issuing

the final guidelines, and statements of intent concerning detailed

provisions in the final guidelines.

In the final guidelilnes published in September 2001, OMB also

requested additional comment on the ``capable of being substantially

reproduced'' standard and the related definition of ``influential

scientific or statistical information'' (paragraphs V.3.B, V.9, and

V.10), which were issued on an interim final basis. The final

guidelines published today discuss the public comments OMB received,

the OMB response, and amendments to the final guidelines published in

September 2001.

In developing agency-specific guidelines, agencies should refer

both to the Supplementary Information to the final guidelines published

in the Federal Register on September 28, 2001 (66 FR 49718), and also

to the Supplementary Information published today. We stress that the

three ``Underlying Principles'' that OMB followed in drafting the

guidelines that we published on September 28, 2001 (66 FR 49719), are

also applicable to the amended guidelines that we publish today.

In accordance with section 515, OMB has designed the guidelines to

help agencies ensure and maximize the quality, utility, objectivity and

integrity of the information that they disseminate (meaning to share

with, or give access to, the public). It is crucial that information

Federal agencies disseminate meets these guidelines. In this respect,

the fact that the Internet enables agencies to communicate information

quickly and easily to a wide audience not only offers great benefits to

society, but also increases the potential harm that can result from the

dissemination of information that does not meet basic information

quality guidelines. Recognizing the wide variety of information Federal

agencies disseminate and the wide variety of dissemination practices

that agencies have, OMB developed the guidelines with several

principles in mind.

First, OMB designed the guidelines to apply to a wide variety of

government information dissemination activities that may range in

importance and scope. OMB also designed the guidelines to be generic

enough to fit all media, be they printed, electronic, or in other form.

OMB sought to avoid the problems that would be inherent in developing

detailed, prescriptive, ``one-size-fits-all'' government-wide

guidelines that would artificially require different types of

dissemination activities to be treated in the same manner. Through this

flexibility, each agency will be able to incorporate the requirements

of these OMB guidelines into the agency's own information resource

management and administrative practices.

Second, OMB designed the guidelines so that agencies will meet

basic information quality standards. Given the administrative

mechanisms required by section 515 as well as the standards set forth

in the Paperwork Reduction Act, it is clear that agencies should not

disseminate substantive information that does not meet a basic level of

quality. We recognize that some government information may need to meet

higher or more specific information quality standards than those that

would apply to other types of government information. The more

important the information, the higher the quality standards to which it

should be held, for example, in those situations involving

``influential scientific, financial, or statistical information'' (a

phrase defined in these guidelines). The guidelines recognize, however,

that

[[Page 8453]]

information quality comes at a cost. Accordingly, the agencies should

weigh the costs (for example, including costs attributable to agency

processing effort, respondent burden, maintenance of needed privacy,

and assurances of suitable confidentiality) and the benefits of higher

information quality in the development of information, and the level of

quality to which the information disseminated will be held.

Third, OMB designed the guidelines so that agencies can apply them

in a common-sense and workable manner. It is important that these

guidelines do not impose unnecessary administrative burdens that would

inhibit agencies from continuing to take advantage of the Internet and

other technologies to disseminate information that can be of great

benefit and value to the public. In this regard, OMB encourages

agencies to incorporate the standards and procedures required by these

guidelines into their existing information resources management and

administrative practices rather than create new and potentially

duplicative or contradictory processes. The primary example of this is

that the guidelines recognize that, in accordance with OMB Circular A-

130, agencies already have in place well-established information

quality standards and administrative mechanisms that allow persons to

seek and obtain correction of information that is maintained and

disseminated by the agency. Under the OMB guidelines, agencies need

only ensure that their own guidelines are consistent with these OMB

guidelines, and then ensure that their administrative mechanisms

satisfy the standards and procedural requirements in the new agency

guidelines. Similarly, agencies may rely on their implementation of the

Federal Government's computer security laws (formerly, the Computer

Security Act, and now the computer security provisions of the Paperwork

Reduction Act) to establish appropriate security safeguards for

ensuring the ``integrity'' of the information that the agencies

disseminate.

In addition, in response to concerns expressed by some of the

agencies, we want to emphasize that OMB recognizes that Federal

agencies provide a wide variety of data and information. Accordingly,

OMB understands that the guidelines discussed below cannot be

implemented in the same way by each agency. In some cases, for example,

the data disseminated by an agency are not collected by that agency;

rather, the information the agency must provide in a timely manner is

compiled from a variety of sources that are constantly updated and

revised and may be confidential. In such cases, while agencies'

implementation of the guidelines may differ, the essence of the

guidelines will apply. That is, these agencies must make their methods

transparent by providing documentation, ensure quality by reviewing the

underlying methods used in developing the data and consulting (as

appropriate) with experts and users, and keep users informed about

corrections and revisions.

Summary of OMB Guidelines

These guidelines apply to Federal agencies subject to the Paperwork

Reduction Act (44 U.S.C. chapter 35). Agencies are directed to develop

information resources management procedures for reviewing and

substantiating (by documentation or other means selected by the agency)

the quality (including the objectivity, utility, and integrity) of

information before it is disseminated. In addition, agencies are to

establish administrative mechanisms allowing affected persons to seek

and obtain, where appropriate, correction of information disseminated

by the agency that does not comply with the OMB or agency guidelines.

Consistent with the underlying principles described above, these

guidelines stress the importance of having agencies apply these

standards and develop their administrative mechanisms so they can be

implemented in a common sense and workable manner. Moreover, agencies

must apply these standards flexibly, and in a manner appropriate to the

nature and timeliness of the information to be disseminated, and

incorporate them into existing agency information resources management

and administrative practices.

Section 515 denotes four substantive terms regarding information

disseminated by Federal agencies: quality, utility, objectivity, and

integrity. It is not always clear how each substantive term relates--or

how the four terms in aggregate relate--to the widely divergent types

of information that agencies disseminate. The guidelines provide

definitions that attempt to establish a clear meaning so that both the

agency and the public can readily judge whether a particular type of

information to be disseminated does or does not meet these attributes.

In the guidelines, OMB defines ``quality'' as the encompassing

term, of which ``utility,'' ``objectivity,'' and ``integrity'' are the

constituents. ``Utility'' refers to the usefulness of the information

to the intended users. ``Objectivity'' focuses on whether the

disseminated information is being presented in an accurate, clear,

complete, and unbiased manner, and as a matter of substance, is

accurate, reliable, and unbiased. ``Integrity'' refers to security--the

protection of information from unauthorized access or revision, to

ensure that the information is not compromised through corruption or

falsification. OMB modeled the definitions of ``information,''

``government information,'' ``information dissemination product,'' and

``dissemination'' on the longstanding definitions of those terms in OMB

Circular A-130, but tailored them to fit into the context of these

guidelines.

In addition, Section 515 imposes two reporting requirements on the

agencies. The first report, to be promulgated no later than October 1,

2002, must provide the agency's information quality guidelines that

describe administrative mechanisms allowing affected persons to seek

and obtain, where appropriate, correction of disseminated information

that does not comply with the OMB and agency guidelines. The second

report is an annual fiscal year report to OMB (to be first submitted on

January 1, 2004) providing information (both quantitative and

qualitative, where appropriate) on the number, nature, and resolution

of complaints received by the agency regarding its perceived or

confirmed failure to comply with these OMB and agency guidelines.

Public Comments and OMB Response

Applicability of Guidelines. Some comments raised concerns about

the applicability of these guidelines, particularly in the context of

scientific research conducted by Federally employed scientists or

Federal grantees who publish and communicate their research findings in

the same manner as their academic colleagues. OMB believes that

information generated and disseminated in these contexts is not covered

by these guidelines unless the agency represents the information as, or

uses the information in support of, an official position of the agency.

As a general matter, these guidelines apply to ``information'' that

is ``disseminated'' by agencies subject to the Paperwork Reduction Act

(44 U.S.C. 3502(1)). See paragraphs II, V.5 and V.8. The definitions of

``information'' and ``dissemination'' establish the scope of the

applicability of these guidelines. ``Information'' means ``any

communication or representation of knowledge such as facts or data * *

*'' This definition of information in paragraph V.5 does ``not include

opinions, where the agency's presentation makes it clear that what is

[[Page 8454]]

being offered is someone's opinion rather than fact or the agency's

views.''

``Dissemination'' is defined to mean ``agency initiated or

sponsored distribution of information to the public.'' As used in

paragraph V.8, ``agency INITIATED * * * distribution of information to

the public'' refers to information that the agency disseminates, e.g.,

a risk assessment prepared by the agency to inform the agency's

formulation of possible regulatory or other action. In addition, if an

agency, as an institution, disseminates information prepared by an

outside party in a manner that reasonably suggests that the agency

agrees with the information, this appearance of having the information

represent agency views makes agency dissemination of the information

subject to these guidelines. By contrast, an agency does not

``initiate'' the dissemination of information when a Federally employed

scientist or Federal grantee or contractor publishes and communicates

his or her research findings in the same manner as his or her academic

colleagues, even if the Federal agency retains ownership or other

intellectual property rights because the Federal government paid for

the research. To avoid confusion regarding whether the agency agrees

with the information (and is therefore disseminating it through the

employee or grantee), the researcher should include an appropriate

disclaimer in the publication or speech to the effect that the ``views

are mine, and do not necessarily reflect the view'' of the agency.

Similarly, as used in paragraph V.8., ``agency * * * SPONSORED

distribution of information to the public'' refers to situations where

an agency has directed a third-party to disseminate information, or

where the agency has the authority to review and approve the

information before release. Therefore, for example, if an agency

through a procurement contract or a grant provides for a person to

conduct research, and then the agency directs the person to disseminate

the results (or the agency reviews and approves the results before they

may be disseminated), then the agency has ``sponsored'' the

dissemination of this information. By contrast, if the agency simply

provides funding to support research, and it the researcher (not the

agency) who decides whether to disseminate the results and--if the

results are to be released--who determines the content and presentation

of the dissemination, then the agency has not ``sponsored'' the

dissemination even though it has funded the research and even if the

Federal agency retains ownership or other intellectual property rights

because the Federal government paid for the research. To avoid

confusion regarding whether the agency is sponsoring the dissemination,

the researcher should include an appropriate disclaimer in the

publication or speech to the effect that the ``views are mine, and do

not necessarily reflect the view'' of the agency. On the other hand,

subsequent agency dissemination of such information requires that the

information adhere to the agency's information quality guidelines. In

sum, these guidelines govern an agency's dissemination of information,

but generally do not govern a third-party's dissemination of

information (the exception being where the agency is essentially using

the third-party to disseminate information on the agency's behalf).

Agencies, particularly those that fund scientific research, are

encouraged to clarify the applicability of these guidelines to the

various types of information they and their employees and grantees

disseminate.

Paragraph V.8 also states that the definition of ``dissemination''

does not include ``* * * distribution limited to correspondence with

individuals or persons, press releases, archival records, public

filings, subpoenas or adjudicative processes.'' The exemption from the

definition of ``dissemination'' for ``adjudicative processes'' is

intended to exclude, from the scope of these guidelines, the findings

and determinations that an agency makes in the course of adjudications

involving specific parties. There are well-established procedural

safeguards and rights to address the quality of adjudicatory decisions

and to provide persons with an opportunity to contest decisions. These

guidelines do not impose any additional requirements on agencies during

adjudicative proceedings and do not provide parties to such

adjudicative proceedings any additional rights of challenge or appeal.

The Presumption Favoring Peer-Reviewed Information.As a general

matter, in the scientific and research context, we regard technical

information that has been subjected to formal, independent, external

peer review as presumptively objective. As the guidelines state in

paragraph V.3.b.i: ``If data and analytic results have been subjected

to formal, independent, external peer review, the information may

generally be presumed to be of acceptable objectivity.'' An example of

a formal, independent, external peer review is the review process used

by scientific journals.

Most comments approved of the prominent role that peer review plays

in the OMB guidelines. Some comments contended that peer review was not

accepted as a universal standard that incorporates an established,

practiced, and sufficient level of objectivity. Other comments stated

that the guidelines would be better clarified by making peer review one

of several factors that an agency should consider in assessing the

objectivity (and quality in general) of original research. In addition,

several comments noted that peer review does not establish whether

analytic results are capable of being substantially reproduced. In

light of the comments, the final guidelines in new paragraph V.3.b.i

qualify the presumption in favor of peer-reviewed information as

follows: ``However, this presumption is rebuttable based on a

persuasive showing by the petitioner in a particular instance.''

We believe that transparency is important for peer review, and

these guidelines set minimum standards for the transparency of agency-

sponsored peer review. As we state in new paragraph V.3.b.i: ``If data

and analytic results have been subjected to formal, independent,

external peer review, the information may generally be presumed to be

of acceptable objectivity. However, this presumption is rebuttable

based on a persuasive showing by the petitioner in a particular

instance. If agency-sponsored peer review is employed to help satisfy

the objectivity standard, the review process employed shall meet the

general criteria for competent and credible peer review recommended by

OMB-OIRA to the President's Management Council (9/20/01) (http://www.whitehouse.gov/omb/inforeg/oira_review-process.html), namely, `that

(a) peer reviewers be selected primarily on the basis of necessary

technical expertise, (b) peer reviewers be expected to disclose to

agencies prior technical/policy positions they may have taken on the

issues at hand, (c) peer reviewers be expected to disclose to agencies

their sources of personal and institutional funding (private or public

sector), and (d) peer reviews be conducted in an open and rigorous

manner.' ''

The importance of these general criteria for competent and credible

peer review has been supported by a number of expert bodies. For

example, ``the work of fully competent peer-review panels can be

undermined by allegations of conflict of interest and bias. Therefore,

the best interests of the Board are served by effective policies and

procedures regarding potential conflicts of interest, impartiality, and

panel balance.'' (EPA's Science Advisory

[[Page 8455]]

Board Panels: Improved Policies and Procedures Needed to Ensure

Independence and Balance, GAO-01-536, General Accounting Office,

Washington, DC, June 2001, page 19.) As another example, ``risk

analyses should be peer-reviewed and accessible--both physically and

intellectually--so that decision-makers at all levels will be able to

respond critically to risk characterizations. The intensity of the peer

reviews should be commensurate with the significance of the risk or its

management implications.'' (Setting Priorities, Getting Results: A New

Direction for EPA, Summary Report, National Academy of Public

Administration, Washington, DC, April 1995, page 23.)

These criteria for peer reviewers are generally consistent with the

practices now followed by the National Research Council of the National

Academy of Sciences. In considering these criteria for peer reviewers,

we note that there are many types of peer reviews and that agency

guidelines concerning the use of peer review should tailor the rigor of

peer review to the importance of the information involved. More

generally, agencies should define their peer-review standards in

appropriate ways, given the nature and importance of the information

they disseminate.

Is Journal Peer Review Always Sufficient? Some comments argued that

journal peer review should be adequate to demonstrate quality, even for

influential information that can be expected to have major effects on

public policy. OMB believes that this position overstates the

effectiveness of journal peer review as a quality-control mechanism.

Although journal peer review is clearly valuable, there are cases

where flawed science has been published in respected journals. For

example, the NIH Office of Research Integrity recently reported the

following case regarding environmental health research:

``Based on the report of an investigation conducted by [XX]

University, dated July 16, 1999, and additional analysis conducted

by ORI in its oversight review, the US Public Health Service found

that Dr. [X] engaged in scientific misconduct. Dr. [X] committed

scientific misconduct by intentionally falsifying the research

results published in the journal SCIENCE and by providing falsified

and fabricated materials to investigating officials at [XX]

University in response to a request for original data to support the

research results and conclusions report in the SCIENCE paper. In

addition, PHS finds that there is no original data or other

corroborating evidence to support the research results and

conclusions reported in the SCIENCE paper as a whole.'' (66 FR

52137, October 12, 2001).

Although such cases of falsification are presumably rare, there is

a significant scholarly literature documenting quality problems with

articles published in peer-reviewed research. ``In a [peer-reviewed]

meta-analysis that surprised many--and some doubt--researchers found

little evidence that peer review actually improves the quality of

research papers.'' (See, e.g., Science, Vol. 293, page 2187 (September

21, 2001.)) In part for this reason, many agencies have already adopted

peer review and science advisory practices that go beyond journal peer

review. See, e.g., Sheila Jasanoff, The Fifth Branch: Science Advisers

as Policy Makers, Cambridge, MA, Harvard University Press, 1990; Mark

R. Powell, Science at EPA: Information in the Regulatory Process.

Resources for the Future, Washington, DC., 1999, pages 138-139; 151-

153; Implementation of the Environmental Protection Agency's Peer

Review Program: An SAB Evaluation of Three Reviews, EPA-SAB-RSAC-01-

009, A Review of the Research Strategies Advisory Committee (RSAC) of

the EPA Science Advisory Board (SAB), Washington, DC., September 26,

2001. For information likely to have an important public policy or

private sector impact, OMB believes that additional quality checks

beyond peer review are appropriate.

Definition of ``Influential''. OMB guidelines apply stricter

quality standards to the dissemination of information that is

considered ``influential.'' Comments noted that the breadth of the

definition of ``influential'' in interim final paragraph V.9 requires

much speculation on the part of agencies.

We believe that this criticism has merit and have therefore

narrowed the definition. In this narrower definition, ``influential'',

when used in the phrase ``influential scientific, financial, or

statistical information'', is amended to mean that ``the agency can

reasonably determine that dissemination of the information will have or

does have a clear and substantial impact on important public policies

or important private sector decisions.'' The intent of the new phrase

``clear and substantial'' is to reduce the need for speculation on the

part of agencies. We added the present tense--``or does have''--to this

narrower definition because on occasion, an information dissemination

may occur simultaneously with a particular policy change. In response

to a public comment, we added an explicit reference to ``financial''

information as consistent with our original intent.

Given the differences in the many Federal agencies covered by these

guidelines, and the differences in the nature of the information they

disseminate, we also believe it will be helpful if agencies elaborate

on this definition of ``influential'' in the context of their missions

and duties, with due consideration of the nature of the information

they disseminate. As we state in amended paragraph V.9, ``Each agency

is authorized to define `influential' in ways appropriate for it given

the nature and multiplicity of issues for which the agency is

responsible.''

Reproducibility. As we state in new paragraph V.3.b.ii: ``If an

agency is responsible for disseminating influential scientific,

financial, or statistical information, agency guidelines shall include

a high degree of transparency about data and methods to facilitate the

reproducibility of such information by qualified third parties.'' OMB

believes that a reproducibility standard is practical and appropriate

for information that is considered ``influential'', as defined in

paragraph V.9--that ``will have or does have a clear and substantial

impact on important public policies or important private sector

decisions.'' The reproducibility standard applicable to influential

scientific, financial, or statistical information is intended to ensure

that information disseminated by agencies is sufficiently transparent

in terms of data and methods of analysis that it would be feasible for

a replication to be conducted. The fact that the use of original and

supporting data and analytic results have been deemed ``defensible'' by

peer-review procedures does not necessarily imply that the results are

transparent and replicable.

Reproducibility of Original and Supporting Data. Several of the

comments objected to the exclusion of original and supporting data from

the reproducibility requirements. Comments instead suggested that OMB

should apply the reproducibility standard to original data, and that

OMB should provide flexibility to the agencies in determining what

constitutes ``original and supporting'' data. OMB agrees and asks that

agencies consider, in developing their own guidelines, which categories

of original and supporting data should be subject to the

reproducibility standard and which should not. To help in resolving

this issue, we also ask agencies to consult directly with relevant

scientific and technical communities on the feasibility of having the

selected categories of original and supporting data subject to the

reproducibility standard. Agencies are encouraged to address ethical,

feasibility, and confidentiality issues

[[Page 8456]]

with care. As we state in new paragraph V.3.b.ii.A, ``Agencies may

identify, in consultation with the relevant scientific and technical

communities, those particular types of data that can practicably be

subjected to a reproducibility requirement, given ethical, feasibility,

or confidentiality constraints.'' Further, as we state in our expanded

definition of ``reproducibility'' in paragraph V.10, ``If agencies

apply the reproducibility test to specific types of original or

supporting data, the associated guidelines shall provide relevant

definitions of reproducibility (e.g., standards for replication of

laboratory data).'' OMB urges caution in the treatment of original and

supporting data because it may often be impractical or even

impermissible or unethical to apply the reproducibility standard to

such data. For example, it may not be ethical to repeat a ``negative''

(ineffective) clinical (therapeutic) experiment and it may not be

feasible to replicate the radiation exposures studied after the

Chernobyl accident. When agencies submit their draft agency guidelines

for OMB review, agencies should include a description of the extent to

which the reproducibility standard is applicable and reflect

consultations with relevant scientific and technical communities that

were used in developing guidelines related to applicability of the

reproducibility standard to original and supporting data.

It is also important to emphasize that the reproducibility standard

does not apply to all original and supporting data disseminated by

agencies. As we state in new paragraph V.3.b.ii.A, ``With regard to

original and supporting data related [to influential scientific,

financial, or statistical information], agency guidelines shall not

require that all disseminated data be subjected to a reproducibility

requirement.'' In addition, we encourage agencies to address how

greater transparency can be achieved regarding original and supporting

data. As we also state in new paragraph V.3.b.ii.A, ``It is understood

that reproducibility of data is an indication of transparency about

research design and methods and thus a replication exercise (i.e., a

new experiment, test, or sample) shall not be required prior to each

dissemination.'' Agency guidelines need to achieve a high degree of

transparency about data even when reproducibility is not required.

Reproducibility of Analytic Results. Many public comments were

critical of the reproducibility standard and expressed concern that

agencies would be required to reproduce each analytical result before

it is disseminated. While several comments commended OMB for

establishing an appropriate balance in the ``capable of being

substantially reproduced'' standard, others considered this standard to

be inherently subjective. There were also comments that suggested the

standard would cause more burden for agencies.

It is not OMB's intent that each agency must reproduce each

analytic result before it is disseminated. The purpose of the

reproducibility standard is to cultivate a consistent agency commitment

to transparency about how analytic results are generated: the specific

data used, the various assumptions employed, the specific analytic

methods applied, and the statistical procedures employed. If sufficient

transparency is achieved on each of these matters, then an analytic

result should meet the ``capable of being substantially reproduced''

standard.

While there is much variation in types of analytic results, OMB

believes that reproducibility is a practical standard to apply to most

types of analytic results. As we state in new paragraph V.3.b.ii.B,

``With regard to analytic results related [to influential scientific,

financial, or statistical information], agency guidelines shall

generally require sufficient transparency about data and methods that

an independent reanalysis could be undertaken by a qualified member of

the public. These transparency standards apply to agency analysis of

data from a single study as well as to analyses that combine

information from multiple studies.'' We elaborate upon this principle

in our expanded definition of ``reproducibility'' in paragraph V.10:

``With respect to analytic results, `capable of being substantially

reproduced' means that independent analysis of the original or

supporting data using identical methods would generate similar analytic

results, subject to an acceptable degree of imprecision or error.''

Even in a situation where the original and supporting data are

protected by confidentiality concerns, or the analytic computer models

or other research methods may be kept confidential to protect

intellectual property, it may still be feasible to have the analytic

results subject to the reproducibility standard. For example, a

qualified party, operating under the same confidentiality protections

as the original analysts, may be asked to use the same data, computer

model or statistical methods to replicate the analytic results reported

in the original study. See, e.g., ``Reanalysis of the Harvard Six

Cities Study and the American Cancer Society Study of Particulate Air

Pollution and Mortality,'' A Special Report of the Health Effects

Institute's Particle Epidemiology Reanalysis Project, Cambridge, MA,

2000.

The primary benefit of public transparency is not necessarily that

errors in analytic results will be detected, although error correction

is clearly valuable. The more important benefit of transparency is that

the public will be able to assess how much an agency's analytic result

hinges on the specific analytic choices made by the agency.

Concreteness about analytic choices allows, for example, the

implications of alternative technical choices to be readily assessed.

This type of sensitivity analysis is widely regarded as an essential

feature of high-quality analysis, yet sensitivity analysis cannot be

undertaken by outside parties unless a high degree of transparency is

achieved. The OMB guidelines do not compel such sensitivity analysis as

a necessary dimension of quality, but the transparency achieved by

reproducibility will allow the public to undertake sensitivity studies

of interest.

We acknowledge that confidentiality concerns will sometimes

preclude public access as an approach to reproducibility. In response

to public comment, we have clarified that such concerns do include

interests in ``intellectual property.'' To ensure that the OMB

guidelines have sufficient flexibility with regard to analytic

transparency, OMB has, in new paragraph V.3.b.ii.B.i, provided agencies

an alternative approach for classes or types of analytic results that

cannot practically be subject to the reproducibility standard. ``[In

those situations involving influential scientific, financial, or

statistical information * * * ] making the data and methods publicly

available will assist in determining whether analytic results are

reproducible. However, the objectivity standard does not override other

compelling interests such as privacy, trade secrets, intellectual

property, and other confidentiality protections. '' Specifically, in

cases where reproducibility will not occur due to other compelling

interests, we expect agencies (1) to perform robustness checks

appropriate to the importance of the information involved, e.g.,

determining whether a specific statistic is sensitive to the choice of

analytic method, and, accompanying the information disseminated, to

document their efforts to assure the needed robustness in information

quality, and (2) address in their guidelines the

[[Page 8457]]

degree to which they anticipate the opportunity for reproducibility to

be limited by the confidentiality of underlying data. As we state in

new paragraph V.3.b.ii.B.ii, ``In situations where public access to

data and methods will not occur due to other compelling interests,

agencies shall apply especially rigorous robustness checks to analytic

results and document what checks were undertaken. Agency guidelines

shall, however, in all cases, require a disclosure of the specific data

sources that have been used and the specific quantitative methods and

assumptions that have been employed.''

Given the differences in the many Federal agencies covered by these

guidelines, and the differences in robustness checks and the level of

detail for documentation thereof that might be appropriate for

different agencies, we also believe it will be helpful if agencies

elaborate on these matters in the context of their missions and duties,

with due consideration of the nature of the information they

disseminate. As we state in new paragraph V.3.b.ii.B.ii, ``Each agency

is authorized to define the type of robustness checks, and the level of

detail for documentation thereof, in ways appropriate for it given the

nature and multiplicity of issues for which the agency is

responsible.''

We leave the determination of the appropriate degree of rigor to

the discretion of agencies and the relevant scientific and technical

communities that work with the agencies. We do, however, establish a

general standard for the appropriate degree of rigor in our expanded

definition of ``reproducibility'' in paragraph V.10: ``

`Reproducibility' means that the information is capable of being

substantially reproduced, subject to an acceptable degree of

imprecision. For information judged to have more (less) important

impacts, the degree of imprecision that is tolerated is reduced

(increased).'' OMB will review each agency's treatment of this issue

when reviewing the agency guidelines as a whole.

Comments also expressed concerns regarding interim final paragraph

V.3.B.iii, ``making the data and models publicly available will assist

in determining whether analytic results are capable of being

substantially reproduced,'' and whether it could be interpreted to

constitute public dissemination of these materials, rendering moot the

reproducibility test. (For the equivalent provision, see new paragraph

V.3.b.ii.B.i.) The OMB guidelines do not require agencies to reproduce

each disseminated analytic result by independent reanalysis. Thus,

public dissemination of data and models per se does not mean that the

analytic result has been reproduced. It means only that the result

should be CAPABLE of being reproduced. The transparency associated with

this capability of reproduction is what the OMB guidelines are designed

to achieve.

We also want to build on a general observation that we made in our

final guidelines published in September 2001. In those guidelines we

stated: ``... in those situations involving influential scientific[,

financial,] or statistical information, the substantial reproducibility

standard is added as a quality standard above and beyond some peer

review quality standards'' (66 FR 49722 (September 28, 2001)). A

hypothetical example may serve to illustrate this point. Assume that

two Federal agencies initiated or sponsored the dissemination of five

scientific studies after October 1, 2002 (see paragraph III.4) that

were, before dissemination, subjected to formal, independent, external

peer review, i.e., that met the presumptive standard for

``objectivity'' under paragraph V.3.b.i. Further assume, at the time of

dissemination, that neither agency reasonably expected that the

dissemination of any of these studies would have ``a clear and

substantial impact'' on important public policies, i.e., that these

studies were not considered ``influential'' under paragraph V.9, and

thus not subject to the reproducibility standards in paragraphs

V.3.b.ii.A or B. Then assume, two years later, in 2005, that one of the

agencies decides to issue an important and far-reaching regulation

based clearly and substantially on the agency's evaluation of the

analytic results set forth in these five studies and that such agency

reliance on these five studies as published in the agency's notice of

proposed rulemaking would constitute dissemination of these five

studies. These guidelines would require the rulemaking agency, prior to

publishing the notice of proposed rulemaking, to evaluate these five

studies to determine if the analytic results stated therein would meet

the ``capable of being substantially reproduced'' standards in

paragraph V.3.b.ii.B and, if necessary, related standards governing

original and supporting data in paragraph V.3.b.ii.A. If the agency

were to decide that any of the five studies would not meet the

reproducibility standard, the agency may still rely on them but only if

they satisfy the transparency standard and--as applicable--the

disclosure of robustness checks required by these guidelines.

Otherwise, the agency should not disseminate any of the studies that

did not meet the applicable standards in the guidelines at the time it

publishes the notice of proposed rulemaking.

Some comments suggested that OMB consider replacing the

reproducibility standard with a standard concerning ``confirmation'' of

results for influential scientific and statistical information.

Although we encourage agencies to consider ``confirmation'' as a

relevant standard--at least in some cases--for assessing the

objectivity of original and supporting data, we believe that

``confirmation'' is too stringent a standard to apply to analytic

results. Often the regulatory impact analysis prepared by an agency for

a major rule, for example, will be the only formal analysis of an

important subject. It would be unlikely that the results of the

regulatory impact analysis had already been confirmed by other

analyses. The ``capable of being substantially reproduced'' standard is

less stringent than a ``confirmation'' standard because it simply

requires that an agency's analysis be sufficiently transparent that

another qualified party could replicate it through reanalysis.

Health, Safety, and Environmental Information. We note, in the

scientific context, that in 1996 the Congress, for health decisions

under the Safe Drinking Water Act, adopted a basic standard of quality

for the use of science in agency decisionmaking. Under 42 U.S.C. 300g-

1(b)(3)(A), an agency is directed, ``to the degree that an Agency

action is based on science,'' to use ``(i) the best available, peer-

reviewed science and supporting studies conducted in accordance with

sound and objective scientific practices; and (ii) data collected by

accepted methods or best available methods (if the reliability of the

method and the nature of the decision justifies use of the data).''

We further note that in the 1996 amendments to the Safe Drinking

Water Act, Congress adopted a basic quality standard for the

dissemination of public information about risks of adverse health

effects. Under 42 U.S.C. 300g-1(b)(3)(B), the agency is directed, ``to

ensure that the presentation of information [risk] effects is

comprehensive, informative, and understandable.'' The agency is further

directed, ``in a document made available to the public in support of a

regulation [to] specify, to the extent practicable--(i) each population

addressed by any estimate [of applicable risk effects]; (ii) the

expected risk or central estimate of

[[Page 8458]]

risk for the specific populations [affected]; (iii) each appropriate

upper-bound or lower-bound estimate of risk; (iv) each significant

uncertainty identified in the process of the assessment of [risk]

effects and the studies that would assist in resolving the uncertainty;

and (v) peer-reviewed studies known to the [agency] that support, are

directly relevant to, or fail to support any estimate of [risk] effects

and the methodology used to reconcile inconsistencies in the scientific

data.''

As suggested in several comments, we have included these

congressional standards directly in new paragraph V.3.b.ii.C, and made

them applicable to the information disseminated by all the agencies

subject to these guidelines: ``With regard to analysis of risks to

human health, safety and the environment maintained or disseminated by

the agencies, agencies shall either adopt or adapt the quality

principles applied by Congress to risk information used and

disseminated pursuant to the Safe Drinking Water Act Amendments of 1996

(42 U.S.C. 300g-1(b)(3)(A) & (B)).'' The word ``adapt'' is intended to

provide agencies flexibility in applying these principles to various

types of risk assessment.

Comments also argued that the continued flow of vital information

from agencies responsible for disseminating health and medical

information to medical providers, patients, and the public may be

disrupted due to these peer review and reproducibility standards. OMB

responded by adding to new paragraph V.3.b.ii.C: ``Agencies responsible

for dissemination of vital health and medical information shall

interpret the reproducibility and peer-review standards in a manner

appropriate to assuring the timely flow of vital information from

agencies to medical providers, patients, health agencies, and the

public. Information quality standards may be waived temporarily by

agencies under urgent situations (e.g., imminent threats to public

health or homeland security) in accordance with the latitude specified

in agency-specific guidelines.''

Administrative Correction Mechanisms. In addition to commenting on

the substantive standards in these guidelines, many of the comments

noted that the OMB guidelines on the administrative correction of

information do not specify a time period in which the agency

investigation and response must be made. OMB has added the following

new paragraph III.3.i to direct agencies to specify appropriate time

periods in which the investigation and response need to be made.

``Agencies shall specify appropriate time periods for agency decisions

on whether and how to correct the information, and agencies shall

notify the affected persons of the corrections made.''

Several comments stated that the OMB guidelines needed to direct

agencies to consider incorporating an administrative appeal process

into their administrative mechanisms for the correction of information.

OMB agreed, and added the following new paragraph III.3.ii: ``If the

person who requested the correction does not agree with the agency's

decision (including the corrective action, if any), the person may file

for reconsideration within the agency. The agency shall establish an

administrative appeal process to review the agency's initial decision,

and specify appropriate time limits in which to resolve such requests

for reconsideration.'' Recognizing that many agencies already have a

process in place to respond to public concerns, it is not necessarily

OMB's intent to require these agencies to establish a new or different

process. Rather, our intent is to ensure that agency guidelines specify

an objective administrative appeal process that, upon furthercomplaint

by the affected person, reviews an agency's decision to disagree with

the correction request. An objective process will ensure that the

office that originally disseminates the information does not have

responsibility for both the initial response and resolution of a

disagreement. In addition, the agency guidelines should specify that if

the agency believes other agencies may have an interest in the

resolution of any administrative appeal, the agency should consult with

those other agencies about their possible interest.

Overall, OMB does not envision administrative mechanisms that would

burden agencies with frivolous claims. Instead, the correction process

should serve to address the genuine and valid needs of the agency and

its constituents without disrupting agency processes. Agencies, in

making their determination of whether or not to correct information,

may reject claims made in bad faith or without justification, and are

required to undertake only the degree of correction that they conclude

is appropriate for the nature and timeliness of the information

involved, and explain such practices in their annual fiscal year

reports to OMB.

OMB's issuance of these final guidelines is the beginning of an

evolutionary process that will include draft agency guidelines, public

comment, final agency guidelines, development of experience with OMB

and agency guidelines, and continued refinement of both OMB and agency

guidelines. Just as OMB requested public comment before issuing these

final guidelines, OMB will refine these guidelines as experience

develops and further public comment is obtained.

Dated: December 21, 2001.

John D. Graham,

Administrator, Office of Information and Regulatory Affairs.

Guidelines for Ensuring and Maximizing the Quality, Objectivity,

Utility, and Integrity of Information Disseminated by Federal

Agencies

I. OMB Responsibilities

Section 515 of the Treasury and General Government Appropriations

Act for FY2001 (Public Law 106-554) directs the Office of Management

and Budget to issue government-wide guidelines that provide policy and

procedural guidance to Federal agencies for ensuring and maximizing the

quality, objectivity, utility, and integrity of information, including

statistical information, disseminated by Federal agencies.

II. Agency Responsibilities

Section 515 directs agencies subject to the Paperwork Reduction Act

(44 U.S.C. 3502(1)) to--

1. Issue their own information quality guidelines ensuring and

maximizing the quality, objectivity, utility, and integrity of

information, including statistical information, disseminated by the

agency no later than one year after the date of issuance of the OMB

guidelines;

2. Establish administrative mechanisms allowing affected persons to

seek and obtain correction of information maintained and disseminated

by the agency that does not comply with these OMB guidelines; and

3. Report to the Director of OMB the number and nature of

complaints received by the agency regarding agency compliance with

these OMB guidelines concerning the quality, objectivity, utility, and

integrity of information and how such complaints were resolved.

III. Guidelines for Ensuring and Maximizing the Quality, Objectivity,

Utility, and Integrity of Information Disseminated by Federal Agencies

1. Overall, agencies shall adopt a basic standard of quality

(including objectivity, utility, and integrity) as a performance goal

and should take appropriate steps to incorporate information quality

criteria into agency information dissemination practices. Quality is to

be ensured and established at levels appropriate to the nature and

timeliness of the information to be disseminated. Agencies shall adopt

[[Page 8459]]

specific standards of quality that are appropriate for the various

categories of information they disseminate.

2. As a matter of good and effective agency information resources

management, agencies shall develop a process for reviewing the quality

(including the objectivity, utility, and integrity) of information

before it is disseminated. Agencies shall treat information quality as

integral to every step of an agency's development of information,

including creation, collection, maintenance, and dissemination. This

process shall enable the agency to substantiate the quality of the

information it has disseminated through documentation or other means

appropriate to the information.

3. To facilitate public review, agencies shall establish

administrative mechanisms allowing affected persons to seek and obtain,

where appropriate, timely correction of information maintained and

disseminated by the agency that does not comply with OMB or agency

guidelines. These administrative mechanisms shall be flexible,

appropriate to the nature and timeliness of the disseminated

information, and incorporated into agency information resources

management and administrative practices.

i. Agencies shall specify appropriate time periods for agency

decisions on whether and how to correct the information, and agencies

shall notify the affected persons of the corrections made.

ii. If the person who requested the correction does not agree with

the agency's decision (including the corrective action, if any), the

person may file for reconsideration within the agency. The agency shall

establish an administrative appeal process to review the agency's

initial decision, and specify appropriate time limits in which to

resolve such requests for reconsideration.

4. The agency's pre-dissemination review, under paragraph III.2,

shall apply to information that the agency first disseminates on or

after October 1, 2002. The agency's administrative mechanisms, under

paragraph III.3., shall apply to information that the agency

disseminates on or after October 1, 2002, regardless of when the agency

first disseminated the information.

IV. Agency Reporting Requirements

1. Agencies must designate the Chief Information Officer or another

official to be responsible for agency compliance with these guidelines.

2. The agency shall respond to complaints in a manner appropriate

to the nature and extent of the complaint. Examples of appropriate

responses include personal contacts via letter or telephone, form

letters, press releases or mass mailings that correct a widely

disseminated error or address a frequently raised complaint.

3. Each agency must prepare a draft report, no later than April 1,

2002, providing the agency's information quality guidelines and

explaining how such guidelines will ensure and maximize the quality,

objectivity, utility, and integrity of information, including

statistical information, disseminated by the agency. This report must

also detail the administrative mechanisms developed by that agency to

allow affected persons to seek and obtain appropriate correction of

information maintained and disseminated by the agency that does not

comply with the OMB or the agency guidelines.

4. The agency must publish a notice of availability of this draft

report in the Federal Register, and post this report on the agency's

website, to provide an opportunity for public comment.

5. Upon consideration of public comment and after appropriate

revision, the agency must submit this draft report to OMB for review

regarding consistency with these OMB guidelines no later than July 1,

2002. Upon completion of that OMB review and completion of this report,

agencies must publish notice of the availability of this report in its

final form in the Federal Register, and post this report on the

agency's web site no later than October 1, 2002.

6. On an annual fiscal-year basis, each agency must submit a report

to the Director of OMB providing information (both quantitative and

qualitative, where appropriate) on the number and nature of complaints

received by the agency regarding agency compliance with these OMB

guidelines and how such complaints were resolved. Agencies must submit

these reports no later than January 1 of each following year, with the

first report due January 1, 2004.

V. Definitions

1. ``Quality'' is an encompassing term comprising utility,

objectivity, and integrity. Therefore, the guidelines sometimes refer

to these four statutory terms, collectively, as ``quality.''

2. ``Utility'' refers to the usefulness of the information to its

intended users, including the public. In assessing the usefulness of

information that the agency disseminates to the public, the agency

needs to consider the uses of the information not only from the

perspective of the agency but also from the perspective of the public.

As a result, when transparency of information is relevant for assessing

the information's usefulness from the public's perspective, the agency

must take care to ensure that transparency has been addressed in its

review of the information.

3. ``Objectivity'' involves two distinct elements, presentation and

substance.

a. ``Objectivity'' includes whether disseminated information is

being presented in an accurate, clear, complete, and unbiased manner.

This involves whether the information is presented within a proper

context. Sometimes, in disseminating certain types of information to

the public, other information must also be disseminated in order to

ensure an accurate, clear, complete, and unbiased presentation. Also,

the agency needs to identify the sources of the disseminated

information (to the extent possible, consistent with confidentiality

protections) and, in a scientific, financial, or statistical context,

the supporting data and models, so that the public can assess for

itself whether there may be some reason to question the objectivity of

the sources. Where appropriate, data should have full, accurate,

transparent documentation, and error sources affecting data quality

should be identified and disclosed to users.

b. In addition, ``objectivity'' involves a focus on ensuring

accurate, reliable, and unbiased information. In a scientific,

financial, or statistical context, the original and supporting data

shall be generated, and the analytic results shall be developed, using

sound statistical and research methods.

i. If data and analytic results have been subjected to formal,

independent, external peer review, the information may generally be

presumed to be of acceptable objectivity. However, this presumption is

rebuttable based on a persuasive showing by the petitioner in a

particular instance. If agency-sponsored peer review is employed to

help satisfy the objectivity standard, the review process employed

shall meet the general criteria for competent and credible peer review

recommended by OMB-OIRA to the President's Management Council (9/20/01)

(http://www.whitehouse.gov/omb/inforeg/oira_review-process.html),

namely, ``that (a) peer reviewers be selected primarily on the basis of

necessary technical expertise, (b) peer reviewers be expected to

disclose to agencies prior technical/policy positions they may have

taken on the issues at hand, (c) peer reviewers be expected to disclose

to agencies their sources of personal and

[[Page 8460]]

institutional funding (private or public sector), and (d) peer reviews

be conducted in an open and rigorous manner.''

ii. If an agency is responsible for disseminating influential

scientific, financial, or statistical information, agency guidelines

shall include a high degree of transparency about data and methods to

facilitate the reproducibility of such information by qualified third

parties.

A. With regard to original and supporting data related thereto,

agency guidelines shall not require that all disseminated data be

subjected to a reproducibility requirement. Agencies may identify, in

consultation with the relevant scientific and technical communities,

those particular types of data that can practicable be subjected to a

reproducibility requirement, given ethical, feasibility, or

confidentiality constraints. It is understood that reproducibility of

data is an indication of transparency about research design and methods

and thus a replication exercise (i.e., a new experiment, test, or

sample) shall not be required prior to each dissemination.

B. With regard to analytic results related thereto, agency

guidelines shall generally require sufficient transparency about data

and methods that an independent reanalysis could be undertaken by a

qualified member of the public. These transparency standards apply to

agency analysis of data from a single study as well as to analyses that

combine information from multiple studies.

i. Making the data and methods publicly available will assist in

determining whether analytic results are reproducible. However, the

objectivity standard does not override other compelling interests such

as privacy, trade secrets, intellectual property, and other

confidentiality protections.

ii. In situations where public access to data and methods will not

occur due to other compelling interests, agencies shall apply

especially rigorous robustness checks to analytic results and document

what checks were undertaken. Agency guidelines shall, however, in all

cases, require a disclosure of the specific data sources that have been

used and the specific quantitative methods and assumptions that have

been employed. Each agency is authorized to define the type of

robustness checks, and the level of detail for documentation thereof,

in ways appropriate for it given the nature and multiplicity of issues

for which the agency is responsible.

C. With regard to analysis of risks to human health, safety and the

environment maintained or disseminated by the agencies, agencies shall

either adopt or adapt the quality principles applied by Congress to

risk information used and disseminated pursuant to the Safe Drinking

Water Act Amendments of 1996 (42 U.S.C. 300g-1(b)(3)(A) & (B)).

Agencies responsible for dissemination of vital health and medical

information shall interpret the reproducibility and peer-review

standards in a manner appropriate to assuring the timely flow of vital

information from agencies to medical providers, patients, health

agencies, and the public. Information quality standards may be waived

temporarily by agencies under urgent situations (e.g., imminent threats

to public health or homeland security) in accordance with the latitude

specified in agency-specific guidelines.

4. ``Integrity'' refers to the security of information--protection

of the information from unauthorized access or revision, to ensure that

the information is not compromised through corruption or falsification.

5. ``Information'' means any communication or representation of

knowledge such as facts or data, in any medium or form, including

textual, numerical, graphic, cartographic, narrative, or audiovisual

forms. This definition includes information that an agency disseminates

from a web page, but does not include the provision of hyperlinks to

information that others disseminate. This definition does not include

opinions, where the agency's presentation makes it clear that what is

being offered is someone's opinion rather than fact or the agency's

views.

6. ``Government information'' means information created, collected,

processed, disseminated, or disposed of by or for the Federal

Government.

7. ``Information dissemination product'' means any books, paper,

map, machine-readable material, audiovisual production, or other

documentary material, regardless of physical form or characteristic, an

agency disseminates to the public. This definition includes any

electronic document, CD-ROM, or web page.

8. ``Dissemination'' means agency initiated or sponsored

distribution of information to the public (see 5 CFR 1320.3(d)

(definition of ``Conduct or Sponsor'')). Dissemination does not include

distribution limited to government employees or agency contractors or

grantees; intra- or inter-agency use or sharing of government

information; and responses to requests for agency records under the

Freedom of Information Act, the Privacy Act, the Federal Advisory

Committee Act or other similar law. This definition also does not

include distribution limited to correspondence with individuals or

persons, press releases, archival records, public filings, subpoenas or

adjudicative processes.

9. ``Influential'', when used in the phrase ``influential

scientific, financial, or statistical information'', means that the

agency can reasonably determine that dissemination of the information

will have or does have a clear and substantial impact on important

public policies or important private sector decisions. Each agency is

authorized to define ``influential'' in ways appropriate for it given

the nature and multiplicity of issues for which the agency is

responsible.

10. ``Reproducibility'' means that the information is capable of

being substantially reproduced, subject to an acceptable degree of

imprecision. For information judged to have more (less) important

impacts, the degree of imprecision that is tolerated is reduced

(increased). If agencies apply the reproducibility test to specific

types of original or supporting data, the associated guidelines shall

provide relevant definitions of reproducibility (e.g., standards for

replication of laboratory data). With respect to analytic results,

``capable of being substantially reproduced'' means that independent

analysis of the original or supporting data using identical methods

would generate similar analytic results, subject to an acceptable

degree of imprecision or error.

[FR Doc. 02-59 Filed 1-2-02; 1:36 pm]

billing code 3110-01-M

Editorial Note: Due to numerous errors, this document is being

reprinted in its entirety. It was originally printed in the Federal

Register on Thursday, January 3, 2002 at 67 FR 369-378 and was

corrected on Tuesday, February 5, 2002 at 67 FR 5365.

[FR Doc. R2-59 Filed 2-21-02; 8:45 am]

BILLING CODE 1505-01-D