Report of Tabulations of Population to States and Localities Pursuant to Title 13 U.S.C., Section 141(c), and Availability of Other Population Information; the Executive Steering Committee for Accuracy and Coverage Evaluation Policy (ESCAP) Report; and the Census Bureau Director's Recommendation

Federal RegisterMar 8, 2001

Ask Donna

What actually matters in this document.

Text

DEPARTMENT OF COMMERCE

Bureau of the Census

Report of Tabulations of Population to States and Localities Pursuant to Title 13 U.S.C., Section 141(c), and Availability of Other Population Information; the Executive Steering Committee for Accuracy and Coverage Evaluation Policy (ESCAP) Report; and the Census Bureau Director's Recommendation

AGENCY:

Bureau of the Census.

ACTION:

Notice of recommendation and report.

SUMMARY:

This notice provides the United States Census Bureau (Census Bureau) Director's recommendation on methodology and the Executive Steering Committee on Accuracy and Coverage Evaluation (A.C.E.) Policy (ESCAP) report analyzing the methodologies that may be used in making the tabulations of population reported to states and localities pursuant to Title 13 U.S.C., Section 141(c), and the factors relevant to the possible choices of methodology. Concurrent with this notice to the public, the Census Bureau Director's recommendation and the ESCAP report have been delivered to the Secretary of Commerce. The recommendation and the report are attached as exhibits to the

SUPPLEMENTARY INFORMATION

section of this notice. In addition to publication in the

Federal Register

, the recommendation and the report will be posted on the Census Bureau Web site at <

http://www.census.gov/dmd/www/2khome.htm>.

FOR FURTHER INFORMATION CONTACT:

John H. Thompson, Associate Director for Decennial Census, U.S. Census Bureau, SFC-2, Room 2018, Washington, DC 20233. Telephone: (301) 457-3946; fax: (301) 457-3024.

SUPPLEMENTARY INFORMATION:

Background Information

The decennial census is mandated by the United States Constitution (Article I, Section 2, Clause 3) to provide the population counts needed to apportion the seats in the United States House of Representatives among the states. By December 28, 2000, the Census Bureau fulfilled its Constitutional duty by delivering to the Secretary of Commerce the state population totals used for congressional apportionment. In accordance with the January 25, 1999, Supreme Court ruling,

Department of Commerce

House of Representatives

, 119 S.Ct. 765 (1999), the Census Bureau did not use statistical sampling to produce the state population totals used for congressional apportionment.

However, the Census Bureau did consider the use of statistical methods to produce the more detailed data required for legislative redistricting. The Census Bureau designed the A.C.E. to permit correction of the initial census results to account for systematic patterns of net undercount and net overcount. The Census Bureau preliminarily determined that the A.C.E., if properly conducted, should produce more accurate census data by improving coverage and reducing differential undercounts. A senior-level committee, the Executive Steering Committee for A.C.E. Policy (ESCAP), was formed to evaluate whether the data produced in Census 2000 support this initial determination. The ESCAP used analysis from reports on topics chosen for their usefulness in informing the decision on the suitability of using the A.C.E. data for legislative redistricting. The Committee also drew upon work from other Census Bureau staff, as appropriate.

As required by final rule, Title 15, Code of Federal Regulations, Part 101, issued by the Secretary of Commerce (66 FR 11232, February 23, 2001), the ESCAP has submitted its report (attached below), accompanied by the recommendation of the Director of the Census Bureau to the Secretary of Commerce. The Secretary will make the final determination regarding the methodology to be used in calculating the tabulations of population reported to states and localities for legislative redistricting. By April 1, 2001, the Census Bureau must provide these tabulations, as required by Public Law 94-171, to each state so that they can redraw congressional, state, and local legislative districts.

Dated: March 1, 2001.

William G. Barron, Jr.,

Acting Director, Bureau of the Census.

Attachment 1 to Preamble

March 1, 2001

Memorandum for Donald L. Evans, Secretary of Commerce

From: William G. Barron, Jr., Acting Director

Subject: Recommendation on Adjustment of Census Counts

I am forwarding the report of the Executive Steering Committee for A.C.E. Policy (ESCAP) on whether the Accuracy and Coverage Evaluation (A.C.E.) should be used to adjust the Census 2000 counts. I asked the ESCAP to provide a recommendation in its report because I rely on the knowledge, experience, and technical expertise of the Committee and Census Bureau staff who have worked extremely hard with tremendous dedication and expertise through every phase of Census 2000.

As a member of the ESCAP and as Acting Director, I concur with and approve the Committee's recommendation that unadjusted census data be released as the Census Bureau's official redistricting data. The law requires that the Census Bureau issue data for use in redistricting by April 1, 2001 (13 U.S.C. 141(c)). The Committee reached this recommendation because it is unable, based on the data and other information currently available, to conclude that the adjusted data are more accurate for use in redistricting. The primary reason for arriving at this conclusion is the apparent inconsistency in population growth over the decade as estimated by the A.C.E. and demographic analysis. These differences cannot be resolved in the time available for the Committee's work. The importance of completing this type of analysis has been emphasized clearly and explicitly in the Census Bureau's public presentations outlining the scope, intent, and purpose of ESCAP deliberations. For example, the June 2000 Feasibility Document contained various references to the importance of demographic analysis and demographic estimates as key components of data and analysis to inform the ESCAP recommendation. This point was reinforced in materials the Census Bureau presented on October 2, 2000, at a public workshop sponsored by the National Academy of Sciences. The inconsistency raises the possibility of an unidentified error in the A.C.E. estimates or Census 2000. This possibility cannot be eliminated by the legally mandated deadline.

I believe the attached report and this cover memo meet the requirements set forth in regulation 66 Fed. Reg. 11231 (February 23, 2001), “Report of Tabulations of Population to States and Localities Pursuant to 13 U.S.C. 141(c) and Availability of Other Population Information; Revocation of Delegation of Authority.”

Please let me know if I can provide you with additional information on these matters.

Attachment 2 to Preamble

Report of the Executive Steering Committee for Accuracy and Coverage Evaluation Policy

Recommendation Concerning the Methodology to be Used in Producing the Tabulations of Population Reported to States and Localities Pursuant to 13 U.S.C. 141(c), March 1, 2001

Recommendation

The Executive Steering Committee for A.C.E. Policy (ESCAP) is unable to conclude, based on the information available at this time, that the adjusted Census 2000 data are more accurate for redistricting. Accordingly, ESCAP recommends that the unadjusted census data be released as the Census Bureau's official redistricting data.

The Census Bureau publicly set forth the criteria it would use to evaluate the success of the Accuracy and Coverage Evaluation (A.C.E.), stating that the adjustment decision would be based on: (1) a consideration of operational data to validate the successful conduct of the A.C.E.; (2) whether the A.C.E. measures of undercount were consistent with historical patterns of undercount and independent demographic analysis benchmarks; and (3) a review of quality measures.

The ESCAP spent many weeks examining voluminous evidence, and has debated at great length whether adjustment based on the A.C.E. would improve Census 2000 data for use in redistricting. As described in the following Report, the Committee considered a wide variety of evidence relating to the accuracy of Census 2000 and the A.C.E. After careful consideration of the data, the Committee has concluded that there is considerable evidence to support the use of adjusted data, and that Census 2000 and A.C.E. operations were well designed and conducted. However, demographic analysis comparisons, and possible issues related to synthetic and balancing error, preclude a determination at this time that the adjusted data are more accurate.

As described in detail in the Report, demographic analysis indicates fundamental differences with the A.C.E. In particular, demographic analysis estimates are significantly lower than the A.C.E. estimates for important population groups. The Committee investigated this inconsistency extensively, but in the time available could not adequately explain the result.

The inconsistency between the A.C.E. and the demographic analysis estimates is most likely the result of one or more of the following three scenarios:

1. The estimates from the 1990 census coverage measurement survey (the Post-Enumeration Survey), the 1990 demographic analysis estimates, and the 1990 census were far below the Nation's true population on April 1, 1990. This scenario means that the 1990 census undercounted the population by a significantly greater amount and degree than previously believed, but that Census 2000 included portions of this previously un-enumerated population.

2. Demographic analysis techniques to project population growth between 1990 and 2000 do not capture the full measure of the Nation's growth.

3. Census 2000, as corrected by the A.C.E., overestimates the Nation's population.

The inconsistency between the demographic analysis estimates and the A.C.E. estimates raises the possibility of an as-yet undiscovered problem in the A.C.E. or census methodology, scenario 3, above. The Census Bureau must further investigate this inconsistency, and the possibility of a methodological error, before it can recommend that adjustment would improve accuracy. Similarly, concerns with synthetic and balancing error must be more fully investigated and addressed.

The ESCAP's recommendation to use the unadjusted data was a difficult one. The Committee conducted a number of analyses directed at understanding the inconsistency with demographic analysis and the synthetic and balancing error issues, but could not find a complete explanation in the time available. The Committee believes it likely that further research may establish that adjustment based on the A.C.E. would result in improved accuracy. However, the uncertainty due to these concerns is too large at this time to allow for a recommendation to adjust. The Committee believes that further research will verify that Census 2000 improved on the coverage levels of past censuses, but that the unadjusted census totals will still reflect a net national undercount. The Committee further believes the evidence will confirm that the differential undercount (the lower than average coverage of minorities, renters, and children) was reduced, but not eliminated, in Census 2000.

The ESCAP finds that both the census and the A.C.E. were efficient and effective operations that produced high quality data. The Committee is proud of the Census Bureau's design work on both the census and the A.C.E. and believes that both produced measurably better results. The high quality of the census has made the adjustment decision more difficult than in 1990. The closeness of the A.C.E. and the census heightens the concern that an undiscovered problem with Census 2000 or the A.C.E. will result in a decrease in accuracy from adjustment. Today's recommendation is, however, in no way a reflection of weaknesses in data quality or in the quality of staff work.

The ESCAP makes this recommendation in light of the information now available. Additional evaluations, research, and analysis may allow the Census Bureau to resolve the noted concerns. The Census Bureau will continue to investigate these issues and will make the results of this research available, as is consistent with the Bureau's long-standing policy of openness.

Executive Summary

The ESCAP cannot recommend adjustment at this time

. The Executive Steering Committee for Accuracy and Coverage Evaluation (A.C.E.) Policy (ESCAP) is required by regulation to prepare a written report analyzing the methodologies and factors involved in the adjustment decision. The Acting Director of the Census Bureau asked the ESCAP to include a recommendation in its Report. The ESCAP spent many weeks examining voluminous evidence, and has debated at great length whether adjustment would improve Census 2000 data for use in redistricting. After having evaluated a wide variety of evidence relating to the accuracy of Census 2000, and developed an extensive record of its deliberations, the ESCAP is unable to conclude, based on the information available at this time, that the adjusted Census 2000 data are more accurate for redistricting.

While the majority of the evidence indicates both the continued existence of a differential undercount of the population and the superior accuracy of the adjusted numbers, the ESCAP has concerns. There is a significant inconsistency between the A.C.E. estimates and demographic analysis estimates. Additionally, possible synthetic and balancing errors may affect the accuracy of the adjusted numbers. Until these concerns are more fully investigated and addressed, the ESCAP cannot recommend using adjustment. Accordingly, ESCAP has recommended that unadjusted census data be released as the Census Bureau's official redistricting data.

The ESCAP makes this recommendation in light of the information now available

. Additional evaluations, research, and analysis may alleviate these concerns and support the evidence that indicates the superior accuracy of the adjusted data. Accordingly, the Census Bureau intends to continue its research into these concerns.

The Census Bureau relied on three prespecified decision criteria

. The ESCAP based its adjustment recommendation on: (1) a consideration of operational data to validate the successful conduct of the A.C.E.; (2) whether the A.C.E. measures of undercount were consistent with historical patterns of undercount and independent demographic analysis benchmarks; and (3) a review of quality measures. These criteria were specified in advance in the Census Bureau's June, 2000 “Accuracy and Coverage Evaluation: Statement on the Feasibility of Using Statistical Methods to Improve the Accuracy of Census 2000.”

Both Census 2000 and the A.C.E. were of high quality

. The ESCAP's recommendation against adjustment in no way suggests serious concern about the quality of the census or the A.C.E. operations, as the ESCAP believes that both Census 2000 and the A.C.E. were efficient and effective operations that produced high quality data. All major programs in the census were completed on schedule and within budget, and design improvements in both Census 2000 and the A.C.E. produced measurably better results. An innovative advertising and partnership program encouraged public participation, and adequate staffing and pay contributed to improved data quality. The ESCAP concludes that the unadjusted census data are of high quality.

The A.C.E. was also a design and operational success. The A.C.E. included a variety of design improvements that resulted in better data quality, including enhanced computer processing and bettering matching. The Census 2000 adjusted data have lower variances and comparable or improved missing data rates compared to the 1990 adjusted data. The Census Bureau followed the A.C.E.”s prespecified design except for two specific instances that are easily explained by good and normal statistical practice. Both of these changes should be considered enhancements. The ESCAP has concluded that both Census 2000 and the A.C.E. were effective and efficient operations.

Demographic analysis estimates were inconsistent with the adjusted data

. The demographic analysis estimates indicate fundamental differences with the results of the A.C.E. In particular, the demographic analysis estimates are significantly lower than both Census 2000 and the A.C.E. estimates for important population groups. The Committee investigated this inconsistency extensively, but in the time available could not adequately explain it. The inconsistency between the A.C.E. and the demographic analysis estimates is most likely the result of one or more of the following three scenarios:

2. Demographic analysis techniques to project population growth between 1990 and 2000 do not capture the full measure of the Nation's growth.

3. Census 2000, as corrected by the A.C.E., overestimates the Nation's population.

Quality measures indicate the adjusted data are more accurate overall, but concerns were identified

. The ESCAP directed the preparation of several total error models and loss function analyses to evaluate whether the adjusted data are more accurate than the unadjusted data. The Committee examined the loss functions for evidence of a clearly measurable improvement under a variety of scenarios and found the following:

1. Under what the Committee considered reasonable assumptions, state, congressional district, and county level analyses showed a marked improvement for adjustment.

2. However, some less likely scenarios indicated that the unadjusted census was more accurate at all geographic levels.

3. The analysis of accuracy for counties with populations below 100,000 people indicated that the unadjusted census was more accurate.

The ESCAP believes that under reasonable scenarios, and absent the concerns noted above, adjustment would result in more accurate data at the state, congressional district, and county levels. Even though smaller counties would have been less accurate, the analysis indicated an overall improvement in accuracy from adjustment. However, the concerns noted above are all potentially indicative of undetected problems. The ESCAP is unable to conclude at this time that the adjusted data are superior because further research on these concerns could reverse the finding of the adjusted data's superior accuracy.

The ESCAP assessed other factors that might affect accuracy

. The ESCAP examined the issues of synthetic and balancing error and concluded that the potential for these errors cannot be ignored, particularly when considered in conjunction with the inconsistency with demographic analysis. Finally, the ESCAP reviewed the treatment of late census additions and whole person imputations, because the number of these cases significantly increased from 1990, concluding that these cases did not raise serious new concerns.

Additional issues were considered

. The ESCAP reiterated that the Census Bureau does not consider block-level accuracy to be an important criterion with which to evaluate either Census 2000 or the A.C.E., and explained that had adjusted data files been released, adjustments for overcounts would not have resulted in the removal of any records from Census 2000 files.

Table of Contents

Introduction

Census and A.C.E. Results in Brief

ESCAP Procedure and Process

Findings

Conduct of Key Operations

Census Quality Indicators

Address List Development

Questionnaire Return—Census 2000 Mail Return Rates

Nonresponse Follow-up

Housing Unit Unduplication Program

Data Processing

A.C.E. Quality Indicators

Historical Measures of Census Coverage-Comparison with Demographic Analysis

Measures of Census and A.C.E. Quality

Total Error Model

Loss Function Analysis

Other Factors That May Affect Accuracy

Synthetic Error

Balancing Error

Late Adds and Whole Person Imputations

Misclassification Error

Additional Issues

Block Level Accuracy

Adjustments for Overcounts

Attachments

Introduction

This report fulfills the responsibility of the Executive Steering Committee for Accuracy and Coverage Evaluation (A.C.E.) Policy (“the ESCAP” or “the Committee”) to prepare a “written report to the Director of the Census analyzing the methodologies that may be used in making the tabulations of population reported to States and localities pursuant to 13 U.S.C. 141(c), and the factors relevant to the possible choices of methodology.”

As is required by regulation, the Director of the Census will forward this report and his recommendation regarding adjustment to the Secretary of Commerce. This report is also being released to the public at the same time that it is being forwarded to the Secretary of Commerce.

The Secretary of Commerce will make the final determination about whether to adjust the data that will be released pursuant to P.L. 94-171.

The phrase “the methodologies that may be used in making the tabulations of population reported to States and localities pursuant to 13 U.S.C. 141(c)” refers to the decision about whether the Census Bureau should release adjusted or unadjusted data for the states to use in redistricting. Rather than repeating this cumbersome legal phrase, this document will often refer simply to “the adjustment decision.”

In addition to the requirement to make this report public, the Census Bureau firmly believes that full disclosure and a vigorous and informed debate will improve both the Census Bureau's internal processes and the public's understanding of statistical adjustment. Accordingly, the Bureau is also making available on its Internet site the documentation supporting the ESCAP report. This additional documentation includes the analytical reports outlined publicly to the National Academy of Sciences Panel to Review the 2000 Census in October, 2000, along with underlying data, analysis, and supporting documentation. An index to the supporting documentation is attached.

The Census Bureau released in June 2000 the report “Accuracy and Coverage Evaluation: Statement on the Feasibility of Using Statistical Methods to Improve the Accuracy of Census 2000,” (the Feasibility Document). The Feasibility Document stated that “the Census Bureau will make the determination to use the A.C.E. to correct Census 2000 after evaluating (1) the conduct of key operations, (2) the consistency of the A.C.E. to historical measures of undercount, and (3) measures of quality.”

This report will, accordingly, evaluate the conduct of key operations, compare the Accuracy and Coverage Evaluation Survey (A.C.E.) estimates to historic measures of the undercount, and evaluate the quality of both the A.C.E. and the census.

Feasibility Document, p. 33.

Census and A.C.E. Results in Brief

As the Census Bureau has stated publicly, Census 2000 was an operational success, meeting or exceeding goals. This success may be attributed to a number of improvements, including the following:

• A multi-faceted marketing and partnership program that encouraged householders to complete and mail back their census forms,

• The ability to hire and retain enough highly skilled temporary staff throughout the course of the census, permitting timely completion of operations,

• The timely completion of nonresponse follow-up, which provided sufficient time and resources to conduct other operations designed to improve coverage, and

• The use of digital imaging and optical character recognition technology for the first time to recognize handwritten answers in addition to marks on the form, a vast improvement that allowed the Census Bureau to process the data faster and permitted multiple response options.

The A.C.E. was also an operational success that met or exceeded goals. The A.C.E. was completed on time and generally produced data equal or superior in quality to prior coverage measurement surveys.

The A.C.E. supports the conclusion that the quality of the initial census was generally good, finding that Census 2000 reduced both net and differential undercoverage from 1990 census levels. The A.C.E. estimates that the net national undercount was reduced from the 1990 rate of 1.61 percent to 1.18 percent in 2000.

This reduction is substantial and reflects high census quality. The A.C.E. further found that not only was the net undercount reduced, but there was a reduction in the differential undercount. According to the 1990 Post-Enumeration Survey, minorities, renters, and children were differentially undercounted in the 1990 census, and other methods indicate a differential undercoverage of minorities in earlier censuses. While these groups still have higher undercount rates than the population as a whole, the differential has dropped considerably.

These figures compare the 1990 and 2000 undercount rates as measured by coverage measurement surveys. The coverage measurement survey conducted in connection with the 1990 census was called the Post-Enumeration Survey (PES). As will be discussed below, Demographic Analysis presents an alternative measure of census coverage.

The A.C.E. did not judge Census 2000 quality to be perfect, however. The A.C.E. indicated that while differential coverage was reduced, it was not eliminated, and that Census 2000 continued longstanding patterns of differential coverage, with minority groups, renters, and children all exhibiting lower coverage rates.

The percent net undercount for owners was 0.44 percent compared to 2.75 percent for renters, and the non-Hispanic White undercount rate of 0.67 percent was lower than the rates for non-Hispanic Blacks (2.17 percent) and Hispanics (2.85 percent).

Coverage measurement surveys such as the A.C.E. are not the only method available to estimate census coverage; the Census Bureau also uses demographic analysis (DA) to assess net and differential population coverage. DA uses records and estimates of births, deaths, legal immigration, and Medicare enrollments, and estimates of emigration and net undocumented immigration to estimate the national population, separately from the census. The Census Bureau has long relied on DA as an important independent benchmark for validation of the accuracy of both the census and coverage measurement surveys such as the A.C.E. Initial DA results, however, presented a major inconsistency with the A.C.E. results—instead of confirming a net undercount, DA estimates that Census 2000 overcounted the national population by 1.8 million individuals. Even an alternative DA that assumed a doubling of net undocumented immigration during the 1990's (compared with the initial DA) showed a small net undercount of 0.9 million, substantially below the net undercount of 3.3 million shown by the A.C.E. These inconsistencies and DA in general will be discussed in more detail later in this report. The DA and A.C.E. estimates did agree, however, that Census 2000 perpetuated the historical phenomenon of the differential undercount.

The following table sets forth the A.C.E.'s results in summary fashion:

Table 1

.—

Percent Net Undercount for Major Groups: 2000 A.C.E.

Estimation grouping

Net undercount

(percent)

Standard Error

(percent)

Total population in Households

1.18

0.13

Race and Hispanic Origin:

American Indian and Alaska Native (on reservation)

4.74

1.20

American Indian and alaska Native (off reservation)

3.28

1.33

Hispanic Origin (of any race)

2.85

0.38

Black or African American (not Hispanic)

2.17

0.35

Native Hawaiian and Other Pacific Islander (not Hispanic)

4.60

2.77

Asian (not Hispanic)

0.96

0.64

White or Some Other Race (not Hispanic)

0.67

0.14

Age and Sex:

Under 18 years

1.54

0.19

18 to 29 years:

Male

3.77

0.32

Female

2.23

0.29

30 to 49 years:

Male

1.86

0.19

Female

0.96

0.17

50 years and over:

Male

−0.25

0.18

Female

−0.79

0.17

Housing Tenure:

In owner-occupied housing units

0.44

0.14

In nonowner-occupied units

2.75

0.26

Notes:

• The race and Hispanic categories shown on this table represent estimation groupings used in developing estimates based on the A.C.E. Survey and do not conform with race and Hispanic categories that will appear in the redistricting (P.L. 94-171) files and other Census 2000 data products. In developing the estimation groupings used to evaluate the coverage of Census 2000, the principal consideration was to combine people who were expected to have the same probability of being counted in Census 2000. Consequently, the race and Hispanic origin groupings used to create the A.C.E. estimates of coverage are exceedingly complex. For a complete description of the estimation groups, see DSSD Memorandum Q-37, which will be provided on request.

• In general, American Indians and Alaska Natives (AIAN) are included in that category, regardless of whether they marked another race or are Hispanic. A few exceptions apply, especially for those who do not live on a reservation, on trust lands, or in an AIAN statistical area.

• Similarly, Native Hawaiians and Other Pacific Islanders (NHPI) generally are included in that category, unless they lived outside of Hawaii and marked more than one race or marked Hispanic.

• Hispanics are mostly in that category, unless they marked AIAN and lived on a reservation, on trust lands, or in an AIAN statistical area, or marked NHPI and lived in Hawaii.

• People who marked Black or African American are generally in that category unless they fell in the categories described above; similarly those who marked Asian are generally in that category, unless they fell in the categories described above.

• The final category includes most people who marked only White or only Some Other Race or marked three or more races but did not fall into the categories described above.

• The data in this table contain sampling and non-sampling error; a minus sign denotes a net overcount.

The following table presents the results from the 1990 Census Post-Enumeration Survey:

Table 1

.—

Percent Net Undercount for Major Groups: 1990 PES

Estimation grouping

Net undercount

(percent)

Standard error

(percent)

Total Population

1.61

0.20

Race and Hispanic Origin:

White or Some Other Race not Hispanic)

0.68

0.22

Black or African American

4.57

0.55

Hispanic Origin

4.99

0.82

Asian and Pacific Islander

2.36

1.39

American Indian and Alaska Native (on reservaton)

12.22

5.29

Age and Sex:

Under 18 years

3.18

0.29

18 to 29 years:

Male

3.30

0.54

Female

2.83

0.47

30 to 49 years:

Male

1.89

0.32

Female

0.88

0.25

50 years and over:

Male

−0.59

0.34

Female

−1.24

0.29

Housing Tenure:

In owner-occupied housing units

0.04

0.21

In nonowner-occupied housing units

4.51

0.43

Notes:

• The data in this table contain sampling and non-sampling error.

• The race and Hispanic categories shown on this table represent selected population groupings used in conducting the PES and do not conform exactly with race and Hispanic tabulations that were released from the 1990 census.

Includes household population and some Group Quarters; excludes institutions, military group quarters.

Includes American Indians off reservations.

Excludes Blacks or African Americans, Asian and Pacific Islanders, and American Indians on reservations.

The following table summarizes DA's estimates for Census 2000:

Table 2.—Demographic Analysis Estimates of Percent Net Undercount by Race, Sex and Age: 2000

Category

DEMOGRAPHIC ANALYSIS—2000

Average

Model 1

Model 2

Black Male

Total

5.10

6.94

3.26

0-17

1.47

4.86

−1.92

18-29

6.45

8.02

4.88

30-49

9.18

10.11

8.25

50+

3.29

4.08

2.49

Black Female

Total

0.63

2.52

−1.27

0-17

1.92

5.39

−1.56

18-29

0.12

1.93

−1.70

30-49

0.98

2.06

−0.10

50+

−1.31

−0.45

−2.16

Nonblack Male

Total

−0.93

−1.21

−0.65

0-17

−0.90

−1.56

−0.23

18-29

−4.17

−4.45

−3.89

30-49

0.10

−0.04

0.24

50+

−0.16

−0.24

−0.08

Nonblack Female

Total

−1.44

−1.74

−1.14

0-17

−0.32

−1.01

0.38

18-29

−3.66

−4.00

−3.32

30-49

−1.21

−1.38

−1.04

50+

−1.45

−1.54

−1.35

(A minus sign denotes a net overcount.)

Note:

Model 1 uses 2000 census tabulations for Blacks that include people who reported “Black” and no other race. Model 2 uses 2000 census tabulations for Blacks that include people who reported Black, whether or not they reported other races. People who reported only “Some other race” are reassigned to a specific race category (to be consistent with 1990 DA estimates and the historical demographic data series).

ESCAP Procedure and Process

After the Supreme Court ruled in January, 1999 that the Census Act barred the use of statistical sampling for reapportioning the House of Representatives,

the Census Bureau redesigned its plan for the census to assure that sampling was not used to arrive at the apportionment counts, and to provide for the possible use of sampling for all other purposes. This action was in accordance with the advice of the (then) General Counsel of the Department of Commerce that “Section 195 of the Census Act requires the Census Bureau, if feasible, to produce statistically corrected numbers from the decennial census for all non-apportionment purposes.”

Dept. of Commerce v. House of Representatives

, 119 S.Ct. 765 (1999).

Memorandum to the Secretary and the Director of the Census from Andrew J. Pincus, General Counsel, dated June 12, 2000 and entitled “Legal Obligation to Produce Statistically-Corrected Non-Apportionment Census Numbers.”

The Associate Director for Decennial Census originally chartered the ESCAP on November 26, 1999 and charged the Committee to “advise the Director in determining policy for the A.C.E. and the integration of the A.C.E. results into the census for all purposes except Congressional reapportionment.” Thereafter, on October 6, 2000, the Department of Commerce delegated to the Director of the Census Bureau the final determination “regarding the methodology to be used in calculating the tabulations of population reported to States and localities pursuant to 13 U.S.C. 141(c).” This regulation further required the ESCAP to “prepare a written report to the Director of the Census Bureau recommending the methodology to be used in making the tabulations of population reported to States and localities pursuant to 13 U.S.C. 141 (c).”

The initial regulation was revised on February 14, 2001 to provide that the Secretary of Commerce would make the final adjustment decision for the redistricting data, but only after receiving the recommendation, if any, of the Director of the Census Bureau, together with the ESCAP's report.

Accordingly, this document constitutes the official report of the ESCAP to the Director analyzing the adjustment methodologies and setting forth the relevant factors in the adjustment decision. The Acting Director of the Census Bureau asked the ESCAP to include a recommendation in the Report. This Report is limited to an analysis of whether adjustment would produce improved data for legislative redistricting.

65 Federal Register 59713, “Report of Tabulations of Population to States and Localities Pursuant to 13 U.S.C. 141(c) and Availability of Other Population Information,” October 6, 2000.

66 Federal Register 11231, “Report of Tabulations of Population to States and Localities Pursuant to 13 U.S.C. 141(c); Revocation of Delegation of Authority,” February 23, 2001.

The ESCAP held its first meeting on December 8, 1999 and met regularly until the date of this Report, meeting over 45 times, sometimes with more than one meeting per day. The analysis set forth in this document is supported by extensive staff work and many analytic reports on various aspects of the census and the A.C.E. The documents in these “B-series” reports represent diligent and thorough statistical, demographic, and analytic work conducted over many months of intensive effort. These more detailed reports are summarized in Report B-1, “Data and Analysis to Inform the ESCAP Report,” from which this Report draws heavily.

A list of these reports is attached to this Report.

The ESCAP's membership was originally set forth in its charter and repeated in the regulations. There are twelve members on the ESCAP, with the Director functioning in an ex officio role. The Committee solicited needed assistance from the Associate Director for Field Operations, recognizing his unique contribution to the Committee's awareness of field operations and procedures. He contributed valuable input to the deliberative process and was, in effect, a member of ESCAP. The ESCAP represents a body of senior career Census Bureau professionals, with advanced degrees in relevant technical fields and/or decades of experience in the Federal statistical system. All are highly competent to evaluate the relative merits of the A.C.E. data versus the census data and are recognized for their extensive contributions to the professional community.

The Committee proceeded through four distinct but overlapping stages. The Chair arranged that minutes be prepared for all sessions, except for the final sessions which were private deliberations. The early sessions were educational, designed to make the Committee members aware of the details of the upcoming operations and to explain possible adjustment issues. The second phase was devoted to the presentation of evidence. As data from the census and the A.C.E. became available, knowledgeable individuals in the Census Bureau made presentations to the Committee. The Committee reviewed data from all relevant census and A.C.E. operations, sometimes asking staff to provide additional and new information. The third phase was the deliberation phase. Unlike the first two phases, the deliberations were closed to all but Committee members and individuals invited for a specific purpose: individuals with specialized knowledge who could respond to specific inquiries from the Committee members. The final and briefest stage was the review stage, where Committee members circulated and commented on the draft report.

During the education and evidence presentation phases, the Chair generally arranged presentations on major issues, issues that he identified on his own initiative or on the suggestion of Committee members. During the evidence presentation stage, authors of the analysis reports known as “the B-series” presented their data and conclusions to the Committee. The deliberation and review phases were less structured with various members raising topics for discussion and asking for evidence. No formal vote was held; this Report reflects a consensus of the ESCAP.

This report and the analysis preceding it were prepared in light of the statutory April 1, 2001 deadline.

The Census Bureau clearly would have preferred to have additional time to analyze the data before it, and may well have reached a different recommendation had it had more time; however, the ESCAP believes that it has analyzed the available data sufficiently to make the findings contained in this report. This report is based on the best data available at the time. More data will be produced in the months and years to come that could affect the matters discussed in this report. As in past censuses, the Census Bureau will prepare a large number of detailed evaluations of both the census and the A.C.E. These evaluations will not be available for months, or in some cases, years, after the Census Bureau is required by law to provide redistricting data to the states. These final evaluations, as distinguished from the analysis reports that informed the ESCAP Committee, will be accomplished without the pressure of a legal deadline, will be based on additional information, and may, in some instances, reach conclusions different from those in the analysis reports.

The Census Act requires that redistricting data be “completed, reported, and transmitted to each respective State within one year after the decennial census date.” 13 U.S.C. § 141(c).

A list of the planned Census 2000 final evaluations can be found at Attachment 3.

Findings

The ESCAP has evaluated the conduct of key operations in both the census and the A.C.E., the consistency of the A.C.E. to historical measures of undercount, and measures of both census and A.C.E. accuracy. Accordingly, this section will evaluate:

• The conduct of key operations (Census Quality Indicators, A.C.E. Quality Indicators),

• Historical measures of census coverage—comparison with Demographic Analysis,

• Measures of census and A.C.E. accuracy (Total Error Models and Loss Function Analysis), and

• Other factors that may affect accuracy.

Conduct of Key Operations

Census Quality Indicators

The ESCAP concludes that the unadjusted census was well designed and executed and that the results are of a high quality. There had been considerable concern about potential operational problems, given that the Census Bureau finalized its plans for Census 2000 very late in the census cycle in response to the Supreme Court ruling in January, 1999. However, Census 2000 was an operational success; all major programs were completed on schedule and within design parameters. Although there were some local problems and minor operational shortcomings, census operations were implemented in a controlled manner and within design expectations.

The ESCAP reviewed the results of the initial census to determine whether improved census operations could be expected to yield high quality results. The ESCAP heard presentations on the results of each major census operation and evaluated the extent to which these operations were under control. The discussion in this document is not meant to be a complete evaluation of census operations, but rather focuses on information relevant to the level and pattern of census omissions or erroneous inclusions, because this information is directly relevant to understanding and assessing the results of the A.C.E.

While several major improvements were introduced for Census 2000, including improved marketing, better questionnaire design, more ways to respond, higher pay rates, and improved processing,

the basic design of Census 2000 was similar to the design of the last two censuses. Address lists were prepared from a variety of sources. Questionnaires were delivered to each address on the list. Questionnaires were principally delivered by the U.S. Postal Service. In areas with rural-style addresses, census workers delivered the questionnaires. Households were asked to return the questionnaires by mail. Those addresses that did not return a questionnaire by mail were followed up by census workers in the nonresponse follow-up (NRFU) operation. NRFU was followed by a coverage improvement follow-up operation. Each major operation had its own quality control procedures.

For more detail see “Census 2000 Operational Plan—December, 2000.”

The following is a brief discussion of the quality indicators associated with some of the major Census 2000 operations.

Address List Development.

A foundation of the decennial census process is the list of housing units representing every known residence in the country. The address list is dynamic, with updates occurring at a number of phases throughout the census. One important measure of its quality is the time at which housing units were added. It is preferable for the address list to be largely complete before the majority of census operations begin, as this would indicate that the building of the address list had been successful, by using operations such as address listing and block canvassing, and local government input in the Local Update of Census Addresses (LUCA) program. The data confirm that the address list was largely complete early in the process, as census enumerators found few new addresses in the field. The address list was nearly 97 percent complete (overall and in each region of the nation) before the census forms were mailed out or delivered. The two fastest growing regions, the South and the West, not surprisingly, had slightly lower percentages of housing unit coverage before the census and higher

rates of added housing units during questionnaire delivery. (See B-2, “Quality Indicators of Census 2000 and the Accuracy and Coverage Evaluation.”)

Questionnaire Return—Census 2000 Mail Return Rates.

One of the most important quality indicators for the census is the mail return rate, the proportion of occupied housing units that mailed back their questionnaires. A high mail return rate is crucial to the success of the census—operationally, budgetarily, and also in terms of data quality; data from mailback questionnaires tend to be more complete and of higher quality than the data from forms completed by enumerators.

Public cooperation is critical for the success of any census. The Census Bureau had projected that the mail return rate would be lower in Census 2000 than in 1990 and had accordingly developed an enhanced marketing and partnership program designed to increase awareness of the decennial census and public cooperation. The marketing program was designed around a first-ever paid advertising campaign, including a national media campaign aimed at increasing mail response, targeted advertising directed at raising mail response among historically undercounted populations, and special advertising messages and campaigns targeted to hard-to-enumerate populations. In the partnership program, the Census Bureau worked nationwide with state and local partners to encourage all individuals to respond to the census. Additionally, the Census Bureau worked with states and local jurisdictions to encourage residents of the jurisdictions to raise their mail response rates over their 1990 levels.

The success of the advertising campaign and the partnership program is reflected in the final Census 2000 national mail return rate of 72 percent. In 1990, this figure was 74 percent, but the mailback universe was different in Census 2000, including the addition of approximately five million (mostly rural) housing units to the universe in 2000. These units were enumerated differently in 1990 and the two figures are, thus, not wholly comparable. It is fair to say that the level of public cooperation in Census 2000 roughly equaled that of 1990, despite projections of lower cooperation.

Nonresponse Follow-up.

The nonresponse follow-up (NRFU) operation involved field follow up of about 42 million housing units that did not return a census form within the specified time after Census Day. For most LCO's, NRFU was completed as scheduled in a 9-week period between April 27 and June 26. This performance is a significant improvement over 1990 when NRFU was generally conducted over a 14-week period from April 26 through July 30. The Census Bureau believes, based on past research, that NRFU interviews conducted closer to Census Day are likely to be of higher quality. Thanks in large part to adequate funding provided by the Congress, pay rates and levels of staffing in 2000 were far higher than in the past two censuses. We believe that this increased funding and the ability to hire adequate staff contributed to an improvement in NRFU quality, and thus improved Census 2000 data in general.

The Census Bureau identified local NRFU problems at a few Local Census Offices (LCOs), including the LCO in Hialeah, Florida. The Census Bureau responded to the localized problems in the Hialeah office by re-enumerating certain areas that were believed to have faulty data and does not believe that net coverage in the Hialeah or any other LCO was substantially affected by these local problems. The limited local imperfections do not detract from the conclusion that NRFU as a whole was successful. The local problems experienced were similar to problems encountered in previous censuses, and should be expected in any non-recurring operation of this magnitude.

Housing Unit Unduplication Program.

The Census Bureau became concerned that the address list might contain a significant number of duplicate addresses, or duplicated persons living in duplicated addresses. The Census Bureau responded to this problem by designing and conducting the Housing Unit Unduplication Program. This program was a special operation designed and instituted to reduce the level of housing unit duplication. While this program was not prespecified, the Census Bureau believed that failure to address this potential problem could have seriously impaired the accuracy of the apportionment numbers. Using the results of an address matching operation and a person matching operation, 2,411,743 addresses were identified as potential duplicates and the person and housing records associated with these addresses temporarily removed from the census file. Based on more detailed analysis, 1,392,686 addresses of these were permanently removed from the address list and 1,019,057 addresses were re-instated and included in the census results. Although this operation certainly made mistakes of both exclusion and inclusion, the operation was necessary and resulted in improved census accuracy.

Data Processing.

The large number of address sources used to compile the address list, along with an increased number of response opportunities, increased the chance of duplicate returns. Census 2000 included several data processing steps designed to handle multiple census returns for a single housing unit. More than 90 percent of Census 2000 housing units had only one census return. For households returning two or more forms, the Census Bureau conducted a computer operation to identify and remove duplicated responses. Imputation is discussed later in this Report.

A.C.E. Quality Indicators

The A.C.E. is based on an independent coverage measurement survey, meaning that it collects information in operations separate from the census to allow comparison with the initial census enumeration. The goal is to determine what proportion of the people living in the A.C.E. sample blocks were correctly included in the census, what proportion were erroneously included in the census, and what proportion were not included in the census, so that corrected data can be prepared.

The Census Bureau selected a stratified random sample of blocks to include in the A.C.E. and created an independent list of housing units in those blocks. Enumerators conducted initial A.C.E. interviews at the housing units on this independent list. Households with discrepant information between the A.C.E. and the census received a follow-up interview to find the correct answer or “true” situation. This process led to a determination for each individual regarding whether the A.C.E. response or the census response was correct. Missing data for households and/or individuals was supplied using prespecified procedures, including imputation.

The individuals in the A.C.E. sample were then categorized by age, sex, tenure (owner or renter) and other predefined variables into groupings called post-strata, and coverage correction factors (CCFs) were calculated for each post-stratum. The methodology used to create the coverage correction factors is called Dual System Estimation or DSE. The coverage correction factors measure the extent to which the total of people in each post-stratum is over-or undercounted in the initial census. These factors can be used

to correct the initial census data and to produce tabulated results.

B-7 “Missing Data Results” contains a description of the three types of missing data in the A.C.E. and the processes used to correct for them.

The Census Bureau incorporated a variety of improvements into the 2000 A.C.E. compared with the 1990 PES:

• In order to reduce variance (sampling error), the Bureau doubled the size of the sample from the 1990 PES.

• The design included enhancements to the matching process, such as a more fully automated matching system with built-in edits and quality checks, centralization of matching in one site, and a change in the treatment of movers.

• Computer processing was improved in a number of ways, such as adoption of software validation and verification procedures, standardized nomenclature, and improved documentation for technical issues.

• Enhancements to minimize missing data were added to the design, including allowance of an additional two weeks for attempts to revisit any nonresponding households.

The ESCAP spent many hours reviewing the elements of A.C.E. quality and has concluded that these enhancements succeeded in their goal of improving the A.C.E., and that the operational quality of the A.C.E. was good.

The quality of the A.C.E. operations is particularly evident from the fact that the A.C.E. was completed on schedule without any major difficulties. That operations of the massive size of both the initial census and the A.C.E. could be finished on time and under budget is testimony to thoughtful design and careful implementation. Listing, interviewing, matching, and follow up were all conducted as designed and in a controlled manner. A.C.E. interview response rates met or exceeded expectations. The Quality Assurance operations were carried out as planned and assured that the A.C.E. was in control, resulting in few outliers.

Computer programs were thoroughly tested and improved from 1990. This evidence indicates that the A.C.E. was a clear operational success.

Outliers are extreme blocks with high effect on the estimates.

B-1, “Data and Analysis to Inform the ESCAP Report.”

A.C.E. prespecified procedures were followed except in two specific instances.

Both of these instances were actually enhancements to the A.C.E. design permitted by earlier than anticipated availability of data; both are consistent with good statistical practice and both improved the accuracy of the A.C.E. results.

B-1, Ibid.

Briefly, the first change was a modification of A.C.E. collapsing rules to permit the inclusion of variance as a criterion to collapse data cells. The second enhancement to the prespecified rules deals with imputation cell estimation, the process by which resident status, match status, or enumeration status is imputed for unresolved cases. Imputation cell estimation was modified because the results of the A.C.E. follow-up forms became available during the missing data estimation process. The changes were discussed with the ESCAP and documented.

The ESCAP was pleased with the reduction in sampling variance from 1990 levels. The A.C.E. was designed so that the coefficients of variation (CV) would be lower than in 1990 because of the increased sample size, because better measures of population size were available for the selection of sample clusters, and because sample weights were less variable. The overall CV decreased about 40 percent from 1990 levels, and forty-seven states saw their CV decline, with an average reduction of 37 percent. The A.C.E. design expectation of state-level CV of less than 0.5 percent was achieved. CVs at the congressional district, place, and county level all showed similar levels of improvement, as detailed in Analysis Report B-11, “Variance Estimates by Size of Geographic Area.”

Other important quality indicators for the A.C.E. operations include the following:

• Consistent reporting of Census Day address may have been somewhat better than achieved in 1990 due to the better interview made possible by being held closer to Census Day and an improved interviewing instrument.

• Matching error in the A.C.E. was low, with indications that it is substantially lower than that achieved in 1990. Additionally, other processing errors are probably lower than those measured in 1990.

• A.C.E. fabrication was tightly controlled in 2000; an improved interviewing instrument, tighter management of field operations, and better detection of falsification through targeting, likely lowered the level of fabrication below 1990 levels.

• The level and pattern of missing data in the A.C.E. were near or below that in the 1990 PES and the effect of missing data on A.C.E. quality is similar to that experienced in 1990.

In short, the A.C.E. operations appear to have been in control, performed as expected, and produced data as good or better than the data produced by the 1990 PES.

Historical Measures of Census Coverage—Comparison With Demographic Analysis

By far the largest issue facing the ESCAP has been the surprising inconsistency between the DA and A.C.E. estimates. The initial DA figures estimate that Census 2000 resulted in a net overcount of 1.8 million individuals, that Census 2000 overcounted the population by 0.7 percent. DA has long provided the standard against which the accuracy of both censuses and coverage measurement surveys are measured, making this inconsistency troubling. The inconsistency between the A.C.E. estimates and the demographic analysis estimates is most likely the result of one or more of the following three scenarios:

2. Demographic analysis techniques to project population growth between 1990 and 2000 do not capture the full measure of the Nation's growth.

3. Census 2000, as corrected by the A.C.E., overestimates the Nation's population.

The inconsistency between the demographic analysis estimates and the A.C.E. estimates raises the possibility of an as-yet undiscovered problem in the A.C.E. or census methodology. The Census Bureau has determined that it must further investigate this inconsistency, and the possibility of a methodological error, before it can recommend that adjustment would improve accuracy.

DA assesses accuracy in a fundamentally different manner from the survey-based approach used in the A.C.E. Instead of comparing the results of an independent survey, DA uses administrative records of births, deaths, legal immigration, and Medicare enrollments along with calculated estimates of legal emigration and net undocumented immigration to estimate the national population. Most of these components of population change are well measured (especially for recent decades), but undocumented immigration is not directly measured and must be estimated by comparing detailed data between two consecutive censuses with administrative data on legal immigration. Given the uncertainty of the initial DA results, the Census

Bureau has reexamined certain of these components and created an alternative set of DA estimates that allows for additional undocumented immigration in the 1990s. This alternative set estimates a net undercount of 0.9 million individuals. This would imply that the 2000 Census undercounted the population by 0.3 percent.

Similarly, legal emigration from the U. S. must be measured indirectly and several scenarios were run that varied this component as well as undocumented immigration. Those scenarios did not fit the observed data as well as the those that simply varied undocumented immigration. Scenarios that changed smaller components such as legal temporary immigration will be examined in future research.

For the decade between 1990 and 2000, the base demographic analysis relied on extrapolations of net undocumented immigration derived from data reflecting the changes between the 1980 and 1990 censuses. This analysis estimated the flow of undocumented immigration during the 1990's at 2.8 million. The accuracy of that assumption can only be assessed once the Census 2000 questions on country of birth and year of immigration become available. However, related data that examine the percent Hispanic and Non-Hispanic in Census 2000, and data on the percent foreign-born from the re-weighted March 2000 Current Population Survey (CPS), provide an indication of the accuracy of the original assumptions about immigration and emigration during the 1990's. These data show that the base DA implies a foreign-born percentage of the population below the value reported in the March 2000 CPS (10.3 percent versus 10.6 percent). Similarly, the base DA implies a percent Hispanic (12.1 percent) that is below the Census 2000 percent Hispanic (12.6 percent). Since the undocumented population has recently been predominantly Hispanic, these numbers would be consistent with an underestimate of the undocumented population component in the base DA.

Census Bureau researchers have therefore assumed that the base DA is a reasonable low estimate of net undocumented immigration in the 1990s, and examined several different scenarios to create a reasonable high estimate. For purposes of simplicity, researchers assumed a doubling of net undocumented immigration over the decade for the alternative DA. Doubling net undocumented immigration implies a percent foreign-born of 11.1, which is higher than the 10.6 percent from the re-weighted CPS, and a percent Hispanic of 12.7, which is higher than the 12.6 percent in the unadjusted Census 2000 results. Until data from Census 2000 on country of birth and year of immigration are available to recalibrate DA in detail, this alternative assumption should be considered a reasonable higher bound on net undocumented immigration during the 1990's.

DA and the A.C.E. do not differ completely. DA and the A.C.E. agree on a reduction in the net undercount in Census 2000 compared with 1990, but DA implies a greater change. DA estimated a 1.8 percent net undercount in 1990, compared with either a 0.7 percent net overcount (base set), or a 0.3 percent net undercount (alternative set) in 2000. The A.C.E. estimates show that the net undercount was reduced from 1.6 percent in 1990 to 1.2 percent in 2000. DA and the A.C.E. also concur that Census 2000 succeeded in reducing the differential undercount. Both DA and the A.C.E. measured a reduction in the net undercount rates for Black and non-Black children (aged 0-17) compared to 1990. Both methods also measure a reduction in the net undercount rates of Black men and women (aged 18 and over).

The DA estimates indicate that correlation bias has not been reduced from 1990 levels.

The A.C.E. sex ratios

for Black adults are lower than DA “expected” sex ratios, implying that the A.C.E. did not capture the high undercount rates of Black men relative to Black women. Historically, DA's important strength has been its ability to measure sex ratios accurately. (The ESCAP believes that correlation bias cannot be ignored. The correlation bias for 2000 measured by DA is about the same magnitude as that measured in 1990.)

Correlation bias is discussed in B-12, “Correlation Bias.”

The ratio of men per 100 women.

It is important to understand the limitations and uncertainties associated with the DA estimates:

• Like the A.C.E., DA has an associated level of uncertainty; the ranges of DA uncertainty are a matter of judgment.

• DA estimates do not provide independent coverage benchmarks for all of the characteristics estimated in the A.C.E.

DA estimates can be tabulated by year-specific age, sex, and Black/Non-Black; the A.C.E. permits tabulation for additional racial categories and other characteristics, such as whether the housing unit is owned or rented.

• DA has difficulty in estimating the sub-national population.

• The DA method requires reconciling the reporting of race in the vital statistics system with race as reported in the census. The Census 2000 questionnaire used the instruction “mark one or more races,” introducing a new consideration into the reconciliation of reported race data.

• DA provides estimates for the total population (people living in households and group quarters (GQ)), while the A.C.E. provides estimates only for the housing unit population, but excludes the group quarters population, which includes college dormitories and prisons.

DA estimates for the 1980 and 1990 censuses did not immediately confirm the results of the coverage measurement surveys in those censuses either. Initial DA estimates for the 1980 census implied a net overcount of 0.4 percent, but were later revised upward, partially to account for an increase in undocumented immigration. DA estimated a 1.8 percent undercount for the 1990 census, leading Secretary Mosbacher and others to question the accuracy of the 1990 adjusted counts. The Census Bureau, however, concluded that the differences between DA and the 1990 PES were explainable as within the bounds of DA uncertainty.

Bureau of the Census, “Technical Assessment of the Accuracy of Unadjusted Versus Adjusted 1990 Census Counts,” 4.

However, in Census 2000 the differences between DA and the A.C.E. are larger than in 1990, with DA measuring an undercount from 1.9 to 0.9 percentage points less than the A.C.E. The Census Bureau acknowledges DA's inconsistency with the A.C.E. estimates and will continue to research this important issue.

Measures of Census and A.C.E. Accuracy

Total Error Model

The total error model and loss function analysis are methods used to compare the accuracy of the adjusted and unadjusted 2000 data. The total error model brings together all of the components of error that can be measured for the A.C.E. The total error model is used to correct the A.C.E. for biases and thus produces a measure of “truth” that can be used to assess the accuracy of both the adjusted and unadjusted census. The measures of the truth are referred to as targets since the components of error must be estimated. By using a range of targets as the basis of comparing the A.C.E and Census 2000, calculations can be done that indicate whether the adjusted or unadjusted census results are more accurate. Situations are defined by the methods and assumptions that are used to vary the components of error in the total error model.

The total error model identifies and estimates the various components of error and their variances for groups of the A.C.E. post-strata designated as evaluation post-strata.

Estimates of the component errors are derived for each evaluation post-stratam, then a simulation methodology is used to create a range of target populations. Loss functions, described in the next section, are then used to determine which of the adjusted or unadjusted census populations is closer to the targets, taking into account the uncertainty in the targets and in the adjustment.

The Census Bureau can only estimate error components for at most sixteen evaluation post-strata. The error components reflect, for the most part, measures of nonsampling error. Estimation of nonsampling error requires extensive methodology carried out by extremely well qualified staff. Because few such staff exist, this limits the size of the sample for which measures can be obtained. Therefore direct estimates of the targets can only be obtained for a smaller number of evaluation post-strata.

Mary Mulry and Bruce D. Spencer. “Accuracy of the 1990 Census and Undercount Adjustments.”

Journal of the American Statistical Association

88 (September 1993): 1080-91; B-19, Mulry and Spencer.

The components of error for the total error model are as follows:

Ibid.

1. P-sample matching error

2. P-sample data collection error

3. P-sample fabrication

4. E-sample data collection error

5. E-sample processing error

6. Correlation bias

7. Ratio estimator bias

8. Sampling error

9. Imputation error

The Census Bureau has data from DA, Census 2000, and the A.C.E. that can be used to produce estimates of components 6, 7 and 8 (Correlation Bias, Ratio Estimator Bias, and Sampling Error), and is relying on 1990 data to estimate the remaining components. The ESCAP discussed the use of 1990 measures for these error components, and determined that doing so would provide conservative estimates of the level of error in the A.C.E. The ESCAP noted that the A.C.E. is similar in design and operation to the 1990 PES, except that the A.C.E. was conducted with higher quality as noted above.

B-19, Mulry and Spencer.

The ESCAP analyzed the sensitivity of various components of the total error model, particularly the office processing components, because the Committee believes that it achieved better results for these components in 2000 than in 1990. Also, the ESCAP used a number of models of correlation bias in the total error model, given the importance of this component, and the understanding of the significant influence that this component has on the estimates of total error and thus on the target populations.

Loss Function Analysis

Loss function analysis is used to compare the adjusted and unadjusted census populations to the target populations derived from the total error model as described above. Loss functions are constructed to measure the loss or error associated with differences from the targets. Loss functions are defined to measure the loss in accuracy due to differences from the target populations. Loss functions are also specified based on various criteria related to the intended uses of the data. A general description of loss functions is as follows:

EN08MR01.005

Where:

n represents the number of entities for which the comparison is conducted;

Cen

, and ACE

represent the target population, unadjusted census population, and the adjusted census population, respectively for the i

entity; and

represents a weight defined for the criterion to be studied for a particular use of the data.

If the Census loss is greater than the ACE loss, then the adjusted data are determined to be more accurate for the criterion represented by the loss function.

The Census Bureau believes that both numeric and distributive accuracy are important measures of census accuracy and accordingly designed loss functions to measure both types of accuracy. Numeric accuracy refers to how close the overall count of a particular geographic area is to the truth, whereas distributive accuracy refers to how close the relative proportion or share of a geographic area is to its true share relative to other areas.

As discussed in B-13, “Comparing Accuracy,” the ESCAP directed the preparation of four types of loss functions:

The relationship between numeric and distributive accuracy is discussed in the Feasibility Document, pp. 15-18.

1. Squared Error Loss

2. Weighted Squared Error Loss

3. Relative Squared Error Loss

4. Equal Congressional District Squared Error Loss

The Committee determined that the second and fourth loss functions, weighted squared error loss and equal CD squared error loss, were the most appropriate to measure accuracy for redistricting data. The ESCAP directed the preparation of loss functions at the state, congressional district and county levels, believing these geographic levels most relevant to the decision before the Secretary. County level data is intended to simulate state legislative districts, because these districts are usually smaller than congressional districts.

The ESCAP studied the sensitivity of the loss functions by varying the assumptions for various of the components in the total error model. As described above, extensive sensitivity analysis was conducted for the various models and levels of correlation bias that were used to generate the target populations.

B-13, “Comparing Accuracy.”

Loss functions that measure only a small gain in accuracy for the A.C.E. may be problematic, given the associated uncertainty with these estimates. Accordingly, the Committee examined the loss functions for evidence of a clearly measurable improvement and found the following:

Ibid.

1. At the state and congressional district level, when only sampling variance was included, the loss functions showed that the change due to adjustment was significant in comparison to sampling error, that is, if sampling error were the only concern, adjustment would result in more accurate data. The ESCAP recognizes, of course, that sampling error is not the only error in the A.C.E., and thus this analysis was conducted to determine whether sampling error alone would result in finding that adjustment was less accurate. This was not the case so the ESCAP proceeded with more extensive analyses.

2. Correlation bias is a significant factor in influencing the results of the loss functions, and a variety of models were used to test the sensitivity of the analysis to correlation bias effects in creating the target populations. When full components of estimated correlation bias were used to construct the target populations, at the state, congressional district, and county levels, the loss functions showed a marked improvement for adjustment, regardless of the model. When only 50 percent of the estimated correlation bias was used in constructing the target populations, the loss functions continued to show a clear improvement. The ESCAP

considered this to be an important finding because while there may be disagreement regarding the existence of correlation bias, assuming no correlation bias is clearly an unlikely possibility.

3. When either no or only a modest amount of correlation bias is factored into the loss functions, they tend to favor the unadjusted census at all geographic levels. The ESCAP noted that assuming no correlation bias would result in a lower bound for the degree of improvement for adjustment, since as noted above, it is not reasonable to assume no correlation bias.

4. The loss functions for counties with populations below 100,000 indicated that the unadjusted census was more accurate regardless of the level of correlation bias assumed. This caused some concern, since this was not the case for the 1990 census adjustment. However, the ESCAP found that the adjustment was more accurate when considered for all counties using both numeric and distributive accuracy. Therefore, the adjustment was improving the data for areas in which the majority of the population resided. This is further indication of the closeness of the A.C.E. estimates and Census 2000.

The conclusion that can be drawn from the loss function analysis is that, absent the concerns with consistency between DA and the A.C.E., the adjustment would result in data that are more distributively and numerically accurate at the state and congressional district levels if correlation bias is recognized at a likely level, but that the data are not more accurate for smaller counties. Even though smaller counties would have been less accurate; the analysis indicated an overall improvement in accuracy from adjustment. However, the ESCAP notes its concern regarding the unexplained differences between DA and the A.C.E. estimates, which may be indicative of an unmeasured problem in Census 2000 or in the A.C.E. The potential for a reversal of these findings is strong enough to preclude a conclusion at this time that adjustment would improve accuracy. When considering the additional concerns described below and taking into account the inconsistencies with DA, the Committee was not prepared to recommend at this time that adjustment would improve accuracy.

Other Factors That May Affect Accuracy

Synthetic Error

The A.C.E. methodology produced estimated coverage correction factors for each of the post-strata. These factors were carried down within the post-strata to the census block level in a process called synthetic estimation. The key assumption underlying synthetic estimation is that the net census coverage is relatively uniform within the post-strata. In other words, the probability that people in a particular post stratum will be missed by the census is assumed to be roughly the same. The failure of this assumption causes synthetic error.

The design underlying synthetic estimation methodology is directed at correcting a systematic under or over count in the census. The synthetic estimates will not correct random counting errors that occur at any geographic level (blocks, tracts, counties, etc). Therefore, the synthetic estimate will not result in extreme changes in small geographic entities, nor will it correct for extreme errors. Synthetic estimation is designed to remove the effects of systematic errors, so that when small entities are aggregated, systematic and differential coverage errors can be corrected.

The ESCAP was concerned with synthetic error, because this type of error is not included as a component of the total error model (which estimates error in post-stratum level DSE's, where there is, by definition, no synthetic error). Furthermore, synthetic error cannot be estimated directly, as direct estimation would require more sample observations for the A.C.E than practicable.

The ESCAP analyzed the effects of synthetic error by conducting artificial population analysis. This analysis creates artificial populations with surrogate variables thought to reflect the distribution of net coverage error. These surrogate variables are known for the entire population. An analysis of these artificial populations for the effect of synthetic error is the basis on which this otherwise unknown effect is studied.

The detailed analysis of synthetic error is described more fully in reports B-1, “Data and Analysis to Inform the ESCAP Report,” and B-14, “Assessment of Synthetic Assumptions.” Briefly, four artificial populations were constructed based on census variables thought to be related to census coverage. The Census Bureau calls these variables “surrogates.”

The Census Bureau distributed the post-stratum level gross undercount (gross overcount) in proportion to the gross undercount surrogate variable (gross overcount surrogate variable) to the geographic levels to be studied. This process results in a population with surrogate values for coverage error which are known at all levels. Unlike other approaches, artificial population analysis provides measures of net coverage for all local areas, within a post-stratum. Therefore the effect of synthetic error can be assessed for these artificial populations.

The methodology used is similar to that suggested by Freedman and Wachter (1994, Statistical Science).

The four artificial populations are described in Table 3 below:

Table 3.—Surrogate Variables Used To Create Artificial Populations

Undercount surrogate

Overcount surrogate

Artificial Population 1

(# non-GQ persons)— (# persons in whole household substitutions)

(# non-GQ persons)—(# persons for whom date of birth was allocated consistent with reported age).

Artificial Population 2

(# non-GQ persons)—(# persons in whole household substitutions)

(# non-GQ persons)—(# persons in whole household substitutions).

Artificial Population 3

# non-GQ persons with 2 or more item allocations

# persons for whom date of birth was allocated consistent with reported age.

Artificial Population 4

# non-GQ persons whose household did not mail back the questionnaire

# non-GQ persons whose household did not mail back the questionnaire.

GQ = Group Quarters

Three types of analysis were conducted using these artificial populations:

See Analysis Report B-14, “Assessment of Synthetic Assumptions” for a detailed discussion.

1. The effect of relative bias for synthetic estimation was assessed by calculating the ratio of the absolute unadjusted census error to the absolute adjusted census error for state and congressional district population totals and shares. An analysis of the distribution of these relative biases indicated that within the artificial populations, synthetic estimation improved the majority of entities.

2. Biases for synthetic estimation were also calculated and compared to the level of bias in the dual system estimates, including correlation bias from the total error model. Since the total error model does not include synthetic bias, the purpose of this analysis was to determine whether the level of synthetic error was small enough to be ignored when compared to the other errors estimated for the A.C.E. This analysis showed that the level of synthetic error could not be ignored for several of the artificial populations. This finding led to the third analysis.

3. Because synthetic error affects both the unadjusted and adjusted census, the Census Bureau studied the effect of synthetic error on both unadjusted and adjusted census loss, as measured by the loss functions, and concluded that synthetic error would increase the loss of both the unadjusted and adjusted census. The question was how this error would affect the relative losses for the adjusted and unadjusted data.

Therefore, the ESCAP directed the addition of synthetic error to the loss measured for both the adjusted and unadjusted census. This study indicated that synthetic error could, in certain situations, affect the relative comparison of adjusted or unadjusted loss.

For the analyses based on several of the artificial populations for state and congressional district counts, the loss function analysis understated the true gains from adjustment. However, for some of the analyses, the loss function results understate the true gain for the unadjusted census. In these situations, the effect could be as high as 58 percent.

The ESCAP noted that a conservative view of the loss function results should be used in assessing the gain in accuracy from adjustment. Given the concerns described above, the ESCAP believes that this finding must be fully understood before recommending for an adjustment.

Balancing Error

The A.C.E. actually consists of two surveys, based on two samples: “ the P-sample and the E-sample. The P-sample is an enumeration independent from the census, used to measure omissions or missed persons. The E-sample is a sample of census records that are reexamined to measure erroneous inclusions. Balancing error occurs when cases are handled differently in the P-and E-samples. For example, the effort spent to identify gross omissions should be comparable to the effort spent to identify erroneous enumerations. The ESCAP examined whether balancing error may have been introduced during the Targeted Extended Search (TES) operation. TES was the A.C.E. operation designed to look for matches in surrounding A.C.E. block clusters. The DSE model attempts to match people in the A.C.E with people in the census. Balancing error occurs when the search area for the P-sample matching does not agree with the search area for E-sample erroneous enumerations. Specifically, if A.C.E. records are allowed to match to records that were not in the common area of search, the DSE ratio will be incorrectly estimated.

One can assess TES balance by seeing if the proportions of errors of inclusion and of exclusion are approximately equal after completion of the search, assuming that there is no geocoding error in the P-sample. In other words, the number of TES people found on the P-sample (coded as a Match) and E-sample (coded as a Correct Enumeration) sides should be about equal. In Census 2000, the much greater increase in the match rate (3.8 percent) than the correct enumeration rate (2.9 percent) may indicate that some aspect of A.C.E. is out of balance. The ESCAP directed a review of this situation. Preliminary results from an early A.C.E. evaluation indicate that a number of E-sample cases coded as correct enumerations were in fact outside of the search area. That means that they should have been coded as Erroneous Enumerations and subtracted from the DSEs. This error could introduce an upward bias in the DSE. In addition, there are also concerns that the search for census duplicate enumerations in surrounding blocks could have understated the estimate of duplicates used in the DSE. The net effect of correcting these two errors could have the effect of reducing the A.C.E. estimate of total net undercount. However, additional work must be completed to quantify this effect.

The ESCAP was concerned about the possibility of balancing error. The ESCAP noted that some measures of this error were included in the total error model. However, this result, in combination with the inconsistency between DA and the A.C.E., added to the concerns that adjustment could not be shown to improve accuracy at this time. The Committee also believes that balancing error must be further investigated before a recommendation can be made.

Late Adds and Whole Person Imputations

There are records included in Census 2000 that do not contain information sufficient for matching to the A.C.E independent sample. The methodology that has been established and used to produce coverage estimates, given that this situation will occur, is to produce the dual system estimate based on the census population that has sufficient information to be included into the A.C.E. matching process. In effect, this excludes records that do not contain sufficient information for matching from the dual system estimation. The dual system estimate then produces a measure of the correct population total. The undercount (or overcount) is estimated by comparing the complete census count to the dual system estimate of the correct population total. Therefore, the effect of these census records is included in the estimates of undercount produced by dual system estimation.

The key assumption underlying this methodology of estimating coverage error is that the probability of including the people represented by these records in the A.C.E. P-sample is the same as the probability of including the people who report sufficient information to be included in the matching procedures.

Census 2000 contains over five million records where imputation procedures were used to create all of the information. These are referred to as whole person imputations. Since these records do not contain information sufficient to be included in the matching, they are handled as described above. The Census Bureau plans to evaluate the causes for these imputations.

In addition, as discussed in the preceding section on census quality, the Housing Unit Unduplication Operation reinstated over a million previously removed housing units (representing over two million individual person records) into the census files. These reinstated “Late Additions” were incorporated into the estimates of coverage error using the same process as described for census records that do not contain sufficient information for

matching. The same assumption underlies this treatment of late additions as described above for records without sufficient information for matching. That is, the probability of inclusion in the P-sample for the people Census 2000 correctly enumerated in the universe of late additions is assumed to be the same as for correctly enumerated people not in this universe.

The ESCAP reviewed the treatment of late additions and whole person imputations because the number of these cases had increased significantly from 1990. The ESCAP concluded that the key assumptions underlying the methodology for including these records into the estimates of A.C.E coverage error could be expected to hold. However, the ESCAP noted that these assumptions would not hold perfectly and examined the effects of deviations from this assumption. The ESCAP concluded that three effects were likely to result (1) the sampling variance of the dual system estimator would be increased; (2) the heterogeneity of the A.C.E inclusion probabilities would be increased, leading to increased correlation bias; and (3) to the extent that these records clustered geographically within the A.C.E. post-strata, synthetic error would be increased.

The ESCAP was comfortable that the measures available for assessing the effects of sampling variance and correlation bias would include the effects of the treatment of late additions and whole person imputations. However, the ESCAP was concerned that synthetic error might be increased and continued its review of the effect of increased synthetic error. The committee reviewed tabulations of late census additions and whole person imputations for A.C.E post-strata by census region. The committee found that these data did indicate some degree of geographic clustering within post-strata. The committee noted that the synthetic error analysis included the effect of clustering of whole person imputations. The committee concluded that there was a possibility for increased synthetic error, and that it was reflected to some degree in the analysis based on artificial populations. The committee concluded further that a higher degree of conservatism should be used in reviewing the results of the loss function analysis. The committee did not view the effect of increased synthetic error as large enough to change the findings described previously.

Misclassification Error

Misclassification error occurs when an individual is classified into different post-strata in the census and the A.C.E. While the Census Bureau has never detected a significant impact of misclassification error in earlier post-enumeration surveys, the introduction of multiple race reporting in both the census and the A.C.E. raised concerns about this type of error. The evidence reveals that misclassification error affected only two groups, the domains of American Indians off reservation and Native Hawaiians and Pacific Islanders. ESCAP has concluded that for these two groups, it appears that inconsistency may have contributed to having lower than anticipated undercount rates because of how they were classified. The misclassification error in these two domains had little or no effect on the validity of the dual system estimates as a whole, given their small sizes. Misclassification error, in general, was not a problem.

Additional Issues

There are several issues or concerns that have been raised regarding census adjustment. These issues did not concern the ESCAP, but are briefly discussed below.

Block Level Accuracy

Block level accuracy is not an important criterion to evaluate either Census 2000 or the A.C.E. The population of stand-alone blocks is not used to determine either congressional or state legislative districts, nor is block-level data used to distribute funds. Rather, blocks are added together to form the more meaningful levels of aggregation studied by the loss functions: states, congressional districts, and counties.

Block level accuracy has two components, random error and systematic errors or biases. Random error can be minimized through the conduct of census operations aimed at improving quality. Systematic biases, on the other hand, are caused by systematic errors that occur during the conduct of census operations. Random errors at the block level diminish greatly as blocks are added together to form larger aggregations of the data. Systematic errors, if not corrected, will remain in the data at all levels of aggregation, leading to data that systematically over or understate affected population groups. Therefore, it is more important for adjustment to remove systematic errors from block level data.

Adjustment for Overcounts

It is important to emphasize that the statistical correction of Census 2000 would involve some amount of downward adjustment for overcounts. While the A.C.E. would mostly result in an increase in the estimated size of most undercounted geographic entities, there are likely to be a small number of overcounted areas that would require decreasing the estimated size. The 2000 A.C.E. data do not show that any state or congressional district was overcounted; all states and congressional districts would increase in measured size. The data do reveal, however, that certain substate entities were overcounted and would thus be subject to downward adjustment.

There are concerns that an adjustment for overcounts removes people from Census 2000 data files. This is not the case; the downward adjustment is accomplished by creating statistical records with negative weights that, when added to Census 2000 tabulations, reduce the count to reflect overcounts. No records would have been removed from the Census 2000 files. However, the effects of the adjustment for overcounts may subtract a person's individual characteristics from the Census 2000 tabulations.

The ESCAP discussed the downward adjustment for overcounts, and noted that it was subject to the same concerns that are related to adjustment for undercounts. The ESCAP concluded that the analysis of the accuracy of the adjustment included the effects of uncertainties for adjustments of over and undercounts, and that any final determination on the potential improvement of accuracy would reflect these uncertainties.

Attachments

1. Index of Supporting Documentation

2. List of “B-Series” Analysis Reports

Attachment 1—Outline of Documents/Records Underlying the Report of the Executive Steering Committee for Accuracy and Coverage Evaluation Policy Regarding the Methodology To Be Used To Produce the Tabulations of Population Reported to States and Localities Pursuant to 13 U.S.C. 141(c)

A. Reports Supporting the Recommendation—Chapter B (Final reports)

B. Detailed Specifications for A.C.E. Methodology

C. Executive Steering Committee for A.C.E. Policy (ESCAP) Meetings

D. October 2, 2000, Presentation to the National Academy of Sciences

E. Prior Steps to Determining Feasibility

F. Prior Documentation of Adjustment Research

G. Census 2000 Decision Memoranda

H. Census 2000 Dress Rehearsal Evaluations Summary (1999)

I. Census 2000 Informational Memoranda

J. U.S. Census Monitoring Board Reports

K. U.S. General Accounting Office Reports

L. U.S. Department of Commerce Office of the Inspector General Reports

Note:

Because of the large volume of underlying documentation, not all will be posted to the Census Bureau's website at the time that the ESCAP report is made available. The remaining documents will be posted in the near future.

Attachment 2_B-Series Documents

February 27, 2001

Title

Author

1. Accuracy and Coverage Evaluation: Data and Analysis To Inform the ESCAP Report

Hogan.

2. Quality Indicators of Census 2000 and the Accuracy and Coverage Evaluation

Farber.

3. Quality of Census 2000 Processes

Baumgardner/Moul/Pennington/Piegari/Stackhouse/Zajac/Alberti/Reichert/Treat.

4. Accuracy and Coverage Evaluation: Demographic Analysis Results

Robinson.

5. Accuracy and Coverage Evaluation: Person Interviewing Results

Byrne/Imel/Ramos/Stallone.

6. Accuracy and Coverage Evaluation: Person Matching and Follow-up Results

Childers/Byrne/Adams/Feldpausch.

7. Accuracy and Coverage Evaluation: Missing Data Results

Cantwell/McGrath/Nguyen/Zelena k.

8. Accuracy and Coverage Evaluation: Decomposition of Dual System Estimation Components

Mule.

9. Accuracy and Coverage Evaluation: Dual System Estimation Results

Davis.

10. Accuracy and Coverage Evaluation: Consistency of Post-Stratification Variables

Farber.

11. Accuracy and Coverage Evaluation: Variance Estimates by Size of Geographic Area

Starsinic/Sissel/Asiala.

12. Correlation Bias

Bell.

13. Comparing Accuracy

Mulry/Navarro.

14. Accuracy and Coverage Evaluation: Assessment of Synthetic Assumptions

Griffin/Malec.

15. Census 2000: Service Based Enumeration Multiplicity Estimation

Griffin.

16. Demographic Full Count Review: 100% Data Files and Products

Batutis.

17. Census 2000: Missing Housing Unit Status and Population Data

Griffin.

18. Accuracy and Coverage Evaluation: Effect of Targeted Extended Search

Navarro/Olson.

19. Overview of Total Error Modeling and Loss Function Analysis

Mulry/Spencer.

Attachment 3 to Preamble

DSSD Census 2000 Procedures and Operations Memorandum Series B-1, March 1, 2001

Accuracy and Coverage Evaluation: Data and Analysis To Inform the ESCAP Report

Howard Hogan, U.S. Census Bureau

Table of Contents

Summary Table

Introduction

Background

Accuracy and Coverage Evaluation (A.C.E.) Dual System Estimates

Review of the Quality of the Census Operations

Conclusions for this Section

Analysis reports important to this section

Discussion

Address List Development

Questionnaire Delivery and Return

Nonresponse Follow-up

Be Counted Campaign

Coverage Edit Follow-up

Coverage Improvement Follow-up

Housing Unit Duplication Operation

Primary Selection Algorithm

Unclassified Unit and Missing Data Estimation

Review of A.C.E. Operations

Proper execution of the steps between processing and estimation

Conduct and control of the A.C.E. operations

Conclusions for this section

Analysis reports important to this section

Discussion

Interviewing

Matching and Follow-up

Review of A.C.E. Quality

Individual components of A.C.E. quality

Sampling variance

Consistent Reporting of Census Day Residence

Matching error

A.C.E. Fabrications

Missing data

Balancing error

Errors in Measuring Census Erroneous Enumerations

Correlation bias

Synthetic Assumptions

Other Measurement and Technical Errors

Synthesizing A.C.E. Quality

Comparison with demographic analysis and demographic estimates

Post-Enumeration Survey—A.C.E. error of closure

Comparing the accuracy of the A.C.E. to the accuracy of the uncorrected census

References

Tables

Table 1. Summary Table: Data and Analysis to Inform the ESCAP Report

Table 2. Percent Net Undercount for Major Groups: 2000 A.C.E. and 1990 PES

Table 3. Distribution Interviews by Week—Unweighted

Table 4a. Census 2000 A.C.E. 64 Post-Stratum Group-Percent Net Undercount

Table 4b. Census 2000 A.C.E. 64 Post-Stratum Group-Standard Error of the Net Undercount

Table 5. A Comparison of the 1990 PES Total Population with the A.C.E. Accounting for Population Change

Table 6. Relative Loss by Degree of Processing Error and Correlation Bias

Table A-1. Census 2000 A.C.E. 64 Post-Stratum Groups by Region—Percent Late Adds

Table A-2. Census 2000 A.C.E. 64 Post-Stratum Group by Region—Percent Iis

Table A-3. Census 2000 Evaluations Program Category Report Schedule

Table 1.—Summary Table: Data and Analysis To Inform the ESCAP Report

Finding

Evidence

Implication

Limitations

What do we know about Census 2000?

Census 2000 was similar in design to the 1980 and 1990 censuses

Procedural histories and Census 2000 Operational Plan, 12/00

We expected similar patterns of coverage error.

Census 2000 included some important improvements such as paid advertising, intensive community outreach, local involvement with address list development, and competitive pay scales

Census 2000 Operational Plan 12/00

We expected these programs to have a modest impact in reducing the undercount.

While there were some local problems and minor operational shortcomings, Census 2000 was generally well designed and executed

Report B-3, “Quality of Census Processes”

We expect generally good and uniform patterns of coverage, with perhaps local exceptions.

There is evidence of geographic heterogeneity in the application of some census processes

Presentation at ESCAP, 12-20-99, 4-12-00, 5-24-00 and B-14, Feasibility, pp. 46-48

Local heterogeneity will affect the accuracy of both the census and the adjusted estimates

We cannot know the effects of this differential pattern on census net undercount at the local level.

There was a high level of advertising and outreach targeted at minority populations

Census 2000 Operational Plan 12/00

We expected these programs to have a modest impact in reducing the undercount.

What does the A.C.E. tell us about Census 2000?

The level and patterns of coverage in Census 2000 are substantially similar to those of the past two censuses, with incremental improvements rather than wholesale discontinuities

Report B-2, “Overall Census and A.C.E. Quality Indicators;” Report B-3, “Quality of Census Processes;” Feasibility Doc

This finding is consistent with what is known about the design and implementation of Census 2000.

While Census 2000 reduced both the net and the differential undercount, the A.C.E. estimates that the census undercounted the total population by approximately 1.18 percent and continued previous patterns of differential coverage, with lower coverage rates for minorities, renters, and children

Report B-9, “Dual System Estimation Results”

A lower undercount in the Census means the benefits from adjusting a loss

All results are subject to sampling and nonsampling errors.

Was the A.C.E. conducted as designed?

The A.C.E. was carried out as pre-specified with only minor modifications, which were warranted and documented when important information became available earlier than expected

Report B-7, “Missing Data Results;” Report B-8, “Decomposition of Dual System Estimate Components;” Report B-9, “Dual System Estimation Results”

The results of the A.C.E. were not manipulated

There were two changes to prespecification, one concerning the collapsing rules and the other affecting the missing data imputation.

Was the A.C.E. an operational success?

The A.C.E. was an operational success; listing, interviewing, matching, and follow-up were all conducted as designed and were well controlled

Report B-5, “Person Interviewing Results;” Report B-6, “Person Matching and Followup Results;” Report B-7, “Missing Data Results”

There were no unforseen operational difficulties with a significant effect on the quality of the data.

The A.C.E. significantly reduced sampling variance

Report B-9, “Dual System Estimation Results;” Report B-11, “Variance Estimates by Size of Geographic Area”

There were no unforseen operational difficulties with a significant effect on the quality of the data.

Consistent reporting of Census Day addresses may have been somewhat better than that achieved in 1990 due to better interviews made possible by the Computer Assisted Personal Interviewing instrument. Interviewing was conducted closer to Census Day

Report B-6, “Person Matching and Followup Results”

Data collection error probably lower than in 1990.

Matching error in the A.C.E. was low, with indications that it is substantially lower than that achieved in 1990

Report B-6, “Person Matching and Followup Results;” Presentation to ESCAP, 11-30-00 and 2-2-01, Feasibility, p. 43

A.C.E. processing errors are probably less than those measured in 1990.

A.C.E. fabrication was more tightly controlled than in the 1990 PES; tighter field management reduced the opportunity for fabrication

Report B-6, “Person Matching and Followup Results”

Data collection error probably lower than in 1990.

The level and pattern of missing data in the A.C.E.'s comparable to that in the 1990 PES

Report B-7, “Missing Data Results”

Effect on A.C.E. quality is similar to that experienced in the 1990 PES.

Questions remain concerning the level of balancing error

Report B-8, “Decomposition of Dual System Estimate Components;” Minutes, Feasibility p. 50, Report B-18, “Effect of Targeted Extended Search”

Increased balancing error could make the adjustment less accurate

A full analysis has not been completed.

E-Sample coding errors were controlled comparable to 1990, except, perhaps, for e-sample geocoding

Series T-6: “Additional Geographic Coding for Erroneously Enumerated Housing Units”

A.C.E. might over estimate the census undercount

There was evidence of some A.C.E. mis-geocoding.

Correlation bias is almost certainly present for both Black and non-Black populations. The switch to PES-C may have increased correlation bias over 1990 levels, but the evidence on the level of correlation bias is weak

Report B-4, “Demographic Analysis Results;” Report B-12, “Correlation Bias;” Presentation to ESCAP, 7-12-00, Feasibility pp. 35-36

A.C.E. could underestimate the undercount

Limited data on females, children etc.

The A.C.E. contains bias due to synthetic estimation

Report B-14, “Synthetic Assumptions”

The A.C.E. will not remove local variations in the net undercount that are not correlated with the poststrata

Only indirect evidence is available.

What does Demographic Analysis say about the census?

Initial Demographic Analysis estimates indicate a net census overcount of 0.7 percent with large overcounts for the non-Black population age 18-29

Report B-4, “Demographic Analysis Results”

The level and pattern of the A.C.E. estimates differs from the initial Demographic Analysis estimates

The Demographic Analysis estimates are subject to their own patterns of uncertainities.

Alternate Demographic Analysis benchmarks indicate a net undercount of 0.9 million, or 0.32 percent

The A.C.E. may be overestimating the population size

The Demographic Analysis estimates are subject to their own patterns of uncertainities.

Both the initial and revised Demographic Analysis indicates an improvement in coverage from the 1990 to the 2000 censuses

B-4, “Demographic Analysis Results”

Census 2000 net coverage is higher than 1990.

Both the initial and the revised Demographic Analysis indicate a differential undercount in Census 2000

B-4, “Demographic Analysis Results”

Census 2000 did not eliminate the differential undercount.

What does loss function analysis tell us about the relative accuracy of the adjusted and unadjusted census?

If there is little or no correlation bias and the level of A.C.E. errors is the same as the 1990 PES, the A.C.E. is less accurate than the census

Report B-13, “Comparing Accuracy”

If these conditions are true, the census is probably the more accurate

These results are dependent on the model assumptions being approximately true.

If there is moderate correlation bias or if the level of A.C.E. processing errors is substantially reduced, the A.C.E. is more accurate

B-13, “Comparing Accuracy”

If these conditions are true, the adjusted figures are probably the more accurate

These results are dependent on the model assumptions being approximately true.

Accounting for local census heterogeneity is unlikely to reverse the findings for the loss function analysis

B-14, “Synthetic Error”

Heterogeneity is a concern but probably not a deciding factor

Measuring the effect of local variation is dependent on finding observable variables that have similar geographic distributions as the net undercount.

What does an analysis of the A.C.E./PES error of closure tell us about the level and pattern of DSE errors?

The level and pattern of errors in the A.C.E. may differ from that of the 1990 PES

B-13, “Comparing Accuracy” B-14, “Synthetic Error,” and Overview of Total Error Modeling and Loss Function Analysis

The findings from the loss function analysis, which depend upon an assumption of A.C.E./PES similarity in error structure, may be misleading

This result depends upon Demographic Analysis's ability to place an upper bound on the level of population change between 1990 and 2000.

All findings are based on the best available evidence as of today; further evaluations could modify them.

Accuracy and Coverage Evaluation: Accuracy and Coverage Evaluation: Data and Analysis To Inform the ESCAP Report

Prepared by Howard Hogan

Introduction

Background

The Census Bureau designed the Accuracy and Coverage Evaluation (A.C.E.) to permit correction of the initial census results to account for systematic patterns of net undercount. The Census Bureau preliminarily determined that the A.C.E., if properly conducted, should produce more accurate census data by improving coverage and reducing differential undercounts; the purpose of this document is to evaluate whether the data produced in Census 2000 support this initial determination.

This document summarizes and synthesizes the more detailed analysis reports that were written to inform the adjustment decision. No one analysis report is determinative; rather the information in the analysis reports, taken together, permits evaluation of the quality of both the census and the A.C.E. The topics of the analysis reports were selected because the Census Bureau believed that the information in those reports would provide the basis for informing the Census Bureau's adjustment decision. In the course of evaluating the conduct of both the census and the A.C.E., it became evident that other analyses should be completed; thus, two additional reports have been added to the 16 formal reports originally specified. The information in the analysis reports, and the reports themselves in draft form, have been shared with the Executive Steering Committee on A.C.E. Policy (ESCAP) on a flow basis so that the Committee could evaluate the data as they became available. The Committee has sometimes asked for additional information, either from the authors of the analysis reports or from other Census Bureau staff. Much of the analysis in the attached reports is applicable for all possible uses of adjusted data, but in some instances the reports focus on the ESCAP Committee's initial regulatory charge: to make a recommendation on the suitability of using the A.C.E. data for redistricting.

As this document is written, the ESCAP is in the process of evaluating which set of numbers, the adjusted or the unadjusted, is more accurate for redistricting purposes. If more than one set of numbers is available, each of the 50 states will then make its own decision on which set of data to use. The Census Bureau believes it is appropriate for it to make one determination on which set of data is more accurate, rather than 50 separate determinations, because the statistical determination of the relative accuracy of the census versus the A.C.E. results is meaningful when summarized across jurisdictions. However, we have not attempted, nor do we think it possible, to establish the relative accuracy of a particular state.

The authors of the attached reports have analyzed the best data available at this time. It should be noted that in the years following Census 2000, as in past censuses, the Census Bureau will prepare an extensive array of detailed evaluations of many aspects of both the initial census and the A.C.E. A list of the evaluation categories and their projected completion dates is attached. These evaluations will not be available for months, or in some cases, years, after the Census Bureau is required by law to provide redistricting data to the states. These final evaluations, as distinguished from these analytical reports that inform the ESCAP Committee, will be accomplished without the pressure of a legal deadline, will be based on additional information, and may, in some instances, reach conclusions different from those in certain of these reports.

Accuracy and Coverage Evaluation (A.C.E.) Dual System Estimates

The A.C.E. indicates that Census 2000 reduced both net and differential undercoverage over the levels measured by the 1990 Post-Enumeration Survey (PES). The net national undercount is estimated to have been reduced from the 1990 rate of 1.61 percent (0.20 percent standard error) to 1.18 percent (0.13 percent standard error). The estimated undercount rate for the Non-Hispanic Blacks domain dropped from 4.57 percent (0.55 percent standard error) to 2.17 percent (0.35 percent standard error), and the estimated undercount rate for the Hispanics domain dropped similarly from 4.99 percent (0.82 percent standard error) to 2.85 percent (0.38 percent standard error). In addition, the estimated undercount rate for children dropped from 3.18 percent (0.29 percent standard error) to 1.54 percent (0.19 percent standard error). (Report B-9, “Dual System Estimation Results”)

Nonetheless, the improvements demonstrated in Census 2000 do not mean that complete coverage has been achieved or that differential coverage has been eliminated. On the contrary, the A.C.E. indicates that Census 2000 perpetuated longstanding patterns of differential coverage, with minority groups and children exhibiting lower coverage rates. The Census 2000 percent net undercount for the non-Hispanic Black and the Hispanic domains, 2.17 percent and 2.85 percent respectively, remain significant, as does the Census 2000 percent net undercount for children of 1.54 percent.

Tenure continues to be an important characteristic to distinguish coverage. The A.C.E. indicates that the pattern of differential coverage continues despite improvements in Census 2000. The percent net undercount for non-owners was 2.75 percent (0.26 standard error) as compared with an estimated net undercount for owners of 0.44 percent (0.14 standard error). While this is a distinct improvement over the percent net undercount for non-owners in the 1990 census, which is estimated at 4.51 percent (0.43 standard error), the A.C.E. indicates that the estimated undercount for this population is significant as well.

In addition, the undercount for minority renters also remains high. The

non-owner undercount for non-Hispanic Blacks was estimated to be 3.58 (0.48 standard error), for Hispanics 4.32 (0.55 standard error), for Asians 1.58 (0.98 standard error), for Hawaiians and Pacific Islanders 6.58 (4.07 standard error), and for American Indians not on reservations 5.57 (2.02 standard error).

Tables 2a and 2b provide the percent net undercount for the race/origin domains, tenure, and age/sex groups for Census 2000 and the 1990 census.

Table 2

—

Percent Net Undercount for Major Groups: 2000 A.C.E.

Estimation grouping

Net undercount (percent)

Standard error

(percent)

Total population in Households

1.18

0.13

Race and Hispanic Origin:

American Indian and Alaska Native (on reservation)

4.74

1.20

American Indian and Alaska Native (off reservation)

3.28

1.33

Hispanic Origin (of any race)

2.85

0.38

Black or African American (not Hispanic)

2.17

0.35

Native Hawaiian and Other Pacific Islander (not Hispanic)

4.60

2.77

Asian (not Hispanic)

0.96

0.64

White or Some Other Race (not Hispanic)

0.67

0.14

Age and Sex:

Under 18 years

1.54

0.19

18 to 29 years:

Male

3.77

0.32

Female

2.23

0.29

30 to 49 years:

Male

1.86

0.19

Female

0.96

0.17

50 years and over:

Male

−0.25

0.18

Female

−0.79

0.17

Housing Tenure:

In owner-occupied housing units

0.44

0.14

In nonowner-occupied units

2.75

0.26

Table 2

—

Percent Net Undercount for Major Groups: 1990 PES

Estimation grouping

Net undercount (percent)

Standard error

(percent)

Total Population

1.61

0.20

Race and Hispanic Origin:

White or Some Other Race (not Hispanic)

0.68

0.22

Black or African American

4.57

0.55

Hispanic Origin

4.99

0.82

Asian and Pacific Islander

2.36

1.39

American Indian and Alaska Native (on reservation)

12.22

5.29

Age and Sex:

Under 18 years

3.18

0.29

18 to 29 years:

Male

3.30

0.54

Female

2.83

0.47

30 to 49 years:

Male

1.89

0.32

Female

0.88

0.25

50 years and over:

Male

−0.59

0.34

Female

−1.24

0.29

Housing Tenure:

In owner-occupied housing units

0.04

0.21

In nonowner-occupied housing units

4.51

0.43

Review of the Quality of the Census Operations

Conclusions for This Section

While many elements of the design of Census 2000 were fundamentally similar to the 1990 census, there were numerous major changes. These included involving local governments in the address list building process, increasing methods for answering the census, designing a simplified questionnaire, developing a multi-step mailing strategy, creating a paid advertising campaign, and restructuring the pay scale for temporary workers. The paid advertising campaign (over $100 million dollars) allowed for a saturation of census awareness across the nation, particularly for the minority communities. The restructured pay scale meant that the census could compete successfully with other employers to hire the number and quality of field workers it needed to conduct the census well.

Operationally, Census 2000 was a success. The census data collection was accomplished on schedule with only a few exceptions. A review of the evidence from field reports and quality assurance processes indicates that Census 2000 programs functioned effectively within design parameters.

Analysis Reports Important to This Section

(All Analysis Reports cited in the text are in the DSSD Census 2000 Procedures and Operations Memorandum Series B)

• Report B-2: “Quality Indicators of Census 2000 and the Accuracy and Coverage Evaluation,” by James Farber.

• Report B-3: “Quality of Census 2000 Processes,” by James B. Treat, Nicholas S. Alberti, Jennifer W. Reichert et al.

Discussion

As documented extensively by Census Bureau and outside statisticians, every census since at least 1940 has experienced both a net undercount and a substantial differential undercount. In particular, the data reveal a persistent differential undercount between the Black and non-Black populations, as well as differential undercounts for other minority groups and for children.

Many elements of the design of Census 2000 were fundamentally similar to the design of the 1990 census. Address lists were prepared using a variety of sources, and questionnaires were delivered to each address on the list. Questionnaires were principally delivered by the U.S. Postal Service; however, in areas with rural style addresses, census workers delivered the questionnaires. Households receiving questionnaires were asked to return the questionnaires by mail, although in some very rural or isolated areas households were interviewed by census enumerators as the enumerators verified and updated the address list. Those addresses that did not return a questionnaire by mail were followed up by census workers in the Nonresponse Follow-up operation (NRFU). NRFU was followed by special coverage improvement follow-up operations, which, among other things, included contacting addresses listed as vacant or nonexistent by the NRFU field staff. Each of these operations had its own quality control procedures.

The Census 2000 plan, however, included several important innovations to the census process designed to improve census accuracy. Prior to Census 2000, the Census Bureau worked closely with local and tribal governments through the Local Update of Census Addresses (LUCA) program to review and update the address list. During LUCA, local and tribal government officials were given the opportunity to review the Census Bureau's address list and identify missing addresses for inclusion in the census. The Census Bureau also implemented the New Construction Program, during which local governments were invited to submit addresses for housing units that had been built subsequent to the completion of the address list in January 2000. The “Be Counted” program was also new in Census 2000. “Be Counted” forms were provided to individuals who believed that they might have been missed in the initial distribution of census questionnaires, as well as to individuals without any usual residence. The “Be Counted” forms were made available to the public at walk-in Census 2000 assistance centers and at a variety of public locations identified through consultation with local organizations. In addition, Census 2000 questionnaires were available upon request in six languages and language assistance guides were available in more than forty languages. Households also were given the opportunity to respond to Census 2000 by telephone or via the Internet.

To encourage households to respond to Census 2000, the Census Bureau initiated the largest promotion and outreach effort in its history for a decennial census. The Census Bureau established approximately 140,000 partnerships with a wide range of government and nongovernment organizations at the national and local levels. Organizations throughout the United States and Puerto Rico implemented promotional activities to educate the public about the importance of participating in the census. Then, starting in November 1999, the Census Bureau launched the first-ever paid advertising campaign for a census. This campaign was extended in targeted cities to encourage cooperation with enumerators during the NRFU operation. Other efforts included the distribution of numerous news releases and a number of video news feeds tailored to local areas to media outlets to generate media coverage during the various stages of Census 2000.

The Census Bureau then implemented the A.C.E. because it expected that, while these innovations would improve the results of the census, the phenomenon of the differential undercount would continue. The A.C.E. is designed to serve as a quality check on the census counts obtained after all other operations planned for Census 2000 were completed. In effect, the goal of the A.C.E. is to make a good census even better.

The discussion in this document is not meant to be a complete evaluation of census operations, but rather focuses on information relevant to the question of the level and pattern of census omissions or erroneous inclusions, because this information is directly relevant to understanding and assessing the results of the A.C.E.

We will discuss what is known about the following major operations:

• Address List Development

• Questionnaire Delivery and Return

• Nonresponse Follow-up

• The “Be Counted” Campaign

• Coverage Edit Follow-up

• Coverage Improvement Follow-up

• Housing Unit Duplication Operation

The Housing Unit Duplication Operation was a special operation designed and instituted after the Coverage Improvement Follow-up to reduce the level of housing unit duplication. This operation has special implications for census coverage and the conduct of the A.C.E.

• Primary Selection Algorithm

• Unclassified Unit and Missing Data Estimation

Address List Development

Address list development was conducted over several years, and the vast majority (96.7 percent) of addresses were listed before questionnaire delivery. One major change from previous censuses was the inclusion of the Local Update of Census Addresses (LUCA) program, during which the Census Bureau solicited the help of local governments in the address list building operation. LUCA was successful in adding approximately five million housing units to the address list

. However, anecdotal evidence suggests that the LUCA program may also have contributed duplicate addresses to the Master Address File (MAF). Duplicate addresses may have been erroneously added because the Census Bureau and local governments refer to the same address in different ways.

Many of the adds were also added by other operations. At this time, we do not know the extent of the overlap. That is, the five million figure cannot be considered as a net addition.

The address list development process included several quality assurance programs. These programs had the following objectives: to prevent errors due to lack of knowledge or understanding on the part of the lister, to control coverage and content errors, and to promote continuous improvement of performance. In general, the preliminary quality

assurance results for address list development are within the expected range for each of the programs.

Questionnaire Delivery and Return

The United States Postal Service or census workers delivered questionnaires to the vast majority of addresses on the address list. As in previous censuses, a certain number of questionnaires were misdelivered. For example, the questionnaire intended for Apartment A might have been delivered to Apartment B and vice versa. We have not quantified the level of questionnaire mis-delivery.

Householders were asked to return the questionnaires by mail. Since the Census Bureau does not expect mail responses from vacant or nonexistent housing units, the relevant measure of cooperation is the return rate, that is, the proportion of occupied housing units that returned their questionnaire. This measure differs from the response rate, which while available earlier in the census process, includes vacant and nonexistent housing units in the denominator. As measured by the return rate, the cooperation of the public with Census 2000 was approximately the same as in the 1990 census. The 2000 return rate (72 percent) is approximately the same as the 1990 return rate (74 percent). The comparison is not exact because the universes are slightly different.

Considering the general trend downward in return rates between censuses and for survey interviews in general, the Census Bureau considers the Census 2000 return rate to be a major success.

Nonresponse Follow-up

During Nonresponse Follow-up (NRFU), census workers visited each household that the address list identified as not yet having returned a mail questionnaire. In Census 2000, approximately 42 million households were included in the NRFU process. Thanks in large part to adequate funding provided by Congress, pay rates and levels of staffing in 2000 were far better than in the past two censuses. We believe that this increased funding and the ability to hire adequate staff contributed to an improvement in NRFU quality.

For most LCOs, NRFU was completed as scheduled in a nine-week period between April 27, 2000, and June 26, 2000. This performance compares favorably with 1990, when NRFU was conducted over a 14-week period from April 26 though July 30. The Census Bureau believes that, all other things being equal, NRFU interviews conducted closer to Census Day are likely of higher quality.

Local NRFU problems were identified in a few local census offices, including the local census office in Hialeah, Florida. The Census Bureau responded to the localized problems in the Hialeah office by re-enumerating certain areas that were believed to have faulty data. The Census Bureau does not believe that net coverage in the Hialeah or any other local census office was substantially affected by these local problems; the NRFU operation for the nation as a whole was good to excellent.

The NRFU quality assurance program was conducted through a random and targeted reinterview program which had the following three objectives:

• Prevent errors due to lack of knowledge or understanding

• Control coverage and content errors

• Promote continuous improvement of performance

Preliminary NRFU quality assurance results show that the reinterview workload was 6 percent, slightly above the expected workload of five percent. Discrepant cases were found in approximately three percent of the reinterview cases. Some local census offices experienced delays in starting their reinterview programs, which may have hindered the reinterviewers' ability to accurately verify the census data. A significant number of quality assurance forms were lost and/or completed incorrectly. (Report B-3, “Quality of Census 2000 Processes”)

In spite of local imperfections, the NRFU program as a whole was largely successful. The better pay and staffing seemed to have resulted in a more professional and controlled labor force. The local problems and quality assurance shortcomings were similar to problems encountered in previous censuses and should be expected in any nonrecurring operation of this magnitude.

Be Counted Campaign

The “Be Counted” campaign was designed to allow people who thought they may have been missed by the census to send in a “Be Counted” form, listing themselves and their April 1, 2000, address. The Census Bureau had hoped that this campaign would allow for improved cooperation and coverage. The National Academy of Science and others feared that large numbers of “Be Counted” forms would overwhelm the system and lead to increased person duplication.

Neither the hopes nor the fears relating to the “Be Counted” campaign were realized. The Be Counted workload was only approximately 600,000, with no large local clusters observed. Its impact on net coverage for any group or area was minimal, and it is not believed to have contributed to housing unit duplication.

Coverage Edit Follow-up

Under certain circumstances, the Census Bureau would call a responding household on the telephone to gain additional information. This extra effort, called Coverage Edit Follow-up (CEFU), was designed to improve within-household coverage, especially for large households. The census questionnaire had room to collect data for six people and asked the respondent living in a household with more than six people to list the additional residents. In CEFU, enumerators called these households and gathered the required information about the additional residents. In addition, CEFU was designed to follow up count discrepancies, or cases where the population count on the front of the questionnaire differed from the number of person responses inside the questionnaire.

Due to computer problems, the start of CEFU was delayed until May 8, 2000. It ran through August 13, 2000. Originally, it was planned for April 5, 2000, through June 19, 2000. This delay may have made it more difficult to obtain good information from households with more than six residents because some of the residents may have moved. In addition, CEFU had no provision to contact large households without telephones. When the Census Bureau could not secure good CEFU data on listed additional residents, it imputed their characteristics; to do otherwise would have decreased net coverage. Thus, the CEFU operation may have resulted in some small coverage loss compared to previous censuses, but this possible loss has not yet been quantified and is not expected to be significant, given the use of imputation.

Coverage Improvement Follow-up

Coverage Improvement Follow-up (CIFU) was designed as a check on addresses that were determined during the NRFU operation to be vacant or deleted (nonexistent). CIFU was also used for addresses requiring follow-up that were identified too late to be included in NRFU. CIFU was conducted from June 26 until September 13. Both the 1980 and the 1990 censuses included similar operations.

CIFU was conducted on 6.5 million addresses for which the housing unit was listed as vacant or non existent in NRFU. CIFU determined that 1.5 million of these units were, in fact,

occupied. In addition, CIFU included 2.2 million other addresses that had been added to the MAF after the initial mail out, such as those that resulted from the New Construction or Update/Leave programs.

The quality assurance procedures on CIFU included a questionnaire review, a dependent review and data entry quality assurance. The dependent review was conducted on housing units identified as vacant or nonexistent and excluded certain occupied units for time and budgetary considerations. Some districts may have had a difficult time completing all of the dependent reviews. A significant number of quality assurance forms were lost and/or completed incorrectly. These lost/incorrect forms make any analysis of outgoing quality difficult. (Report B-3, “Quality of Census 2000 Processes”)

Housing Unit Duplication Operation

The Census Bureau observed tentative indications as the census progressed that the MAF might contain a significant number of duplicate addresses. The Census Bureau also concluded that the Hundred Percent Census Unedited File (HCUF) might contain a significant number of duplicated persons, many of which are assigned to duplicated addresses. The Census Bureau responded to this problem by designing and conducting the Housing Unit Duplication Operation (HUDO). While this program was not prespecified, the Census Bureau believed that failure to address this potential problem could impair the accuracy of the apportionment numbers. Using the results of an address matching operation and a person matching operation, 2,411,743 address listings (address ID's) were analyzed on an aggregate basis to see whether these addresses were likely to correspond to other addresses already contained in the listing. Based on this analysis, 1,392,686 address IDs were permanently removed from the HCUF; after further review to identify units that may have been removed in error, the remaining 1,019,057 addresses were reinstated and included in the census results. The HUDO was designed solely to remove address/housing unit duplication. The software used for this process was carefully checked.

Primary Selection Algorithm

Census questionnaires contain a unique ID, an identifier that the Census Bureau uses to make sure it records the information for each household only once. Nonetheless, the Census Bureau sometimes receives more than one questionnaire for a single address ID. For example, a household might mail back its questionnaire after the Census Bureau had already created NRFU assignment lists; a NRFU interviewer would then get an interview for a household that had already mailed back its response. As a further example, a “Be Counted” form might be received for a household with a completed census questionnaire. Since NRFU households identified as vacant are sent to CIFU, sometimes multiple questionnaires are generated by design. That is to say that we expect to have one questionnaire from NRFU and another from CIFU. The Primary Selection Algorithm (PSA) examines these multiple questionnaires to form one household to represent the housing unit in the census, sometimes by combining information from more than one questionnaire. The PSA was designed to prevent both overcoverage (including people more than once) and undercoverage (deleting too many people).

Multiple returns were received for less than 10 percent of the address IDs. However, many of these multiple returns were from vacant units or multiple listings of the same people on two IDs. The number of people in a household was found to be larger than the number reported on the most complete questionnaire for fewer than 300,000 IDs. In other words, PSA resulted in an increase of individuals in fewer than 300,000 housing units.

Although no formal evaluation has been completed, the PSA was well programmed and well tested. The results are consistent with the overall design of the PSA and of the census.

Unclassified Unit and Missing Data Estimation

As in the past, Census 2000 had some housing unit records listed on the MAF for which the Census Bureau could not gain information. In addition, there were a small number of housing units which the Census Bureau knows to be occupied but for which it could not secure precise information about the individuals living in that unit. The census process could not always determine whether other units are occupied or vacant. Sometimes, the unit was determined to be occupied, but the number of residents could not be determined. In each of these cases, a statistical process known as “imputation” is used to estimate the number of people living in these units.

Preliminary results indicate that almost 0.4 percent of person records were in housing units on the preliminary HCUF were missing a status of occupied, vacant or nonexistent, indicating that the residents of the housing unit were imputed. For states, the imputation percent ranged from 0.2 percent to 1.1 percent. In 1990, about 0.02 percent of people in unclassified units were imputed.

In addition, Census 2000 encountered whole households where the number of people could be determined, but the person records for these residents were missing. In accordance with past practice, the Census Bureau used imputation techniques to estimate characteristics for these people. About 0.8 percent were imputed with this technique.

The total person substituted persons in the Census 2000 is approximately 1.3 percent. The percent of substituted persons in 1990 was only about 0.7 percent.

Review of A.C.E. Operations

Similar to its review of the operations in the initial census, the Census Bureau has reviewed the A.C.E. operations to identify any deviations from specified procedures and to assess the extent to which the operations were under management control.

Proper Execution of the Steps Between Processing and Estimation

Conclusions for This Section

The A.C.E. was carried out as designed, with only minor modifications. Each modification was well documented and justified by good statistical practice. No steps were skipped because of lack of time or resources, and there was no manipulation of the results or distortions resulting from outside pressures. There is a clear and traceable path from the data collected by the interviewer to the final results. The Census Bureau carried out the A.C.E. according to its public plan, and the steps between processing and estimation were properly executed.

Analysis Reports Important to This Section

• Report B-7: “Accuracy and Coverage Evaluation Survey: Missing Data Results,” by Patrick J. Cantwell.

• Report B-8: “Accuracy and Coverage Evaluation Survey: Decomposition of Dual System Estimate Components,” by Thomas Mule.

• Report B-9: “Accuracy and Coverage Evaluation Survey: Dual System Estimation Results,” by Peter P. Davis.

Discussion

The A.C.E. methodology planned for Census 2000 involves comparing (matching) the information from an independent sample survey to initial census records. In this process, the

Census Bureau conducts field interviewing and computerized and clerical matching of the records. Using the results of this matching, the Census Bureau applies the statistical methodology of Dual System Estimation (DSE) to develop coverage correction factors for various population groups. The results are then applied to the census files to produce the adjusted census data.

One concern sometimes expressed about statistical correction is that statistical processes could be subject to manipulation. The Census Bureau believes that this notion is not well founded. The A.C.E. was publicly prespecified to assuage these concerns. The first step in reviewing the A.C.E. is to evaluate A.C.E. operations to determine whether the prespecified procedures were followed and documented. The Census Bureau's analysis found that all planned A.C.E. operations were carried out in close adherence to the prespecified design, with the two exceptions noted below.

The supporting analysis reports review each of the steps in the A.C.E. operation from the creation of the A.C.E. micro-records to the computation of the final adjustment factors. In particular, Report B-8, “Decomposition of Dual System Estimate Components,” presents an accounting of the A.C.E. estimation components so that the results can be independently verified. Beginning with records with complete data (meaning records with both post-stratification variables and enumeration status) the accounting then proceeds through each stage of missing data adjustment and sample weighting until the final weighted “matched” results are provided (which are the results that are the input data for the dual system estimates). Report B-8 allows an informed reader to see clearly how the final results were derived and to understand the relative effect of the estimation steps on the results.

Report B-7, “Missing Data Results,” shows in detail the effects of individual missing data estimation steps upon the weighted matching results. Report B-9, “Dual System Estimation Results,” provides detailed DSE computations together with useful “roll-ups” that aggregate the results by age and sex, minority/nonminority, or other useful summations. This document allows the reader to verify how the final coverage correction factors are computed from the input data.

These three documents, taken together, demonstrate how the final coverage correction factors were derived from the micro-level data and document that the prespecified procedures were followed, with the following exceptions.

The following two changes from the prespecified procedures arose from the unexpected availability of important information in time to improve the A.C.E. estimation:

• The A.C.E. plan provided that cells could be collapsed because of cell size but did not explicitly include variance as a reason for collapsing. We modified these rules because the estimated variance for one cell was unusually large. The design had not anticipated having variance estimates available in time to permit their use in collapsing. When the variances became available earlier than anticipated, the Census Bureau's statistical staff determined the collapsing of “outlier” poststrata was appropriate. This change did not deviate from the purpose or spirit of the prespecified collapsing rules but allowed a more precise application. The change was discussed with the ESCAP and documented.

• Our method for imputing unresolved match and residency status, namely imputation cell estimation, was modified because the results of the A.C.E. follow-up forms became available during the missing data estimation process (Report B-7, “Person Matching and Follow-up Results”). The prespecified design had not anticipated that these data would be available in time to be used in missing data estimation. Analysis of the data indicated that some cases grouped together in the initial missing data design could be separated based on the keyed follow-up results, allowing for a more precise imputation. This change is consistent with normal statistical practice and was discussed with the ESCAP and documented.

Conduct and Control of the A.C.E. Operations

Conclusions for This Section

The A.C.E. was an operational success; it was properly conducted and encountered no unanticipated difficulties. Listing, interviewing, matching, and follow-up were all conducted as designed and were all in control.

A more extensive description of the A.C.E. can be found in Howard Hogan's paper, “Accuracy and Coverage Evaluation: Theory and Application,” prepared for the February 2-3, 2000, DSE Workshop of the National Academy of Sciences Panel to Review the 2000 Census; and Danny R. Childers and Deborah A. Fenstermaker, “Accuracy and Coverage Evaluation: Overview of Design,” DSSD Census Procedures and Operations Memorandum Series S-DT-02, U.S. Census Bureau, Washington, D.C., January 11, 2000.

Analysis Reports Important to This Section

• Report B-5: “Accuracy and Coverage Evaluation Survey: Person Interviewing Results,” by Rosemary L. Byrne, Lynn Imel, and Phawn Stallone.

• Report B-6: “Accuracy and Coverage Evaluation Survey: Person Matching and Follow-up Results,” by Danny R. Childers, Rosemary L. Byrne, Tamara S. Admas, and Roxanne Feldpausch.

• Report B-7: “Accuracy and Coverage Evaluation Survey: Missing Data Results,” by Patrick J. Cantwell.

Discussion

The second aspect to this review is to establish that the A.C.E. operations were well conducted and well controlled. Reports B-5, “Person Interviewing Results,” B-6, “Person Matching and Follow-up Results,” and B-7, “Missing Data Results,” taken together, establish that the operational quality of the A.C.E. was generally good and that the prespecified design was well followed.

Interviewing

One change from 1990 was the introduction of telephone interviewing. The Census Bureau implemented a telephone program to enhance the efficiency and quality of the A.C.E. interview. The Census Bureau believed that shortening the elapsed time from Census Day to the A.C.E. enumeration would improve data quality and that beginning interviewing early in a more easily controlled environment would allow the A.C.E. supervisors to gain valuable experience in conducting interviews and in operating their laptop computers before training the enumerators. The Census Bureau designed this process to maintain the independence between the A.C.E. and the other Census 2000 operations.

A.C.E. interviewing was an operational success. The A.C.E. interviewing finished on schedule by September 1, 2000, in every local census office except the Hialeah office, where census NRFU interviewing finished late (September 11, 2000) due to local difficulties. There were no major disruptions or delays introduced by the Computer Assisted Personal Interviewing (CAPI) instrument. The timely interviews allowed the Census Bureau to have an orderly completion of interviewing was a major accomplishment.

Twenty-nine percent of the total A.C.E. workload was completed during the telephone phase (April 24 through June 13). These A.C.E. interviews were conducted much closer to Census Day

(April 1) than had been possible in 1990, thereby reducing recall bias (the phenomenon of a respondent not remembering the actual situation several months earlier). By design, the telephone phase was restricted to a limited universe of households that were deemed unlikely to have any exposure to continuing census operations. These were primarily households that had mailed back their questionnaires, that had included a telephone number on the questionnaire, and that did not live in certain multi-unit or rural structures. The Census Bureau's conservative use of this interview mode meant that more than 99 percent of the telephone cases were classified as complete or partial interviews and were conducted with a household member.

Table 3.—Distribution of Interviews by Week—Unweighted

Phase

Week starting on

Number of cases

Cumulative percent of person interviewing workload

Telephone

April 23, 2000

7,699

2.6

April 30, 2000

20,590

9.4

May 7, 2000

25,638

17.9

May 14, 2000

19,7282

4.5

May 21, 2000

10,497

28.0

May 28, 2000

3,232

29.1

June 4, 2000

1,154

29.5

June 11, 2000

29.5

Personal Visit

June 18, 2000

45,204

44.5

June 25, 2000

57,241

63.5

July 2, 2000

41,642

77.3

July 9, 2000

31,344

87.7

July 16, 2000

17,038

93.4

July 23, 2000

7,764

96.0

July 30, 2000

5,057

97.7

Aug 6, 2000

3,982

99.0

Aug 13, 2000

1,756

99.6

Aug 20, 2000

939

99.9

Aug 27, 2000

336

100.0

Sept 3, 2000

100.0

Sept 10, 2000

100.0

Source: Accuracy and Coverage Evaluation Survey 2000— housing unit data collected by the Computer Assisted Personal Interviewing (CAPI) instrument. Report B-5, “Person Interviewing Results.”

The automated Computer Assisted Person Interviewing (CAPI) increased the quality of the data captured in the A.C.E. interviews, as the instrument included data edits to ensure a predetermined quality of data before the interview was considered complete. This was not possible with the paper and pencil 1990 instrument. It insured that the interviewer followed the correct path through the interview. CAPI also allowed quick feedback to the interviewers. The Census Bureau's observations and debriefings indicated that CAPI instilled the interviewers with a sense of professionalism and purpose. Observations also indicated that the use of laptop computers enhanced the respect and cooperation exhibited toward the interviewers by the respondent households thereby leading to improved A.C.E. data quality. However, there were a couple of small problems with the CAPI instrument that had minor impacts on quality.

The Nonresponse Conversion Operation (NRCO) was designed to “convert” nonresponse cases, that is, to obtain A.C.E. information for nonresponding households. On a national basis, the NRCO operation successfully converted 70.8 percent of its cases to complete interviews and 14.1 percent to partial interviews. Only 2.2 percent of the cases finished as refusals.

A.C.E. interview rates were very high. The A.C.E. asked questions about both the household living at the address on Census Day and the current household. Because of this, there are two measures of household nonresponse. The rate for occupied housing units on Census Day was 97.1 percent; on the date of the A.C.E. interview, the rate for occupied housing units was 98.8 percent.

These rates compare favorably to the approximately 98.4 percent (unweighted) in the 1990 Post-Enumeration Survey. The unweighted rates for 2000 were 97.0 and 98.9, respectively. Due to the high rate of response, most of the noninterview adjustment factors were very close to one. Consequently, this operation did not change the final weights very much. This helps to keep down the variance of the survey weights.

Missing data rates for characteristic data were very low, ranging from 1.4 percent to 2.4 percent. Compared to the 1990 PES, the rates of characteristic missing data are slightly higher for the age and sex characteristics and slightly lower for tenure and race. Again, this is indicative of good quality interviewing.

The goal of A.C.E. interviewing quality assurance was to ensure that the interviewers did, in fact, visit the designated households, and to prevent systematic errors caused by of lack of knowledge or understanding. The evidence indicates that the A.C.E. interviewing quality assurance operation was properly implemented and successful. A total of 11.6 percent of the cases were subject to random or targeted quality assurance checks. We assume that the 88.4 percent of the cases not in quality assurance share the favorable error rates of the randomly selected cases (0.13 percent). This may have been reduced further as 171 of the remaining errors were corrected in the targeted QA sample.

Matching and Follow-up

Matching refers to the process of determining whether an individual enumerated in the A.C.E. was the same person as an individual enumerated in the census. The matching and follow-up

process also determines whether a census record in the E-Sample

was complete and correct. Errors in matching can significantly affect undercount estimates; highly accurate matching and processing are an important component of A.C.E. methodology.

The E-sample refers to the sample of census data defined person records selected for inclusion in the A.C.E. The P-sample refers to the independent sample of people included in the initial A.C.E. interview.

Although neither Secretary Mosbacher nor the Committee on Adjustment of Postcensal Estimates (CAPE) identified matching error as a significant problem with the 1990 PES, the Census Bureau made significant improvements to the matching process in the 2000 A.C.E. design. The A.C.E. computer matched the P-sample to the census using the Census Bureau's Statistical Research Division Record Linkage System, a system that the Census Bureau has been developing, testing and using for nearly two decades. Clerical personnel at a centralized location reviewed records that were not matched by the computer matcher. The Census Bureau utilized an ample staff of over 200 clerks, 46 technicians, and 16 analysts so that each successive level of review could perform quality assurance on the previous level. Higher level staff independently reviewed a sample of each employee's work, a process designed to identify random matching errors. Each of the matching levels improved on the previous level. The clerks matched what the computer could not. The technicians worked on any cases the clerks could not resolve and performed the quality assurance on the clerks' cases. Then the analysts finished any cases the technicians could not resolve and performed quality assurance on the technicians' cases.

The results indicate computer matching of 69.6 percent of the P-sample and 64.4 percent of the E-sample. The computer matcher assigned matches very conservatively. Numerous studies over the years have shown that this operation produces insignificant numbers of false matches. Therefore, all questionable matches, possible matches, and near matches are left for clerical review. All nonmatches were clerically reviewed.

We have quality assurance results only on the quality of the clerical matching in the before follow-up stage and the first three stages of after follow-up. The Census Bureau measures matching quality relative to the results that would be produced by the Census Bureau's most experienced and best trained matchers, the 16 analysts permanently employed by the Census Bureau. The quality of the matching process is further measured in terms of changes made by the next level of review; this process tends to overstate the matching error, as not all changes are the result of erroneous matching. However, given these caveats, the outgoing quality rate (the final match rate) for before follow-up was well more than 99 percent. For after follow-up, the outgoing quality rate was also well more than 99 percent. These rates are calculated based on the before follow-up and the after follow-up workload and not on the total number of sample cases, that is, they do not include the cases matched by computer. These rates exceed expectations and are indicative of high quality matching.

Person follow-up is also an important A.C.E. process. The follow-up resolves possible matches and, most importantly, determines which E-sample nonmatches are, nonetheless, correctly enumerated in the census. The person follow-up interviews were conducted either by permanent census field staff or by experienced decennial interviewers and the quality assurance operation was targeted at ensuring that the interview was conducted. Of the randomly selected person follow-up quality assurance cases, 0.45 percent resulted in a discrepancy, that is, only 0.45 percent determined that the person follow-up interview may not have been conducted. We can assume that the remaining 84,843 cases not randomly selected for quality assurance have the same rate of failure or roughly 400 cases total that may have not been conducted. In addition, we corrected 84 of those cases in the targeted samples.

Review of A.C.E. Quality

The review in the previous section established that the A.C.E. was conducted as designed. This section will take the next step and evaluate the quality of the A.C.E. as implemented.

Our review of A.C.E. quality has two aspects. First, we review the available data relating to selected individual components of A.C.E. error. The second part of the A.C.E. quality review synthesizes what is known about the components of error into a few indicators of overall relative accuracy for both the adjusted and the unadjusted census results.

Individual Components of A.C.E. Quality

Sampling Variance

Conclusions for This Section

The A.C.E. significantly reduced sampling variance relative to the 1990 PES. This result was achieved by nearly doubling the sample size coupled with significant improvements in the sample design.

Analysis Reports Important to This Section

• Report B-9: “Accuracy and Coverage Evaluation Survey: Dual System Estimation Results,” by Peter P. Davis.

• Report B-11: “Accuracy and Coverage Evaluation Survey: Variance Estimates by Size of Geographic Area,” by Michael D. Starsinic, Charles D. Sissel, and Mark E. Asiala.

Discussion

The dual system estimate shows that the Census 2000 undercount rate for the national household population is 1.18 percent, with a standard error of 0.13 percent. The net undercount for the 1990 census was estimated at 1.61 percent, with a standard error of 0.20 percent (see table 2, above). Comparisons by poststrata between 1990 and 2000 are necessarily inexact as the universe differs (2000 includes only the household population) and the exact poststrata definitions are different. Still, some comparisons are instructive. The standard error for owners was reduced from 0.21 percent to 0.14 percent, and the standard error for non-owners fell from 0.43 percent to 0.26 percent. The measured standard error fell for all comparable race/origin groups and for each age/sex group. The estimated standard error was comparatively high for the two groups estimated separately for the first time: Hawaiian and Pacific Islanders (2.77 percent) and American Indians and Alaskan Natives living off reservation (1.33 percent). As we will see, these groups also had high levels of inconsistent reporting between the census and the A.C.E. The estimated standard error for American Indians living on reservations fell dramatically from 5.29 percent to 1.2 percent. The standard error for Asians was 0.64 percent. For Hispanics, it was 0.38 percent, and for non-Hispanic Blacks it was 0.35 percent.

Table 4 gives the estimated percent net undercount and standard errors for the 64 major poststratum groups. The standard errors for several groups are above 1 percent and for a few small groups are up to 4 percent. Because the populations of these groups are small,

their high variances will have only limited impact on geographic variance.

• For Census 2000, persons can self-identify with more than one race group. For post-stratification purposes, persons are included in a single Race/Hispanic Origin Domain. This classification does not change a person's actual response. Further, all official tabulations are based on actual responses to the census.

• A negative net undercount denotes a net overcount.

BILLING CODE 3510-07-P

EN08MR01.000

EN08MR01.001

• For Census 2000, persons can se

This text is long and has been trimmed here. Open the source document for the complete record.

This is a copy of a public record, reproduced as it was published. It is not legal advice, and it may not be the version a court would rely on. Check the official source before you cite it.