# Petition — Civil Service Commission v. Guardians Ass'n of the New York City Police Department, Inc.

> Briefs, arguments, decisions, and more.

URL: https://www.frixlaw.com/law-library/documents/brief%3Amicro_IA40385007_1629%3A1

## Record

- **Collection:** Supreme Court brief
- **Document type:** Petition
- **Published:** January 1, 1981
- **Citation:** 452 U.S. 940

## Text

Office-Supreme Court, U.
80-1743 FILED
a
be 1 APR 15 1981
ER UL. STEVAS,
IN THE er

Supreme Cowt of the United States

OCTOBER TERM, 1980

>_>

CIVIL SERVICE COMMISSION OF THE CITY OF NEW YORK,
DEPARTMENT OF PERSONNEL OF THE CITY OF NEW YORK,
and THE NEW YORK CITY POLICE DEPARTMENT,

Petitioners,
—_—vV ed

THE GUARDIANS ASSOCIATION OF THE NEW YORK CITY
POLICE DEPARTMENT, INC., THE HISPANIC SOCIETY OF
THE NEW YORK CITY POLICE DEPARTMENT, INC., NYDIA
I. DIAZ, JAMES MICHAEL HIDALGO, WILFRED CEBEL-
LERO, ANDRE LOPEZ, REINALDO SALGADO, DENISE SAN-
TOS, DEBORAH HOLMES and PAMELA OBEY, individually
and on behalf of all those similarly situated,

Respondents.

PETITION FOR A WRIT OF CERTIORARI TO THE
UNITED STATES COURT OF APPEALS
FOR THE SECOND CIRCUIT

ALLEN G. SCHWARTZ,
Corporation Counsel of

the City of New York,
Attorney for Petitioners,
100 Church Street,
New York, New York 10007.
(212) 566-4338/2183

LEONARD KOERNER,
JUDITH A. LEVITT,

of Counsel.

No.

IN THE

Supreme Court of the Wnited States

OCTOBER TERM, 1980

a an

CIVIL SERVICE COMMISSION OF THE CITY OF NEW YORK,
DEPARTMENT OF PERSONNEL OF THE CITY OF NEW YORK,
and THE NEW YORK CITY POLICE DEPARTMENT,

Petitioners,

—_V.—

THE GUARDIANS ASSOCIATION OF THE NEW YORK CITY —
POLICE DEPARTMENT, INC., THE HISPANIC SOCIETY OF
THE NEW YORK CITY POLICE DEPARTMENT, INC., NYDIA
I. DIAZ, JAMES MICHAEL HIDALGO, WILFRED CEBEL-
LERO, ANDRE LOPEZ, REINALDO SALGADO, DENISE SAN-
TOS, DEBORAH HOLMES and PAMELA OBEY, individually
and on behalf of all those similarly situated,

Respondents.

-

PETITION FOR A WRIT OF CERTIORARI TO THE
UNITED STATES COURT OF APPEALS
FOR THE SECOND CIRCUIT

QUESTIONS PRESENTED

1. Do bare statistics demonstrating a different pass rate for
white and minority candidates on the written portion of Exam-
ination No. 8155, which tested for special abilities necessary to

perform the job of police officer, without any supporting
evidence as to the relative qualifications of the white and
minority candidates in the pool, give rise to a presumption of
discrimination and satisfy plaintiffs’ burden of establishing a
prime facie case under Title VII of the Civil Rights Act of
1964, as amended?

2. Does the decision of the Court ot Appeals, holding that
Police Officer Examination No. 8155, although professionally
developed and job-related, could not be used for rank-ordered
selection of the best qualified candidates as mandated by the
New York Constitution and New York State Civil Service Law,
extend the scope of section 703(h) of Title VII beyond that
intended by Congress, in conflict with this Court’s decisions in
New York City Transit Authority v. Beazer, 440 U.S. 568
(1979), and United States v. South Carolina, 445 F. Supp. 1094
(D.S.C. 1977), summarily aff'd, 434 U.S. 1026 (1978)?

3. Assuming that rank-ordered selection of candidates from
Examination No. 8155 is not valid, did the Court of Appeals
err in also holding that the pass score, set at the score of the
12,000th highest scoring candidate, determined by the pro-
jected manpower needs of the Police Department during the
life of the eligible list, violates Title VII, because it may
eliminate certain candidates who have qualifications equal to
those of candidates who passed the examination?

SO OE regen ye Er OEE

eye

TABLE OF CONTENTS

PAGE
sg ETE SS TES AE RD A a eel l
ee Es oC ons endo 6 0X6 oe new ee l
Ps POU es hs 644944.) cs DAs Ace etree l
aPuneeeee SOe Bie CARE coe i ee hide ebe vee cade 2
REASONS FOR GRANTING THE WRIT .......... 6
1. In light of the ultimate burden upon plaintiffs to
prove discrimination, plaintiffs’ bare statistical
analysis was insufficient to establish a prima facie
case raising the presumption that Examination
No. 8155 was discriminatory .................. 8

2. The decision of the Court of Appeals invalidating
the use of Examination No. 8155 for the selection
of the best qualified candidates for the position of
police officer as required by New York law far
exceeds the scope of Title VII ..........ccceee. 10

ee OO eee re eee Sey re ne 14

TABLE OF AUTHORITIES

CASES: PAGE
Albemarle Paper Co. v. Moody, 422 U.S. 405 (1975) ..8, 11,
12
Castaneda v. Partida, 430 U.S. 428 (1977) ........... 9
Commonwealth v. O’Neill, 465 F. Supp. 451 (E.D. Pa.

eee ees de eee ead cae ras aeee 66 11
Firefighters Institute for Racial Equality v. City of St.

Louis, 549 F.2d 506 (8th Cir.), cert. den. sub nom.

United States v. Banta, 434 U.S. 819 (1977) ........ 13
Furnco Construction Co. v. Waters, 438 U.S. 507

SS. By ia Wai ae a a 5 OR Rie Coc ace ee a RA ee 8
Griggs v. Duke Power Co., 401 U.S. 424 (1971) ..... 6, 7, 10
Hazelwood School District v. United States, 433 U.S.

PED ha54 bees be Kee MEER EEDA A> 485060 ROERS 9
International Brotherhood of Teamsters v. United

UNE, Pe GA PO ROEED bwarnd eevee cccsscewrs 8, 9, 11
Jackson v. Nassau County Civil Service Commission,

424 F. Supp. 1162 CE.D.N.Y. 1976) 0... cc vcaccsees 11
Jefferson v. Hackney, 406 U.S. 535 (1972) ........... 8
Keyes v. School Dist. No. 1, 413 U.S. 189 (1973) ..... 8
Kirkland vy. New York State Dept. of Correctional Ser-

Weds, Fae eae See Gee CO, ISTS) ov ccccdascacecves 11
Louisville Black Police Officers v. City of Louisville, 20

Fumes Coes 195s Cem BE IGTD) cinscccccecvcnss 7
McDonnell Douglas Corp. v. Green, 411 U.S. 792

oe RP re er ae any anny Seeger ae 6

National League of Cities v. Usery, 426 U.S. 833 (1976) 13

PAGE

New York City Transit Auth. v. Beazer, 440 U.S. 568
SE so an Ue cued vas bo80seecusen esas cuueere 8, 9, 10, 13

Scott v. City of Anniston, 597 F.2d 897 (Sth Cir. 1979).. 13

Texas Dept. of Community Affairs v. Burdine, —_—
U.S. ____, 49 U.S.L.W. 4214 (March 4, 1981) ...... 8, 12

United States v. City of Chicago, 573 F.2d 416 (7th Cir.
SEE: cD cks tates cacinbecens VE RCEDESS See aeh Rane 7, 13

United States v. Commonwealth of Virginia, 454 F.
Supp. 1077 (E.D. Va. 1978) .....ccccccccccevceees 1]

United States v. South Carolina, 445 F. Supp. 1094
(D.S.C. 1977), summarily aff'd, 434 U.S. 1026 (1978) 10, 13

United States v. State of New York, 21 E.P.D. 4 30, 314

CEDIA. Y., Dat. G, TOT ova vccevctsecdccccncseee 7
Village of Arlington Heights v. Metropolitan Housing

Development Corp., 429 U.S. 252 (1977) .......... 8
Vulcan Society v. Civil Service Commission, 490 F.2d

be fe 8 rere err rrr ee 5
Washington v. Davis, 426 U.S. 229 (1976) ........4-- 8, 10

STATUTES, RULES AND REGULATIONS:

Title VII of the Civil Rights Act of 1964, as amended, 42
tk toe F fF Saree 12,4 7.8 Mm ia. i

Section 703(h), 42 U.S.C. § 2000e-2(h) ....6, 7, 11, 12

Uniform Guidelines on Employee Selection Procedures,

fet i ES oft. eee Perr err re 4,9
4 2 fo o. | Seer rrr ee yee Te 2
Bow Wark Civil Servics LOW «oa cs coccscnccoccavesens 4, 10

| DR errr es tee 1]

vi

PAGE
CONSTITUTIONS:
United States Constitution
Pe I a4 5 ae 66 3 oh os 0h ce ee cee sudo 13
Fourteenth AMENGMENt .......ccccvscscesceces 12
POU E Ue CIIUI oc ccc cece cctacncnee 4, 10
Cg a a ee re ee re 10

OTHER AUTHORITIES:

Lerner, Employment Discrimination: Adverse Impact,
Validity, and Equality, 1979 Supreme Court Review 17 7

Note, Title VII and Public Employees: Did Congress
Exceed Its Powers, 78 COL. L. REV. 372 (1978) ..... 13

STATEMENT

Petitioners, New York City Civil Service Commission, De-
partment of Personnel and Police Department, seek a writ of
certiorari to review a judgment of the United States Court of
Appeals for the Second Circuit which affirmed in part and
reversed in part an order of the United States District Court
for the Southern District of New York and remanded for entry
of a new decree.

Plaintiffs, black and Hispanic applicants for the position of
New York City Police Officer, challenged the written portion
of Police Officer Examination No. 8155 as discriminatory in
violation of 42 U.S.C. § 1983 and Title VII of the Civil Rights
Act of 1964, as amended, 42 U.S.C. §§ 2000e, et seg. The
Court of Appeals held that the test was job-related but that
Title VII prohibited its use for rank-ordered selection of the
best qualified candidates as mandated by the New York State
Constitution and Civil Service Law.

OPINIONS BELOW

The opinion of the Court of Appeals, 630 F.2d 79 (2d Cir.
1980), is annexed as Appendix A. The order of the Court of
Appeals, dated January 15, 1981, denying plaintiffs’ petition
for rehearing (not officially reported) is annexed as Appendix
B. The revised opinion of the District Court, 484 F. Supp. 785
(S.D.N.Y. 1980), is annexed as Appendix C. The order of the
District Court, entered January 11, 1980, is annexed as Appen-
dix D.

JURISDICTION

The judgment of the Court of Appeals was entered on July
31, 1980. Plaintiffs’ timely petition for rehearing was denied
on January 15, 1981. This Court’s jurisdiction is invoked
under 28 U.S.C. §§ 1254(1) and 1331.

2

STATEMENT OF THE CASE

On June 30, 1979, defendants administered the written
portion of Examination No. 8155 for the position of New York
City Police Officer. Reflecting the success of an intensive
recruitment drive aimed at attracting minority candidates,
approximately one-third of the nearly 37,000 candidates who
took the examination were black and Hispanic (App. 8a).*
That examination resulted in the establishment of an eligible
list based on test scores containing the names of approximately
13,000 candidates (App. 8a). Plaintiffs, black and Hispanic
applicants, commenced this action on October 5, 1979, chal-
lenging the written test as discriminatory in violation of 42
U.S.C. § 1983 and Title VII of the Civil Rights Act of 1964, as
amended, 42 U.S.C. §§ 2000e, ef seg., and seeking declaratory
and injunctive relief (A. 4-16).

After a trial on the merits, the United States District Court
for the Southern District of New York rendered a decision**
and order on January 11, 1980 (App. D), holding that the
examination violated Title VII and was intentionally dis-
criminatory. The District Court enjoined use of the list result-
ing from this examination unless 50% of the candidates ap-
pointed therefrom were black and Hispanic.

Defendants timely appealed to the Court of Appeals for the
Second Circuit, and on July 31, 1980, that Court entered a
judgment affirming in part and reversing in part the order of
the District Court and remanding for entry of a new decree
(App. A). The Court of Appeals reversed the finding of
intentional discrimination but held that the use of the examina-

° Unless otherwise indicated, numbers in parentheses preceded by
“App.” refer to the Appendices annexed hereto, numbers preceded by
“T.” refer to the original trial transcript, and numbers preceded by
“A.” refer to the Appellants’ Appendices submitted to the Court of
Appeals.

oe This decision was withdrawn and a revised opinion filed on January
23, 1980, during the pendency of defendants’ appeal to the Court of
Appeals, and after the appellants’ brief had been filed in that Court
(App. C).

3

tion for rank-ordered selection of the best qualified candidates
violated Title VII.

As found by the Court of Appeals (App. 4a-7a), the job
analysis and test preparation were thorough and done in a
professional manner. Petitioners respectfully refer this Court
to the opinion of the Court of Appeals for a detailed statement
of the procedures followed by the New York City Department
of Personnel and Police Department in developing Examina-
tion No. 8155.

Extreme caution was exercised throughout the test develop-
ment procedures to ensure that the examination was fair and
unbiased. Both in the job analysis and the test writing and
review, all samples and panels of police officers were repre-
sentative of the Police Department racially, ethnically and
sexually (A. 33-84). The language in the examination questions
was kept as simple as possible, and was, overall, on an eighth
grade reading level so that vocabulary would not be an obstacle
(App. 35a; T. 211; A. 359-62). In addition, extra time was
given for completion of the test so that reading difficulties
would not prevent candidates from competing equally (App.
35a).

Candidates were rank-ordered on the basis of their scores on
the examination (App. 7a). The passing grade was set at the
score of the 12,000th highest scoring candidate based upon the
projected manpower needs of the Police Department over the
course of the life of the eligible list (App. 8a).

Of the total candidates, 16.7% were identified as black,
14.2% were identified as Hispanic, 53.8% were identified as
white, and 13.2% did not identify themselves. Of those who
passed the examination, 7.6% identified themselves as black,
7.8% as Hispanic, and 66.6% as white (App. 8a).

Plaintiffs offered no statistics reflecting the ethnic composi-
tion of the 13.2% of the candidates who chose not to identify
themselves (16.5% of those who passed the examination), nor
evidence that the self-identification was accurate. No evidence
was submitted concerning the relative qualifications, education

4

Or motivation of the white and minority applicants who took
the test. And no evidence was offered to establish that the
qualifications necessary to become a police officer are
uniformly present in the labor pool.

Based upon the analysis of the bare statistics under both the
“four-fifths rule” established by the Uniform Guidelines on
Employee Selection Procedures, 29 C.F.R. § 1607.4(D) (1979),
and standard deviation analysis, the Court of Appeals held
that plaintiffs established a prima facie case of discrimination
under Title VII. in so doing, the Court found that to require
plaintiffs to substantiate the accuracy of the data submitted or
to offer evidence of relative qualifications would be too oner-
ous a burden upon plaintiffs (App. 13a, n.7).

The Court found, however, upon review of the job analysis
and test preparation, that Examination No. 8155 was content
valid and reasonably related to the job of police officer (App.
35a). Despite this finding of job-relatedness, the Court further
held that there was not “a demonstration of such substantial
test validity” (App. 37a) to support the use of the examination
for rank-ordered selection for the position of police officer as
required under the applicable provisions of the New York Civil
Service Law, and, to the extent practicable, New York’s Con-
stitution (App. 35a-45a). The Court also held that the cut-off
score—based upon the Police Department’s projected man-
power needs, rather than some concept of minimum compe-
tence—was invalid under Title VII (App. 45a-48a).

In reaching its conclusions regarding rank order and the pass
score, the Court relied in significant part on its finding that
one of the abilities necessary to perform the job of police
officer, i.e., human relations skills, identified as 30% of the
job, was inappropriately tested for on a written examination.
While it also indicated that if human relations skills had not
been tested for in some manner, the test would have been
unrepresentative of the job and therefore invalid, neither
plaintiffs nor the Court offered any alternative method for
testing for this ability (App. 30a-3la, 34a). Yet, based on this
“inappropriateness,” the Court found that defendants did not

prove that the examination was sufficiently job-related to
demonstrate that one-point differences in test score would
predict significant differences in job performance (App. 36a).*
The Court also relied on the fact that significant numbers of
candidates, including minority candidates, were bunched just
below the cut-off score and within the range of the standard
error of measurement (App. 46a-47a). Therefore, the Court
concluded, the use of this cut-off score might exclude from
employment certain candidates with qualifications equal to
those who passed the examination (App. 46a-48a).

It was not disputed that plaintiffs’ experts were at a loss to
identify any other, more valid testing device which could have
been used to select the most qualified candidates (T. 321-22,
389, 394, 496). Nor did plaintiffs offer any evidence that, had
Examination No. 8155 been prepared in a more perfect man-
ner, the disparate impact would have been reduced. Plaintiffs’
expert, Dr. Richard Barrett, who, in his extensive experience in
testing litigation, has never testified that an examination was
valid, in response to questioning by the District Court, stated
that he knew of no better test for the position of police officer
than the one administered by defendants (T. 394):

“THE COURT: Then let me take off. How would you
construct a job-related test? Maybe I can get rid of the
content.

THE WITNESS: [Dr. Barrett]: I don’t know how I would
do it. The issue as posed in the guideline has two facets to

° The Court noted, in addition to the defects in the test preparation,
the lack of evidence of reliability. In fact, the Court overlooked the
evidence in the record that the test had a reliability coefficient of .93,
admitted by plaintiffs’ expert to be very high (T. 477; A. 68-69).

The other evidence of validity which the Court found lacking was
either unavailable or unreliable, also as conceded by plaintiffs’ expert
(T. 367). A predictive validity study could not have been performed
since only 415 of the highest ranking candidates had been appointed. A
concurrent validity study would have been unreliable because of the
experience and motivation of the incumbent police officers, and any
pre-testing would have jeopardized the security of the examination. See
Vulcan Society v. Civil Ser: ice Commission, 490 F.2d 387, 395 n.10 (2d
Cir. 1973) ( Friendly, J.).

6

it, one is what test is used, and is there an alternative test
to be used?

And the other is, is there an alternative use of the same
test? And I can’t think of any alternative test. I have read
literature and talked to people who have been developing
them, and I have not been persuaded that these tests are
any better than what we had here.”

Despite this concession the Court of Appeals ignored the
testimony of defendants’ expert that evidence of validity of an
examination indicates that higher scorers will, in general,
perform better, and, therefore, not to use rank-ordered selec-
tion is to discard valuable information (T. 173-74, 225). More-
over, absent any evidence of feasible alternative selection
devices or discriminatory intent, the Court nullified the man-
date under New York law that, wherever practicable, candi-
dates for public employment should be selected in rank order.
It did this despite its acknowledgment that “[i]f a test is
content valid [as this was held to be], it may be reasonable to
infer that the test scores make some useful gross distinctions
between candidates” and that “[c]andidates with high scores
may well be expected to perform the job better than candidates
with low scores” (App. 36a).

REASONS FOR GRANTING THE WRIT

Ten years ago, in the context of the use of an intelligence test
to select employees “on the Company’s judgment that [such
tests] generally would improve the overall quality of the work
force,” this Court first interpreted that portion of section
703(h) of Title VII, 42 U.S.C. § 2000e-2(h), which authorizes
the use of professionally developed tests to select employees.
Griggs v. Duke Power Co., 401 U.S. 424 (1971). In that case,
this Court held that, upon the establishment of what McDon-
nell Douglas Corp. v. Green, 411 U.S. 792 (1973), would later
characterize as a “prima facie case of discrimination,” which
there consisted of evidence of the disproportionate impact of

7

the test upon minority applicants and a history of intentional
discrimination, Title VII requires an employer to demonstrate
that an examination has a “manifest relationship to the em-
ployment in question”—that it is “job-related.” Griggs, supra,
at 413, 436.

In sharp contrast to Griggs, the Court of Appeals in this case
has acknowledged that there is here no history of intentional
discrimination nor any evidence of present discriminatory
intent, and that the examination in question is a “second
generation” selection procedure. That is, the City clearly
sought to develop a test which meets professional standards
and eliminates any elements of unfairness or bias.

In response to this “second generation” case, the Court of
Appeals has radically extended Griggs, both with respect to
plaintiffs’ obligation to establish a prima facie case and defen-
dants’ burden of demonstrating job-relatedness. In so doing,
the Court lost sight of the fact that the ultimate issue in any
Title VII case is whether plaintiffs have proved discrimination.
The Court effectively has read into section 703(h) the addi-
tional requirement that an employer utilizing a written employ-
ment examination to select best qualified applicants, which
examination produces a “disparate impact,” must demonstrate
that its examination is the best and indeed more feasible than
any examination currently in existence. This burden far ex-
ceeds that which petitioners believe was intended by Congress
in enacting section 703(h) and in extending Title VIi to the
States and their subdivisions in 1972, or which this Court ever
contemplated in Griggs.* The far-reaching effects of this deci-
sion on the entire public employment sector, which is, by and
large, under the mandate of civil service laws, and on the

° The Second Circuit is not alone in requiring employers to demon-
strate that they have used the most perfect test. See, e.g., United States
v. City of Chicago, 573 F.2d 416 (7th Cir. 1978); Louisville Black
Police Officers v. City of Louisville, 20 FEP Cases 1195 (W.D. Ky.
1979); United States v. State of New York, 21 E.P.D. 430,314
(N.D.N.Y., Sept. 6, 1979). See, also, Lerner, Employment Discrimina-
tion: Adverse Impact, Validity, and Equality, 1979 Supreme Court
Review 17.

8

private sector, require review and clarification by this Court of
the scope and obligations of Title VII in a “disparate impact”
case.

1. In light of the ultimate burden upon plaintiffs to prove
discrimination, plaintiffs’ bare statistical analysis was
insufficient to establish a prima facie case raising the
presumption that Examination No. 8155 was discrimina-

tory.

This Court has clearly stated that the ultimate burden of
proving discrimination is on the plaintiffs. See New York City
Transit Auth. v. Beazer, 440 U.S. 568, 587, n.31 (1979). See,
also, Albemarle Paper Co. v. Moody, 422 U.S. 405, 425
(1975). The “division of intermediate evidentiary burdens bring
the litigants and the court expeditiously and fairly to this
ultimate question.” Texas Dept. of Community Affairs v.
Burdine, __. U.S. ___, 49 U.S.L.W. 4214, 4216 (March 4,
1981). The first of these intermediate burdens is the establish-
ment of a prima facie case which creates a presumption of
discrimination. See McDonnell Douglas Corp. v. Green, supra.
The evidence necessary to satisfy plaintiff’s burden must be
sufficient to eliminate the most common nondiscriminatory
reasons for the disparate results of the selection device used by
the employer. See /nternational Brotherhood of Teamsters v.
United States, 431 U.S. 324, 358 and n.44 (1977); Furnco
Construction Co. v. Waters, 438 U.S. 507, 577 (1978).

In a wide variety of contexts this Court has cautioned
against reliance on bare statistics alone to establish a prima
facie case, finding that in many instances, to recognize the
limited probative value of disproportionate impact is merely to
acknowledge ‘the heterogeneity of the nation’s population.’ ”
Village of Arlington Heights v. Metropolitan Housing Devel-
opment Corp., 429 U.S. 252, 266 n.15, quoting Jefferson v.
Hackney, 406 U.S. 535 (1972). See Washington v. Davis, 426
U.S. 229 (1976) (employment examination); Keyes v. School
Dist. No. 1, 413 U.S. 189 (1973) (school segregation). Yet it
was precisely this type of bare analysis that was accepted by the
Court of Appeals in this case.

9

Here, plaintiffs’ sole evidence to establish their prima facie
case was applicant flow data indicating that of the candidates
who chose to identify themselves racially or ethnically, white
candidates passed the written portion of Examination No. 8155
at a higher rate than minority candidates. The total ‘ack of
evidence of qualifications renders meaningless both plaintiffs’
comparison of white and minority pass rates in terms of the
“four-fifths rule,” as well as the application of the standard
deviation analysis approved by this Court in Castaneda v.
Partida, 430 U.S. 428 (1977). In Hazelwood School District v.
United States, 433 U.S. 299, 308 n.13 (1978), this Court
specifically cautioned that “[w]hen special qualifications are
required to fill particular jobs, comparison to the general
population (rather than to the smaller group of individuals
who possess the necessary qualifications) may have little pro-
bative value.” This is particularly true where, as here, the
employer has conducted an intensive recruitment effort
directed at attracting protected group members. Uniform
Guidelines on Employee Selection Procedures, sec. 4D. While
it may not be possible for plaintiffs to refute every circum-
stance that could explain the disproportionate impact shown
by the bare statistics, it is required, contrary to the ruling of
the Court of Appeals, that plaintiffs eliminate the most com-
mon nondiscriminatory factors.

Furthermore, while applicant flow data is relevant in evalu-
ating allegations of employment discrimination, standing alone
such data is insufficient to establish a prima facie case. In
Hazelwood, supra, this Court held that such data must be
evaluated in light of all other statistical and nonstatistical
evidence to determine their usefulness. Accord, Teamsters,
supra; Beazer, supra.

Particularly in the absence of a history of intentional dis-
crimination, plaintiffs cannot rely on the rigid application of
statistical analyses to overcome their burden of establishing a
prima facie case of discrimination. There is no such history in
the City of New York and this test did not result in the
exclusion of minority candidates. Under these circumstances,
and in accord with the approach adopted by this Court in

10

Beazer, supra, the evidence utterly fails to support any pre-
sumption of discrimination.

2. The decision of the Court of Appeals invalidating the use
of Examination No. 8155 for the selection of the best
qualified candidates for the position of police officer as
required by New York law far exceeds the scope of Title
VII.

Under any fair reading of Title VII it must be held that
defendants satisfied their burden of demonstrating that Exami-
nation No. 8155 has a “manifest relationship” to the job of
police officer. Griggs, supra. Yet, the Court of Appeals, al-
though finding the examination to be job-related, invalidated
rank-orders< selection of the best qualified candidates and the
cut-off score based upon projected manpower needs on the
theory that the examination could have been prepared in a
more perfect fashion. This Court has never construed Title VII
in this manner, imposing upon an employer the obligation to
prove that the selection device used is the best that could have
been devised. See New York City Transit Auth. v. Beazer,
supra; United States v. South Carolina, 445 F. Supp. 1094
(D.S.C. 1977), summarily aff’d, 434 U.S. 1026 (1978). Cf.
Washington v. Davis, 426 U.S. 229 (1976). If permitted to
stand, the decision of the Court of Appeals will effectively
preclude the City from seeking to hire the most qualified
employees for public service as required by the New York State
Constitution and Civil Service Law.

It cannot be disputed that the City’s interest in employing
the candidates best qualified for positions within the public
service is legitimate and nondiscriminatory. This public policy
is embodied in both the New York State Constitution and Civil
Service Law § 61(1). Article 5, § 6 of the New York Constitu-
tion provides in pertinent part:

“Appointments and promotions in the Civil Service of the
state and all of the civil divisions thereof, including cities
and villages, shall be made according to merit and fitness

11

to be ascertained, as far as practicable, by examination
which, as far as practicable shall be competitive. . . .”

Rank-ordered selection ensures that the most qualified can-
didates will enter public service. Rank-ordered selection is
particularly important in this regard when considering that an
eligible list may have a life of four years under Civil Service
Law § 56. It is likely that, if rank order is discarded, after a
year or more from the giving of the examination, candidates
who are better qualified and have not been reached for ap-
pointment will have found other positions and lost interest in
public employment. Rank-ordered selection also serves the
important interest of objectivity in public employment deci-
sions. And, in combination with a cut-off score based on
projected manpower needs, it reduces the public safety and
cost risks inherent in hiring unqualified or minimally qualified
employees.

The courts, including the Court of Appeals for the Second
Circuit, have traditionally given great deference to these goals
and the aims of the civil service law. See Kirkland v. New York
State Dept. of Correctional Services, 520 F.2d 420, 428 (2d Cir.
1975), cert. den., 429 U.S. 823 (1976); Jackson v. Nassau
County Civil Service Commission, 424 F. Supp. 1162, 1166
(E.D.N.Y. 1976); Commonwealth v. O’Neill, 465 F. Supp. 451,
463-65 (E.D. Pa. 1979); United States v. Commonwealth of
Virginia, 454 F. Supp. 1077, 1101 (E.D. Va. 1978). Indeed, the
concept of merit selection is embodied in the language of
section 703(h), which protects from challenge a “bona fide
seniority or merit system” unless designed to discriminate. See
International Brotherhood of Teamsters v. United States, su-
pra.

The burden imposed by the Court of Appeals, to prove that
one-point differentials in test scores will accurately predict
demonstrable differences in job performance, is, simply, insur-
mountable. It must be kept in mind that no pre-employment
test approaches such accuracy of prediction. Albemarle Paper
Co. v. Moody, 422 U.S. 405, 449 (1975) (Blackmun, J.,

12

concurring). This is not surprising since the psychological
standards for employee selection are constantly evolving as
evidenced by the series of guidelines issued by the EEOC which
are frequently contradictory. See Albemarle Paper Co., supra,
at 451-52 (Burger, C.J., concurring in part and dissenting in
part). Neither the Court nor the plaintiffs in this case were able
to point to any alternative selection device which would meet
the legitimate employment goals of the City and which would
have resulted in a lesser disparate impact. Absent such evi-
dence, the City cannot be required to abandon these goals in
favor of seeking mere minimum competency in its employees
in order to achieve proportionate minority representation.

Most recently, in the context of a “disparate treatment” case,
this Court stated its view of the substantive limitations of Title
VIL:

“The statute was not intended to ‘diminish traditional
management prerogatives.’ [Steelworkers v. Weber, 433
U.S. 193, 207 (1979)]. It does not require the employer to
restructure his employment practices to maximize the
number of minorities and women hired. Furnco Construc-
tion Co. v. Waters, 438 U.S. [567, 577-78 (1978)]. ...

Rather, the employer has the discretion to choose
among equally qualified candidates, provided the decision
is not based upon unlawful criteria. The fact that a cour
may think that an employer misjudged the qualifications
of the applicants does not in itself expose him to Title VII
liability... .”

Texas Dept. of Community Affairs v. Burdine, supra, ——
U.S. ___, 49 U.S.L.W., at 4217. Nothing in section 703(h) of
Title VII suggests that Congress, in authorizing the use of
professionally developed tests, intended any greater restrictions
upon the employer.

Certainly, the Congress which extended Title VII to the
States and their subdivisions in 1972 under the authority of the
Fourteenth Amendment, mindful of the principles of federal-

13

ism found in the Tenth Amendment, could not have intended
the restriction upon the City’s ability to govern itself in an
effective manner which results from the decision of the Court
of Appeals in this case. See, generally, National League of
Cities v. Usery, 426 U.S. 833 (1976); Scott v. City of Anniston,
597 F.2d 897 (5th Cir. 1979); United States v. City of Chicago,
573 F.2d 416 (7th Cir. 1978); Firefighters Institute for Racial
Equality v. City of St. Louis, 549 F.2d 506 (8th Cir.), cert. den.
sub nom. United States v. Banta, 434 U.S. 819 (1977); Note,
Title VII and Public Employees: Did Congress Exceed Its
Powers, 78 Col. L. Rev. 372 (1978). Although this constitu-
tional tension was not an articulated ground for this Court’s
holdings in Beazer, supra, and United States v. South Carolina,
supra, we submit that it affords some explanation for the
analytic approach there adopted.

The decision of the Court of Appeals in this case places an
enormous, and, indeed, impossible burden on both public and
private employers, unrelated to the goal of Congress in enact-
ing Title VII to ensure minority group members equal access to
employment opportunities. Petitioners urge this Court to clar-
ify the obligations imposed upon employers by Title VII in the
context of this “disparate impact” case.

14

CONCLUSION
For these reasons, a writ of certiorari should be granted.
April 15, 1981.
Respectfully submitted,

ALLEN G. SCHWARTZ,

Corporation Counsel of
the City of New York,

Attorney for Petitioners.

LEONARD KOERNER,
JUDITH A. LEVITT,

of Counsel.

la

Appendix A

Opinion of the Second Circuit

UNITED STATES COURT OF APPEALS
FOR THE SECOND CIRCUIT

aos

No. 849—September Term, 1979
(Argued February 6, 1980 Decided July 31, 1980)
Docket No. 80-7027

+

THE GUARDIANS ASSOCIATION OF THE NEW YORK CITY
POLICE DEPARTMENT, INC., THE HISPANIC SOCIETY OF
THE NEW YORK CITY POLICE DEPARTMENT, INC., NYDIA
I. DIAZ, JAMES MICHAEL HIDALGO, WILFRED CEBEL-
LERO, ANDRE LOPEZ, REINALDO SALGADO, DENISE SAN-
TOS, DEBORAH HOLMES and PAMELA OBEY, individually
and on behalf of all those similarly situated,

Plaintiffs-A ppellees,
—_—vV —

CIVIL SERVICE COMMISSION OF THE CITY OF NEW YORK,
DEPARTMENT OF PERSONNEL OF THE CITY OF NEW YORK,
and THE NEW YORK CITY POLICE DEPARTMENT,

Defendants-Appellants.
—$+>—

Before:

MANSFIELD and NEWMAN, Circuit Judges,
and SIFTON,* District Judge.

i

° Of the United States District Court for the Eastern District of New
York, sitting by designation.

2a

Appeal from a decision by the United States District Court
for the Southern District of New York (Robert L. Carter,
Judge), 484 F. Supp. 785, holding that New York City violated
Title VII of the Civil Rights Act of 1964, as amended, 42
U.S.C. § 2000e-2, in administering an examination to appli-
cants for positions in the New York City Police Department.

Affirmed in part, vacated in part, and remanded.

i

PETER BIENSTOCK, New York, N.Y. (M.D. Tara-
cido, Kenneth Kimmerling, Robert L.
Becker, Puerto Rican Legal Defense & Edu-
cation Fund, New York, N.Y., on the brief),
for plaintiffs-appellees.

L. KEVIN SHERIDAN, New York, N.Y. (Allen G.
Schwartz, Corporation Counsel, Judith A.
Levitt, Steven M. Goldberg, Maureen M.
McCabe, New York, N.Y., on the brief), for
defendants-appellants.

DAVID L. ROSE, Washington, D.C. (Robert B.
Fiske, Jr., U.S. Atty., Nancy E. Friedman,
Richard N. Papper, Dennison Young, Jr.,
Asst. U.S. Attys., New York, N.Y., Drew S.
Days III, Asst. Atty. Gen., Steven H. Rosen-
baum, Washington, D.C., on the brief), for
the United States as amicus curiae.

(Ira H. Leibowitz, Barry Lasky, Garden City,
N.Y., submitted a brief for The Policewom-
en’s Endowment Assoc., Inc., as amicus cu-
riae.)

(H. Elliot Wales, New York, N.Y., submitted a
brief for Seven Civil Service Organizations as
amicus curiae.)

+

3a

NEWMAN, Circuit Judge:

This employment discrimination suit pursuant to Title VII of
the Civil Rights Act of 1964, 42 U.S.C. § 2000e-2, once again
requires this Court to venture into the complex realm of testing
and test validation. The test at issue was designed by New York
City officials and administered on June 30, 1979 to 36,797
applicants for positions on the City’s police force. Plaintiffs
are the Guardians Association of the New York City Police
Department, Inc., an organization of Black police officers, the
Hispanic Society of the New York City Police Department,
Inc., an organization of Hispanic police officers, and eight
individual Black or Hispanic applicants. Defendants are the
New York City Department of Personnel, which performed
much of the test preparation, the New York City Civil Service
Commission, and the New York City Police Department. The
United States District Court for the Southern District of New
York (Robert L. Carter, Judge) found that use of the test
unjustifiably discriminates against Blacks and Hispanics in
violation of Title VII. Guardians Ass’n v. Civil Service Com-
mission, 484 F. Supp. 785 (S.D.N.Y. 1980). The Court ordered
a broad remedy, including a 50% minority hiring quota. We
affirm the District Court’s finding that the City’s specific use
of the test violates Title VII, but vacate the remedy and
remand for entry of a revised decree.

I. Factual Background

The test in question, designated Exam No, 8155, was de-
signed to select candidates for hiring as entry-level police
officers. Those who pass the exam are selected, in rank order
of their test scores, to complete the other aspects of the hiring
process—a medical examination, a physical agility test, a
psychological test, and a character investigation. These last
four components of the hiring process are scored only on a
pass/fail basis. Thus, an appellant’s score on Exam No. 8155 is
a major determinant of his prospects for becoming a police
officer. It is also the only feature of the process alleged to have

4a

a discriminatory impact. Once an applicant scores high enough
to be selected for the final four hiring steps and successfully
completes those steps, he or she becomes a sworn police officer
and enters the police academy for five months of training.
While successful completion of the training program is a
requirement of continuing as a police officer, the Department
does not use the training program as a selection device, but
anticipates that nearly all academy entrants will go on to active
duty.

The exam was developed by a fairly elaborate two-stage
process; the first stage was an analysis of the police officer's
job, and the second was construction of the test itself. The job
analysis consisted of five separate steps. First, the Department
of Personnel identified 71 tasks that police officers generally
perform, based on interviews with 49 police officers and 49
supervisors. Second, a panel of seven officers and supervisors
reviewed the list to add any tasks that had been omitted, and to
eliminate those items that were duplicative, or too specialized
to be performed by entry-level officers. The result was a
consolidated list of 42 entry-level tasks.

Third, a questionnaire was distributed to 5,600 police offi-
cers, requesting them to rate each of the 42 tasks on the basis
of its frequency of occurrence, its importance, and the amount
of time normally spent in performing it. The 2,600 responses
that were received were then analyzed by computer to yield a
ranking of the 42 tasks, according to the combined rating of all
the responses. In addition, faculty members of John Jay
College were asked to observe police officers during an entire
tour of duty and record the tasks that they performed; their
survey generally confirmed the identification of the 42 tasks.

In the fourth step of the job analysis, the Department of
Personnel divided the list of 42 ranked tasks into clusters of
related activities. Five such clusters were established: the arrest
process, providing assistance to people, police operations,
stationhouse activities, and handling unusual and other occur-
rences. The fifth step was an analysis of all five clusters, each
one by a separate panel of police officers, to identify the
“knowledge, skills and abilities” required to perform these

Sa

tasks at the entry level, and to assign percentages reflecting the
relative importance of each of the identified knowledges, skills,
and abilities for the cluster as a whole. One panel listed five
such qualities for its cluster, all of which are properly charac-
terized as “abilities” or “skills” (hereafter referred to as “abili-
ties”): recalling facts, filling out forms, understanding and
applying statutory definitions of crimes, understanding written
instructions and applying appropriate procedures, and human
relations skills, including communication techniques. Each of
the other four panels used the first panel’s list of abilities, but
developed its own percentages to express the relative impor.
tance of each ability to the tasks within its cluster.

The second major stage in developing Exam No, 8155, the
process of test construction, consisted of four identifiable
steps. First, the percentages of the five abilities necessary to
perform each of the five task clusters were multiplied by the
weightings that had been given to each task in Step 3 of the job
analysis on the basis of frequency, importance, and time spent.
This yielded a general measurement for the importance of each
of the five abilities for performance of the job of police officer.
As a result of this computation, the Department of Personnel
concluded that on a test with 100 questions, 15 questions
should test for the ability to recall facts, 9 questions for filling
out forms, 14 questions for understanding and applying sec-
tions of the criminal law, 32 questions for understanding
written instructions and applying appropriate procedures, and
30 questions for human relations skills. Next, a group of eleven
police officers was selected to write multiple-choice questions
that tested for the five abilities, as they related to the 42
identified tasks. The officers wrote many of these questions
from Police Academy materials and similar sources, however,
without having access to descriptions of the five identified
abilities, or the 42 ranked tasks. In the third step, Department
of Personnel staff members who did have access to the descrip-
tion of abilities and the ranking of tasks reviewed the questions
written by the police officers to assure that the questions were
not ambiguous, overly complex, overly specialized, or depen-
dent on prior knowledge. As a result of this review, some

vw

6a

questions were discarded, others were revised, and still others
were added. Finally, the resulting questions were subjected to a
further review by a panel of six police experts, and by various
members of the Department of Personnel.

The test that resulted consisted of 100 multiple-choice ques-
tions, designed so that the candidate could answer correctly
without knowledge of any information beyond what was pro-
vided on the test itself. The test materials were determined by
the Department of Personnel to require an eighth-grade read-
ing level, on the average, although the 14 questions on law
required college-level reading ability. The estimated time for
completing the exam was 1% hours, but 3% hours were
allowed.

The first part of the exam, designed to measure the ability to
recall facts, consisted of a page-and-a-half description of a
burglary, and a series of 15 questions to be answered without
referring back to the description. In the second part, testing
ability to fill out forms, the candidates were given a simplified
arrest form, and a page-long description of both a robbery and
an arrested suspect, and then asked 9 questions about the
proper entries to be made in filling out the form. Part three,
intended to test ability to apply provisions of law, consisted of
14 questions, each briefly presenting the facts of an incident,
and then requiring the candidate to identify the precise
criminal offense involved on the basis of definitions provided
in the test materials.' The remaining 62 questions, of which 32

| One example:

37. Chris Hart and Larry Burns are walking by a big museum
late one night. Hart notices that, although the museum is closed to
the public, one of the doors is unlocked. Hart suggests that for a
prank they go into the museum to see the exhibits. Hart has a
flashlight with him while Burns has an illegal gun hidden under his
clothing. Hart does not know that Burns has a gun illegally in his
possession, About five minutes after they enter the museum, they
hear footsteps and leave the museum the same way they entered.
According to the definitions given,

(A) Burns committed the crime of criminal trespass, but Hart did
not

(B) Hart committed the crime of criminal trespass, but Burns did
not

7a

were intended to measure the ability to follow appropriate
procedures and 30 were intended to measure human relations
skills, consisted of general instructions as to procedures or
appropriate responses for certain types of situations, a descrip-
tion of a specific situation, and then one or more questions
asking the proper response to the situation presented. Three of
the questions dealing with appropriate procedures, for exam-
ple, involved the proper response to a bomb threat. Four of the
questions in the human relations section involved the proper
way to deal with a person who appears to be mentally ill.

The test was scored from zero to one hundred, with one
point given for each correct answer, and bonus points given for
veterans.* The candidates were then rank-ordered on the basis
of their scores. Scores were generally high, with 13% of the
applicants scoring 98 or above, and fully 50% scoring 91 or
above. Because of the number of candidates taking the test and
the bunching of candidates in the upper range of scores, each
point a candidate achieved made a substantial difference in his

(C) both Hart and Burns committed the crime of criminal
trespass

(D) neither Hart nor Burns committed the crime of criminal
trespass.

The definition provided for crimina! trespass is as follows:

The crime of criminal trespass is committed when a person know-
ingly enters or remains in a building in which he has no right to be
and while in the building, possesses, or knows that another person
accompanying him possesses, an explosive or a gun.

It will be noted, simply as an indication of the difficulty of the test
construction enterprise, that there is a slight ambiguity in this question.
The applicant is told that Hart is unaware that “Burns has a gun
illegally in his possession,” but not whether Hart was unaware that
Burns had a gun at all. This could affect the answer, since, according
to the definition given, knowledge of any gun, even of a legal one,
would render Hart’s action a criminal trespass. While it would be
unreasonable to suggest that a test must be free of every possible
ambiguity in order to be acceptable, the ease with which such ambi-
guities can appear emphasizes the value of confirming the test’s
reliability by some empirical procedure. See Section IV infra.

2 As a result of these bonus points, it was possible to achieve a score
as high as 110. Some 482 applicants, or 1.3% of those taking the test,
scored above 100.

8a

position on the rank-ordering list. More than 2,000 applicants
achieved a score at each numerical grade from 92 to 97.

The passing grade was determined in the following manner.
The Police Department first estimated that 4,000 police offi-
cers would be hired during the four-year period for which the
eligibility list resulting from Exam No. 8155 would be valid.
The Department further estimated that only one out of three
applicants who passed Exam No. 8155 would successfully
complete all the remaining steps in the hiring process. There-
fore, if this eligibility list was to meet the Department’s needs,
12,000 persons had to pass the exam to provide the 4,000
needed police officers. With all this in mind, the Department
simply set the passing grade at the score achieved by the
12,000th highest scoring candidate, which turned out to be 94.
Because of the bunching phenomenon, a large number of
candidates, 2,124, achieved this same score, so that the actual
number who received a passing grade was 13,749.

Of the 36,797 applicants who took the test, 6,142 identified
themselves as Black, 5,239 identified themselves as Hispanic,
19,798 identified themselves as white, and 4,847, or 13.2% did
not identify their race. Thus, identified Blacks constituted
16.7% of the total applicants, and 19.7% of all those who
identified their race, while the equivalent percentages for
identified Hispanics were 14.2% and 16.8%, and for identified
Whites, 53.8% and 64.5%.

Of those who passed the exam, i.e., scored 94 or better,
7.6% had identified themselves as Black and 7.8% had iden-
tifed themselves as Hispanic, for a known minority population
in the passing group of 15.4%, against a known minority
population in the applicant pool of 30.9%. In contrast, 66.6%
of the passing applicants had identified themselves as White,
although Whites comprised only 53.8% of the applicant pool.

Viewed in another and more revealing way, the figures show
that, among those who had identifed themselves by race, the
passing rate for Whites was 45.9% compared to 17% for
Blacks and 20.5% for Hispanics. The combined minority pass
rate was thus about two-fifths of the pass rate for Whites.

9a

The Police Department accepted 415 candidates from the list
in November, 1979 and planned to hire another 380 in January,
1980. Of this group of 795 candidates, 89.2% were White,
3.5% were Black, and 6.8% were Hispanic. The selection rates
(number chosen compared to number of applicants) for these
first two uses of the list were 0.5% for Blacks, 1% for
Hispanics, and 3.6% for Whites.

Il. The District Court’s Prior Proceedings and Decision

The plaintiffs filed their complaint in this suit, together with
a motion for a preliminary injunction, in October, 1979, before
any candidates had been accepted on the basis of the list. They
charged that the intended use of the lists by the Police Depart-
ment constituted discrimination against Blacks and Hispanics
in violation of the Fourteenth Amendment, Title VII, and
various other Federal and state laws. The District Court, by
consent of the parties, consolidated the hearing on the prelimi-
nary injunction and the trial on the merits. This proceeding
was held on November 13, 14, and 15, 1979, shortly after the
Police Department’s first use of the list. On January 11, 1980,
three days before the Department intended to use the list to
accept a second group of trainees, the District Court held a
second hearing. That same day, the Court issued an opinion,
which was subsequently re-issued in revised form on January
aaa

3 By the time of the November hearing, the Police Department had
already proceeded to use the list to accept 415 trainees, as described
above. In order to forestall any further hiring on the basis of the list,
and to assist the City in making alternative arrangements, the District
Court informed the parties on December 17, 1979 that the test violated
Title VII, and orally enjoined the defendants from its further use. On
December 27, 1979, the City filed an order to show cause and a motion
for a stay, in which it stated that the Police Department needed to
accept an additional 380 trainees from the list on January 14, 1980.
When the Court denied its motion, the City filed a petition in this
Court for a writ of mandamus. This Court granted the petition,
ordering the District Court to issue findings of fact and conclusions of
law, pursuant to Fed. R. Civ. P. 52(a), at least 48 hours before any
injunction against the City was to take effect. The District Court then

10a

The Court’s basic conclusion was that Exam No. 8155
violated Title VII. In reaching this conclusion, the Court used
the common mode of Title VII analysis, in which the plaintiff
is first required to establish a prima facie case on the basis of
disparate impact, and then the defendant is required to rebut
the plaintiff’s case by proving that the disparity results from
legitimate, job-related selection procedures. The Court first
found that the disparity between the percentage of minority
group members who achieved a passing score and the percent-
age of minority groups members in the applicant pool was
sufficient to establish a prima facie case. It based this finding
of disparate impact on the standards developed by the Su-
preme Court in Castaneda v. Partida, 430 U.S. 482 (1977),
and by the Equal Employment Opportunity Commission
(EEOC) in its Uniform Guidelines on Employee Selection
Procedures, 29 C.F.R. § 1607 (1979). Castaneda stated that, in
cases involving large samples, “if the difference between the
expected value [from a random selection] and the observed
number is greater than two or three standard deviations,” a
prima facie case is established. 430 U.S. at 496 n.17.* The
Uniform Guidelines provide that “[a] selection rate for any

held its second hearing, which was devoted to the issue of relief. Since
the Court’s decision, issued that day, preceded the January 14 action
by more than 48 hours, it fulfills the requirements of this Court’s
mandamus.

4 The standard deviation for a particular set of data provides a
measure of how much the particular results of that data differ from the
expected results. In essence, the standard deviation is a measure of the
average variance of the sample, that is, the amount by which each item
differs from the mean. The number of standard deviations by which
the actual results differ from the expected results can be compared to
the normal distribution curve, yielding the likelihood that this dif-
ference would have been the result of chance. The likelihood that the
actual results will fall more than one standard deviation beyond the
expected results is about 32%. For more than two standard deviations,
it is about 4.6% and for more than three standard deviations, it is
about .03%. On this basis, the Supreme Court concluded in Casteneda
that when actual results fell more than three standard deviations from
the expected result (that is, a race-neutral selection), the deviation
could be regarded as caused by some factor other than chance.

lla

race, sex, or ethnic group which is less than four-fifths (*/.) (or
eighty percent) of the rate for the group with the highest rate
will generally be regarded by the Federal enforcement agencies
as evidence of adverse impact.” § 1607.4(D) [hereinafter
Guideline sections are cited only by the subdivisions of 29
C.ER. § 1607]. The District Court then noted that the discrep-
ancy between the percentage of minority group members in the
applicant pool and the percentage of minority groups members
who passed the test was 39 standard deviations. The evidence
also showed that the passing rate of the minority group
members was 44.3% of the passing rate of Whites, or about
two-fifths.

Having concluded that the plaintiffs had established a prima
facie case, the Court next concluded that the test was not
sufficiently valid to constitute a legitimate attempt to choose
those applicants who would become better police officers.
Relying primarily on the EEOC Guidelines, the Court stated
that a determination of validity based on the content of the test
would be inappropriate, first because the test purported to
measure abilities that the accepted applicants would be trained
to acquire, see Guidelines, § 14(C)(1), and second, because the
test actually measured constructs, not abilities, see id.* More-
over, the Court concluded that the job analysis was not
sufficiently precise to satisfy the Guideline requirement even
for content validation, see Guidelines § 14(C)(2). Since content
validation was the only method of validation which the City
attempted, the Court concluded that the test was invalid, and
thus an inadequate rebuttal to the plaintiffs’ prima facie case.

In fashioning relief, the District Court noted that a previous
examination administered by the New York City Police Depart-

5 A construct is generally defined as “an idea developed or ‘con-
structed’ as a work of informed, scientific imagination; that is, it is a
theoretical idea developed to explain and to organize some aspects of
existing knowledge.” American Psychological Association, Inc., Stan-
dards for Educational & Psychological Tests 29 (1974) [hereinafter
APA Standards]. Neither the APA Standards nor the Guidelines
appear to include a definition of content, apart from the concept of
content validity, although the Guidelines do describe content as involv-
ing “knowledges, skills, or abilities.” Guidelines § 14(C)(1).

12a

ment had been found to be in violation of Title VII. Guardians
Association v. Civil Service Commission, 431 F. Supp. 526
(S.D.N.Y.), vacated and remanded on other grounds, 562 F.2d
38 (2d Cir. 1977). Concluding that the defendants had “per-
sisted in devising and utilizing testing procedures that continue
to discriminate against blacks and hispanics,” the District
Court found that the defendants’ “studied adherence to dis-
criminatory procedures at this point must be deemed conscious
and deliberate.” 484 F. Supp. at 798. On this basis, the Court
held that “[a]ffirmative action is mandated as an interim
measure either until such discrimination has been totally elimi-
nated or until defendants proceed to select police officers
under procedures that are in full compliance with Title VII.”
Id. at 27. It enjoined the City from using Exam No. 8155,
although permitting it to use the eligibility list from that exam
for purposes designated by the Court. In its order, also issued
on January 11, the Court ordered the Police Department to
achieve at least 30% minority composition of the force, a level
comparable to the percentage of minorities in the labor force
of the relevant hiring area. To achieve this goal, the Court
further ordered that the defendants should “as an interim goal
appoint 50% of their entry level police officers from among
qualified black and hispanic applicants.” Finally, the Court
awarded the plaintiffs attorneys’ fees and costs, and retained
jurisdiction “for such further relief or other orders as may be
necessary or appropriate to enforce and insure rights to equal
employment opportunity within the New York City Police
Department.”

The City moved to stay the District Court’s order pending
consideration of its appeal from the decision. This Court
denied the motion. However, it granted a conditional stay, in
view of the City’s declared need to hire new policy officers®
and set an expedited schedule for the appeal.

6 The terms of this conditional stay were that, if the City wished to
hire, pending appeal, it should establish two pools of candidates, one
consisting of all the minority applicants who passed the exam, and the
second consisting of all others who passed, and select trainees from
these pools in the ratio of one minority applicant for every two others.
The 415 applicants already hired were to be counted in determining
whether the new hires conformed to this | to 2 ratio.

13a
III. The Framework of Title VII Analysis

As the District Court concluded, the accepted procedure for
Title VII cases is to require the plaintiffs to establish a prima
facie case, and then to require the defendants to rebut this
showing with proof that the test was legitimately job-related.
See Albemarle Paper Co. v. Moody, 422 U.S. 405 (1975);
McDonnell Douglas Corp. v. Green, 411 U.S. 792 (1973);
Griggs v. Duke Power Co., 401 U.S. 424 (1971). The Court
correctly concluded that a prima facie case had been es-
tablished. By any reasonable measure, including the standard
deviation rule of Castaneda, supra, or the four-fifths rule of
the EEOC Guidelines, Exam No. 8155 had a disparate racial
impact.

The City argues that statistics alone, specifically a compari-
son of the racial composition of the passing group to that of
the applicant group, are not sufficient to establish a prima
JSacie case. But statistics showing a significantly disparate racial
impact have consistently been held to create a presumption of
Title VII discrimination. See /nternational Brotherhood of
Teamsters v. United States, 431 U.S. 324, 339 (1977); United
States v. City of Chicago, 549 F.2d 415, 428 (7th Cir.), cert.
denied, 434 U.S. 875 (1977); Kirkland v. New York State
Department of Corrections, 520 F.2d 420, 425 (2d Cir. 1975),
cert. denied, 429 U.S. 823 (1976); Vulcan Society v. Civil
Service Commission, 490 F.2d 387, 392-93 (2d Cir. 1973);
Chance v. Board of Examiners, 458 F.2d 1167, 1172-73 (2d Cir.
1972).’

7 The City hypothesizes some situations in which statistics could be
misleading (e.g., if some of the candidates taking the test had not been
eligible to apply for the position) but presents no evidence to show that
this occurred. To accept such unsupported possibilities, and require the
plaintiffs to refute every circumstance that could explain the disparate
impact shown by the statistics, would create an onerous burden of
proof, far in excess of the Title VII standards as interpreted by the
Supreme Court. See Dothard v. Rawlinson, 433 U.S. 321, 329-30
(1977); Jones v. New York City Human Resources Administration, 528
F.2d 696, 698 (2d Cir. 1976).

l4a

The City also claims that finding a prime facie Title VII
violation by state or local governments without a showing of
discriminatory intent violates the Tenth Amendment. This view
has been definitively rejected by the Seventh Circuit in United
States v. City of Chicago, 573 F.2d 416, 422-24 (7th Cir. 1978),
and we agree with that analysis. Congress may enforce the
Fourteenth Amendment by legislation that prohibits practices
the Amendment might not of its own force condemn. See
Katzenbach v. Morgan, 384 U.S. 641 (1966).

The real issue in this case, therefore, is whether the defen-
dants have rebutted the plaintiffs’ prima facie case by proving
that its test was job-related: that the test accurately selected
applicants who would be better police officers. Adjudication
of this issue presents a more complex problem in the present
case than it has in many previous Title VII suits. Many of the
previous suits involved tests that were so artlessly constructed
that they could be judged invalid without extensive inquiry,
fine distinctions, or a precise notion of where the line between
validity and invalidity was located. See, e.g., Griggs, supra,
401 U.S. at 431 (intelligence tests used “on the Company’s
judgment that they generally would improve the overall quality
of the work force”); United States v. N.L. Industries, Inc., 479
F.2d 354, 371 (8th Cir. 1973) (test given to one applicant
“consisted of four or five mathematical problems which a
Company employee jotted down on a sheet of yellow paper”);
Brito v. Zia Co., 478 F.2d 1200, 1205-06 (10th Cir. 1973) (test
based almost entirely on subjective judgments of supervisors,
not administered or scored under controlled and standardized
conditions); Vulcan Society, supra, 490 F.2d at 396-98 (no job
analysis, test measured abilities that were clearly of secondary
importance to job).

Exam No. 8155, in contrast, is a “second generation”
selection procedure. Despite the various flaws in construction
of the test, it is clear that some attempt was made to develop
the test with recognition of at least some of the standards that
courts have established in the first wave of Title VII cases.
Aware that the validity of the test would likely have to be
demonstrated, the City performed an extensive job analysis,

EP

lSa

consciously used Guideline concepts in determining the quali-
ties that were being tested for, and attempted to eliminate
extraneous variables, such as the applicant’s prior knowledge,
his reading level, and his ability to complete the test in a
relatively short amount of time.

Nevertheless the plaintiffs have alleged and the District
Court has concluded that the construction and use of Exam
No. 8155 failed in several respects to meet test validity stan-
dards, particularly those specified in the Guidelines. Whether
or not these deficiencies are fatal, they are plainly more
substantial than the defects deemed not to defeat validity in
prior cases. Detroit Police Officers Association v. Young, 446
F. Supp. 979, 990-91, 1007-08 (E.D. Mich. 1978); Bridgeport
Guardians v. Bridgeport Police Department, 431 F. Supp. 931
(D. Conn. 1977); cf. Washington v. Davis, 426 U.S. 229,
248-52 (1976) (Fourteenth Amendment case involving some
Title VII concepts). Consequently, assessment of Exam No.
8155 necessarily carries this Court into difficult areas of
judging test validity. We must determine, with some care, what
the general standards are for judging validity, and how these
standards are to be applied in a specific factual situation.

The study of employment testing, although it has necessarily
been adopted by the law as a result of Title VII and related
statutes, is not primarily a legal subject. It is part of the
general field of educational and industrial pychology, and
possesses its owe methodology, its own body of research, its
Own experts, and its own terminology. The translation of a
technical study such as this into a set of legal principles
requires a clear awareness of the limits of both testing and law.
It would be entirely inappropriate for the law to ignore what
has been learned about employment testing in assessing the
validity of these tests. At the same time, the science of testing
is not as precise as physics or chemistry, nor its conclusions as
provable. While courts should draw upon the findings of
experts in the field of testing, they should not hesitate to
subject these findings to both the scrutiny of reason and the
guidance of Congressional intent.

l6a

The need to modify rigid technical conclusions from the
field of testing is indicated by the view of certain testing
experts, including those who testified for the plaintiffs in this
case, that there is no test that can be considered completely
valid to select candidates for any but the most rudimentary
tasks. If this view guided interpretation of Title VII, then at
the current stage of the technology of testing, no test that
produces a disparate racial impact could be used for positions
such as police officers.

While this position is a conceivable one, it is supported
neither by the statutory language nor by judicial precedent.
Had Congress felt that testing for virtually all employment was
invalid, it would not have made a specific exception to Title
VII for the proper use of professionally designed employment
tests.* Clearly, Congress did not intend that the standard used
in interpreting Title VII would reject every test with a disparate
racial impact. Nor have the courts permitted this to occur;
although they have subjected employment testing to the careful
scrutiny required by the statute, they have found a variety of
tests to be valid, despite a disparate racial impact. See, e.g.,
Sims v. Sheet Metal Workers, Local 65, 489 F.2d 1023, 1025-26
(6th Cir. 1973); Detroit Police Officers Association v. Young,
supra, 446 F. Supp. at 1007-08; Friend v. Leidenger, 446 F.
Supp. 361 (E.D. Va. 1977), aff’d, 588 F.2d 61 (4th Cir. 1978);
United States v. South Carolina, 445 F. Supp. 1094 (D.S.C.
1977), aff’d, 434 U.S. 1026 (1978); Bridgeport Guardians v.
Bridgeport Police Department, supra, 431 F. Supp. at 936-39;
Jackson v. Nassau County Civil Service Commission, 424 F.
Supp. 1162 (E.D.N.Y. 1976); Buckner v. Goodyear Tire &

8 Title VII states, in relevant part:

Notwithstanding any other provision of this subchapter, it shall
not be an unlawful employment practice . . . for an employer to
give and to act upon the results of any professionally developed
ability test provided that such test, its administration or action upon
the results is not designed, intended or used to discriminate because
of race, color, religion, sex or national origin.

42 U.S.C. § 2000e-2(h). Cf. International Brotherhood of Teamsters,
supra, 431 U.S. at 348-56 (interpreting Title VII’s seniority exception,
§ 2000e-2(h), to give operative effect to exception’s full scope).

17a

Rubber Co., 339 F. Supp. 1108, 1113-16 (N.D. Ala, 1972),
aff'd, 476 F.2d 1287 (Sth Cir. 1973).

The danger of too rigid an application of technical testing
principles is that tests for all but the most mundane tasks
would lack sufficient validity to permit their use. At least that
is the risk given the current state of the art of employment
testing. This risk can be appreciated by considering the one
example even most test critics acknowledge to have substantial,
though not complete, validity. This is a typing test given to a
group of applicants for jobs as typists.’ Such a test substan-
tially meets all the criteria suggested by plaintiffs’ experts for
content validation, but the very success of this test casts doubt
on the usefulness of the example. To begin with, typing is a
task that readily yields to quantitative measurement. The
quality of a typist’s job performance depends on two factors,
both of which can be captured with precision in numbers: how
fast he types, and how many errors he commits. Most jobs
involve tasks whose performance can be evaluated only in the
more subjective light of judgment. Surely this is true of nearly
all the tasks required to be performed by police officers. In
addition, there is a more basic problem with the typing test
example. Typing is one of the few activities that a test-taker
can perform in virtually the same manner as he will be required
to perform on the job. That is obviously an ideal testing
situation, but it is not one that is frequently available, and such
“on-the-job” testing could not possibly be done to select police
officers. Yet the force of the typing test example easily leads to
one of the conclusions of the District Court in this case: that
Exam No. 8155 lacked validity because it measured perform-
ance in an artificial classroom setting and did not necessarily
indicate who would perform well on the job.'®

9 See EEOC, Uniform Employee Selection Guidelines: Interpretation
and Clarification (Questions and Answers) Q. 62 1979 [hereinafter
cited as EEOC Questions and Answers]; APA Standards, supra at 29;
accord, Vulcan Society, supra, 490 F.2d at 395.

10 In criticizing the questions involving application of the law, the
District Court stated: “In a real situation the officer sees activity and
must determine rather quickly whether the activity is illegal, with no

18a

Closely related to the question of the proper weight to be
given to technical conclusions of testing theory is the question
of the proper weight to be given to the EEOC Uniform
Guidelines, which are largely based on these technical conclu-
sions. See Guidelines § 5(C). The District Court drew its
methodology from the Guidelines, concluding that the City’s
test was invalid because it failed to satisfy all of the Guidelines.
The Supreme Court has relied upon some of the Guidelines in
several of the leading cases, see Albemarle, supra, 422 U.S. at
431; Espinoza v. Farah Manufacturing Co., 414 U.S. 86, 94
(1973); Griggs, supra, 401 U.S. at 433-34, but the Court has
not ruled that every deviation from any of the Guidelines
automatically results in a violation of Title VII. The Court
appears to have applied the Guidelines only to the extent that
they are useful, in the particular setting of the case under
consideration, for advancing the basic purposes of Title VII.
See Espinoza, supra, 414 U.S. at 94; Guardians Association v.,
Civil Service Commission, 490 F.2d 400, 403 n.1 (2d Cir, 1973),
United States v. Georgia Power Co., 474 F.2d 906, 913 (Sth
Cir, 1973). To the extent that the Guidelines reflect expert, but
non-judicial opinion, they must be applied by courts with the
same combination of deference and wariness that characterizes
the proper use of expert opinion in general. See A/bemarle,
supra, 422 U.S. at 449 (Blackmun, J., concurring) (Guidelines
“have never been subjected to the test of adversary comment.
Nor are the theories on which the Guidelines are based beyond
dispute.”) Thus, the Guidelines should always be considered,
but they should not be regarded as conclusive unless reason
and statutory interpretation support their conclusions. As this
Court has previously stated: “If the EEOC’s interpretations go

definitional aids before him, He must operate on instinct and experi-
ence.” 484 F, Supp. at 797, That is true, but it is a criticism of all
testing. In any situation, there are generally at least two steps that are
necessary to produce the correct behavioral response, The first is to
know what to do, and the second is to act accordingly, Clearly the
limits of any test, no matter how well designed, is to determine whether
the applicant knows or can determine what to do, Only a probationary
period can determine if the applicant will act correctly in a real life
situation,

19a

beyond congressional intent, the Guidelines must give way.”
Guardians Association, supra, 490 F.2d at 403 n.1.

In addition to their force as the expression of expert opinion,
the Guidelines also possess legal force. But here too, it is
necessary to keep their limits in mind. The primary purpose of
the Guidelines is to indicate the standards that various Federal
agencies, such as the EEOC, the Civil Service Commission,
and the Department of Justice are to use in enforcing Title VII
and related statutes, See Guidelines § 2(A). But the fact that an
agency or group of agencies has announced the standards they
will use does not convert those standards into mandatory legal
rules,

A second legal basis for following the Guidelines is that they
represent the “administrative interpretation of the Act by the
enforcing agency,” and are “entitled to great deference” on
that basis. Griggs, supra, 401 U.S. at 433-34; see Albemarle,
supra, 422 U.S. at 431. However, the Court has also recognized
that the Guidelines “are not administrative ‘regulations’ pro-
mulgated pursuant to formal procedures established by Con-
gress.” /bid, They are entitled to deference, not obedience, See
Espinoza, supra, 414 U.S, at 94 (1973) (Guideline rule on
discrimination against non-citizen “is no doubt entitled to
great deference . . ., but that deference must have limits
where, as here, application of the guideline would be inconsis-
tent with an obvious congressional intent”). Moreover, the
Court in Griggs was following the Guidelines only to make the
straightforward distinction between general intelligence tests
and job-related tests; it is not at all clear that Griggs requires
observance of all the intricate details of the Guidelines. It
might be desirable for all employers to follow the more careful
practices required of the Federal Government, but there is no
reason to think that Congress intended to impose such prac-
tices, in their full rigor, when it enacted Title VII.

With these considerations in mind, we turn to the validity of
Exam No, 8155.

20a

IV. The Validity of Exam No, 8155

A. Selecting the Validation Technique

The threshold task in determining the validity of a chal-
lenged examination is to select the appropriate method for
assessing its job-relatedness. The Guidelines describe three
techniques: content validation, construct validation, and crite-
rion-related validation. Guidelines §§ 5(B), 14. The Guidelines
specify when each technique is appropriate and also specify the
requirements for successfully validating an exam by use of
each technique, Defendants have attempted to justify Exam
No. 8155 by content validation, a technique appropriate for
tests that measure “knowledges, skills or abilities” representa-
tive of the “content” of the job. Guidelines § 14(C)(1). Plain-
tiffs contend that construct validation must be used to assess
this exam because, in their view, the exam attempts to measure
“constructs,” that is, inferences about mental processes or
traits, such as “intelligence, aptitude, personality, common-
sense, judgment, leadership and spatial ability.” /bid.

This content-construct distinction has a significance beyond
just selecting the proper technique for validating the exam; it
frequently determines who wins the lawsuit. Content Validation
is generally feasible while construct validation is frequently
impossible. Even the Guidelines acknowledge that construct
validation requires “an extensive and arduous effort.” Guide-
lines § 14(D)(1). The principal difficulty with construct valida-
tion is that it requires a technique that includes a criterion-re-
lated study, Guidelines § 14(D)(4)—a demonstration from em-
pirical data that the test successfully predicts job perform-
ance.'' Developing such data is difficult, and tests for which it
is required have frequently been declared invalid.'? As a result,

11 Such data may be obtained by studying correlations of test scores of
accepted candidates with their subsequent job performances, or corre-
lations of the test scores of present employees with their current job
performances,

12 See United States v. City of Chicago, supra, $49 F.2d at 430-32;
Douglas v. Hampton, §12 F.2d 976, 985-86 (D.C. Cir. 1975); Malean
Society, supra, 490 F.2d at 395 & n.10; Bridgeport Guardians, Inc. vy.

2la

a conclusion that construct validation is required would often
decide a case against a test-maker, once a disparate racial
impact has been demonstrated.

To determine whether defendants are entitled to use content
validation, we examine the Guidelines’ criteria for that tech-
nique, but we do so bearing in mind our cautionary approach
to the Guidelines, previously expressed. The Guidelines specify
two basic conditions that must be met before content valida-
tion may be used. First, it must appear that what the test
attempts to measure is knowledge or an ability, and not a
general trait, such as intelligence. Guidelines § 14(C)(1). Sec-
ond, the test must not measure knowledge or ability that an
employee will be expected to learn on the job. J/bid.; see also
§ S5(F). The District Court rejected content validation, conclud-
ing both that Exam No. 8155 measures constructs, not abili-
ties, and that, even if what was tested for could be considered
abilities, they could be learned in the five-month training
program.

In specifying how the selection of validation techniques is to
be made, the Guidelines adopt too rigid an approach, one that
is inconsistent with Title VII’s endorsement of professionally
developed tests. Taken literally, the Guidelines would mean
that any test for a job that included a training period is almost
inevitably doomed: if the attributes the test attempts to

Civil Service Commission, 354 F. Supp. 778 (D. Conn), aff'd in part,
rev'd in part, 482 F.2d 1333 (2d Cir. 1973); cf. Schmidt & Hunter, The
Future of Criterion-Related Validity, 33 Personnel Psych. 41, 48 (1980)
(“criterion related validity studies will frequently, perhaps typically, be
technically infeasible”).

A rare example of a criterion-related study that was found accept-
able is Washington v. Davis, supra, 426 U.S, at 249-52, 251 n.17. This
was not a Title VII case, however, and the Court's use of Title VII
concepts, in dictum, to assess the validity of the test in question under
the Fourteenth Amendment does not indicate that the Court was
reviewing the test with the stringency that Title VII requires. See
Guardians Ass'n v, Civil Service Commission, No, 79-7377 (2d Cir.
1980) (slip op. at _._-__). In fact, a less demanding standard was
almost certainly being used, as is clear from the comparison between
Davis and the Court's Title VII decision the preceding term in
Albemarle.

22a

measure are too general, they are likely to be regarded as
constructs, in which event validation is usually too difficult to
be successful; if the attributes are fairly specific, they are likely
to be appropriate for content validation, but this too will prove
unsuccessful because the specific attributes will usually be
learned in a training program or on the job.

The origin of this dilemma is not any inherent defect in
testing, but rather the Guidelines’ definition of “content.” This
definition makes too sharp a distinction between “content”
and “construct,” while at the same time blurring the distinction
between the two components of “content”: knowledge and
ability. The knowledge covered by the concept of “content”
generally mean factual information. The abilities refer to a
person’s capacity to carry out a particular function, once the
necessary information is supplied. Unless the ability requires
virtually no thinking, the “ability” aspect of “content” is not
closely related to the “knowledge” aspect of “content”; instead
it bears a closer relationship to a “construct.” Some researchers
regard content tests as nothing more than assessments of
particular kinds of constructs, e.g., Tenopyr, Content-Con-
struct Confusion, 30 Personnel Psych. 47 (1977); others regard
any ability that is evidenced by observable behavior as suffi-
ciently non-inferential to be considered content, see Ebel,
Comments on Some Problems of Employment Testing, 30
Personnel Psych. 55 (1977). See generally Catell, Validity and
Reliability: A Proposed More Basic Set of Concepts, 55 J. Ed.
Psych. | (1964). Whichever view is adopted, it would seem that
abilities, at least those that require any thinking, and con-
structs are simply different segments along a continuum re-
flecting a person’s capacity to perform various categories of
tasks. This continuum starts with precise capacities and ex-
tends to increasingly abstract ones—from the capacity for
filling out forms to the capacity for exercising judgment.

Recognition that abilities and constructs are not entirely
distinct leads to a conclusion that a validation technique for
purposes of determining Title VII compliance can best be
selected by a functional approach that focuses on the nature of
the job. The crucial question under Title VII is job related-

23a

ness—whether or not the abilities being tested for are those
than can be determined by direct, verifiable observation to be
required or desirable for the job. See Griggs, supra, 401 U.S.
at 431; Vulcan Society, supra, 490 F.2d at 394-95; Chance,
supra, 458 F.2d at 1177. If the job in question involves
primarily abilities that are somewhat abstract, content valida-
tion should not be rejected simply because these abilities could
be categorized as constructs. However, if the test attempts to
measure general qualities such as intelligence or common
sense, which are no more relevant to the job in question than
to any other job, then insistence on the rigorous standards of
construct validation is needed. Since tests of this kind are often
biased in favor of a person’s familiarity with the dominant
culture, permitting them to be used without a showing of
predictive validity would perpetuate the effects of prior dis-
crimination. But as long as the abilities that the test attempts to
measure are no more abstract than necessary, that is, as long as
they are the most observable abilities of significance to the
particular job in question, content validation should be avail-
able. To lessen the risks of perpetuating cultural disadvantages,
the degree to which content validation must be demonstrated
should increase as the abilities tested for become more ab-
stract.

This functional approach, which adjusts the distinction be-
tween content and construct to the nature of the job being
tested for, expands the opportunity for both employers and
courts to rely on content validation. It also avoids making a
threshold choice between content and construct validation
based solely on the nature of the quality tested for unrelated to
the job, a choice that might make content validation seem
inappropriate. To base the content-construct determination on
the nature of the job, it is necessary first to analyze the job to
see if it requires abilities appropriate for content validation.
Instead of choosing between content and construct validation
at the outset, as the Guidelines seem to require, employers and
courts can start the content validation inquiry and use its
results to determine both whether content validation is appro-
priate and whether it has been achieved. Should the attempted

24a

content validation be found inadequate, the reason may be
that this method of validation was not appropriate because of
the pertinent job abilities revealed by the job analysis. On the
other hand, this approach will sometimes indicate that content
validation is appropriate, even though the abilities tested for
could be considered constructs.'?

Just as lessening the severity of the Guidelines’ distinction
between content and construct reduces the likelihood that a
test is invalid because it measures constructs, so sharpening the
distinction between knowledge and ability, now obscured by
the Guidelines, reduces the problem that the test is invalid
because it duplicates the training period, i.e., tests for what
will later be learned. Unlike knowledge, some abilities are
appropriate for testing confirmed by content validation despite
their overlap with post-selection training. A valid measurement
of some abilities can select applicants who will ultimately use
their training to perform their tasks more effectively or who
will more effectively perform similar tasks for which they have
not been specifically trained. On the other hand, content
validation remains inappropriate for tests that measure knowl-
edge of factual information if that knowledge will be fully
acquired in a training program. Approval of such tests,
without predictive validation, risks favoring applicants with
prior exposure to the information, a course likely to discrimi-
nate against a disadvantaged minority. For example, it would
be duplicative of the Police Department’s training program,
and thus invalid, to test applicants for their knowledge of the
Department’s arrest form. Testing for their ability to fill out
the form, however, can be expected to select applicants who

13 There will be some tests whose character is sufficiently clear so that
the content-construct distinction can be applied at the threshold,
without the need to place the test in the context of the job it tests for. A
general intelligence test, see Griggs, supra (Wonderlic Personnel Test),
will almost always need to be assessed by construct validation, since it
necessarily measures for an inferred ability, regardless of the context.
The much-vaunted typing test, in constrast, can always be regarded as
amenable to content validation. However, there are a large number of
tests, including virtually all the “second generation” tests for jobs such
as the one considered here, that will fall into the middle range.

25a

can be successfully trained to perform well at that task and
others like it.

Applying the approach just outlined, we conclude, at least as
an initial matter, that content validation may properly be
selected as the appropriate technique for assessing Exam No.
8155. The exam tests for three basic abilities (although it
purports to test for five): the ability to remember details, the
ability to fill out forms, and the ability to apply general
principles to specific facts. This third ability is assessed in three
contexts: the application of general statements of criminal
offenses to the facts of specific events, the application of
procedures and standards to the facts of specific situations
involving human relations problems. These three basic abilities
are not so abstract, on their face, as to preclude content
validation, provided subsequent consideration of the job anal-
ysis does not demonstrate that important and more concrete
abilities necessary for the job were needlessly omitted from
those considered for measurement. Though all three abilities
involve some inference about mental processes, they are based
on observable behaviors and are far less abstract than such
traits as intelligence, leadership, or judgment. Moreover, test-
ing for these three abilities sufficiently avoids the objection
that the test duplicates the Department’s training program.
Though all three abilities can be trained to some extent, the
test-makers were entitled to select applicants with existing
ability so that training would both enhance their abilities and
prepare them for other tasks requiring similar talents. The vice
of testing for knowledge readily taught in the training program
was totally avoided.

B. Assessing the Content Validity of Exam No. 8155

Since content validation appears to be an appropriate
method for assessing Exam No. 8155, we proceed to consider
whether the use of this method indicates that the exam has
sufficient validity to select applicants for the job of police
officer. The Guidelines describe various aspects of content
validation, but do not neatly list ingredients of an adequate

26a

exam. From our study of the Guidelines, we distill five attri-
butes of an exam with suf*icient content validity to be used
notwithstanding its disparate racial impact. The first two
concern the quality of the test’s development: (1) the test-
makers must have conducted a suitable job analysis, and (2)
they must have used reasonable competence in constructing the
test itself.'* The next three attributes are more in the nature of
standards that the test, as produced and used, must be shown
to have met. The basic requirement, really the essence of
content validation, is (3) that the content of the test must be
related to the content of the job. In addition, (4) the content of
the test must be representative of the content of the job.
Finally, the test must be used with (5) a scoring system that
usefully selects from among the applicants those who can
better perform the job. We consider each of these five matters
in turn.

The Job Analysis

According to the Guidelines, a job analysis involves an
assessment “of the important work behavior(s) required for
successful performance and their relative importance.”
§ 14(C)(2). The job analysis performed by the City, while
somewhat flawed as the District Court pointed out, is nonethe-
less adequate to meet this standard. As far as the first part of
the standard is concerned, the work behaviors involved in
being a police officer were identifed by extensive interviewing,
and subjected to serious review (Job Analysis, Steps 1 and 2).
The District Court found that these work behaviors “were not
defined with precision.” J. Supp. A. at 18. In fact, the
descriptions of the 42 tasks that ultimately appeared on the job
analysis list vary considerably in the level of precision. Some
are complete and unambiguous, such as “1. Checks the condi-

14 These considerations are particularly crucial to an exam validated on
content grounds. See APA Standards at 29: “Content validity is
determined by a set of operations, and one evaluates content validity
by the thoroughness and care with which these operations have been
conducted.”

27a

tion of personal and department equipment such as radio,
patrol car, weapons, etc.”; “35. Attends training sessions.”
Others are more open-ended, but do manage to fulfill their
function by defining the behaviors associated with the task,
such as “3. Performs foot patrol”; “40. Controls various types
of crowds.” Still others are so vague that they communicate
very little real information, such as “10. Interacts with juve-
niles in non-arrest situations”; “39. Performs duties in hostage
situations.”

While greater precision might have been achieved, a com-
plete description of the observable tasks associated with being
a police officer would be a reworded version of the entire
training manual. The Police Department’s list of tasks, despite
some lapses in specificity, contains a sufficient amount of
meaningful information to satisfy the relevant requirement.

The second part of the Guideline standard for a job analysis
requires determination of relative importance of the identified
work behaviors. The City performed this function by means of
an extensively distributed questionnaire, specifying the criteria
to be used in ranking the 42 tasks (Job Analysis, Step 3). The
process as a whole appears to be reasonably accurate, and
neither the plaintiffs nor the District Court raised any serious
objection to it.

Having determined the work behaviors and established their
relative importance, the City then grouped the 42 tasks into
five clusters and asked panels of police officers to identify the
knowledges, skills, or abilities necessary to the effective per-
formance of these tasks. (Job Analysis, Steps 4 and 5). This
function was implemented in a much less satisfactory manner.
Only one of the panels identified the abilities; the other four
used the list of five abilities that the first panel had developed.
This lessened the value of having five independent panels make
this complicated and subjective determination. Moreover, no
effort was made to explain the relationship between any of the
five abilities and the 42 job tasks from which they were
ostensibly derived.'’

15 _ In some instances the relationship is obvious. Plainly the ability to
fill out forms is needed for task 16, “Processes arrests using appropri-

28a

The plaintiffs criticize the required abilities identified by the
City for being undefined. But the type of definition suggested
by the Guidelines—one that describes the abilities in terms of
“observable behaviors and outcomes,” Guidelines § 15(C)(3)—
seems repetitive, since the work behaviors are already defined
in this way. The five identified abilities, with the possible
exception of “human relations, including communication tech-
niques,” are comprehensible enough. Their appropriateness for
measurement would have been considerably clearer, however, if
each panel had explained which tasks required which abilities.
While the Guidelines may be unnecessarily stringent in regard-
ing the identification of this relationship of ability to task as
“essential,” see ibid., such identification does go far toward
eliminating the ambiguities that are otherwise inherent in
generalized descriptions of abilities. Only if the relationship of
abilities to tasks is clearly set forth can there be confidence that
the pertinent abilities have been selected for measurement.

The Test Construction Process

With a job analysis of questionable sufficiency, the City then
proceeded to the test construction stage. As an initial matter,
we note that Exam No. 8155 was developed “in-house,” by
staff members of New York City’s Police Department and
Department of Personnel; there was litile input from any
outside source, and no participation by anyone specializing in
test preparation. Of course, the law should not be designed to
subsidize specialists. But employment testing is a task of
sufficient difficulty to suggest that an employer dispenses with
expert assistance at his peril. Certainly, the decision to forgo
such assistance should require a Court to give the resulting test
careful scrutiny. See Kirkland, supra, 520 F.2d at 425-26;
Vulcan Society, supra, 490 F.2d at 395-96.

ate police department forms and notifications.” But it is not evident,
for example, why any of the five listed abilities are critical to task 32,
“Searches for lost children, runaways, etc.” or task 19, “Guards and
transports prisoners.”

29a

While the determination of how many questions should be
included for each identified ability was made by a fairly careful
numerical analysis (Test Construction, Step 1), the process of
writing the questions themselves was rather haphazard. The
questions were initially framed by police officers, who may
have had expertise in identifying tasks involved in their job but
were amateurs in the art of test construction. In addition, the
officers did not have access to the job analysis material during
much of the process. Finally, the questions, although they were
reviewed, were not tested on a sample population. To be sure,
a complete determination of the questions’ accuracy in measur-
ing the identified abilities would be equivalent in its complexity
to a criterion-related study. But the City did not even perform
the minimal sample testing to ensure that the questions were
comprehensible and unambiguous.

Not surprisingly, the test construction process did not fully
succeed in meeting even its own goal of testing for all the
identified abilities. As previously indicated, Exam No. 8155
does appear to test for the three identified abilities of remem-
bering details, filling out forms, and applying general princi-
ples to specific facts. However, the fourth identified ability,
human relations skill, proved more troublesome. In deciding
how to test for this ability, the City faced a dilemma inherent in
testing for all but the most mundane jobs. To be fully repre-
sentative of the job, a test should measure all the significant
abilities needed for successful job performance, yet some
abilities, especially in jobs of any complexity, are far along the
construct end of the content-construct continuum where
successful validation is difficult. If a test tries to be representa-
tive and measure all significant abilities, including those that
are clearly constructs, it risks the use of inadequate assessment
devices, because the rigorous standard for construct validation
will rarely be met. On the other hand if the test-makers
acknowledge the difficulty of satisfactorily measuring con-
structs and test only for those abilities that are appropriate for
content validation, they encounter the objection that the test is
not sufficiently representative of the job.

30a

Recognizing the difficulty of construct validation, yet reluc-
tant to omit assessment of an important characteristic of
successful job performance, the City attempted to resolve the
dilemma by treating human relations skill as an ability suitable
for content validation and devoting 30 questions, nearly one-
third of the exam, to an effort to assess this ability. Mindful of
an important requirement of content validity, the City carefully
avoided rewarding a test-taker’s prior knowledge and, instead,
supplied in the test itself all the information necessary to select
the correct answers to the human relations questions. Included
before each group of questions was a set of appropriate
standards—essentially “do’s” and “don’ts”—for handling a
particular type of human relations matter. But supplying this
guidance rendered the 30 questions primarily a further assess-
ment of a candidate’s ability to apply written standards to
specific fact situations, and only slightly a measure of his talent
for human relations. Anyone with minimal analytic ability
needed to apply the standards to the various fact situations
could select the one correct answer, even if his intuitive reac-
tion to a human relations problem might be woefully inade-
quate.

Assessing human relations skill will always be a difficult
enterprise, but the deficiency of the City’s attempt does not
mean that a context validation approach is necessarily imper-
missible nor impossible to achieve. As indicated above, at least
within the middle range of the content-construct continuum,
the distinction between context and construct should be deter-
mined functionally, in relation to the job. If the quality
measured is not unduly abstract, and if it constitutes a signifi-
cant aspect of the job, content validation of the test compo-
nent used to measure that quality should be permitted. But that
component must be designed in an extremely careful way.
Test-makers will be well advised to obtain highly qualified
assistance in constructing this portion of an exam.

One desirable approach would be to confront applicants
with simulated real life situations and assess the appropriate-
ness of their volunteered responses. See Firefighters Institute
for Racial Equality v. City of St. Louis, __. F.2d ____ (8th

3la

Cir. 1980). That technique is normally too costly for large
numbers of applicants, but might have usefulness as a testing
device to be used toward the end of the overall selection
procedure, after an initially large group of applicants has been
narrowed down by the results of a written exam and a back-
ground check. If the test component is limited to traditional
pencil and paper methods, it may be preferable to forgo any
pretense of being able to make fine differentiation among
candidates’ human relations skills and instead adopt a pass/
fail approach, rejecting those whose demonstrably inappro-
priate responses to human relations questions mark them as
unsuitable for police work. Another possibility is to recognize
that questions in this area for which only one answer is correct
are likely to be too easy, as were most of the questions on
Exam No. 8155, and therefore of little use in making selections
from among applicants. Instead questions can be designed for
which some answers are appropriate responses and others are
inappropriate. As feasible techniques in this area evolve, em-
ployers will be expected to use them.

With these strengths and weaknesses of the job analysis and
the test construction in mind, we now consider how well the
test, as constructed and used, met the basic requirements of
content validity.

The Direct Relationship Requirement

The central requirement of Title VII, relationship of test
content to job content, was sufficiently satisfied by Exam No.
8155. The job analysis procedure provides adequate assurance
that the identified tasks are in fact the tasks that a police
officer performs. While the procedure for identifying the
abilities required for those tasks was less satisfactory, the three
abilities that were actually tested for appear adequately related
to most of the identified tasks. The list of tasks confirms one’s
intuitive assumption that police officers are required to fill out
forms (see, e.g., “16. Processes arrests using appropriate police
department forms and notifications”), to remember facts (see,
e.g., “18. Gives testimony in court (oral and written)”), and to

32a

apply general principles to specific fact situations (see, e.g.,
“26. Executes warrants.”).

Moreover, these abilities are among the most concrete ones
that can be derived from the list; they are certainly more
concrete than human relations skills, which the test purported
to measure, but did not. Two of the abilities tested for, filling
out forms and remembering facts, are as specifically stated as
they could be without resort to trivial distinctions about
particular kinds of forms and facts. The ability to apply
general standards is somewhat more problematical, since it is a
relatively abstract skill that is relevant to many jobs. However,
if there is any job for which ability in applying and following
rules is an especially important requirement, it is the job of a
law enforcement officer.'®

The Representativeness Requirement

The second requirement established by the Guidelines is that
the test must be a “representative sample of the content of the
job.” As presented by the Guidelines, this representativeness
requirement has two different meanings. The first is that the
content of the test must be representative of the content of the
job; the second is that the procedure, or methodology, of the
test must be similar to the procedures required by the job itself.
The Guidelines express this dual requirement in the following
somewhat inscrutable language: “For any selection procedure
measuring a knowledge, skill, or ability the user should show
that (a) the selection procedure measures and is a representa-
tive sample of that knowledge, skill, or ability. . . .” Guide-
lines § 14(C)(4) (emphasis added).

16 The fact that the factual subject matter of the exam questions (as
opposed to their purpose in measuring abilities) was related to the
subject matter of the job is not a major indicator of the test’s validity,
since the test measured abilities, not knowledge. But it does suggest
that the test has avoided the dangers inherent in using irrelevant
factual material. Such material could skew the test for ability in
directions unrelated to the job, a phenomenon that even the best
designed test might not be able to avoid. The present test avoids that
problem by ensuring that any distorting effects resulting from the
subject matter of the questions are themselves job-related.

33a

Both aspects of the representativeness requirement, if inter-
preted rigorously, would once again foreclose any possibility of
constructing a valid test. The United States, as amicus, argues
that the requirement that the content of the exam be represent-
ative means that all the knowledges, skills, or abilities required
for the job be tested for, each in its proper proportion. This is
not even theoretically possible, since some of the required
capacities cannot be tested for in any valid manner. Even if
they could be, the task of identifying every capacity and
determining its appropriate proportion is a practical impossi-
bility.

It is similarly impossible for the procedures of the test to be
truly representative of the actual job procedures. Tests, by
their nature, are a controlled, simplified version of the job
activities, not the activities themselves. As a practical matter,
virtually any realistic test, except one that directly measures a
physical skill, like lifting 50-pound sacks, is likely to be a
pencil and paper activity, quite different from the job it tests
for. An elaborate effort to simulate the actual work setting
would be beyond the resources of most employers, and
perhaps beyond the capacities of even the most professional
test-makers.

More reasonable interpretations of the representativeness
requirement are appropriate in light of Title VII’s basic pur-
poses. The reason for a requirement that the content of the
exam be representative is to prevent either the use of some
minor aspect of the job as the basis for the selection procedure
or the needless elimination of some significant part of the job’s
requirements from the selection process entirely; this adds a
quantitative element to the qualitative requirement—that the
content of the test be related to the content of the job. Thus, it
is reasonable to insist that the test measure important aspects
of the job, at least those for which appropriate measurement is
feasible, but not that it measure all aspects, regardless of
significance, in their exact proportions. The reason for a
requirement that the test’s procedure be representative is to
prevent distorting effects that go beyond the inherent distor-
tions present in any measuring instrument. For example, al-

34a

though all pencil and paper tests are dependent on reading,
even if many aspects of the job are not, the reading level of the
test should not be pointlessly high. Similarly, the instructions
should not be overly complex, and the exam should not place
candidates under excessive time pressure unless such time
pressure is an identifiable aspect of the job.

Exam No. 8155 meets these representativeness requirements
to an adequate degree. While it did not test for all the skills
involved in being a police officer nor adequately test for the
human relations skill that the job analysis identified as impor-
tant, the ones it did measure—memory, the ability to fill out
forms, and the ability to apply rules to factual situations—are
all significant aspects of entry-level police work. To be sure,
this conclusion would have been easier to reach if the City had
spelled out the relationship between the abilities that were
tested for and the job behaviors that had been identified. But
the relationship is sufficiently apparent to indicate that the
City was not seizing on minor aspects of the police officer’s
job as the basis for selection of candidates. The inadequate
assessment of human relations skill lessens the representative-
ness of the exam and consequently lessens its degree of content
validity, but this deficiency is not fatal, especially in light of the
difficulty of assessing such an abstract ability. Though human
relations skill was deemed so important as to warrant 30 of the
exam’s 100 questions, the City could just as plausibly have
concluded that equally important for the job of policing are
such other abstract qualities as common sense, leadership
potential, sound judgment, or ability to resist provocation.
When a police exam inadequately tests for any of these
abstract abilities, it simply recognizes the limits of the art of
testing. Indeed, the more a test concerns itself with relatively
concrete abilities identified as necessary for successful job
performance, the more likely it is to achieve a sound basis for
assessment of applicants.

Similarly, the procedure that the test employed was not
needlessly unrepresentative of the job itself. In electing to use a
pencil and paper test, the City did not forgo any readily
available and realistically feasible alternative procedure that

35a

would have been more representative of the job. Moreover, the
risks of using a written test were substantially minimized. The
reading level necessary to understand the questions was in
some cases equal to, but generally well below, the training
materials used in the Police Academy. The instructions were
clear enough, and employed an ordinary four answer multiple
choice format, perhaps the most familiar standardized test
technique. In addition, ample time was allowed for taking the
exam, thereby avoiding an unnecessarily pressured situation.

Thus the exam is adequately related to the content of the
policy officer’s job, and adequately representative. The com-
bined effect of this assessment might support a conclusion that
the exam as a whole has content validity, though it would be a
close question whether a test with the disparate racial impact of
this one can be validated when its development departs in some
significant respects even from reasonably attainable require-
ments of the Guidelines. However, even if the construction of
the exam passes muster, the way in which it was used to
distinguish among candidates seriously departs from the third
requirement for content validity and defeats any claim of
validity for a testing process that produces disparate racial
results.

The Scoring Requirement

Essentially, the City used the results of the exam to compile a
rank-ordering of all the applicants, and then selected a passing
score sufficient to generate the required number of potential
trainees. Neither the rank-ordering nor the passing score con-
forms to even the most minimal standards for these two
devices.

Rank-Ordering. The Guidelines provide that rank-ordering
should be used only if it can be shown that “a higher score

. . is likely to result in better job performance.” Guidelines
§ 14(C)(9). This requirement is reasonable and consistent with
Title VII’s provision that the “results” of a test may not be
“used to discriminate.” 42 U.S.C. § 2000e-2(h). If test scores
do not vary directly with job performance, ranking the candi-

36a

dates on the basis of their scores will not select better employ-
ees. It is possible to read the Guidelines’ standard for rank-
ordering as if the required relationship between better scores
and better job performance had to be demonstrated by a
criterion-related study. However, the EEOC’s interpretation of
the Guidelines disclaims such a high standard. The relationship
between higher scores and better job performance may permis-
sibly rest on an inference, but where, as here, the test scores
reveal a disparate racial impact, and that disparity is greater at
high passing scores than at low passing scores, the appropriate-
ness of inferring that higher scores closely correlate with better
job performance must be closely scrutinized.

This close scrutiny is required because rank-ordering makes
such a refined use of the test’s basic power to distinguish
between those who are qualified to perform the job and those
who are not. If a test is content valid, it may be reasonable to
infer that the test scores make some useful gross distinctions
between candidates. Candidates with high scores may well be
expected to perform the job better than candidates with low
scores. See Science Research Associates, Validation: Proce-
dures and Results (1972) (use of criterion “tails” identifying
best and worst candidates more justifiable than continuous
rating). And it may even be that within some range of scores,
some incremental improvements in scores show some positive
correlation with improvements in job performance. But neither
of these propositions provides confidence for inferring that
one-point increments among those who took Exam No. 8155
are a valid basis for making job-related hiring decisions,
especially in the range of scores between 94 and 100. The
reason such a precise inference cannot be so readily drawn is
that content validity is not an all or nothing matter; it comes in
degrees. A test may have enough validity for making gross
distinctions between those qualified and unqualified for a job,
yet may be totally inadequate to yield passing grades that show
yositive correlation with job performance.

Overlooking this point, the City earnestly contends that if
the appropriate abilities were tested for, it makes eminent sense
to select candidates strictly on the basis of ranked scores, even

37a

to the extent of concluding that a candidate scoring 98 will
perform better as a police officer than a candidate scoring 97.
The frequency with which such one-point differentials are used
for important decisions in our society, both in academic assess-
ment and civil service employment, should not obscure their
equally frequent lack of demonstrated significance. Rank-or-
dering satisfies a felt need for objectivity, but it does not
necessarily select better job performers. In some circumstances
the virtues of objectivity may justify the inherent artificiality
of the substantively deficient distinctions being made. But
when test scores have a disparate racial impact, an employer
violates Title VII if he uses them in ways that lack significant
relationship to job performance.

Permissible use of rank-ordering requires a demonstration of
such substantial test validity that it is reasonable to expect one-
or two-point differences in scores to reflect differences in job
performance. Our prior conclusion that the test itself may have
had enough validity to be used does not, therefore, lead to
approval of using its results for rank-ordered selections. On
the contrary, the defects we noted in the job analysis and the
test construction are substantial enough to preclude an in-
ference that passing scores will correlate with job performance
closely enough to justify rank-ordered selections. While we do
not criticize the City’s efforts as extensively as did the District
Court, we agree that the identification of pertinent abilities,
the demonstration of their relationship to the job tasks, and
the process of developing the questions, were flawed.

These shortcomings take on added significance when it is
recognized that the test just barely satisfied even our lenient
construction of the Guidelines requirement of procedural rep-
resentativeness. As the EEOC has advised, it is “easier” to
make the inference of a relationship between higher scores and
better job performance “[t]he more closely and completely the
selection procedure approximates the important work behav-
iors.” EEOC Questions and Answers, supra, Q. 62. Unlike the
District Court, we are not willing to reject any use of a police
exam simply because the pencil and paper procedure of the test
is not a close approximation of the job. Nor are we willing to

38a

preclude rank-ordering because a pencil and paper procedure
was used. Given the current state of the art in employment
testing, we think it would be unrealistic to condemn pencil and
paper tests. Alternative procedures have not beer shown to be
readily available within the limitations of time and resources
confronting most employers. Nevertheless, we cannot ignore
the Guidelines’ criticism of assessing ability to perform com-
plex tasks by a test procedure so different from the work
setting. When the selection procedure does not closely approx-
imate the important job tasks, it becomes especially important
to insist upon a strong showing that other aspects of content
validity have been demonstrated. And that demonstration must
be very substantial when a test procedure that does not closely
approximate the job is sought to reflect the fine gradations
required for rank-ordering. In short, while we might not agree
with the District Judge that the defects in the test preclude a
finding of sufficient content validity to permit its use, we agree
that content validity has not been shown to the extent neces-
sary for rank-ordering.

In addition to inadequate demonstration of validity, the test
may not be used for rank-ordered selections because of the
total absence of any evidence that the exam possessed another
vital feature—reliability, that is, the extent to which the exam
would produce consistent results if applicants repeatedly took
it or similar tests. Of course, there is no expectation that
applicants will take any given test more than once. But if an
exam lacks reliability to such an extent that results would be
significantly inconsistent if the same applicants were to take it
again, that is an important indication that the test is not
especially useful in measuring their abilities. Although not
explicitly mentioned in the Guidelines reliability is prominently
identified in the APA Standards (to which the Guidelines refer
in § 5(C)) to be as basic for evaluating an exam as validity
itself. See APA Standards, supra as 48-55. Like content valid-
ity, reliability is not an all or nothing matter. It too comes in
degrees. What is required is not perfect reliability, but rather a
sufficient degree of reliability to justify the use of being made
of the test results. Without some substantial demonstration of

39a

reliability it is wholly unwarranted to make hiring decisions,
with a disparate racial impact, for thousands of applicants that
turn on one-point distinctions among their passing grades.

Two aspects of reliability deserve consideration in assessing
the use of rank-ordering. The first is the quality of the exam
questions. The more skillfully they have been formulated, the
more likely it is that results on one question will correlate with
results on the other questions and that successive test scores
would be consistent. This will avoid the tendency of scores to
vary because of extraneous factors such as test administration.
Whether this aspect of reliability has been achieved to an
extent sufficient to justify rank-ordering need not be left to
general consideration of the quality of the test construction
process. A basic demonstration of this aspect of reliability can
easily be made by the test-maker, before the test is adminis-
tered to job applicants. The test-maker can pre-test his exam
by giving it twice to a sample of persons generally approximat-
ing the characteristics of the population where the test is
expected to be used for employment selection. To avoid distor-
tion due to recollection, the test given at a later date to such a
sample can use similar but not identical questions. Another
somewhat useful indicator of reliability is a technique known
as a split-half correlation—dividing each component of the test
into equal halves and observing how consistent were an indi-
vidual’s scores on each half.'’ This tec

[Text truncated at 120,000 characters. The full text is on the page linked above.]

---

Source: Frix Law Library, https://www.frixlaw.com/law-library/documents/brief%3Amicro_IA40385007_1629%3A1. Public record. Not legal advice.