National Assessment Governing Board

DEPARTMENT OF EDUCATION

National Assessment Governing Board

AGENCY: National Assessment Governing Board; Department of Education.

ACTION: Notice of information collection activity; request for comment.

-----------------------------------------------------------------------

SUMMARY: In compliance with the Paperwork Reduction Act (44 U.S.C. 3501

et seq.), this notice announces a proposed information collection

request (ICR) of the National Assessment Governing Board. The

information collection is to conduct validity studies in conjunction

with the pilot study of the proposed national tests in 4th grade

[[Page 49558]]

reading and 8th grade mathematics, in March 1999. Before submitting the

ICR to the Office of Management and Budget (OMB), the Governing Board

is soliciting comments on the information collection as described

below.

DATES: Comments must be submitted on or before November 16, 1998.

ADDRESSES: Submit written comments identified by ``ICR: Voluntary

National Test-Pilot Validity Studies'' by mail or in person addressed

to Ray Fields, Assistant Director for Policy and Research, National

Assessment Governing Board, Suite 825, 800 North Capitol Street, NW.,

Washington, DC 20002. Comments may be submitted electronically by

sending electronic mail (e-mail) to [email protected]. Comments sent

by e-mail must be submitted as an ASCII file avoiding the use of

special characters and any form of encryption.

All written comments will be available for public inspection at the

address given above from 8 a.m. to 4:30 p.m., Monday through Friday,

excluding legal holidays.

FOR FURTHER INFORMATION CONTACT:

Ray Fields, Assistant Director for Policy and Research, National

Assessment Governing Board, Suite 825, 800 North Capitol Street, NW.,

Washington, DC 20002, Telephone: (202) 357-0395, e-mail:

[email protected].

SUPPLEMENTARY INFORMATION:

I. Information Collection Request

The National Assessment Governing Board is seeking comments on the

following Information Collection Request (ICR).

Type of Review: New.

Title: Validity Studies of the Voluntary National Tests in 4th

Grade Reading and 8th Grade Mathematics.

Affected Entities: Parties affected by this information collection

are state, local, Tribal Government or non-public education agencies.

Abstract: Pub. L. 105-78 vests exclusive authority to develop the

voluntary national tests in the Governing Board and also prohibits the

use of Fiscal Year 1998 funds for pilot testing, field testing,

implementation, administration, or distribution of voluntary national

tests. If Congress does not prohibit further development of the

voluntary national tests after September 30, 1998, the Governing Board

intends to begin pilot testing of items, (i.e. test questions) and

conduct validity studies of test procedures in March 1999.

Pub. L. 105-78 also requires the Governing Board to make four

determinations about the voluntary national tests: (1) The extent to

which test items selected for use on the tests are free from racial,

cultural, or gender bias, (2) whether the test development process and

test items adequately assess student reading and mathematics

comprehension in the form most likely to yield accurate student

achievement in reading and mathematics, (3) whether the test

development process and test items take into account the needs of

disadvantaged, limited English proficient, and disabled students, and

(4) whether the test development process takes into account how

parents, guardians, and students will appropriately be informed about

testing content, purposes and uses.

The purpose of the validity studies is to assess procedures for

administering the proposed voluntary national tests in reading and

mathematics. Since test administration can affect student performance,

the validity studies determine if characteristics of test performance,

such as non-standard conditions for students with special needs, affect

student performance. Three studies are included in this information

collection. Since the VNT is designed to be administered in two, 45-

minute sessions, the Effect of Break Length between Testing Sessions

validity study will examine this impact on examinee test scores and

make recommendations for future administrations of the test. The

Effects of Calculator Type validity study will investigate how student

familiarity with a particular calculator affects test performance.

Design specifications for the 8th grade mathematics test call for the

use of a calculator in one testing session. The National Assessment

Governing Board undertakes this study to inform policy, for proposed

future field and operational tests, about whether to issue standard

calculators or permit students to take the test with their own

calculator. The Effects of Extended Time and Small Group Administration

Accommodations validity study investigates non-standard test

administration procedures for the inclusion of students with

disabilities and students with limited English proficiency in the

voluntary national tests. This study considers the testing

accommodations of extended time (up to two times the standard

administration test length) and small group administration for students

with disabilities, students with limited English proficiency (LEP) and

comparison groups of non-disabled, non-LEP students. This study will

measure the magnitude of the effect of test accommodations on student

performance.

Effects of Break Length between Testing Sessions. The

specifications for the VNT call for the test to be administered in two-

45 minute sessions given on the same day but do not dictate the

specific scheduling of these two sessions. As school personnel will

administer the operational VNT, it is likely that there will be some

variation in test administration procedures at participating schools.

We anticipate that most schools will opt for two morning sessions with

a short break between, but some schools may have logistical reasons for

administering the test in one morning and one afternoon session. Data

will be collected from test administrators on the test administration

schedule and activities that occurred during testing sessions. This

validity study would describe break lengths and activities, investigate

their effects on test performance, and would allow AIR to advise NAGB

on scheduling options for the VNT field-test and operational

administration.

The Effects of Calculator Type validity study will use released

NAEP questions for which national statistics are available, rather than

test questions for the VNT pilot test, Eighteen 8th grade students

would be recruited to participate in a ``think-aloud'' procedure.

Students would take one group of mathematics questions using either

their own or a standard issue calculator, followed by another group of

questions using the opposite calculator. While answering these

questions, examinees would respond to interviewer prompts about their

thinking processes. Students would also complete a brief questionnaire

about their calculator use in schools. Data would be analyzed to

determine the ways in which calculator use and familiarity with

calculator features affects student performance.

The Effects of Extended Time and Small Group Administration

Accommodations validity study will investigate the impact of

accommodations on the test performance of special populations; it will

be conducted in two parts. In the first part, 900 students at each

grade level will be added as an augmentation to the original pilot

sample (300 students with disabilities who would take the VNT either

unaccommodated or with accommodations that do not require altered test

formats (e.g., large print, oral presentation), 300 students with

limited English proficiency, and 300 non-disabled, non-LEP students)

would either take the VNT under the standard time condition (two 45-

minute

[[Page 49559]]

sessions) or under an extended time accommodation (two sessions, each

up to 90 minutes). Large schools and schools with large numbers of LEP

students would be recruited to participate and would be assigned to one

of the two conditions. Because of the small number of eligible students

with disabilities in a given school, 110 schools (60 for 4th grade

reading, 50 for 8th grade mathematics) will be recruited to participate

in this study.

For the second part, the accommodations of extended-time and small-

group administration would both be considered for a sample of 750

students with disabilities eligible for small-group or extended time

testing accommodations at each grade from schools participating in the

pilot sample, both from students in classrooms that will be sampled

from those schools for pilot study participation and from the students

in the remaining classrooms in those schools. Thus, all students will

be selected from schools selected to participate in the main pilot test

and some of the students selected for participation in this study will

also have been selected for inclusion in the main pilot test.

Small group accommodations are often offered to students with

disabilities but the manner of the accommodation varies. The small

group accommodation could be a ``pull-out'' session in another

classroom, or, due to space or staffing restrictions, the small group

may be ``embedded'' in a larger setting (such as a library or

cafeteria) where other activity is present. This study distinguishes

between these two methods of providing small group accommodations for

students with disabilities. A subsample of schools that are already

participating in the main pilot VNT would be assigned to one of five

conditions: (1) Standard time, small group ``pull-out'' administration,

(2) extended time, small group ``pull-out'' administration (3) standard

time, ``embedded'' small group administration, (4) extended time,

``embedded'' small group administration, (5) standard time, standard

(large) group administration. All eligible students with disabilities

in the school would then take the VNT under the prescribed condition. A

total of 750 students with disabilities would be recruited at each

grade level, resulting in 150 students in each of the five conditions

mentioned above. Due to the small number of eligible students with

disabilities in a given school, it is estimated that 290 schools (4th

grade) and 267 schools (8th grade) would be needed to complete the

sample size. Parental consent will be sought for all students selected

to participate under conditions different from those identified in

their IEPs.

All students in both parts of this study who take a test under the

extended time would, at the end of the first 45 minutes of a testing

session, be asked to switch to a different color pencil. Students in

this condition could then answer remaining items or return to skipped

items, using the different color pencil, until they have completed the

test or until the end of the extended time period. All students (all

conditions) would be asked to complete a short questionnaire about the

length of the test.

Data from this study would be analyzed to determine which groups

benefit from the accommodation of extended time, which method of small

group administration maximizes performance for students with

disabilities, and how much accommodations affect test performance.

Analyses would also be done to determine if the current test

administration procedures provide sufficient time for students to take

the VNT. Recommendations will be made for providing appropriate

accommodations for students with disabilities and students with limited

English proficiency on future field tests and operational tests of the

VNT.

In order to ensure adequate control and proper identification of

the booklets of test items, and conduct necessary analyses of the data

that results from the information collection, the following background

information will be collected on the cover of the booklets of test

questions: student name, date of birth, race/ethnicity, and sex (all to

be supplied by the student), and special education status, limited

English proficiency status, disadvantaged status, test administration

accommodations, and primary language (collected by the test

administrator under contract). Although students will write their name

on each booklet for identification purposes during the administration

of the pilot test, the students' names will be removed from the booklet

shortly after the pilot test. Student names will not be included in the

database for analysis and will not leave the school building where

pilot testing is taking place. Instead, a unique numeric or

alphanumeric identifier will be assigned to each booklet for tracking

and analysis purposes. No third party notification or public disclosure

burden is associated with this collection.

Burden Statement

Effects of Break Length between Sessions: This study will not

require any increase in burden for students above that required for the

pilot study. School staff burden for this study is approximately 10

minutes per school (672 for reading and 372 for math) or 174 hours

total.

Effects of calculator use study: The respondent burden for this

study is 40.5 hours, or 2.25 hours for each of the 18 students

participating in the Cognitive Laboratory study. This estimate is based

on 90 minutes of cognitive think aloud and 45 minutes of test

administration procedures by research staff.

Testing accommodations validity study: The annual burden respondent

estimate is based on 90 minutes of testing and 30 minutes of test

administration activities (e.g., delivering instructions, handing out

and collecting booklets, and providing background information as

described above) per student, or two hours per student, in the standard

time condition. Students who take the test in the extended time

condition have up to three and a half hours each: 30 minutes of test

administration activities, and up to 180 minutes of testing time,

although this may be less if students finish early. Five hundred

seventy-three of the students participating in this study will not

require any increase in testing burden above that required for the

pilot study, 382 will require an increase of 1.5 hours of testing

burden, 327 will be new students with a total testing burden of 2

hours, and 218 will be new students with a total testing burden of 3

hours. In addition, there is an additional 6 minutes of burden per each

of the 3,300 students to answer questions about the length of the test.

Total student burden for the 3,300 students participating in the 4th

grade reading test and 8th grade mathematics test is 7,270 hours.

School staff burden for this study, for both reading and mathematics

tests, is 225 hours. This includes questions about classroom practices

to be asked of school staff by test administrators, 5 minutes each for

the 2,700 students with disabilities and students with limited English

proficiency included in the study. There is no school staff burden for

the 600 students without disabilities or limited English proficiency.

Participation in the pilot test and these validity studies is

voluntary. State, local, and non-public education agencies are not

mandated or required to participate.

[[Page 49560]]

Summary

The total number of students involved in the validity studies,

above that which has already been requested in the pilot VNT

collection, is 3,318 with a total burden of 7310.5 hours. Total school

staff burden for these validity studies is 399 hours.

II. Request for Comments

The National Assessment Governing Board solicits comments to:

(a) Evaluate whether the proposed collection of information is

necessary for the proper performance of the functions of the Governing

Board, including whether the information will have practical utility;

(b) Evaluate the accuracy of the Governing Board's estimates of the

burden of the proposed collection of information;

(c) Enhance the quality, utility and clarity of the information to

be collected;

(d) Minimize the burden of the collection of the information on

those who are to respond, including through the use of appropriate

automated, mechanical or other technological collection techniques or

other forms of information technology, e.g., permitting electronic

submission of responses.

III. Public Record

A record has been established for this action. A public version of

this record, including printed, paper versions of electronic comments,

is available for inspection from 8 a.m. to 4:30 p.m., Monday through

Friday, excluding legal holidays. The public record is located in Suite

825, 800 North Capitol Street, NW., Washington, DC 20002. Comments may

be submitted electronically by sending electronic mail (e-mail) to

[email protected]. Comments sent by email must be submitted as an

ASCII file avoiding the use of special characters and any form of

encryption.

The official record for this action, as well as public version, as

described above will be kept in paper form. Accordingly, the National

Assessment Governing Board will transfer all comments received

electronically into printer, paper form as they are received and will

place the paper copies in the official record which will also include

all comments submitted directly in writing. The official record is the

paper record maintained at the National Assessment Governing Board,

Suite 825, 800 North Capitol Street, NW, Washington DC 20002.

List of Subjects

Pilot tests for the voluntary national tests in 4th grade reading

and 8th grade mathematics, validity studies, and Information Collection

Request.

Dated: September 11, 1998.

Roy Truby,

Executive Director, National Assessment Governing Board.

[FR Doc. 98-24849 Filed 9-15-98; 8:45 am]

BILLING CODE 4000-01-M