Butting Heads With the Test Makers

The ranking officials of the College Board held their annual meeting in Chicago last week and decided that they are still pretty pleased with their best-known product, the Scholastic Aptitude Test (SAT). There were reaffirmations that the exam is an effective and fair means of predicting college performance and testimony designed to discredit critics who say that it puts some minorities and the underprivileged at a disadvantage. The experience was "reassuring," as one participant put it.

And there is a good reason for confidence at the College Board. Despite all of the recent controversy over the SAT and progress on some fronts by test reformers, college admissions officers seem committed to the exam and unshaken in their loyalty to the Board and to Educational Testing Service (ETS), the company hired to actually design and administer the SAT. Some schools have conducted small-scale evaluations of the SAT on their own; others are only vaguely aware of the specifics of the debate over standardized testing. But the people who govern admissions tend to share a common perspective on the SAT debate: You can find a handful of studies to endorse any position--pro or con--and at this point, there isn't enough evidence to prompt a retreat from reliance on the test.

Surprisingly, perhaps, none of this has discouraged the critics. Test reform groups succeeded last year in pushing through a New York state law that requires ETS for the first time to provide students with their answer sheets, the questions themselves, and an explanation of scoring formulae. ETS responded by voluntarily expanding the so-called "truth-in-testing" provision to all states, and a House of Representatives subcommittee will hold hearings this week on whether to require disclosure by federal law. In the words of none other than the College Board's executive director of research and development, Robert G. Cameron, truth in testing "has gone a long way in de-mystifying" the SAT.

On other fronts, the reformers have had less impact. Investigative groups ranging from Ralph Nadar's organization to the Federal Trade Commission (FTC) have raised serious doubts about the test makers' claims, but policies here at Harvard and elsewhere haven't changed significantly as a result. Two leading critics of the SAT, and of ETS itself, are Harvard-affiliated doctors Warner V. Slack and Douglas Porter, who have combined their respective backgrounds in computer medicine and psychology to produce a detailed critique of the test and its administrators. Their most important research on the issue appeared last year in the Harvard Educational Review, but Slack began the campaign in 1977 with a New York Times Op-Ed page piece, declaring with characteristic enthusiasm, "High school students unite. Save your money. Leave your No. 2 pencils at home. Boycott these tests."

Armed with extensive statistical evidence, culled mostly from ETS's own studies, Slack and Porter have since used every opportunity to preach that message. They acknowledge that their appearances at academic conferences and on radio and television talk shows haven't as yet turned admissions offices upside down; but like other test reformers, they point to the five-year battle over disclosure and predict ultimate success.

Slack, an associate professor of Medicine, serves as co-director of the Computer Medicine Laboratory at Beth Isreal Hospital, where Porter is a principal associate. They share, in Porter's words, "a keen interest in the formation and use of questions" and work together on ways to broaden the computer's ability not only to store and transfer medical information, but also to diagnose illness and prescribe treatment by communicating with patients and doctors.

They also share a general skepticism of exercises described as "aptitude" tests and advertised as being impervious to specific preparation. When ETS challenged Slack's 1977 article with the findings of "all known studies," Slack and Porter requested copies of said studies and went to work checking up on the claims of the Princeton, N.J.-based firm. "We went in assuming the research they did wasn't all that good," says Slack. "What we didn't know was that they had consistently misrepresented and omitted information."

In brief, here are the arguments Slack and Porter make, along with responses from ETS and the College Board.

I The Harvard-affiliated doctors accuse the test makers of implying in technical publications and literature for students that test preparation has less of an impact on scores than studies have shown. Moreover, Slack and Porter argue, ETS has only recently begun to distinguish between short-term "coaching," which it calls ineffective, and long-term "intensive training," which it now concedes can raise scores. Before making this distinction, Slack and Porter say, ETS misled students into thinking that even a school year's worth of special preparation would not affect math or verbal aptitude as measured by the SAT. Cameron of the College Board agrees that his organization has shifted its rhetoric on the effects of long-term coaching, but he denies any deliberate misrepresentation, saying, "We came to the realization that the distinction was not clear to students." He joins ETS vice president Rex Jackson in maintaining that short-term drill-and-technique instruction will have insignificant results.

In the more obscure debate over the validity of the SAT as a predictor of academic success, Slack and Porter argue that research has been consistently misinterpreted by ETS. The testers, on the other hand, charge their critics with intentionally using improper statistical methods in re-analyzing existing data. The bottom line is that ETS and the College Board contend that using the SAT "improves" prediction by 27 per cent over a system in which high school grades are used alone; Slack and Porter set the factor at "an insignificant 1 to 4 per cent."

I Finally, there is disagreement over how ETS describes what the SAT actually tests. Again, Slack and Porter say that ETS implies that it is measuring innate and unchanging abilities, even if its more recent literature places a heavy emphasis on skills developed during schooling. Mary Churchill, an ETS spokesman, responds that almost every aspect of the voluminous information packet received by students emphasizes that the SAT is not an intelligence test, but rather a measure of "learned academic skills."

This is very much how college officials view the entire debate: a series of semantic and statistical contradictions which so far haven't warranted much of a reaction from schools. Bowdoin College in Maine remains the only prominent institution that does not require the SAT for admission. Elsewhere, administrators point out that the exam has never been more than one aspect of their decision-making processes, that they interpret scores only as rough estimates (thus allowing for small increases resulting from special coaching), and that the SAT is "close enough to standardized," in one official's words, to provide a check on differing high school grading scales. "There will always be disparities in life," says Lawrence J. Momo, Columbia University's associate director of admissions, "and in this controversy I see it as a matter of who you believe; we really can't get bogged down in all the minutiae unless something is really proved wrong."

This is where Slack and Porter believe they will eventually change minds. "Even if you can't follow the statistics, there is clear evidence that ETS, a monopolistic company, has lied about its product," Slack says. Once admissions officers begin to doubt ETS's word, "it will open up all of the other questions about what 'aptitude' means and about whether we want to weight education so much toward a test that involves little more than tricky math and little-used vocabulary," he continues. An example:

Until 1978, ETS did not acknowledge or refer to two studies conducted in 1961 and 1965 that attributed gains of 40 to 85 points to special test preparation. ETS's Jackson says in his formal written response to the Slack-Porter Educational Review article that the studies "examined score gains by students involved in lengthy educational programs that go well beyond what is ordinarily described by the term 'coaching.'" But in reviewing the mountain of research conducted on the issue, Slack and Porter found that one of the programs analyzed was only 17 minutes longer than those Jackson calls "short-term." Asks Slack: "How much coaching will be done in 17 minutes; wouldn't you say they're deceiving their customers?"

ETS counters that the studies in question didn't use proper control groups. Slack and Porter argue that they compensated for this in their analysis, adding that ETS itself quotes studies lacking concurrent controls when the findings suit its ends. Not so, responds Cameron, who accuses his critics of allowing "social aims" to shape their research and makes passing reference to the fact that Slack was a classmate of Ralph Nadar's at Princeton in the early 1950s, "and maybe that means something."

The most recent FTC report on the SAT, compiled by Joan Gerrity of the agency's Boston office and released last April, concurs with the Slack-Porter conclusion that "ETS and the College Board have not fully disclosed the potential benefits of coaching," Gerrity says. The report also raises questions about "how useful schools should find a test that appears not to be very standardized" and "how much high school education has become oriented toward a test that may not be valid for a lot of students," she adds. Yet admissions officers do not question ETS's practices, describing the company in such terms as "eminently reputable" and "honorable."

"There is a certain vindictiveness about the Slack-Porter argument," says Dean K. Whitla, director of the Office of Instructional Research and Evaluation and the longest standing member of Harvard's admissions committee. Whitla, who has enjoyed periodic research support from ETS, made a statement in 1962 that Slack and Porter now take pleasure in quoting: "If this thesis [that SAT scores can be increased by an intensive tutoring program] were to be proved true, its proof would challenge the validity and essential purposes for which the test was constructed." Today, Whitla says, Harvard is continuing its standard practice of review and reevaluation; but despite criticism, the SAT is still considered a "fair judge of intellectual abilities." Ian Hodos, associate director of admissions at New Jersey's public institution, Rutgers University, says his school "is, if anything, increasing reliance on the SAT as an indicator of students' capacities."

Slack and Porter endorse what Whitla describes as a growing reliance on Achievement Tests at Harvard and other schools. But they warn that too much emphasis on these exams--which test knowledge of individual academic subjects on terms dictated by ETS and the College Board if teachers felt compelled to prepare students specifically for the achievements. The answer for Slack and Porter is one that admissions offices and ETS would consider a step backwards: abandon the SAT, use other standardized exams to a limited extent, and place even more stock in high school grades and recommendations.

"The question we'd like every admissions officer to ask himself is, 'Does this test [the SAT] do me enough good to make it worth all of the negative aspects?'" says Slack. "And we think the answer is 'no.' "