In Search of Good Research (2)

“There are three kinds of lies: lies, damned lies and statistics.” (Benjamin Disraeli or not, sic!)

       In his introduction to Reading Educational Research – How to Avoid Getting Statistically Snookered, Gerald W. Bracey states, “Misleading statistics abound. “ His entire book is an illustration of this claim, starting from the very chapter (Data, Their Uses and Their Abuses) to the final part (Testing: A Major Source of Data – and Maybe Child Abuse).

*As a side note, I loved reading in the Foreword about his struggle to expose statistics abuse for years as he demystified the politics behind many U.S. educational programs, institutions and policies (No Child Left Behind, NAEP – National Assessment for Educational Progress and more).  At the time of writing this book he was losing a part-time job at George Mason University because, as Jay Mathew remarked, “The school couldn’t take the heat that often follows Bracey in his scholarly travels.” – in other words, he was too direct in his approach to truth.

The book is full of examples that support the 32 Principles of Data Interpretation and guides the reader to a deeper understanding of how data is collected, presented and used. I will only selectively refer to some of them as I cannot obviously make a summary in a single blog post.

…………………………………………………………………………………………………………………………………………………………………………………………………

Principle 1: Do the arithmetic

Failure in doing the math occurs mostly in examples that track changes in data over time.

One of the most notorious example was the statistical assertion:

“Every year since 1950, the number of American children gunned down has doubled.”

There are a number of issues, aside from math, with this statement:

–          How does the author define “children”? Does he mean those up to 16 or up to 19? That would make an enormous difference.

–          What does he mean by “gunned down”? Does he include suicides and accidents?

Bracey goes on saying, “It’s an American thing. Americans in particular ascribe to numbers an accuracy, precision and objectivity that isn’t there.” It is the same story of rice we all know. If we do the math in this case…we would arrive at the ridiculous case of the American children exceeding by 1987 the total human population in all of history.

Another example is the NRP (National Reading Program) repeated claim that it has received more than 100,000 studies in the course of its work. Given that duration of the NRP and the length of a work day this works out to about… 1.6 studies per minute.

Also, deliberate distortions can impede understanding of the data as they are more insidious and serve an agenda.

“Nationally, about half of high school graduates have not mastered 7^th grade arithmetic.” (William Bennett, former secretary of education, 2005 – Washington Post)

–          High school graduates are not tested, so how could he know?

–          Bennett does not specify what he means by “mastered”. Must a student get 100% or a lesser score would suffice?

–          What does “7^th grade arithmetic” mean? Again, lack of specificity.

Moreover, Bracey notes that there is a strong tendency for people to accept anything that shows public schools in bad light (he calls it “Neurotic Need to Believe the Worst”) over anything that shows improvement.

…………………………………………………………………………………………………………………………………………………………………………………………………

Principles 2 and 3: Show me the data! Beware of selectivity in the data.

Selective use of statistics

Bracey points to this as the “the most common abuse of all statistics” – choosing those data that, while they may be accurate in themselves, reveal only part of the picture.

He illustrates it thorough several examples, the most poignant being the publication of A Nation at Risk (1983) that, in addition to its warrior-like rhetoric (“Our nation is at risk…It is threatened by a rising tide of mediocrity.”), undermines the very concept of data integrity by selecting statistics to support its agenda.

“There was a steady decline in science achievement scores of the U.S. 17-year-olds as measured by national assessments in 1969, 1973, and 1977.”

Why did they select only science and only 17-y.o.? Because (and Bracey shows the complete charts) it was the only group that showed decline.

Another one from the same publication:

“Average achievement of high school students on most standardized tests is now lower than 26 years ago when Sputnik was launched.”

Guess what? The commission didn’t even have data from “most standardized tests” because most standardized tests did not permit trend analysis over such a long period as 26 years.

………………………………………………………………………………………………………………………………………………………………………………………………….

There are more principles illustrated with clear examples from the field of education, after which Bracey discusses more research methodology and specific terminology (such as the differences between rates, numbers or scores) that are critical to understanding research reports. It is amazing how easily we can be deceived on the sole basis of ignorance or lack of attention.

It was interesting to grasp the “achievement gap” theory. After he dissects the inaccuracy of the data (I won’t bore you with charts and mathematical calculations), he analyzes  how certain student groups (blacks, for instance) will be even more damaged by the policies and classroom practices currently implemented in the U.S.

People typically talk about the achievement gap in terms of scores. But Cohen and Gandal (Do Graduation Tests Measure Up?”, 2004) present the achievement gap in terms of rates which, the author demonstrates, is “neither logical nor legitimate.” The gap between whites and blacks was huge – 40%. The 10^th grade test liberated 77% of whites to study material other than covered by the test – and here is the trick- while the non-passing black students had to be removed from the classroom to receive drilling on test objectives. There you go: white students are free to move ahead with the curriculum (and learning) while the black students are held back to study the same things over and over to pass the test.

Result? The black students pass the test but in terms of real achievement the gap has widened. But then the newspaper boost with pride because they did pass:

“In sum, the passing rates will tell you nothing about what is happening to any achievement gap.” (p.57)

…………………………………………………………………………………………………………………………………………………………………………………………………..

Another example is the discussion on the concept of “rank”. I chose this one because I see the hysteria surrounding PISA scores and other international tests.  Ranking can be so easily misread without a context (i.e. ranking last on Graduate Record Examination still places you among a fairly elite 25% of Americans who have at least a bachelor’s degree) and it has been a tool to induce anxiety on many occasions.

Following the TIMSS (science assessment) in 1995, the newspapers decried the low rank of the U.S. (19^th place). Upon a closer examination, the American students actually ranked 6^th – see all the countries clustered together with similar percentage. More, “if American students had managed a mere 5% more correct, they would have shot all the way the ladder to rank 5^th in the world. Conversely, had they slipped a mere 5% points they would have fallen all the way to 29^th.” (see chart)

Rank	Country	Percent Correct
1	Singapore	70
2	Korea	66
3	Japan	65
4	Czech Republic	64
6	Bulgaria	62
6	Netherlands	62
6	Slovenia	62
9	England	61
9	Hungary	61
9	Austria	61
11.5	Belgium (Flemish)	60
11.5	Australia	60
14	Slovak Republic	59
14	Sweden	59
14	Canada	59
19	Ireland	58
19	UNITED STATES	58
19	Russian Federation	58
19	New Zealand	58
19	Norway	58
19	Hong Kong	58
19	Germany	58
23.5	Thailand	57

…………………………………………………………………………………………………………………………………………………………………………………………………

At the end of this blog (parts 1 and 2), and after wrapping my head around sometimes obscure research jargon and complicated mathematical formulas, I still have a question:

Who has the time to analyze every piece of research/study we come across?

As I said on Twitter, even if armed with background knowledge on reading data, with the necessary awareness about its potential of being manipulated, who can really devote so much time in analyzing every single aspect of the respective study/ report/ research?

The teacher workload is overwhelming at times (I should know as I have been writing report comments for two weeks now and still haven’t finished) and we cannot look into research as we should. Between schoolwork, Twitter, and reading educational blogs, it is more than challenging to make more time for an in-depth understanding.  On the other hand, allowing misleading research to drive education policies and programs is more than damaging and it does not take far history to teach us that.