Texas Schools Reading Exams Fail National Test
Parents should view the results of state reading tests with a critical eye: Texas students may be passing their state assessments, but that doesn’t mean they are competitive with their peers nationwide.
A Texas Tribune analysis of results on the state and national tests since 2003 show wide gaps in results, with the Texas TAKS test showing proficiency rates up to 18 percentage points higher than the National Assessment of Educational Progress, or NAEP. The gaps are apparent in reading at the fourth and eighth grades, the only grades the national test measures. The state's math results tracked ...
Comments (6)
AdamColligan
"Plus, [Criss Cloudt] says, teachers daily emphasize preparation for the state exams. Students don't have the same motivation to do their best work on the federal test."
__________
Seriously? She's arguing that 10-year-olds are, what - not bothering to write what they think is the right answer? ...because they understand the intricate relationship between federal and state assessments of their ability and so disdain the federal one that they will actually sit in front of the test and mark it with a pencil but *not* mark it with their best answer?
I'll held off on calling this a severely lame excuse...if the TEA can publish some out-of-this-world civics testing results for these cohorts.
KaraThrace
Although I appreciate that someone is reporting this story, and the facts are, for the most part, accurate (there are real issues with the reading TAKS and its focus on low level literacy), I have some concerns about the literacy (or proofreading ability) of Ms. Abby Rapoport.
"Site unseen" That's just sloppy. And if it's a direct quote from Ratcliffe (in writing), then it should have a [sic] next to it.
This sentence, which is a paraphrase, couldn't be any more awkward: "Plus, she says, teachers daily emphasize preparation for the state exams."
How about, "Teachers emphasize preparation for the state exams daily"?
Dr. Meiers is from A&M University? Not Texas A&M University?
But more importantly, I'd like to see this reporter push the story harder.
Talk to the people involved in the writing of the ELA standards last year, who had all of their efforts erased in a coup d'etat by the State Board, giving their favorite right wing honey from Houston the power to rewrite the standards to their pleasure. Standards for the 18th century was what the SBOE majority wanted.
Talk to an actual professor of literacy education who might talk to you about the content of the NAEP and the TAKS (without a vested interest like people at TEA). Basic, which is compared to meets expectations on the TAKS, is the lowest possible passing level. It is defined as "Partial Mastery." And our state scores don't even measure up to that! In the two higher levels, Proficient and Advanced, our passing numbers drop precipitously.
Talk to teachers about how testing has taking over their teaching of reading (close to 6 weeks of instructional time is lost in most districts to "benchmarking" and practice testing with released TAKS tests). They'll also talk about why students do well on the TAKS (teachers teach to it, just as Ratcliffe says), but not on the NAEP, which has more assessment of higher order reading skills of analysis and critique.
And finally, check out the actual "Nation's Report Card" that will let you know that the NAEP is also administered in 12th grade (it just wasn't a part of the report you got because we don't have an equivalent assessment at 12th grade because our graduation exam is given earlier in high school). http://nces.ed.gov/nationsreportcard/reading/
psymetric
I was very excited to learn about the Texas Tribune yesterday because I saw it as having the potential to exceed the high-quality coverage of issues by the likes of the New York Times and the Washington Post. However, if you limit your coverage of education to quotes from press releases and brief interviews with spokespersons and others, you will not have the impact on the State that you could. Your reporters must have a desire to truly understand what is happening in education and the resources to explore issues in depth. The simplistic understanding of education held by politicians, the media, and consequently by the general public has added more to the problems of schooling than it has helped. This shallow understanding controls the agenda of educational improvement much to the frustration of educators who must deal with the realities of that misunderstanding daily. Real improvement requires reporting that understands the complex and often ambiguous factors that influence learning so schools can be viewed from a more realistic point of view. There must be more investigation and analysis and less easy reporting.
Some thoughts to consider when reading stories like this one.
1. Test of general achievement like the NAEP or the TAKS are less measures of achievement than of developed abilities, especially at the higher grade levels. They reflect the influence of many factors other than recent instruction—socio-economic status, childhood health, support for families in economic distress, the family and community milieu in which children are raised, etc. To lay all responsibility at the doorstep of the schools is wrong because schools do not control all factors affecting student learning. Furthermore, the schools have very little power to affect the scores over a short period of time. Take a look at NAEP scores over the past few years for evidence that No Child Left Behind has had an impact on scores.
Look at the literature on the prediction of state NAEP scores based on such indicators as average income levels, percent of students in poverty, etc. These studies make it clear that factors other than school quality are also important in test scores.
2. With regards to the NAEP’s achievement levels such as "Basic," the Government Accountability Office and others have roundly criticized the achievement levels as being invalid. Given the way the assessments are created, the achievement levels simply reflect cut points on a normal distribution on ability measures. There is no basis for the descriptions of student capabilities the NAEP provides for each level.
3. Spokespersons for agencies (TEA, school districts, etc.) are generally poor sources of information because of the pressure to put a positive spin on the results and because of their lack of understanding of assessment. It is very likely that the TAKS is an easier test than the NAEP and talk about different standards is largely irrelevant because tests such as the TAKS and the NAEP are primarily measures of developed verbal and mathematical abilities not the mastery of standards. When the test is given is irrelevant if one wants to compare Texas students with those in other states because students in other states are also taking the NAEP in January and February. If recent instruction had anything to do with NAEP and TAKS scores, then the difference between the tests is even greater than reported because the Texas students has several months of additional instruction after taking the NAEP.
4. No test or test series is an adequate measure of an educated person. While the scores are not without meaning, their meaning is widely misunderstood, and the focus on accountability measures and the focus on raising scores divert energy and resources that would be better spend in other endeavors such as adopting and implementing a coherent and comprehensive model of instruction.
Abby Rapoport
Thanks for the proof-reading corrections, KaraThrace (or Starbuck?). Thanks for the suggestions everyone. This won't be the last story we have on testing — promise.
KaraThrace
Psymetric, got some questions for you:
As per your # "2. With regards to the NAEP’s achievement levels such as "Basic," the Government Accountability Office and others have roundly criticized the achievement levels as being invalid. Given the way the assessments are created, the achievement levels simply reflect cut points on a normal distribution on ability measures. There is no basis for the descriptions of student capabilities the NAEP provides for each level.'
The GAO knows almost nothing about reading development (in fact I might go as far as saying it knows nothing about reading development). The reading achievement levels are tied to higher level and synthesis questions (which do not even appear on the TAKS until high school). The NAEP uses Item Response Theory to indicate the kinds of questions students answered correctly, not just their overall score. So it is not just a "cut score."
As per your # "3....If recent instruction had anything to do with NAEP and TAKS scores, then the difference between the tests is even greater than reported because the Texas students has several months of additional instruction after taking the NAEP."
Actually the opposite, if the TAKS were administered in January, then (following the scenario above) fewer students would pass. Then the TAKS passing rates would be closer to the NAEP passing rates. Believe me, I'm not defending the TAKS, which has led to the gutting of meaningful reading instruction in this state (well TAKS and C-Scope); I'm just following the logic.
As for #4, I absolutely agree. What this study by NAEP did was show how states, when allowed to set their own standards, tend to set them pretty low, so more students can pass. When given the opportunity to join a national standards group, to develop stronger expectations for Texas students, Texas was one of the two states that opted out. Of course because we think we know better.
Because we lied about our drop-out rates to create the Texas Miracle (Thanks for that Secretary of Education, Rod Paige!).
Because we still insist that a bunch of half-baked crazy dentists know more about defining standards for our students than teachers.
Because we allow the head of our education agency to be a political appointee rather than someone qualified for the job.
Because we rank number one (woohoo) in child homeless, and toward the bottom in literacy.
Let's keep this streak going, Texas!
psymetric
KaraThrace,
Your comment about the test dates is exactly correct. I don’t know what I was thinking.
With regards to the validity of the NAEP achievement levels, I have more faith in the conclusions reached by the GAO than in the pronouncement of the NAEP Governing Board or the Department of Education. In keeping with the spirit of full disclosure, I must say that my opinion of the GAO has been shaped in part by the fact that our younger daughter is an analyst for the GAO, and what she has told me about the procedures they use to produce their reports makes me believe they are highly valid and objective. Furthermore, the achievement levels have been criticized by others who have looked at them closely. In fact, the following text is taken from the 2007 NAEP read report.
“As provided by law, NCES, upon review of congressionally mandated evaluations of NAEP, has determined that achievement levels are to be used on a trial basis and should be interpreted with caution. The NAEP achievement levels have been widely used by national and
state officials.”
How many years does a “trial” last? They are “widely used by national and state officials” because they give the impression that the NAEP is instructionally sensitive and measures more than developed abilities.
The achievement levels are based on cut scores. Item response theory is used to produce a scale score that ranges from 0-500. Both students and items can be placed on that scale, and the distribution of the students on the scale is uniform and like the normal distribution in shape—few at the top and bottom, most near the middle. The achievement levels represent relative position on the scale and are not marked by any discontinuities.
The NAEP Governing Board creates the achievement cut scores following input from “policymakers, educators, and members of the general public.” I do not know exactly how they accomplish this, but the general idea is to present a set of items that span the range of difficulty from 0-500 and have the participants debate where along that distribution cut points should be placed. The 2007 report illustrates this relationship between scale scores and items, but unfortunately, interpretations of what the items are measuring are shown rather than the items themselves. I have always found such interpretations of items to be of questionable validity. The cut scores for the 2007 reading test at grade 4 were as follows:
0-207 Below Basic
208-237 Basic
238-267 Proficient
268-500 Advanced
The achievement levels are nothing more than arbitrary designations for ranges of scores on the underlying scale that are set by the policymakers, educators, and members of the general public. They bear no relationship to any underlying understanding of literacy development and represent, in my view, an attempt to disguise a measure of developed verbal ability into an instructionally relevant accountability measure.
The tests used in accountability systems are appropriate for the uses to which they are put. In my view (and that of others) IRT methods produce instructionally insensitive tests that are saturated with ability. The tendency is reinforced by creating tests of general content (e.g. reading and mathematics) rather than focusing content more closely aligned with instruction. The saturation with ability means that scores on the resulting tests are not easily raised, especially at the higher grade levels where they reflect life experiences over a longer time span. This view is reinforced by NAEP progress in recent years. The average fourth grade reading score for 2007 (221) falls near the top of the Basic achievement level, and has risen by only 6 points since 1998. Six points on a 500-point scale! At this rate of gain, the average score will move into the Proficient achievement range in about 25 years.
The bottom line is that as long as schools and school systems are judged by inadequate measures, they will never be able to demonstrate significant improvement. Unfortunately, business leaders, politicians, and journalists have taken control of the education debate and forced educators into the background, and educational leaders have defensively adopted their rhetoric even as teachers and campus administrators realize the uselessness of most improvement initiatives, or at least the uselessness of judging improvement using such measures.
PS You have a great username. So say we all.