Teaching to the Test?

"Teaching to the Test" refers to the practice of teaching in a manner designed to improve test results at the expense of some superior form of learning.  Currently, the phrase is most commonly used as an attack on certain standardized tests.  In New York, for example, students take annual standardized tests in grades 3 through 8. 

If the standardized tests are competency tests, which I think they should be, the risk of "teaching to the test" should be related to the treatment of slower students that are at risk of failing to reach measured competency unless the teaching methods are targeted to the particular testing content.  Then, for these students, the risk is that focusing on their ability to answer some form of basic competency questions is materially worse than educating them in some other manner that won't allow them to best answer these same questions.  I have always been skeptical that the probability and cost of this scenario could outweigh the benefits of basic competency testing.  

To better inform my opinion on this issue, I reviewed the New York 2008 math and ELA tests for grades 3 through 8.  I was surprised to learn that all of the recent tests are well-organized on the state website.  I randomly picked a few pages from each test and compiled them in two files (math and ELA).  To me, the questions seem to be fair and straightforward.  I have difficulty understanding what is meant by "teaching to the test" with these sorts of questions and how it could be damaging to the students.  So, I ask the reader: What are some examples of how someone might "teach to the test" with respect to these particular questions?  How would such a method be harmful to the student?  I would love to hear from teachers and school leaders in New York that are particularly concerned with the "teaching to the test" phenomenon.   

Comments

[Anon]

I am generally pro-testing, but it can be taken too far. One way that schools teach to the test is that they teach only those items that will show up on the test. They sometimes sacrifice skills that are less-often measured by the tests (e.g. critical thinking, problem solving) or key content (in social studies, for instance) that won't show up on any state-sanctioned test.

Some schools also focus so much on the tests that the students get the sense that one learns in order to pass the end-of-year test, which erodes their sense of learning for learning's sake. Again, I'm not so pie-in-the-sky to think that we should only teach the "untestable" - or at least untested - skills, but I also think that schools have reacted to having a single, very short-term, accountability metric in a very predictable way: they have oriented all they do toward the test.

Think about it in relation to your industry: What would happen if all of Wall Street were oriented toward one short-term metric, such as profitability?

I imagine you'd get some banks that chase that metric to the exclusion of long-term thinking and risk management. The pressure to do so is tremendous. But as we found out this year, no one metric can tell you how good a business is, and similarly no one metric can tell you how good a school is.

The consequences of rating ourselves with just one short-term measuring stick has dire consequences regardless of the industry.

[Sam]

Great response, Anon. I would add a few things:
1) A single test measures a finite number of skills, as Anon points out. Anon points out that other skills may get left out if they are not tested. In addition, whole content areas can be left out. In New York, the emphasis on testing in ELA and Math that all other subject areas receive less or no attention.
2) Curious2 seems to suggest that "slower" students cannot grasp higher level concepts until they get the basics. Maybe I'm misreading him, but either way, it's a common argument. It's also been disproven. Students with low-skills can engage in higher level thinking while catching up on basic reading and writing at the same time. In fact, I would argue that it's the best way to keep them engaged.
3) Teaching to the test often leads to "test prep," which amounts to practicing test questions repeatedly. This may translate to marginally higher test scores, but it's uncertain whether it results in students being able to apply the tested skills in a non-testing environment.
4) It comes down to this: what would you want for your own children? Would you argue that because they are not "slower" they do not require high stakes testing, but others do?
And who are the slower students? I think that slower, in this case, is not the issue. When we're talking about reforming schools and high stakes accountability, we're mostly talking about urban public schools where the students are of color and poorer than the general population. Those students are not, by and large, "slower." I've taught two large urban school districts. My students were not slow, and I did not see how--even for those whose academic skills were low--that testing was more beneficial than detrimental.
If testing is not good enough for my (middle class) children, it's not good enough for my students, either.

[fmb]

I think it's incontrovertible that "teaching to the test" is an effect. Consider SAT prep classes -- some of this is drilling on content, but a lot of it is also test-taking strategies. It seems these classes are effective (and certainly it's a big biz), which strongly suggests that test-taking is a trainable skill orthogonal to other kinds of learning.

To the extent that this is so, schools that teach this skill to some degree will be selected for (**).

And, as the previous commenters describe, material that is harder to measure on a test will become relatively deemphasized without some compensating incentive.

So, I think disproving the effect is a losing battle. And that it's critical to design tests carefully to minimize the effect. But, I agree that the magnitude of the effect (for a well-designed test) is greatly exaggerated, and that there's an important tradeoff that justifies some "loss" due to teaching to the test.

**Random idea:

Maybe the tests should have content-less or impossible questions that only measure test-taking skill, then they can make some adjustment?

[Ken]

Thanks for the comments, Anon and Sam.

Anon, both of your concerns make a lot of sense to me. I view the New York State tests as competency exams to help detect school environments that are not meeting minimum standards. I agree that there is a danger that school leaders and teachers only focus on these tests, although I am curious to learn how often that happens in practice. I have visited many charter schools over the years. At schools that get good scores, the schools seem to teach a wide variety of subjects outside of those tested. (I would like to explore more deeply how they teach the tested subjects, though.) For schools that are not getting good scores, their problems have always (so far) seemed to go beyond any issue of teaching to the test. Of course, my experiences are too few and shallow to represent proof of any kind, so your opinions and experiences are very helpful to me.

Sam, my post focuses on "slower" students because these students seem to be the most likely to cause teachers to "teach to the test". If a child could get passing scores on the test easily with a wide variety of curriculum and teaching techniques, the teacher wouldn't have an incentive to teach to the test. I am only defining "slower", in this case, as students that would have trouble passing the test without some particular, potentially sub-optimal approach. In any case, my biggest concern with my (future!) children would be along the lines of some of your concerns: I would hope that their education would allow them to grow well beyond the confines of the test material. However, I would want my children to take standardized tests as one measure of their progress. In fact, I would hope that some of those tests were considerably more challenging than the New York State exams.

[Ken]

Thanks for the comment, fmb.

I agree with your thoughts, although I would make some comments:

1. The SAT is not a competency test. In other words, it is expected that the vast majority of students will not be able to answer many of the questions. I think the optimization for the best system changes when the exam in question is easier, the content is more uncontroversially worth knowing, and the main focus is on "passing".

2. I am sure that some schools are "teaching to the test", but one must ask if this is damaging versus teaching something else (or nothing at all). For students that can easily pass the test, there shouldn't be a strong incentive to "teach to the test". For students that have trouble passing the test, one should ask what the optimal curriculum and teaching techniques are for those students. When the test material involves basic math and reading comprehension questions (like the questions on the NY State exams), I wonder what better curriculum exists. I am not saying such a curriculum doesn't exist, but having reviewed the exams, I am not sure what it might be. Meanwhile, I think standardized subject-matter tests like history tests are more controversial, because it is more questionable what content is most "worth knowing".

[fmb]

Yeah, I basically agree with what you're saying. I'm making an argument about the margin, but if the test is intended to be passed easily by most, then they're not on the margin and there really isn't a material incentive to teach differently.

And, everybody teaches for some objective. For this sort of test and for lower quality teachers, the test might well be a better measurement of the true objective than whatever subjective process would otherwise be used.

The marginal cases arise in (at least) 3 ways:

with (as you say) "slower" students, in which case they may well be harmed by not having a more tailored curriculum (I think this is mostly an empirical question, though);

with poor teachers, in which case perhaps it might be a good idea to at least cover some basic material (until you can replace them);

with poorly designed tests (too hard, too trainable, too unaligned with a school's curriculum, etc). As you say, a history test would likely be more controversial for something like this reason.

Note, though, that a basic competency test that most pass is only useful for distinguishing the low end of the distribution, and won't tell you much about whether a school is adequate, good, or great. And, given the number of poor performing schools, it seems that in practice a lot of students/schools are on these margins where there is an incentive to change passing rates, perhaps by "teaching to the test."

It certainly seems appropriate to acknowledge and be vigilant against the risk that a test might actually incent suboptimal teaching, and demonstrate for any particular test that this risk is both small and justified. I agree that that test seems to be met for the examples you've provided.

[Sam]

Thanks for clarifying, Ken. The way that things are structured in New York and many states right now, the incentive is actually not to focus test-prep on the lowest-performing students. It's to focus on the students who are on the verge of moving up a level. Students can score a 1, 2, 3, or 4, and schools are rewarded for moving students. Therefore, a school would have incentive to focus on a student who scored a high 2 as opposed to the student who scored a 1.

[M]

I'm really late to the game, but one improvement I think NYC has made with the progress report is to diminish the incentive to bump kids up a level, as mentioned by Sam. The city has done this by assigning an exact score, 1.00 to 4.60 (something like that), to each exam, and then comparing that score with a (presumably) equivalently-scored test the following year. If the student receives the same or higher score, they are credited as making 1 year of growth in that subject. A great deal (I think 60% or so) of a school's grade on the progress report comes from the school's performance in attaining a full year of growth for each student.

I don't think this is ideal, mostly because the presumption that the tests are equivalently scored and scaled year to year, test to test, is highly suspect. Nonetheless, what this method accomplishes is ensuring that all students are encouraged to improve to the greatest extent possible. I've heard many coaches in the past say "don't worry about the 3s and 4s, they'll be fine." Not so anymore. A student that received a 4 the previous year must receive a 4 to have made 1 year of progress, just as a student who received a 1 must match or improve that score. There is still an incentive to move as many 1s to 2s and 2s to 3s as possible, but that incentive is tempered by the new system.