Follow LA Observed on Twitter here
In its recent story “Who’s teaching L.A.s kids” the Los Angeles Times has singled out two teachers as being bad at their jobs, basing the verdict on insufficient evidence. The same fate may be awaiting many more Los Angeles Unified School District teachers.
The public pillorying of Karen Caruso of Third Street Elementary in Hancock Park and John Smith of Broadous Elementary in Pacoima was what first caught my attention when I read the story.
Caruso, Smith and about 6,000 other teachers were evaluated by the Times according to a method called value added. As the Times story said, “In essence, a student's past performance on tests is used to project his or her future results. The difference between the prediction and the student's actual performance after a year is the "value" that the teacher added or subtracted.
“For example, if a third-grade student ranked in the 60th percentile among all district third-graders, he would be expected to rank similarly in fourth grade. If he fell to the 40th percentile, it would suggest that his teacher had not been very effective, at least for him. If he sprang into the 80th percentile, his teacher would appear to have been highly effective.”
Using this analysis, the paper said Smith “ranked among the least effective of the district’s elementary school teachers.” For emphasis, a photo of him in the class room is posted on the Times web site. Caruso, the paper said, “ranked among the bottom 10 per cent of elementary school teachers in boosting students’ test scores.”
These damning conclusions were stated with great certainty. But such certainty was not reflected in a research paper that appears to have been the intellectual backup for the Times project. http://www.latimes.com/teachermethod
In the research paper, Richard Buddin of Rand, who conducted the statistical analysis for the Times, wrote:
“At the outset, we acknowledge several limitations of value-added measures. First, student achievement test are not administered until 2nd grade, so the measures provide no indication of the effectiveness of kindergarten or 1st grade teachers. Second, annual tests are only given in English Language Arts (ELA) and math. These subjects are important and key building blocks for other subjects, but the tests do not provide a comprehensive indication of what students learned or everything that they should know at their grade
level. Third, standardized tests are imperfect measures of learning because students may misunderstand what is expected or because individual students may have test anxiety or other issues on the day of the test. “
Buddin wrote “Some of these problems will ‘average’ out across students in a classroom or school.”
But the process of averaging, along with other statistical aspects of the process is subject to error. As someone who has been involved with political polling for several years, I know that there are errors in surveys in which a great amount of data is collected from many people. The probability of these errors is reported in the poll as “the margin of error.” This is so the reader of the poll knows that a candidate shown to be two points behind may actually be two points ahead if the margin of error is four.
Error is also a factor in value added teacher evaluations. “Policymakers must carefully consider likely system error rates when using value-added estimates to make high-stakes decisions regarding educators.” wrote Peter Z. Schochet and
Hanley S. Chiang of Mathematica Policy Research in a study for the U.S. Education Department’s Institute of Science.
What was the margin of error in the evaluations of teachers Karen Caruso and John Smith? Were they at the bottom? Were they just a couple points away from the next bracket, within the margin of error? This information should have been in the story. It certainly should be included when the Times makes available the names and evaluation scores of the some 6,000 other teachers in the study.
As a matter of fact, the State Board of Education, in its application for Obama administration Race To The Top Funds, made in conjunction with the LAUSD, has proposed making value added just 30 per cent of a teacher evaluation. Ben Austin, a state board member and a leading Los Angeles reformer as head of Parents Revolution, said he "wants this type of information to evaluate my daughters' teachers. Why wouldn't I." But he said assigning the 30 per cent figure to value added as a percentage of teacher evaluation "strikes me as incredibly reasonable."
Yet the way the Times story came out, value added seemed to be given much more weight.
The Times story correctly said that “no one suggests using value-added analysis as the sole measure of a teacher.” So reporters were sent into classrooms to observe the teachers. “The surest sign of a teacher’s effectiveness was the engagement of his or her students—something that often was obvious from the expressions on their faces,” they reported.
I have been in many classrooms as a reporter for the Times and now as a columnist for the Jewish Journal, and I would not judge a teacher’s effectiveness from my interpretation of the expressions on students’ faces based on a few visits to a classroom.
But from this thin evidence and from the value added scores with their chance of error, the Times is delivering verdicts on the effectiveness—and the careers--of thousands of teachers.
