Gary Rubinstein

Was thinking the same thing. I think it is: "...correlation coefficient of .35..." Found it following the link: http://garyrubinstein.teachforus.org/2012/02/26/analyzing-released-nyc…

Gary Rubinstein wrote an excellent series of posts debunking the value added metrics http://garyrubinstein.teachforus.org/2012/02/26/analyzing-released-nyc…

The data has proved to be virtually useless in assessing teachers. See http://garyrubinstein.teachforus.org/2012/02/26/analyzing-released-nyc… They're useful for showing students are underperforming. But you can't say anything about teacher quality. In fact I suggest if you took the teachers from the "worst schools" and put them in the "best schools" and vice-versa the results the following year wouldn't look noticeably different. Does that mean that teacher have no impact at all. I don't believe that. But I do think the large gaps in achievement are systemic problems more than the problem of teacher quality at certain schools.

This article reminds me of Bill Gate's emphasis on measuring outcomes - here's a quote from the Gates Foundation 2013 letter [1]: -------------- Begin Gates Quote --------------- Over the holidays I read The Most Powerful Idea in the World, a brilliant chronicle by William Rosen of the many innovations it took to harness steam power. Among the most important were a new way to measure the energy output of engines and a micrometer dubbed the "Lord Chancellor," able to gauge tiny distances. Such measuring tools, Rosen writes, allowed inventors to see if their incremental design changes led to the improvements-higher-quality parts, better performance, and less coal consumption-needed to build better engines. Innovations in steam power demonstrate a larger lesson: Without feedback from precise measurement, Rosen writes, invention is "doomed to be rare and erratic." With it, invention becomes "commonplace." Starting around 1805, the “Lord Chancellor” micrometer, according to author William Rosen, was “an Excalibur of measurement, slaying the dragon of imprecision,” for inventors in the Industrial Revolution. (© Science Museum, London) Of course, the work of our foundation is a world away from the making of steam engines. But in the past year I have been struck again and again by how important measurement is to improving the human condition. You can achieve amazing progress if you set a clear goal and find a measure that will drive progress toward that goal-in a feedback loop…

"In New York City, the value-added score is actually not based on comparing the scores of a group of students from one year to the next, but on comparing the ‘predicted’ scores of a group of students to what those students actually get. The formula to generate this prediction is quite complicated, but the main piece of data it uses is the actual scores that the group of students got in the previous year." http://garyrubinstein.teachforus.org/2012/03/06/analyzing-released-nyc… Some qualitative comments (from the UK) An experienced and resourceful teacher may be allocated more 'problem' students than a relatively new teacher as a matter of policy. Teaching students at the extremes of the ability range is harder than the median as the reasons for (say) severe under achievement can be very diverse (learning difficulties, home situation, undiagnosed medical issues - diabetes in my recent experience with one student - and so on) and each student may need a different approach. I have taught in a few settings over the years. The best teaching happens in my experience in settings where people work together. Performance related pay based on dodgy statistics in an atmosphere of public ridicule does not seem the best way to encourage sharing of practice and teamwork! "Good luck with that" as I believe the late Mr Jobbs used to say to competitors.

Many of the reasons why every value-added scoring system in use is terrible: http://garyrubinstein.teachforus.org/2012/02/26/analyzing-released-nyc…

The title is wrong, data is objective but the models are not and which data is used is not (and she argues both those points). She links to Gary Rubinstein's blog[1] that is mind boggling in its analysis of the results of an actual model of teacher's effectiveness and the resulting data. 1) The model that NYC used to evaluate a teacher's effectiveness is not available. It appears that the input data is not available either, only the output data of the model. That isn't transparency. 2) The output of the model is random. Having a feedback loop on random output data isn't going to improve the model. Maybe Bill Gates' model and data will be better and will be transparent. I'm skeptical. [1] http://garyrubinstein.teachforus.org/2012/02/26/analyzing-released-nyc…

Unless you are comfortable with tweaking opacity, jitter and point size until you get a graph that is representative, don't use scatterplots. A 2d histogram with a decent color scheme is far more reliable and easy. For dense data, scatterplots can obscure more than they show: http://www.chrisstucchio.com/blog/2012/dont_use_scatterplots.html Even an author who was well aware of the problem made the same mistake: http://garyrubinstein.teachforus.org/2013/01/09/the-50-million-dollar-…

A heatmap would be more revealing, but I'd eat my hat if it weren't a standard distribution (and I had a hat). It's nature (but hey, anything's possible). The scales are only slightly skewed, but we're not talking logarithms. The merit of the scales ("value-added") on the other hand, is very questionable. But the relation, whatever it is describing, is pretty weak since the ellipse isn't very irregular (it's only "slightly" irregular). http://www.emeraldinsight.com/content_images/fig/0870280109015.png Think Kepler's laws. Edit: The first post in the series shows a scatter of the direct (non-delta) scores, and it looks like San Fransisco in the morning: http://garyrubinstein.teachforus.org/2012/02/26/analyzing-released-nyc…

I, like many, hope the MET project yields fruit. However, see this post for a strong critique of one of the key claims of the MET study: http://garyrubinstein.teachforus.org/2013/01/09/the-50-million-dollar-… Namely, in the scatterplot correlating predicted effectiveness and actual effectiveness, they pooled teachers into 5% groups. This naturally reduces the variance, yielding a much straighter line of points than would an ordinary scatter plot. This is not, to my knowledge, an accepted or ordinary technique.

If you want to see how basically meaningless these teacher evaluations really are, check out the graphs in these posts: http://garyrubinstein.teachforus.org/2012/02/28/analyzing-released-nyc… The evaluations of the same teachers teaching the same subjects in two different classes are almost completely uncorrelated. The distribution of the scores on the scatterplots look almost random. It's a travesty that perfectly fine teachers are being publicly shamed based on these unreliable metrics.

I am all for edumetrics but there doesn't seem to a way to get a good signal on a general "teaching skills" metric. Does such a metric even make sense? I would assume that a proper metric for a teacher would be dict like {"cares":"5*", "energy":4, "perceptibility":3, "subject_knowledge":5, general_knowledge:3}. Furthermore I am not sure how school boards and schools will use the metrics. Should you fire a teacher because some data fit decided that you are a bad teacher? No way! Anyone who is willing to put in the energy and spend time with kids teaching them stuff should continue to do it. Metrics for self assessment YES, but metrics for firing teachers NO. Also, the whole idea of "Value added" score has been called bullshit upon here https://news.ycombinator.com/item?id=5059737 --> http://garyrubinstein.teachforus.org/2013/01/09/the-50-million-dollar-… . [ quote: ... the correlation is so low that I, and many others who have created similar graphs, concluded that this kind of measurement is far from ready to be used for high-stakes purposes like determining salaries and for laying off senior teachers who are below average on this metric. ] The author basically says that there is no correlation of the "value added" metric that a teacher brings from year to year. This lack of correlation is masked in the report "Measures of Effective Teaching" because "they averaged the predicted and actual scores in five percentile groups. In doing this, they mask a lot of the…

Edit: since ColinWright explicitly asked us to indicate experience with hard statistics: Check the link in my profile. I work with very messy data. I've also found that the more experience people have with something, the less confident they are about their expertise so I won't claim to be NotWrong, but the statistics here really aren't hard. It's linear regression with data that appears, cursorily, to be gaussian, close-ish to iid, etc. The question isn't "did Gary do the math wrong?" it's more "is he interpreting the data reasonably?" I will make a strong claim: If doctor effectiveness were this stable, and it were the only data I had available, I would choose a doctor on this metric, ceteris paribus. I found a scatterplot by the author that uses smaller marks so it looks a lot less blob-ish and more heat-map-ish: http://garyrubinstein.teachforus.org/2012/02/26/analyzing-released-nyc… I'm trying to track down the original data to verify this myself, but a couple of quick points: There definitely seems to be good evidence of a relationship. It might be "weak" but that's not the same thing as "ineffective". Considering the number of data points, it makes it unlikely that the trend occurred by chance. This plot also makes it clear why the averaging made it look so pretty: the average in just about any percentile group in 08-09 matched 09-10 regardless of how finely it's divided (to a point). It's certainly the case that there's a lot of "noise".…

One alternative is to not use these flawed/complicated/expensive metrics. Imagine if I came up with a metric that had an 0.1 correlation coefficient with your performance at your job. If you're a programmer, maybe that'd be something like measuring the lines of code written per day. Would you put much faith in that? Of course not, there would be myriad problems with such a system. Same thing here. The status quo in teaching is to evaluate teachers based on qualitative observations from management, same as it is in most other industries. As the same blogger points out in another post http://garyrubinstein.teachforus.org/2013/01/13/50-million-3-years-no-… "Conspicuously missing from the various weighting schemes they compare is one with 100% classroom observations. As this is what many districts currently do and since this report is supposed to guide those who are designing new systems, wouldn’t it be scientifically necessary to include the existing system as the ‘control’ group? As implementing a change is a costly and difficult process, shouldn’t we know what we could expect to gain over the already existing system?" The status quo also has the advantage of not being ridiculously expensive and complicated. Ask Bill Gates to release the data if you're curious about its performance, but given what I've seen, I'd be shocked if it was terribly different than these complicated new metrics.

Looking at the correlation in students between year to year makes clear what's going on - http://garyrubinstein.teachforus.org/2012/03/06/analyzing-released-nyc… There's such a high correlation between an individual student's score from one year to the next, and the number of data points is so small, that trying to tweeze out the teacher's contribution in a statistically meaningful way is fairly hopeless. What I'd like to see are correlates with a teacher's own scores on the tests of the material they're supposed to be teaching (oh you think they all can get perfect scores easily? Hah!) and whether teachers banned from giving homework do any worse, and whether dumping the enforced curriculum in favor of letting students study at their own pace makes any difference. Given how little what the teachers actually do seems to make, it would be logical to at least dump the things which make school dull and unpleasant.

Interesting, I didn't catch from the article that this three-year bump model had been tested on multiple incidents and found to correlate. It mentions in the third paragraph their methodology used "a pattern that, in Atlanta, indicated cheating in multiple schools". It seems Atlanta was the only place they are claiming there was a correlation, but even of this single data point they offer no specific evidence beyond the red dot in their graph. I did find it interesting that the biggest recent case where there was systematic corruption within a system was in Atlanta, and the newspaper here trying to show that cheating is common throughout the rest of the US and not just something special in Atlanta is Atlanta's leading newspaper. Recent analyses of data on 18,000 teachers in New Yock City has shown that year to year testing results are not as consistent as one would assume. http://news.firedoglake.com/2012/03/07/data-analysis-of-value-added-te… > Gary Rubinstein has been analyzing the data at his site, and he’s found some incredible results. First, he found that there’s almost no correlation among teachers year-over-year in the data. A teacher is likely to be judged as effective one year and ineffective the next. The average change in performance was a relatively large 25 points, and it did not fit with commonly accepted beliefs that teachers improve performance over time, particularly between the first and second…