Explain It Like I'm Nine: Low Substitute Fill Rates
Can’t find enough substitute teachers to cover teacher absences? Seems...
Over the summer of 2018, the RAND Corporation released a report on the results of the Bill & Melinda Gates Foundation’s Intensive Partnerships for Effective Teaching Initiative. The big question: Did teacher evaluation reforms move the needle on student achievement? In this interview, we speak with researcher, professor and author Dr. James Stronge about his perspective and reaction to the RAND Corporation’s report.
In your school system, are teacher evaluations primarily used to simply gauge performance, or as a means of improving practice? Take our instant poll and see how your organization stacks up to others.
All over the U.S., people are doing amazing things in the world of K-12 education.
JAMES STRONGE: The big factor, the number one factor that’s within our control as educators, is in fact the teacher. We can have two teachers working in classrooms side by side, earning the same salary, using the same whiteboards, ostensibly implementing the same curriculum, and getting remarkably different results.
They’re creatively solving problems, making decisions, finding new ways to meet the needs of students and teachers alike. And we’re talking to them – experts in the field, thought leaders, superintendents, principals, special educators and more.
JAMES STRONGE: Evaluation is not an easy reform. It’s not like buying a new textbook or putting in place a new curriculum, asking people to implement a new teaching standard. It is a long road that we have to stay on.
Every other Friday, we share those conversations here. From Frontline Education, you’re listening to Field Trip.
Here’s the question: the way they’re done today, do teacher evaluations work? That is, do they actually help to advance teaching practice, support teacher improvement, ultimately make a difference in student learning?
To dig into those questions, I sat down with Dr. James Stronge, who has spent years doing research in this area.
JAMES STRONGE: I think probably the best way to start is to say that I’m a teacher, and I’ve always been a teacher, and hope I always am one. I started teaching in middle school and as a professor at the College of William and Mary in Virginia. I teach doctoral students in educational policy planning and leadership.
He’s also president and CEO of his company, Stronge & Associates Educational Consulting.
JAMES STRONGE: We do work in the US and internationally on issues related to teacher effectiveness and leader effectiveness, specifically a lot of work on teacher and principal evaluation, teacher and educator hiring, and professional development work around qualities of effective teachers.
Over a period of about 6 years, the Bill and Melinda Gates Foundation poured hundreds of millions of dollars into something called the Intensive Partnerships for Effective Teaching Initiative. They worked with several school districts, such as Memphis, Pittsburgh, and Hillsborough County, Florida, as well as some charter schools in California.
The goal was to improve teaching and ultimately to impact student achievement and graduation rates. Over the course of the project, the schools and districts involved put together new evaluation systems that included student growth measures and classroom observations. They adjusted professional development based on those evaluation results. They offered performance-based incentives, looked at their hiring practices. But the RAND report basically said, “We don’t think it did all that much. We didn’t see a significant change in student performance and graduation rates.” And we wanted to know what Dr. Stronge thought about all of that.
JAMES STRONGE: I don’t think there’s much in the report to disagree with. They’re reporting the facts and then interpreting what they found.
The place that I would disagree, Ryan, is on the causes. I doubt seriously that the design of the evaluations in those districts was the serious problem. I think instead it had to do with flaws in implementation. There are multiple teacher evaluation systems in place. Hillsborough used one, Memphis used something else and so forth. Any good evaluation system is built on the research of what we know makes a good teacher. Now, it’s divided up differently, I think some systems are more efficient than others. So it’s not the difference so much in the design of what makes a good teacher. I think it has to do with implementation in two ways:
One of the problems that I see in the implementation in the Gates Foundation-funded projects is the emphasis that was placed on how data are collected and interpreted, what data were used. And in many of those districts, there was still a very heavy emphasis on classroom observation. Classroom observation is flawed as well. There’s been a great deal of writing and controversy around using value-added measures, but what people sometimes forget is that observation is flawed.
And so the design, in that respect of not “What is a good teacher?” but “How do you document the work of a good teacher?” is very likely problematic and reflected to some degree in the findings from the RAND Corporation.
The second flaw and way to interpret this report and the implementation by the Gates Foundation is in the fidelity of actual implementation, regardless of how the designs were created. Were principals following the implementation procedures properly? The end result that you can’t argue with is that in the vast majority of cases, teachers are perfect. All teachers receive whatever the highest rating is. If it’s a five point scale, they get a five. If it’s a four point, they get a four.
Ultimately what happens in many schools, then, is that the evaluations get reduced to a one- or two-point scale. The districts think they have a four point summative scale, but they don’t. They never use the bottom two points. And consequently, the only distinction is between “effective” and “highly effective.” That’s a flaw in implementation.
Do you think that that is a result of evaluations simply being time-consuming for administrators to do thoroughly and well? What might be playing into the fact that these teachers are receiving “effective” or “highly effective” ratings almost all the time?
JAMES STRONGE: I’ve talked to a lot of principals about that very question, and there are multiple issues. If you take it from a policy vantage point or researchers’ vantage point, they claim a flawed system. And that’s what the RAND report is saying: essentially this is a flawed effort, it wasn’t worth doing.
If you take it from a policy, public interest standpoint, there is a keen interest in distinguishing “effective” from “less effective.” And I’m not certain that the public would want to give up on that. Think about it for yourself: if you have a son or a daughter in a classroom, do you want that teacher to be good? And what are the mechanisms that we can use to ensure the effectiveness in the classroom? That has to implicate evaluation at some point.
The intent is not off-base at all. If you talk to principals, however, they will tell you that it is indeed time-consuming. The typical span of control for principal and evaluation is around 30 to 1, and that’s a lot of work to do, a lot of people to cover. And what happens if a principal has, out of those 30, five or six or seven who are new teachers? Most evaluation designs will put a premium on additional attention, more intensive support, for novice teachers. And additional observations are required, additional data collection and other aspects of the evaluation. So it becomes overwhelming in terms of actual time. There is some evidence that shows the principals only spend about 13 percent of their time on instructional supervision issues. So 87 percent of the time that they have isn’t even spent on instruction.
Being a principal is a tough job. There are so many things that are required and so they’re pinched in the amount of time that they do have. That’s clearly an issue. Another issue is that when there are ineffective teachers — this is a place that many principals have commented to me when I’ve talked with them — is that if they recommend removing an ineffective teacher, the HR office in their school district will often push back and will say, “You have to give us more data. You have to tell us more. You have to keep that teacher longer.” They go through this extraordinarily challenging process, and in the end have to keep the teacher anyway. I think that’s a flaw in organizational management in school districts where HR offices are risk-averse. They don’t want to be sued, and they require far more than a court would require in order to remove a teacher.
Most states have a pre-tenure period, and even in those instances, principals have told me over and over that HR departments are requiring them to document far more than the statute in the state expects to remove those teachers.
Another issue that principals usually don’t acknowledge, but that is real, is the personal relationship side. I do think there is an issue of a culture of kindness that is intended to be positive, but can be harmful. The more we protect teachers and think that we have zero defects, and “If anybody gets on our bus, they can ride on our bus for 30 years — we’ll keep them there and we’ll support them and we’ll help them grow and improve,” there’s something flawed in that thinking. At some point you have to think, “How much do I help teachers continue to grow and improve when they’re not at the bar of acceptability, when we’re exposing children to those teachers year upon year upon year?” So the human relations side of not wanting to deal with conflict is problematic. There’s a psychological element.
And then finally, if we were to ask districts about the issue, there’s a huge cost and involved in it. There’s a cost of removing an ineffective teacher. I don’t know the cost in various states any longer. I was talking with someone yesterday who suggested that in one of the states in the middle of the country, it was about $300,000 — a unionized state, $300,000 to remove a teacher who isn’t effective.
What we’re missing though, Ryan, is the opportunity cost that we have for exposing kids to ineffective teachers. Now I’m talking on the summative side. Evaluation has to be for 100% of employees, and that means helping everybody grow and improve. What the report does talk to is that teachers are seeing growth in the formative side. I think we’re going to make a mistake if we throw away summative, but I think there absolutely has to be a balance between the formative and the summative purposes with a heavy emphasis on the formative, on helping people grow. But when people aren’t capable, or they will not do the work that’s necessary to be an effective teacher, they shouldn’t be teaching children.
Is the RAND report missing anything important, or not taking something important into consideration in your point, whether that be data or… is there anything that you look at it and say, “Here’s a glaring omission that really wasn’t considered”?
JAMES STRONGE: I think the glaring omission is putting all the weight on design issues and saying, “This simply is not a good investment.” I mean, I can’t argue with the fact that if you ended up with the same results that you started with, it’s not a good investment. So I don’t argue with that at all, but the reasoning for why, I think, may be fallacy.
For example, Dan Goldhaber, who is a professor at the University of Washington and the director of the Center for Education Data & Research, stated about the report, and this is a quote from him, “These findings don’t undermine any of the papers that this initiative was built on. It undermines the notion that we have the political will to do this.” And he’s correct. When we have a system that is well-designed but it is implemented without fidelity, you’re not going to get good results. So the flaw that I see in the practice and that I think is overlooked in the RAND report is not that value-added or multiple data sources or summative evaluation will work. It’s that, just as Dan said, they’re not being implemented properly. There’s something wrong when we end up with 99 percent of our employees being almost perfect, — it’s just not quite honest.
Based on what you know about the work that the Gates Foundation has done in this area, what do you think this effort did well, and what might you have liked to have seen done differently? I guess another way of asking that is, what positive work do you think came from this work by the Gates Foundation, and what can we learn from it?
JAMES STRONGE: I think the Gates-funded projects served as an impetus for the states throughout the US. Many states took on, at a state board of education level or a legislative level, the issue of teacher evaluation in wanting to address teacher effectiveness. I don’t think there was a mean-spirited aspect to this. I don’t think legislatures were out to get teachers. I think they were out to support families and children in having effective teachers, and a driver for that in the view of the legislatures is, or was, that evaluation would be a good mechanism for verifying. They were essentially saying, “Trust, but verify,” in my thinking. And that’s where I think the Gates Foundation work was right on point. It was a clear impetus for those states. What I would like to have seen done differently though, was more thoughtful work.
There was a rush to implementation, a rush to design, to judgment, to implementation. And I think there were problems with that. For example, many of the states mandated that 40 to 50 percent of the evaluations be based on measures of student progress. I absolutely concur that teachers, principals, any educator, should be accountable for their work, and some aspects, some portion of evaluation, should be predicated on not how I do my job, but the results of my job. But the percentages that were selected were arbitrary. They were just political numbers. 40 percent in some states, 50 percent in some states. Arizona comes to mind. It allowed a range of, if I remember, 33 to 50 percent of the evaluation to be based on outcome measures to be decided by the local school district. So what’s the foundation for determining those percentages? It was pretty arbitrary.
And consequently, in more recent years, there’s been huge pushback from the unions. Less so from charter schools it appears, but from unions in public school districts, and for good reason. There was too much political engagement without thoughtful research behind what those percentages should be and how the system should be implemented. So I think that’s something I would have preferred to have seen done differently.
At about the same time Gates was doing this, if we step back just a bit, there was Race to the Top that was being implemented. The US Department of Education was spending hundreds of millions of dollars in selected states, about 10 states initially, to implement various programs. And there was a race to spend the money. And so without that thoughtfulness, we end up at a place where we are today.
We know that the quality of a teacher has an enormous impact on student achievement, but we also know that it’s not the only factor. What percentage of that pie would you attribute to teacher effectiveness?
JAMES STRONGE: If I use the research from John Hattie, a very well respected researcher at the University of Melbourne in Australia who has done meta-analyses on all sorts of educational interventions, the percentage that Hattie would attribute is about 30 percent. About half of the variability is attributable to the child, what the kid brings into the classroom — backgrounds are different, behaviors are different, aspirations at various levels can differ. About 5-10 percent of the variability seems to be attributable to the leadership in the school. And there are many studies that show that range, some going a little bit higher, some up to about 14 percent. But the big factor, the number one factor that’s within our control as educators, is in fact the teacher. We can have two teachers working in classrooms side by side, earning the same salary, using the same whiteboards, ostensibly implementing the same curriculum, and getting remarkably different results.
In a study that I conducted with some colleagues in North Carolina a few years ago with fifth grade teachers, a little over 300 fifth grade teachers, in the study. We found that teachers who could be indeed working side by side, where kids started at the same point statistically at the beginning of the year in both reading and math, by the end of a single year, there could be a 30 percent difference in achievement. That’s a huge difference. That’s a whopping difference. And there are many studies similar to that finding. Teachers do indeed have an have an enormous impact, an impact that last measurably three to five years, in some studies it’s shown to last a lifetime. If a child has an ineffective or a highly effective Algebra teacher at about eighth grade, that child is going to be turned on to math or turned off, and that’s going to implicate what math classes that child takes for the rest of his education career or her education career, and what happens in that individual’s life in the future.
Teachers have an enormous impact. We know that, and we know what good teachers do. The fallacy comes in how do we assess that effectiveness. My worry, Ryan, is that because of reports like the RAND Corporation and pushback that is constantly being felt across all the states, that this initiative will be completely dropped and we won’t return to it for another decade or so, and we’ll realize the mistake we’ve made. There were mistakes made in implementation, mistakes made in rushing to design, but to throw all of that out and to say that, “Well, okay, evaluation is not worth doing. What’s next?” — that’s a huge mistake.
That’s a great segue to my next question. I want to ask you about a different report — also from the RAND Corporation — and it was also released around the same time as the first report we talked about. In this new report, they look at results from a nationwide survey of teachers which took place in October 2016. In it, most teachers — 76 percent, in fact — said that they believed that they are better teachers, that they have improved their teaching, as a direct result of their evaluations. What do you think about this, especially in light of the report that says evaluations weren’t effective?
JAMES STRONGE: I think it says that we need to keep working on it, not throw out evaluation and go back to where we were 10 or 15 years ago. There were flaws in some of the design, but greater flaws in implementation. Those can be fixed if there’s the political will to do so. Evaluation is not an easy reform. It’s not like buying a new textbook or putting in place a new curriculum, asking people to implement a new teaching standard. It is a long road that we have to stay on, and unfortunately, educators and policy makers look too frequently for quick fixes.
Policymakers look for four-year fixes, and educators want to move on to what’s next. And that’s a mistake. In a study that I conducted investigating the highest-performing international school districts or school systems in the world such as Finland, the Netherlands, South Korea, Japan, Canada, one of the elements that rose to the top is that in those high achieving schools, there is a consistency in staying with whatever the initiative is that’s being implemented.
Number one, they limit the number of new reform efforts, the new initiatives that are being rolled out. And number two, they stay with them. They simplify it, and make it work, and stay with it, and continue to work with it.
I don’t think that evaluation is the only a reform that we need by any means. In fact, I have a bit of skepticism about evaluation being the best place to start. But I do think it’s an important reform, and quitting at this point will waste not hundreds of millions of dollars, it will waste billions of dollars. And more importantly, it’s going to impact the lives of kids. If we give up on distinguishing between effective and highly effective and less effective teaching, we’re never going to improve in our schools. We’re going to be exactly where we were in the past and we’re going to remain there in the future, if we give up on that effort.
How should we look at teachers who say, “Yes, I’ve made improvements in my practice,” if there don’t appear to be corresponding gains for students? What do you think is going on in a situation like that?
JAMES STRONGE: That is an age old question that has both a simple and a complex answer, I think. The simple answer is that if children don’t change, nothing changed. There was a school administrator with who was visiting in my office some time back, and he told me that in his school they had conducted 800-something walkthroughs, classroom visits, to that point, and it was in November. What I was thinking when he was telling me is that that doesn’t really matter. So what? Who cares how many walkthroughs have been conducted? Show me the change in teachers — legitimate, real change in teachers. Teacher behavior, how teachers instruct, how teachers plan, how teachers create robust, safe learning environments, how teachers use assessment to drive instruction. Show me the change in teachers and then when teachers change, students change.
The report from the RAND Corporation is self-reported from teachers where teachers are saying, “Yes, it helped me. Yes, I did change.” I would like to go beyond that and document the actual changes that happened in teacher behaviors. When that happens, kids change. Teachers are indeed the lever for reform. Reform doesn’t happen in the White House. It doesn’t happen in the state house. It happens in the schoolhouse, one classroom at a time. So when we can reach that step, we’ll see student change.
The report also said that teachers answered in the survey that it’s more helpful for them to receive feedback from their peers rather than from school leaders. 86 percent said it’s helpful to receive feedback from their peers, while 74 percent said it’s helpful to receive feedback from school leaders. Why do you think that might be? Why the greater desire for peer feedback?
JAMES STRONGE: I think there are multiple elements related to it. I think on the one hand, there is some validity to it. On the other hand, I would offer a cautionary note.
In terms of validity — the report didn’t say teachers, it said teachers, instructional coaches, others like that. If there are people who are closer to the teacher, like an instructional coach, someone who knows good teaching, they’re closer to the classroom, they have more time to invest in the observation and the follow-up support, it makes sense that that would be useful and helpful. Secondly, it may offer an opportunity to create a true learning partnership where that learning coach or that team of teachers works collaboratively to improve their practices. Whereas the principal is dropping in for 10 minutes or 30 minutes and passing some kind of judgment and then moving on. It’s not intentional on the part of the principal, but if principals can only spend 13 percent of their time on instructional supervision, no wonder it isn’t sticky or it isn’t making a greater impact on teachers.
Another reason is that, I think, the teacher to teacher review is very soft, softer-sided. It doesn’t have any evaluative component to it. It’s, “I’m here to watch you and to give you some feedback.” There can be flaws in that, however. If the feedback that I’m getting from my neighboring teacher is from a teacher who is getting comparable student results as I am, who teaches like I do, it’s probably going to be affirmation that what I’m doing is good. Whereas it may not be. And keep in mind, Ryan, what the RAND report is basing their study on are survey data, perceptual data. It’s teachers saying, “Here’s how I think I’ve changed,” or “Here’s how I think I’ve grown.”
There are other ways that we know that are better. There have been various studies that do show us that some sources are far better. For example, very specifically, teacher self-ratings of their performance when they’re correlated with their own students’ reading achievement, especially reading achievement, the correlation is very low. It’s about a 0.2, which explains about between four and five percent of the variability of students’ reading achievement growth.
If it’s been a while since you’ve taken statistics, a correlation looks at the strength of a linear relationship between two variables, and is measured on a scale of -1.0 (perfect negative linear correlation) to 1.0 (perfect positive linear correlation). A coefficient of zero says that the one variable has no linear relationship to the other. Looking at the relationship between teachers’ self-ratings of their performance to their students’ reading achievement, the correlation coefficient of 0.2 is a positive correlation — that is, there is some linear relationship between the two – but just barely.
JAMES STRONGE: That’s not very good. Teachers are much better at self-rating if they’re math teachers. There’s a higher correlation between teachers rating, “Am I good at teaching math?” and how much students are learning in math. The correlations with principal observation ratings are abysmal. So no wonder teachers don’t trust them. And what it means is that however many observations are being conducted, those observation ratings do not relate to whether students are succeeding or not succeeding in reading. It’s not a very useful use of time.
Summative evaluations by principals are a bit higher, but the source that’s really good, over and over, is when students rate teachers. The correlation between student feedback on whether my teacher is effective and student achievement gains in reading has a correlation coefficient, in some studies, of about 0.75. That explains a large percentage of the variability of student achievement. Kids are better, more valid evaluators of teacher effectiveness than teachers are of themselves and, I would conjecture, of teachers watching other teachers or principals rating teachers. The correlation with math achievement is also high, not quite as high as reading, but it’s still very high. Kids know good teaching.
You’ve done a lot of research into teacher effectiveness. You’ve created a rubric, the Stronge Effectiveness Performance Evaluation System. Based on your research and experience, what can you tell us about the role of evaluations in supporting great education, whether that’s teaching or whether that’s other roles within the building, as you’ve mentioned already?
JAMES STRONGE: I alluded earlier, Ryan, to the fact that I think evaluation is important, but I don’t think it’s the end-all and in fact, it isn’t even the place where I would start. So I’m giving you a somewhat different answer.
The first evaluation, I think, is the most important evaluation, and that’s how we hire. If we hire quality people, induction is much more successful, professional development is going to be much easier to implement, and future evaluations will be relatively easy — if we hire quality people. We have some problems there, there is a very thin applicant pool available and if there are no big fish in the applicant pool, you can’t catch them. There are flaws in how teachers are hired. It’s very nonscientific. If there are 100 schools in a school district, they’re often are 100 or more different interview protocols, we aren’t even asking the right questions.
So onboarding the right people is where I would start, and I think that’s a mistake that was made in policy. Put more emphasis at the beginning and the end will be easier. In the end, though, to go back to your question, evaluation must play a role in any profession. There are two hallmarks of a profession. Number one, a profession must have a specialized body of knowledge. People can’t walk in off the street and do what that professional does. And that’s true in teaching. People can’t walk in and just do what a teacher does. You can’t just have an undergraduate degree in history and think you can be an effective history teacher. It doesn’t work that well. It takes about 10 years to reach mastery in a complex profession. So we own that.
The second hallmark of a profession is that it must be able and willing to monitor its own ranks. And when we give up on evaluation, we fail abysmally and we will never, ever be the profession that we hope and claim to be if we cannot assess who is effective and who is less effective. So in terms of the teacher, the principal, the counselor, the superintendent, anyone, I think we must have quality evaluation that’s about growth, about improvement and still about who’s doing good work.
So what are the traps that people tend to fall into most often when evaluating teachers? What are the things that schools and districts tend to do that decrease the effectiveness of this process?
JAMES STRONGE: Number one, I would suggest, is the lack of investment in it. You get what you pay for in life. And if you have one single training session on how to evaluate and you leave it at that, it won’t be long before people are back to their old practices.
There’s something called rater drift that occurs. In the training session there may have been good inter-rater or intra-rater reliability, but when without additional follow-up, regular calibration, people drift back to where they used to be. If we’re going to be effective in evaluation or any big reform, there must be an ongoing investment in it.
Secondly, and these are probably not rank ordered, but a second issue is the political will to distinguish effective teaching and to support it. And that has multiple angles that are problematic. Living with what we have, I’m not convinced that we do have the political will to stay with a tough reforms. Some school districts are, they’re not giving up. Miami-Dade, Florida is one that comes to mind. There are a number of New Jersey school districts, in Wisconsin, and others that I work with that have invested heavily in quality teacher effectiveness initiatives and focus on effectiveness with evaluation being a tool to help with that.
That’s the right way to go. So we can do something about it. We can be effective. A problem, however, on political will that’s outside our control, is the tendency for states to focus on cheap public policy. If we have a shortage of teachers, we’ll have alternate licensure routes to get people into the classroom rather than raising salaries. A recent report showed that starting salaries of teachers in Germany can be higher than the starting salaries of doctors. We don’t do anything near that in America. We believe in cheap public policy. You get what you pay for.
I want to ask that question and flip it on its head. Is there one thing that our listeners, that schools, that districts can do that will help to most maximize the impact that evaluations have on the classroom? Is there any positive action that can be taken right now by any given school system out there?
JAMES STRONGE: Yes, hire and retain the absolute best principals. There’s some evidence that suggests it takes four or five years for a principal to begin to have a real footprint in that school, and if it’s an effective principal, that’s going to be a positive footprint. We’ve known for a long time from research from the Dallas public schools that the quickest way to turn around a school, for good or bad, is to change the principal. Quality principals get quality results. And a quality principal will know good teaching. That person will be a medical practitioner, essentially, be able to diagnose what’s effective and what’s not effective that’s occurring in a classroom. And then be able to prognose and say, “Here’s what you can do to get better and to improve.” That good principle will follow up, give support, not just say, “You need to get better,” but will say precisely, “Here’s how you can get better, and we’re going to be there to support you in getting better.”
And then finally, we’ll follow up to verify that that improvement has occurred. One other thing we know from research on the best principals is that they will not condone poor teaching. They help teachers find another job if they don’t get better. Principals, while their direct influence on student achievement is in the range of five to 10 percent of the total amount, they have a much greater influence. They hire teachers, they support teachers, they develop teachers, they keep teachers. If I could do anything immediately, that would be it, and that’s within our control to a degree. The second thing that would require a policy change is to greatly enrich the applicant pool of quality teachers.
We been speaking with Dr. James Stronge about teacher evaluation and effectiveness. James, thank you for joining us today.
JAMES STRONGE: My pleasure. Thank you.
Did you know new episodes of Field Trip are released every other Friday? Don’t miss a single one – you can subscribe wherever you get your podcasts.
Field Trip is a podcast from Frontline Education. Frontline’s industry-leading software is designed exclusively for the K-12 market. That includes Frontline Professional Growth, a holistic solution to help school and district leaders manage the entire educator growth cycle in one system, including employee evaluations and professional learning, and provide tools for educators to collaborate online. For more information, visit FrontlineEducation.com/FieldTripPodcast.
For Frontline Education, I’m Ryan Estes. Thanks for listening, and have a great day.