Blog

Today’s article, focused on the five phases of program evaluation, is Part 2 of a seven-part series on evaluating professional development. In the rest of the series, you’ll learn how you can apply these phases to evaluate your own programs.

What is Program Evaluation?

In Part 1, we learned that program evaluation is applying systematic methods to collect, analyze, interpret and communicate data about a program to understand its design, implementation, outcomes, or impacts. In the field, evaluators talk about evaluation as a way to determine the merit, worth and significance of a program. Simply put, it’s about understanding the quality of our professional learning programs, how valuable they are to our schools or districts and how important they are to what we are trying to accomplish with teachers and students.

[ctt template=”9″ link=”5Xvam” via=”no” ]PD program evaluation: using data to understand the quality of our programs, value to our schools and importance to our goals.[/ctt]

Program Evaluation Has 5 Key Phases

Evaluating a program consists of five interdependent phases that don’t necessarily occur in a perfectly linear fashion.

Phase 1: Engagement and Understanding

Program evaluation is most successful when stakeholders are involved and work collaboratively. Who should be involved? Administrators and teachers connected to the program, along with instructors and participants, will likely be called upon to offer or collect data, and should be included in planning the evaluation.

Think of a professional learning program in your school or district. Is there a clearly articulated description of the program? Are there stated and measurable outcomes? Does everyone involved with the program know what participants are expected to learn, how they might change their practice, and what student outcomes are expected as a result? Does everyone agree on these? Don’t worry! It’s quite common to answer “no” to one or all of these questions.

In a future post, you will learn about the importance of program descriptions and logic models. I’ll share how these tools can be easily created to promote shared understanding of our professional learning programs and how this sets us up to conduct high quality evaluation.

Phase 2: Questions

Developing evaluation questions is foundational to effective program evaluation. Evaluation questions form the basis for the evaluation plan and drive data collection.

Conducting evaluation is much like conducting a research study. Every research study starts with one or a few broad questions to investigate. These questions inform how and from whom we collect data. The following are examples of the types of questions we might pursue in evaluating our professional learning programs:

• To what extent is the program changing teacher practice?
• What evidence do we have (if any) of student learning that may be attributable to the program?
• How might the program be improved?

Phase 3: Data Collection

We collect data on professional learning programs to answer our evaluation questions, and all decisions about data collection strategies to use rest squarely on these.

• Most people are familiar with surveys (also called questionnaires; check out my book on designing surveys), interviews, or focus group interviews, but data collection can go far beyond asking people questions.
• Observations of a professional learning program in action can yield important insights into how the program is going and whether or not it appears to be on track to achieving its objectives.
• Classroom observations can help us understand if and how well teachers are implementing new practices, whether there are barriers to implementation, and what might be getting in the way.
• Teachers journaling about their learning, or creating lesson plans or other artifacts, can also demonstrate whether a professional learning program is working well.
• And of course, student data — achievement, attendance, discipline, work samples, etc. — can also serve to help answer the evaluation questions.

Later on in this series we’ll offer a more in-depth look at the advantages and disadvantages of specific data collection strategies, along with ideas for exploring more innovative data sources.

Phase 4: Data Analysis

This is the phase that scares people the most. People often think they need to understand statistics or have advanced spreadsheet skills to do data analysis. They worry when their datasets aren’t perfect or whether they have collected data in a “scientific” enough way. They are concerned about whether their data is reliable and valid, especially if it is qualitative and perceptual data, such as answers to open-ended questions from surveys or interviews.

[ctt template=”9″ link=”jXp38″ via=”no” ]You don’t need a Ph.D. in statistics to understand and analyze data around your professional learning program. [/ctt]

These concerns are understandable, but in truth, there’s no need to get worked up.  In a future post, we will put to rest all of these fears!

Given the types of data we use to evaluate professional learning programs, we rarely need statistics beyond simple frequencies and averages. And datasets are seldom perfect. When we conduct surveys, for example, we find that some people don’t answer some questions. Others misinterpret questions, or it’s clear they make mistakes answering them.

On one feedback form after a very well-received workshop, a participant checked “Strongly disagree” for every statement when it was clear that “strongly agree” was the intended answer. How did I know this? Between the statements were comment boxes filled with glowing praise about how much the participant enjoyed the workshop, valued the materials and loved the instructor. It was clear the person mistook “Strongly disagree” for “Strongly agree” based on the location of those responses on the sheet.

Phase 5: Reporting and Use

Evaluation should be conducted with an emphasis on use. Once we interpret the data and glean insights that can inform future decisions about a program, we need to consider how to report new learning to key stakeholders. The formula for effective reporting includes:

• identifying appropriate audiences for evaluation reports,
• understanding their information needs, and
• knowing how they consume information.

Are you reporting to a Board of Education? A superintendent? A group of administrators and teachers? Do they need all the details, just a few key data points, or a brief summary of results? Knowing our audience and how to engage them informs how we create reports, and reports can come in a wide variety of formats. Here are just a few examples:

• Presentations
• Documents
• Infographics
• Podcasts
• Webpages

Evaluation as an Iterative Process

Earlier, I mentioned that these phases aren’t necessarily linear. In the graphic, you see them as a cycle where Reporting and Use points back to Engagement and Understanding. As we complete an evaluation for a program and make decisions about its future, we may enter another evaluation cycle. Also, as we collect data, we may analyze and report on it even as the evaluation work continues, thus revisiting Phases 3, 4 and 5 multiple times in one evaluation cycle.

Next up in the series

In Part 3, we go deep into ensuring that everyone has a shared understanding of how our professional learning programs are designed to influence change in teaching practice and student learning.

Investing in professional learning for educators comes with the expectation that you’ll be able to evaluate the gains from that investment. But all too often, rolling out a learning plan without an evaluation method means that the second part never happens.

That’s why the Frontline Research & Learning Institute, along with Learning Forward, worked with six districts to examine the best way to measure the impact of professional learning.

Those districts were:

• Boston Public Schools, Massachusetts
• Greece Central School District, New York
• Jenks Public Schools, Oklahoma
  Metro Nashville Public Schools, Tennessee
• Prior Lake Savage Area Schools, Minnesota
• Shaker Heights City Schools, Ohio

Each district brought an existing professional learning program to this small-scale study with the intention of collaboratively determining which changes to the programs would most likely benefit educators at their school. Their five essential findings might surprise you.

1. Plan evaluation as a holistic part of the program

To evaluate effectiveness of professional learning programs on both educators and their students, the program needs to be designed with clear targeted, measurable outcomes and indicators of success. In wrestling with this fact, the districts fell into three buckets of program development:

• Existing programs with evaluable outcomes in place.
• Existing programs that required reworking for more clarity of outcomes.
• In-development programs that didn’t yet have targeted outcomes.

Similar to how educators develop units of learning, planning their final assessments first, these district leaders found it was necessary to do that same. While a “one-off” professional learning event may seem like a good idea, the bigger question is: how does it all fit together?

2. Develop targeted outcomes with existing data sources in mind

In developing their target outcomes, Boston Public School leaders looked to the multiple data sources they already had at their fingertips in the district. They considered which might be useful as indicators of the impact of their new professional learning program. Repurposing data sources in this way can help you more easily and quickly evaluate the program without creating a new data burden (Killion, 2018).

The down side: data sources already in place are approximations of measures of the targeted outcomes of the professional learning program. So, district leaders worked to analyze and interpret the results, then form conclusions about the impact of the program.

Weigh how important exacting data tailor-made for your program is vs. the functional ease of using existing data sources. Consider a combination of the two, if necessary.

3. Consider a systems-approach to better evaluate effectiveness. from its inception.

When professional learning within a district or school lacks coherence — that is, when each event or initiative feels ad hoc or separate — it’s pretty difficult to measure its effects. That’s why Jenks Public Schools took the opportunity to revise their program using a systems-approach. They reworked the planning process to ensure that professional learning met their criteria for quality

Using this planning model, the district and school leaders aligned professional learning with identified needs, provided adequate implementation support, and monitored implementation to increase the likelihood of results.

4. Continually evaluate both new and existing programs

The reasons for evaluating a new program are obvious:

• Determine if it’s worth the investment to continue into a second year
• See how to refine the program
• Incorporate feedback from participants

But what about after the first year or so, when you feel it’s going well?

The districts in the study found it helpful to continue to evaluate existing programs in the following ways:

• Run annual data collection from multiple sources about the program to inform continuous upgrades, even after it’s been refined for a year or so.
• Go a step further to measure the impact of the program on student achievement, connecting the dots between program outcomes and changes in student learning.

Metro Nashville Schools collected data using the Collaborative Inquiry Process in partnership with REL Appalachia, a system for collecting, analyzing, and using a variety of data to improve programs.

These modes of evaluation helped the districts stay freshly engaged with programs, even if they had been running for more than a year or so. Continuous data collection meant they could go a step further in guiding teachers who were implementing learnings, too.

5. Use reliable and flexible systems for data collection and evaluation

Useful evaluations can take time, resources, and effort. Districts with data systems that allow them to gather, track, analyze and access data quickly are able to monitor the program’s success or needed adjustments more easily.

Data systems that generate analytics using multiple types of educator and student data allow district leaders to see the best way to adjust a program more clearly. This ease shifts the focus from collecting data to analyzing it — a much more effective use of time.

In this small-scale study, the six districts looked at their professional learning programs, however established or nascent, to collaboratively examine methods of evaluating those programs. Rolling evaluation into the holistic design process, beginning with targeted outcomes, taking a systems approach, continually evaluating the program, and using a solid data system all felt like the most important pointers to take away from the study to run an impactful professional learning program for educators.

Wouldn’t it be great if we knew when our professional learning programs were successful? What if we knew more than just the fact that teachers liked the presenter, were comfortable in the room or learned something new? Wouldn’t it be better to know that teachers made meaningful changes in teaching practice that resulted in increased student learning?

We can ascertain all of this and more by conducting program evaluation.

Every day we engage in random acts of evaluation – multiple times per day, in fact. When we get dressed in the morning, we implicitly ask ourselves a set of questions and gather data to answer them.

• Will it be warm or cold?
• Will the temperature change throughout the day?
• Will there be precipitation?
• Which clothes do I have that are clean?
• What do I have on my schedule? How should I dress for that?

Of course, getting dressed is pretty low stakes. At worst, we might find ourselves too warm or cold, or under- or overdressed for an occasion. Buying a new car, however, is a higher stakes proposition. We could end up with a lemon that costs us a lot of money, or even worse, is unsafe. When we evaluate, we are more or less systematic about it depending on the context. For the car, we may create a spreadsheet and collect data on different models, their price, performance, safety features and gas mileage. Or, at the very least, we would read up on this information and note it in our heads.

But what about our professional learning programs? What does it mean to evaluate a program?

What is Program Evaluation?

Program evaluation is applying systematic methods to collect, analyze, interpret and communicate data about a program to understand its design, implementation, outcomes or impacts. Simply put, program evaluation is gathering data to understand what’s going on with a program, and then using what we learn to make good decisions.

Program evaluation gives us key insights into important questions we have about our professional learning programs that help inform decisions about them. For example, we may want to know:

• How well does the program work? Is it changing teacher practice?
• Is the program meeting the needs of the participants?
• To what extent has there been progress toward the program’s stated objectives?
• Do we have evidence of student learning attributable to the program?
• How can the program be improved?

Part of the innate beauty of program evaluation lies in its abundant flexibility. First, there are numerous forms and approaches, and second, evaluation can be conducted both before and during a program, as well as after the program ends.

Systematic Methods

What do we mean by systematic methods? Much like a high-quality lesson or unit plan, program evaluation is the result of good thinking and good planning. It’s knowing what we want our programs to accomplish and what types of assessment will help us determine if we are successful. Being systematic means:

• Ensuring that we understand what our programs do and what they are expected to do for both educators and students
• Identifying which questions need to be answered
• Knowing what data we need to collect to answer those questions
• Identifying the primary users of our evaluation results – those who rely on the answers to be able to make good decisions

There are myriad strategies for collecting data. Surveys, interviews or focus group interviews, and observations or walkthroughs are common methods. We can also look at student achievement data, student work samples, lesson plans, teacher journals, logs, video clips, photographs or other artifacts of learning. The data we collect will depend on the questions we ask.

5 Reasons to Evaluate Professional Learning Programs

Learning Forward offers a set of standards, elements essential to educator learning that lead to improved practice and better results for students. The Data Standard in particular calls for professional learning programs to be evaluated using multiple sources of data. While adhering to a set of standards offers justification for action, there are specific advantages of program evaluation that substantiate its need:

1. Evaluating professional learning programs allows leaders to make data-informed decisions about them.When leaders have evaluation results in hand they can determine the best course of action for program improvement. Will the program be expanded, discontinued, or changed?
2. Evaluating professional learning programs allows all stakeholders to know how the program is going.How well is it being implemented? Who is participating? Is it meeting participants’ learning needs? How well is the program aligned to the Every Student Succeeds Act (ESSA) definition of professional learning?
3. Evaluation serves as an early warning system.It allows leaders to peek inside and determine the degree of progress toward expected outcomes. Does it appear that program goals will be achieved? What’s going well? What’s going poorly? Evaluation uncovers problems early on so that they can be corrected before the program ends.
4. Program evaluation helps us understand not only if the program has been successful (however “success” is defined) but also why the program is or is not successful.It allows us to know what factors influence success of the program.
5. Program evaluation allows us to demonstrate a program’s success to key stakeholders such as boards of education and community members, or potential grant funders.Evaluation results allow us to document accomplishments and help substantiate the need for current or increased levels of funding.

All Evaluation is NOT the Same

The word “evaluation” can strike fear into the hearts of teachers and administrators alike. People naturally squirm when they think they are being evaluated. Although personnel or employee evaluation shares some characteristics with program evaluation — such as collecting and analyzing data, using rubrics, assigning a value or score and making recommendations — they serve entirely different purposes.

Program evaluation focuses on program data, not on an individual’s personal performance. The focus of the evaluation is on how the program performs. In education, we take great care not to let program evaluation results influence personnel evaluation. And remember the example about buying a car? That’s product evaluation, and it too shares traits with program evaluation but serves a different purpose.

Are you convinced that program evaluation will help you generate insights that inspire action to improve professional learning in your school or district?

Next up in the series

In Part 2, we’ll take a deeper dive into program evaluation and understand the big picture of how evaluation is conducted, the forms it can take, and how it relates to research.

Tips to Boost Teacher Agency

20 miles east of Portland, Oregon sits Gresham-Barlow School District. With 18 schools, “We’re too small to be big, and too big to be small,” says Sarah Hayden, an instructional coach at the district. Sarah and her colleagues work one-on-one with teachers, but also work at the district level in to provide support where needed.

They wear many hats, and just like many districts, they’re asked to do a lot with limited resources. In response, her team has come up with some creative ways to provide educator-driven, make-an-honest-to-goodness-difference-in-the-classroom professional learning opportunities for teachers.

 [Note: this interview has been edited for brevity and clarity.]

Collaboration Walks

FRONTLINE EDUCATION: We’re here today to talk about something you’re doing at Gresham-Barlow called “Collaboration Walks.” Tell me about that — what are they?

SARAH HAYDEN: Collaboration walks are something that we started in our district about three years ago to promote teacher voice, teacher agency and teacher professional growth. On a given day, we get about ten teachers together and explore different classrooms around the elementary schools in our district. Then, teachers sit together to talk and collaborate with each other about what they’ve seen in the classrooms — how they can take what they’ve learned and internalize it.

FRONTLINE: What led to you starting these? 

SARAH: We wanted our rubric (Charlotte Danielson’s Framework for Teaching) to be a model for professional growth and not just evaluation. We took all the numbers away from the rubric, and we used it to talk about instruction in a way that was meaningful, safe and promoted growth for teachers. We even rebranded [the walks]. Instead of ‘Learning Walks’, we call them ‘Collaboration Walks.’ We talk about how the collective voice of the teachers in the room is what is needed for everyone to grow — this is not something that’s top-down. It’s a collaborative way to talk about teaching.

Want more details? Listen to our full interview with Sarah Hayden about Collaboration Walks at Gresham-Barlow School District.

FRONTLINE: Describe the process — who’s there? Where do you go? What do you do?

SARAH: There are usually about ten teachers who sign up, usually within a day and a half. We meet at one of the schools in the morning, and we talk about our goals for the day: What do we want to get out of today? How are we going to be reflective? How are we going to move forward collectively?

We focus on two or three of the different standards in our rubric and we ask, “What does best practice look and sound like in the classroom? What does best practice surrounding discussion and question techniques look and sound like in the classroom? What does setting purposeful intentions for students look like and sound like in the classroom?” And in a collaborative way, we come up with, “What does best practice with these indicators, these standards, really mean?”

Then we go into the classrooms with this lens in mind. Teachers bring their cameras, they talk with students, they work alongside teachers. We’re in a classroom for anywhere from 15 to 20 minutes, and we observe. Then the most exciting part of what we do, we sit and we talk, and we talk, and we talk about teaching — what I do in my classroom, what you do in your classroom, what we observed in the teacher’s classroom that we saw.

We keep it really safe and non-evaluative. We use sentence stems that just say, “This is what I observed… This is what I saw… This is what I wonder….” Through this process of collaborative discussion come amazing points about how teachers are going to move their practice forward.

FRONTLINE: What is it about the structure of what you’re doing that makes these effective?

SARAH: It is 100% teacher-driven, teacher-centered, and the entire goal is to elevate teacher voice across our system. The caliber of the teachers in our district is amazing, and if you get some like-minded individuals in a room, we can solve the problems of the world. That’s why I think it’s been so successful, because it’s about meeting the teachers where they’re at, and helping them continue on their personal journey.

Whether they’re a first-year teacher or a veteran of 20+ years in our district, every single person in that room can support their colleagues through collaborative conversations.

Reflective Conversations

FRONTLINE: Collaboration walks aren’t the only thing that you’re doing in professional learning, of course. How else are you working to make professional learning more teacher-centered?

SARAH: We’ve taken the idea of collaboration walks and we’re doing what we call “reflective conversations,” where teachers videotape themselves conducting a lesson in their classroom, and then they come to our professional development session with a trusted peer from their school or their grade level, who has also videotaped themselves.

We had one teacher comment, “In all of my 20 years of teaching, I’ve never had professional development as meaningful as what I experienced with my colleague at this professional learning.” — Sarah Hayden

Then we spend some time talking about what a reflective conversation is, and how to support your colleague in a way that promotes their professional growth. The teachers watch the videos alongside each other and use these reflective conversation skills to discuss their practice. We had one teacher comment, “In all of my 20 years of teaching, I’ve never had professional development as meaningful as what I experienced with my colleague at this professional learning.” 

FRONTLINE: What is it about the use of video for these reflective conversations that makes it important?

SARAH: Videotaping yourself as a teacher is absolutely terrifying, which is why we bring that trusted peer in. When you watch yourself teach, you are your own worst critic. And everything that you see, you don’t realize that you do on a day to day basis. Once you get past, “Oh my gosh, I really sound like that?” you see exactly what you’re doing and how your students respond to you. Things that aren’t usually visible in the classroom are very visible when you watch yourself on video.

It’s a chance to go deep into what you’re doing every day, and see how the things you do affect student outcomes. It’s completely and totally career-altering. And the teachers communicated that to us, even after one video.

FRONTLINE: Can you talk more about the training you provide for these reflective conversations?

SARAH: The reflective conversations training is a day long. In the morning, we don’t watch any videos — the trusted peer and the teacher sit with us, and we talk about what a reflective conversation is. “What are ways that you can pose an open question that invites inquiry from your partner?” Because as a teacher, it’s very easy to watch a video alongside a colleague and say, “Well, in my classroom, I…” or “Have you ever tried…?” — which can stifle what that teacher needs to understand about their own practice. So we ask questions in an invitational way, where the trusted peer is asking questions so that the teacher can develop their own understanding.

The trusted peer is never telling, never answering. The trusted peer is facilitating and prompting their peer to think about their teaching in a deeper, different way. The trusted peer is the one who can help you draw out what you need to investigate about your own practice. So it’s way more meaningful when you have someone to support you in that and to ask those questions that you didn’t even know to ask yourself.

FRONTLINE: I would imagine receiving that kind of feedback is both helpful and scary.

SARAH: Exactly. As we’re setting up the day, we talk about what a reflective conversation is, and we say, “It’s rigorous. It’s not mere support group talk.” It’s talking about teaching and giving meaningful feedback. We’ve heard from our teachers that so often in our profession, this is missed. Deep, rich, meaningful feedback is missed by teachers, and they crave it. That’s why they feel so good at the end of the day —they’re finally getting something that’s going to help them, that’s going to take them to the next level.

You’re the trusted peer for your colleague, and then they are the trusted peer for you. So not only are you getting, you’re also giving. Additionally, the teachers have found that being the trusted peer and watching their colleague’s video and asking questions allows them to be reflective of their practice as well.

Inquiry Teams

SARAH: I think one of my favorite things that we’re doing in our district is what we call “inquiry teams.” Inquiry teams are a way for us to allow teachers to experience professional learning completely and totally on their own terms.

In teams, teachers put together a proposal about something that they want to learn about. It could be anything from mindfulness in the classroom to new math strategies for STEM to exploring questions of equity within our school. Then, we put them together with a facilitator and give them time and space to inquire about what they want to learn about in a meaningful way, and then share what they’ve learned with their colleagues across our district.

FRONTLINE: How did they do that? How did they share out those practices?

SARAH: At the end of the inquiry team process, the teams and the facilitators put together a ten-minute presentation about what they learned, and then we have an inquiry celebration where teachers can go and learn from their colleagues in a forum. There’s cake involved, which always is exciting, and then they share out their project and what they’ve learned.

This is our second year of inquiry teams, and last year, some of the presenters said, “You know what? We learned a lot about what doesn’t work through inquiry. We hit some roadblocks, which was completely meaningful for our way of learning. Investigating those things and finding out what doesn’t work was the most beneficial type of learning that we could have done.”

It’s just…it’s amazing. Going over just the inquiry proposals this year was inspiring. What teachers are grappling with, what they want to learn, how they feel that they can move their practice forward, and then seeing how they bring their knowledge through their inquiry team back to their buildings, back to the school district as a whole, is so exciting.