11.+Evaluating+the+Evaluation


 * Evaluation of the L. B. Primary School evaluation of Guided Reading program**

We would like to thank the staff and students of L. B. Primary School for allowing us to come into the school to perform the evaluation. The evaluation team was told by the Board office staff that the staff at the school may be hostile to our visit. We did find that the stakeholders were suspicious of the evaluation process. We were able to relax the staff by involving them in the process and also bribing them with wine, chocolate, and pedicures. The stakeholders were involved in all parts of the evaluation from the beginning to the end. Some staff members did mention that they were going to Cuba for a vacation and that the evaluation was the last thing on their minds, so perhaps our timing could have been better. In addition, we would have preferred to interview all of the students but, due to time constraints, we compromised by having student focus groups in each grade.

The evaluation team met to reflect on the evaluation process at L. B. Primary School. One of the main factors that influenced the efficacy of the process is that of time. The evaluators felt that insufficient time was allocated to this evaluation and that all parties would have benefited from an extension. The team was effective in that the unique skills and contributions of each person were honoured and instrumental in developing this final report. The evaluation design aligned well with our evaluation questions and the team believes that appropriate data were gathered to provide fully supported and justified answers. Triangulation of data allowed us to develop a rich, broad understanding of the context, with input from each of our stakeholders. The results from our quantitative measures aligned with the results from our qualitative measures, which served to increase confidence in our conclusions. The participatory nature of the endeavour allowed stakeholders to feel more in control and less threatened by the process.

The stakeholders were not surprised by our findings. They needed evidence to take to their board office staff so that they could get the support they needed. The board office staff and the administrator did not understand that the teachers need ongoing in-servicing and support to effectively implement the program. Three months after the report was published, three of the four recommendations have been implemented. The resources have been replaced, a committee to purchase more resources has been set up, and workshops for DRA and DIBELS training have been booked. Guided Reading workshops have yet to be booked. The sexagenarian administrator has taken the entire evaluation to heart and has enrolled in intensive DRA, DIBELS, and Guided Reading Program administration training marathon, to take place for two full months in the summer. He is willing to take on the role of trainer at the school, thus overcoming the difficulty in finding the appropriate trainer. In addition, he, in his spare time, is becoming a certified massage therapist and pedicurist to cater to the whims and desires of his staff: A happy staff is a productive staff, well-able to deliver a Guided Reading Program with fidelity.

Personal Reflections
=Evaluation Reflections by Wanda Quigg=

Oh, the things that I have learned.

I did not realize how complex an evaluation is. There are so many different styles and each have their strengths and limitations. Evaluation is an evolving practice. I never considered how important evaluations were because someone else did the work. I hope they were experts. I relied on the integrity of the group who contracted the evaluators to make the right decision. To create a credible evaluation takes a lot of work.

Some forms of evaluation appeals to me more than others due to the features they have. I particularly like participatory evaluation. This process involves all parties and I consider if someone is going to use the program then it makes sense to be involved in the implementation and evaluation of the program. I have seen school boards where the programs were mandated by the administration. There was very little ‘buy in’ from the staff and lots of grumbling. On the other hand, when the staff members were involved in the decision making, the implementation of the program was smoother.

In my last course, I wrote about Heritage Fairs. (Heritage Fairs are very similar to science fairs but instead of a science project being presented, a project about a person’s heritage or family history is presented.) I noticed that the project’s visual presentation at the territorial fair were very similar to each other. I attributed the similarity of the features of the projects to the Rubric the projects being measure with. I feel that Objective-oriented evaluation was used in evaluating Heritage Fair projects. If the projects met certain ‘objectives’ at the school fair, they went on to the regional fair. The same thing happened again, if the projects met the ‘objectives’ at the regional fair, they went on to the territorial fair. The mandate of the Heritage Fairs is to allow the student to present their research in any manner that they wish yet while I was judging at the Territorial Fair, only 1 out of about 75 projects was not on a presentation board. How to value creativity in a Rubric is something that I have to consider. I have agreed to work on a committee to re-do the rubric.

There are several items I wish to explore more. I definitely want to learn more about DIBELS. I want to spend time examining the DIBELS home page that was created by the University of Oregon. It looks like it is a quick and easy assessment yet so much information can be attained. Another item I wish to expand on is the analysis of the assessments to measure skills after guided reading sessions. Currently, in our region, the teachers measure reading levels and use this information to created reading group. There is so much more information that can be gained from miscue analysis whether it being DRA or running records that we currently don’t utilize. I will have to consider this point further and see what I can do to make the process more than ‘numbers to give to the administration’.

I found it very difficult to synthesis all of the information that was delivered in this course. The speed that the course progressed was overwhelming. There was never any time for me to reflect upon what I was learning. This maybe due to the fact that this course was delivered in a condensed spring session rather that in a regular session.

The situation we were evaluating is hypothetical and I found it difficult to not to be distracted by ‘what ifs’. It might have been easier to remain focused if we set up the criteria for the hypothetical situation before hand. I know that it would have been difficult to anticipate everything that might have happened but I think that we would not have spent so much time going back and forth.

Working on the final project forced me to become more familiar with Skype and to learn new programs such as Google.docs and Wikispaces. Being forced to collaborate on-line caused us to be creative in keeping in touch. I wish to thank my classmate for sharing the software with me and being patient with me while I learned how to use the programs.

I now will read //Consumer Reports// in a different light.

One of the main things that I learned in this course is that program evaluations require a significant amount of time and that an ‘evaluation’ done in 6 weeks is a rushed evaluation. I suppose that our hypothetical evaluation turned out well, however it was what we made it in that all of the data was invented. Did I learn a lot about the program? Perhaps, but GR is a program that I have implemented in my classroom for years. I have a good sense of what it can do, so for the purpose of this class, I would say that creating the logic model was a good way to develop a better understanding of how the various components of a GR program interact and serve to contribute to the final outcome, which ultimately improved my understanding. I think that this beginning stage of the evaluation (the logic model) would be most beneficial to me in undertaking future evaluations. Developing a solid understanding of what a program is intended to do serves as a framework for an entire evaluation – it allows you to see just where in the process things are working well, or, on the other hand, where things are falling apart. In our case, we contrived to make many things fall apart and it was easy to see how a gap in the model would affect things further along in the process.
 * Personal Reflections:**
 * Robin **

I think that having gone through this process has better prepared me for future evaluations, yet so much of the information had to be processed so quickly that I would definitely need to rely on a guide, such as Alkin or Killion. If I had to identify some of the ‘stumbling blocks’ that I encountered, I would say that one was simply dealing with so much new information in such a short period of time and then having to ensure that our group had developed a common understanding of the processes and task demands. Another ‘stumbling block’ was dealing with fabricated data and hypothetical stakeholders. It really is difficult to engage in a participatory evaluation when you have to imagine what your stakeholders might say, might think, and might bring to the process, and then strive for alignment across a number of different measures that were contrived by different people. A final challenge (although I wouldn’t call it a stumbling block) was communication, simply because our ‘evaluation team’ was comprised of members from across the country. We dealt with differing schedules, time zones, and personal demands. In a true evaluation setting I would imagine that the evaluators are all ‘present,’ which would serve to facilitate communication. I am very thankful for Skype technology, as the several Skype meetings that we had were invaluable in developing a common approach and understanding. As well, our group utilized Google docs and Wikispaces to ensure that we had a common work area.

What seemed easier than I anticipated? To tell you the truth, nothing seemed easier than I anticipated, with the exception of creating Box and Whisker plots. I found the entire process to be extremely demanding of my time, energy, and cognitive capacity – not necessarily because of difficult content, but because I needed to develop an understanding of unfamiliar content in a short amount of time, along with the ability to apply that understanding to the hypothetical evaluation. In fact, I found the entire process to be more difficult than I had anticipated.

What lessons or learnings will I take with me? As already noted, evaluations are time consuming and a significant amount of time needs to be devoted to effectively conduct one, especially if it is participatory in nature. As well, how to develop a logic model is something that I will ‘take with me,’ as is an understanding of the importance of clear and frequent communication amongst group members. The understanding that I developed of Box and Whisker plots may also serve useful at some time in the future. Finally, I learned that our quantitative measures, the DRA and DIBELS, both are effective instruments to use in the formative assessment of student literacy skills. I will definitely be looking at them in the future and recommending them to colleagues. ~ Robin

**Georgie’s Program Evaluation Reflection ** Throughout this course I realized the complexities of conducting a program evaluation. There are many factors to consider such as selecting a program to evaluate, involving all stakeholders, determining the design and procedure of the evaluation, formulating evaluable questions and collecting the appropriate data. This is simply the planning stage as once the program has been evaluated interpreting the data to allow stakeholders to use the results and managing the evaluation needs to be considered. I learned that program evaluation is a lengthy and time consuming process which is necessary to determine the merit of a program. Analyzing and interpreting the data collected provides invaluable information in determining the effectiveness of the program. I also learned about the Internal Review Board as I was not aware of the protocol and considerations that are required when human subjects are involved. There is a lot to consider when planning a program evaluation.
 * What have you learned? **

Our hypothetical evaluation was certainly a learning curve for me. I was fortunate to work with a group of knowledgeable people that were able to assist me through this assignment as the topic of Guided Reading was also new to me. I did find that there was much repetition and overlapping in the sections of information that was required for our hypothetical evaluation. Since we all worked on separate pieces within the assignment it required greater effort to align all of our ideas so that we could have a consistent flow throughout the evaluation.
 * How did your hypothetical evaluation turn out? **

I really enjoyed learning about the Guided Reading program as I can see the benefits of using this program to enhance independent reading. Observing and analyzing students’ reading levels, comprehension, fluency, phonological awareness and vocabulary are integral to determining and understanding where the student is at with their reading skills and abilities. I feel that although guided reading is used mostly in elementary schools it would be extremely useful in higher grades. Fountas and Pinnell (1996) wrote that GR is not static and will vary over time as readers grow in knowledge, skill, and experience. Similarly, the materials and instruction provided by teachers will also change depending of the instructional levels of the students. According to**‍‍ ****Fountas and Pinnell **,“it is important for all students to receive guided reading instruction at a level that allows them to process texts successfully with teacher support”. Therefore, it is possible that students in older grades (e.g., grade 4 or 5) may require GR instruction that is similar to the instruction provided in earlier grades. In their continuum of literacy learning, Fountas and Pinnell (1996) identify curriculum goals and characteristics of texts, for grades 3 to 8 that could be used to inform GR instruction with older students. After conducting a hypothetical evaluation on the Guided Reading program I have decided that I would like to learn more about this program and seek professional training so that I may apply the program in my classroom next year. As a middle school teacher Dibels and DRA assessments were all foreign to me (which is too bad because these assessments could be useful in middle school).
 * Did you learn about your topic? **

Learning how to evaluate a program that was unfamiliar to me was difficult. Not only was I trying to keep up with the required readings for the course, I was also constantly reading about Guided Reading. There were times when I felt extremely inadequate as I knew very little about the processes and activities in evaluating programs and I knew even less about the Guided Reading program. I felt that this course was quite demanding as so much information needed to be covered in such a short time frame. The course required a lot of time, effort and commitment which was difficult while trying to work full time, finishing report cards, being a parent/wife and dealing with life’s tragedies all at the same time. It certainly was a stressful time. I did appreciate working on the hypothetical evaluation with group members as I learned so much from my group. I also found that Skyping was an effective way to talk with everyone. I did like working on the project in incremental steps (ie. logic model, gathering data, data analysis etc.) as this provided a way to focus on one piece of the evaluation at a time.
 * <span style="color: black; font-family: 'Times','serif'; font-size: 15px;">What were some of the big stumbling blocks for you? **
 * <span style="color: black; font-family: 'Times New Roman','serif'; font-size: 16px;">What seemed easier than you anticipated? **

<span style="font-family: 'Times New Roman','serif'; font-size: 16px;">I realized how vital evaluating a program is to the success of the program and how important it is to collect data as evidence to students’ learning. Too often teachers continue using a program because other teachers have recommended it or because it is a school initiative without considering the effectiveness of the program or determining whether the program achieved its intended results. Through our hypothetical evaluation I learned that much can be learned from a program that failed to meet the intended results. The negative results are ways to improve the program or learn from the program to determine what can be done differently to produce positive results. <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Sincerely, <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">Georgie
 * <span style="font-family: 'Times New Roman','serif'; font-size: 16px;">What lessons and learnings will you take with you? **

Considering I knew very little about program evaluation before starting this course, I feel as though my learning curve was enormous. I learned that a program evaluation is a well though out, comprehensive process that can provide critical information to the stakeholders. I had never even heard of logic models before and feel clear on their purpose and value. I gained an understanding of how to triangulate data and the importance of analyzing both qualitative and quantitative results. I have learned an enormous amount of information but I am not sure that it is all synthesized and organized in my brain. However, I know where to look in the future to support my new (very green) knowledge in this area. Our hypothetical evaluation turned out fantastic! It was an enormous task to complete and having a clear picture of it from the beginning was overwhelming. However, as the work progressed it also became clearer. Luckily the five of us all brought varying talents to the table. I already had a very strong understanding of guided reading and how a successful program could and should run in an elementary school. I liked having the opportunity to learn more about the DIBELS and recognize its value as an assessment tool. I can see encouraging teachers and schools to use it in my future work. I think assimilating all the new knowledge in such a short amount of time was difficult. However as I already mentioned, if I ever am involved in a program evaluation in the future, I know what resources to access. I admit to finding on line courses more difficult as I am a people person and love the personal interaction. That being said, I thoroughly enjoyed working on line (thank goodness for Skype and email mailboxes that can hold hundreds of emails at time!) with this group. We were a diverse group with varying ideas and levels of expertise and I learned something from everyone. I do think it was challenging to create the evaluation because it was a hypothetical situation. I would like to actually participate in a real program evaluation of guided reading with real stakeholders. Hmmmm..I am not sure anything was easy in this course. It was advantageous to have some of our weekly posts be pieces of the overall assignment. Honestly, I found this course difficult. Not that I couldn’t grasp the concepts but because it was all new information for me and I had little background knowledge to support my new findings. I have learned the value of program evaluations and the detail and attempted precision with which they are carried out. I feel as though I will be more aware of their use in the future – perhaps suggesting one is done or participating in an evaluation of an ongoing program. I feel as though I could make a valuable contribution as part of an evaluation team. I definitely realize that a program evaluation needs to be done in a team. It is a huge process and all team members bring something to the process.
 * Alison’s Reflection**
 * What have you learned?**
 * How did your hypothetical evaluation turn out?**
 * Did you learn about your topic?**
 * What were some of the big stumbling blocks for you?**
 * What seemed easier than anticipated?**
 * What lessons and learnings will you take with you?**


 * Cavelle's Reflections**


 * What have you learned?**

Having already taken two previous research courses, Qualitative and Quantitative, I started this course thinking that I would already have some understanding of research and various research methods, however, that turned out to be a false assumption. While having that background knowledge did give me a better understanding of those two particular research methods I still feel as though this course was a huge learning curve. This course was an eye-opener, into the complexities of conducting a program evaluation. There are a multitude factors to consider such as selecting a program to evaluate, involving all stakeholders, determining the design and procedure of the evaluation, formulating evaluable questions and collecting the appropriate data. This is simply the planning stage as once the program has been evaluated interpreting the data to allow stakeholders to use the results and managing the evaluation needs to be considered. I also learned about the Internal Review Board as I was not aware of the protocol and considerations that are required when human subjects are involved. There is a lot to consider when planning a program evaluation. I had no idea what a logic model was before this course and now after working with an example I feel I have a better insight into their purpose. I feel as though I started this course way behind and perhaps now I’m on a better footing, just as it’s coming to an end.


 * How did your hypothetical evaluation turn out?**

I thank my lucky stars for the wonderful group I had the privilege to be a part of. Our hypothetical evaluation was a huge learning curve and I felt overwhelmed many times by the vastness. Thankfully, our group of five was supportive and we managed to stay on top of deadlines and submissions. I found that there was a lot of repetition and overlapping in the sections of information that was required for our hypothetical evaluation. Everyone in the group worked on separate pieces within the assignment therefore it required greater effort to align all of our ideas so that we could have a consistent flow throughout the evaluation. Through various Skype meetings and hundreds of emails and wiki comment posts, our group supported and encouraged each other with edits, friendly reminders and questions. In the end, I think our evaluation proved successful and I’m very proud of the hard work and effort every one of my group members and myself poured into this program evaluation.


 * Did you learn about your topic?**

I thought I knew a lot about Guided Reading, which is why I was thrilled when our group chose to do a program evaluation on Guided Reading. As a Literacy clinician, I encourage my teachers to engage their students in Guided Reading sessions on a regular basis and I myself conduct various Guided Reading sessions with struggling readers on a daily basis. Our school used the Fountas and Pinnell resource book and utilized the Nelson’s PM Benchmark reading record assessment tool, however, thanks to this project I’ve now been introduced to the DIBELS assessment tool and will be doing a lot of convincing to ensure that my school gets access to this wonderful tool. I had heard tell of DIBELS before this course but had never taken the time to look at it more closely but now I know that when I recommend this tool to my teachers for use next year, they will be thrilled.


 * What were some of the big stumbling blocks for you? **

One of the biggest stumbling blocks for me was the overwhelming amount of information and new knowledge that was covered in such a short amount of time was difficult. As I mention earlier, I started this course feeling as though I was already way behind and I never got over that feeling of running to catch up. Thankfully, my group was so supportive and encouraging, as I tried to be for them, and we managed to successfully work through this program evaluation. On a positive note, I do feel as though if I am ever asked to be a part of a program evaluation, I will at least now be able to make some knowledgeable contributions.


 * What seemed easier than you anticipated? **

In all honesty, nothing was easier than I anticipated. I thought this course was going to be easier than it was, based upon my assumption that having already successfully completed two research courses that this course would be a refresher. I honestly did not think any course could be more challenging than my Quantitative Research course. However, I must say that once the feelings of anxiety passed about this hypothetical program evaluation I did come to appreciate the hypothetical evaluation and the opportunity to work with such a diverse group of people. Thank goodness, for Skype (if your mic worked). I, also, appreciated that this course was built upon completing the project in incremental steps (ie. logic model, gathering data, data analysis etc.) as this provided a way to focus on one piece of the evaluation at a time and receive feedback to alleviate stress about whether or not the group was on the right track in our evaluation of a Guided Reading program. The comments and suggestions from our peers and the professor really helped to ensure our project was as thorough as possible.


 * What lessons and learnings will you take with you?**

I have learned many lessons that I will be taking with me as I complete this course and move on to other courses and as I continue to work in my field of literacy education. I have learned the value of program evaluations and the vast detail and precision with which they require to be carried out with confidence. .I realized how vital evaluating a program is to the success of the program and how important it is to collect data as evidence to students’ learning. Through our hypothetical evaluation, I learned that much can be learned from a program that failed to meet the intended results. That just because a program is failing that does not mean that the program itself is not valid or a good tool, just that the intended implementation and the actual implementation could be out of sync. Also, I’ve learned that I could make a valuable contribution as part of an evaluation team. A program evaluation is a huge process and requires a supportive and invested team where, all team members bring something to the process.