Meta-Analysis

Evaluation Tasks

I found this graphical representation of Evaluation Tasks from Better Evaluation very useful for thinking about the evaluation process.

(Click on the pic for a larger version).

In my experience the “synthesize findings across evaluations”-bit gets neglected. In my work as an evaluator contracted to many corporate donors, I am usually required to submit an evaluation report for use by the client. I often have to sign a confidentiality agreement that prohibits me from doing any formal synthesis and sharing, even if I am doing similar work for different clients. Informally, I do share from my experience, but the communication is based on my anecdotal retellings of evidence that has been integrated in a very patchy manner.I try to push and prod clients into talking to each other about common issues, but this rarely results in a formal synthesis.

It is not always feasible for the clients who commission evaluations to do this kind of synthesis. Their in-house evaluation capacity rarely includes the meta-analysis skill, and even if they contract a consultant to conduct a meta-analysis based on a variety of their own evaluations, there are some problems: Aggregating findings from a range of evaluations that do not pay attention to the possibility that a meta-analysis will be done somewhere in the future, requires a bit of a “fruit-salad approach” where apples and oranges, and even some peas and radishes, are thrown together. Another obvious problem is that donors who do not care to share the good, bad and ugly of their programs with the entire world, would be hesitant to make their evaluations available for a meta-analysis conducted by another donor.

Perhaps we require a “harmonization” effort among the corporate donors working in the same area?

22 Seems to be the Magic Number in Solving Education problems!

In a previous post, I introduced the book by Stuart S. Yeh Entitled “The Cost-Effectiveness of 22 Approaches for Raising Student Achievement”.
Now the World Bank released a report entitled “Making Schools Work – New Evidence on Accountability Reforms” which is based on 22 recent impact evaluations of accountability-focused reforms in 11 developing countries. I wonder why this fascination with the number 22?

In the book (written by Barbara Burns, Deon Filmer and Harry Anthony Patrinos) they investigate strategies to address “service delivery failures” where increased spending does not lead to a concomitant change in education output (completion) or outcomes (learning). The idea is that if people in the schooling system are held accountable, things will improve.

This book focuses specifically on

three key strategies to strengthen accountability relationships in school systems—information for accountability, school-based management, and teacher incentives

and looks into how these can affect school enrolment, completion, and student learning.

Main findings about the three strategies include:

Information for accountability (for example – providing “school report cards”) seems to work, but it isn’t a solution to all the problems. Which information is shared, who it is shared with and how it is shared are important considerations which could help parents, communities and other role players identify where the weaknesses in the system is.

School based management reforms (e.g. implementing effective school governance, and school based management) are effective, but these “reforms need at least five years to bring about fundamental changes at the school level and about eight years to yield significant changes in test scores”

Teacher incentives of two kinds have been investigated: Contract teachers (where teachers are contracted on condition that they deliver certain results), and pay for performance reforms (bonuses from meeting targets) seem to be successful too, but perverse behaviours (Such as gaming, cheating or teaching to the test) are likely to abound and eventually negate the overall success of this strategy.

We’ve seen some progress in this regard in the South African schooling system: School Management and Governance training remains an important component of “whole school” development, and the implementation of the Annual National Assessments (ANA) is likely to evolve into an “information for accountability” initiative. (Also see this article about the ANA’s in the local press). Perhaps its time to take the hand of the labour unions and see how incentivising teachers can be implemented?

22 Approaches for Raising Student Achievement

I’m working my way through a book by Stuart S. Yeh Entitled “The Cost-Effectiveness of 22 Approaches for Raising Student Achievement” (Also available as an ebook http://ow.ly/4R62s or on paper from www.loot.co.za see: http://ow.ly/4RRll)

The book (based on Studies in the States) concludes that:

The review of cost-effectiveness studies suggests that rapid assessment is more cost effective with regard to student achievement than comprehensive school reform (CSR), cross-age tutoring, computer-assisted instruction, a longer school day, increases in teacher education, teacher experience or teacher salaries, summer school, more rigorous math classes, value-added teacher assessment, class size reduction, a 10% increase in per pupil expenditure, full-day kindergarten, Head Start (preschool), high-standards exit exams, National Board for Professional Teaching Standards (NBPTS) certification, higher teacher licensure test scores, high-quality preschool, an additional school year, voucher programs, or charter schools

I find this interesting, because this makes a very compelling argument for using computer based learning in schools. The first chapter of the book presents a nice theoretical overview that indicates how kids become disheartened if they don’t consistently have mastery experiences. If assessment and teaching can be individualized so that each learner progressively improves compared to their previous performance (rather than a comparison with peers) they are likely to feel that they are in control of their learning, and they would be more likely to stay engaged in the learning process. The chapter states:

A theory of learning may be deduced: Individualization of assessment, task difficulty and performance expectations for each student on a daily basis, in combination with performance feedback, autonomy in task execution and an accelerating standard of performance, ensures that students achieve success and feel successful on a daily basis, fostering student engagement, increased effort, and further improvements in achievement in a virtuous cycle.

Software can automate the provision of corrective feedback, and assigning of content (on a daily basis) and this has been shown to have powerful effects on learning and achievement. Yeh reports that in a study of Math Assessment, (involving 1,880 students in grades 2 through 8, 80 classrooms and seven states) they found an effect size of 0.324 Standard deviations over a 7 month period. This is a huge increase in learner performance.

Of course we need to remember that this is based on the American school system that is very different from ours. And note the study is about “Cost Effectiveness”. It is not saying that the other strategies are not effective – In their context the other strategies were effective but at a higher cost than Rapid Assessment

Of course there are a few other assumptions that will have to be checked if a similar intervention is implemented in South Africa:
1) Computer infrastructure, technical support, and teacher’s abilities must be supportive of successful implementation. It is a lot harder to implement an ICT based project than one might imagine at the outset.
2) Rapid assessment cannot replace the role of the teacher – It can help learners improve, but they still have to get quality tuition from a qualified teacher.
3) Learners’ ability to interact with the software must not be blocked by poor reading / language capabilities.

“Development “

Aid on the edge posted a little satirical cartoon that made me reflect on the value of “development” as we do it these days.
See: http://aidontheedge.info/2011/03/23/there-you-go/

But that is only one side of the coin:

A friend once told me that he worked in development in Africa, and admitted to forcing a Western idea of education into a local village many years ago. The contemporary wisdom is that one should rather embrace indigenous knowledge systems and community structures. When he went back many years later, he wanted to “repent” of his youthful folly.

When he got there, there wasn’t much left of the village. Sadly climate change has wreaked havoc and the village’s agricultural lifestyle could not longer be sustained. If it wasn’t for the “out of place” education initiative that my friend started many years earlier, there would have been no livelihood for the villagers. At least the kids who got educated, got out. They were now earning a living in another way, sending back some money.

I’m not suggesting that forcing education was the right thing to do, but we can also not blame the education and local development for climate change that is happening on a global scale and would have had an effect irrespective of whether development came to that corner of Africa or not.