The Growing Practice of Rapid Evaluations in Africa – Insights from Three Countries
In a 2017-19 research study by Twende Mbele to assess the state of M&E culture in five participating countries which included, Benin, South Africa, Uganda, Kenya and Ghana a question regarding the ‘[timeliness of] information provided to decision-makers’ was asked to more than 462 managers.
According to a respondent, information needed to make key decisions does not make it to them on time and when it does, it is not relevant anymore. This suggested that there was a problem with key evidence being available when needed to make decisions. This and other reasons led Twende Mbele to begin an initiative to look at rapid evaluations to plug this gap.
Rapid evaluations are intended to reduce the costs of evaluation projects and the time they take (DPME, 2020). These are evaluations which can produce a result that can feed into policy and practice quickly, but yet is sufficiently robust to provide good guidance for decision-making.
They address the need to quickly assess policy/programme/strategy/function delivery, and establish the main performance data, with main recommendations for improvements (Hercules, 2019). And they also help to understand and learn from what works, what doesn’t, when and for whom.
The last couple of years has seen the advancement of methods for doing Rapid Evaluations, many African countries have taken up the opportunities to conduct this type of evaluation for quicker results. This blog gives a short insight and update of three Twende Mbele country governments in doing the type of evaluation.
At the start of 2019, Twende Mbele designed a Rapid Evaluation Toolkit to provide a framework to think about rapid evaluations, and example questions, indicators and tools to do rapid evaluations. This toolkit is intended for use by officials in evaluations, research and program delivery working in all spheres of government.
In South Africa, the Department of Planning Monitoring and Evaluation (DPME) introduced a Rapid Evaluation Guideline in May 2020. The guideline was developed collaboratively between the DPME, the Western Cape Department of the Premier (DoTP) and the Twende Mbele. The Guideline is an introduction to doing rapid evaluations, either conducted internally or externally. It was designed to provide guidance for implementing rapid evaluations in a government context.
Borrowing from the South African Rapid Evaluation Guideline, Benin adapted the guidelines to its existing national guidelines. A training of National Evaluation System (NES) actors took place in June 2020, with the aim of developing the capacities of national actors in this new form of evaluation for use in the event of an emergency, or as part of a preliminary analysis to help determine priorities, identify emerging problems and trends, and enable decision-making to support adjustments to an intervention. With that said, the Bureau of Evaluation of Public policy and Analysis of Government Action BEPPAAG will be undertaking a Rapid Evaluation of the recent effects of COVID-19 on the informal sector.
In Ghana, the Ministry of Monitoring & Evaluation conducted their first rapid evaluation with IDInsight in 2019. This was the Rapid Evaluation of the One Village One Dam (1V1D) project, an intervention which seeks to increase access to reliable source of water for livestock watering, domestic activities and dry season farming as a means of contributing to poverty eradication and addressing the various forms of inequalities with particular emphasis on rural and deprived communities.
The aim of the Rapid Evaluation was to assess the progress of implementation, emerging outcomes and potential impacts of the 1V1D intervention. It also sought to document emerging issues and challenges that will require immediate attention of the Cabinet, Ministry for Special Development Initiative (MSDI) and other stakeholders.
For critical decision-making, Rapid Evaluation have been used in response to unplanned or planned actions by governments for urgent information needs. Inevitably, there is a trade-off for one to use Rapid Evaluation instead of more rigorous evaluation in the public sector. While there is potential for a greater use of findings when they are timely, a longer evaluation allows for more data points, deeper investigation of the literature, etc.
The Growing Practice of Rapid Evaluations in Africa: Government Experiences (Webinar)
The Growing Practice of Rapid Evaluations in Africa: Government Experiences (Webinar)
Rapid evaluations are intended to reduce the costs of evaluation projects and the time they take (DPME, 2020). These are evaluations which can produce a result that can feed into policy and practice quickly, but yet is sufficiently robust to provide good guidance for decision-making. They address the need to quickly assess policy/programme/strategy/function delivery, and establish the main performance data, with main recommendations for improvements (Hercules, 2019). And they also help to understand and learn from what works, what doesn’t, when and for whom. But are they really the panacea government thinks they are?
In the last couple of years, with the advancement of methods to doing Rapid Evaluations, many African countries have taken up the opportunities to conduct this type of evaluation for quicker results. This webinar will highlights some of the efforts by these governments.
Click the link below to register for this webinar:
Efforts to Improve M&E Culture in African governments
Insights from the Survey
Many countries in Africa are using monitoring and evaluation (M&E) as part of their efforts to improve performance of the public sector (Porter and Goldman, 2013). A key foundational component of this is the idea of an ‘M&E Culture’. Monitoring and evaluation culture is composed of perception, underlying assumptions, beliefs and values, reflected in the degree of support by senior management, people’s behaviour and institutional practices and embedded in policies, guidelines, tools and procedures (Mayne, 2010). The predominant M&E culture drives the extent to which the institutionalisation of national monitoring and evaluation systems is entrenched within and across government ministries, departments, or agencies. This culture may reflect values of compliance, or evidence use for learning and improvement.
Three pioneer countries in establishing government-led national evaluation systems are Uganda, Benin and South Africa. These countries have been working together to share experiences and learn from each other around M&E, a partnership now formalised through Twende Mbele – an African M&E peer-learning partnership. More recently Ghana, Niger and Kenya have joined the partnership.
Twende Mbele partner countries have been attempting to foster a culture of learning and evidence use within their respective country-level governments. A baseline assessment of five Twende Mbele countries assessed the state of M&E culture by seeing how the various M&E systems interact to improve performance and accountability, with a specific focus on policy, approach, concepts, framework and organisational arrangements in the their public sector. Over 26 national departments and ministries participated in the questionnaire with 442 respondents from the 5 countries. Overall, the baseline study found relatively high levels of evidence use, however, this was mostly embedded in a culture of compliance, rather than learning.
A culture of compliance values making sure it is covering the regulations and requirements necessary to do business. In this form of culture, organisation-wide values, norms, and expectations make compliance a way of life. A culture of learning on the other hand goes beyond compliance. In this type of culture, learning is valued and people are encouraged to use evidence to improve performance.
The results of the baseline study showed some strong points and some areas of weakness in terms of an M&E culture operating in the five countries. Although there was significant demand for M&E evidence from ministers and senior managers, in over a 1/3 of the cases findings were concealed, and around 40% of respondents perceive senior management as not championing honesty about performance.
Trends also indicate limited respect for evidence-based decision-making by managers with them not liking certain conclusions which they deem as ‘unhelpful’, and 50% and upwards of manager’s fear admitting mistakes. This does not create a great environment for learning or incentives for improvement in policy or programmes. Respondents also suggested that M&E units are seen as not being influential which may indicate that there is not a generalised, diffused culture of reflection or using evidence in decision-making or for learning.
Addressing some of the barriers to evidence use will take a concerted effort to change the organisational culture toward openness to reflection, evaluative thinking and accountability for performance. Given that systems often reinforce cultural attitudes, countries looking to institutionalise evaluation will need to actively scrutinise how the system interacts with existing, and desired, cultures within government. Increasing funding and capacity for internal evaluation may assist meeting demand for evidence in timely fashion, improve manager’s ability to understand and use findings, and reduce the reliance on external knowledge brokers.
Cultural change takes a long time, and is situated in various systems that both inhibit and foster different types of culture. However, as the five countries continue to overcome some of the barriers to use of M&E – and particularly through addressing the cultural challenges – Twende Mbele expects to see the culture change gradually.
This blog was originally posted on the African Evidence Network (AEN) website. Click on the following link to read the original blog: bit.ly/2Q3zNRq
Are black evaluators breathing?
A screening check on the utility of evaluation approaches under COVID19
Writing this article in a time of a global Covid19 pandemic is amongst the most difficult tasks to do. As a proponent of the Made in Africa Evaluation (MAE), it is an opportune time to reflect on how the monitoring and evaluation landscape of Africa is being shaped. Such reflections come at a time when another heart-breaking moment in black history occurred in the USA with the murder of George Floyd at the hands of white policemen – who are meant to administer safety and security to society. His expression, “I can’t breathe” became a slogan for the black lives matter movement as an expression of the sad state of racial affairs in the world.
It is the global response to the “black lives matter” campaign that African scholars and Evaluators ought to leverage to transform evaluation to a more inclusive and accommodative of African centred approaches. This article argues that African evaluators’ ‘breathing’ is impaired, and that the global pandemic is an opportune moment to check the symptoms of marginalisation, exclusion and lack of opportunities which African evaluators are subjected to.
It is undeniable that for centuries the ‘knee’ of colonisation has been inflicted upon black people – a narrative often continued and reinforced through the ‘development’ sector. This effectively means that their collective breathing patterns have been under siege based on their blackness. It is not surprising that Black researchers/evaluators who wish to contribute to shaping the development and evaluation sectors continue to encounter systematic exclusions. More concerning, there seems to be ignorance and implicit acceptance of this racially bias system which thrives at the exclusion of the people it is reliant upon. We all agree that this exclusion is unfair and unjust, but the question of why this continues remains unanswered.
This blog, therefore, calls for the immediacy in the implementation of a tangible (moves beyond [1]talking left and walking right) transformational agenda in the field of evaluation. This urgent agenda should strive to ensure that the voices, lenses, nuances and ideologies of black researchers/evaluators are firstly recognised, respected and subsequently given space to influence the practice.
Monitoring and evaluation is an important approach and tool for measuring outcomes of development programmes, yet it remains a ‘nice to have’ for the most part. It can play a role of screening ‘symptoms’ of malfunction in development programmes and interventions. It has been bestowed by various experts with the mandate to ascertain what works and does not work including why things do not work in development. Sometimes referred to as the evaluation praxis, it has both the competencies and capabilities to foretell answers to complex problems. As the continent responds to Covid19, there remains a space for African evaluators to measure the effectiveness and efficiency of the programmes and interventions.
As a black evaluation professional located in South Africa, I have been preoccupied with more questions than answers during this pandemic. Since evaluation involves measurement and making a judgement, what is the temperature of this nascent but growing profession in the era of Covid-19? Using the “I cannot breathe” analogy from George Floyd’s last words, is evaluation in South Africa – and across the continent more broadly – breathing?
Mokgophana Ramasobana is an Independent M&E Expert and a South African Monitoring and Evaluation (SAMEA) board member. He writes this article in his personal capacity.
Transforming Evaluations for Africa – Webinar
The call for transformation and decolonization is nothing new in African vocabulary, there’s been calls for decolonization education and healthcare sectors, and the transformation of political and economic participation in terms of more women being represented. The evaluation development space has not been exempted from these calls. Over the years, there’s been growing calls for the transformation of the evaluation landscape with more female representation and the use of more black evaluators in the space. Phrases such as: Made In Africa Evaluation; Indigenous Evaluation; and Decolonizing Evaluations have been touted more and more frequency.
Do they all mean the same thing? If not, then what do they mean? This webinar titled ‘Transforming Evaluations for Africa’ will look to unpack the meanings of phrases such as ‘Made In Africa Evaluation’, ‘Indigenous Evaluation’ and ‘Decolonizing Evaluations’.
Click the following link to register for this webinar: http://bit.do/fGPgc
CreativeBox
