SAMEA Conference 2022 Panel Discussion
On Thursday the 22 September, Twende Mbele hosted a panel discussion title,‘Using experience to adapt a guideline on Rapid Evaluations in the public sector’ at the SAMEA Biannual Conference.
The purpose of this session was to update the existing Twende Mbele Rapid Evaluation Guideline from the lessons learned in the adaption and piloting of rapid evaluations in the partner countries. This session facilitated by Mine Pabari, included presentations by the Chief Director of the DPME, Ms Thokozile Molaiwa who spoke to the lessons emerging from South Africa, and Dr Ayabulela Dlakavu who presented on the lessons emerging from three partner countries.
For a quick read on the findings, click the links below for the respective presentations:
Mrs Thokozile Molaiwa: https://bit.ly/3ruXXXR
Dr Ayabulela Dlakavu: https://bit.ly3STw9YY
DPME Evaluation Design Clinic – 6, 7 September 2022
As part of the National Evaluation System (NES), a system of Evaluation Design Clinic is used to facilitate stakeholder participation in the conceptualisation and design of the evaluations prioritised in the National Evaluation Plan (NEP). The last Design Clinic was held in September 2021 with custodian departments and proposing parties on the evaluations identified for the periods 2021/22 and 2022/23. The Design Clinic facilitated stakeholder engagements on problem analysis, programme theory building or refinements where it existed, development of terms of reference (scope, focus, evaluation purpose, questions and methodology), and identification of key stakeholders for each evaluation. The DPME will be hosting a Design Clinic in 2022 for the NEP evaluations planned for 2023/24 and 2024/25.
The objectives of the evaluation Design Clinic is to ensure that the following are developed and achieved:
- A draft theory of change for the programmes/ policies being evaluated in 2023/24 and 2024/25,
- Elements of a draft TOR including the evaluation purpose and evaluation questions are developed, and
- Workshop participants are introduced to the 1-1.5 pager summary for evaluations included in the National Evaluation Plan.
Below are some blogs written with regards to the design clinic. Click the link below for a quick read:
Formation sur l’évaluation rapide pour les organisations de la société civile, ministère de l’éducation et ministère du développement – 7 et 8 juillet 2022
Les 7 et 8 juillet 2022
Le Bénin a accueilli une formation sur l’évaluation rapide pour les organisations de la société civile, le ministère de l’éducation et le ministère du développement.
Évaluation rapide pour les programmes d’investissement dans les infrastructures du ministère de l’enseignement secondaire – 19, 20, 21 juillet 2022
Date de l’atelier : 19, 20 et 21 juillet 2022
Atelier de 3 jours sur la théorie du changement pour le cadre d’évaluation et l’élaboration de matériel de collecte de données pour l’évaluation rapide des programmes d’investissement dans les infrastructures du ministère de l’enseignement secondaire.
Rapid Evaluations in Uganda: The view of a Team Leader
Introduction:
I was a team leader on two rapid evaluations in Uganda. These evaluations included:
- assessing the extent to which the use of the remitted 20% of national park entry fees has been effective in improving the livelihood of the communities in Local Government districts surrounding the wildlife protected areas;
- exploring the challenges in the Local Government staffing while focusing on the filling of critical positions in the Local Government structures in Uganda.
This blog highlights my experiences of successfully undertaking two rapid evaluations, paying special attention to: benefits of the use of a hybrid/facilitated model for doing rapid evaluation; developing rapid evaluation topics; ensuring buy-in for rapid evaluation and use; and lastly, ensuring the quality assurance of the rapid evaluation in Uganda.
Benefits of the use of a hybrid model for doing rapid evaluations
As outlined in the Rapid Evaluation Guidelines (add hyperlink here) there are a few options for governments wanting to start doing rapid evaluations. In Uganda’s case, they have moderate team capacity and wanted to use an external facilitator to assist the internal team with the technical aspects of the evaluations. This is what we call a ‘hybrid model’.
There have notable benefits along the process of using this capacity develop approach to undertaking rapid evaluations. Some of these include;
i) a mix of ideas and expertise from the mainstream government technical teams and the independent consultant, which is a good practice in evaluations;
ii) a quick consensus on the clarity of the study scope, methodologies and expected evaluation results;
iii) knowledge sharing and transfer is quite evident;
iv) an increasing impetus for buy-in, internal clearances, uptake of the evaluation findings to inform policy change as well as improve implementation of government programmes/projects.
Developing rapid evaluation topic
Developing rapid evaluation topics and questions requires demand-driven approaches. In most cases, it begins with identification of the knowledge gap/evaluation problem that seeks for quick solutions to better improve service delivery. These gaps are often identified through quarterly, semi-annual and annual reviews of various government programmes and projects, or from Cabinet and Ministers. The Office of the Prime Minister is then tasked with the process of drafting the terms of reference (ToR), securing funding, setting-up internal and/or external evaluation teams including the quality assurance procedures.
In this case, the internal team had developed selected the evaluation topics through consultation with relevant departments and ministers, and designed the evaluations during a Twende Mbele-led training on how to do rapid evaluations. As the team leader, my job was to ensure the design was feasible and rigorous, and that it could be done within the desired timelines.
Ensuring rapid evaluation buy-in and use
Throughout the rapid evaluation process, it is absolutely necessary that the commissioners seek clearance from both the technical and policy organs of government. All the processes from evaluation design to dissemination should include participation of the relevant stakeholders/consumers of the evaluation findings. It is important that several forms of disseminations are conducted such as reports, policy briefs, press conferences, symposia, website blogs, etc.
There should be follow-up meetings after the rapid evaluation reports disseminations to ensure that there is not only uptake but also use of the evaluation results to improve service delivery. As a team leader, I was responsible for ensuring there is a plan for dissemination and for quality assuring some of the written outputs done by the team.
Quality assurance
Ensuring high quality rapid evaluations is about managing the risks and errors along the processes. A clear multi-sectoral quality assurance framework was utilised to keep track of the evaluation processes. In Uganda, the quality assurance mechanisms are premised on the national M&E strategy framework of a country’s M&E eco-system. Within a quality assurance framework are principles and standards to follow in conducting rapid evaluations. The rapid evaluations in Uganda are effectively coordinated by the national M&E multi-sectoral working group guided by the national M&E strategy. In some cases, reference groups are set up to provide unique technical input to enrich the evaluation processes and findings.
Conclusion
As we can see, there are many elements that come together to ensure the success of undertaking a rapid evaluation – many of which are not dissimilar to traditional evaluations. In this blog I have highlighted some of these elements which was critical in the process of Uganda successfully doing two rapid evaluations utilising a facilitated/hybrid model.
CreativeBox
