Call for Proposal (Previous applicants need not apply)
Organization/Firm
Final Evaluation of Phase III of the Strengthening Pre-service Teacher Education Programme in Myanmar (STEM)
Terms of Reference
Date Issued: 17th July 2023
Organization: UNESCO – Myanmar Project Office
Unit: Strengthening Pre-service Teacher Education in Myanmar (STEM)
Functional Title: Contract for Services
Tentative Duration: August 2023 to November 2023
Application Deadline: 16th August 2023 at 17:00 Hours (Yangon Time)
- Background
UNESCO’s support to teacher education in Myanmar is delivered through the ‘Strengthening Pre-service Teacher Education in Myanmar’ (STEM) programme. STEM has been in existence for approximately 9 years. Phase I was launched in July 2014 with support from the Government of Australia; the second phase, funded by the Governments of Australia, Finland and the United Kingdom, commenced in January 2017 and came to an end in June 2020. On 01 July 2020, STEM entered Phase III, extending the programme through December 2023, under a multi-partner trust fund (MPTF) with the Government of Finland joining a basket fund with a contribution of EUR 5,000,000. To date, Finland remains the sole contributor to the MPTF, with Phase III likely to be the final period of the programme.
Through STEM, UNESCO’s assistance is generally executed via the following four (4) key areas. According to the approved programme document for STEM Phase III, these outcomes cover the period July 2020 to December 2023.
- Outcome 1 - formulate a policy framework of pre-service teacher education and providing technical advice on developing teacher policies and teacher competency standards framework;
- Outcome 2 - improve the system and operations of pre-service teacher education by restructuring and redesigning the curriculum and building networks amongst them for knowledge and experience sharing;
- Outcome 3 - develop institutional and human resource management capacities of pre-service teacher education; and
- Outcome 4 - mainstream inclusion and equity issues in teacher education.
Since the approval of STEM Phase III’s programme document, the operating environment in Myanmar has changed significantly. The COVID-19 global pandemic contributed to significant learning losses (educational institutions across the country closed for an academic year), while the ongoing political situation has presented an additional layer of challenges to the education system. Consequently, the Results Framework outlined in STEM’s programme document, and its corresponding outcomes, outputs and activities had to be re-programmed to align with the changing operational and implementation environment.
While STEM has contributed significantly to changing the landscape of pre-service teacher education, prior and future achievements have been threatened by unravelling political, social and economic events. With Phase III of STEM ending in December 2023, a final evaluation of the programme is to be conducted in the form of an external evaluation. The evaluation report will be reviewed by the Evaluation Reference Group which will include the Government of Finland, with UNESCO providing the Secretariat, to inform decisions as necessary to adjust or refine project priorities, costed annual work plans and deliverables.
- Purpose and Use of the Final Evaluation
The evaluation will focus on the entire implementation period for Phase III (July 2020 – December 2023). The purpose of the evaluation is to assess the extent to which the outcomes, outputs and targets of STEM Phase III have so far been achieved, and the programme’s overall ability to effectively re-orient programming and operational modalities in the context of the COVID-19 global pandemic and the unravelling political situation in Myanmar.
Specifically, the findings of the external evaluation shall be used to:
- Assess the value and merit of the achievements of Phase III for the overall capacity of the teacher force and project stakeholders;
- Outline the achievements attained during Phase III against the M&E Framework and the extent to which these have so far contributed to Phase III’s expected outcomes to be attained by December 2023;
- Analyze the extent to which the reorientation of the programme is likely to enable a pre-service teacher education reform that is sustainable and how it could evolve to further secure its sustainability in view of the current operating environment;
- Enhance the programme’s efficiency and effectiveness, providing key recommendations to reorient aspects of the programme towards improvements and greater impact in view of the current operating environment and in the event that funding is secured to extend some of the programme’s activities;
- Assess how effectively STEM contributed to gender mainstreaming, equity and social inclusion;
- Identify lessons learned in terms of the UNESCO’s ability to reorient STEM’s programming and operating modalities in the context of rapidly changing needs and operating environment;
- Assess partnerships, governance mechanisms and level of collaboration with other local partners to ensure quality control, the successful reorientation of the programme in view of the changing operational environment, as well as the contribution to strengthening impact and the effectiveness of such collaboration; and
- Document the programme’s progress, highlighting challenges and areas that need to be addressed should the programme receive funding to be extended to Phase IV.
Intended users of the Evaluation
While this final evaluation of Phase III will seek to address a set of relevant evaluation questions, the findings of the evaluation will be used for promoting accountability and organizational learning among the various stakeholders identified below.
- Education administrators[1] and educators[2] involved in teacher education
- The Donor (Finland)
- UNESCO HQ
- UNESCO Asia and Pacific Regional Bureau for Education
- UNESCO Project Office in Myanmar
- STEM Programme team
- Objectives and Scope of Evaluation
The Contractor will perform the tasks under the overall authority of the Director of UNESCO Asia and Pacific Regional Bureau for Education, the direct supervision of the Head of Office of UNESCO Myanmar Project Office, and in close collaboration with the STEM programme team. The evaluation will cover the implementation period July 2020 – December 2023. It shall identify key outputs that have led to the attainment of outcomes (intended and unintended), analyze the enabling factors and obstacles, and scrutinize the challenges encountered and their causes. The intended outcomes of Phase III were set to be attained by 2023; as such, the evaluation should assess the progress as well as the challenges and lessons learnt so far towards achieving the intended outcomes.
As the implementation of STEM Phase III was affected by double crises – the COVID -19 global pandemic and the ongoing political situation – the evaluation should also consider how effectively the outcomes and outputs were reprogrammed and remained relevant towards the reform of teacher education in Myanmar. The evaluation will further assess to what extent the monitoring and evaluation tools are able to effectively identify achievements and challenges, as well as what remedial actions were taken or can possibly be taken to address challenges moving forward. This evaluation is expected to be completed virtually with potential for field visits depending on the situation in Myanmar and/or the ability to secure visas.
The STEM programme would be assessed against the following evaluation questions structured against the OECD-DAC criteria. The evaluation questions will be adapted and finalized in consultation with the ERG during the inception phase of the evaluation.
Effectiveness:
- How effectively did the adjusted/reprogrammed activities and outputs contribute to the achievement of outcomes considering the changed operational and implementation environment? What were the most significant results (outputs and outcomes) of the programme?
- What constraints were encountered in implementing programme activities? How were they addressed and what was the impact on the achievement of the programme’s outcomes?
- What impact did STEM have on the different beneficiaries or target groups, including contribution to gender equality and equity and social inclusion in teacher education?
- How realistic was the programme’s theory of change and M&E Framework in response to the changing operational environment?
Relevance:
- To what extent did STEM Phase III produce results that are relevant to and resulted in benefits to beneficiaries such education administrators and educators?
- To what extent did the programme respond to the needs of the country and as well as the objectives and priorities of the Myanmar Joint Response Framework for Education 2022-2025 (JRF) and the UN’s Socioeconomic Resilience and Response Plan (SERRP) for Myanmar?
- How effective were the reorientation/adaptations to the programme in continuing to meet the most important needs and priorities of pre-service teacher education reform?
- Was the program’s approach for promoting human rights, gender equality and social inclusion relevant for the implementation of the international, regional and national conventions and strategies?
Efficiency:
- To what extent did the management structure, financial and human resources support efficient programme implementation? Do the results justify the resources invested into the programme?
- Was the programme implemented in a timely way? How efficiently were resources (financial and human) reallocated in response to constraints encountered in implementing programme activities?
Impact:
- Have interventions through STEM Phase III contributed or expected to contribute to long lasting changes at the societal, institutional and individual level?
- To what extent did the programme contribute to the broader medium and longer-term responses to teacher education reform in Myanmar?
- To what extent did the interventions of the programme contribute to observed changes? Did all the intended target groups, including the most disadvantaged and vulnerable, benefit from these changes?
- What were the unintended results (positive or negative) of STEM Phase III? How did this affect beneficiaries?
Sustainability:
- To what extent are the benefits/impact of STEM Phase III likely to continue including results in social inclusion and gender equality? Are there already signs of results being sustained? What are the major factors that will affect continuity?
- What evidence can be found of organizational, social and financial structures that will support sustaining the results achieved by STEM?
- What measures have been undertaken to strengthen capacity at the individual and organizational level to sustain results?
- Design and Methodology
The evaluation will follow UN Evaluation Group (UNEG) Evaluation Norms and Standards and incorporate gender and human rights perspectives into the evaluation process and methods. For details see http://www.uneval.org/document/detail/980.
The Contractor is requested to propose a methodological approach but the evaluation should include, but not limited to:
- Desk study of all relevant documents including the progress reports, surveys, documents, guidelines and templates that were published or issued in the course of project implementation. The Myanmar Joint Response Framework for Education 2022-2025, as well as the Final Evaluation of STEM Phase II and the management response to the recommendations should also be considered;
- Interviews of key stakeholders and participants, as well as possible other sources of relevant information, such as consultation through online focus groups, case studies and surveys. Information should be gathered from the following sources (this list is not exhaustive).
- Administrators and Educators including, but not limited to: ethnic and indigenous basic education schools, monastic and faith-based schools, community-based education initiatives and learning spaces.
- Representatives from STEM programme donors.
- Representatives from relevant development partners as well as civil society.
- Representatives of vulnerable and disadvantaged groups.
Contractors are encouraged to use a methodology that combines breadth in data review and analysis with in-depth understanding of key stakeholders’ experiences along the assessment criteria and evaluation questions outlined earlier. Innovative approaches in the presentation of evaluation results are welcome.
Owing to the evolving political situation in Myanmar, a degree of flexibility is required. The methodology developed should take this into consideration. Contractors are required to keep UNESCO informed of risks and mitigation measures in case the ongoing political situation and any other unforeseen crisis may affect evaluation activities.
- Roles and Responsibilities
The evaluation will be managed by UNESCO following the UNESCO Evaluation Manual and carried out by an external evaluation team.. The Final Evaluation Report will be published and made available as a public document, following a careful assessment of situational context in Myanmar. Representatives from UNESCO and the donor government (Finland) will be the reference group for this evaluation, and will be responsible for overseeing and advising on the evaluation process and methodology. All deliverables will be reviewed by representatives of each organization, and they will provide feedback to the evaluators as appropriate.
The evaluation team will abide by the UNEG Ethical Guidelines for Evaluation, available at http://www.uneval.org/document/detail/2866. Responsibilities of the Contractor include but are not limited to:
- Treating documents in a confidential manner;
- Returning all documents to UNESCO;
- Asking for permission before giving any information on the evaluation to third parties;
- Providing all logistics such as office space, telecommunication, printing of documentation, etc.
- Ensuring availability for a briefing via teleconference with representatives of UNESCO and Finland prior to data collection;
- Ensuring availability for team members to partake in in-country mission of sufficient duration for all needed data collection;
- Situating the final evaluation of the programme within a holistic understanding of the context in which it operates;
- Maintaining positive relationships during all stakeholder interactions and take steps to ensure their safety and confidentiality during such interactions;
- Ensuring that the final evaluation and all interactions are grounded in political and cultural sensitivity; and
- Responding to queries from the Reference Group in a timely manner.
Responsibilities of UNESCO include but is not limited to:
- Providing key documents for desk review, as well as during data collection as needed;
- Providing contact information in a secure manner and facilitating participation of stakeholders during data collection;
- Provide general guidelines to ensure the safety and confidentiality of stakeholders during the data collection process;
- Ensuring availability of staff to cooperate with the evaluation team;
- Arranging travel and field visits for the evaluation team (if needed);
- Liaising with Reference Group members to provide updates on evaluation progress;
- Providing updates on political developments within the country that may have implications for the evaluation;
- Liaising with both the evaluation team and Reference Group members to schedule meetings as required; and
- As part of the Reference Group, providing feedback on the inception report, data collection tools, preliminary findings, and draft evaluation report.
Responsibilities of the donor include but is not limited to:
- Ensuring availability of representatives from the Embassy of Finland for both data collection and feedback to evaluation deliverables; and
- As part of the Reference Group, providing feedback on the inception report, data collection tools, preliminary findings, and draft evaluation report.
- Deliverables and Timeline[3]
The evaluation deliverables (Evaluation Inception Report, Evaluation Report) will be developed and quality assessed based on guidance in the UNESCO Evaluation Manual, available at https://unesdoc.unesco.org/ark:/48223/pf0000383948.locale=en
Deliverable 1. A draft inception report which contains a summary of the theory of change of the project and synopsis of the programme’s operational context drawn from the desk study, an evaluation design matrix, a framework for assessing the impact of STEM Phase III, a list of stakeholders to be consulted, and a list of reviewed documents. The evaluation design should contain the proposed data collection methods and data sources to be used for answering each evaluation question and a plan for their validation. The evaluation design should also contain a timeline and key deadlines by 30th August 2023.
Deliverable 2. Inception meeting to discuss the proposed evaluation methodology by 8th September 2023.
Deliverable 3. Revised and final inception report following stakeholder feedback by 14th September 2023.
Deliverable 4. Draft evaluation report of no more than 40 pages, excluding annexes. It should be based on the following structure: Executive Summary, Introduction, Purpose, Methodology, Findings, Conclusion, Lessons Learnt, Recommendations, Annexes (data collection instruments, key documents consulted, Terms of Reference) by 16th October 2023.
Deliverable 5. Debrief meeting/presentation to discuss the preliminary evaluation findings and conclusions by 25th October 2023.
Deliverable 6. Final Evaluation Report by 1st November 2023.
The deliverables must be written in English. The annexes should provide an adequate level of evidence to sustain the findings and recommendations.
In line with UNESCO’s overall gender mainstreaming strategy, the Contractor is expected to integrate a gender perspective in all activities and apply gender analysis and mainstreaming concepts whenever feasible.
- Qualifications for Specialized Knowledge/Experience:
This evaluation should be conducted by an evaluation team. Preference will be given to multicultural evaluation teams with appropriate gender balance and geographic representation. It is mandatory that the team members have no previous involvement in any of the programme activities under review.
Organization
Required qualifications:
- A minimum of 8 years of experience in project/programme evaluation; - A minimum of 3 evaluations and/or projects successfully implemented in support of education in developing countries, including in teacher education; - A minimum of 2 evaluations and/or projects successfully implemented with UNESCO and/or the UN System including World Bank Group; and - Demonstrated experience in evaluation methodologies and techniques, both qualitative and quantitative, and experience in evaluation of capacity development projects.
Desirable qualifications:
- Experience undertaking evaluation projects in Myanmar;
- Experience working in post-conflict and/or conflict-affected contexts;
The personnel: Team Leader
Required qualifications:
- Advanced university degrees in specialized fields of social sciences, humanities, public policy, or related fields;
- A minimum of 7 years of experience in applying qualitative and quantitative evaluation methods;
- Experience as a Team Leader for a minimum of 5 previous project evaluations;
- Experience as part of an evaluation team for a minimum of 2 project evaluations focused on capacity development in education;
- Demonstrated experience with human-rights based approach, including a focus on gender equality; and
- Excellent knowledge (written and spoken) of English language.
Desirable qualifications:
- Experience as part of an evaluation team in evaluating projects with activities taking place in Myanmar;
- Experience working in post-conflict and/or conflict-affected contexts;
- Experience as part of an evaluation team evaluating projects focused on teacher education; and
- Experience as part of an evaluation team evaluating projects implemented by UNESCO and/or the UN System including World Bank Group.
The personnel: Team Members
Required qualifications:
- The total number of team members, including the Team Leader, is at least three. It is not expected that more than four team members are required to complete the assignment;
- University degrees in specialized fields of social sciences, humanities, public policy, or related fields;
- Proficiency in English language (written and spoken);
- At least one team member with a minimum of 3 years of experience in applying qualitative and quantitative evaluation methods, including excellent data analysis skills;
- At least one team member with a minimum of 3 years of experience in gender equality and social inclusion programming, particularly in the area of social development;
- At least one team member with a minimum of 5 years of experience in support of education in developing countries; and
- At least one team member with excellent knowledge (written and spoken) of Myanmar language, with preference for multiple team members with excellent knowledge of Myanmar language.
Desirable qualifications:
- At least 5 years combined experience with the education sector in Myanmar; and
- At least one team member with a minimum of 3 years of experience working in post-conflict and/or conflict-affected contexts.
[1] Those overseeing the day-to-day management and administrative functions in a variety of contexts including, but not limited to: ethnic and indigenous basic education schools, monastic and faith-based schools, community-based education initiatives and learning spaces (run by volunteer/community organizations), and temporary learning centres.
[2] Individuals who are teaching or preparing to teach in a variety of contexts including, but not limited to: ethnic and indigenous basic education schools, monastic and faith-based schools, community-based education initiatives and learning spaces (run by volunteer/community organizations), and temporary learning centres.
[3] Timeframes are subject to change. A more precise timeframe will be discussed before the ToR agreement and contracting.
How to apply
Interested organizations are invited to submit both Technical and Financial Proposals (detailed cost breakdown in USD, e.g. daily rate, number of team members, number of days, using their own template template), accompanied by the undermentioned documents to yangon@unesco.org . The technical and financial proposal must be sent in two separate files and in compliance with instructions in the UNESCO Solicitation Documents.
The Financial Proposal should outline two options – a) one that includes the cost of the evaluation with field visits to and within Myanmar; and b) another that includes the cost of the evaluation without field visits.
- Profile of the organization, including the organization structure, registration and CVs of the team members proposed to work on the evaluation; and
- Evidence of designing and conducting/leading evaluations pertaining to capacity building in education through the submission of two examples of recently completed evaluation reports.
UNESCO places great emphasis on ensuring that the objectives of the work assignment, as described in the Terms of Reference, are met. Accordingly, in evaluating the proposals for the assignment, attention will focus first and foremost on the technical elements. From those proposals deemed suitable in terms of the criteria in the Terms of Reference, UNESCO shall select the proposal that offers the Organisation best value for money.
The deadline for receipt of proposals is 16th August 2023 no later than 5:00pm Myanmar time.
Only short-listed organizations will be contacted. (Previous applicants need not apply.)