.jpeg)
Global Proficiency Framework
The UK Foreign, Commonwealth & Development Office (FCDO) and the World Bank contracted with the American Institutes for Research (AIR) under the Data and Research in Education (DARE) project to implement the policy linking method with the Pakistan Institute of Education’s National Assessment Wing (PIE/NAW) in Islamabad. There were two objectives: 1) build the capacity of PIE/NAW officials in policy linking, and 2) set global benchmarks for linking the National Achievement Test (NAT) with the Global Proficiency Framework (GPF).1 AIR subcontracted to the Society for the Advancement of Education (SAHE) for the coordination and logistics of the capacity building training (“training”) and policy linking workshops (“workshops”). FCDO, World Bank, PIE/NAW, AIR, and SAHE (“team”) collaborated on the activities.
Background
In 2017, the United Nations (UN) defined Sustainable Development Goal (SDG) Indicator 4.1.1 as the proportion of students reaching global minimum proficiency in reading and mathematics. The UNESCO Institute of Statistics (UIS) was designated as the UN agency responsible for reporting on Indicator 4.1.1. Between 2017 and 2021, donor agencies including UNESCO, the World Bank, UNICEF, the US Agency for International Development (USAID), FCDO, and the Gates Foundation funded the development of an operational method – called policy linking – for providing data on Indicator 4.1.1. Policy linking allows countries to link their national assessments to a common measurement scale through global benchmarks so that UIS can compare, aggregate, and track assessment results on a global basis. In addition, countries can use policy linking for their internal tracking of assessment results in relation to the common scale.
Implementation
The AIR facilitators provided a three-day training to the PIE/NAW officials on the background to policy linking and the three tasks: 1) checking the alignment of the 2019 NAT booklets with the GPF using a standardized method (Frisbie, 2003); 2) matching the 2019 NAT items with the GPF levels and descriptors; and 3) setting benchmarks on the 2019 NAT booklets using a standardized method (Angoff, 1971). During two five-day workshops, the panelists set three global benchmarks for Global Minimum Proficiency (GMP) on each of the 2019 NAT booklets: Partially Meets GMP, Meets GMP, and Exceeds GMP. Whether a student score reaches or surpasses the benchmark determines whether the score is placed in the level.
Results
The benchmarks resulted in score ranges for the four GPLs for each of the grades and subjects. Note that the maximum score for Grade 4 English was 36 points, Grade 8 mathematics was 45 points, and Grade 8 Sindhi and Grade 8 Urdu were 50 points (Exhibit 1).
When the final benchmarks were applied to the datasets, the totals meeting or exceeding GMP were 30% in English, 29% in mathematics, 47% in Sindhi, and 34% in Urdu.
The reliability statistics for the benchmarking were generally above acceptable thresholds. The workshop evaluation by the panelists showed that they were comfortable with the process and their benchmarks. They considered that the organization of the workshop was successful
Conclusions
AIR drew the following conclusions from the workshop results:
• All four grades and subjects of the 2019 NAT were judged as aligned with the GPF, with Grade 4 English, Grade 8 Sindhi, and Grade 8 Urdu as strongly aligned.
• About three out of ten students were proficient in Grade 4 English (30%), Grade 8 mathematics (29%), and Grade 8 Urdu (34%). Almost five out of ten students were proficient in Grade 8 Sindhi (47%).
• The benchmarking reliability indicators, i.e., the standard errors (SEs) and the inter-rater consistencies, were acceptable.
• The workshop evaluation showed that the panelists were satisfied with the training and benchmarking, and they viewed the workshops as well-organized and successful.
Policy Recommendations
AIR has the following policy recommendations from the training and workshop:
• Continue with policy linking for the other key grades and subjects of the NAT, e.g., Grade 4 mathematics, Sindhi, and Urdu and Grade 8 English, when data are available.
• Use the reading and mathematics content standards of the GPF as a complement to the new 2020 national curriculum to help increase the alignment of the NAT.
• Use the reading and mathematics performance standards of the GPF as a complement to the 2020 national curriculum in the test specifications, item development, and test construction for the NAT.
• Apply the global benchmarks to future NATs for tracking student performance over time as an indicator of improvement in students achieving global minimum proficiency.
Lessons Learned
AIR has the following lessons learned from the training and workshop:
• The training was highly valuable in orienting the PIE/NAW officials to the policy linking method and global benchmarking, and a lesson learned is to continue with capacity building training for officials in future workshops as PIE/NAW continues to reorganize.
• The gathering of the participants in a central location was critical in encouraging the interaction needed for mutual understanding of the process, and a lesson learned is to ensure in-person training and workshops in the future, with in-person lead facilitation.
• The interpretation of key parts of the workshop into Urdu and Sindhi increased panelist understanding, and a lesson learned is to conduct more sessions in national languages.
• The interaction between PIE/NAW, donor agencies, AIR, and SAHE was highly useful, and a lesson learned is to continue with joint support for future training and workshops.
Training and Workshop Implementation
Methodology
Policy linking with global benchmarking is illustrated in the three figures below. The first figure shows the GPF’s common global reporting scale with four GPLs and GPDs. The red arrow represents the scale from lowest to highest achievement. The top three levels have descriptors, with the lowest level described as knowledge and skill below partially meets (Exhibit 10).
Benchmarking is conducted for each assessment based the descriptions of GMP for the specific grade and subject in the GPF. Three global benchmarks are set on each assessment: Partially Meets GMP, Meets GMP, and Exceeds GMP (Exhibit 11). Reaching or surpassing the benchmark determines whether a score is placed in the level.
The global meets benchmark (in red) is the most important since it separates does not meet from meets GMP. When reporting based on two levels, the below partially meets and partially meets levels are combined into the does not meet level, while the meets and exceeds levels are combined into the meets level. In the example, 55% (20% + 35%) of student scores were in the does not meet level and 45% (30% + 15%) of scores were in the meets level.
Conclusions
AIR drew the following conclusions from the workshop results:
• Three of the four subjects of the 2019 NAT were strongly aligned with the GPF: Grade 4 English, Grade 8 Sindhi, and Grade 8 Urdu. Grade 8 mathematics was strongly aligned for depth and but only minimally aligned for breadth, which was due to the large number of subconstructs in the GPF. • About three out of ten students were proficient in Grade 4 English (30%), Grade 8 mathematics (29%), and Grade 8 Urdu (34%). Almost five out of ten students were proficient in Grade 8 Sindhi (47%).
• The benchmark reliability indicators were generally acceptable for all grades and subjects. The standard errors (SE) of the benchmarks were acceptable. This demonstrated the overall accuracy of the benchmarks. The inter-rater consistencies for MC in all subjects and for CR items in mathematics were acceptable. This showed that the panelists achieved an overall consensus in their item ratings.
• The workshop evaluation showed that the panelists were satisfied with the training and comfortable with their final benchmarks, and they viewed the workshops as well-organized and successful.
Recommendations
AIR has the following recommendations from the training and workshop:
• Continue with the reorganization and reinvigoration of the PIE/NAW so that the next iterations of the NAT can be conducted, analyzed, reported, disseminated, and used at an international standard of quality.
• Use the reading and mathematics content standards of the GPF as a complement to the 2020 national curriculum to increase the alignment of the NAT with global standards.
• Use the reading and mathematics performance standards of the GPF in the test specifications, item development, and test construction for the next iterations of the NAT to ensure that the assessments are measuring learning at the right levels.
• Apply the global benchmarks to future NATs for tracking student performance over time as an indicator of improvement in students achieving global minimum proficiency and for monitoring system performance in relation to GMP
Lessons Learned
AIR has the following lessons learned from the training and workshop:
- The training was highly valuable in orienting the PIE/NAW officials to the policy linking method and global benchmarking, and a lesson learned is to continue with capacity building training for officials in future workshops.
- The organization of the participants in a central location was critical in encouraging the interaction needed for mutual understanding of the process, and a lesson learned is to ensure in-person training and workshops in the future, with in-person lead facilitation.
- The translation of key parts of the workshop into Urdu and Sindhi increased panelist understanding and application of the policy linking method, and a lesson learned is to conduct more of the sessions in national languages.
- The interaction between PIE/NAW, donor agencies, AIR, and SAHE was instrumental in planning and implementing the training and workshop, and a lesson learned is to continue with this kind of multi-organizational support for future training sessions and workshops.