Brian Annan 

This is the summary of the paper I will present for discussions at the University of Lyon re “What are the strategic challenges behind implementation of accountability in education policies?”

The use of assessment as a strategy to track and lift student achievement within and across schools has only moved to centre stage in recent years in schooling improvement projects in New Zealand.  Furthermore, it only became a government priority to improve all schools late last year.  One main reason is because New Zealand’s national assessment policy concentrated on formative assessment in the classroom.  Of utmost importance has been the relationship between the teacher and the student in setting learning goals and providing students with relevant feedback and feed forward for next steps in the learning process.  Tracking achievement trends beyond the classroom was left to national and international sampling surveys.  Results from those surveys guided systems-level decisions about hotspots in the student population and curriculum areas where professional development resources would be best spent.  Literacy and underachievement became national spotlights due to recent results which indicated New Zealand had a high average in reading literacy but a significant number of students spread through all schools tracking well below where they should be (references).  Although the sampling techniques avoided the traps attached to high-stakes national testing and punitive accountability measures, it allowed school leaders and teachers to disassociate themselves from the national underachievement problem in reading literacy.  I’ve heard many principals and senior teachers in many districts where the underachievement problem is most serious say “sure we have a problem in literacy in New Zealand, but we are doing a great job.”

Schooling improvement projects have been one lever among others charged with the task of sorting out whether those claims are true. It has been a long game of supporting small groups of schools (referred to as “clusters” in this paper) to come around to the truth about how effective teaching and learning is in their schools and to make improvements where necessary.  The incoming government has signalled the long game is over with its standards push in literacy and numeracy.  Government is determined to get on top of the underachievement problem at an accelerated rate by involving all schools in working out the way forward.  The push is not intended to be a pendulum swing away from in-class formative assessment to out-of-class tracking of achievement trends.  It is about finding the right balance.  Policy details are currently being formulated to make a start in 2010.  This paper is not intended to foreshadow what that start might look like.  Rather, it outlines schooling improvement’s contribution for policy consideration. 

Schooling improvement’s main contribution is information from its latest venture into building evaluative capability across the leaders of 18 clusters, which together involve about 200 schools (about 8 percent of all New Zealand schools).  The venture has two aims. The first aim is to identify and build on existing evaluative capability among all principals, lead teachers, professional development providers and local Ministry officials participating in cluster interventions as well as national Ministry staff who make decisions about on-going support.  The second aim is to evaluate the overall effect of schooling improvement as a policy intervention on raising student achievement.  This evaluation serves as both a value-for-money exercise and an investigation into the likely explanatory variables to assess the value that involvement with schooling improvement has given, and can give, students and schools.   Work towards both aims is expected to merge into more effective ways of doing schooling improvement within and across those schools with the most serious underachievement trends. 

Early findings against the first aim of building evaluative capability indicate five areas of development. These are leaders’ capability in (i) self-review and development of themselves against key dimensions of the inquiry and knowledge building cycle,     (ii) developing effective and practical plans, (iii) making explicit and developing the theory for improvement underpinning their plans (iv) analysing and using student achievement data, (v) supporting teachers to engage students in the improvement process, and (vi) using a lot more change talk to make sure changes do occur in classrooms and across the system. 

A feasibility study has been completed to shed light on whether the second aim of assessing the overall effectiveness of schooling improvement is possible or not.  Achievement data received and analysed from 187 of 199 schools indicates that it is possible to assess overall effectiveness even though the clusters use a variety of tests at different times during the school year.  This first attempt to pool data found that progress is being made across all the clusters at a fairly similar rate. It also indicated that all schools will need to provide essential demographic data, e.g. gender and ethnicity, in order to get a rich picture of variables influencing the achievement trends. 

In the process of doing this work, we are learning to deal with some of the challenges that all schools as well as systems-level educators are about to face as they enter a standards based environment.  The following three challenges have strategic relevance in New Zealand and seem relevant to improvement efforts in the United States, England and France; 
(i)    participants focusing on a learning agenda so that they get better at their particular roles rather than trying to get a good score from evaluators,
(ii)    achieving role clarity between school leaders, professional providers and Ministry officials, and
(iii)    commercially-competing evaluation and professional provider organisations collaborating effectively. 

The paper will outline in more detail the schooling improvement journey in New Zealand, our latest venture into the evaluative capability work and its associated strategic challenges.