173 / 2018-03-28 14:53:24

Standards in and out: Setting rubrics for a task assessing core competencies

Standards; rubrics; core competencies

全体主题 > Standards in validating language assessments

The National English Curriculum Standards for Senior High Schools (NECS) (MOE, 2018) has proposed core competencies (CCs) as the goal of English education in schools. The four components are language competence, cultural awareness, thinking capacity and learning ability. The competency-based standards entails both innovative prospect and challenges for foreign language teaching and assessment, as it serves as the foundation based on which students are to be taught and assessed (Cheng, 2017).
Assessment plays a critical part in the implementation of newly developed standards, and it is important for task development and validation to follow the standards in the first place. In light of the definitions and descriptors of CCs in the NECS, a prototype task composed of four sub-tasks (9 items) is developed aiming to assess English CCs of senior high school students. As the CCs are largely latent (especially the latter three) and intertwined, the scoring of students’ task performance meets challenges in distinguishing among the competencies.
Therefore, this study focuses particularly on the rubric setting for this task with responses of sixty-six students in a pilot test. Four experts, including one member of the NECS revision panel, two scholars in language testing and one senior high school teachers, were invited to make judgements on the CCs and the specific competency aspects being assessed by each sub-task with support of the descriptions from the NECS and a response sample. Data collected from the expert judgement stage were analyzed to form draft rubric. Then two raters made revision of the rubric informed by the scaling practice, which provided better understanding of that how different aspects of competencies could manifest in and be identified from test-takers’ responses in a more valid and reliable way. The rubric defined language competence, thinking capacity and cultural awareness as the constructs of this task.
The rubric drawn from the above qualitative data was further examined quantitatively. A generalizability study was conducted to estimate the effects of different facets on the test scores. The results indicate that the major variance is accounted by the students (40.75%) and followed by the interaction between students and questions (21.53%), and the rater facet exerts little effect (0.01%) on scores. More data is under process for a comprehensive claim. The rubric development provides empirical insights into the CCs definition prescribed in the standards.
Notes: This study is funded by a Beijing's 13th five-year plan key project “A longitudinal study on the development of senior high school students’ English subject core competencies” (CADA17076).