关键词: 新HSK，写作测评，信度，概化理论

Abstract: New HSK is the official Chinese proficiency test for speakers of other languages, which is administered by China’s Hanban (i.e. National Office for Teaching Chinese as a Foreign Language). It was first launched in 2010, and held five times annually. Being a rather new test, research and development focusing on an improvement in the test’s quality is essential. Writing test scores of the New HSK, similar to those of any other language proficiency tests, are most vulnerable to reliability criticism. The study collected writing samples (n=52) from two sets of mock tests of the writing part of the New HSK, level 5. Data analysis was performed from the perspective of Generalizability Theory using Genova and mGenova software. Variance components were estimated for possible influencing effects of item, raters, rating speed, and their interactions. Dependability coefficient (Phi) was estimated for current test settings. Moreover, D studies with various rating patterns were conducted to explore approaches promoting the Phi coefficient. Major findings are: (a) According to current allocation of quantity for each item type, the descending order of the Phi coefficient for each item type is: writing based on keywords given, writing based on the given photo, ordering of the inner-sentence components; (b) For the Phi coefficient to be at least 0.8 for each item type, ordering items needs to increase from 8 items to 20 while the other two need to increase one item each; (c) If item quantities are increased as specified in b, then, the ordering items needs only one rater, writing based on keywords needs two, and writing based on photo needs three. However, if writing based on photo has three items, two raters will be enough to serve the need of keeping Phi coefficient as least 0.8; (d) With current allocation of item quantities for each item type, if calculation of the comprehensive score of writing uses weight proportional to the raw scores, then the Phi coefficient for the writing test could marginally reach the level of 0.8. But the study explored various approaches reaching a Phi coefficient at least 0.85 with relatively lower costs (for details, please refer to section 3.2.2 of this paper). For such estimations, the analysis applied solver functions of Microsoft Excel. (e) The study did not find effect of rating speed significant. However, this conclusion was limited to the two different speeds investigated by this study: a speed that each individual rater feels comfortable, and a speed that raters feel a little pushy but still have confidence about the rating quality under that speed. Furthermore, more evidence is needed to support such a conclusion, especially real data from professional raters. The study suggested to investigate effect of rating speeds on generalizability of writing test scores with more rigorous designs and more mature data collection techniques, and called for more attention to reliability issues of writing tests.

Key words: new HSK, writing assessment, reliability, Generalizability Theory