Paper short abstract:

We investigated the relationship between multiple-trait (MT) ratings of "Purpose and Content" and holistic (H) ratings for 55 essays. We found 43.6% concordance for H and MT ratings. The low group had high H ratings, and the high group had high MT ratings.

Paper long abstract:

This report investigates the relationship between multiple-trait ratings (MT) of "Purpose & Content" and holistic (H) ratings for 55 essays of European learners of Japanese as a second language in the I-JAS; essays compared fast food and home cooking, and offered opinions on food in daily life.

For MT rating, we used a flow chart designed around MT rating standards. Points were assigned by Level (L) 1-6 from lowest to highest for presence or absence of the following: main idea (comparison and opinion); support for the main idea; consistency and appropriateness of the main idea; social perspective and objectivity; and originality. The essay levels were distributed as follows: 5 L1s, 20 L2s; 12 L3s; 16 L4s; 2 L5s; and none at L6.

Our results of the rating relationships show:

(1) H and MT ratings for "Purpose & Content" aligned for 24 of 55 essays, for a 43.6% concordance rate.

(2) Results by level show that for the middle group on MT ratings for "Purpose & Content" (L3), there was a high level of concordance between H and MT ratings. For the group that rated low on MT ratings for "Purpose & Content" (L1-2), H ratings were higher than MT ratings; for the group that rated high on MT ratings for "Purpose & Content" (L4-5), MT ratings were higher than H ratings.

(3) There were only four essays that had a point spread of two or more between H and MT ratings. Of these, three essays had lower MT than H ratings, and one essay had a higher MT than H rating. In the cases where MT ratings were lower than H ratings, essays showed problems such as new ideas introduced in the conclusion; difficulty communicating the main idea because of regular use of questions such as "and you?"; and topics departing from the main theme. The one essay that had a higher MT rating effectively supported arguments with clearly-grounded assertions and specific examples, but had low Japanese language ability.