Paper short abstract:

A statistical method was used to investigate the effectiveness of the automated scoring system "jWriter". The results of the survey revealed that jWriter is particularly effective at the beginner level.

Paper long abstract:

Our research group is developing a web system called "jWriter" (https://jreadability.net/jwriter/). This system is capable of automatically scoring learners' proficiency in writing and providing feedback comments to help them improve their writing. In order to clarify the validity of this system, we report the results of automatically scoring the data of 26 Japanese language learners at Y University in Croatia. The two specific research questions are as follows.

To what extent does the system "jWriter" capture the learners' operational proficiency?

To what extent do the feedback comments of the "jWriter" system affect the learners' writing?

To answer these questions, we conducted three surveys of 26 Japanese-language learners at Y University.

Survey 1: The students were asked to take the "SPOT" objective test to measure their Japanese language proficiency.

Survey 2: We asked them to write an opinion essay titled "住みやすい国の条件と理由(Conditions and reasons for a country to be a good place to live).

Survey 3: The opinion essays written in "Survey 2" were input into the "jWriter" automatic scoring system, and automatically scored. The students were then asked to rewrite their essays after receiving feedback comments from "jWriter.

As a result of analysis, the correlation coefficient between the scoring scores of Survey 1 and Survey 2 was r=.518, and the correlation coefficient between the scoring scores of Survey 1 and Survey 3 was r=.653. There was a strong correlation between the automatically scoring scores and the objective test scores both before and after the rewriting of the opinion essays. Next a Wilcoxon signed rank test was conducted on the data from Survey 2 and Survey 3, and a significant difference was found (V=72.0, p=0.015). For further analysis, we checked the data at the beginner, intermediate, and advanced levels and found significant changes, especially in the beginner data, and can therefore conclude that the automatic scoring system is particularly effective at the beginner level.