Send message to Author

Paper short abstract

This paper examines challenges in incorporating social media data into BCCWJ2, an extension of the Balanced Corpus of Contemporary Japanese. Using data from microblog-style platforms, it discusses issues of representativeness and metadata design in building a balanced reference corpus.

Paper long abstract

This paper examines the challenges involved in incorporating social media data into BCCWJ2, an ongoing extension of the Balanced Corpus of Contemporary Japanese. While texts produced on social media have become a central component of contemporary written communication, their inclusion in a public reference corpus raises methodological, conceptual, and ethical issues that differ substantially from those associated with traditional published materials.

The paper focuses on microblog-style platforms such as Bluesky and Misskey, which are structurally comparable to Twitter (X) and are currently being considered as candidates for inclusion in BCCWJ2. After outlining the current state of large-scale data collection from these platforms, the paper discusses key issues related to platform selection, data volume, and representativeness. Given that social media data can be collected on a scale far exceeding that of the corpus as a whole, careful selection and sampling are required in order to maintain balance within BCCWJ2.

Particular attention is paid to criteria for selecting social media texts, including the distribution of post types (e.g. original posts, replies, automated posts), temporal dispersion to mitigate event-driven bias, and the potential overrepresentation of language use by a small number of highly active users. The paper also addresses the treatment of metadata, considering which information should be retained within an XML-based corpus structure to support linguistic research while avoiding unnecessary personal identification.

By highlighting these issues, the paper argues that incorporating social media data into a balanced reference corpus is not simply a matter of increasing data volume. Rather, it requires reconsideration of fundamental design principles such as balance, representativeness, and usability. The discussion aims to contribute to broader debates on how large-scale reference corpora can adapt to new forms of digitally mediated language use.

Send message to Author

Paper short abstract

This talk outlines the design of BCCWJ2, a 200-million-word expansion of BCCWJ. We describe NDC-stratified annual book sampling (5M words/year; ~1,000 books), streamlined metadata/annotation and copyright-aware policies, and demo the release data in Chunagon.

Paper long abstract

This presentation reports on the design of BCCWJ2, the ongoing expansion of the Balanced Corpus of Contemporary Written Japanese (BCCWJ) from approximately 100 million to 200 million words. The core addition is a large-scale collection of book data published between 2006 and 2025, sampled at a fixed rate of 5 million words per year. To preserve the principle of balance while ensuring transparency and replicability, sampling is stratified by the Nippon Decimal Classification (NDC), resulting in approximately 1,000 sampled books per year.

We focus on design choices that balance continuity with BCCWJ1 and practical feasibility under current conditions. First, we explain the revised sampling workflow: BCCWJ2 defines the population and sampling unit at the book level, simplifying procedures compared to earlier page- or character-based randomization while keeping NDC-based stratification. Because complete bibliographic information for the entire 2006–2025 period was not available at the outset of the project, the sampling frame is operationalized year by year, allowing steady annual growth and timely releases.

Second, we outline metadata and annotation policies. To improve construction efficiency while maintaining usability, we prioritize the metadata items exposed in the Chunagon interface and streamline less-used structural tag sets. The talk also introduces new metadata for safer secondary use (e.g., adult-content filtering in an rated R scheme) and explains how these decisions support research and educational contexts.

Third, we address copyright-aware design. Building on Japan’s post-2018 legal environment for text and data mining (TDM), BCCWJ2 adopts operational constraints such as excluding short poetic works and carefully controlling the length of contexts displayed in web services.

The presentation concludes with a live demonstration of the currently constructed subset (focusing on the 2006–2010 portion) in Chunagon, showing continuity with BCCWJ1 workflows and illustrating how users in and outside Japan can access and make use of the expanded corpus.

Send message to Author

Paper short abstract

This presentation outlines the morphological annotation of BCCWJ2 and compares it with earlier BCCWJ data to identify newly emerged words and significant frequency changes, demonstrating how BCCWJ2 supports the analysis of diachronic lexical change in contemporary Japanese.

Paper long abstract

This presentation provides an overview of the morphological annotation in BCCWJ2 and examines the lexical characteristics of BCCWJ2 through a comparison with data from earlier periods represented by BCCWJ. Using this morphological information, we extract lexical items that have newly emerged or have shown statistically significant frequency changes, thereby clarifying distinctive features of the expanded corpus.

BCCWJ2 adopts two types of word units, short-unit words and long-unit words, and annotates each with morphological information. Short-unit words are linguistic units defined with a focus on morphological structure; they are characterized by clear criteria and minimal variation in segmentation. Morphological information for short-unit words is annotated by applying manual corrections to the results of automatic morphological analysis using the UniDic morphological dictionary. Long-unit words, by contrast, are defined with a focus on syntactic structure, and their morphological information is constructed by combining the morphological information of short-unit words and applying manual corrections to the output of a newly developed long-unit analyzer. All morphological annotations in BCCWJ2 follow the same specifications as those used in BCCWJ, enabling direct and reliable lexical comparison between the two corpora.

In the latter part of the presentation, we focus on short-unit morphological information to compare data from the 2006–2010 portion of BCCWJ2 with data from earlier periods in BCCWJ. Specifically, we compare lexical frequencies across the two periods using statistical measures in order to identify lexical items that have newly emerged as well as those that have become markedly more or less frequent. Based on these results, we discuss how social and technological changes have influenced Japanese vocabulary in the late 2000s (2006–2010), and how processes of lexical diffusion, stabilization, and decline are reflected in the corpus. Through this analysis, we demonstrate that BCCWJ2 provides an effective foundation for research into diachronic changes in contemporary Japanese vocabulary.