In this blog post I want to describe the work of the Learner Corpus Metadata working group and how it is connected to Språkbanken. I am actively contributing to technical implementations of a proposed metadata standard for the learner corpus research community. My colleague Elena Volodina on the other hand is interested in the output of the working group as a corpus creator with her work on the SweLL (Swedish Learner Language) corpus.
Learner corpora and metadata
Before diving deeper into the main topic I would like to clarify what I mean by both learner corpora and metadata. The Cambridge Handbook of Learner Corpus Research (Granger et al. 2015, p 1) gives a good idea what learner corpora are:
Written and spoken data produced by learners has always been a key resource for the study of second language acquisition (SLA). [...] Learner corpora gave rise to a flurry of studies, which have come to be grouped under the umbrella term of ‘learner corpus research’ (LCR). This new research strand emerged in the late 1980s as an offshoot of corpus linguistics [...]
As you can see it talks specifically about learners of languages, especially learning any language that is not their first language. The texts produced by learners can then be analyzed using methods of corpus linguistics and natural language processing to study second language acquisition. Text is used here in the broadest possible sense of the word because the textual representation researchers are working on can either be written directly by the learners or they could be transcribed from audio or video recordings.
To make the learner data more reusable we need to describe the data itself. That's the role of metadata. To quote Paquot et al. (2024, pp 280-281) specifically for learner corpus research:
Metadata can broadly be defined as ‘data about data’. When conducting a learner corpus study, metadata is crucial at every single step of the research process, from (1) study design, (2) corpus selection or compilation to (3) data analysis, and interpretation of the results.
But in the same paper they broaden the role of metadata using a statement by Higgins:
To put it differently, “metadata is the backbone of digital curation. Without it a digital resource may be irretrievable, unidentifiable or unusable” (Higgins, 2007).`
These are strong words. The two quotes highlight the different roles of metadata, both for the researchers and the research infrastructures. These roles are subsumed by the FAIR criteria as Paquot et al. (2024, pp 281-282) point out:
Metadata is considered a key element of the FAIR principles, essential for maximising optimal reuse of research data. According to the FAIR principles, data and metadata should be Findable (for both humans and computers), Accessible (with clear guidance on how to access the data, possibly including authentication and authorization), Interoperable (with applications and workflows for analysis, storage and processing) and Reusable (containing any information needed by other researchers to effectively interpret and disseminate findings from subsequent use of the data) (Wilkinson et al., 2016).
The FAIR principles as stated by Wilkinson et al. (2016) highlights the importance of metadata to be both readable by both humans and machines to make research data as useful as possible. The FAIR principle
R1.3: states that (Meta)data meet domain-relevant community standards. Unfortunately, until recently as Paquot et al. (2024, pp 282) state in reference to the community:
To date, however, Learner Corpus Research (LCR) has not developed community standards or best practices for data collection, archiving and sharing (cf. König et al., 2021; Stemle et al., 2019; Volodina et et al., 2018).
The metadata standard and working group
People have been aware about the problem of a missing community standard for the learner corpus community for a while. In 2017 an attempt for a metadata standard had been presented at the CLARIN workshop Workshop on Interoperability of Second Language Resources and Tools. Sylvain Granger gave a keynote titled Towards standardization of metadata for L2 corpora based on joint work with Magali Paquot. Unfortunately, this standard never got properly published. The work was continued by Magali Paquot and colleagues which led to a first public version of the Learner Corpus Metadata (LC-meta) which they presented at the Learner Corpus Research conference 2022. After some revisions the second version of the standard was published in May 2024 as Core Metadata Schema for Learner Corpora (version 2). The authors of the standard describe it in a lot of detail in an article that was published later in 2024 (Paquot et al. 2024).
It would go beyond the scope of this blog post to describe the standard in detail. However, I can highlight some design decisions. The goal of the metadata standard is to be sufficiently broad and flexible to cover as many applications as possible. This leads to a quite intimidating number of 168 fields. However as a creator of such metadata you most likely never have to touch all of them, many of them are optional and only used for specific use-cases. The metadata fields are sorted into eight groups ranging from general administrative metadata to specific information about the learners and texts. Fields and whole categories can be optional or repeated and some fields only have to be filled when other fields have a certain value, for example you only can give results of a placement test if a placement test has been recorded. When this metadata schema was published a spreadsheet format was chosen to represent the metadata. Spreadsheets are a common sight for many people working with computers. They can be edited quite easily but are also limited in their use and are not really considered machine-readable or actionable.
(A section of the Learner Corpus Metadata spreadsheet showing the beginning of the administrative section)
Following the release of the metadata schema, in November 2024 a call for interest in joining a working group was spread to relevant mailing lists. Both Elena and I followed the call and joined an initial meeting. Afterwards I actively joined the working group meeting more or less regularly since beginning of 2025.The LC-meta working group has members from more than five different organizations (see Figure 1). Even the Learner Corpora Association participates in an observing role.
(A diagram of the structure of the LC-meta working group)
The work of the group is split in two main aspects. Some people work on the technical details while others works on more social aspects such as community involvement. The technical working group mostly works on providing machine-readable representations for the LC-meta schema as well as tools for handling these formats. The other group tries to involve the community to help establish LC-meta as an actual community standard.
Our involvement and contributions
Elena has been involved in the topic of Learner corpus metadata for a long while. She was leading the development of the Swedish Learner Language corpus called SweLL which required work with extensive metadata description. In the context of this work she co-organized the CLARIN workshop where Sylvaine Granger gave the previously mentioned keynote. Despite the metadata standard never being published but it influenced the development of SweLL.
I first got in contact with learner corpora at the NLP4CALL workshops organized by Elena during my PhD research on language learning applications. At the workshops she and her colleagues presented their work on SweLL and related infrastructure. In my work in the QUEST project at the University of Hamburg I worked extensively with metadata standards for a wide range of community. With my background in language learning I joined my then colleague Elena Arestau who looked into learner corpus metadata as part of multimodal corpora. Some frustration about the situation with missing community standards got channeled in a paper I submitted to NLP4CALL (Lange 2022), just when the first metadata standard had been presented. When I learned about the new working group on Learner Corpus Metadata it was obvious that I wanted to join it and do my part to improve the situation. So far I worked on two problems: convert the LC-meta schema into a machine readable format and create a prototype for a web-based editor.
The machine readable format of choice is CMDI (Component Metadata Infrastructure), the standard metadata format within the CLARIN infrastructure. It is hierarchical and stored as XML. Together with a few other features CMDI is very suitable for representing the LC-meta schema. The tree structure can help us avoid some duplication and it has support for optional and repeated elements. The information that some fields are only relevant when other fields have a certain value can be stored in a CMDI profile but it is up to tools to handle this information properly.
(A section of the Learner Corpus Metadata CMDI profile showing the beginning of the administrative section)
This brings us to the next topic: tools. Because CMDI is XML-based we can use all the existing XML tools to edit and validate CMDI. And because it is the standard metadata standard used by CLARIN it is very well integrated in the CLARIN infrastructure which makes metadata in the CMDI format findable. And CLARIN actively develops tools handling CMDI. For us the most useful is CLARIAH CMDI form. Given a metadata schema as a CMDI profile it creates web forms for editing the metadata, taking care of all the constraints and helping the user as far as possible.
(A section of the Learner Corpus Metadata CMDI forms showing the beginning of the administrative section)
By converting the LC-meta schema into a CMDI profile and by adapting the CMDI forms we now have both a machine-readable metadata format for LC-meta and a user-friendly editor that keeps a lot of technical complexities hidden from the users.
Other initiatives within the group and next steps
We already build on the metadata standard used within CLARIN. Consequently, we try to push for better integration of LC-meta in the CLARIN infrastructure. The CLARIN Virtual Language Observatory (VLO) is a treasure trove to find language resources, especially by using filtering on so-called facets. Facets are metadata categories that help to quickly filter out unwanted results. But its strength depends on the metadata provided for these resources. At the same time it can only respect the most commonly provided metadata to cover as many resources as possible. LC-meta goes far beyond the usually considered information which makes it difficult to find the learner corpora relevant for your research. We are working with CLARIN on a way to have community-specific versions of the VLO that allow to filter by the full range of LC-meta relevant categories, e.g., learner L1 or learner background.
(The CLARIN VLO showing some learner corpora published by Eurac)
Our work is not the only work done in the group. Other members have explored different methods to collect and edit relevant metadata. One idea is to use an online survey based on the free and open source platform LimeSurvey to already collect information about learners, an important part of the LC-meta schema, while the texts for the corpus are created or collected. This avoids the problem with metadata that if you don't collect it in time it is almost impossible to go back and do it later.
In parallel to our work on CMDI and the CMDI-based editor two colleagues work on alternative approaches. On the one hand, CMDI is not the only possible machine-readable format, and not necessarily the best. Instead the metadata can be expressed as linked open data and stored as RDF. Linking data supports the ideas behind the FAIR principles but requires more knowledge than filling in the CMDI form. While CMDI is strongly connected to CLARIN and strictly limited to metadata, linked data and RDF build on standards by the World Wide Web Consortium (W3C). Another big bonus of using linked data and RDF is that the same format can be used to represent data and metadata and link both.
(This is how we imagine the LC-meta ecosystem in the near future)
An alternative to the CMDI editor we explore is the metadata editor released by the World Bank. It currently supports a bunch of metadata schemas, among others Dublin Core and a custom schema developed by the World Bank. Additional schemas can be define using JSON schemas. Work on creating a schema for LC-meta is in progress.
The development of all these formats and editors is pretty useless when they are not properly tested on real data and used in practice. Therefor we plan to test the representations and tools on both existing and newly created learner corpora. This brings us back to the SweLL corpus which can be a good test case. Together with Elena I plan to convert the existing SweLL metadata into the CMDI format.
Presentation at the Learner Corpus Conference 2026
One major success of the working group is that a paper to the Learner Corpus Research Conference in Prague with me as a co-author was accepted for presentation. This conference is the main venue for everyone working with learner corpora and it is essential to raise awareness about our work there. The review scores have been very good, almost intimidatingly, calling the submission seminal. This shows how important our work is for the community. The current results are presented in Prague on September 16th to 20th, 2026.
References
- Granger, Sylviane et al., 2015: "Introduction: Learner Corpus Research -- Past, Present and Future." in The Cambridge Handbook of Learner Corpus Research, edited by Sylviane Granger et al., Cambridge University Press (CUP), pp. 1–6, https://doi.org/10.1017/CBO9781139649414.001
- Wilkinson, Mark D. et al., 2016: "The FAIR Guiding Principles for scientific data management and stewardship", Sci Data 3, https://doi.org/10.1038/sdata.2016.18
- Lange, Herbert, 2022: "Metadata Formats for Learner Corpora: Case Study and Discussion" Proceedings of the 11th Workshop on Natural Language Processing for Computer-Assisted Language Learning (NLP4CALL 2022), https://doi.org/10.3384/ecp190011
- Paquot, Magali et al., 2024: "The Core Metadata Schema for Learner Corpora (LC-Meta)." International Journal of Learner Corpus Research, vol. 10, no. 2, pp. 280–300, https://doi.org/10.1075/IJLCR.24010.PAQ