What are corpus sentence sets?

For many of Språkbanken's corpora downloadable sentence sets are available. The sentence sets are collections of sentences from each corpus, with automatic annotation of, e.g., part-of-speech and syntactic structure. The sentences have been scrambled for copyright reasons and thus appear in a randomized order, so that the original texts cannot be recreated.

This is approximately what Sparv's "standard" XML format for (sentence sets) looks like:

<text>
 <paragraph>
   <sentence>
     <token _tail="\s">text</token>
   </sentence>
 </paragraph>
</text>

Notes:

• Not all corpora have paragraphs.
• Older corpora (those that were annotated with versions of Sparv older than 4.0) have <w> tags instead of <token> and they never contain _tail attributes.
• The _tail attribute inside <token> holds information about the whitespaces (spaces "\s", tabs "\t" or newlines "\n") that follow that token in the source material.
• In addition to the tags and attributes shown above there can be any amount of additional tags and attributes in the data, depending on what the input looked like and what annotations were added by Sparv.

What is a bz2 file?

Since the sentence sets can be very large, the XML files are compressed using the bzip2 format to reduce the file size and the amount of data that needs to be downloaded.

Processing the bz2 file with Python

If you want to process the file programmatically, you do not need to decompress the entire file first. Python can read XML data directly from the compressed bz2 file. Since the file can be very large, we recommend processing it incrementally rather than loading the entire XML document into memory.

Here is an example using Python's standard library:

import bz2
import xml.etree.ElementTree as ET
with bz2.open("filename.xml.bz2", "rb") as f:
   for _event, elem in ET.iterparse(f, events=("end",)):
       # Replace "paragraph" with the relevant XML element
       if elem.tag == "paragraph":
           # Add your processing code here
           print(elem.text)
           # Free memory when done with the element
           elem.clear()

Decompressing the file manually

If you prefer to decompress the XML file, you can do so as follows:

• Windows: Use 7-Zip or another program that supports bz2 files.
• macOS: bz2 files can usually be decompressed directly in Finder. You can also use an application such as The Unarchiver.
• Linux: From the command line, you can use bunzip2 filename.xml.bz2.

Note that the decompressed XML file can be significantly larger than the bz2 file and may therefore require a considerable amount of disk space.