Reading ELAN files to R

What this is about

Update from May 2017: Lots of things have changed, for example, at the moment I would prefer doing things with purrr::map instead of plyr, on the other hand this would leave complete rewrite of many functions and the purrr approach just is so convenient. Even more, the list columns in dataframes have been gaining popularity lately, and that may be very convenient for language documentation data as well, which doesn’t necessarily fit so neatly into a tabular representation.

This treatment may appear little bit chaotic and unstructured, but this is only because the topic in hand is relatively complicated. I try to explain it to different audiences, but I also understand that following the whole text demands knowledge about some documentary linguistic practices and programming language R. The main idea is to present ways to use R to work with ELAN files. The main idea is to get from something like this:

Into something like this:

As one can see, not all data in this data frame comes from ELAN file directly. In my opinion this is one of the major reasons to use R to work with ELAN files. Associating session name or speaker id with any arbitrary pieces of information suddenly become trivially easy. In this case I have just merged some metadata with the tokens, but naturally there is no end to what one can have there. I have sorted the data here by speaker and the time, whereas sorting it only with time would of course bring to the table the natural order in which the utterances were spoken. In principle one could use more complicated data structures to store the ELAN file’s content, but in my opinion a data frame is just fine since it is very easy to work with. I will go through now only how one reads ELAN files – metadata comes later. I can just say briefly that what I’m usually doing is to open database connection from R to FileMaker Pro and to read the metadata from there. Now perfect, but works fine and is easy.

I wrote already some year ago a small R package that can be used to parse ELAN files. We’ve called it FRelan, as it is developed with Freiburg based language documentation traditions. There is just one small problem – it works out of the box only with our ELAN files. The reason is that ELAN files can be structured in almost any possible way. As far as I see it, it is almost impossible to write a package which would just work automatically. However, I think it is possible to write some kind of generic tools which can be adapted into most of the files, and following the same logic one can also parse more exotic tier structures if the need arises.

FRelan package can be installed from GitHub with devtools package:

install.packages("devtools")
devtools::install_github("langdoc/FRelan")

However, one should probably read this post thoroughly before attempting to use it. (And do not try to do that before this post is ‘officially’ published, because it is guaranteed to be work very weirdly!)

In an ideal world one could read ELAN file into R just typing:

FRelan::read_eaf(file = "elan_file.eaf")

And this is how it works with our ELAN files, but I want to help others to achieve the same with their corpora! Of course usually the idea would be to read in the whole corpus, which can be achieved easiest this way:

eaf_files <- list.files(path = "path/to/corpus/main/folder", 
                        pattern = "eaf$", 
                        full.names = T, 
                        recursive = T, )
corpus <- plyr::ldply(eaf_files, FRelan::read_eaf) # could be nicer with purrr::map

What happens here is also relatively simple. I use list.files function to find all ELAN files recursively. Those files are stored in a character vector which has one element for each ELAN file. After that I use ldply function from plyr package, which basically loops the function read_eaf across each path stored in eaf_files and coerces the result into a dataframe. In this point the data frame corpus one row for each token in the ELAN files with all information that is present in those files. However, reality is often less beautiful, and the parsing tends to be interrupted in one point or another. If each ELAN file would be perfectly similar in their structure, the loop would work. The majority of problems arise from tiny hidden inconsistencies which at least my files tend to be ridden with. I would call these an invisible problem, because they never really pop up in our daily work with ELAN corpus, but actually cause very large number of issues behind the scenes. So in the end very much about parsing ELAN files into R is actually about finding out which ELAN files are inconsistent with R.

Lots of common tasks use packages plyr, dplyr, xml2 and XML, so installing those is necessary. Especially two first mentioned are very useful in general. Update from 2017: Now it is possible to install most of these with install.packages('tidyverse'). It is good to note that plyr is not part of tidyverse, although few years ago it was the most convenient tool for plenty of tasks.

library(tidyverse)
librar(FRelan)

In this tutorial I use ELAN file called kpv_izva20140404IgusevJA.eaf, which is a long interview I made with Komi researcher Jevgeni Aleksandrovich Igushev in 2014.

test_file <- "kpv_izva20140404IgusevJA.eaf"

Introduction

This writing is part of the wider topic of how to get documentary linguistic data into R. This means not only reading in ELAN files, but doing the same for metadata and other accompanying datasets we may have with our corpus. There are many reasons one may want to have documentary linguistic corpus in R. It makes searching the data very easy, and one is not limited to the predefined possibilities of individual user-interface. Besides this it is easy to make the study entirely reproducible, as there is no unnecessary mouse clicking involved. One can do at least the following tasks entirely in R:

• Searching the ELAN corpus
• Detecting structural mistakes or typos in ELAN corpus
• Opening ELAN files directly from R
• Associating linguistic data with metadata
• Plotting the search results or corpus content in different ways
• Doing the statistical evaluations of your results

R has already for many years been used in variationistic sociolinguistics, and also in that world use of ELAN has been around (Nagy and Meyerhoff 2015). In their present paper Nagy and Meyerhoff suggest a possible workflow. However, this workflow is also based upon exporting the ELAN files into CSV format, which is then being read into R. Reading these files into R is trivially easy, but it creates uneasy situation where doing changes in the data demands creation of a new export file. The situation gets even more problematic if there are further annotations done to those spreadsheets, as it can get quite messy to append the changes from modified ELAN files into annotated tables. This is very typical problem, in my opinion, and there are very few satisfactory solutions to the general problem (which also persists in ELAN when the transcriptions are modified but there are annotations already on the tiers below).

All treatments of data analysis with R start with the notion that one has to get the data into R (see, for example (Wickham and Grolemund 2016)). Getting data into R is basically a data transformation, and the method of doing the import is to define this transformation and run it whenever needed. So the question can also be turned into a statement: if data can be transformed, then it can be read into R. By transformation I mean any kind of operation where the data is parsed, in its totality or with some subset, and turned into whatever other data structure we need without information being lost. Or if the information is lost, it should be clearly understood and acknowledged.

In order to be able to make this kind of transformations to the corpus, the data structures have to be perfectly described and systematically present in the files. Actually making sure that this is the case would be very easy, for example, with parameterized RMarkdown reports. The R functions to parse ELAN files are kind of backbones of this kind of testing, about which I should probably be writing more soon.

I think often when there is talk about data formats there is some confusion over the fact that the data should be in different structures in different parts of its life. Of course these stages are not really distinct from one another, but there are clear conceptual differences and requirements which I try to outline below.

• data editing: minimal repetition of fields
• data archiving: maximal usability for different purposes
• data analysis: wide applicability for different statistics and plotting

All these stages have different demands for the ideal data structures, and we should not imagine that there is just one solution for this. Naturally this is a two way street, so that during data analysis one often needs to edit the data and come back to analysis. If we think about the resulting dataset in terms of tidy data, where we would ideally have one row per one observation (Wickham 2014), then I think we often should consider token as the basic unit here. The resulting data is relatively ineffective in the sense that the same information is repeated all over the place, but it is tidy according to some specifications. It can be described also as long, as there are relatively few columns and the data is stored along the rows. This kind of data structure is not supposed to be edited by human, but computers crunch it very happily.

Technical part

There are many ways to parse ELAN file into R, and ultimately it boils down to the question what do you want to do? Usually I want to search from the data in R or I want to merge some content with metadata, either to make more complex searches or to check what is going on. However, the scope varies. If I think in terms of ELAN’s tier structure, there are following attributes we can use:

• linguistic type (ideally each file would have one tier of one type per speaker)
• speaker
• tier name (although this is complicated – what is tier name but combination of attributes above?)

Reading ELAN files into R is not trivially easy and it gets fast very complicated. The main question is always how uniform the ELAN files are in their basic structure. This is really where all problems begin. If all ELAN files are identical, then one can just specify the pattern to parse one and loop it over all files. However, in reality the situation is much more complicated, and ELAN files tend to be somewhat heterogenic. There is nothing in ELAN that would prevent small creeping in here and there, and what may look harmless for a human, for example, slightly different name of a linguistic type, is of course impassable barrier for a computer which tries to read the file. So the code to parse ELAN files often needs to have all kinds of bells and whistles attached into it to handle errors, which makes the code very complicated and there is always possibility that something fails in a new way.

However, this is what makes parsing ELAN files to R particularly rewarding as well. It tells where the problems are, and makes it easier to address them. It can thereby be tested also as a robust testing method which helps to keep ELAN files organized. As far as I see it, the fact that all files adhere to uniform structure should be among the most basic definitions we have for corpus, differentiating it from eclectic file collections. This comes back to what I was describing above: if corpus is parsable, it is also transformable, and this is actually guarantees its longevity. It is not likely that in thirty years we want data to be in exactly this kind of XML files, but if they are perfectly uniform doing the transformation will never be more than a fast and easy task, whereas currently that can be quite a nightmare.

FRelan::read_tier(eaf_file = test_file, linguistic_type = "wordT") %>% 
        dplyr::slice(1:15) %>% # This slices 15 rows
        knitr::kable() # This displays table nicely

FRelan::read_tier(eaf_file = test_file, linguistic_type = "refT") %>% 
        dplyr::slice(1:5) %>%
        knitr::kable()

# kable() thinks the id's are email addresses, have to look into that in some point...

|-ref (id)
\- orth (kpv)
  \- word (tokenized from orth)
\- ft-rus (Russian translation)
\- ft-eng (English translation)
\- note (different notes on utterance level)

elan_file <- FRelan::read_tier(eaf_file = test_file, linguistic_type = "wordT") %>%
        dplyr::select(content, ref_id, participant) %>%
        dplyr::rename(token = content) %>%
        dplyr::rename(annot_id = ref_id) %>%
        dplyr::left_join(FRelan::read_tier(eaf_file = test_file, linguistic_type = "orthT")) %>%
        dplyr::select(token, content, participant, ref_id) %>%
        dplyr::rename(utterance = content) %>%
        dplyr::rename(annot_id = ref_id) %>%
        dplyr::left_join(FRelan::read_tier(eaf_file = test_file, linguistic_type = "refT")) %>%
        dplyr::select(token, utterance, content, participant, time_slot_1, time_slot_2) %>%
        dplyr::rename(reference = content)

## Joining, by = c("annot_id", "participant")

## Joining, by = c("participant", "annot_id")

elan_file %>% dplyr::slice(1:15) %>% knitr::kable()

# Note: add also some translation tiers to this

elan_file %>% dplyr::left_join(FRelan::read_timeslots(eaf_file = test_file), 
                               by = c("time_slot_1" = "time_slot_id")) %>%
        dplyr::rename(time_start = time_value) %>%
        dplyr::left_join(FRelan::read_timeslots(eaf_file = test_file), 
                         by = c("time_slot_2" = "time_slot_id")) %>%
        dplyr::rename(time_end = time_value) %>%
        dplyr::select(-time_slot_1, -time_slot_2) -> elan_content

elan_content %>% dplyr::slice(1:15) %>% knitr::kable()

elan_content %>% count(participant) %>% knitr::kable()

elan_content %>% FRelan::find_token(".+иг.+")

## # A tibble: 5 x 6
##             token
##             <chr>
## 1      кыпӧдчигас
## 2     Григорьевна
## 3         керигӧн
## 4     Григорьевна
## 5 оригинальнӧйӧсь
## # ... with 5 more variables: utterance <chr>, reference <chr>,
## #   participant <chr>, time_start <dbl>, time_end <dbl>

# About the same as:
# elan_content %>% filter(str_detect(token, '.+иг'))