Lessons learned from creating a balanced corpus from online data

Roberts Daragis; Kristīne Levāne-Petrova; Ilmārs Poikāns

doi:10.3233/FAIA200614

Lessons learned from creating a balanced corpus from online data

Roberts Daragis
, Kristīne Levāne-Petrova
, Ilmārs Poikāns

University of Latvia

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › Research › peer-review

6 Citations (Scopus)

Abstract

This paper describes lessons learned from developing the most recent Balanced Corpus of Modern Latvian (LVK2018) from various online sources. Most of the new corpora are created from data obtained from various text holders, which requires cooperation agreements with each of the text holders. Reaching these cooperation agreements is a difficult and time consuming task and may not be necessary if the resource to be created is not of hundred millions of size. Although there are many different resources available on the Internet today for a particular language, finding viable online resources to create a balanced corpus is still a challenging task. Developing a balanced corpus from various online sources does not require agreements with text holders, but it presents many more technical challenges, including text extraction, cleaning and validation.

Original language	English
Title of host publication	Human Language Technologies - The Baltic Perspective - Proceedings of the 9th International Conference Baltic HLT 2020
Editors	Andrius Utka, Jurgita Vaicenoniene, Jolanta Kovalevskaite, Danguole Kalinauskaite
Place of Publication	Amsterdam
Publisher	IOS Press
Pages	127-134
Volume	328
ISBN (Print)	9781643681160
DOIs	https://doi.org/10.3233/FAIA200614
Publication status	Published - 15 Sept 2020

Publication series

Name	Frontiers in Artificial Intelligence and Applications
Volume	328
ISSN (Print)	0922-6389
ISSN (Electronic)	1879-8314

Keywords

Balanced corpus
Corpus development
General corpus
Metadata

OECD Field of Science

1.2 Computer and Information Sciences

Access to Document

10.3233/FAIA200614

Cite this

Daragis, R., Levāne-Petrova, K., & Poikāns, I. (2020). Lessons learned from creating a balanced corpus from online data. In A. Utka, J. Vaicenoniene, J. Kovalevskaite, & D. Kalinauskaite (Eds.), Human Language Technologies - The Baltic Perspective - Proceedings of the 9th International Conference Baltic HLT 2020 (Vol. 328, pp. 127-134). (Frontiers in Artificial Intelligence and Applications; Vol. 328). IOS Press. https://doi.org/10.3233/FAIA200614

Daragis, Roberts ; Levāne-Petrova, Kristīne ; Poikāns, Ilmārs. / Lessons learned from creating a balanced corpus from online data. Human Language Technologies - The Baltic Perspective - Proceedings of the 9th International Conference Baltic HLT 2020. editor / Andrius Utka ; Jurgita Vaicenoniene ; Jolanta Kovalevskaite ; Danguole Kalinauskaite. Vol. 328 Amsterdam : IOS Press, 2020. pp. 127-134 (Frontiers in Artificial Intelligence and Applications).

@inproceedings{29a26ed35a584fec92e2126d50273e6e,

title = "Lessons learned from creating a balanced corpus from online data",

abstract = "This paper describes lessons learned from developing the most recent Balanced Corpus of Modern Latvian (LVK2018) from various online sources. Most of the new corpora are created from data obtained from various text holders, which requires cooperation agreements with each of the text holders. Reaching these cooperation agreements is a difficult and time consuming task and may not be necessary if the resource to be created is not of hundred millions of size. Although there are many different resources available on the Internet today for a particular language, finding viable online resources to create a balanced corpus is still a challenging task. Developing a balanced corpus from various online sources does not require agreements with text holders, but it presents many more technical challenges, including text extraction, cleaning and validation.",

keywords = "Balanced corpus, Corpus development, General corpus, Metadata",

author = "Roberts Daragis and Kristīne Levāne-Petrova and Ilmārs Poikāns",

note = "Publisher Copyright: {\textcopyright} 2020 The authors and IOS Press.",

year = "2020",

month = sep,

day = "15",

doi = "10.3233/FAIA200614",

language = "English",

isbn = "9781643681160",

volume = "328",

series = "Frontiers in Artificial Intelligence and Applications",

publisher = "IOS Press",

pages = "127--134",

editor = "Andrius Utka and Jurgita Vaicenoniene and Jolanta Kovalevskaite and Danguole Kalinauskaite",

booktitle = "Human Language Technologies - The Baltic Perspective - Proceedings of the 9th International Conference Baltic HLT 2020",

}

Daragis, R, Levāne-Petrova, K & Poikāns, I 2020, Lessons learned from creating a balanced corpus from online data. in A Utka, J Vaicenoniene, J Kovalevskaite & D Kalinauskaite (eds), Human Language Technologies - The Baltic Perspective - Proceedings of the 9th International Conference Baltic HLT 2020. vol. 328, Frontiers in Artificial Intelligence and Applications, vol. 328, IOS Press, Amsterdam, pp. 127-134. https://doi.org/10.3233/FAIA200614

Lessons learned from creating a balanced corpus from online data. / Daragis, Roberts; Levāne-Petrova, Kristīne; Poikāns, Ilmārs.
Human Language Technologies - The Baltic Perspective - Proceedings of the 9th International Conference Baltic HLT 2020. ed. / Andrius Utka; Jurgita Vaicenoniene; Jolanta Kovalevskaite; Danguole Kalinauskaite. Vol. 328 Amsterdam: IOS Press, 2020. p. 127-134 (Frontiers in Artificial Intelligence and Applications; Vol. 328).

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › Research › peer-review

TY - GEN

T1 - Lessons learned from creating a balanced corpus from online data

AU - Daragis, Roberts

AU - Levāne-Petrova, Kristīne

AU - Poikāns, Ilmārs

PY - 2020/9/15

Y1 - 2020/9/15

N2 - This paper describes lessons learned from developing the most recent Balanced Corpus of Modern Latvian (LVK2018) from various online sources. Most of the new corpora are created from data obtained from various text holders, which requires cooperation agreements with each of the text holders. Reaching these cooperation agreements is a difficult and time consuming task and may not be necessary if the resource to be created is not of hundred millions of size. Although there are many different resources available on the Internet today for a particular language, finding viable online resources to create a balanced corpus is still a challenging task. Developing a balanced corpus from various online sources does not require agreements with text holders, but it presents many more technical challenges, including text extraction, cleaning and validation.

AB - This paper describes lessons learned from developing the most recent Balanced Corpus of Modern Latvian (LVK2018) from various online sources. Most of the new corpora are created from data obtained from various text holders, which requires cooperation agreements with each of the text holders. Reaching these cooperation agreements is a difficult and time consuming task and may not be necessary if the resource to be created is not of hundred millions of size. Although there are many different resources available on the Internet today for a particular language, finding viable online resources to create a balanced corpus is still a challenging task. Developing a balanced corpus from various online sources does not require agreements with text holders, but it presents many more technical challenges, including text extraction, cleaning and validation.

KW - Balanced corpus

KW - Corpus development

KW - General corpus

KW - Metadata

UR - http://ebooks.iospress.nl/volumearticle/55535

UR - https://www.scopus.com/pages/publications/85093363245

U2 - 10.3233/FAIA200614

DO - 10.3233/FAIA200614

M3 - Conference paper

SN - 9781643681160

VL - 328

T3 - Frontiers in Artificial Intelligence and Applications

SP - 127

EP - 134

BT - Human Language Technologies - The Baltic Perspective - Proceedings of the 9th International Conference Baltic HLT 2020

A2 - Utka, Andrius

A2 - Vaicenoniene, Jurgita

A2 - Kovalevskaite, Jolanta

A2 - Kalinauskaite, Danguole

PB - IOS Press

CY - Amsterdam

ER -

Daragis R, Levāne-Petrova K, Poikāns I. Lessons learned from creating a balanced corpus from online data. In Utka A, Vaicenoniene J, Kovalevskaite J, Kalinauskaite D, editors, Human Language Technologies - The Baltic Perspective - Proceedings of the 9th International Conference Baltic HLT 2020. Vol. 328. Amsterdam: IOS Press. 2020. p. 127-134. (Frontiers in Artificial Intelligence and Applications). doi: 10.3233/FAIA200614

Lessons learned from creating a balanced corpus from online data

Abstract

Publication series

Keywords

OECD Field of Science

Access to Document

Other files and links

Fingerprint

Cite this