Morphological dictionary

Inspired by the success of the Universal Dependencies for cross-linguistic annotation of syntactic dependencies, similar efforts have emerged for morphology, e.g., UniMorph[1] and UDer.[2] These feature simple tabular (tab-separated) formats with one form in a row, and its derivation (UDer), resp., inflection information (UniMorph):

aalen   aalend  V.PTCP;PRS

aalen   aalen   V;IND;PRS;1;PL

aalen   aalen   V;IND;PRS;3;PL

aalen   aalen   V;NFIN

(UniMorph, German. Columns are LEMMA, FORM, FEATURES)

In UDer, additional information (part of speech) is encoded within the columns:

abändern_V      Abänderung_Nf   dVN07>

Abarbeiten_Nn   abarbeiten_V    dNV09>

abartig_A       Abartigkeit_Nf  dAN03>

Abart_Nf        abartig_A       dNA05>

abbaggern_V     Abbaggern_Nn    dVN09>

(UDer, German DErivBase 0.5. Columns are BASE, DERIVED, RULE)

At the time of writing (2021), all of these are non-aligned morphological dictionaries (see below). Their simplistic format is particularly well-suited for the application of machine learning techniques, and UniMorph in particular, has been subject of numerous shared tasks.

Finite State Transducers (FSTs) are a popular technique for the computational handling of morphology, esp., inflectional morphology. In rule-based morphological parsers, both lexicon and rules are normally formalized as finite state automata and subsequently combined. They thus require morphological dictionaries with specific processing instructions (which often have a linguistic interpretation, but, technically, are just treated like arbitrary string symbols).[3] Popular FST packages such as SFST[4] (as available from the fst package in Debian and Ubuntu) allow to define application-specific file formats for morphological lexica, that bundle different pieces of morphological information with every individual morpheme. These are thus aligned morphological dictionaries, but very rich (and also, idiosyncratic) in structure.

Sample data from SMOR[5] (German SFST grammar):

<Base_Stems>Aachen<NN><base><nativ><Name-Neut_s>

<Base_Stems>Aal<NN><base><nativ><NMasc_es_e>

<Base_Stems>Aarau<NN><base><nativ><Name-Neut_s>

<Suff_Stems><suffderiv><gebunden><kompos><NN>nom<>:e<>:n<NN><SUFF><kompos><frei>

<Suff_Stems><suffderiv><gebunden><kompos><NN>nom<NN><SUFF><base><frei><NMasc_en_en>

<Suff_Stems><suffderiv><gebunden><kompos><NN>nom<NN><SUFF><deriv><frei>

Interlinear Glossed Text (IGT) is a popular formalism in language documentation, linguistic typology and other branches of linguistics and the philologies. Although IGT can be created without any specialized software (but just with a conventional editor), such specialized software has been developed, with notable examples such as Toolbox,[6] the FieldWorks Language Explorer (FLEx)[7] or open source alternatives such as Xigt.[8] Toolbox and FLEx support semi-automated annotation by means of an internal morphological dictionary. Whenever a morphological segment is encountered for which an annotation in the dictionary can be found, this annotations is applied. Whenever a morphological segment is newly annotated, the annotation is stored in the dictionary. FLEx and Toolbox provide different editor functionalities for annotating text and editing dictionaries, so that additional information beyond that found in annotations can be added, but at its core, their formats provide aligned morphological dictionaries.

FLEx and Xigt are based on XML formats, Toolbox uses a plain text format with idiosyncratic "markers". FLEx and Toolbox are not directly interoperable with each other, but a semiautomated converter for Toolbox to FLEx does exist. Xigt comes with FLEx and Toolbox importers, but is less widely used that either FLEx or Toolbox. Their formats of FLEx and Toolbox are not intended for human consumption, nor are they well-supported by any processing software other than their native tools.

OntoLex is a community standard for machine-readable dictionaries on the web. In 2019, the OntoLex-Morph module has been proposed to facilitate data modelling of morphology in lexicography, as well as to provide a data model for morphological dictionaries for Natural Language Processing.[9] OntoLex-Morph does support both aligned and non-aligned morphological dictionaries. A specific goal is to establish interoperability between and among IGT dictionaries, FST lexicons and morphological dictionaries used for machine learning.

In an aligned morphological dictionary, the correspondence between the surface form and the lexical form of a word is aligned at the character level, for example:

(h,h) (o,o) (u,u) (s,s) (e,e) (s,⟨n⟩), (θ,⟨pl⟩)

Where θ is the empty symbol and ⟨n⟩ signifies "noun", and ⟨pl⟩ signifies "plural".

In the example the left hand side is the surface form (input), and the right hand side is the lexical form (output). This order is used in morphological analysis where a lexical form is generated from a surface form. In morphological generation this order would be reversed.

Formally, if Σ is the alphabet of the input symbols, and ${\displaystyle \Gamma }$ is the alphabet of the output symbols, an aligned morphological dictionary is a subset ${\displaystyle A\subset 2^{(L^{*})}}$, where:

${\displaystyle L=((\Sigma \cup {\theta })\times \Gamma )\cup (\Sigma \times (\Gamma \cup {\theta }))}$

is the alphabet of all the possible alignments including the empty symbol. That is, an aligned morphological dictionary is a set of string in ${\displaystyle L^{*}}$.

A non-aligned morphological dictionary (or full-form dictionary) is simply a set ${\displaystyle U\subset 2^{(\Gamma ^{*}\times \Sigma ^{*})}}$ of pairs of input and output strings. A non-aligned morphological dictionary would represent the previous example as:

(houses, house⟨n⟩⟨pl⟩)

It is possible to convert a non-aligned dictionary into an aligned dictionary. Besides trivial alignments to the left or to the right, linguistically motivated alignments which align characters to their corresponding morphemes are possible.

Frequently there exists more than one lexical form associated with a surface form of a word. For example, "house" may be a noun in the singular, /haʊs/, or may be a verb in the present tense, /haʊz/. As a result of this it is necessary to have a function which relates input strings with their corresponding output strings.

If we define the set ${\displaystyle E\subset \Sigma ^{*}}$ of input words such that ${\displaystyle E={w:(w,w')\in U}}$, the correspondence function would be ${\displaystyle \tau :E\rightarrow 2^{\Gamma ^{*}}}$ defined as ${\displaystyle \tau (w)=w':(w,w')\in U}$.