Statistical machine translation

The idea behind statistical machine translation comes from information theory. A document is translated according to the probability distribution ${\displaystyle p(e|f)}$ that a string ${\displaystyle e}$ in the target language (for example, English) is the translation of a string ${\displaystyle f}$ in the source language (for example, French).

The problem of modeling the probability distribution ${\displaystyle p(e|f)}$ has been approached in a number of ways. One approach which lends itself well to computer implementation is to apply Bayes Theorem, that is ${\displaystyle p(e|f)\propto p(f|e)p(e)}$, where the translation model ${\displaystyle p(f|e)}$ is the probability that the source string is the translation of the target string, and the language model ${\displaystyle p(e)}$ is the probability of seeing that target language string. This decomposition is attractive as it splits the problem into two subproblems. Finding the best translation ${\displaystyle {\tilde {e}}}$ is done by picking up the one that gives the highest probability:

${\displaystyle {\tilde {e}}=arg\max _{e\in e^{*}}p(e|f)=arg\max _{e\in e^{*}}p(f|e)p(e)}$.

For a rigorous implementation of this one would have to perform an exhaustive search by going through all strings ${\displaystyle e^{*}}$ in the native language. Performing the search efficiently is the work of a machine translation decoder that uses the foreign string, heuristics and other methods to limit the search space and at the same time keeping acceptable quality. This trade-off between quality and time usage can also be found in speech recognition.

As the translation systems were not able to store all native strings and their translations, a document was typically translated sentence by sentence, but even this was not enough. Language models were typically approximated by smoothed n-gram models, and similar approaches have been applied to translation models, but there was additional complexity due to different sentence lengths and word orders in the languages.

The statistical translation models were initially word based (Models 1-5 from IBM Hidden Markov model from Stephan Vogel[5] and Model 6 from Franz-Joseph Och[6]), but significant advances were made with the introduction of phrase based models.[7] Later work incorporated syntax or quasi-syntactic structures.[8]

The most frequently cited benefits of statistical machine translation over rule-based approach were:

• More efficient use of human and data resources
  • There were many parallel corpora in machine-readable format and even more monolingual data.
  • Generally, SMT systems were not tailored to any specific pair of languages.
• More fluent translations owing to use of a language model

• Corpus creation can be costly.
• Specific errors are hard to predict and fix.
• Results may have superficial fluency that masks translation problems.[9]
• Statistical machine translation usually works less well for language pairs with significantly different word order.
• The benefits obtained for translation between Western European languages are not representative of results for other language pairs, owing to smaller training corpora and greater grammatical differences.

In phrase-based translation, the aim was to reduce the restrictions of word-based translation by translating whole sequences of words, where the lengths may differ. The sequences of words were called blocks or phrases, however, typically they were not linguistic phrases, but phrasemes that were found using statistical methods from corpora. It has been shown that restricting the phrases to linguistic phrases (syntactically motivated groups of words, see syntactic categories) decreased the quality of translation.[10]

The chosen phrases were further mapped one-to-one based on a phrase translation table, and could be reordered. This table could be learnt based on word-alignment, or directly from a parallel corpus. The second model was trained using the expectation maximization algorithm, similarly to the word-based IBM model.[11]

Syntax-based translation was based on the idea of translating syntactic units, rather than single words or strings of words (as in phrase-based MT), i.e. (partial) parse trees of sentences/utterances.[12] Until the 1990s, with advent of strong stochastic parsers, the statistical counterpart of the old idea of syntax-based translation did not take off. Examples of this approach included DOP-based MT and later synchronous context-free grammars.

Hierarchical phrase-based translation combined the phrase-based and syntax-based approaches to translation. It used synchronous context-free grammar rules, but the grammars could be constructed by an extension of methods for phrase-based translation without reference to linguistically motivated syntactic constituents. This idea was first introduced in Chiang's Hiero system (2005).[8]

Problems that statistical machine translation did not solve included:

• Annotated list of statistical natural language processing resources — Includes links to freely available statistical machine translation software